It is messy to directly compare feature engineering (FE) to the transform step in ETL; they exist on different levels of abstraction. A "traditional" ETL pipeline looks more like ETTTTL in practice because data is piped between multiple tables before it ends up in something like a dashboard or ML model. In the non-ML use case, we design that last "T" by consulting analysts and stakeholders to understand what subset of the data they need at which cadence for reporting/tracking/etc. In the ML use case, we design the last "T" using feature engineering to match the data to the requirements of the algorithm / ML model we are using.
I found this video a bit abstract. Does anyone know a good comparison between Airflow and Kubeflow (or TFX)? That may help provide concrete examples of Data Engineering vs ML pipelines
It is messy to directly compare feature engineering (FE) to the transform step in ETL; they exist on different levels of abstraction. A "traditional" ETL pipeline looks more like ETTTTL in practice because data is piped between multiple tables before it ends up in something like a dashboard or ML model. In the non-ML use case, we design that last "T" by consulting analysts and stakeholders to understand what subset of the data they need at which cadence for reporting/tracking/etc. In the ML use case, we design the last "T" using feature engineering to match the data to the requirements of the algorithm / ML model we are using.
Is it possible to become both a backend developer and a data engineer at the same time
Its possible, but I guess I wonder what your goals are
@@SeattleDataGuyi am doing Masters in CS ... is SWE - DE - MLE a good path or there are other quick stepping stones role to MLE?
I found this video a bit abstract. Does anyone know a good comparison between Airflow and Kubeflow (or TFX)? That may help provide concrete examples of Data Engineering vs ML pipelines
Oh that's actually a good topic, Airflow vs Kubeflow!
I can tell all of that definitions you’ve put up there in the video is generated by chatgpt 😅
Hmm, I am actually not sure. I'll ask Sarah though!
What is your opinion on Godfather of ai statement after resigned Google