Thanks for your sharing your thoughts. If possible, Do you mind sharing your thoughts on data governance , data quality etc . Including policies and enforcing techniques.
AWS Managed Airflow is very easy to use. You just need to understand how to write DAGs and chain the Tasks and off you go. Same with Astronomer. Both are much easier than Databricks IMO. What Databricks did for EMR was less than the huge overhead that AWS did for Managed Airflow as it's a much more difficult setup to set up Airflow. I would not do an unmanaged Airflow service, but I would have no hesitations if someone wanted to build your own Apache Spark or use EMR (or Synapse).
Thanks. Had to Google Argo. :-) The question comes down to do you really need it?, is it the best solution? I would recommend being skeptical of Argo initially. Do you use it?
![[UNCUT]“ฅนตื่นธรรม”ปะทะ“หมอปลา-ปู่มหามุนี-อั๋นโอกิ”สะเทือนกระทรวงเวทมนตร์ Iคนดังนั่งเคลียร์I10 ตค.67](http://i.ytimg.com/vi/peMD_SAkwNc/mqdefault.jpg)
Great video, thanks Bryan. I would add Prefect as Airflow competitor. It even has a paragraph on "Why not Airflow". Pretty easy to start with.
This is an excellent video, not just in the data engineering space...but in any IT engineering field....everything is relatable on spot on.
Thanks for your sharing your thoughts. If possible, Do you mind sharing your thoughts on data governance , data quality etc . Including policies and enforcing techniques.
Temporal seems to be a workflow solution that could be an airflow. Still trying to evaluate it and these tips helped.
AWS Managed Airflow is very easy to use. You just need to understand how to write DAGs and chain the Tasks and off you go. Same with Astronomer. Both are much easier than Databricks IMO. What Databricks did for EMR was less than the huge overhead that AWS did for Managed Airflow as it's a much more difficult setup to set up Airflow. I would not do an unmanaged Airflow service, but I would have no hesitations if someone wanted to build your own Apache Spark or use EMR (or Synapse).
These solutions are too expensive for people starting on small budget
i love this man
Great advice. Don't get carried away by new tech, focus on core tech like spark, python, sql What's your take on Argo.
Thanks. Had to Google Argo. :-) The question comes down to do you really need it?, is it the best solution? I would recommend being skeptical of Argo initially. Do you use it?
What job are these techs most fit for? Data engineer? Data Scientist? Or Data Analyst?
Did you watch the video?
Great content
Thank you!