"Airflow is hard" seems strange to me as a misconception :) I found it very easy when I started with Airflow a few years ago, and it was one of my selling points to implement it in my company at the time
It is hard for people used to GUI based orchestration tools - like Informatica Power Center , SSIS etc. The truth is that a lot of developers/analysts involved in data processing are not proficient in Python and very often in programming as well. In Airflow you have to like to code and and have proper mindest to appreciate it.
Just to clarify, so the modern way to write dags is with [@dags][@task] decorator. And the old way is [with DAG() as dag:] and [task_a]. I,ve taken your course. A great as it is I still get confused because there seems to be multiple ways to write dags and tasks.
Hi Kevin, Correct, the taskflow api with the decorators gives a much easier way to author your DAGs. However, you can still use the with dag syntax. It’s a matter of preference 😉
Is datasets time aware? If I load a table in Redshift on 2023-01-01 and 2023-01-02, will the downstream tasks that depend on these datasets start on those given days? Will the downstream run on 2023-01-02 wait for the 2023-01-02 dataset or it will run if the upstream run only ran for 2023-01-01.
This was a "what's new in airflow" but worded as misconceptions. Got me to watch through and learn nothing new, sadly. Well produced, all in all, but a little misleading.
I’m actually happy that you didn’t learn anything new. That means you are fully aware of what Airflow is today and I’m glad to know that 🙏 if there is any topic you would like to see, please let me know
This video has definitely given a better understanding of Airflow and now have some ideas on how to use it more effectively for projects.
What a wonderful thing to hear! Happy to help
Thanks for the video! I have not used Airflow for some years and it is great to see the evolution :)
"Airflow is hard" seems strange to me as a misconception :)
I found it very easy when I started with Airflow a few years ago, and it was one of my selling points to implement it in my company at the time
Agree, but some people think it is 🥹
It is hard for people used to GUI based orchestration tools - like Informatica Power Center , SSIS etc. The truth is that a lot of developers/analysts involved in data processing are not proficient in Python and very often in programming as well. In Airflow you have to like to code and and have proper mindest to appreciate it.
Just to clarify, so the modern way to write dags is with [@dags][@task] decorator. And the old way is [with DAG() as dag:] and [task_a]. I,ve taken your course. A great as it is I still get confused because there seems to be multiple ways to write dags and tasks.
Hi Kevin,
Correct, the taskflow api with the decorators gives a much easier way to author your DAGs. However, you can still use the with dag syntax. It’s a matter of preference 😉
We can't use task api for bash or sql query execute operators. We are forced to use xcom in those cases.
Is datasets time aware? If I load a table in Redshift on 2023-01-01 and 2023-01-02, will the downstream tasks that depend on these datasets start on those given days? Will the downstream run on 2023-01-02 wait for the 2023-01-02 dataset or it will run if the upstream run only ran for 2023-01-01.
Limitations of airflow
This was a "what's new in airflow" but worded as misconceptions.
Got me to watch through and learn nothing new, sadly.
Well produced, all in all, but a little misleading.
I’m actually happy that you didn’t learn anything new. That means you are fully aware of what Airflow is today and I’m glad to know that 🙏 if there is any topic you would like to see, please let me know