Why Airflow? The Top 5 Reasons To Use It!

Why Airflow? The 5 reasons why Apache Airflow is for you!
🏆 BECOME A PRO WITH AIRFLOW: www.udemy.com/course/the-comp...
👍 Smash the like button to become an Airflow Super Hero!
❤️ Subscribe to my channel to become a master of Airflow
🚨 My Patreon: / marclamberti
Curious about Apache Airflow? 🚀
In this video, we're breaking down the top 5 reasons why Airflow should be your ultimate choice for orchestrating your data workflows.
From efficient orchestration and dynamic workflows to seamless integrations, robust monitoring, and a thriving community, discover how Airflow empowers you to master the art of data orchestration with reliability, scalability, and customization.
#ApacheAirflow #DataOrchestration #DataEngineering
00:00 Introduction
00:20 The data ecosystem
00:53 A typical data pipeline
01:24 The first reason
03:57 The second reason
05:10 The third reason
06:58 The fourth reason
08:32 The fifth reason

[@MarcLamberti]

If you want start learning Airflow today: www.udemy.com/course/the-complete-hands-on-course-to-master-apache-airflow/

[@arjunanand1]

Hi Marc, great video yet again! You mention around 3:35 that you can have multiple replicas of the metadata DB, I thought this was not possible by design. How did you set that up?

[@MarcLamberti]

Airflow needs a database to work and there is nothing that prevents you from making your database highly available. If you use postgres, you can look at the following doc: www.postgresql.org/docs/current/high-availability.html
To be clear, it's not an Airflow thing, it's db thing and it depends on the database you use :)

[@oshebase]

Hi Marc, I love the way you make Airflow interesting to jump at for a beginner like myself. Am completely new to Airflow world, as a matter of fact, your video on online retail is my first bite. But unfortunately, i got choked 😔 along the line 😭 tried to figure things out but couldn't. The task the error was coming from was in the task that upload gcs to bigqery, this task failed with error decodedencoded utf-8. But when i check BigQuery the raw_invoices table was created but no dataset 😢 wish i could fix this or find way around this. Could you as well be kindly make a video on dynamic task maping as mentioned in this video. Maybe from API to data warehouse. Thanks for the great and concise videos. 🎉

[@rozhindoroudi3514]

Hi Marc, I took your course last year, and I don't remember any discussion around dynamic workflow and data lineage. Are these new concepts? Do you have any videos about these concepts? Thanks in advance :)

[@MarcLamberti]

Great question. I’m planning to update the courses this year. Stay tuned 🥹

[@rozhindoroudi3514]

@@MarcLamberti looking forward to it :)

[@Shrew_Bucket]

what you think about prefect ?

[@MarcLamberti]

Didn’t try yet

[@sanishthomas2858]

to learn airflwo which all tools we haveto learn like python, sql?

[@MarcLamberti]

Python essentially