Apache Airflow Tutorial for Data Engineers
ฝัง
- เผยแพร่เมื่อ 26 ก.ย. 2024
- Chapters:
2:57 Pre-requisites
4:15 Airflow UI tour
11:04 Write your first DAG (that copies data files from AWS S3 to a Postgres DB)
23:16 Write your own Custom Airflow Operator (using Python inheritance)
30:08 TriggerDagRun Operator (Learn how to trigger one DAG from another)
33:06 BranchPythonOperator (Learn how to create dynamic workflows)
37:54 Set up DAG failure Slack alerts and handle errors in Airflow
47:03 CICD pipeline for Airflow (Automatic code deployment from GitHub to AWS Airflow)
All code in my Git repository github.com/sou...
TH-cam API python code inspired by this video - • TH-cam API for Python...
Thank you for creating a very crisp video on Airflow!
Glad you liked it!
the coffee making intro was unnecessary
🤣 but cool dude 🙌
Haha agree, totally unnecessary. I was so obsessed with my coffee machine when I made this.
And thank you!
This is an amazing video over apache airflow. I am waiting for some more amazing contents around data engineering. Thank you so much for the informative and engaging tutorial.❤
Thank you Chandan for such a thoughtful comment. I will try my best. Glad you liked it!
Thank you. This is definitely what I need. Better than other videos.
You are very welcome!
If possible could you please create a Kafka or Cassandra video tutorial next and Thanks for this wonderful Airflow tut.
You are very welcome! And I will try to make something similar. Thank you for the input!
Best part you have linked your GitHub repo in description, can you try to add chapters to the video.
Thank you the suggestion Diwakar. I will add chapters. Usually YT does it automatically. Not sure why it did not do it for this video
This video was amazing. I am a beginner and i learnt a lot many things.
But where i struggle most is setting up this framework of docker and airflow on my local. Can you help with a tutorial on this.
Hey! I already made a video last year for this th-cam.com/video/GmN1emEe4Jo/w-d-xo.html
If you have any particular question, please feel free to message me on instagram @techtalksourav
Awesome tutorial!!
Thank you!
Very useful tutorial 👌🏽
Thank you 🙏
So helpful, so cool!
Glad it was helpful!
Please make such more tutorials on Data Engineering tools.
Thanks, I will try my best!
@@souravtechie 😆 Waiting for the next tutorial!!
Haha hey Manish! Please feel free to let me know what tutorial you are interested in
@@souravtechie DBT or maybe on the concept of Delta Lake (Databricks). And a project on Airflow. Anything will work. Your way of explaining gives me a deep understanding and it's an awesome experience to watch your tutorials.
please improve your mic quality
Thank you for your patience. I will fix the audio in the upcoming videos
Awesome!
Kafka is next!
It very well could be brother as soon as I have a moment
do a series on kafka and snowflake also please
I was actually thinking of doing a Snowflake video. Thank you for the suggestion Sarthak. Really appreciate it!
The content is really good but I recommend you to enunciate words more clearly as it was difficult to follow.
Thank you!
😅 Still waiting for a full Airflow course with a project.
Hey Manish! Honestly, I'm not planning to make any more deep dive Airflow videos. However, I'm happy to answer any specific questions if you have any on Instagram or LinkedIn. Or any other Data Engineering tool 🙂
nice!!!!! thank you
would be nice if setting Airflow up actually worked.....
I did not talk about setting up Airflow in this video at all. However, I do have another video about running Airflow on a Mac that you may find useful.
Thank you so much Sourav for all your great efforts 👍
You're very welcome Zavier 🙂