00:05 A data engineer builds data pipelines to extract, transform, and load data onto a target location. 01:50 Apache Airflow is a viral and widely adopted open-source project for managing data pipelines. 03:33 A workflow in Apache Airflow is called a DAG (Directed Acyclic Graph). 05:10 Apache Airflow has different operators and executors which make task creation and execution easier. 06:46 DAG with Dummy Operator in Airflow 08:12 Apache Airflow allows you to manage DAGs and track their statuses in one place 09:40 Airflow allows you to create complex pipelines and execute Python functions 11:16 Airflow overview and recommended project
4:11 - In the context of DAG, “graph” is a data structure consisting of nodes & edges. It’s a mathematical field in itself. In the physical world, graphs can be seen as “networks” such as social network or transportation network. It’s should not be confused with graphs used for visualization such as bar graphs or line graphs.
Wow! The amount of value you bring to the viewer in just 12 minutes is just incredible! Like you were saying "the hassle of managing pipelines in python scripts", I made a data project a while back, and I really can understand what you mean. I heard that Airflow makes everything easier, but I kept deferring Airflow learning, partly because "maybe this is good for large data pipelines, but my one consists of just a few small tasks," and also because most TH-cam videos of Airflow out there was a couple of hours long. And that's great if you are already committed to learn Airflow, but I was really looking for a video like yours - something that will give me a practical demo of how Airflow actually works, but it will be around 10-15 minutes. And this video was just perfect. Once again, thanks a whole lot! I'll definitely be integrating Airflow into my project now! And my resume too, God willing!
Basically, we can also create a python script and do the same work, the only thing which airflow is helping developers is that we don't have to track these scripts individually, here we are getting one portal kind of thing which manages and it also has different functions which can track individual script and if something fails we can track it.
Hey Darshil, I find your videos really helpful both conceptually and practically. I have been able to explain these concepts in my interviews and crack them. Keep on doing your amazing work with such explanatory videos
All of the videos I have seen till now from your channel has never confused me while learning new things. Thank you for explaining the concepts with simplicity and with examples. This gives us learners to understand the concepts quickly. Wish you the best. :)
I like these kind of small videos to get to know about a new thing. If we want to master it, then we can go for the full course or other videos. Thank you!
Hi Drashil, literally I understood about airflow, and learnt almost all topics in a high level,you explained it easier and simple way.Other videos having lengthy contents ..is very difficult to understand.Thanks...looking forward to more videos from you
Please make thesr type of videos on Snowflake and Apache Kafka. This would be very useful. You can explain the topic in simple way so that a learner can understand it easily. 👌👌👌
will you be uploading similar kinds of videos for Jenkins and Kafka as well? I feel there is always confusion between airflow, Jenkins, and Kafka and what a new data engineer should be learning when starting.
Thank you for making this video Darshil. I just watched the snowflake video and then couldn’t, resist to watch this one. Keep making such awesome content.
Can you please make a crash course of Apache Airflow in which following topics are discussed. Installation (Using Docker, EC2 and PIP) Multiple DAG creation Executors Different configurations like using airflow along with kafka how we can use airflow on EC2.
Please don't use the retro distorted video filter for the slides as it is very distracting. And the background instrumental music is highly distracting too. Otherwise, it is an extremely informative video. Thank you!
Hey darshil, very informative and detailed video, thanks a lot for this one.. 😊 have a request though, please upload more such videos consisting of small E2E projects on other apache tools.
Video content is so helpful. Than You Which AI platform or video editor are you using to create this video? Also for background music which platform are you using?
Thank you for your fruitful content and I have smashed the like button of the video. May I ask why you can directly save the csv to s3 without using boto3?
@@DarshilParmar Bro you are a Data Engineer Guru. Thanks for making life simple. Bro can you create a video how we get a full time role as a Data Engineer permanent Remote. You are doing freelancing but can you make how we get remote jobs and earn in USD. Because on TH-cam everyone is just faking and what they mentioned are for highly skilled people
Hi darshil, My coding skills are not so much good and currently i am working as data engieer and i am not using any cloud platform. Could please tell me which cloud platforms is best where we have less coding for pipline,transformation and deployment . So i can start learning for that cloud platform like azure ,gcp, aws
Hi, Cloud platforms has nothing to do with amount of coding you will be doing, as most of them are similar at fundamental level, it totally depends on the tool you are working on. If you are working with tools like Alteryx then it is mainly drag and drop but Airflow, Writing Spark script all of them requires coding If your coding is not good then I suggest start working on it as it is important skillset to have
@@DarshilParmarI am a beginner at learning Python and data engineering fundamentals in general with hopes of being one in the near future. I was tempted to see how these tools work and to be honest, I am a bit intimidated by the coding and complexity of it all. I pray I am able to learn it all.
Hey Drashil, As you said Twitter API is not valid anymore. Now which API we can use for Twitter Data Pipeline project? How to use another API like tweepy? Could you please explain about this?
Hi @DarshilParmar, I have gone through couple of your video and posts on youtube and linkedin. All are very helpful for me. I need your help to resolve my one problem as below: I need to create aws quicksight dashboard using my developer team designed ui template. I didnt not any option to use custom Ui option in qucksight? Please help me.
100k when? 😭 Hit that LIKE button, Subscribe if you are new and Comment something for better reach 🥹
Soon❤
All the best ❤for 1m subs
Can I get opportunity if i know only Airflow???
00:05 A data engineer builds data pipelines to extract, transform, and load data onto a target location.
01:50 Apache Airflow is a viral and widely adopted open-source project for managing data pipelines.
03:33 A workflow in Apache Airflow is called a DAG (Directed Acyclic Graph).
05:10 Apache Airflow has different operators and executors which make task creation and execution easier.
06:46 DAG with Dummy Operator in Airflow
08:12 Apache Airflow allows you to manage DAGs and track their statuses in one place
09:40 Airflow allows you to create complex pipelines and execute Python functions
11:16 Airflow overview and recommended project
This 10 mins video is much better than lengthy but hollow videos of hours. Thanks bro
4:11 - In the context of DAG, “graph” is a data structure consisting of nodes & edges. It’s a mathematical field in itself. In the physical world, graphs can be seen as “networks” such as social network or transportation network.
It’s should not be confused with graphs used for visualization such as bar graphs or line graphs.
Hi Darshil , please make these type of videos for Hive , cassandra , kafka ,data bricks ,snowflake,snowpark,mongodb
Yes, I am planning to do the same, already did it for Spark and Airflow (now), I will do it for DataBricks and others too
@@DarshilParmardo for kafka using azure services as well
Wow! The amount of value you bring to the viewer in just 12 minutes is just incredible! Like you were saying "the hassle of managing pipelines in python scripts", I made a data project a while back, and I really can understand what you mean. I heard that Airflow makes everything easier, but I kept deferring Airflow learning, partly because "maybe this is good for large data pipelines, but my one consists of just a few small tasks," and also because most TH-cam videos of Airflow out there was a couple of hours long. And that's great if you are already committed to learn Airflow, but I was really looking for a video like yours - something that will give me a practical demo of how Airflow actually works, but it will be around 10-15 minutes. And this video was just perfect. Once again, thanks a whole lot! I'll definitely be integrating Airflow into my project now! And my resume too, God willing!
Basically, we can also create a python script and do the same work, the only thing which airflow is helping developers is that we don't have to track these scripts individually, here we are getting one portal kind of thing which manages and it also has different functions which can track individual script and if something fails we can track it.
Got clear understanding of basics of Apache Airflow.. Very helpful
Hey Darshil, I find your videos really helpful both conceptually and practically. I have been able to explain these concepts in my interviews and crack them. Keep on doing your amazing work with such explanatory videos
Thank you Suraj
Definitely best overview of Airflow till date
Thank you!
Can you share he the emoji bro
All of the videos I have seen till now from your channel has never confused me while learning new things. Thank you for explaining the concepts with simplicity and with examples. This gives us learners to understand the concepts quickly. Wish you the best. :)
I like these kind of small videos to get to know about a new thing. If we want to master it, then we can go for the full course or other videos. Thank you!
Love your reason you start thís TH-cam channel : "Simplify everything"
please dont apply background music
Super useful, really point to point explanation. No non-sense in between. Thumbs up for your work 👍
Hi Drashil, literally I understood about airflow, and learnt almost all topics in a high level,you explained it easier and simple way.Other videos having lengthy contents ..is very difficult to understand.Thanks...looking forward to more videos from you
Please make thesr type of videos on Snowflake and Apache Kafka. This would be very useful. You can explain the topic in simple way so that a learner can understand it easily. 👌👌👌
Thank you for the clear and concise video. It's greatly appreciated!
Thank you for all the help brother... I have started learning from your videos..🙏🏻
Wonderful video! Concise but explained in a way that it's digestible for anyone, thank you!
One of the best video. Lots of things in just 10 min.
will you be uploading similar kinds of videos for Jenkins and Kafka as well? I feel there is always confusion between airflow, Jenkins, and Kafka and what a new data engineer should be learning when starting.
Just Smart video, to the point, no unnecessary content. thanks a lot
Super useful video. Best beginner video available on this topic.
I'll give this a try. Thank you so much bro.
@darshil what’s the difference between control m and this air flow.?
Thank you for making this video Darshil. I just watched the snowflake video and then couldn’t, resist to watch this one. Keep making such awesome content.
Thank you so much 😀
Dude, your video is awesome, thank you for share your knowledge :D
Really clear and interesting explanation. Thank you
Hi Mr Parmar,
Thank you for this short introduction to Airflow.
You are most welcome
Excelent explanation!, very insightful and useful. Thanxs
Where can we install airflow? Either on cloud or on-premises
You can do it on both
Excellent explanation Darshil! Keep it up
Great video, thank you..
How it's different from SSIS or Control-M ?
On what compute service the python code runs?
Great job explaining in just 10mnts. Thanks!
Does your data engineering class on your website teach airflow? Does it teach how to debug python code too?
what's difference between Airflow and Autosys? both are schedulers?
Airflow is better in feature but they are quite similar
Can you please make a crash course of Apache Airflow in which following topics are discussed.
Installation (Using Docker, EC2 and PIP)
Multiple DAG creation
Executors
Different configurations like using airflow along with kafka
how we can use airflow on EC2.
Please don't use the retro distorted video filter for the slides as it is very distracting. And the background instrumental music is highly distracting too. Otherwise, it is an extremely informative video. Thank you!
Thanks buddy, very useful 🙌🏻
Hello Darshil!!
Your content is very good.
Please start pyspark tutorial.
It will be very helpful for me.
Thank you, I am working on it
Thanks for this video.. very informative and educative.. thanks again
Great content. One suggestion- Remove/tone down the background music. Had trouble paying attention to your voice.
Hey darshil, very informative and detailed video, thanks a lot for this one.. 😊
have a request though, please upload more such videos consisting of small E2E projects on other apache tools.
Working on it
Is this similar to AWS step functions?
Yes, you can say that!
Do We have to code in Apache airflow
Yes
Amazing Explanation!!!! :)
Brother, This is Maruthi… good work. I subscribed and referred my friend also. Looking forward to Master class on Fundamentals.
Sure
Really helpful.keep it up good work.
Realy its a nice explanation
Thanks. Quick and clear 👍
Hi Darshil, are you also proving course on Azure for data engineering?
Very informative video Darshil, btw which font style do you use?
It is very helpful, learned a lot.
Thanks for the video🙂🙂🙂
Thank you Harsh
Airflow is really tricky for me, got problems calling functions in my dag from other files that are not in airflow folder structure ...
Thank you so much for this video
Very helpful . Thank you
when we have ADF and aws glue, is airflow still make sense in market ?
Yes, OpenSource
Great explanation!
Video content is so helpful.
Than You
Which AI platform or video editor are you using to create this video? Also for background music which platform are you using?
I have editor, they use Premier pro, bg music can be found from youtube music lib or google
Awesome!
Thank you for your fruitful content and I have smashed the like button of the video. May I ask why you can directly save the csv to s3 without using boto3?
Really crisp explanation
Glad you liked it
@@DarshilParmar Bro you are a Data Engineer Guru. Thanks for making life simple. Bro can you create a video how we get a full time role as a Data Engineer permanent Remote. You are doing freelancing but can you make how we get remote jobs and earn in USD. Because on TH-cam everyone is just faking and what they mentioned are for highly skilled people
Hi darshil,
My coding skills are not so much good and currently i am working as data engieer and i am not using any cloud platform. Could please tell me which cloud platforms is best where we have less coding for pipline,transformation and deployment . So i can start learning for that cloud platform like azure ,gcp, aws
Hi,
Cloud platforms has nothing to do with amount of coding you will be doing, as most of them are similar at fundamental level, it totally depends on the tool you are working on.
If you are working with tools like Alteryx then it is mainly drag and drop but Airflow, Writing Spark script all of them requires coding
If your coding is not good then I suggest start working on it as it is important skillset to have
@@DarshilParmarI am a beginner at learning Python and data engineering fundamentals in general with hopes of being one in the near future.
I was tempted to see how these tools work and to be honest, I am a bit intimidated by the coding and complexity of it all. I pray I am able to learn it all.
Bro can i get trained in jspiders as a full stack developer is it worth . Does its certificate have value
can you explain ICEDQ???
Amazing stuff
thank you so much ♥ ♥
Hey Drashil, As you said Twitter API is not valid anymore.
Now which API we can use for Twitter Data Pipeline project? How to use another API like tweepy? Could you please explain about this?
TH-cam, Maps, Spotify, Crypto there are so many just search about it
Hi @DarshilParmar, I have gone through couple of your video and posts on youtube and linkedin. All are very helpful for me. I need your help to resolve my one problem as below: I need to create aws quicksight dashboard using my developer team designed ui template. I didnt not any option to use custom Ui option in qucksight? Please help me.
very nice video bro. keep it up
Nice Video!!!
Glad you liked it
Jai Jinendra 🙏
You must talk about internals and workings too.
hello @DarshilParmar can you make video about how to install apache airflow on windows
100s of resources are already there brother, just search about it
does anyone get problem with 'unexpected wsl error' with docker and done fix it ? pls help me
Nice explanation
Hi darshil, can you please make a tutorial on pyspark
Please make an intro course for Apache using Windows
can airflow automate javascript
You can host js script somewhere in sever and call from airflow if needed
Hi, can you release a building project using Airflow? that would be very handy, thank you.
There is already one project on airflow
Bring on end-end course on airflow with projects @Darshil
Working on it
Many Thanks. Remove background music
Sorry for that
Thanks bro
how to install Apache airflow???
There is a complete project linked at the end of the video
Great video! I hope you can make a comparison between airflow, prefect and dagster
Great, I will put this in my list
good video
Please create video on Mage AI.
Bhai yeh Airflow kya bala hai ab? 😂 Alright. Thanks man. 👍🏻
Good video but music is a bit distracting
Thank you, will keep this in mind
Hey, this video is not for beginners. If you're just starting out, I suggest you skip this one. It might confuse you and make you feel discouraged.
You got boggled dint you?
dude are you still at school 😂 brilliant knowledge though 💯
Found a Goldmine!
Bro started the video with a little beard and handed the video with the more beard
xd
Dude this is worth 10min.
No need of background music. It is irritating
Will keep this in mind
One of the poorest scheduler , airflow
Uc4 is better
@@immortal348 there is nothing better than control m, best scheduler
pronunciation😁