Seriously, awesome video as I prep for an interview tomorrow morning to refresh/learn about Airflow and Docker. Thank you! The world is so much better with people willing to give their time like you! THANKS
among all the airflow tutorials on youtube, this is THE ONLY ONE that I can follow and get the exact same result as my life-saver Soumilshah does. others just throw some random and weird errors which made me wanna die. Thank you, life-saver!!!!
Thank you very much! I really tried to find a video just with this Airflow basics and all others got errors. This is the first one that run nice and well.
sorry to be off topic but does anybody know a tool to log back into an Instagram account? I was dumb forgot the password. I would love any assistance you can give me.
@Noe Soren I really appreciate your reply. I got to the site on google and I'm trying it out now. Seems to take a while so I will get back to you later with my results.
Good day Soumil, I hope you saw and red my messages. Please can you do a scratch beginner tutorial on Data engineer and how to install all the necessary tools because in this video I'm completely lost. You didn't show how you installed all the tools you are using for data engineer (i.e. python, airflow, docker and it's container). Please it will be grateful if you do consider. Thanks ☺️
Hello, I am new at apache airflow. Your videos of airflow are awesome and helped me to understand it. I have a request to you, I don't know is it possible to use airflow for php/mysql simple application Cron task. If yes, then it would be a great help for us if you make a video on it step by step like other videos.
One suggestion to beat the almighty algorithm. Don't swear in the video. It's not that I have a problem with profanity, but TH-cam after it flags you, restricts regions and demotes you in ranking. :( Amazing tutorial
@soumil shah can we create Airflow Rest API using Nodejs as the backend if so can you please how I can config the env for that?Thanks for the great course:)
Hi, I followed through the video carefully but I seem to be getting this delay from making my web server come up: "project-webserver-1 | Sun 06 Mar 2022 07:59:39 AM UTC - waiting for Postgres... 9/20 ". after a while, i get this: "postgres:5432 still not reachable, giving up".
I have a question, can you use trained machine learning model which create DAG, and integrate it with Airflow or spark, can you make a video on that for job scheduling part.
Nice presentation bro! I have a requirement where we have the schedule interval stored in a table and I need to schedule some python jobs based on the interval fetched from the table. Is it possible to pass the schedule_interval as parameter or is that need to be coded in DAG manually?
you can dynamically set the schedule interval for DAGs using the schedule_interval parameter, but it's typically set during DAG definition and remains static. However, for dynamic scheduling based on data from a table, you can create a DAG that reads the schedule intervals from the table at runtime, using an operator like PythonOperator. Within this operator, fetch the schedule intervals from the table, calculate the next execution time, and dynamically generate DAG runs accordingly. This approach allows for flexible scheduling based on your table data.
Hey @soumil , i tried running the script..I am getting import error -ModuleNotFoundError: No module named 'termios'.. i did some googling and found that windows doesnt have termios. I am getting this error while importing python operator. Did you get this error, if yes let me know how you solved this.
Hello sir! How to stop triggering next schedule if previous fail, using wait_for_downstream next schedule is running even though task is not started but i need not to trigger the next schedule at all How to do that sir plz help me.
i have tried in the same way but i think we did n't mentioned any role but getting Error " webserver_1 | psycopg2.OperationalError: FATAL: role "airflow" does not exist"
do we have to repeat the steps everytime we are creating a different project? it seems like we are installing Airflow again and again. Do we have any solution for that?
Hi Soumil, Thanks for the knowledge sharing. I'm facing an issue while running "docker-compose up --build" command. Help me to fix this error. Below is the full error. ERROR: for webserver Cannot start service webserver: failed to create shim task: OCI runtime create failed: runc creat e failed: unable to start container process: error during container init: error mounting "/run/desktop/mnt/host/c/Ranji th/Learn_Airflow/project/config/airflow.cfg" to rootfs at "/usr/local/airflow/airflow.cfg": mount /run/desktop/mnt/host /c/Ranjith/Learn_Airflow/project/config/airflow.cfg:/usr/local/airflow/airflow.cfg (via /proc/self/fd/14), flags: 0x500 0: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified ho st path exists and is the expected type ERROR: Encountered errors while bringing up the project.
Do we need to create multiple python files to create multiple dags. For ex: I've 10 jobs to execute at different time intervals. I want to create 10 DAG's. How to do that ?
Hello brother, I try follow your step, but it show error below: (Lerning_Airflow) D:\Mala\Project\Lerning_Airflow\project>docker-compose up --build ERROR: The Compose file '.\docker-compose.yml' is invalid because: services.webserver.volumes 'type' is a required property Could you help to solve it?
how to access csv file from my local folder in windows 10? I am okay running bash operator and calling the python script to run but I am having issue accessing local file it gives me error FileNotFoundError: [Errno 2] airflow. Thank you for your help.
Hey soumil. at 17:43 in the video. I'm getting 'docker-compose' is not recognized as an internal or external command, operable program or batch file. How to fix this in windows 10. I tried installing docker in windows but no use. Please help me with this.
Seriously, awesome video as I prep for an interview tomorrow morning to refresh/learn about Airflow and Docker. Thank you! The world is so much better with people willing to give their time like you! THANKS
Hands down the best video of Airflow and Docker on TH-cam.
among all the airflow tutorials on youtube, this is THE ONLY ONE that I can follow and get the exact same result as my life-saver Soumilshah does. others just throw some random and weird errors which made me wanna die. Thank you, life-saver!!!!
Man, you can't even imagine how helpful for me was your video, thank you
Thank you very much! I really tried to find a video just with this Airflow basics and all others got errors. This is the first one that run nice and well.
04:56 - Web UI overview
12:30 - Install Airflow
15:47 - docker-compose.yml
20:47 - build first_dag
27:58 - pass kwarg parameter to the executor
30:36 - second function executor
32:38 - xcom_push
Thanks man I will add this in comment sections
Which software he is using to write docker yml code
This video really helps me to understand the basics of airflow very well , thanks
Really helpful for those who want to learn airflow.
Excellent one, Thanks Soumil for all your efforts. It's slightly advanced and it's a great place to have the initial understanding
Thank you Soumil, this video has saved me at least weeks of research!!!
It was really nice watching your video :) Good material, and good sense of humor too.
You sir are very good at explaining minor details. It helped me a lot, so thank you.
Great intro to this topic Soumil.
I did have an error with "start_date" in the DAG(). Solved it by move it to the second param spot.
It covers very basic but definitely a great one to start with. Thanks
Excellent video, I was struggling to setup airflow from their official site. This made it so easy
Wonderful tutorial video. Soumil is talking super fast, which saved us a lot of time learning. Usually it takes about 2 hours, I guess?
Excellent video and you made me take off fast in airflow. Appreciate your help friend
Thanks Soumil, this is really helpful to start with airflow.
Thanks a lot for your efforts and enthusiasm! Thanks to people like you community expands their horizons!
From piles of official docs, you just brought the crisp and clear material for kickstarting! liked and subscribed :)
Love it and would be eve nicer if you can add timeline to the video and have the command line commands in the description!
Greetings from Buenos Aires, Argentina, I really learned a lot. I´ll share your video.
This video was a great resource. Thanks for the tutelage and your take on it.
nice video bro, greetings from Brazil!
Perfect video to set up your environment and start your Airflow journey. Thanks Soumil, that was really helpful.
sorry to be off topic but does anybody know a tool to log back into an Instagram account?
I was dumb forgot the password. I would love any assistance you can give me.
@Erik Toby Instablaster :)
@Noe Soren I really appreciate your reply. I got to the site on google and I'm trying it out now.
Seems to take a while so I will get back to you later with my results.
@Noe Soren It did the trick and I now got access to my account again. I am so happy:D
Thank you so much, you really help me out :D
@Erik Toby happy to help xD
Good day Soumil,
I hope you saw and red my messages. Please can you do a scratch beginner tutorial on Data engineer and how to install all the necessary tools because in this video I'm completely lost. You didn't show how you installed all the tools you are using for data engineer (i.e. python, airflow, docker and it's container). Please it will be grateful if you do consider. Thanks ☺️
Hello, I am new at apache airflow. Your videos of airflow are awesome and helped me to understand it. I have a request to you, I don't know is it possible to use airflow for php/mysql simple application Cron task. If yes, then it would be a great help for us if you make a video on it step by step like other videos.
loved ur humility man, nice one
DAG means "Directed Acyclic Graph" that should have been told in the beginning of Airflow demo. I think you have missed it!
Nicely Compiled and presented Soumil !
Thanks, Soumil. It helps me a lot!
Thank you so much it helped me a lot 😃😃😃😃
One suggestion to beat the almighty algorithm. Don't swear in the video.
It's not that I have a problem with profanity, but TH-cam after it flags you, restricts regions and demotes you in ranking. :(
Amazing tutorial
Hi Thanks for good demo. I see you are running Airflow in Windows pc. do we need to install Docker Desktop before we start ?
many thanks for your videon it was very easy to follow:)
@soumil shah can we create Airflow Rest API using Nodejs as the backend if so can you please how I can config the env for that?Thanks for the great course:)
Can you please share a video or setup for using Apache Airflow with Databricks.
Man you hit that Enter HARD! :-D
Really great video, thanks!!😁
hello, merciii :)
is it possible to have the tasks in nodejs for example ??
That is verz helpful , thanks for your effort
Hi, I followed through the video carefully but I seem to be getting this delay from making my web server come up: "project-webserver-1 | Sun 06 Mar 2022 07:59:39 AM UTC - waiting for Postgres... 9/20
". after a while, i get this: "postgres:5432 still not reachable, giving up".
Helpful video!! Bookmarking it.
Great Video Soumil. Do you have any sql alchemy or Pyodbc DAG
Great video. Thank you!
yeah, this is a wonderful video and perfect
thank you
Hi Soumil,
I am working on Tidal Enterprise Scheduler
tool now, is Airflow is same as Tidal?
Nice video
I found it fruitful 👍
How can I import a local module (created with a setup.py) to use in a DAG?
I have a question, can you use trained machine learning model which create DAG, and integrate it with Airflow or spark, can you make a video on that for job scheduling part.
Nice presentation bro! I have a requirement where we have the schedule interval stored in a table and I need to schedule some python jobs based on the interval fetched from the table. Is it possible to pass the schedule_interval as parameter or is that need to be coded in DAG manually?
you can dynamically set the schedule interval for DAGs using the schedule_interval parameter, but it's typically set during DAG definition and remains static. However, for dynamic scheduling based on data from a table, you can create a DAG that reads the schedule intervals from the table at runtime, using an operator like PythonOperator. Within this operator, fetch the schedule intervals from the table, calculate the next execution time, and dynamically generate DAG runs accordingly. This approach allows for flexible scheduling based on your table data.
do you lot need to restart airflow just to add a dag?
I'm able to add Dags without restarting airflow takes a few second but it updates automatically
Thanks Soumil, do you know how to install libraries like pip install tensorflow inside airflow container?
Awesome tutorial!
Thank you for this video
Hey @soumil , i tried running the script..I am getting import error -ModuleNotFoundError: No module named 'termios'..
i did some googling and found that windows doesnt have termios. I am getting this error while importing python operator. Did you get this error, if yes let me know how you solved this.
Well done Man, awesome vedio.
For me, after creating the 3 docker containers, 8080 is listed as a port but is inaccessible. The other two ports work. Looking into it now.
sir please show some more code and also how to connacet to pyspark data frame
how to launch air flow?
what commands should I enter?
Thanks for the video!
hey, awesome video.
which coding studio do you use in this video ?
Pycharm
Thanks a lot bro
Hello sir!
How to stop triggering next schedule if previous fail, using wait_for_downstream next schedule is running even though task is not started but i need not to trigger the next schedule at all
How to do that sir plz help me.
Which software your using writing code “IJ”
Keep going, keep growing.
How to install apache airflow in windows please !!
Hello Soumil, lovely video!
how can we get data ?? i mean after doing preprocessing using pandas can we get data through airflow ??
i have tried in the same way but i think we did n't mentioned any role but getting Error " webserver_1 | psycopg2.OperationalError: FATAL: role "airflow" does not exist"
do we have to repeat the steps everytime we are creating a different project? it seems like we are installing Airflow again and again. Do we have any solution for that?
Hi, Can I installed Apache airflow using .tar.gz file on production server? Thanks for the video.
do I have to build the docker container every time i add new dag?
Nice video. Thank you
Amazing bro
love it
Hi Soumil, Thanks for the knowledge sharing. I'm facing an issue while running "docker-compose up --build" command. Help me to fix this error. Below is the full error.
ERROR: for webserver Cannot start service webserver: failed to create shim task: OCI runtime create failed: runc creat
e failed: unable to start container process: error during container init: error mounting "/run/desktop/mnt/host/c/Ranji
th/Learn_Airflow/project/config/airflow.cfg" to rootfs at "/usr/local/airflow/airflow.cfg": mount /run/desktop/mnt/host
/c/Ranjith/Learn_Airflow/project/config/airflow.cfg:/usr/local/airflow/airflow.cfg (via /proc/self/fd/14), flags: 0x500
0: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified ho
st path exists and is the expected type
ERROR: Encountered errors while bringing up the project.
Hi soumil, Is there way we can store the resulset of postgres read from pghook conmection to txt file and upload it to s3 bucket
Do we need to create multiple python files to create multiple dags. For ex: I've 10 jobs to execute at different time intervals. I want to create 10 DAG's. How to do that ?
Hello brother, I try follow your step, but it show error below:
(Lerning_Airflow) D:\Mala\Project\Lerning_Airflow\project>docker-compose up --build
ERROR: The Compose file '.\docker-compose.yml' is invalid because:
services.webserver.volumes 'type' is a required property
Could you help to solve it?
Hi Soumil,
where df.to_csv() will be stored ?
Excellent One.
damn. Straight to the point.
Yup I don’t like going round and round
Pls post your linkedin link and blog link in description
Awsome video ... Pace is good ...
Intro to Apache Airflow or you can call it Wind flow or Airpipe or Airpass it’s upto you really
how to access csv file from my local folder in windows 10?
I am okay running bash operator and calling the python script to run but I am having issue accessing local file it gives me error FileNotFoundError: [Errno 2] airflow. Thank you for your help.
is "ti" special reserved keyword? 34:23
Hi,
We installed postgres image and we didn't saved anything to DB
Does airflow store any dag information in db
Thank you
Error Couldn't find env file ...\LOAD_EX=n
excellent!
Hi sir can you help with without using docker which one will be helpful debugging application with windows ( VScode or pycharm)
Windows 10 + Ubantu sub system + local airflow server
Hi I'm facing the dag import error
Hey soumil. at 17:43 in the video. I'm getting 'docker-compose' is not recognized as an internal or external command,
operable program or batch file.
How to fix this in windows 10. I tried installing docker in windows but no use. Please help me with this.
is ur docker up and running in local ?. if not check ur wsl version,virtulisation seeting enabled.
yaml: line 25: did not find expected '-' indicator
plshelp
I am getting the cgitb error
Mam dowód że Google nadal używa Apacze 2.0 na Python
bro can you tell me how to debug this ?
Is anyone having issue while importing PythonOperator?
thanks friend! :)