ฝัง
- เผยแพร่เมื่อ 10 ก.ค. 2024
- Airflow DAG: create dag with bash operator!
#Airflow #AirflowTutorial #Coder2j
========== VIDEO CONTENT 📚 ==========
Today I am going to show you how to create our first dag using the bash operator! By watching this video, you will know:
👉 How to create a dag and set parameters as your need
👉 Define tasks using BashOperator
👉 How to build the task dependencies
Video Request: forms.gle/UMp4GA3krcSMMWzy9
You don't have a running Airflow in your local setup? Check out the Apache Airflow 2.0 Installation Tutorials.
Run Airflow locally 👉 • Airflow introduction a...
Run Airflow in Docker 👉 • Airflow Docker: run Ai...
========== L I N K S 🔗 ==========
GitHub Repo 👉 bit.ly/3HD5oTX
Airflow Documentation 👉 bit.ly/3wbTqv4
========== T I M E S T A M P ⏰ ==========
00:00 - Launch airflow using docker
01:08 - Remove all the airflow example dags
02:16 - Create our first dag and set the parameters
05:00 - Create dag tasks using bash operator
07:26 - Trigger our first dag run
08:05 - Define multiple tasks for the dag
10:15 - Three different ways to build tasks dependencies
========== Connect with me 👏 ==========
Twitter 👉 / coder2j
Website 👉 coder2j.com
GitHub 👉 github.com/coder2j
Thanks man, you're the best instructor in Airflow on TH-cam ever.
Looking forward to the next one!
Thanks. Stay tuned! 🙌
I really enjoyed this video. I learned more today than the past two weeks with another youtube video. Also I reinstalled Airflow with your instructions from another video and it has never been this stable. Thank you!
Glad it was helpful!
Amazing videos, very simply explained
Thankyou for this this tutorial.
I am following your instructions and have been able to understand Airflow at an intermediate level.
I would love to see your writing DAGs for PythonOperator with Xcoms. Eagerly waiting for it!
Nice to hear that. Python operator is already planned. Stay tuned! 😉
Same here 😊
Nice AirFlow video ever in you Tube, heartfelt Thanks
You are welcome!
Awsomee!!! Thank you for all this broo, its really helpful
Thanks. Have fun learning.
Thanks for posting far easy and clearly explained then paid material . Please keep postings if possible on airflow integeration with aws services such as red shift and databricks .
Thanks for the feedback ☺️
Thanks man.
OMG, you are my hero!
🤗🤗
this is the best video ever.
Thanks for your nice comment.
Thx
More please
how do i import a json config file that is dependent on another python script in airflow
Good stuff! Thanks for the tuto! I am running the dag which is status success but the task 1, 2, and 3 are blanks and if I select one task to run it independently, it tells me ''Only works with the Celery or Kubernetes executors, sorry'' any clues how to fix this? I should add some settings in the .yaml file?
If you want to rerun the dag or one of the task, you need to click the dag or task run, then click the clear button. Wait a second, the scheduler will schedule it and run it again. Check out the airflow tutorial part6, in which I also dodi the same.
Nicee .. please create one for passing one variable from 1st function to another function using these python operators
Thanks for the feedback. Will include that in the coming episode. Subscribe and Stay tuned! 😉
Always man 🙂.. would like to see you covering more stuff's like kafta flume spark and all
Please include branchpyhtonoperator
Hello) Cool videos! Thank you!
Why is the old version of DAG not rebooting, but a new one is created?
And how to remove old versions, leaving only the latter?
Because a new dag ID is created. You can simply delete the old dag on the UI.
大哥，用visual studio编辑DAG，然后在airflow portal refresh，我这里创建不了新的DAG，而且，也删除不了原有的31个DAG。怎么解决呢？我用的mac
Hi is it possible to get task/dag duration from airflow context
Yes, you can use the on_success_callback in operators. Check the documentation here: bit.ly/3uxpxER
Please include how we can install more python package?
Thanks for your great feedback. We, as data engineers/scientists need a lot of python packages. It definitely deserves an episode! Stay tuned! 😉
@@coder2j Also interested in this topic. XD Maybe dockerizing an application with all its dependencies, then run it using docker operator?
@@leamon9024 Will cover it in the future episode! Stay tuned! 😉
i am getting error as- Import "airflow" could not be resolved. possible solutions plz
Did you get this error from the airflow webserver or from local python lint? And are you running airflow locally or in docker?
@@coder2j I get this error in vscode whenever I write " from airflow import DAG". I am running using docker. I think it is not able to find airflow module which is installed in docker.
@Akshay That's normal. Because the airflow package is installed in the docker container. The vs code will complain it since there is no airflow package installed. One tip to bypass this error is to create a local python environment, install airflow in that and tell the vs code the path of the python environment you created. But you can ignore this as long as you can launch airflow webserver successfully!
when I try to import airflow, it says that "Import "airflow" could not be resolved". I looked up over the internet and it seems like the problem is my local python interpreter doesnt have airflow installed. How can I use python interpretor from the container instead in visual studio code so that I wont get this error?
You can install vscode remote Container Plugin and select python interpreter within the container.
Hi this has been really helpful. However the error in the logfiles I get when I run my dag is that says it doesn't have access to my python file - INFO - python: can't open file '. I am using the Bash Operator and the bash command is 'python fulldirectorypath/myscript.py' how can I fix this. I have tried all version of the writing the full path. How do I give airflow permission?
If you want to execute python functions, it is recommended to use PythonOperator. I covered that in the Airflow Tutorial Part6.
Sir the airflow webserver has already running but when i tried to import DAG from airflow it says no module named airflow, should i just pip install it? or did i miss something that make the airflow module not automatically installed?
Is the error coming from the IDE lint or runtime?
@@coder2j ide, but i alreadY just pip install apache-airflow in the terminal
Hi, really good videos. Shall I know how to dm you. I want to ask a doubt regarding sql sensor and hive operator
Please leave it directly in the comment section. So that we can share the info in case others may also have the same question.
Hi, am using bash operator and passing multiple commands inside bash_command.Its a mixture of if else loops and commands.I don't know how to correctly incorporate it.l am using" to separate it. Could you guide me here if you have worked with something similar please thanks in advance
Can you try wrapping the multiple lines command into 3 double quotes? Like """
"""
@@coder2j yes but I have nested if loops and inside one loop I'm executing a s3a command.So my qn is do I put all of these ifs and else's in one ''' ''' or after each if else?also do I need to use + with '''? Like- ''' echo''' + '''if else'''
Hi, you can save your bash commands in a .sh file under the dags folder, then just put the relative path of .sh file as the value of the bash_command parameter.
@@coder2j if I have to pass arguments which has macro values can I do so by-
Date=some macros #global variable
Bash_command='''
"File.sh Date "
'''
You can use Jinja template to insert parameters. airflow.apache.org/docs/apache-airflow/stable/howto/operator/bash.html
bash_task = BashOperator(
task_id="bash_task",
bash_command="echo \"here is the message: '$message'\"",
env={"message": '{{ dag_run.conf["message"] if dag_run else "" }}'},
)
i use BashOperator run bash_command='date > ./date.txt' but when i run task i cannot find 'date.txt' in folder. so where is it?
You can save it under dags folder: date > dags/date.txt
how can i find where it's saved when i run task in webserver?
It should be under your dags folder.
@@coder2j i tried to run task, it was successfull but i can find this file under dags folder