Airflow Python Operator and XCom: Airflow Tutorial P6

Airflow Python Operator and XCom: Airflow Tutorial P6
#Airflow #AirflowTutorial #Coder2j
========== VIDEO CONTENT 📚 ==========
Today I am going to show you how to use Airflow PythonOperator and XComs. By watching this video, you will know:
👉 How to run a python function as a task using the python operator
👉 How to pass parameters to the python function
👉 How to share values between different tasks using Xcoms
Video Request: forms.gle/UMp4GA3krcSMMWzy9
You don't have a running Airflow in your local setup? Check out the Apache Airflow 2.0 Installation Tutorials.
Run Airflow locally 👉 • Airflow introduction a...
Run Airflow in Docker 👉 • Airflow Docker: run Ai...
========== L I N K S 🔗 ==========
GitHub Repo 👉 bit.ly/3HD5oTX
Airflow Documentation 👉 bit.ly/3wbTqv4
========== T I M E S T A M P ⏰ ==========
00:00 - Introduction and launch airflow using docker
00:50 - Create a new DAG file
02:40 - Using the python operator to run a python function
03:50 - Pass parameters using op_kwargs
05:10 - How XCom works
07:10 - Push and pull values using XCom
09:18 - Push multiple values using key and value
12:42 - Only share small values via XCom
========== Connect with me 👏 ==========
Twitter 👉 / coder2j
Website 👉 coder2j.com
GitHub 👉 github.com/coder2j

[@pebl4269]

Excellent job! Finally, Airflow DAGs have been demystified.

[@leamon9024]

Finally!! Thanks for your hard work. Would love to see more too.

[@coder2j]

Working on it!

[@Jeganbaskaran]

It's just short and covers the core concepts nicely. Expecting more videos and Thanks for your time to prepare the valuable content.

[@coder2j]

You are welcome! I am glad that you liked it. 😉

[@achrafbattiwa1654]

Really useful.Thank you !

[@wagnermichaell4776]

Boom! It's really interesting and so useful! Thanks so much!!! 💕

[@coder2j]

I am glad that you like it. Thanks for your warm comment!! :-)

[@siqiongli2632]

Loving your tutorials! It would be great if you could share more about Airflow on Kubernetes

[@digambarmishra]

Very easy to Understand and Practice.

[@feeelgoood9580]

Great job!

[@con7745]

I have studied easily thanks to your video. Thank you very much!

[@coder2j]

You are welcome!

[@LearnRight]

Thankyou so much for clear, simple and very useful video.

[@coder2j]

Thank you.

[@brunodupont8494]

very very good, I am inching my way along, this is helpful

[@coder2j]

Thanks for watching.

[@anilkumar-xp9et]

Boom Super... nice explanation for beginners for airflow... pls do more videos on this.

[@coder2j]

Thanks, will do!

[@MahmoudBahnasawy-yk2fj]

Great

[@hungaoquang9716]

good

[@aniqatiq731]

Amazing work

[@coder2j]

Thanks a lot 😊

[@user-zc7hl4db7g]

thank you very much from korea peninsula

[@coder2j]

You are welcome

[@faysalahmedsiddiqui]

Very nice and concise. I have dict object that i want to pass as kwargs to pythonoperator callable function along with ti. How xan i do that?? Do i have to add 10 dictionary keys as function parameter and 11th paramter as ti or there is some other way..

[@coder2j]

You can just pass the dict object as a python parameter. Something like this: op_kwargs={'some_dict': {'a': 1, 'b': 2}} for python callable def greet(some_dict, ti):

[@pythonmathui3057]

Sir,, why everytime i save a new dag, the previous dag that i saved before got lost in the dags list in the webserver? i mean the v01, v02, and v03 doesnt appear soon after i saved the v04 and v05, did i miss something, please answer ....

[@coder2j]

That's normal. When you saved it to new version. The scheduler will refresh and get the latest version only.

[@abhishekpoudel500]

If not using XCOM for larger values, what to use instead?

[@coder2j]

You can save intermediate large files to local disk or remote storage like S3.

[@surenderraja1304]

I have seen a python method like def get_spark_config(**context):
ti = context['ti'] what is the purpose of context , why there are ** . What is the meaning for ti = context['ti']

[@coder2j]

context is a dictionary which contains a lot of dag info the airflow provides. With **, you basically put every key and values in dictionary context as function parameters. So context['ti'] basically get ti (task instance) value from context with key ti.

[@surenderraja1304]

@@coder2jUnderstood now , Thanks for your clear videos and explanation

[@Khc514]

Hi do you know why 'docker-compose up -d' doesn't start my localhost:8080? Instead I simply use "docker compose up' and it works, is there a reason for this?

[@coder2j]

It should work. Can you check if the container is up when you run the 'docker-compose up -d'. Keep in mind, it may take some minutes to bring the containers up

[@Khc514]

yes, it does thank you, if you don't mind me asking what's the difference between these two?@@coder2j