Big data in Airflow? Discover the XCom Backends with AWS S3!

Big data in Airflow? Discover the XCom Backends!
👍 Smash the like button to become an Airflow Super Hero!
❤️ Subscribe to my channel to become a master of Airflow
🏆 Take my course : www.udemy.com/... to join the legends of Airflow
🚨 My Patreon: / marclamberti to support my work and be friend for life
The materials:
www.notion.so/...
How to run Airflow locally with Docker
• Running Airflow 2.0 wi...
All you need about XComs:
marclamberti.c...
The TaskFlow API or the NEW WAY of creating DAGs:
• TaskFlow API in Airflo...
When you begin with Apache Airflow, one of the first question you ask yourself is:
How to share data between tasks in Airflow?
And for that, there is a mechanism called XCOMs.
Sharing data with XComs is simple. You just have to call two methods, xcom_push and xcom_pull. The data you want to share will encapsulated into a XCom object. But there is a problem...
XComs are limited in size. Depending your metadata database, you won't be able to store the same amount of data.
To solve this, Airflow 2.0 introduced a new feature...
The XCom Backends!
Apache Airflow allows you to define your own XCom backend. Which means, instead of storing your XComs into the metadata database of Airflow, you can store them into any storage system you want.
In this video, you are going to discover how to set up your own XCom Backend with AWS S3!
Enjoy!

[@darsh_shukla]

Marc you always surprise me. 🙏

[@MarcLamberti]

I do my best 😉

[@MrIgorbpf]

Great video man!!! Thanks a lot! I've been struggling to deploy airflow 2 on AWS ECS. Can you make a video of it? I believe other people will enjoy because deploying airflow is tough!

[@manqobadlamini2207]

Brilliant video as always! Thanks.

[@AbdurrahmanKocukcu]

Awesome :)

[@candyskullxoxo4660]

Hi, i love your vids. Can you show how to integrate minio xcom backend in Airflow running on kubernetes? Do i need to modify the Pod executer?

[@MrQsam]

very useful! ty!

[@pavelpetkun5269]

Merci beaucoup!

[@sunilpotu6028]

How can we use xcom between two SparkOperator's. In spark command I am generating value in python file so want to use in sencond SparkOperator python value?How can push value to xcom in spark submit?

[@luiztauffer8513]

thanks for the great content!
wouldn't it be safer to store the secrets as ENV variables and retrieve them when needed?

[@fahadshoaib8735]

Hey man nice video but I have one query when the extract task shares the dataframe into the process task the dataframe becomes None and we are not able to perform any pandas operations on it. What's the use of sharing info between tasks if it not usable.

[@chetansurwade]

Low or apparently no documentation for Azure Blob and GCS custom xcoms.

[@shubham_chourasia]

How to achieve this in AWS MWAA environment?

[@murugangan6817]

Xcom with mssql

[@Arnob_111]

You skip some things in your video such as while creating an IAM user when need to create a new policy you opened a new browser tab for IAM and created the policy and then came back to your user and attached the policy. These little things may look unimportant but makes it hard to follow you.

[@SoumilShah]

I sent you messages on LinkedIn through email
Tried contacting you no reply sad to say all you are doing is just for views