How to Pass Data Between Tasks in Airflow

In this video I'll go through ever possible way I know of to pass data between tasks, whether they are Traditional Operators or TaskFlow API Tasks! Check out the repo below if you want code examples for any of these methods!
github.com/ast...

[@thepakcolapcar]

hi George, nice video
I am looking for a way to pass an json, yaml or some way to pass the configurations to dag.
And then should be able to read those values, overwrite in task and then get the updated value of that variable/config in another task of same dag
However I dont want to use xcom or variables. Is there any way to handle this? PLease let me know

[@thedataguygeorge]

Yes definitely! You would want to use DAG params for that, you can define variables that are passed at runtime, and you can use them throughout your DAG! Check out the docs for some examples: airflow.apache.org/docs/apache-airflow/stable/core-concepts/params.html

[@thepakcolapcar]

thank you@@thedataguygeorge
I will try follow above link and keep you posted

[@thepakcolapcar]

hello @thedataguygeorge The lik provided by you is very useful. But it seems like values of params passed at dag when modified in a task_a does not appear in next task_b. The task_b again gets the default value that were passed to DAG.
I am trying to change values of dag params in task_a and the subsequently need to get the changed values from task_a when accessed in task_b. Hope I am not confusing.

[@thedataguygeorge]

Ahhhhhh I see, thank you for the clarification that makes much more sense! For that approach you would typically want to use Xcom's or Variables, but if you want to avoid both of those, you'll need to save those saved values from task a in an intermediary location as part of task a, and then load them into task b explicitly before using them. Check out the 'intermediary data storage' section of the following link for more info on how to do this docs.astronomer.io/learn/airflow-passing-data-between-tasks

[@0kazaki]

If you have a DAG A that triggers DAG B via-Datasets, can DAG B get information from DAG A? such as run_id or other custom information, like for example, DAG A's Task1 proccessed CSV files [example1.csv, example2.csv[

[@thedataguygeorge]

Yeah definitely, you would just need to dynamically pass the info from dag A as parameters in the API request!

[@thedataguygeorge]

Can get that info you described from reading the DAG A task instance

[@ornachshon1]

How can I pass a csv (small) between tasks?

[@thedataguygeorge]

Does it need to be a CSV? If you use the Astro Dataframe decorators you can pass dataframes between tasks, otherwise i'd recommend saving it some where in the first task and then loading it from that location for the second task!