Airflow DAG dependencies: The Datasets, TriggerDAGRunOperator, and ExternalTaskSensor

Airflow DAG dependencies: The Datasets, TriggerDAGRunOperator and ExternalTaskSensor
A DAG dependency in Apache Airflow is a link between two or multiple data pipelines.
There are three ways of creating DAG Dependencies in Apache Airflow:
👉 Datasets
👉 TriggerDagRunOperator
👉 ExternalTaskSensor
In this video, you will discover an overview of those 3 ways and how to use them.
Materials: robust-dinosaur-2ef.notion.si...
🏆 BECOME A PRO: www.udemy.com/course/the-comp...
👍 Smash the like button to become an Airflow Super Hero!
❤️ Subscribe to my channel to become a master of Airflow
🚨 My Patreon: / marclamberti
Enjoy ❤️

[@claudiodeluca2357]

Great!!!

[@MarcLamberti]

Thank you

[@akshitapanyam]

Hi, I have a question. If I have a DAG which is scheduled to run on daily basis. Can I keep a dependency on the previous run of the same dag?

[@SfZq]

How can I use ontology with Graph to extract dependency 😖 ???

[@revazg4071]

Thank you for the great tutorials. Could you please consider touching email-sending methods? There are multiple ways to send an email and it will be interested, which one do you prefer.

[@user-gg9nz5ns1k]

Hey...I have a question. If 2 DAGs remain within 2 different airflow clusters.....which of these methods can be used for managing cross DAG dependency?

[@MarcLamberti]

Check out the REST API

[@Sir_Dave_Kennedy]

Hi Marc, I love you Airflow videos. Do you have a solution for file watching files matching pattern like filename_YYYYMMdd.csv (eg: filename_20240501.csv) in S3 and also onprem? Thanks

[@luongky06]

Why we don't use API to trigger another DAG instead of use the datasets?

[@MarcLamberti]

Because datasets are easier and more flexible to use than the API. The API is useful if you have a dependency between two DAGs that are in two DIFFERENT Airflow instances

[@TheMisterDonald]

Max i pay u train udemy.. my case i wanna copy data base from postgres server to local server .. and update every day.. please give me example...i dont understand..

[@reviewmaster9435]

ok

[@MarcLamberti]

ok

[@reviewmaster9435]

@@MarcLamberti i really like your videos and hard work, you are awesome, love from india Marc😇

[@viniciuskreischer1868]

Hi, Marc. Is it posdible to trigger specific indtances of a DAG using datasets?