Apache Airflow: Adios SubDAGs! Welcome TaskGroups!

[@saritkumarsi4166]

Thanks again Marc. Great addition to my knowledge:)

[@MarcLamberti]

Glad it was helpful!

[@sunil._sharma]

As usual, you picked a great topic to share

[@MarcLamberti]

Glad you enjoyed it

[@user-ni5gp4km4v]

Need to watch it

[@xuantuongvu2020]

Thanks but if you have a airflow cluster, how to gurantee all the tasks in a taskgroup to run in the same airflow woker node?

[@jasjyotsingh2007]

Is it not possible to group multiple dags instead of mutiple tasks inside TaskGroup. We have multiple ETL processes and a separate DAG for each of them if we chose to group the tasks directly instead of a dag we might end up with DAG file with far too many tasks encompassing tasks from different processes . And it also effectively means owners of different processes changing the same DAG file whenever they want to change / reposition tasks for their subgroup which they could have accomplished initially by changing their respective Dag files

[@punitgaikwad4858]

Hi, can you tell how to call a specific task group and pass a parameter to it if we have a multiple task group inside our dag

[@techcode_man]

Hi Marc, is there a video you uploaded for subdags? I am not finding it.

[@MarcLamberti]

don't use subdags :)

[@smitatanwar3229]

Thanks Marc. It is very useful. I was just searching some vedio with airflow.providers.grpc.operators.grpc

[@vandanasingh3678]

@Marc Lamberti, How can we skip taskGroups and also they should be marked as 'skipped' in airfow UI. I have looked into skipping tasks using PythonOperator, I wanted to know if there is a way to skip taskGroups

[@abimaeldominguez4126]

Hello Marc, I have a question, if one of the task inside of a task group fails, can I tell Airflow to re-run ALL the tasks inside the task group ( not only re-run failed tasks), thank you.

[@sanjairajv]

+1

[@adityabasak533]

Is it possible to have trigger rules in task groups

[@eldarrozencvaig2013]

Hi Marc,
what about repetitive tasks?
using subtags help me define the tasks oney once,
can we do the same with Task Groups ?

[@MarcLamberti]

Of course you can. Just put your taskgroup within a function that returns the taskgroup and call it wherever you need

[@eldarrozencvaig2013]

@@MarcLamberti Thanks for the quick response!
Can you please share an example for using the returned task_group from the function in the DAG itself ?

[@punitgaikwad4858]

Hi, I have a question. What if we want to run each tasks inside task group to run at different time. Suppose after 10 seconds of completion of previous task. Can we do that... If yes then tell how

[@MarcLamberti]

Why would you want to do that?

[@punitgaikwad4858]

@@MarcLamberti Hi, I have some set of jobs which are identical in some way so i want to group them using TaskGroup. But as of now while I am triggering my dag all tasks grouped inside TaskGroup are executing at same time. I want them to run one after other ( having some execution time difference in between them) . So how can i do that!

[@neerajvyas3947]

Can we create n number of task groups dynamically?

[@neerajvyas3947]

By taking input from dag run. Conf?

[@MarcLamberti]

Yes you can :)
Try. The same way you can create tasks dynamically, you can create taskgroups dynamically as well. At the end, it a python object.

[@pavithrasudhakar6699]

can we create dependency on these tasks in TaskGroup ?

[@lxkakkarot3689]

Have you found how to create dependencies in TaskGroup?

[@FernandoCulen]

@@lxkakkarot3689 did you find something about this?

[@rafaelsantana616]

Task groups are great, but you can't run each one separately. This is the only benefit of using subdags I can see.