วีดีโอ

Great webinar! Thanksss

how to handle zombie task? for large file copy

The title is misleading. This video is not about aieflow 3. It's about an individual contributors journey to becoming an airflow contributor. That's cool, but not what I came here for. Please do better in the future.

ma'am when i run the command astro dev start i get the following error Error: error building, (re)creating or starting project containers: Error response from daemon: error while creating mount source path '/host_mnt/Users/Bingumalla Likith/Desktop/MLOPS/airflow-astro/dags': mkdir /host_mnt/Users/Bingumalla Likith/Desktop: operation not permitted can you help me out with it. Im using mac.

how about the logs ?

I am able to install astro but getting access denied error when using astro dev init or any astro command

Insightful 🙏

​ @Astronomer, Hello! Could you please tell me how you open the .html documentation that is generated inside the Airflow Docker container through the web interface? When I navigate to "data_docs_url": file:///opt/airflow/gx/uncommitted/data_docs/local_site/index.html," I get a 404 error.

Gonzalez Betty White Scott Anderson Jennifer

Thompson Helen Martinez Helen Lee Laura

Hernandez Brian Lewis Angela Clark Thomas

The issue with the KubernetesExecutor is that you cannot view the task logs in the Airflow UI because, with KubernetesExecutor, workers are terminated after their job finishes. This issue is not present with the Celery or CeleryKubernetesExecutor. I tried different solutions with Persistent Volumes (PV) and Persistent Volume Claims (PVC), but they didn’t work for me. At the end of the video, Marc also presented the issue, but no solution was provided. Does anyone here know how to resolve it?

Say I have a dag run happening and dynamically I updated the dag tasks.. Will it break the existing dag run? Say that particular dag run has 10 tasks to do, and I update dag when it's doing 1st task. Will it implement old tasks and newly run dag run implement new updated tasks??

What I would love to see is how to do this when the dbt and Airflow are in separate repositories. Due to different dependencies between airflow and dbt, this seems like a common use case.

Que bacana, meu filho!!! Parabéns.

The best❤️‍🔥

Excelente 👏👏

Hi, will the dag.test() applied for complex tasks such as SparkSubmitOperator() ? 🙏

Great job, thanks for sharing it.

Are presented examples available in any code repo?

Thank you very much for the great presentation and hands-on session. We are going to use Airflow in EKS, and our development Team needed a way to simulate their local environment to test their DAGs during development and become familiar with airflow on Kubernetes. Your guide was extremely helpful.

Gracias

After adding the python file and html file, and restarting the web server plugin details are visible in Admin > Plugins path. But the View is not populating in cloud composer. Is there anything else need to be performed?

After adding the python file and html file, and restarting the web server and postgres from docker. But the View is not populating in my local airflow, Is there anything else need to be performed? Running airflow from docker setup, My airflow version is 1.10.15, pretty old, but can't switch to newer version right now

HI Is it possible to schedule the Task using dataset ? or its controlled at Dag level. I mean if i hv 2 task in downstream Dag , do I hv option to customised the schedule on the basis of Task's upstream dataset

Do you have LinkedIn?

Great!

Can you provide more context on the batch inference pipeline ? Airflow is an orchestrator, you will need a different framework to perform batch inference ?

Very informative, thank you!

Really helpful thank you😍

Tons of information. Any chance this can be thrown in a github for us engineers who need more time to digest?

thank you, well explained. Created an express application to create DAGs programatically but the endpoints are not working

This is really awesome and I love the entire video and always love content from you guys and girls but could I please give some constructive feedback?

Is it possible to get the list of variables pushed through xcom push in first task (here extracting lets say) And can we pull that varibales list xcom_pull and have it as a group Dynamically (instead of A, B, C)??

what about if any of the subtasks fails ? how to trigger the error than but also the remining parallel tasks to be run.

@Astronomer Please share a direct link to the CLI library you mention (for proper files strcuture) th-cam.com/video/zVzBVpbgw1A/w-d-xo.htmlsi=HiJa9Afi-53yLZOG&t=873

Do we have a video on how to run airflow using docker on cloud containers. Running locally is fine to learn and test. But the real work is to see how on cloud. Am a consultant and for my clients easier setup is the goal. With airflow i dont see that

great work, keep it up.

Hi, this video is really beneficial. I have some question about the best practive of handling data transmission btw tasks. I am building MLops using airflow. In my model training dag, it contains data preprocess-> model training. So there would be massive data transmission btw this 2 dags. I am using Xcom to transmit data btw them. But there's like a 2G limitation in Xcom. So what's the best practice to deal with this problem? Using a S3 to sned/pull data from tasks? Or should I simply combine these 2 tasks(data preprocess-> model training)? Thank you.

Great video. Would also be interested in a webinar regarding scaling the Airflow database since I'm having some difficulties of my own with that.

great video. I'm trying to make this work with LivyOperator do you know if it can be expanded or partial arguments supplied to it?

32:29 why "test' connection button is disabled. SO frustrating. Aifrflow makes it so hard to connect to anything. Not intuitive at all. And your video just skipped on how to enable "test". And ask me to contact my deployment admin. lol, I am the deployment admin. Can you show me how? I checked its website and the documentation is not helpful at all. I have been stuck for over a week on how to connect airflow to an MSSQL Sever.

*promosm* 💔

Is it good to return df many times in Airflow?

Hi, I have already an existing airflow project, so how can use Astro CLI to run my project ?

is the git repository public?

please, share repository

That is great intro and overview of Airflow for beginners! I very much like the datasets concepts and the ability to see data lineage. However, I haven't found the solution for how to make a triggered pipe, that is dataset aware, to be executed with the parent dag execution date. Is it even possible at the moment?

Hi, Thank you for detailed demo. I just started exploring dynamic task mapping and I have below requirement where I need to get the data from metadata table and create list of dictionary. [ { 'colA' : 'valueA', 'colB' : 'valueB', 'colC' : 'valueC', 'colD' : 'valueD', }, { 'colA' : 'valueA', 'colB' : 'valueB', 'colC' : 'valueC', 'colD' : 'valueD', }, { 'colA' : 'valueA', 'colB' : 'valueB', 'colC' : 'valueC', 'colD' : 'valueD', }, ] The above structure can be generated using fetch_metadata_task (combination of BigQueryHook and PythonOperator). Now the Question is, how do I generate the dynamic tasks using the above list of dictionary. for each dictionary I want to perform set of tasks ex:GCSToBigQueryOperator, BigQueryValueCheckOperator, BigQueryToBigQueryCopyOperator etc. The sample dag dependancy look like this: start_task >> fetch_metadata_task fetch_metadata_task >> [GCSToBigQueryOperator_table1 >> BigQueryValueCheckOperator_table1 >> BigQueryToBigQueryCopyOperator_table1 >> connecting_dummy_task ] fetch_metadata_task >> [GCSToBigQueryOperator_table2 >> BigQueryValueCheckOperator_table2 >> BigQueryToBigQueryCopyOperator_table2 >> connecting_dummy_task ] fetch_metadata_task >> [GCSToBigQueryOperator_table3 >> BigQueryValueCheckOperator_table3 >> BigQueryToBigQueryCopyOperator_table3 >> connecting_dummy_task ] connecting_dummy_task >> BigQueryExecuteTask >> end_task

Hi All, Thank you for detailed demo. I just started exploring dynamic task mapping and I have below requirement where I need to get the data from metadata table and create list of dictionary. [ { 'colA' : 'valueA', 'colB' : 'valueB', 'colC' : 'valueC', 'colD' : 'valueD', }, { 'colA' : 'valueA', 'colB' : 'valueB', 'colC' : 'valueC', 'colD' : 'valueD', }, { 'colA' : 'valueA', 'colB' : 'valueB', 'colC' : 'valueC', 'colD' : 'valueD', }, ] The above structure can be generated using fetch_metadata_task (combination of BigQueryHook and PythonOperator). Now the Question is, how do I generate the dynamic tasks using the above list of dictionary. for each dictionary I want to perform set of tasks ex:GCSToBigQueryOperator, BigQueryValueCheckOperator, BigQueryToBigQueryCopyOperator etc. The sample dag dependancy look like this: start_task >> fetch_metadata_task fetch_metadata_task >> [GCSToBigQueryOperator_table1 >> BigQueryValueCheckOperator_table1 >> BigQueryToBigQueryCopyOperator_table1 >> connecting_dummy_task ] fetch_metadata_task >> [GCSToBigQueryOperator_table2 >> BigQueryValueCheckOperator_table2 >> BigQueryToBigQueryCopyOperator_table2 >> connecting_dummy_task ] fetch_metadata_task >> [GCSToBigQueryOperator_table3 >> BigQueryValueCheckOperator_table3 >> BigQueryToBigQueryCopyOperator_table3 >> connecting_dummy_task ] connecting_dummy_task >> BigQueryExecuteTask >> end_task