Airflow CI/CD: Github to Cloud Composer (safely)

Jacob Ferriero
Deploying bad DAGs to your Airflow environment can wreak havoc. This talk provides an opinionated take on a mono repo structure for GCP data pipelines leveraging BigQuery, Dataflow and a series of CI tests for validating your Airflow DAGs before deploying them to Cloud Composer.

[@daibhead]

This is an excellent talk. Really useful information. Migrating from a self-managed airflow setup to composer and this is exactly the type of overview I was looking for. Very well spoken presenter too. Thanks!

[@pulkitgoel2968]

I was looking for a CICD process for our ETL pipelines which uses Dataflow and Composer.. I have a better idea now..Thanks

[@xsamuelx3603]

nice video :D

[@prakashmudliyar4834]

I don't understand what you did at 28:50 to trigger the build. Could you explain?

[@DKR-oh5zz]

Hi i was deploying the airflow Dag in gcp cloud i got this error if you have any idea please let me know