Airflow on Kubernetes - Scaling DAG Workflows | Daniel Imberman, Seth Edwards @ PyBay2018

This talk was presented at PyBay2018 - the Bay Area Regional Python conference. See pybay.com for more details about PyBay and click SHOW MORE for more information about this talk.
Description
Over the past year, we have developed a native integration between Apache Airflow and Kubernetes that allows for dynamic allocation of DAG-based workflows and dynamic dependency management of individual tasks.
Abstract
Apache Airflow is a highly popular Directed Acyclic Graphs (DAG) based workflow engine that allows users to deploy complex DAGs as python code. It is considered a natural progression of the "code as configuration" philosophy of DevOps and ETL.
With the addition of the native "Kubernetes Executor" and "Kubernetes Operator", we have extended Airflow's flexibility with dynamic allocation and dynamic dependency management capabilities of Kubernetes and Docker.
Slides: goo.gl/tU1cJ9
Speaker Bio
Daniel Imberman is a Big Data Engineer on the Spark & NoSQL Team at Bloomberg LP. He received a BS/MS in distributed Machine Learning from UC Santa Barbara in 2016.
Seth Edwards is a Staff Software Engineer at PubNub and responsible for Data platforms, ETL, and DevOps.
This and other PyBay2018 videos are brought to you by our Gold Sponsor Cisco!

[@arunselvamani2237]

Skip to 21.30 for demo

[@pruff3]

Actually @22:23 is where the demo starts, they screw around with the projector for a minute

[@Nikita11035]

So what the reason to use KubernetesPodOperator against KubernetesExecutor? Both allow you to specify custom Docker image.

[@renalapriansyah2735]

how you did gcloud auth application default login on airflow? isn't it should login by UI?

[@Cal97g]

'we were able to use airflows literally in built solution to store task state'