Airflow on Kubernetes: Containerizing your workflows

Presented by Michael Hewitt at Airflow Summit 2020.
At Nielsen Digital we have been moving our ETLs to containerized environments managed by Kubernetes. We have successfully transferred some of our ETLs to this environment in production. In order to do this we used the following technologies: Helm to easily deploy Airflow on to Kubernetes; Airflow’s Kubernetes Executor to take full advantage Kubernetes features; and Airflow’s Kubernetes Pod Operator in order to execute our containerized Tasks within our DAGs. To automate a lot of the deployment process we also used Terraform. Lastly, Kubernetes features were used to gain much more fine grained control of Airflows infrastructure.

[@risebyliftingothers]

Awesome 👍 The best part is QnA

[@rfrost4708]

Great to see activity on the TH-cam channel. Keep em coming!!!

[@stefanw8203]

Great and compact explaintions. Thank you very much sharing this with us. Like more from you.

[@Blablablateelbal]

Thanks for the talk!
Any chance to see some of the infrastructure as code or manifest files you talked about?

[@KG_Mrt]

Great stuff, now we have dag sync. Can we attach two different git repo to same namespace - like multi tenancy?

[@lostfrequency89]

Always had this question in mind, is it possible though? Have you tried by any chance ?

[@lostfrequency89]

Always had this question in mind, is it possible though? Have you tried by any chance ?

[@KG_Mrt]

@@lostfrequency89 it's not possible. Airflow is a single tenancy.

[@lostfrequency89]

@@KG_Mrt there’s a bitnami airflow helm chart which provides the option to add more repos though. Never tried but sounds interesting. Any thoughts on that?

[@kimted3272]

thanks for the great info :)

[@ruwanvimukthimettananda7847]

How we can tolerate the pod from KubernetesPodOperator?