Any working example or demo on k8spodoperator? Is it possible to run airflow mgmt on linux vm but have dag tasks run on a remote k8s cluster as well as a non default namespace? Any pointers to examples
k8spodoperator will be covered next :) Regarding dag tasks execution on remote eks, there isn't executor specifically designed for that use-case. You can choose airflow namespace by setting following environment variable AIRFLOW__KUBERNETES__NAMESPACE airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#namespace. There are other namespace configurations that you can choose e.g. AIRFLOW__KUBERNETES__MULTI_NAMESPACE_MODE
Hi, I really need to know if we put the python script into a container and run it using DockerOperator, how can we pass the values of Variables from AirFlow Admin UI t the container ?? Please help.
Question Can I say airflow is a cicd tool? Are we automating the code using kubernets in airflow? Out of topic what is difference between airflow and open stack? Please clarify
Airflow is a scheduling tool. Yes, nothing stops you to schedule your CICD operations in airflow (but only if its worth!) e.g. if steps involved in cicd pipeline has complex dependencies. Otherwise, existing tools such as Bitbucket, and GitLab are usually sufficient for cicd tasks.
I absolutely loved the whole series. It's the first educational content I have binge-watched! Really good stuff!
Glad you liked it. Thanks :)
I love this series as well. I hope you publish more airflow contents for data etl and work with various cloud data warehouse.
Thankyou Li. Definitely we have more airflow contents coming very soon.
Superb content !!
Any working example or demo on k8spodoperator? Is it possible to run airflow mgmt on linux vm but have dag tasks run on a remote k8s cluster as well as a non default namespace? Any pointers to examples
k8spodoperator will be covered next :)
Regarding dag tasks execution on remote eks, there isn't executor specifically designed for that use-case.
You can choose airflow namespace by setting following environment variable AIRFLOW__KUBERNETES__NAMESPACE
airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#namespace.
There are other namespace configurations that you can choose e.g. AIRFLOW__KUBERNETES__MULTI_NAMESPACE_MODE
Hi, I really need to know if we put the python script into a container and run it using DockerOperator, how can we pass the values of Variables from AirFlow Admin UI t the container ?? Please help.
Question
Can I say airflow is a cicd tool?
Are we automating the code using kubernets in airflow?
Out of topic what is difference between airflow and open stack?
Please clarify
Airflow is a scheduling tool. Yes, nothing stops you to schedule your CICD operations in airflow (but only if its worth!) e.g. if steps involved in cicd pipeline has complex dependencies. Otherwise, existing tools such as Bitbucket, and GitLab are usually sufficient for cicd tasks.