Someone correct me if I'm wrong, but can't you run a KPO from any airflow instance that has access to the kubernetes cluster via it's api? For example, I have access to a cluster on a dev machine, and can interact with it via the api (via kubectl). KPO should work just fine in a local airflow instance
Generally speaking, yes, you can spin up a pod programmatically using k8s api (given you have access and permissions). If I understand you question correctly, I don't think you can ask your local airflow instance to spin pod using KPO in k8s cluster in dev machine. KPO expects you to have airflow running on a kubernets cluster where it will try to spin up a pod. th-cam.com/video/RqSYh3UI_Is/w-d-xo.htmlsi=VOhMCMPZMZZdfbvO&t=304
what a clear explanation. thank you so much. for almost 2 years i've been using KPO but just understand the behind the scene :)
Nice video and good explanation
KubernetesPodOperator actually works well in AWS EKS with CeleryExecutor
Correct, the video was wrong about the requirement of KubernetesExecutor.
Great video, thanks 💯
Someone correct me if I'm wrong, but can't you run a KPO from any airflow instance that has access to the kubernetes cluster via it's api? For example, I have access to a cluster on a dev machine, and can interact with it via the api (via kubectl). KPO should work just fine in a local airflow instance
Generally speaking, yes, you can spin up a pod programmatically using k8s api (given you have access and permissions). If I understand you question correctly, I don't think you can ask your local airflow instance to spin pod using KPO in k8s cluster in dev machine. KPO expects you to have airflow running on a kubernets cluster where it will try to spin up a pod. th-cam.com/video/RqSYh3UI_Is/w-d-xo.htmlsi=VOhMCMPZMZZdfbvO&t=304
thank you, now i got this )
Can you please provide the link of part 2, or is it still in progress?
Lets code Airflow Kubernetes Pod Operator | Deep dive | Part 2
th-cam.com/video/2JFFzTw0a-U/w-d-xo.html