Really good videos. Can you make one how to create a new spark job to a cloud Kubernetes like EKS cluster running Airflow? Bonus points if you can show how to dockerize, test it locally, deploy on QA environment & eventually promote it to production environment. This will mimic day to day work real closely. I understand it’s massive to undertake, but hope you get around to creating a series for this end to end scenario. Love your content!!
Very interesting deep dives. I was wondering how would you go about specifying resources per DAG step? Say ingest would need at least 32gb mem and X CPUs. Is there any way to have this flexibility at this point? Thanks for the videos. Keep up the good work.
Really good videos.
Can you make one how to create a new spark job to a cloud Kubernetes like EKS cluster running Airflow? Bonus points if you can show how to dockerize, test it locally, deploy on QA environment & eventually promote it to production environment.
This will mimic day to day work real closely. I understand it’s massive to undertake, but hope you get around to creating a series for this end to end scenario.
Love your content!!
Come on bro. It’s been so long don’t keep us waiting :)
nice and simple demonstration.. thanks
You are the best🎉😊
Very interesting deep dives. I was wondering how would you go about specifying resources per DAG step? Say ingest would need at least 32gb mem and X CPUs. Is there any way to have this flexibility at this point? Thanks for the videos. Keep up the good work.
Great explanation, Just one quetion how can we use "envFrom:configMapRef" key:value variables into our docker image business logic.
Great 🎉
hello my friend, Would you mind being able to share the code on github?
Feel free to fork!
Please record previous old video again. Quality is not good.