Hey folk 👋 Here is the updated video with the sound fixed: Dynamic DAGs in Apache Airflow for beginners th-cam.com/video/6eHuOd96unQ/w-d-xo.html Enjoy ❤
I believe, this method is much more reliable if you are going to generate multiple dags with different schedule intervals. I was using dynamic dag generation with globals()[dag_id]=dag, but if you are working with APIs or clients, scheduler & worker don't work as expected. Also in the documentation, isolated DAG files are suggested, therefore Airflow community must be using your methods as well. Generator.py can be scheduled to generate the files using cron job, and it will be good to go :) Thank you for the content you have shared and informed Airflow users. It was extremely helpful for me :)
This is informative but I think the piece that I always struggle with is whats the best way to keep the load on the scheduler down if I need to pull data to build the dag based on whats in a database.
Hi! Thanks for your video. Very helpful. Just a comment, you're recording your voice through just one channel, and music in the other, which makes it a bit quieter than normal.
Hi Marc, thank you very much for sharing great information! one quick question though, this could have also done by using normal replace method as well where it will replace the values it receives from yaml in dag template and generate new dag for each yaml config files. Any specific advantage of using jinja2 templates instead?
Hi Marc, can you please elaborate on the risks of using config with a loop and globals? I would agree that having one file per DAG is more ideal but you also linearly increase your codebase for each config so that con needs to outweigh the risks/downsides of the loop/global approach
Hi Mark, I am new to airflow and I was hoping you’d be able to help provide me guidance on this: I want to create a job that retrieves 1.api key based on params (ie params=api id) 2. make api call 3. load data However I need to do this for many different Apikey (ie do this for 100+ different api keys, but function is the same across all, only difference for each job is params of step1) how would you approach this? Would you dynamically create individual dags based on params for each api key? Would you have a parent dag to generate subdags for each apikey? Or any other ideas? Thanks a lot in advance!
Hi Marc, first of all, thanks a lot for teaching us airflow! I also got your course on Udemy but I had a question, there you ask us to use virtual box but I didn't and I used your TH-cam video to setup airflow using docker in 5 minutes. That shouldn't be a problem for the rest of the course, right? Edit: fixed tracing to teaching
Hey folk 👋
Here is the updated video with the sound fixed: Dynamic DAGs in Apache Airflow for beginners
th-cam.com/video/6eHuOd96unQ/w-d-xo.html
Enjoy ❤
thank you for the great tutorial again. I love the airflow series.
Awesome content, thank you.
awesome! that's what i was looking for)
I believe, this method is much more reliable if you are going to generate multiple dags with different schedule intervals. I was using dynamic dag generation with globals()[dag_id]=dag, but if you are working with APIs or clients, scheduler & worker don't work as expected. Also in the documentation, isolated DAG files are suggested, therefore Airflow community must be using your methods as well. Generator.py can be scheduled to generate the files using cron job, and it will be good to go :) Thank you for the content you have shared and informed Airflow users. It was extremely helpful for me :)
This is informative but I think the piece that I always struggle with is whats the best way to keep the load on the scheduler down if I need to pull data to build the dag based on whats in a database.
Informative ❤️
Hi! Thanks for your video. Very helpful. Just a comment, you're recording your voice through just one channel, and music in the other, which makes it a bit quieter than normal.
Hey. Can I access the airflow instance which is deployed on astronomer using kubernetes?? I mainly want meta data to access
Lovely!
Hi Marc, thank you very much for sharing great information! one quick question though, this could have also done by using normal replace method as well where it will replace the values it receives from yaml in dag template and generate new dag for each yaml config files. Any specific advantage of using jinja2 templates instead?
Hi Marc, can you please elaborate on the risks of using config with a loop and globals? I would agree that having one file per DAG is more ideal but you also linearly increase your codebase for each config so that con needs to outweigh the risks/downsides of the loop/global approach
Can we use JSON also instead of YAML?
of course :)
Do you have any git repo this example?
Hi Mark, I am new to airflow and I was hoping you’d be able to help provide me guidance on this:
I want to create a job that retrieves
1.api key based on params (ie params=api id)
2. make api call
3. load data
However I need to do this for many different Apikey (ie do this for 100+ different api keys, but function is the same across all, only difference for each job is params of step1) how would you approach this?
Would you dynamically create individual dags based on params for each api key?
Would you have a parent dag to generate subdags for each apikey?
Or any other ideas?
Thanks a lot in advance!
Hi did you find a way to do it?
Hi Marc, first of all, thanks a lot for teaching us airflow! I also got your course on Udemy but I had a question, there you ask us to use virtual box but I didn't and I used your TH-cam video to setup airflow using docker in 5 minutes. That shouldn't be a problem for the rest of the course, right?
Edit: fixed tracing to teaching
Not at all and I'm actually thinking of removing this VM
@@MarcLamberti that's great, thank you very much!
may you re-record this?
Why? 👀
Hmm.. why can I only hear music but not the voice from my right earbud.
great video nonetheless thank you !!
My mic :'(
Sorry about that
I never managed to install airflow to start learning…the local host 8080 never opens to me
Audio is not good