Your videos are very good, I would like to know how I can expose the airflow configuration file since all the images that I have downloaded do not have it accessible, thank you very much
The airflow configuration file exists in the airflow container under that directory of /opt/airflow/airflow.cfg, by which you can mount local airflow.cfg with. If you want to change the configuration, it is recommended to use the environment variables: airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html
I will definitely cover how to use Airflow to connect with S3 and customised operator. Stay tuned! SFTP you can check the doc of the SFTP provider. airflow.apache.org/docs/apache-airflow-providers-sftp/stable/_api/airflow/providers/sftp/index.html
Hey @coder2j, I'm newbie Data Engr and learning Airflow. While practing on the same code, I got following error. can you please help me to sort it out? WARNING - Dependency , greet already registered for DAG: DAG5-3 ]
Thank you for doing a great job to share your knowledge!
Glad you like it!
Thank you !! you are the no1 engineer in the world!!😊😊😊😊
Wow, thanks
Great tutorial! Would be great to see custom operator best practices and contrib operators usage. Thanks.
More to come, stay tuned. 😉
Great work so helpful
Glad to hear that!
Thanks man.
Happy to help
i want to know how to use airflow REST services using postman we facing issue for Authentication while sending request from postman
Amazing, learned a lot. thank you for your series. Can you keep it up please so we can learn more stuff?
That's the plan! Thanks for sharing your thoughts.
Your videos are very good, I would like to know how I can expose the airflow configuration file since all the images that I have downloaded do not have it accessible, thank you very much
The airflow configuration file exists in the airflow container under that directory of /opt/airflow/airflow.cfg, by which you can mount local airflow.cfg with. If you want to change the configuration, it is recommended to use the environment variables: airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html
@@coder2j thank u master ;)
It would be great to see Airflow in action on a cluster of workers on AWS free tier account. :)
Great idea! Let me investigate it and see if we can make a video for that.
Using keras with airflow and how to use any docker image for that. Build a deep learning pipeline with keras
Do you mean how to run airflow docker container task?
Yes docker with keras installef
That's a great idea. I will plan something for that. Stay tuned!
Can you show how to download file from s3 bucket and share it via SFTP to remote server by writing custom dag (the workflow in same dag)
I will definitely cover how to use Airflow to connect with S3 and customised operator. Stay tuned! SFTP you can check the doc of the SFTP provider. airflow.apache.org/docs/apache-airflow-providers-sftp/stable/_api/airflow/providers/sftp/index.html
should i use this taskflow api in 2023 ?
There is no right or wrong way to write your dag. Task flow is a good way to organize your code.
is there any better option?
plz let me know, I'm newbie
Does anyone know how to switch to dark mode in airflow ?
Hey @coder2j, I'm newbie Data Engr and learning Airflow. While practing on the same code, I got following error. can you please help me to sort it out?
WARNING - Dependency , greet already registered for DAG: DAG5-3
]
Can you share your dag file?