This is the best video on airflow on TH-cam. You’ve exceptional clarity in teaching! No filler/stammering helps users grasp it in the first go. Thanks.
This is exactly what I was looking for. Thanks a ton for this amazing tutorial! If you can extend this further by creating a dashboard using tableau or power bi from mysql table that you created, that would be great! 🙌
Looking for what to share with my student on Airflow and I came across this. I watched it from start to end because it's like watching John Wick 4. Probably the best Airflow tutorial on TH-cam.
Hello sir, I am new at apache airflow. Your video of airflow for was awesome and helped me to understand it. I have a request to you, I don't know is it possible to use airflow for php application Cron task. If yes, then it would be a great help for me if you make a video on it step by step like this video.... thank you
I have an MS SQL Server (on local/remote) which is connected using my windows authentication in python. How can I connect MS SQL Server using windows authentication in docker? I have used Trusted_Connection=yes but this doesn't seem to be working. The works well outside the docker but not in docker.
Hi Teja, thank you for the video. It really helped me to get on airflow UI, but one of my py file contains bs4 library and 'No module named bs4' error keeps showing. I installed the library in local computer and include the library in dockerfile, but I can't fix this issue. What would be the problem?
i tried importing the pre_processfile but i have an error of module not found on my uri. i wrote my pr_process file to include several function file and i need eah function to run on airflow as tasks. how do i fix this?
I am getting this error 'docker-compose' is not recognized as an internal or external command, operable program or batch file. ## do i need to install docker first and set env path? before extracting the file provided by you.
Hello , thank you for the example. I have a question can you please help? I am not able to view the logs in the AirFlow UI! Tried mounting the log volume in the docker compose but still have the same issue. Can you please point what I am missing. Note: I am running docker in a Mac system. Thank You.
Hello sir when I'm trying to run below command than its giving me error Command:- docker-compose -f .\docker-compose-LocalExecutor.yml up Error:- 'docker-compose' is not recognized as an internal or external command, operable program or batch file.
All of the flaws described at the beginning are handled by most mature ETL platforms. 1."Retrying" only the failed containers in SSIS for example can be achieve with variables and good use of the control flows and precedence constraints... 2. Scheduling is handled by SQL Agent. 3. Logging is available through the SSIS catalogue, etc.. Caricature when trying to explain why you should use tool A vs B is not always the best approach. Not saying there isn't a use case for Airflow, but ETL tools are actually great if you know how to use them...and for a company that already has skills in a given ETL tool, moving to Airflow will probably be a waste of time.
Hi Sir, When I run this run docker-compose command , I am getting some Permission denied errors like this- Error: /snap/docker/2285/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release. from cryptography.hazmat.backends import default_backend Traceback (most recent call last): File "/snap/docker/2285/lib/python3.6/site-packages/urllib3/connectionpool.py", line 710, in urlopen chunked=chunked, File "/snap/docker/2285/lib/python3.6/site-packages/urllib3/connectionpool.py", line 398, in _make_request conn.request(method, url, **httplib_request_kw) File "/snap/docker/2285/usr/lib/python3.6/http/client.py", line 1285, in request self._send_request(method, url, body, headers, encode_chunked) File "/snap/docker/2285/usr/lib/python3.6/http/client.py", line 1331, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/snap/docker/2285/usr/lib/python3.6/http/client.py", line 1280, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/snap/docker/2285/usr/lib/python3.6/http/client.py", line 1046, in _send_output self.send(msg) File "/snap/docker/2285/usr/lib/python3.6/http/client.py", line 984, in send self.connect() File "/snap/docker/2285/lib/python3.6/site-packages/docker/transport/unixconn.py", line 30, in connect sock.connect(self.unix_socket) PermissionError: [Errno 13] Permission denied
This is the best video on airflow on TH-cam. You’ve exceptional clarity in teaching! No filler/stammering helps users grasp it in the first go. Thanks.
Thanks a lot Shivam for your feedback
You have built a well designed channel for data aspirants, Much appreciated. You’ll get traction in coming days Teja.
Thank you Rupesh
this is the best tutorial vedio on youtube thanks a lot bro
Thank you
This is exactly what I was looking for. Thanks a ton for this amazing tutorial! If you can extend this further by creating a dashboard using tableau or power bi from mysql table that you created, that would be great! 🙌
I have enjoyed and thanks for sharing all you need to know about airflow, my concepts about airflow is cleared now..
Thank you
Teja ,,, This is so perfect ,, You are awesome so much clarity .....Great Job. And Thanks for sharing with us.
Thank you Deepak
Nice vedio, Good stuff for new learner. Thaks a lot!!!!
Thanks for the detailed and line by line explanation. hope that we will get more videos regarding airflow in future.
Sure will try
Looking for what to share with my student on Airflow and I came across this. I watched it from start to end because it's like watching John Wick 4. Probably the best Airflow tutorial on TH-cam.
Thanks Olorunfemi for the feedback
Great video! Thanks a lot.
very well explainedd
Awesome video, Thanks a lot
This is diamond 💎. Thanks a lot Teja for the awesome content. This is exactly what I was looking for. Kudos to your efforts 🙏
Thank you
good presentation
Nice videos continue more videos
Thank you
simply superbbbbbbbbb
Thank you
Good content, keep it up
Thank you Abhishek.
love u bro
Could you please bring a series on dataproc pyspark?
Is there any video how to use dags through gcp cloud composer?
bless you x2
Excellent , please share the slides.
docs.google.com/file/d/1hSeHKB2ceBi5yVGiUkO-Ri0a0vzB-7Bc/edit?usp=docslist_api&filetype=mspresentation
Hello sir, I am new at apache airflow. Your video of airflow for was awesome and helped me to understand it. I have a request to you, I don't know is it possible to use airflow for php application Cron task. If yes, then it would be a great help for me if you make a video on it step by step like this video.... thank you
Sure let me check the feasibility
straight to the point . great video . keep it up
Thank you
I have an MS SQL Server (on local/remote) which is connected using my windows authentication in python. How can I connect MS SQL Server using windows authentication in docker? I have used Trusted_Connection=yes but this doesn't seem to be working. The works well outside the docker but not in docker.
Hello sir very nice video.
Could you make a video for call third party post api to upload csv
Please sir tell how to become tech CEO fast as possible.
Learn the technology first😂
amazing. thanks for the session. i have some questions how can i post you ?
Hi is it possible to get task/dag duration from context
Hi Teja, thank you for the video. It really helped me to get on airflow UI, but one of my py file contains bs4 library and 'No module named bs4' error keeps showing. I installed the library in local computer and include the library in dockerfile, but I can't fix this issue. What would be the problem?
i tried importing the pre_processfile but i have an error of module not found on my uri. i wrote my pr_process file to include several function file and i need eah function to run on airflow as tasks. how do i fix this?
I am getting this error 'docker-compose' is not recognized as an internal or external command,
operable program or batch file. ## do i need to install docker first and set env path? before extracting the file provided by you.
Yes of course you have to install docker first
Hello , thank you for the example. I have a question can you please help? I am not able to view the logs in the AirFlow UI! Tried mounting the log volume in the docker compose but still have the same issue. Can you please point what I am missing. Note: I am running docker in a Mac system. Thank You.
Hi Teja,
I'm doing a poc on data robot. and i would like to run the data robot pipeline through apache airflow. can you provide some guidance on it.
Never tried
Nice explanation please make a video on Aws topic like S3,Lambda,Ec2 please make one video
Have plans on that
Hi man , how to get data from Google drive with specific name and run the process as mentioned above
Hi, Thank you
Just a quick question, How does the docker file download the dependencies in the python file (processing.py) such as Pandas
We will have requirements.txt file and on docker file we will run the command to install dependency
@@viswatejaforu Thanks
Hello teja, do you have any video on S3 to snowflake loading via Airflow ?
No, never tried
How do you integrate Pycharm with docker to run code on Airflow ?
Never tried with Pycharm
Hi, Can I installed Apache airflow using .tar.gz file on production server? Thanks for the video.
Can be, but never tried
Could you please send the github repo link for this project. The link provided in the description leads to some other directory
Its the right link, you can click on download button and all the zip file will get downloaded.
Hello sir when I'm trying to run below command than its giving me error
Command:- docker-compose -f .\docker-compose-LocalExecutor.yml up
Error:-
'docker-compose' is not recognized as an internal or external command,
operable program or batch file.
Did you installed docker? If so is docker up and running
Did you installed docker? If so is docker up and running
How should I use on Windows/
How much python we should know to learn airflow ?
basic python would be enough
< bash > ntg
All of the flaws described at the beginning are handled by most mature ETL platforms.
1."Retrying" only the failed containers in SSIS for example can be achieve with variables and good use of the control flows and precedence constraints...
2. Scheduling is handled by SQL Agent.
3. Logging is available through the SSIS catalogue, etc..
Caricature when trying to explain why you should use tool A vs B is not always the best approach.
Not saying there isn't a use case for Airflow, but ETL tools are actually great if you know how to use them...and for a company that already has skills in a given ETL tool, moving to Airflow will probably be a waste of time.
can you send me the data?
HELP
itna haaf kyu rhe ho marathon run karke sikha rhe ho kya 42 km bhaage ho kya
bad sound
Can you please tell me the time line where there is a bad sound
Hi Sir, When I run this run docker-compose command , I am getting some Permission denied errors like this-
Error:
/snap/docker/2285/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.
from cryptography.hazmat.backends import default_backend
Traceback (most recent call last):
File "/snap/docker/2285/lib/python3.6/site-packages/urllib3/connectionpool.py", line 710, in urlopen
chunked=chunked,
File "/snap/docker/2285/lib/python3.6/site-packages/urllib3/connectionpool.py", line 398, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/snap/docker/2285/usr/lib/python3.6/http/client.py", line 1285, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/snap/docker/2285/usr/lib/python3.6/http/client.py", line 1331, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/snap/docker/2285/usr/lib/python3.6/http/client.py", line 1280, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/snap/docker/2285/usr/lib/python3.6/http/client.py", line 1046, in _send_output
self.send(msg)
File "/snap/docker/2285/usr/lib/python3.6/http/client.py", line 984, in send
self.connect()
File "/snap/docker/2285/lib/python3.6/site-packages/docker/transport/unixconn.py", line 30, in connect
sock.connect(self.unix_socket)
PermissionError: [Errno 13] Permission denied