วีดีโอ

And what about installing packages with _PIP_ADDITIONAL_REQUIREMENTS in docker-compose.yaml directly?

amazing! Thanks for the share

I need your help. I downloaded Airflow version 2.10.2. When I select the connection type, it doesn't have Amazon S3 even though I have Amazon provider with version 8.28.0.

When I run the server and open the localhost link, There are no DAGS listed there. What could be the issue?

thank you for this tutorial

I think airflow installation is getting complicated nowadays for a developer with sound knowledge of infra , this video made my day

Excellent work. Thanks so much for the content.

Hi liked your video. Can you also upload the pdfs as well for our reference

How do I stop airflow webserwer?

Gonzalez Carol Gonzalez Melissa Miller Kevin

Hi, if xcom is not suitable for large data. How can we share data then between tasks? If it is mentioned in another video, could you tell me its number?

Good video, thanks for sharing! For anyone wondering how to run spark in Kubernetes i have a workshop/tutorial for it! th-cam.com/video/Vwhaq4ezmaw/w-d-xo.html

Crystal clear, thx!!!🎉

I am unable to see the log details in stdout and stderr. Kindly suggest

what can be done when // spark=SparkSession.builder.appName('Practise').getOrCreate() is keep on running // is keep on running and not getting executed.

[2024-09-14, 10:25:59 UTC] {taskinstance.py:441} ▼ Post task execution logs [2024-09-14, 10:25:59 UTC] {taskinstance.py:2905} ERROR - Task failed with exception Traceback (most recent call last): File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line 465, in _execute_task result = _execute_callable(context=context, **execute_callable_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line 432, in _execute_callable return execute_callable(context=context, **execute_callable_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/baseoperator.py", line 401, in wrapper return func(self, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/operators/python.py", line 235, in execute return_value = self.execute_callable() ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/operators/python.py", line 252, in execute_callable return self.python_callable(*self.op_args, **self.op_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/airflow/dags/dag_with_postgres_hooks.py", line 13, in postgres_to_s3 conn = hook.get_conn() ^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/postgres/hooks/postgres.py", line 175, in get_conn self.conn = psycopg2.connect(**conn_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/psycopg2/__init__.py", line 122, in connect conn = _connect(dsn, connection_factory=connection_factory, **kwasync) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc2 in position 83: invalid continuation byte [2024-09-14, 10:25:59 UTC] {taskinstance.py:1206} INFO - Marking task as FAILED. dag_id=dag_with_postgres_hooks_v01, task_id=postgres_to_s3, run_id=scheduled__2024-09-03T00:00:00+00:00, execution_date=20240903T000000, start_date=20240914T102555, end_date=20240914T102559 [2024-09-14, 10:25:59 UTC] {standard_task_runner.py:110} ERROR - Failed to execute job 144 for task postgres_to_s3 ('utf-8' codec can't decode byte 0xc2 in position 83: invalid continuation byte; 4317) [2024-09-14, 10:25:59 UTC] {local_task_job_runner.py:243} INFO - Task exited with return code 1

Thank you for this tutorial ! Magnificent !!

Thanks for the video, but I have a question, so are we using hooks here instead of the providers you showed before?

Mason Terrace

I found this to be an excellent guided tour, but only because I've gone through a lot of tutorials on webpages, books, and videos. PySpark is so vast and diverse that, after seeing all these other tutorials, one needs a video like this one to knit them all together.

great course, thanks a lot ! I have a slight problem though with my scheduler and webserver containers in the part 1:24:36 , my containers constantly keep switching between restarting and running status , so i am not able to access the webserver , i tried the steps again but i still can't solve the issue. Help would be appreciated

Great course look forward to watching more content

setup tutorial is officially obsolete. if you're reading this go and find another one lol

thank you sooo much this was really awesome

Do i need to create a new env after closing or restarting a session ?

Wasn't expecting a 50 min video to be this informative! Covered all the major topics related to PySpark. Truly, a well structured video and quality content. Thanks!

Wonderful course for beginners like me. Very helpful. Thanks a lot

I have an issue raise error_class(parsed_response, operation_name) botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden How do we resolve it?

If you are using minio local and after doing everything right, you still get this error "botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden". Ensure you send go to "Configuration" in Minio Local and set the "Server Location" to "us-east-1"

Where is the pyspark ML? you promised :'(

13:49 give error for me "spark not defined"

I am unable to get the jupyter notebooks to work. It keeps complaining about py4j. I am able to run the commands in a terminal tho. Not sure what the setup is supposed to be. I wish this video started with how to install everything.

Amazing videos

ERROR: failed to solve: process "/bin/bash -o pipefail -o errexit -o nounset -o nolog -c pip install --user --upgrade pip" did not complete successfully: exit code: 1 this error i get when i build image my script: FROM apache/airflow:2.9.3 COPY requirements.txt /requirements.txt RUN pip install --user --upgrade pip RUN pip install --no-cache-dir --user -r /requirements.txt

Though I have followed all the steps to create DAG with postgres operator, The DAG is not refelecting in the airflow page. Everything before that worked well. What will be the issue? from airflow.providers.postgres.operators.postgres import PostgresOperator somehwere this module is not present it seems. Any help on this will be useful

Thanks for the video. Anyone else got this error? ...{connection_wrapper.py:384} INFO - AWS Connection (conn_id='minio', conn_type='aws') credentials retrieved from login and password. ... ERROR - Task failed with exception ... raise EndpointConnectionError(endpoint_url=request.url, error=e) botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "localhost:9000/***/first_100_rows.csv" ... __________ For some reason it shows the url with *** instead of the bucket name. Maybe this has to do with it? Anyone else had this issue? After half a day, still haven't figured it out

Great videos, thank you very much for the help, it would be great if you continued with more explanations, such as how the daemon works or how to link a Jupyter Notebook to a job and show the logs of the Jupyter Notebook processes. thank you so much

I am trying to follow along, however, i got the error when running df.show(). I am running on windows and use jdk 21 version. Py4JJavaError: An error occurred while calling o42.showString. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0) (PIR-HRHNTW3-XZ.fios-router.home executor driver): org.apache.spark.SparkException: Python worker exited unexpectedly (crashed)

You saved me with the last clip. I had complicated docker-compose and couldn't figure out why the examples keep showing even though I turned everything off. The docker-compose file had the load examples hard coded to true. Duhhhh!~

how could I solve ERROR - Failed to establish connection to Docker host unix:///var/run/docker.sock: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory')) problem? I have tried docker version, unix:///var/run/docker.sock is running well.

This is the latest and most beginner friendly tutorial regarding apache spark. As an aspiring data engineering this is absolutely helpful! If you ever have free time, I would like to request creating a tutorial series regarding docker, dbt, and kafka, also if possible could you add more tutorials for dagster. As a student, out of all the tutorials in youtube I was able to follow you tutorial the best. Thank you for the airflow and pyspark playlist. I hope you still continue recording 😊

How can we setup this for multiple environments like Dev, Prod can you please guide us through?

I learned a lot with this video, thank you

Outstanding tutorial! I hope you can release a Kafka tutorial.

hello brotyer my example dags are visibling after set load_examples to false

is there another way to add these credentials to airflow without doing it from within the UI?

hi great author, i follow your steps from beginning. i have set the AIRFLOW_HOME and looks ok for this, but seems failed after airflow db init, could you please let me know? if i need to install db first or no? or do you have any suggestion? in minute 7:36 thank you so much broo

Noice 👌

Best Airflow tutorial ever👍

please chenage the voice setting you are sounding like a robot