วีดีโอ

Can you please provide the link of part 2, or is it still in progress?

hey brother in my ui dag is not possible

I was able to create the app and install the webhook on slack. Testing the API via POSTMAN was successful. However when I run the airflow job which is failing as intended there is no slack notification sent to the channel. I see the following error - "schannel: next InitializeSecurityContext failed" any idea? and how to solve

Nice! This was very helpful 👏

Hi, I am getting the following error: kubectl logs airflow-webserver-c6c5fd86f-pjcn5 Defaulted container "airflow-webserver-app" out of: airflow-webserver-app, wait-for-mysql (init) Unable to load the config, contains a configuration error. Traceback (most recent call last): File "/usr/local/lib/python3.8/pathlib.py", line 1288, in mkdir self._accessor.mkdir(self, mode) FileNotFoundError: [Errno 2] No such file or directory: '/opt/airflow/logs/scheduler/2024-06-27' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.8/logging/config.py", line 563, in configure handler = self.configure_handler(handlers[name]) File "/usr/local/lib/python3.8/logging/config.py", line 744, in configure_handler result = factory(**kwargs) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 47, in __init__ Path(self._get_log_directory()).mkdir(parents=True, exist_ok=True) File "/usr/local/lib/python3.8/pathlib.py", line 1292, in mkdir self.parent.mkdir(parents=True, exist_ok=True) File "/usr/local/lib/python3.8/pathlib.py", line 1288, in mkdir self._accessor.mkdir(self, mode) PermissionError: [Errno 13] Permission denied: '/opt/airflow/logs/scheduler' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/airflow/.local/bin/airflow", line 5, in <module> from airflow.__main__ import main File "/home/airflow/.local/lib/python3.8/site-packages/airflow/__init__.py", line 47, in <module> settings.initialize() File "/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py", line 561, in initialize LOGGING_CLASS_PATH = configure_logging() File "/home/airflow/.local/lib/python3.8/site-packages/airflow/logging_config.py", line 73, in configure_logging raise e File "/home/airflow/.local/lib/python3.8/site-packages/airflow/logging_config.py", line 68, in configure_logging dictConfig(logging_config) File "/usr/local/lib/python3.8/logging/config.py", line 808, in dictConfig dictConfigClass(config).configure() File "/usr/local/lib/python3.8/logging/config.py", line 570, in configure raise ValueError('Unable to configure handler ' ValueError: Unable to configure handler 'processor'

How should i integrate this with new relic?

Dask Executor ..??

I am trying to test multi-vpc private connectivity, created 2 VPCs, installed MSK in one and then enabled multi-vpc. Then I used managed VPC connections to connect to the cluster, but having difficulties connecting from a test instance over the private link It would be nice if you could create a video on that

Good starter, waiting for the series

Had no idea this was a thing.

when I run sudo docker build . -t airflow_dags:0.1.0 it return error ERROR: The EULA was not accepted. Installation aborted. dpkg: error processing archive /tmp/apt-dpkg-install-FuP4Yf/25-msodbcsql18_18.3.3.1-1_amd64.deb (--unpack): new msodbcsql18 package pre-installation script subprocess returned error exit status 1 Errors were encountered while processing: /tmp/apt-dpkg-install-FuP4Yf/25-msodbcsql18_18.3.3.1-1_amd64.deb E: Sub-process /usr/bin/dpkg returned an error code (1) how I can fix it? I use ubuntu

Very Useful

This was amazing. So easy to setup. Thanks a ton!!

I am having issue while implementing

how can I access airflow container when I am hosting to another server and add more dags?

I appreciate the tutorials. Sadly there are a few steps that make it impossible to run at the moment for a beginner. First the following error: Exception: Can not find valid pkg-config name. which can be solved by installing pkg-config Second in all your tutorials you somehow seem to magically mount your directories without any permission errors. It might be a mac thing that allows you to just mount everything without checking permissions? On linux or wsl you have to solve those issues. By either chown or chmod 777 on those shared directories. Lastly I now get a python ssl error when trying to connect to the mysql server....

The Visual Studio cursor animations when you indent and remove code are awesome lol never seen that before

404 not found (documentation not opening)

Can you suggest how do explain in interview, bcoz 3 times this question has asked interview

Amazing work! I've used it for my project. Thank you!

Hello, just a short improvement to have xcom availble ;) ```` from airflow.exceptions import AirflowException from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator from airflow.sensors.base import BaseSensorOperator class KubernetesPodSensor(BaseSensorOperator, KubernetesPodOperator): def __init__(self, do_xcom_push=False, *args, **kwargs): super().__init__(*args, **kwargs) self.do_xcom_push = do_xcom_push # def poke(self, context): # try: # KubernetesPodOperator.execute(self, context) # return True # except AirflowException: # self.log.error("KubernetesPodSensor failed: possibly because the sensor condition not met. " # "Will retry ...") # return False def poke(self, context): try: result = KubernetesPodOperator.execute(self, context) if self.do_xcom_push and result: context['task_instance'].xcom_push(key='return_value', value=result) return True except AirflowException: self.log.error("KubernetesPodSensor failed: possibly because the sensor condition not met. " "Will retry ...") return False ```

May i ask where did you create mysql:5.7 image?? Do you have a dockerfile for that, i can't find it in the githhub repo

nice video 🤌

When you say access, the pattern is typically "cross-account" access. Meaning that the publisher and the consumer are separate users on separate VPCs. The Consumer user may own the MSK cluster in its VPC. The Producer who is another AWS account on another VPC, needs to be able to access the cluster. Right?

Really very good explanation,Specially why do we need of it ..

Fantastic video, very easy to follow!

How are the worker nodes setup during the installation, is this config in the k8s YAML. Does the # of worker nodes remain static in the infra. If the K8s does load mgmt using Pods and workers remain constant, there is no cost savings in this approach that avoids idle compute costs. Am I getting it right?

Source code with explanation + results of working example. Nice!!! Thank you!

Hi, On failure callback is not working for I suppose because for me it’s not sending any notifications to the slack. The notification works when I use SlackWebhookOperator. Thank you

Excellent Video especially custom metrics

Someone correct me if I'm wrong, but can't you run a KPO from any airflow instance that has access to the kubernetes cluster via it's api? For example, I have access to a cluster on a dev machine, and can interact with it via the api (via kubectl). KPO should work just fine in a local airflow instance

Great video MaxcoTec. Do you have some video to show integrated airflow, k8s and argcocd? Would be incredible!!! Thanks.

Exception: Can not find valid pkg-config name. Specify MYSQLCLIENT_CFLAGS and MYSQLCLIENT_LDFLAGS env vars manually I am getting this error while I am trying to build image. Do you know any solution for that?

Thank you for the video. I have a similar requirement for one of my ongoing project where i have to setup up a Kafka consumer in python to consume from a MSK cluster topic in a different AWS account. Will that still be possible with this solution that you have demonstrated?

Nice explanation. Please make video on how to configure Kubernetes with Podman.

Thanks so much for explaining why we need airflow for some use cases! This is much appreciated.

Do you faced issue with parallelism. I have set airflow core parallelism to 512, max dag run is 1 max task run for one dag is 128. I have a default pool with 128 slot but still my airflow is not running more than the 32 tasks at time. I'm using helm chart to deploy the airflow.

Great going!

P R O M O S M 🤪

I have installed the docker now how to run on comand propmt not able to understand , I am on windows software

Thankyou everyone for liking our content. If you'll find issue "mysql" is unhealthy", find the fix here; github.com/maxcotec/Apache-Airflow/issues/2

Try even a more simpler approach on running airflow on Kubernets by using Rancher Desktop. This takes care of all kubernets deployment by doing all heavy lifting at back of the scene. Watch more here th-cam.com/video/eZBD-G3Ybhs/w-d-xo.html

thank you, now i got this )

What is the method of tracking the laser that you use?

nice and simple demonstration.. thanks

Nice video and good explanation

So if you have 2000+ dags you are saying to have 2000+ repos ? thats sound like nightmare to manage especially during airflow version upgrades when some variables/operators gets deprecated which needs to be updated in the repo for each dag. Imagine creating 2000 separate prs , one for each dag repo. Also wat about common operator/libs/utils which are used across sub set of dags and may have to updated which will mean updating code for each dag repo . If you have common repo say added as separate repo or part of base image and if some updates needs to be done .

Helpful as always 😊 Will you be able to do a scaled down miniature version of this CICD pipeline you mentioned in this video? Will be nice to see all the moving parts together. th-cam.com/video/a-4yRne3ba4/w-d-xo.html

Really good videos. Can you make one how to create a new spark job to a cloud Kubernetes like EKS cluster running Airflow? Bonus points if you can show how to dockerize, test it locally, deploy on QA environment & eventually promote it to production environment. This will mimic day to day work real closely. I understand it’s massive to undertake, but hope you get around to creating a series for this end to end scenario. Love your content!!