In this video, I'll show you how you can use a cool new tool by my colleague Kaxil to set up a local Airflow environment without Docker! github.com/kaxil/airflowctl/t...
Yesterday I needed to carry out a local test on my machine and ended up testing AirflowCTL, the installation was practical and quick, I even had time to create an architectural medallion using Spark.
Thank u so much for this dude, thought I’d have to do all the docket config stuff for my simple script to just extract from an API weekly and load the parquet files into Redshift
This is the quickest airflow installation I have EVER seen, It makes me a bit giddy, cant wait to try it out. You mentioned its not great for production scale, hows that?
Hahahah, def try it out! Mainly because it's pretty lightweight and only supports local executor, so trying to run a lot of DAG's this way will quickly fall apart.
Oh yeah definitely, since I don't think S3 supports unzipping a file within a bucket, you'd want to create a python task in airflow that uses the boto3 library to ingest the file, unzip it in a buffer or on some attached/on-disk storage if you're using a custom xcom backend that's big enough, before uploading the unzipped file into the second bucket! Check out this example below for the python task part of it! If you're still stuck, just let me know and I can help figure out what code you need! medium.com/@johnpaulhayes/how-extract-a-huge-zip-file-in-an-amazon-s3-bucket-by-using-aws-lambda-and-python-e32c6cf58f06
@@thedataguygeorge wow thank you so much! I was able to do it with buffer memory. 🫡😃. However, is there any limitation with buffer? I will be processing mainly around 20-30 mb sizes of file. It’s okay right ?
Docker gives you a more production-like environment, and is more stable when deployed for long periods of time. If you're planning to host your project, definitely would recommend using a docker image!
Can you help? whenever I try to initialize a new project I get error ScannerError: while scanning a double-quoted scalar in "Documents\airflow\test1\settings.yaml", line 10, column 16 expected escape sequence of 8 hexadecimal numbers, but found 's' in "Documents\airflow\test1\settings.yaml", line 10, column 21 I have looked at the yaml file and it seems fine, I even activated the venv and it worked okay, this is really bugging me
Would you mind showing me the file? Try with a fresh yaml file, sometimes characters can get encoded wrong and it can look fine while still causing issues.
On windows 10 I'm getting these Errors ERROR: Could not find a version that satisfies the requirement airlowctl (from versions: none) ERROR: No matching distribution found for airlowctl Any solution ?
If the error you posted here was copied directly from your terminal, it could be because you misspelled "airflowctl", I see airlowctl, could be the issue.
![อยู่ได้แล้ว - LIPTA feat. Mirrr [OFFICIAL MV]](http://i.ytimg.com/vi/-9-hpntNym4/mqdefault.jpg)
Yesterday I needed to carry out a local test on my machine and ended up testing AirflowCTL, the installation was practical and quick, I even had time to create an architectural medallion using Spark.
Wow awesome to hear, will pass the positive feedback along to the developers!
Thank u so much for this dude, thought I’d have to do all the docket config stuff for my simple script to just extract from an API weekly and load the parquet files into Redshift
docker*
No worries man, thats exactly what this approach is for! Get a nice easy set and forget local environment to run your jobs!
This is amazing! Thank you for the video 🙌
Thanks Superdoc!!
Excellent channel 👏
Thanks Harish, appreciate the love!
This is the quickest airflow installation I have EVER seen, It makes me a bit giddy, cant wait to try it out. You mentioned its not great for production scale, hows that?
Hahahah, def try it out! Mainly because it's pretty lightweight and only supports local executor, so trying to run a lot of DAG's this way will quickly fall apart.
Hey nice video. Btw need a suggestion, is there a way to unzip a file in s3 and move the unzipped file to another bucket without using lambda function
Oh yeah definitely, since I don't think S3 supports unzipping a file within a bucket, you'd want to create a python task in airflow that uses the boto3 library to ingest the file, unzip it in a buffer or on some attached/on-disk storage if you're using a custom xcom backend that's big enough, before uploading the unzipped file into the second bucket! Check out this example below for the python task part of it! If you're still stuck, just let me know and I can help figure out what code you need!
medium.com/@johnpaulhayes/how-extract-a-huge-zip-file-in-an-amazon-s3-bucket-by-using-aws-lambda-and-python-e32c6cf58f06
@@thedataguygeorge wow thank you so much! I was able to do it with buffer memory. 🫡😃. However, is there any limitation with buffer? I will be processing mainly around 20-30 mb sizes of file. It’s okay right ?
what is the difference using airflow without docker, and with docker? if my project just using airflow, so docker is not necessary?
Docker gives you a more production-like environment, and is more stable when deployed for long periods of time. If you're planning to host your project, definitely would recommend using a docker image!
Can you help? whenever I try to initialize a new project I get error
ScannerError: while scanning a double-quoted scalar
in "Documents\airflow\test1\settings.yaml", line 10, column 16
expected escape sequence of 8 hexadecimal numbers, but found 's'
in "Documents\airflow\test1\settings.yaml", line 10, column 21
I have looked at the yaml file and it seems fine, I even activated the venv and it worked okay, this is really bugging me
Would you mind showing me the file? Try with a fresh yaml file, sometimes characters can get encoded wrong and it can look fine while still causing issues.
i got the same issue how did you resolve it
which is best to run airflow on linux or docker?
I think docker is always best!
On windows 10 I'm getting these Errors
ERROR: Could not find a version that satisfies the requirement airlowctl (from versions: none)
ERROR: No matching distribution found for airlowctl
Any solution ?
I'm not sure it supports windows machines, so that might be why!
If the error you posted here was copied directly from your terminal, it could be because you misspelled "airflowctl", I see airlowctl, could be the issue.
@@danielopanubi2525 Now I'm using airflow directly on docker without any command line tool. It is better and easy
@@danielopanubi2525 Also as a data engineer you should know docker and containerization concepts
Man! Very, very cool! Muito Foda! In portuguese translation...Bom Demais...rsrs Thank u so much!
Um grande obrigado a você também! Muito amor do outro lado da lagoa!
Portugal is one of my all time favorite countries, so glad to hear I have supporters over there!