This was extremely helpful! I really like that you are able to compress such valuable information in just 8 mins! I think it would be really useful to see how to build an ETL pipeline in a IaC framework. Haven't see many on the web! Thanks!
Subscribed!!! Thank you so much for the great content!! Can you please make dedicated videos on how to use AWS Glue, Triggers, Lambda functions and Athena for ETL pipeline?
Thanks great video! Other examples I have seen used a crawler to write the schema of the redshift table to the data catalog before loading using a Glue Job. If I just wanted to do this using only a Visual Glue Job and without a crawler, is it possible?
I am not exactly sure what are you trying to ask. The crawler, crawls the data and infer schema from it. Do you mean to say you want to infer schema without the crawler? Here is a good read on how a crawler infer schema from AWS. repost.aws/knowledge-center/glue-crawler-detect-schema
It depends on your data size. If you are dealing with small data size then you can trigger the api via lambda and ingest it in your destination service. Otherwise, you need to stage it to s3.
@@BiInsightsInc So if the data size is not that big, do you recommend me to do the ETL with lambda correctly? and what do you think about AWS Step Functions? thanks again for the content
Hi - thanks for such concise content! I noticed that you deployed to S3 without debugging locally. Suppose i wanted to test the etl script before deploying it? is there a way to execute the etl.py script on the local host using aws_cli?
Yes, you can set up a local development environment to test your work prior to deploying it to AWS. I haven’t covered it but here is an article to get you started. medium.com/@bezdelev/how-to-test-a-python-aws-lambda-function-locally-with-pycharm-run-configurations-6de8efc4b206
@@BiInsightsInc Thank you. In your view, what is the best practice? Using a terminal, lambda console or the procedure in that link you posted? I'm a big time NB
@@joegenshlea6827 I would go with the CLI, I have used the AWS CLI to test code locally. Along with AWS CLI AWS provides SAM CLI to test the AWS Lambda function locally. The Lambda developer guide also advocates for this approach.
@@BiInsightsInc Thank you again! I think I understand. Maybe a video idea for you! I like your videos because you skip all the superfluous nonsense and get right to the meat and potatoes. Good work.
Hi KK, you can get the Glue embedded connection details in Python with boto3. Hope this helps. glue = boto3.client('glue', region_name='us-east-1') # get the connection response = glue.get_connection( Name='name-of-embedded-connection', HidePassword=False ) # get specific connection args response['Connection']['ConnectionProperties']['USERNAME'] response['Connection']['ConnectionProperties']['PASSWORD']
Hi Pawan, you can transfer large datasets with AWS Glue. It’s is a distributed platform, uses Spark behind the scene, to process big data. Lambda is for small to medium size datasets.
Link to AWS Playlist: th-cam.com/video/cFO2-gs56d8/w-d-xo.html
This was extremely helpful! I really like that you are able to compress such valuable information in just 8 mins! I think it would be really useful to see how to build an ETL pipeline in a IaC framework. Haven't see many on the web! Thanks!
You're a hero for the well explained content and then answering everyone's comments. :)
Thanks for the motivation!
Subscribed!!! Thank you so much for the great content!! Can you please make dedicated videos on how to use AWS Glue, Triggers, Lambda functions and Athena for ETL pipeline?
Thanks great video! Other examples I have seen used a crawler to write the schema of the redshift table to the data catalog before loading using a Glue Job.
If I just wanted to do this using only a Visual Glue Job and without a crawler, is it possible?
I am not exactly sure what are you trying to ask. The crawler, crawls the data and infer schema from it. Do you mean to say you want to infer schema without the crawler?
Here is a good read on how a crawler infer schema from AWS.
repost.aws/knowledge-center/glue-crawler-detect-schema
thanks!!! but I have a question, If my data comes from an API, is S3 not necessary?
It depends on your data size. If you are dealing with small data size then you can trigger the api via lambda and ingest it in your destination service. Otherwise, you need to stage it to s3.
@@BiInsightsInc So if the data size is not that big, do you recommend me to do the ETL with lambda correctly? and what do you think about AWS Step Functions? thanks again for the content
Hi - thanks for such concise content! I noticed that you deployed to S3 without debugging locally. Suppose i wanted to test the etl script before deploying it? is there a way to execute the etl.py script on the local host using aws_cli?
Yes, you can set up a local development environment to test your work prior to deploying it to AWS. I haven’t covered it but here is an article to get you started.
medium.com/@bezdelev/how-to-test-a-python-aws-lambda-function-locally-with-pycharm-run-configurations-6de8efc4b206
@@BiInsightsInc Thank you. In your view, what is the best practice? Using a terminal, lambda console or the procedure in that link you posted? I'm a big time NB
@@joegenshlea6827 I would go with the CLI, I have used the AWS CLI to test code locally. Along with AWS CLI AWS provides SAM CLI to test the AWS Lambda function locally. The Lambda developer guide also advocates for this approach.
@@BiInsightsInc Thank you again! I think I understand. Maybe a video idea for you! I like your videos because you skip all the superfluous nonsense and get right to the meat and potatoes. Good work.
Hi, how can we read the credentials from connections or secrets in aws glue python shell, it not working for me
Hi KK, you can get the Glue embedded connection details in Python with boto3. Hope this helps.
glue = boto3.client('glue', region_name='us-east-1')
# get the connection
response = glue.get_connection(
Name='name-of-embedded-connection',
HidePassword=False
)
# get specific connection args
response['Connection']['ConnectionProperties']['USERNAME']
response['Connection']['ConnectionProperties']['PASSWORD']
Hi, Can we transfer 1Tb data from s3 to Redshift using Glue or Lambda +Glue ?
Hi Pawan, you can transfer large datasets with AWS Glue. It’s is a distributed platform, uses Spark behind the scene, to process big data. Lambda is for small to medium size datasets.