How to download a S3 File from Lambda in Python | Step by Step Guide
ฝัง
- เผยแพร่เมื่อ 1 ส.ค. 2024
- In this video, I walk you through how to read a JSON file in S3 from a Lambda function with 3 easy steps. I start by taking note of the S3 bucket and key of our file. Secondly, I create a Lambda function with S3 Read permissions. Finally, I code up the Lambda function in Python.
Looking to get hands on experience building on AWS with a REAL project? Check out my course - The AWS Learning Accelerator! courses.beabetterdev.com/cour...
📚 MY RECOMMENDED READING LIST FOR SOFTWARE DEVELOPERS📚
Clean Code - amzn.to/37T7xdP
Clean Architecture - amzn.to/3sCEGCe
Head First Design Patterns - amzn.to/37WXAMy
Domain Driver Design - amzn.to/3aWSW2W
Code Complete - amzn.to/3ksQDrB
The Pragmatic Programmer - amzn.to/3uH4kaQ
Algorithms - amzn.to/3syvyP5
Working Effectively with Legacy Code - amzn.to/3kvMza7
Refactoring - amzn.to/3r6FQ8U
🎙 MY RECORDING EQUIPMENT 🎙
Shure SM58 Microphone - amzn.to/3r5Hrf9
Behringer UM2 Audio Interface - amzn.to/2MuEllM
XLR Cable - amzn.to/3uGyZFx
Acoustic Sound Absorbing Foam Panels - amzn.to/3ktIrY6
Desk Microphone Mount - amzn.to/3qXMVIO
Logitech C920s Webcam - amzn.to/303zGu9
Fujilm XS10 Camera - amzn.to/3uGa30E
Fujifilm XF 35mm F2 Lens - amzn.to/3rentPe
Neewer 2 Piece Studio Lights - amzn.to/3uyoa8p
💻 MY DESKTOP EQUIPMENT 💻
Dell 34 inch Ultrawide Monitor - amzn.to/2NJwph6
Autonomous ErgoChair 2 - bit.ly/2YzomEm
Autonomous SmartDesk 2 Standing Desk - bit.ly/2YzomEm
MX Master 3 Productivity Mouse - amzn.to/3aYwKVZ
Das Keyboard Prime 13 MX Brown Mechanical- amzn.to/3uH6VBF
Veikk A15 Drawing Tablet - amzn.to/3uBRWsN
📚 References:
Getting started with AWS: • Introduction to AWS | ...
S3 to Lambda Trigger - • AWS S3 File Upload + L...
Lambda Code - gist.github.com/djg07/8f8455c...
☁Topics covered include:
- S3 Bucket
- S3 Key
- S3 GetObject
- Lambda Function
- Lambda S3 GetObject
- Lambda Boto3
- Lambda Test Event
- Python JSON processing
🌎 Find me here:
Twitter - / beabetterdevv
Instagram - / beabetterdevv
Patreon - Donations help fund additional content - / beabetterdev
#AWS
#S3
#Lambda
![[TH] GEN vs SEN | FPX vs TH - VALORANT Champions 2024 DAY 1](http://i.ytimg.com/vi/VcyH-0LAlhs/mqdefault.jpg)
This was great. After days of searching your video was exactly what I was looking for thanks.
Glad it was helpful!
You saved my live, man. Thank !
Exactly what I needed!
Glad I was able to help ken!
Your videos are the best!! Keep going. Made my day better today hehe
You're very welcome Danilo! YOUR comment made MY day better! :D
Thank you sir for this tutorial you are a life saver ...❤️❤️
You're very welcome Otakugirl!
Well done man great job
Thank you Mr Beats!
Thanks!
Thanks so much Manuel!
I might be missing something, but this is a reading from the S3 file, not downloading the file. What if you want to actually download that or any other file to your local computer? Besides that, great videos, very concise and easy to understand.
I am also trying to explore how can I automatically download a file to my local machine from S3 using Lambda Trigger
The same concern I am looking how to download not reading
Love your videos man. Suggestion: GET request with API Gateway + Lambda + S3. Path Param of a file name and have lambda retrieve it from S3 and return it through API Gateway! :)
Thanks Daniel! Will add that to my list!
Hey, love the videos. Question: How different would the steps be if I wanted to upload/download an image file rather than JSON?
Hey there!
The process would be similar, you would just need to encode your image during upload, and decode it during download. I found an applicable tutorial that may be helpful for you (link below). It is using NodeJS but the concepts are the same. Hope this helps!
medium.com/think-serverless/image-upload-and-retrieval-from-s3-using-aws-api-gateway-and-lambda-b4c2961e8d1
Did you find how to do it using python? I m unable to do it. If you could help me, it will be great
Very good tutorial.
There is a CSV file in the local computer.
It is uploaded to S3.
Any changes made in the CSV file on the local computer, the changes should reflect in S3 automatically.
How to do that?
Can you do a video that takes this one step further and inserts the results of the JSON into an Aurora table?
Thanks for the suggestion chris. I'll add your request to my list of topics to cover. Cheers.
Hi can you also make a video on load this json record to a table
Hello everyone, I have a task that need to read the content of a file on s3 like this video but the problem is the file was compressed to the .gz type. I cannot read its content directly, Could you please help me what scenario should I use to read the file? I use nodejs, lambda and s3 service.
Hi, your videos are really helpful. I wanted to know how to copy files(csv) from s3 to a windows EC2 instance through a lambda function. What are the ways in which that could be done. I am very new to AWS.
Any help will be appreciated.
Thanks.
Hi Aatisha, Thank you for your kind words.
With regards to your question. Is this a one time operation? Or a functionality that you need to expose to your customers through an API? The approach you take will be different. Maybe if you can describe to me your use case and exactly what you are trying to do I can help more.
Cheers and thanks for watching.
Great , I have one question , Based on Put object on S3 , I want to trigger a Lambda function and that lambda will do some modification to object(type JSON) and will store modified object in another S3 bucket how I can do this ?
Thanks in advance
Hi sir I have some requirement.
How to automate the code to copy the S3 bucket to S3 glacier
My lambda function is unable to read file from S3 bucket which is private and have given s3fullaccess to lambda execution role
good afternoon,
how to combine multiple jason and insert into dynamo db using python.
Please help me on this. @Be A Better Dev
Can you please create a tutorial to copy the files from S3 to ec2 instance using lambda function. Thanks
Thanks Srikar for the suggestion - I've added it to many backlog.
is there any way I can copy files from s3 to sharepoint by using lambda?
Can you tell us how to extract the data using REST API from Lambda function (CSv or json)
Hi Vinooth, the data should be present in the event object that is passed into your lambda. A simple print of the object should show you how you can access the payload.
stupid question. When you click on JSON files in your s3 bucket, they open up in the browser. My computer wants to download them first before I open them. I'm running ubuntu20.04 and chrome and brave browser. Just wondering if you knew a quick solution.
Hi, If my file stored in specific path in s3, how am i going to specific the path to retrieve my file?
Hi Jun,
You would just use the nested path separated by / characters when specifying the Key.
I was hoping this was downloading a file from S3 using Lambda. How does one accomplish this?
how to read excel file
Very good video, but this is a reading from the S3 file, not downloading the file. I want to download my files to my local computer, is that possible?
Hi Vicky,
Lambda is a 'serverless' solution - so the machine that is reading the file from S3 is on the cloud.
To download the file to your local computer, you can use almost the exact same approach except run the code from your machine. This requires a little bit of setup (see some of my other videos on IAM roles/policies), but can be easily done.
Once you read the file from S3 like in the example above, you can write the file to your local disk. Here's an example I found that takes care of this last part: stackabuse.com/reading-and-writing-json-to-a-file-in-python/
Hope this helps.
@@BeABetterDev Oh wow thanks for your quick reply. I want to automate the process, manually I can go to the GUI, see the object in the bucket and download my object, but to automate this process I was thinking on a S3 event which triggers a lambda function that download the object to my "private data center"
Hi Vicky. For a use case like this, I can think of two possible options:
1) send the S3 put notification to an http endpoint that is hosted in your data center. This endpoint can be invoked and can be set up to download the file from S3 and save to a local destination.
2) same idea as 1, but make the SNS topic deliver the message to an sqs queue, and have an on premise machine poll the queue regularly for messages. Alternatively you can use a Cron job that runs periodically and processes all the messages.
I'm more familiar with #2 so it's probably what I would personally go with. However technically #1 seems simpler at face value.
Hope this helps !
Daniel
@@BeABetterDev very useful options, thanks so much for your quick reply and support, new follower
You're welcome !