Tutorial: How to upload files into a S3 Bucket using a REST API via API Gateway
ฝัง
- เผยแพร่เมื่อ 25 ก.พ. 2023
- A quick tutorial on how to use API Gateway to create a REST API to upload files into a S3 Bucket
Topics Covered:
1) S3 Bucket Creation
2) IAM Role Creation
3) IAM Policy Creation
4) API Gateway Creation
5) REST API Creation
This was incredible. It would have taken approximately 1 million years to figure this out by reading the docs alone. You're a hero.
Glad you found it helpful
Glad you found it helpful
Thank you so much for sharing your knowledge. I also want to add that the way you present the process was very satisfactory. I look forward to seeing more videos on the subject.
I have been busy lately, but I plan on doing another video in the next couple of weeks. Thanks for the encouraging comment.
So insightful! Thank you.
Thanks
nice, no nonsense, no silly accent, 10/10 thank you sir
Thank you kindly
Just curious, what's a silly accent?
@@RebornLegacy indian accent
@@zishanchaudhary221
Ohhh, so you guys are neanderthals complaining about free instructional material.
Got it
sir , you are a life saver , thank you
Glad you found it helpful.
genioooooo!!!, gracias loco
Gracias
good one, thank you
Glad you liked it!
Hi @letsfindsolutions
Your video is like, follow me. Crisp and up to the point. It took hardly 20min to reproduce what you have explained.
Wish to see more video from you.
Will eventually add more when I get time.
Why do you use the path override {bucket}/{filename} for your child resource {filename} if you already gave your endpoint these same structure /{bucket}/{filename}. An action name in your method of your {filename} would have been good enough right?
Wow, this a really straightforward solution! I was able to follow your tutorial easily and upload a file to AWS S3 without any trouble. Thank you for sharing your knowledge!
I have one question though. Is it possible to upload a file to a specific folder within a bucket on AWS S3? If so, could you please explain the process of achieving this? I would greatly appreciate your guidance. Keep up the amazing work
Glad it helped! As for your question, I have not tried out specific folders, but looking into some threads I see that S3 is pretty much a flat storage structure. So in this thread, they mentioned including the full path as the file name...I would give this a try. (stackoverflow.com/questions/51214518/upload-files-to-a-specific-folder-in-the-bucket-with-aws-sdk). Essentially when specifying the filename, try passing the FULLPATH and see if that works.
@@letsfindsolutions @lucasvargasquintana8396 I was able to find the solution to this. Having the same configuration as in this video, you can provide the path with %2f. Eg: Suppose you want to upload to path bucket/folder1/folder2/test.txt, so you will be doing: bucket%2ffolder1%2ffolder/test.txt
@@sayedimran5972 Thanks for sharing
@@sayedimran5972 thank you! easy and working well
That was really helpfull, thanks for sharing!@@sayedimran5972
Thank you. This was very informative. Will this video help me in uploading a Json file into aws?
Yes, you can upload any file type into the s3 bucket.
Im really grateful for this tutorial that you created.
Also may I ask if how can I use it or the api so I can use it in php and upload files from the local storage into s3 bucket instead of the server side?
You can have your php grab files from local storage and call the api...similar to having a web page with a file selector and doing ajax call to the api to upload the file.
Hey Thank You for the wonderful explanation, just had one question, is it possible to pass "chacterset=utf-8" while uploading file
I am glad you like the video. As for your question I have never really tried using utf8. I think as long as you just include all the media types it can handle any file encoding you want to handle.
Absolutely the content am looking for..
I also would like to know how to write custom response body in postman when status code is "200 OK"
You would need to write a custom lambda for that.
This is really great video. I have few questions
1. Did you pass any token in header for validation.
2. I can give this url to upload file to anyone, whether he is a AWS user or not
1. There is no security around it so no token is needed. You can add cognito security with an authorizer to validate a token if needed.
2. Yes the url is public and open. It can be called like any other api publicly. They do not need to be anaws user.
@@letsfindsolutions many thanks.
I am glad you found the video helpful...I really do plan to make more...just haven't found the time to lately.
Thank you man
Actually i am trying this its works on postman ,but i want to use API gateway endpoint in my JS code
can you please suggest me .
You could just use a library like axios or just use jquery ajax calls to the api. Keep in mind this expose your keys, so really its best to have it call your own back end layer which then calls your api to keep the keys hidden.
thanks for the great video,
I'm starting a project where I'll need to use S3, why use the API Gateway and not S3's own REST API?
I had a use case to use a custom authorizer, so api gateway allows for this type of configuration.
i want to do OTA update by giving this the url of the uploaded firmware file. is this method ok for that? what happened if I just upload the binary file?
This methid requires a valid jwt token...so as long as you can authenticate and pass along a token this will work.
was there a change in AWS UI, I am having a problem creating api resource as path parameter using {}?
Yes they changed the ui a while back
Does anyone know how to do the same with Terraform?
Thanks a lot for this video. I've a question about authentication. Here, the REST API call from postman worked as we have given the IAM role with putobj policy. How about if I have to restrict the upload to a specific user? Say, the upload happens from a custom web application, the user uploads the file, is there a way to include IAM user authentication here? Could you please give any pointers to this?
You could use a lambda authorizer (formerly known as custom authorizer). Here you have full control of the auth logic and can validate your keys against specific groups or users.
@@letsfindsolutionsThank you!
It is uploading the file into but there are some issues m facing:
1) even if am specifying the content type in Binary-media-type (audio/mp3), still m able to upload any file
2) the file gets corrupted when it's uploaded
I tried form postman and even tried using flutter
Possibly the encoding is messing up somehow...aside from that i. Not really sure what else it could be.
Thanks for this solution. I have implemented this for MS excel file upload but the size got change every time. Can you please help me out with the solution for that. Facing the issue with format change and file size got increased during upload.
Did you follow the video...what media types have you defined?
@letsfindsolutions I have allowed all media types. But only for Excel, it got format change happens
Maybe the encoding ia somehow getting changed on upload...excel would be similar to other binary files.
Thanks it works for me, now problem i am facing is, once file upload to s3 bucket i can see file, but when i use to open that file with object url than my file is not opening, its giving me error like access denied so let me know how to give public writes to view that file simultaneously.
I believe you will need to update the permissions of the s3 bucket to allow for public read.
@@letsfindsolutions Thanks i done yesterday only by myself by creating policy
great video and thanks for sharing. I would like to upload via browser instead of postman, is there any way to api via browser ?
Yes, you would need to build a front end ui...like a react ui web page that can make calls to the api similar to postman.
@@letsfindsolutions can you tell how to do that in Vue.js
, i tried a lot, didnt get any solution
I don't have the "Create policy" button from 2:37 .. any ideas why?
They changed the ui a while back...it might look different now.
How would it be via POST, to send file via base64 string?
I think you can change the method to a POST and just add in additional media types to handle the base64 encoded data being passed.
what are configuration for your stage of dev?
And my problem is that file successfully upload but size is 0kb so any solution?
What are you using to test the upload? In the video I use postman and i believe all its doing is converting the data to base64 encoded binary data. Similar to what a web browser would do.
Thanks
@@letsfindsolutions
While Edit integration request to add URL path parameters when click save it showing Network Failure( Anyone help Currently I am Using ap-south-1 Region ) i don't know what is the problem ??
Not sure im understanding the iasue...but maybe try switching regions. I usually use us east.
@@letsfindsolutions I tried us east also but it's not working in api gateway
i am also facing same problem while creating resource in API gateway can't add brackets with bucket in resource path, same with filename , my resource path is not taking brackets or I can say anything . error invalid mapping expression parameter specified
I resolve my problem put path in single inverted comma '' that will work as i am not getting option to put curly brackets
How can I get the content of the file in the bucket using the api?
I mean GET method
You would need to write a new get method to read and serve back the content.
In this method, can anybody in the world upload to my bucket? What if i wanted to restrict it to just myself? Using an access token or api key?
With api gateway you can enable an Authorizer and have it validate a token i the header. You can setup cognito to work with the authorizer to validate the tokens.
Look at my other video about jwt authorizer....you would use the same approach there.
Hi all, I'm pretty new. I followed all these steps, made sure that my S3 and API endpoint are in the same region and yet, I still keep getting this error: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint
I'm trying to upload a JSON file. Have any of you had this issue?
I have never encountered this issue. In readibg your error, this looks to be endpoint uri related or something wrong with the endpoint permissions.
@@letsfindsolutions Hi, it was something with the permissions :) Thanks for your reply, I was able to resolve it
It should return the link for access the file
That can be customized if needed
When I give a path resource, it does not get it
Maybe try recreating it. Sometimes that helps.
@@letsfindsolutions by the way thak its help
i am also facing same problem while creating resource in API gateway can't add brackets with bucket in resource path, same with filename , my resource path is not taking brackets or I can say anything . error invalid mapping expression parameter specified
can you help me
@@gurpreetkaur-cw8woWhen creating a resource, simply write the name within {} brackets, and it will work.
no it did not work time wasted
Sorry to hear it did not work for you.
i am getting internal server error while calling the api.
is it something to do with WAF or ListWebACLs?
Internal server error leads me to believe maybe code related...did you modify the sample code?
@@letsfindsolutions there's no code..i followed your steps.. m getting a WCL error on the screen tho,,,could it be the issue? or it seems like s3 access isue
Ya could be s3 permission issue