Thank you so much for sharing your knowledge. I also want to add that the way you present the process was very satisfactory. I look forward to seeing more videos on the subject.
Wow, this a really straightforward solution! I was able to follow your tutorial easily and upload a file to AWS S3 without any trouble. Thank you for sharing your knowledge! I have one question though. Is it possible to upload a file to a specific folder within a bucket on AWS S3? If so, could you please explain the process of achieving this? I would greatly appreciate your guidance. Keep up the amazing work
Glad it helped! As for your question, I have not tried out specific folders, but looking into some threads I see that S3 is pretty much a flat storage structure. So in this thread, they mentioned including the full path as the file name...I would give this a try. (stackoverflow.com/questions/51214518/upload-files-to-a-specific-folder-in-the-bucket-with-aws-sdk). Essentially when specifying the filename, try passing the FULLPATH and see if that works.
@@letsfindsolutions @lucasvargasquintana8396 I was able to find the solution to this. Having the same configuration as in this video, you can provide the path with %2f. Eg: Suppose you want to upload to path bucket/folder1/folder2/test.txt, so you will be doing: bucket%2ffolder1%2ffolder/test.txt
This is really great video. I have few questions 1. Did you pass any token in header for validation. 2. I can give this url to upload file to anyone, whether he is a AWS user or not
1. There is no security around it so no token is needed. You can add cognito security with an authorizer to validate a token if needed. 2. Yes the url is public and open. It can be called like any other api publicly. They do not need to be anaws user.
Is there any specific reason for uploading data with a PUT request or does POST work just as fine? I'd like to send json objects to an S3 bucket in a HTTP post request, which is why I'm asking.
Why do you use the path override {bucket}/{filename} for your child resource {filename} if you already gave your endpoint these same structure /{bucket}/{filename}. An action name in your method of your {filename} would have been good enough right?
You could just use a library like axios or just use jquery ajax calls to the api. Keep in mind this expose your keys, so really its best to have it call your own back end layer which then calls your api to keep the keys hidden.
Hi @letsfindsolutions Your video is like, follow me. Crisp and up to the point. It took hardly 20min to reproduce what you have explained. Wish to see more video from you.
I am glad you like the video. As for your question I have never really tried using utf8. I think as long as you just include all the media types it can handle any file encoding you want to handle.
Im really grateful for this tutorial that you created. Also may I ask if how can I use it or the api so I can use it in php and upload files from the local storage into s3 bucket instead of the server side?
You can have your php grab files from local storage and call the api...similar to having a web page with a file selector and doing ajax call to the api to upload the file.
I'm struggling to upload multipart/form-data to a s3 bucket and the metadata to dynamoDB. Suprisingly there is very few info on this, or maybe i didnt look in the right places. ChatGPT was also not very useful and after some time i realised maybe HTTP api gateway cannot do this and REST api must be used. Kindly let me know if this is the case
Hii Its a great video. Just one quick question I'm trying to automate the azure cost report to store in s3 bucket and blob container using the logic apps. I had completed by till generating the report to blob container but from the documentation provided by the Microsoft S3 bucket they don't have post method. Is that something can we use in logic apps the api method to trigger and uploads the file in s3 bucket too? Can you please help me out
Can you clarify your question. Are you looking to trigger s3 storage via api call from Microsoft Azure blob storage? I have not done much cross integration but I'm sure it's possible.
@@letsfindsolutions thanks for your response. I fixed it by using the api gateway endpoint url in using logic apps http trigger to upload the file when it triggers. Thanks for your support and pinging me back.
Thanks a lot for this video. I've a question about authentication. Here, the REST API call from postman worked as we have given the IAM role with putobj policy. How about if I have to restrict the upload to a specific user? Say, the upload happens from a custom web application, the user uploads the file, is there a way to include IAM user authentication here? Could you please give any pointers to this?
You could use a lambda authorizer (formerly known as custom authorizer). Here you have full control of the auth logic and can validate your keys against specific groups or users.
Thanks for this solution. I have implemented this for MS excel file upload but the size got change every time. Can you please help me out with the solution for that. Facing the issue with format change and file size got increased during upload.
i want to do OTA update by giving this the url of the uploaded firmware file. is this method ok for that? what happened if I just upload the binary file?
Thanks it works for me, now problem i am facing is, once file upload to s3 bucket i can see file, but when i use to open that file with object url than my file is not opening, its giving me error like access denied so let me know how to give public writes to view that file simultaneously.
It is uploading the file into but there are some issues m facing: 1) even if am specifying the content type in Binary-media-type (audio/mp3), still m able to upload any file 2) the file gets corrupted when it's uploaded I tried form postman and even tried using flutter
What are you using to test the upload? In the video I use postman and i believe all its doing is converting the data to base64 encoded binary data. Similar to what a web browser would do.
While Edit integration request to add URL path parameters when click save it showing Network Failure( Anyone help Currently I am Using ap-south-1 Region ) i don't know what is the problem ??
i am also facing same problem while creating resource in API gateway can't add brackets with bucket in resource path, same with filename , my resource path is not taking brackets or I can say anything . error invalid mapping expression parameter specified
I followed the steps precisely, but I am encountering the following error: "message": "Missing Authentication Token". I have verified the steps multiple times. May I know what could be the reason for this error?
@@letsfindsolutions there's no code..i followed your steps.. m getting a WCL error on the screen tho,,,could it be the issue? or it seems like s3 access isue
With api gateway you can enable an Authorizer and have it validate a token i the header. You can setup cognito to work with the authorizer to validate the tokens.
Hi all, I'm pretty new. I followed all these steps, made sure that my S3 and API endpoint are in the same region and yet, I still keep getting this error: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint I'm trying to upload a JSON file. Have any of you had this issue?
i am also facing same problem while creating resource in API gateway can't add brackets with bucket in resource path, same with filename , my resource path is not taking brackets or I can say anything . error invalid mapping expression parameter specified can you help me
I do same thing. but when i fill url parameter that not accept it gives me error. I seen many videos but i cant find that soloution . and also aws foramte changed. error is ---Invalid mapping expression specified: Validation Result: warnings : [], errors : [Invalid mapping expression parameter specified: method.request.path.bucket]
This was incredible. It would have taken approximately 1 million years to figure this out by reading the docs alone. You're a hero.
Glad you found it helpful
Glad you found it helpful
one of the smoothest tutorials. i'm shaking your hand virtually!
Happy to hear that!
Thank you so much for sharing your knowledge. I also want to add that the way you present the process was very satisfactory. I look forward to seeing more videos on the subject.
I have been busy lately, but I plan on doing another video in the next couple of weeks. Thanks for the encouraging comment.
nice, no nonsense, no silly accent, 10/10 thank you sir
Thank you kindly
Just curious, what's a silly accent?
@@RebornLegacy indian accent
@@zishanchaudhary221
Ohhh, so you guys are neanderthals complaining about free instructional material.
Got it
Well I daresay ye've earned yourself a new subscriber. Excellent video.
Awesome, thank you!
I fell over around the eight minute mark. Hope you upload this video again for the newer ui
Exactly what i was looking for.. Thanks a lot
Glad it helped
AWS. Quite possibly the most convoluted processes ever invented.
I have been working with Azure lately also, but yes their are nuances to each cloud platform that take getting used to.
Wow, this a really straightforward solution! I was able to follow your tutorial easily and upload a file to AWS S3 without any trouble. Thank you for sharing your knowledge!
I have one question though. Is it possible to upload a file to a specific folder within a bucket on AWS S3? If so, could you please explain the process of achieving this? I would greatly appreciate your guidance. Keep up the amazing work
Glad it helped! As for your question, I have not tried out specific folders, but looking into some threads I see that S3 is pretty much a flat storage structure. So in this thread, they mentioned including the full path as the file name...I would give this a try. (stackoverflow.com/questions/51214518/upload-files-to-a-specific-folder-in-the-bucket-with-aws-sdk). Essentially when specifying the filename, try passing the FULLPATH and see if that works.
@@letsfindsolutions @lucasvargasquintana8396 I was able to find the solution to this. Having the same configuration as in this video, you can provide the path with %2f. Eg: Suppose you want to upload to path bucket/folder1/folder2/test.txt, so you will be doing: bucket%2ffolder1%2ffolder/test.txt
@@sayedimran5972 Thanks for sharing
@@sayedimran5972 thank you! easy and working well
That was really helpfull, thanks for sharing!@@sayedimran5972
This is really great video. I have few questions
1. Did you pass any token in header for validation.
2. I can give this url to upload file to anyone, whether he is a AWS user or not
1. There is no security around it so no token is needed. You can add cognito security with an authorizer to validate a token if needed.
2. Yes the url is public and open. It can be called like any other api publicly. They do not need to be anaws user.
@@letsfindsolutions many thanks.
I am glad you found the video helpful...I really do plan to make more...just haven't found the time to lately.
Is there any specific reason for uploading data with a PUT request or does POST work just as fine?
I'd like to send json objects to an S3 bucket in a HTTP post request, which is why I'm asking.
Thank you. This was very informative. Will this video help me in uploading a Json file into aws?
Yes, you can upload any file type into the s3 bucket.
Why do you use the path override {bucket}/{filename} for your child resource {filename} if you already gave your endpoint these same structure /{bucket}/{filename}. An action name in your method of your {filename} would have been good enough right?
I guess just for demo purposes...but good point.
sir , you are a life saver , thank you
Glad you found it helpful.
Thank you man
Actually i am trying this its works on postman ,but i want to use API gateway endpoint in my JS code
can you please suggest me .
You could just use a library like axios or just use jquery ajax calls to the api. Keep in mind this expose your keys, so really its best to have it call your own back end layer which then calls your api to keep the keys hidden.
So insightful! Thank you.
Thanks
Hi @letsfindsolutions
Your video is like, follow me. Crisp and up to the point. It took hardly 20min to reproduce what you have explained.
Wish to see more video from you.
Will eventually add more when I get time.
thanks for the great video,
I'm starting a project where I'll need to use S3, why use the API Gateway and not S3's own REST API?
I had a use case to use a custom authorizer, so api gateway allows for this type of configuration.
Hey Thank You for the wonderful explanation, just had one question, is it possible to pass "chacterset=utf-8" while uploading file
I am glad you like the video. As for your question I have never really tried using utf8. I think as long as you just include all the media types it can handle any file encoding you want to handle.
Im really grateful for this tutorial that you created.
Also may I ask if how can I use it or the api so I can use it in php and upload files from the local storage into s3 bucket instead of the server side?
You can have your php grab files from local storage and call the api...similar to having a web page with a file selector and doing ajax call to the api to upload the file.
I'm struggling to upload multipart/form-data to a s3 bucket and the metadata to dynamoDB. Suprisingly there is very few info on this, or maybe i didnt look in the right places. ChatGPT was also not very useful and after some time i realised maybe HTTP api gateway cannot do this and REST api must be used. Kindly let me know if this is the case
I would do REST api with form post data.
Hii
Its a great video. Just one quick question I'm trying to automate the azure cost report to store in s3 bucket and blob container using the logic apps. I had completed by till generating the report to blob container but from the documentation provided by the Microsoft S3 bucket they don't have post method. Is that something can we use in logic apps the api method to trigger and uploads the file in s3 bucket too? Can you please help me out
Can you clarify your question. Are you looking to trigger s3 storage via api call from Microsoft Azure blob storage? I have not done much cross integration but I'm sure it's possible.
@@letsfindsolutions Yes, I want to trigger the api gateway endpoint to send file to aws s3 bucket from azure
You may need to write another lambda function to do that.
@@letsfindsolutions thanks for your response. I fixed it by using the api gateway endpoint url in using logic apps http trigger to upload the file when it triggers. Thanks for your support and pinging me back.
Can I use PowerAutomate to call Postman and follow all the steps you show us?
If you mean have postman call the api you setup, then yes that is possible.
Does it restrict the filesize to AWS Gateway file size limit?
I'm not sure, but that would make sense
Thanks a lot for this video. I've a question about authentication. Here, the REST API call from postman worked as we have given the IAM role with putobj policy. How about if I have to restrict the upload to a specific user? Say, the upload happens from a custom web application, the user uploads the file, is there a way to include IAM user authentication here? Could you please give any pointers to this?
You could use a lambda authorizer (formerly known as custom authorizer). Here you have full control of the auth logic and can validate your keys against specific groups or users.
@@letsfindsolutionsThank you!
Thanks for this solution. I have implemented this for MS excel file upload but the size got change every time. Can you please help me out with the solution for that. Facing the issue with format change and file size got increased during upload.
Did you follow the video...what media types have you defined?
@letsfindsolutions I have allowed all media types. But only for Excel, it got format change happens
Maybe the encoding ia somehow getting changed on upload...excel would be similar to other binary files.
Absolutely the content am looking for..
I also would like to know how to write custom response body in postman when status code is "200 OK"
You would need to write a custom lambda for that.
great video and thanks for sharing. I would like to upload via browser instead of postman, is there any way to api via browser ?
Yes, you would need to build a front end ui...like a react ui web page that can make calls to the api similar to postman.
@@letsfindsolutions can you tell how to do that in Vue.js
, i tried a lot, didnt get any solution
i want to do OTA update by giving this the url of the uploaded firmware file. is this method ok for that? what happened if I just upload the binary file?
This methid requires a valid jwt token...so as long as you can authenticate and pass along a token this will work.
good one, thank you
Glad you liked it!
Thanks it works for me, now problem i am facing is, once file upload to s3 bucket i can see file, but when i use to open that file with object url than my file is not opening, its giving me error like access denied so let me know how to give public writes to view that file simultaneously.
I believe you will need to update the permissions of the s3 bucket to allow for public read.
@@letsfindsolutions Thanks i done yesterday only by myself by creating policy
was there a change in AWS UI, I am having a problem creating api resource as path parameter using {}?
Yes they changed the ui a while back
@@letsfindsolutions so can we proceed without giving {bucket}, kindly confirm will it work
@@venkat-0412 I have not tried it.
It is uploading the file into but there are some issues m facing:
1) even if am specifying the content type in Binary-media-type (audio/mp3), still m able to upload any file
2) the file gets corrupted when it's uploaded
I tried form postman and even tried using flutter
Possibly the encoding is messing up somehow...aside from that i. Not really sure what else it could be.
genioooooo!!!, gracias loco
Gracias
what are configuration for your stage of dev?
And my problem is that file successfully upload but size is 0kb so any solution?
What are you using to test the upload? In the video I use postman and i believe all its doing is converting the data to base64 encoded binary data. Similar to what a web browser would do.
Thanks
@@letsfindsolutions
How would it be via POST, to send file via base64 string?
I think you can change the method to a POST and just add in additional media types to handle the base64 encoded data being passed.
How can I get the content of the file in the bucket using the api?
I mean GET method
You would need to write a new get method to read and serve back the content.
While Edit integration request to add URL path parameters when click save it showing Network Failure( Anyone help Currently I am Using ap-south-1 Region ) i don't know what is the problem ??
Not sure im understanding the iasue...but maybe try switching regions. I usually use us east.
@@letsfindsolutions I tried us east also but it's not working in api gateway
i am also facing same problem while creating resource in API gateway can't add brackets with bucket in resource path, same with filename , my resource path is not taking brackets or I can say anything . error invalid mapping expression parameter specified
I resolve my problem put path in single inverted comma '' that will work as i am not getting option to put curly brackets
I followed the steps precisely, but I am encountering the following error: "message": "Missing Authentication Token". I have verified the steps multiple times. May I know what could be the reason for this error?
Are you passing the token in the Authorization header?
@@letsfindsolutions No, I have not.
The token needs to be passed in the Auth header
i am getting internal server error while calling the api.
is it something to do with WAF or ListWebACLs?
Internal server error leads me to believe maybe code related...did you modify the sample code?
@@letsfindsolutions there's no code..i followed your steps.. m getting a WCL error on the screen tho,,,could it be the issue? or it seems like s3 access isue
Ya could be s3 permission issue
I don't have the "Create policy" button from 2:37 .. any ideas why?
They changed the ui a while back...it might look different now.
you can just go to policy and create it instead of trying to do it from roles
It should return the link for access the file
That can be customized if needed
Does anyone know how to do the same with Terraform?
I sure there are some yaml samples out there. Unfortunately I don't have any.
In this method, can anybody in the world upload to my bucket? What if i wanted to restrict it to just myself? Using an access token or api key?
With api gateway you can enable an Authorizer and have it validate a token i the header. You can setup cognito to work with the authorizer to validate the tokens.
Look at my other video about jwt authorizer....you would use the same approach there.
Hi all, I'm pretty new. I followed all these steps, made sure that my S3 and API endpoint are in the same region and yet, I still keep getting this error: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint
I'm trying to upload a JSON file. Have any of you had this issue?
I have never encountered this issue. In readibg your error, this looks to be endpoint uri related or something wrong with the endpoint permissions.
@@letsfindsolutions Hi, it was something with the permissions :) Thanks for your reply, I was able to resolve it
Np
When I give a path resource, it does not get it
Maybe try recreating it. Sometimes that helps.
@@letsfindsolutions by the way thak its help
i am also facing same problem while creating resource in API gateway can't add brackets with bucket in resource path, same with filename , my resource path is not taking brackets or I can say anything . error invalid mapping expression parameter specified
can you help me
@@gurpreetkaur-cw8woWhen creating a resource, simply write the name within {} brackets, and it will work.
i am getting "Internal server error"
I know the ui looks different, but my guess is that this is most likely configuration related
no it did not work time wasted
Sorry to hear it did not work for you.
I do same thing. but when i fill url parameter that not accept it gives me error.
I seen many videos but i cant find that soloution .
and also aws foramte changed.
error is ---Invalid mapping expression specified: Validation Result: warnings : [], errors : [Invalid mapping expression parameter specified: method.request.path.bucket]
Unfortunately you are right about aws ui changes. Without seeing the issue it's hard for me to guide you.