API GATEWAY DEMO | Create API ENDPOINT to fetch S3 DATA using LAMBDA FUNCTION | PYTHON BOTO3
ฝัง
- เผยแพร่เมื่อ 12 ก.ย. 2020
- Today we will work on a small project where we will understand how we can use AWS API GATEWAYS along with AWS LAMBDA to fetch data from AWS S3
USERS at TEAM A were uploading files to AWS S3 Bucket, and the users at TEAM B used to download or use them directly from the bucket.
Post which there was a requirement that all the data that, the users are going to download should be base 64 encoded.
⭐ Kite is a free AI-powered coding assistant that will help you code faster and smarter. The Kite plugin integrates with all the top editors and IDEs to give you smart completions and documentation while you’re typing. I've been using Kite for 6 months and I love it!
www.kite.com/get-kite/?...
So we made a design change and for base64encoding we introduced a simple lambda function and to provide the users a bit of convenience we thought of providing them with an API endpoint resource so that they can pull the data they need.
So let's see how we can get this done.
AWS Solutions Architect Associate Certification 2020 Playlist:
tinyurl.com/y4lr8zu4
Please follow we in the links below to stay updated: 🙌
🍀Click on the link below to subscribe: tinyurl.com/qqebnwz
🍀Instagram: / tougherapollo
🍀Twitter: / tougherapollo1
🍀Discord: / discord
🍀Care to share: / pythoholic
Disclaimer: The content provided in the channel are not affiliated in any way to the organization. We provide information here on the channel based on the knowledge we have on the topic. We advise our viewers to please do their own research and read more about them from the source provided by the organization to get a better outlook on the topic that has been covered.
These videos are just to provide you a platform to learn, and there can be mistakes and we are always trying to improve based on your feedback. we recommend viewers to have an open mind. Please support the channel to get more content like these in the future.
#ApiGatewayDemo #AWSSolutionsArchitectAssociate2020 #Pythoholic - วิทยาศาสตร์และเทคโนโลยี
Great job Pythoholic!
Keep it up by putting this kind of videos having examples.
Bcoz it can be easily understood. 👍
fantastic video bro! this is what I was searching for! Thank you very much!
Awesome video buddy!! This helped a lot with my work which I am doing for Amazon itself :)
Nice walkthru demo. Good job!
Great content. Thank you!
WOW...What a teacher you are! just simply superb. I became an alcoholic after watching pytholic
😂😂😂😂😂
AWS Solutions Architect Associate Certification 2020-2021 Playlist:
tinyurl.com/y4lr8zu4
Please follow we in the links below to stay updated: 🙌
🍀Click on the link below to subscribe: tinyurl.com/qqebnwz
🍀Instagram: instagram.com/tougherapollo/
🍀Twitter: twitter.com/TougherApollo1
🍀Discord: discord.gg/dsZ6Wdh
🍀Care to share: www.patreon.com/pythoholic
@Pythoholic, you are gifted teacher. Please can you make an explicit vide on AWS S3 Accesspoint practical with full demo
very helpful thank you :D
Excellent video . I am using this video for creating my masters project . This was very helfpful
thank u
I want to do something similar but have the base64 encoded data be posted to the api gateway then decode and then retrieve the data from the s3 bucket. Kind of similar to a JSON schema registry it seems. This way is good, but how am I to know what bucket and key they want before hand.
Well, this is really great demo video and I really thanks Pythoholic but some improvement is required specially keeping content of demo in description and improving labs based on feedbacks.
that's a wonderful tutorial. I'm really appreciate. Keep on to teach. +rep
thanks practical examples
Sir,How to build a serverless web application?I tried hands on practice but code is nodejs 6.10 runtime is deprecated from aws lambda so now what can I do?
We can do that i have already tried it. But haven’t made that into a video.
Lets get the sessions over then we go deep into any particular topic.
Thanks - helps much. I do have a question. How do I expand on the example to pull only selected data from the file. So more filter parameters? Would appreciate the help
I think it's better to integrate Athena and query the data that you want and use lambda to invoke it.
Hi francois if the data is of json type you can use json.dumps() to convert it to a dictionary and return specific values within the lambda itself. If your data file is of specific type then you can use supported libraries ex: bs4 for html data, pandas for csv etc.
Your videos are awesome. "isn't it". :)
Yes it is ;)
How did you know it is $input in the template? Do we also need the template when we use a post request with json data instead of query string parameters? Thanks and best
its the syntax its already there i will provide some documentations for this it might be useful
Can we just have the API gateway and user B can access the file from S3 without the Lambda in between ?
Yes please check this , this might be helpful : docs.aws.amazon.com/apigateway/latest/developerguide/integrating-api-with-aws-services-s3.html
Excellent work man ! Can i keep this as real time project in my resume to showcase my knowledge?
this is a small demo. i think u can go for a bigger project which does some operation
@@Pythoholic Thanks brother your videos are helping me lot.
At 32:48 , we have the Query strings : where the bucket name and key is present as pythoholic/sample02.txt.
As this API is the initial point to the user, where did we pass this parameters in the video..I didnt see giving as any test events in API gateway as we did in lambda function.
TO be simple , in this demo how does the API gets the bucket and key value. Can you please clear this Sir?
Its the context, when u call the lambda function you pass the parameter in the call itself. Test does the same thing.
@@Pythoholic Sir from API when the call is made to lambda function, i understand we pass the the parameters in the api call to lamdba. (the parameters we are giving in query string parameters which need to be passed into tha lambda fn)
But my query is from where will be the api requests gets the parameters?
@@hemalathavijay3278 This is valid point even I am also getting same issue as in the video not mentioned from where they have copied formatted string which has pasted in Query string and It would be better if you can share demo content in the description.
i had created the query string
Hi sir. Thanks for the video. I have a doubt. I am sending the live embedded data to the lambda function. I am configuring it using test event. How can I get the real time data in the api endpoint. Will this mapping works for it ?
it would be better to create a data stream and push that data to that stream using kinesis. that should help you get the data.
I want to retrieve all the images I stored in s3 bucket through API gateway how can I do that
Fetch the list of files from s3 ..using lambda.. and then create a base64 of each and download each of the base64 strings and decode it back and save it as image format.
@@Pythoholic any sample code can I get I am very new to it
Great video bro. How to specify the latest file in s3 bucket as value for the key for the template ? As shown at time 31.25sec ? in the video. My front end user needs to pull the latest file from the s3 bucket. How to specify the query strings like at 32.46 in the video. Any suggestions will be helpfull.
You can write the code to pull the lastest one based on time stamp. For it to be the simples
can you do a video on how to upload json to s3 using api and lambda
sure
If I use HTTP APIs can I also use CloudFront to front is to reduce costs ?
Depends on the requirement.
Is speed important or cost
@@Pythoholic If you can explain it a bit more is appreciated ! I thought CloudFront was going to be faster and cheaper
Yeah the thing that I wanted to tell you was if you are working on static content delivery and speed is something that you have in mind, you can create your CDN and share the same endpoint to the users. So that they have low latency.
Cost is a factor if you want to scope latency as a factor. If the data is more dynamic and doesn't need to be cached enough then you should use lambda to perform the operation and use cloud front to cache the resources.
But both are either used in the way the requirement is. That is what it depends on
@@Pythoholic My situation is that I have monthly data that I obtain nightly from a provider, process and keeps in DynamoDB. I need to provide a HTTP or REST endpoint that outputs monthly JSON data for our mobile app to consume. Our users can browse the data by swiping the monthly calendar. some of the data changes nightly; but still fairly static and expecting to use some kind of caching heavily like CDN. What is the most cost effective way to do this? Speed is a concern but secondary.
i am able to to get file without creating template
which template?
But Bro I don't know python language, could you please help me? How can I learn python for AWS
I am making a course on python
it is coming uo
@@Pythoholic thanks bro,make the course as soon as possible
I am a Begginer, who write the Lambda code, is it Developer or cloud engineer.
Sorry ram.i didn't understand your question
@@Pythoholic My query was who will write the code in lambda function which is written in python. Developer or Cloud Engineer will write the code.
Actually its both but mostly its about ur requirement. if its a specific feature then the cloud engineer develop the function and provision it to the devops teams to manage it later on.
and as a person if you are working in a cloud team and want to test out or implement a automation we generally use lambda but its not restricted to any job role
Could u please upload the code here....
code is at github @ pythoholic
this is not scalable at all, base 64 consumes 33% more data making it not usable.
thanks for the feedback,
its not a Proof of concept for scalability but its just a demo to showcase the connectivity. else i would never do this example. i wont even go for this encoding decoding. i would do something much better.
and more over as the title suggests, its an example for fetching data from s3 not the example to flaunt my knowledge on how secure or performance oriented the design is. i will keep in mind ur feedback, it will help improve the content
@@Pythoholic hey dude don't be so salty. I found a way through s3 presigned way which is more better check it out
@@NIKDEFAULT hahah no dude i am just joking.. please dont take it seriously.. i am never rude.. if u felt that way.. i was just messing around :D.. i agree to what you say
@@NIKDEFAULT so long i have heard this word “salty” … good one.. if i was salty i woulnt have thanked you for the feedback. i think words in text convey a different meaning to what the actual expression is sometimes. if u didnt like the video its fine.. i will try and improve…
I really need to know what format I can pass below parameter in Query String while testing
I have been trying with multiple options but no luck
bucket&key=lambda-s3-basav/simple.txt
bucket,key=lambda-s3-basav,simple.txt