Read JSON file from S3 With AWS Lambda in python with Amazon EventBridge Rule

This video is a step-by-step guide on how to configure an EventBridge Rule to trigger a lambda function and read in a JSON file that was uploaded into S3. It also covers how to write a pandas file to read in a JSON file into a Pandas dataframe in python so data transformation could then be prepared on it.
#aws #eventbridge #awsdatawrangler
timeline
00:00 video overview
00:37 Amazon EventBridge configuration
02:19 Lamba configuration
04:14 lambda python code
09:45 end to end test
Useful resources:
Install AWS Data Wrangler on a lambda function: • How to Install AWS SDK...
AWS Data Wranlger Overview: • AWS Data Wrangler Over...

[@markthomas9641]

Great video, thank you. DeltaJSON is really useful for devops working with JSON in AWS, it does compare, merge and graft via API service.

[@DataEngUncomplicated]

Cool I'll check it out, thanks mark!

[@EagleHunt]

Hi, I am trying to follow along but at 3:18 I only have the CloudWatch Logs option. I cannot select s3, what should I do? Any help would be greatly appreciated!

[@DataEngUncomplicated]

Hi, it sounds like you are missing the required iam permissions for your lambda role. Make sure to update it for the permissions you need to s3

[@tekkemganesh9424]

Can we read image from s3 bucket and integrate it with Amazon eventbridge and send it to the lambda function to trigger can any one helpme??

[@DataEngUncomplicated]

Yes, when you set up Eventbridge, the key and bucket should be in the metadata, you can write a lambda function which can then read that image into your lambda function

[@georgelza]

Personal request, do you have an example where the JSON is published onto a Kafka topic ?

[@DataEngUncomplicated]

Hi George, no unfortunately I haven't worked with Kafka

[@georgelza]

@@DataEngUncomplicated bummer... it's such an important building block these days in data streaming, and Lambda is such a nice "publisher" from the AWS ecosystem, thanks. back to Google searching.

[@georgelza]

@@DataEngUncomplicated sorry, another question, I got a potential inbound stream, .json.gz that i first need to uncompress.
, the resulting .json file could then have up to 1million records.... comment from viewpoint of awswrangler handling this and Lambda handling this ?
At any time i might have ,multiple of these files being dropped into the S3 bucket.

[@DataEngUncomplicated]

I have handled more than 1 million records using aws wrangler. You will need to increase the memory from the default to handle it though. Keep in mind, if your lambda functions is doing complex transformations and creating many data frames in memory, it will eat up more memory and you could run out of memory.

[@georgelza]

@@DataEngUncomplicated all that i need to do is loop through the doc and post them as individual messages onto a kafka topic, so little processing.