Practical Projects to Learn Data Engineering On AWS
ฝัง
- เผยแพร่เมื่อ 21 ก.ค. 2024
- This video comes with some practical project ideas to learn data engineering on aws so you can handle a variety of real-world challenges. This includes architecture diagrams of:
1. Event Driven data processing with AWS Lambda,SNS,SQS
2. Batch Processing Data with AWS Glue
3. Build a Dashboard with AWS Quicksights on a Data Lake powered by AWS Athena.
4. DMS To Migrate Data from a Database to S3
5. Process streaming data from Amazon Kinesis Stream
Diagrams and article link:
aws.plainenglish.io/most-comm...
Buy me a coffee : www.buymeacoffee.com/dataengu
Timelines
00:00 Introduction
00:31 Event-driven Data Processing
02:59 Batch Processing with AWS Glue
04:40 Serverless Dashboard
05:23 Database Migration
06:29 Process Streaming Data
#aws #dataengineering
I'm a newcomer to AWS a data engineer, Your channels really help me! Appreciate it!
this is the best channel on TH-cam! thank you! explanations are to the point and easy to grasp.
Hi Rodrigo, wow thank you for your kind words. I'm glad you found this video helpful! thanks for watching.
What an amazing video. Solved an issue I was facing in my own implentation of a data ingestion pipeline. You got a sub today.
Thanks for subscribing!
Excellent video ! I wish I found this earlier.
Yiour videos are so consise and easy to follow. Glad I found you
Thanks Mike, much appreciated!
Awesome !.You cleared all of my doubts in this video...Thanks a lot.
You're welcome! I'm glad it was helpful.
Great video, clear concise and valuable!!!
Thank much appreciated!
one of the best channels on youtube for data engineering! great content, keep the great work
Wow, thanks for the kind words! It made my day 😄
please do a comprehensive course on this! I really love your content 🙂
Thank you for great video!
Hi Tutor, Since I don't know your name, Can you please do a video on end-to-end implementation with sample project to use the AWS Data services like S3, Glue, Kinesis, Redshift, Data pipeline, Crawlers etc. for both batch and streaming data ingestion.
@sjvr1628
My friend, his name is: Adriano Nicolucci
capisce
Your videos are fanstastic. Please bring them more.
Thanks Jasper!
Great video. Well presented.
Thank you kindly!
You're an excellent teacher, keep it up...subscribing now
Thanks for the kind words!
thanks, everything is clear, i like your voice
Thanks Saltanat!
thankyou for providing this video
You're welcome!
The theory is very well explained. Could you please make a video showcasing the implementation practically, with a small example.
Thanks, This is a good idea. I will think about what I can do!
Can you please make a video showing implementation of event driven architecture using event bridge, sns, sqs and lambda? Thank you!
Well presented. For the first pattern, what's the advantage of using EventBridge and implications of skipping it?
Thanks! Well one benefit of Event bridge is event bridge rules is you can have many different AWS events used as the source to start your job. But with glue it is more limited. If you just want to run it on a schedule than glue orchestration should do the job fine!
@@DataEngUncomplicated ace edx
Hi thanks for these project ideas. Would like to ask, can you advice any public data sources/providers for data engineer practice?
You're welcome! I use www.mockaroo.com/ to generate my sample data. It gives full control of the column names and data types for randomly generated data.
Pretty neat and easily explained !! ♥️
Thanks Shrey!
Looking for more examples with practical session
What percentage of real industry work would you say fall into these 4 categories?
Also, thanks again for these sweet sweet videos man. Super concise and informative.
Hi Andrew, that's a good question! I don't have the data to answer this with any certainty but I can speak to my own personal experience. From what I see, the majority are batch processing workflows. However, there is a growing trend towards more event based architectures and streaming architectures as companies mature in the cloud and need real-time analytics grows.
well explained, is there an implementation video ?
No I don't have one yet! but this might make a good follow up video. Thanks!
Hey Adriano, thank you for your content. I am a data engineer and I will take the aws data analytics specialty exam next month. Please what materials do you best advise me to use to study for the exam? Thank you.
Hi Paul, your welcome. I bought Stephane maarek's Udemy course for this exam and I felt it did a great job to prepare me for the exam. I recommend it.
Loving your content. Do you provide any courses on udemy for projects like this ?
Thanks so much! Unfortunately I don't have any courses on Udemy, I would love to make one in the future though! I'm currently just focusing my extra time outside of my full-time job on TH-cam videos for now.
@@DataEngUncomplicated Youre doing friggin great so far -- Please keep it up! At the same time, I am still rooting for you to create a full course sometime in the future.
will you please make an end2end production grade code that will also combine terraform?
Hi, I actually just created a video on terraform with glue jobs. I think you will find this helpful: th-cam.com/video/CYZ3ZjXdQP8/w-d-xo.html
hi can you do a video on working with the incremental data stored in s3
Hey, can you provide more details what you are looking for?
Hii , could u please make an end to end aws data engineer project . It would a great help for everyone .
Hi Hrithik, what pattern do you think would be most useful? It might be a challenge to cram it into one video given there are many services to learn and know about. The video could be several hours.
@@DataEngUncomplicated You can make a series of video of 20 to 30 minutes. Small small steps in each video.
This is a good idea! This would be more manageable as a video series. Thanks!
Can u please help with these project videos. I would love to connect if u plan for a project day luve on youtube
Thanks Krishna, I'm slowly working on technical tutorials for these architecture patterns. My next batch of videos is using AWS Glue for reading, transforming and writing data.
I loved your videos and subscribed them instantly. But one think I didn''t like is the subscribe notification that comes in between and distract the whole focus and then I need to play that particular part again and again. You ask for subscribing in the middle of the video, nobody would pause and will go to subscribe but instead ask for subscribing in the end so that people will remember to subscribe in the end. Thank you.
Thanks your feedback and subscribing! In my newer videos I have added it to the end or at least it's not as distracting. Unfortunately TH-cam doesn't let you edit existing videos without uploading the entire thing again.