this is crazy ❤❤❤ wish i had seen this earlier ! is this how the whole amazon product in a actual work flow look like？ and also could you maybe make another showing azure system? pleaaase
impressive and informative video, good job, go on doing tutorials plss :) Would be very interesting to see a video about spark and snowflake on your channel!
Fantastic tutorial indeed! I did as instructed and I got two fails in deploying the 'add step' part of the EMR Cluster stage, any insights would be appreciated.
Could you share more about project for data engineer beginners? I have start to learn to be a DE recently and I hope to know more about some personal project that help me to enhance my skills. Thank you so much for your sharing and waiting for your next video :> Have a good day
I'm learning data engineering and I find it's hard to find end-to-end data engineering projects to learn building scalable B2B data infrastructure systems that process large amounts of data. Many examples only touch the surface or handle small datasets (well, they say example). Do you have a plan to make a tutorial about actual use cases of data engineering (large and complex data, scalable, cost efficiency system, etc.)?
Yea for sure, I’ll consider making more end to end videos in the future. Hard to mimic actual use case data engineering with toy projects, but will try to bridge that gap
@@jayzern, I'm doing a project to provide a management solution for small businesses. Data such as customer appointment schedules, staff's working hours, etc., will be stored in MongoDB. I'm thinking of a data engineering component: some data will be extracted, transformed and loaded to AWS S3 for visualization on a mobile device. I think it's still simple but, perhaps, good enough
I am wondering if I only needed to do PySpark, is EMR the best tool or is it overkill and Glue serverless would be good enough with a lot less to manage and fewer configurations to worry about. Is it possible to enable better performance with all the options in EMR?
Hey, we typically create VPCs over EMR clusters for more networking control and better security. If I rmb correctly, here we defined a public subnet and internet gateway which connects to S3. You could also use private subnets to avoid attaching internet gateway, but it's trickier to setup and can incur NAT gateway charges. The video is just an example
@@syedmehdi5125 bc most companies want senior data analysts or graduate students when it comes to data science. You’ll waste more money chasing a data science job than you’ll make
Just for the information to all the learners this is not how things to be done in tech industries....you need to understand Terra form scripts along with jenkins which deploys aws services....you will not get access to go on management console and play around and do stuff.
Your content is valuable but I find that your presentation is too fast. Sometimes, you fastforward steps too quickly. It makes the video look not lengthy but it's challenging for audience to closely follow your steps.
That’s super helpful feedback, I’ll try to slow down on the steps and talk more in future videos. Hopefully my latest video is slower. How the audience feels matters a lot
eah good in EMR AWS but an absolute rookie in Videography and equipment use manual focus since you are stationary.... your autofocus keeps looking for something and change light set-up
![AWS EMR Tutorial [FULL COURSE in 60mins]](http://i.ytimg.com/vi/v9nk6mVxJDU/mqdefault.jpg)
![AWS EMR Tutorial [FULL COURSE in 60mins]](/img/tr.png)
I hope you create more videos about AWS services. Loved the way you explain things, perfect for beginners.
This is an outstanding tutorial. Thank you for making this!
Best tutorial I've seen so far. Was confused between Glue and EMR for a future projects requiring big compute power with control over each node.
So sad your channel doesn't have more tutorials like this :( thank you so much!
Same thoughts me too
Great work mate, very crisp!
Thanks man!! Love ur content
We need more videos Jaaay 🙏🏻💪🏻 You're awesome dude!
Absolutely enjoyed watching the entire video. I felt this video is gonna be great start to understand EMR. Thanks for making it jay
This is so goood :). Please keep making these kind of videos! Hello from Seattle
Thanks Isaac from Seattle! Appreciate your support
awesome explanation, simple , subtle and to the point!
Very well explained EMR video, thank you
Very clear! Thank you for sharing this excellent tutorial!
thank you!! I watched the TH-cam demo and it was really helpful. I also want to study spark on eks
great tutorial! can’t wait to see more
Go ahead bro....CONGRATS TUTO
Hey, thank you so much!!.. you really explain very well!
Your video is very interesting!
Hope you release many new videos :)
gnarly stuff man! great content.
Excellent tutorial
this is crazy ❤❤❤ wish i had seen this earlier ! is this how the whole amazon product in a actual work flow look like？ and also could you maybe make another showing azure system? pleaaase
I can see you are brilliant.
thank you, this is great tutorial
Very well explained, kudos
Great Article ! Thanks for sharing..
That was FAST, you are subscribed :D
Any vids related to "Amazon Managed Workflows for Apache Airflow"???
impressive and informative video, good job, go on doing tutorials plss :) Would be very interesting to see a video about spark and snowflake on your channel!
the type of video that makes me wanna quit the field because of how bad i feel about the level I am in , but its a very helpful video though
Are you working as data engineer right now?
@@shafiq_ramli I am looking for a job now as a Data Engineer
@@shafiq_ramliif u could help me find one 😢😅
this is amazing, thank you!
More tutorials 🙏
nice job, great tutorial
More videos on Streaming, Airflow and Spark
when writing the spark script, does it ever change or the skeleton layout remains the same? i truly appreciate this and i cannot wait for more
Great tutorials! thanks for the headup! do you have a git repo or more notion notes? would like some guidance
Fantastic tutorial indeed! I did as instructed and I got two fails in deploying the 'add step' part of the EMR Cluster stage, any insights would be appreciated.
Hi bishop, I got this error too just now - did you manage to fix it? afaik I did everything the same
nice, keep going
You killed it. Loved it! Extremely useful
Thank you man! Hope to create more
Could you share more about project for data engineer beginners? I have start to learn to be a DE recently and I hope to know more about some personal project that help me to enhance my skills. Thank you so much for your sharing and waiting for your next video :> Have a good day
Thanks for sharing man 👌
I'm learning data engineering and I find it's hard to find end-to-end data engineering projects to learn building scalable B2B data infrastructure systems that process large amounts of data. Many examples only touch the surface or handle small datasets (well, they say example). Do you have a plan to make a tutorial about actual use cases of data engineering (large and complex data, scalable, cost efficiency system, etc.)?
Yea for sure, I’ll consider making more end to end videos in the future. Hard to mimic actual use case data engineering with toy projects, but will try to bridge that gap
@@jayzern, I'm doing a project to provide a management solution for small businesses. Data such as customer appointment schedules, staff's working hours, etc., will be stored in MongoDB. I'm thinking of a data engineering component: some data will be extracted, transformed and loaded to AWS S3 for visualization on a mobile device. I think it's still simple but, perhaps, good enough
great video! can you make also for AWS Glue? Thank you!
Great video
Great job
Great!! Thank u so much!
From where you learn that coding part 😢
is this free to use or do i need to have a licensed software in order to use? this is quite interesting.
Please post more videos
Thanks .. Good stuff
Isnt using EMR notebook one of of the ways to trigger EMR job?
Yes it is! Wanted to keep things simple in the video so didn't include it
why do you need to create a VPC?
VPC is for nodes. It allows them to communicate between each other and the master node.
I am wondering if I only needed to do PySpark, is EMR the best tool or is it overkill and Glue serverless would be good enough with a lot less to manage and fewer configurations to worry about. Is it possible to enable better performance with all the options in EMR?
And thank you for this video - I’m studying for AWS certification and it was helpful to see your demonstration
thanks for the video
Hey why did you create vpc?
Hey, we typically create VPCs over EMR clusters for more networking control and better security. If I rmb correctly, here we defined a public subnet and internet gateway which connects to S3. You could also use private subnets to avoid attaching internet gateway, but it's trickier to setup and can incur NAT gateway charges. The video is just an example
I hav done masters of science in biotech, 38 yers of age, want to switch to data science...how shud i do it??? Plz reply.....
Do projects and add them in your resume. Try upwork and do some projects as freelancers. Keep applying
Don’t
@@Ved3sten y , plz reply...
@@syedmehdi5125 bc most companies want senior data analysts or graduate students when it comes to data science. You’ll waste more money chasing a data science job than you’ll make
bro thank you i survive
Great VIdeo
Don't stop
Just for the information to all the learners this is not how things to be done in tech industries....you need to understand Terra form scripts along with jenkins which deploys aws services....you will not get access to go on management console and play around and do stuff.
Your content is valuable but I find that your presentation is too fast. Sometimes, you fastforward steps too quickly. It makes the video look not lengthy but it's challenging for audience to closely follow your steps.
That’s super helpful feedback, I’ll try to slow down on the steps and talk more in future videos. Hopefully my latest video is slower. How the audience feels matters a lot
eah good in EMR AWS but an absolute rookie in Videography and equipment use manual focus since you are stationary.... your autofocus keeps looking for something and change light set-up
Fair point 👍 will work on lighting and camera setup more next time
nice try but its not working
Let me know how I can help
I can add a step for the spark application@@jayzern
Check if
1. the Spark script is encrypted when you upload it inside S3
2. any typos (line 41 should be "add_argument")
I had tried. but it's not working for me @@jayzern
Send me a DM on instagram @jayzern or linkedin, happy to pair up
Love the ways how you demonstrate! so clear and easy to understand! Thanks for sharing @jayzern
thanks so much