AWS Glue ETL Vs EMR - Which one should I use?
ฝัง
- เผยแพร่เมื่อ 6 มิ.ย. 2024
- ℹ️ aws.amazon.com/emr/
❔www.thequestionbank.io
ℹ️ johnnychivers.co.uk
☕ www.buymeacoffee.com/johnnych...
00:00 - Intro
00:36 - What is EMR?
01:26 - What is AWS Glue?
02:11 - When do I use EMR? When do I use Glue?
In this video we take a look at the use cases surrounding AWS Glue and AWS EMR. It's a common question on which one should I use and when. I therefore attempt to answer this question in a highly requested video.
😎 About me
I have spent the last decade being immersed in the world of big data working as a consultant for some the globe's biggest companies. My journey into the world of data was not the most conventional. I started my career working as performance analyst in professional sport at the top level's of both rugby and football. I then transitioned into a career in data and computing. This journey culminated in the study of a Masters degree in Software development. Alongside many a professional certification in AWS and MS SQL Server. - แนวปฏิบัติและการใช้ชีวิต
![AWS Glue Tutorial for Beginners [FULL COURSE in 45 mins]](http://i.ytimg.com/vi/dQnRP6X8QAU/mqdefault.jpg)
![AWS Glue Tutorial for Beginners [FULL COURSE in 45 mins]](/img/tr.png)
![AWS EMR Serverless - What is it? [FULL TUTORIAL in 25mins]](/img/n.gif)
What a fantastic explanation - all killer, no filler. Thanks!
All his videos work. Is this channel in a parallel universe?
Wow this is a fantastic video on EMR and Glue. Thanks.
Damn. You're good. I understood the use cases for both in one swing. Thanks 🙏
Thanks for the clear explanation!
Thanks a lot Johnny, awesome explanation as always!
Thanks for the video!! It really helps a lot
Thanks for watching
Thanks a lot! I got my answer so I think it's a great video!
Thanks for watching Leo!
Excellent! Keep 'em coming!!
Great stuff mate. Subscribed and Liked..!
So clear! Thank you!
Thanks for the explanation! good comparison
Thanks for watching Jorge
Great job comparing the two options, Johnny
Thanks as always bob!
I have not heard PIG in forever, really enjoyed that language.
I don’t know how you only have ~3k subscribers. What a trove of knowledge. Thank you
Thanks for watching Jason.
Just a matter of time for him to get much more subscribers. The content is absolutely great
Crystal clear 🔮 explanation
I do almost the same code in AWS glue as EMR, I mean, I consume from external sources via spark JDBC connectors and publish the results to other warehouses via JDBC, I only have crawlers to detect the intermediate files that are generated at the datalake at the staging or business layer, but I don't use the studio or editor.
I feel AWS glue more integrated in terms of managing the workflows and the status (cpu/ram,etc) than a EMR based service.
I think you can request AWS to increase your quota in Glue to have more than 100 DPU enabled per glue job.
amazing work
Great explanation. How about Choosing between Glue and Emr serverless?
Was just thinking that as well. I think his points will still be valid since EMR serverless will be more expensive to run that the self-managed EMR and we still have the case of not utilizing it fully.
Hi Johnny, since there is EMR server less available now, do you think that the operational overhead is still one of the differentiator between the two? What do you recommend based on the EMR serverless?
Would love if you could do a simple industry grade project starting from beginning to end!
Definitely something I can look into. What would give the most benefit? Glue or EMR? Bearing in mind it would be a very long video as it would be at industry standard.
@@JohnnyChivers glue would be nice since you mentioned its easier for beginner data engineers to learn and use. But yeah i agree it could be a whole course in itself. Perhaps you could split them up into smaller sections/videos if you do decide to do them
@@JohnnyChivers Hey John, great stuff, have there been an update to this please?
BUY THIS MAN A COFFEE. Thanks Johnny!
thanks for watching.
Which one is a better option when building a transactional data lake?
I love your Scottish accent
ETL code in pyspark or scala? Can I have it in Java instead?
I have a requirement to copy the data from One DynamoDB to other DynamoDB within same account. Data in source table is of 2017 version while target is having 2019 version.
Can you please suggest which Service will fit best with no downtime.
You can use AWS DMS in co-ordination with AWS Schema Conversion Tool
For data wrangling only, what are differences between AWS Glue ETL and AWS Sagemaker Data Wrangler?
Great videos!
Sagemaker data wrangler helps you build a workflow using pre-created libraries mainly with the intention of using the data for ML.
Glue ETL is were you write the code and the logic yourself from scratch. Of course you have any spark and python library at your disposal. And whilst Glue ETL can run ML algos, that doesn’t have to be the aim of your wrangling - unlike Sagemaker data wrangler.
Does it make sense to do your proof of concept ETL code in Glue, then have EMR run that process at scale?
Hi Alan, it certainly does. The one factor to be conscious of is load. If using a subset of data to develop your script in glue, there could be performance issue later down the line in EMR once the full dataset is used.
@@JohnnyChivers Thanks, Johnny, for the response! Got it! Performance is always an issue, both in volume and composition. The test case would have to include the full range of expected inputs, but volume... well I guess you just have to run it at full volume and fix as necessary. Or parallelize with multiple Producers?...I'm sure you've dealt with it.
They now have EMR Serverless so I guess you don't need a cluster up and running 24/7 when you only need it every now and then
yes, would interesting to get a comment from Johnny. how close those to each other..costs
hey Im new to AWS...just wondering...even without serverless EMR, cant we use Transient EMR cluster to run and kill once the job is completed...that way we will not have to keep the cluster up and running
AWS Glue is for ETL Purposes and performing ETL Operations is way easier that is get the data into catalog and create jobs in scala or python and AWS will run without needing to manage Clusters, Infrastructure and Apache engine
EMR requires knowledge of clustered computing so it may require a lot of infrastructure cost
AWS glue is 20-40% is overhead cost but 1)But In AWS Glue we pay for what we use only that is it on demand service 2)On the other hand you have to pay AWS EMR all the time and in most companies around 80% there isn't any need to run Amazon EMR cluster
It gives 100 Dpus (16GB ram 4 CPU per DPU)
EMR Is better in AWS Glue and looking data in EMR because Glue requires Jobs for everything
So why do people use either of these when you can just use Databricks?
Especially EMR seems like it can be replaced by Databricks.