I built data pipelines at Netflix that ran 2000 TBs per day, here’s what I learned about huge data!
ฝัง
- เผยแพร่เมื่อ 1 ต.ค. 2024
- Check out my academy at www.DataExpert.io where you can learn all this in much more detail!
You can get use code ZACH15 to get 15% off!
#dataengineering
#netflix
I’m so glad I found this video, I was just sitting here with 60 million gigabytes and was figuring out what joins to use so this was perfect timing.
if all u registered was 60 mil gb & joins ur not flowing
You're kidding, but somehow I just started a data analysis project of two terabytes and this video shows up.
@@aripapas1098 if you think comments must indicate a user registered every aspect of a video, ur not following
@@aripapas1098this is a sad comment
Sarcasm ???? 😂
Can't wait to build hyperscale pipelines for my startup with 0 users
But it sounds powerful when you say it, like you mean business.
Based
1 user (me)
If you build it, they will come.
I have 1k TB data just sitting around in my backyard. Glad your video came up to get me started on atleast something.
Boyfriend simulator: you sit with your bf and he starts talking about this nerdy stuff you have no idea about but need to keep listening because you love him
This is exactly correctly
aww 🥰
After marriage they no longer pretend to listen to
If only a girl would fall for me when I speak nerdy stuff 🫠
@@rajns8643 are you kidding me? This is what most people like the most! Intelligent people are extremely attractive
In the future a wrist watch will have a little blinking light that will have 60 million gigabytes of data in it
You mean an Electron app?
yeah okay crack smoker
And it will still lag and hit 99% singularities
@@dhillaz that will just show current time
@@Ivan-Bagrintsev Yes it will show the time, but with full DRM. Unless you have a license to view certain minutes it will be denied.
What I absolutely love about your videos is that as a beginner in the data engineering field, you often talk about things that I had no conception of. In this video for example, I have never heard of SMBs or broadcast joins. This gives me an oppurtunity to learn these things, even hearing them be mentioned from someone as widely experienced as you.
You need not necessarily have to even go into detail, but these short form videos act as beacons of knowledge that I can throw myself into learning about.
Thanks a lot, and keep these coming Zach!
Really appreciate this comment! It reminds to that the value im putting out there is important!
@@EcZachly_ ✌
Great summation! I was thinking the exact same thing while watching. It's nice hearing even the specialized lingo from technical experts in their fields, it peaks my curiosity.
@@EcZachly_thanks
@@EcZachly_did you already know the importance of these two before Netflix or did you learn that while working at Netflix?
Thanks Zach, hopefully one day I will understand what all of that means
😂😂😂, I’m starting now
2 pita bites a day, the same as me when I’m on a diet.😊
Holy crap. I’m currently learning about data science, the various roles, etc. -with the hope of one day switching careers. But the current state of learning is all about the languages and software used etc, not about the infrastructure and what to do with massive datasets. So this just 🤯
its really about math but no one talks about it. get at least 1 year university math comprehension and then get into the python and tech tools. the most competent and successful data engineers are always people with a good STEM background. for example Zach has a Bachelor's Degree in Applied Mathematics and a Bachelor's Degree in Computer Science so he is a heavy numbers guy. That's what most of Data Science \ Engineering TH-camrs don't tell their viewers cause that will cause them to loose viewers.
learning the tools can be very different from solving real world problems.
@@samuelisaacs7557 True asf
@@samuelisaacs7557Yep, even a business administration bachelors will have a lot of maths and it's nowhere near data science which is 3x that.
Sir this is a Wendy's.
I love that you kept it short and to the point.
He sure wanted to save some data… 😅
Yes but why does he look like a French model
Do u actually like every comment 😮
Dude has beef with Bezos😂
Great content, an honour to be able to listen to someone who has handled that volume of data.
literally 🎉
Have chat gpt explain it too you or some other LLM.
1. Are you a data engineer?
2. What tech is this? AWS, Snowflake?
Thanks for the info Zach. Could you please make an elaboriative video on SMB join.
In the 37 years I’ve been working in data, I’ve never heard anyone call it Peter 😂. PETA
What's wrong with a Peter bite?
Heya Peeda
Could be an accent or a slip 😂
Imma wait for Primeagen to confirm this as well when he reacts to this video inevitably 😁
tbh if youre dealing with this much data, it's likely a good problem to have 💰 💰
My problem is how do people even find out about the careers that they go into?
I dont know anything about data science? Why am i watching this?
You shouldn't write "s" in Terabyte per hour, just TB/hr
"TBs/hr" looks like "Terabyte*second / hour" 😅
Petabyte was misspelled. Great video though.
Damn I just wanted to shuffle like there’s no tomorrow and then I found this video.
If I shuffled all the word in this video, it would still sound same to me.
Thanks Zach for the insightful video. I have a similar use-case. Hence, a few questions:
1. So, with the large volumes of the datasets, do you archive it/compress it/just set a TTL to it? What do you suggest would be the best way for this.
2. With such large datasets, while I join the two tables, bucketing along with partitioning would be the most viable option right? Can you make a video around the joins if possible.
Thanks!
How in the hell did I get to this side of algorithm? 😢 hahaha
I love pita bites as much as the next guy, but I don't think I can take more than 35 before I'm full
As a guy struggling to get a job because entry level roles require ex[erience, I have learned something new and valuable today. Broadcast and SMB joins.
Did Facebook use Databricks or did they have HPC Clusters for you to run Spark on?
you could have made up all of this jargon and it wouldn’t change my perception on what you were talking about 😂
please do not take this as a negative comment, im just completely ignorant
I've never heard of these terms, thank you sharing your real case scenarios(The FB notification example)
Me watching this not knowing anything hes talking about makes me feel like starting a big tech company 😀
My medical science clients called, they need an 800tb imaging data set parsed by end of day (thank you kubernetes)
What the fuck is he talking about? How can you shuffle the pipe and bucket the joins. Who are the tables? Why is bucketing joins?
Wow, if I knew all this, it's pretty amazing content...
If only...
Can you help me get a job as Data Analyst? I have certifications but employers never hire me
That's a ridiculous amount of data, but wait till you see my girlfriend's Messenger 😂
I don't quite understand why Netflix needs data pipelines.
Wth is in that data ? Like seriously i feel like most of that would be redundant shit since even a chemical plant can be run entirely on Excel without ever needing db involvement
Bleh, who actually shuffles these days at these high volumes! Bucket joining ftw!
Peterbytes Bucket joining manage shuffle and fcuk Jeff bezos is what I got out of this
Thanks Zach , but I have a question broadcast join is used when we have a small dimensions joined with big table this is your case? Or are you used hash join with two large table?
Subscribing just for the britto. One of my favourite hoods
Hold my beer while I cross join Amazon to Netflix
100TB keyword and trying to sell courses at end of the day
If you ever want to collab let me know!
Wow I didn’t even know that such joins existed. No one taught me 😮
The way I only know very basic networking and have no idea what you’re talking about
No idea what this guy is talking about, but thankful TH-cam sent me this
I'd like to learn more about these pitabytes. What are they? What do they taste like?
Thank you Zach for taking the time to give us the hard truth and hands down your experience. It helps a lot of enthuastic students/people to know how we can in some way support or help others in the subjects we like. I don't imagine myself processing 2000TBs per day, but it helps give a bigger picture. Once again, appreciate the short video and thank you for sharing
Hey
Data with Zach
.. I have some questions.. So netflix uses AWS servers all over the world.... I am wondering. how many gb is each 4K movies, 1080p movie.. ? :) and what audio mix do they have.. Dolby Atmos, DTX etc. etc. :) Have a good day.. love from sweden :)
For serving videos they use OpenConnect and CloudFront, not AWS servers. This allows them to serve the video from the closest regional spot to you.
Almost all videos can be served in 4k. but are downsampled depending on the current network conditions
I’m trying to get into data analytics and most of this we t over my head but this still sounds lit 🔥
In a way thats not gonna make Jeff Bezoz Millions of dollars 😂😂
Now I just need a billion dollar company to have these kinda problems.
My question would be, why you have table that big? Can't you distribute or cluster your data?
I'm thinking like 10000 users per server. Only stuff around those 10k users gets stored.
No magic needed to query stuff.
Gotta analyze it all together though
You keep that $750k salary. I’d rather sleep better.
How do you get a job in this field? Were you in software engineering?
Wait, i have 200TB/hr what do I do? Please help!
Me a random viewer because yt algorithm said so: 🤔🧐
are you parsing customer facial expressions or something?
I wanna hear about "everything I need to know about extreme high volume" AFTER you compare notes with people from various other organisations processing that much or lots more, that are outside your expertise. I.e. LLNL, CERN etc, chemical industry or any large government etc., and scaled down ones, too where you recognise your lessons.
Otherwise it's just an incomplete angle. Which would still be OK if you reflect on it. But telling everything people need to know? Nah. You cant, i can't, they cant.
It’s a two minute video bro, relax
I built data pipelines at Netflix that ran 2000000000 MBs per day
I know the words, but the sentences are not familiar 🤔
I would be interested in the architecture and content delivery for pre and post cdn from a network design perspective. Are there any examples or presentations regarding networking at netflix?
Minimizing retention and broadcast joins could have been ten seconds of the video, and the rest could have been productively spent explaining SMB joins with a diagram
Make that video and share it with me!
I really wonder how netflix achieves 100tb/hr just with only streaming videos.
But… every day I’m shufflin’ 😢
I have 60 million cross joints but not enough time to smoke em
would you say that using bucketing and basically constraining against “acceptable” throughput as well as risking on creating gazillion files in process is more acceptable approach then more ad hoc ones like: z ordering and bloom filters?
If you come across a scenario to join 2 large datasets. You could do an iterative broadcast join. Basically you are going the break one of the df into multiple dfs and join the dataframe in a loop till all the multiple dfs are joined.
You’ll require a lot of memory and have long start times, no?
Is this only available with sparksql?
No, broadcasts can be leveraged in any processing framework that leverages two sets of processing logic. Your highly parallelized logic as well as a commonly single process. The single process “broadcasts” data for all of the parallel instances. It can be implemented other ways but that is the most common.
He is channeling a young William Benney over here isn't he
Has anyone told that you look a lot like Carlos Sainz Jr.?
Linus looks different somehow 😅
I liked the content, but Jeff Bezos didn't
Wow, didn't know Owen Wilson was working on data
Bro is the PewDiePie of data Engineering
Turn off shuffle service… don’t sort
Wait? So youre making a table? 🤔🥴
Very important concept in such short time.. thank u so very much ❤
What did you do to those massive data sets with your huge mouth?
I suddenly feel like pita bread...
I still bite my gigas when my man hustling meta in peta
Oloko o cara usando camisa de Romero Brito?
This was so fucking interesting
Did you work with Theprimagen 😂
I always shuffle. Like always.
Can you get a tripod for your cam?
All that dara flow just for selfies and self endulgance. 99% crap, but perhaps those 1% pays off.
Perhaps...
Bro 100 TB an hour???? Yo whattt
I love how you acronym Sorted Bucket Merge as SMB. Think you may have had Super Mario Bros on the mind 😂
Small fry honestly.
Honestly, 2000 TB per day isn't the problem. The problem is the cost and how much of the data is burst. If it is not burst it is pretty much always cheaper to do it in-house with your own hardware than to pay and rent the cloud to do it.
Never thought broadcast join is a Netflix saviour
Just started following you. Really appreciate you for sharing your knowledge with the community.
I love technology and I know more than your average user, yet I have no IT qualifications and I am light years away from this knowledge, but for some reason, I love watching these videos as if I was ever going to use the information 😂
What engine were you using to do these massive joins? Spark?
Yep!
Just download more ram
Love the way you tried to make it sound more complicated than it actually is and failed.
What’s a megabyte