My dream FINALLY came True! - Petabyte Project Recovery Part 2
ฝัง
- เผยแพร่เมื่อ 11 ก.ค. 2024
- Go to privacy.com/linus to get $5 off your first purchase!
Check out SignalRGB and sync all your RGB peripherals at bit.ly/SRGB-LMGF
What better way to recover our lost data than to build a single server with over a PETABYTE of storage (1200TB)?!
Check out 45Drives on their website (www.45drives.com) or TH-cam channel ( / 45drives )
Discuss on the forum: linustechtips.com/topic/14091...
Check out 45Drives at the links below
Website: lmg.gg/eGo2K
TH-cam: lmg.gg/6ModQ
Buy Seagate 20TB Exos Drives
On Amazon: geni.us/WFfs
On Newegg: geni.us/XhkNI
Buy AMD EYPC 7402P CPU
Amazon: geni.us/ELHlm5W
Newegg: geni.us/AIBe
Purchases made through some store links may provide some compensation to Linus Media Group.
► GET MERCH: lttstore.com
► AFFILIATES, SPONSORS & REFERRALS: lmg.gg/sponsors
► PODCAST GEAR: lmg.gg/podcastgear
► SUPPORT US ON FLOATPLANE: www.floatplane.com/
FOLLOW US ELSEWHERE
---------------------------------------------------
Twitter: / linustech
Facebook: / linustech
Instagram: / linustech
TikTok: / linustech
Twitch: / linustech
MUSIC CREDIT
---------------------------------------------------
Intro: Laszlo - Supernova
Video Link: • [Electro] - Laszlo - S...
iTunes Download Link: itunes.apple.com/us/album/sup...
Artist Link: / laszlomusic
Outro: Approaching Nirvana - Sugar High
Video Link: • Sugar High - Approachi...
Listen on Spotify: spoti.fi/UxWkUw
Artist Link: / approachingnirvana
Intro animation by MBarek Abdelwassaa / mbarek_abdel
Monitor And Keyboard by vadimmihalkevich / CC BY 4.0 geni.us/PgGWp
Mechanical RGB Keyboard by BigBrotherECE / CC BY 4.0 geni.us/mj6pHk4
Mouse Gamer free Model By Oscar Creativo / CC BY 4.0 geni.us/Ps3XfE
CHAPTERS
---------------------------------------------------
0:00 Intro
1:18 This time vs. last time
3:50 Debate of the Century
5:08 Aboout the Exos X20 drives
6:23 This Epyc Board RULES
7:03 The CPU
8:23 Wait we have to mod this
9:09 "That's horrifying"
11:00 The cooler's mangled
12:05 How long will the transfer take?
12:59 Booting it up
15:20 Are we Protected?
17:20 TECH TIP
18:29 TECH TIP 2
20:05 Outro - วิทยาศาสตร์และเทคโนโลยี
Linus in 2016: loses data
Also linus: this will never happen again
Linus in 2022:
see you in 6 years when Linus loses more data
Probably has some serious PTSD
@@supercool_saiyan5670 Might as well call the series “Linus’ Bizarre Server Adventure” given the recurring nature of server loss over there.
And it's always related to Seagate :-D
@@supercool_saiyan5670
12 years later
Linus: guys my entire server is gone and my IT team left me
i hope this doesn't happen fr tho
I get hyped for seeing more of the long-term projects at LMG/LTT, like Petabyte Project/Server Room stuff, Pyramid PC and Linux Challenge.
Yea man same
It's like the LTT lore and the other videos are filler episodes
@@neuro3423 fair, but at least with LTT the filler is entertaining
Always fun when they have trouble...
Nothing will ever live up to ‘Whole room water cooling”. Event if it didn’t actually work.
Can't wait until 15 years later when there's going to be "The Exabyte Project", which theoretically can fit 1000(or 1024) of these petabyte servers in the same space
The fact that the zettabyte project will probably be a thing in our lifetime just blows my mind.
@@TH_5094 Unless we ditch computers entirely by then
@@drakonua Yep, it's back to rocks and sticks by then.
Linus: *coughs* "well these random coughs sure gets to you when you are old, sorry"
by then we will be using crystals for storage.
Probably one of my favourite LTT episodes purely because of the back and forth between Linus and Jake 😂 just like my boss and I at work. I love it.
You should marry the boss.
@@fynkozari9271 So true
Not sure about IT department but what you guys REALLY need is a montage of every hard drive locked in with nice clicking sound like they do on LEGO building channels. 100% satisfaction guaranteed.
I watch a guy tear down engines and he edits his videos so that the cracking loose of the head bolts are ripple-edited for that effect. It's one of my favorite things he does.
@@TechGorilla1987 channel name?
@@muzameela2845 I Do Cars is the channel name. He does teardowns on various engines.
Its on their onlyfans
Too bad fan noise would ruin it
I can’t wait to see the result of this project.
Also, the bot destroyer seems to be working well.
About bot destroyer, we can't be sure bcz we don't know what comments are being deleted and how many genuine ones are being flagged as spam, but I believe ltx did some monitoring on it so ye just be careful with it I guess
@@ncb4_69 Mine got deleted, and it was just me talking if the drives were sent to LS for free.
So yeah, we will have to deal with it. for the greater good
@@Ave-S But you don't know if it was the bot destroyer or YT itself.
@@Ruhrpottpatriot Oh shit. fair point
@@ncb4_69 we can’t measure the collateral damage, yes, but we can clearly see a night and day difference in the amount of spam in the comment section.
Linus 2 years ago: "Why you DON'T want a 20TB Hard Drive"
Linus now: Buys 60 20TB Drives
And says filling them will take at least 2 weeks
What did he put in those petabytes of hard disk drives??
@@fynkozari9271 They basically save footage, everything, which isn't worth it in my opinion, I would only save the final edit and a few important clips. Especially if they don't take care of it like they did and let the data degrade.
@@farkasambrus5741 lots of film and tv got lost to time with that thinking
@@farkasambrus5741 Yep, not really worth it just for the footage itself. They say it themselves, partly it's more for the fun of it, and as a pretext to getting these absurd servers so they can make videos out of it.
Now thats a healthy work relationship between a boss and employee..
Where they both rip at each other without whole "Remember who you are talking to"
Hopefully you can recover the data and that it will take even longer until another data loss happens.
1 Petabyte of Data: $37k
lacking an IT department: $0
Linus making multiple Petabyte project videos: Priceless
to be fair $37k is like, six months worth of a good IT technician, so...
@@wertigon LPT: just brute force more storage to avoid paying a tech admin.
$37K for a regular person. $0 for him. He got all those drives for free.
@@wertigon Oh gawd not even close.
My company would charge maybe a few hundred a month for server monitoring. It's actually quite inexpensive, you don't need a dedicated employee. It really depends on how many hours of work that's needed.
Usually for management, it's a couple hours a week tops. If there's a failure, just get someone on site to do a swap, press a few buttons and you're good to go.
For heavy errors, there are several alternate methods.
@@lilkittygirl But then you're not paying for an IT staff but for one guy to come over every once in a while, just like an electrician or a plumber.
A full time decent IT guy can easily make $60k a year, and that's just counting what the IT guy gets in his pocket.
PSA: zfs also lets you add in a hot spare that will _automatically_ replace a failed drive. You can configure it to be the hot spare for multiple zpools. When a drive fails, it drops itself into the array to restore parity, then when you've replaced/fixed the bad drive it'll go back to being a spare until the next time it's needed.
Not a PRO. Hot Spare additions are not automatic because a complete failure needs to flagged, and its rarely the case.
Really hope Linus can take advantage of this. Even if it’s not automatic, just telling the server to rebuild with the spare already inside could be really convenient!
Question, will it be on while it's sitting there doing nothing?
@@fffUUUUUU Linus knows a lot of things. Not necessarily well in any, though.
Old Linus: Oh noes, we lost data because we did not have sufficient redundancy to overcome our poor preventive maintenance habits.
New Linus: Lets deploy a new cluster, with even more storage to maintain, but with less redundancy, and then have the same people manage it.
These 2 in a video together with the lovely banter, gets me Everytime!! I just love them both!!
Guys, I'm absolutely loving the fact that you have a new Petabyte project but please get a new IT guy.
As an entire team of people who are well-versed at networking and computers, it doesn't make sense to get an IT guy at first, but in the grand scheme of things, you need someone who's able to attend to that equipment 24/7
Proper IT is a different skill set than "tech stuff" and I don't think anyone we've seen qualified. Yet either way they definitely need someone even if only because of limited time for current employees to fix the servers.
Get a new IT guy? They need to get someone as a dedicated IT guy first, they've never had one, as far as I remember.
They keep going with Seagate drives too... seesh it's like they don't learn.
Ya he went over this in the last video on the subject
Not even a guy, just a monthly up keeper
IT person interviewing for job: What is your data retention policy?
Linus: Yes
IT:How many years?
Linus: Yes
Yes, except when we misconfigure our storage and lose hundreds of TB ¯\_(ツ)_/¯
With single point of failure, both answers are "no"
@@tim3172 I think they determined to NOT do that and just rely on alerting from the software raid controller. Their first video says it's to expensive to back it up and the data isn't that important, so still No.
@@ScottZupek it's on youtube. That's their backup lol. They dont need literally any of this
@@johnathanera5863 Not quite, because your missing all the files needed to make them as well, from adobe templates, to the specific components of the video that get put into their video editor of choice.
Their is a lot more to it than just 'its on TH-cam, why care' because your seeing the finale product, not every part used in the whole process.
One of the best videos in a long time. Entertaining to see Jake and Linus mocking each other. Also, great editing transferring the chemistry between them. It never gets boring, even though such a topic has the potential to do precisely that. Thanks!
Just get one of those lifters that are used for motors by car mechanics. Use suspensions, dumpers and springs to minimize vibration and you are set! You can even make your own to carry heavy stuff around the office.
Didn't Linus say in Wanshow at some point that "No one would want 20Tb harddrives?" Because of their sata interface it will take stupidly long to rebuild parity drives in case of a failure. So it would be quite likely that another drive fails during the rebuild. But hey I guess another data loss will make for more content that they don't have to plan.
lmao great point.
That's fine they don't use monitoring or scrubs so they never need to rebuild!
they're using raidZ2 tho, so if another drive fails during a rebuild it's still ok. 2 drives failing while the 1st one rebuilds is still very unlikely
bah, just made that comment myself and then saw yours. So much of what they do is eyerolling but at least it's entertainment.
It made e shudder seeing the 20TB drives, Linus did say you dont want 20TB drives in a regular server, this is archive data and hopefully they will actually manage the the server and swap it out in time.
I can remember installing hard drives in that form factor -- 1" tall, 3.5" wide -- that could hold 120 MEGABYTES. And I couldn't believe then how much data they contained, considering that there were 80MB drives that were 2.5" tall and 5.25" wide still knocking around. And now these drives will store more than 160,000 TIMES as much data in the same space. Boggles the mind!
Wow that must have been a while ago
@@nesyboi9421 indeed, it was. I’ve been doing PC upgrades/repairs since 1988 or so. The first PC I ever built from scratch was a 386DX running at 25 MHz, 4 MB of RAM, an 80 MB SCSI hard drive, and a VGA card with 512 KB of VRAM on it, plus a 15” VGA color monitor. Cost a stupid amount of money, too - a bit over $3K in April of 1989. That’s the equivalent of about $7K today. 😳
I still have one of those hard drives!
At this rate, i wouldnt be surprised if this amount of data today could fit in a small usb later on the future
I remember paying an additional $700 to have a 20MB HD in my computer instead of a 2nd 3 1/4 inch floppy.
Jake is a treasure. Calm and confident descriptions and excellent advice.
I love seeing Jake in these kind of showcase videos. My FAVORITE channel on TH-cam. I love you guys!!!
I love Jake and Linus' banter and chemistry. please do more videos with you two, and only you two. it is hilarious. one of your best videos comedy wise in my opinion.
James & Riley. Alex & Linus. Brian the Electrician and Brian the Electrician.
absolutely love the videos with Jake...seems like one of the few that can go jab for jab with Linus..fun to watch...more Jake vids
That guy is a joke he knows nothing. This setup will fail after 2 drives. It's a mess.. also he wears shorts to work.
@@xzaz2 what is wrong with shorts?? that comment alone is why you are absolutely so wrong
@@EddyWhitaker it's unprofessional
@@xzaz2 I suppose that depends on if your work says so..but I don't think it's unprofessional at all.. obviously it's not unprofessional at LTT because you see various in employees in shorts
@@EddyWhitaker it is. Ltt are not professionals at this topic. They have no clue what they are doing lol
Linus and Jake poking fun at each other adds quite a bit to the entertainment of this. It makes a workday a lot more pleasant aswell as opposed to some super serious strict work only approach
I can appreciate the knowledge , walkthrough, and banter in these kinds of vids.
What knowledge? Everything they're doing here is wrong.
5:36 I found that too, I ended up getting EXOS drives cheaper from the US imported to Australia than the NAS grade stuff was anywhere... wut
I feel like it's because of the failure rate of Seagate? May be wrong, though
@@ArensLive It might be economy of scale at work
@@ArensLive so WD do better than Seagate?
@@shawno8253 annoying thing is thats self fulfilling prophecy, "oh it didn't do well in Australia 10 years ago, so we aren't going to ever try again"
I just hate how that is the excuse, cause it does mean as an Aussie tech, its really fricking hard to source parts cause "we don't usually get much call for that"
then again, we also don't get a lot of movies or shows cause apparently "we pirate everything"...its almost like if they released it over here, we wouldn't have the need to consider that?
I've been trying to import Dell constellation ES.3 (3tb) from the states to Australia too as its pretty cheap per tb (about $25). I haven't had a failure from them yet.
I love when Linus contradicts himself just to put together the LARGEST storage server ever used by a TH-camr ever 😁
With technology, 'ever' is not something that lasts very long.
@@bongosbongos ever just means till that very moment
but not in high performance computing.... :) PBs is small potatoes.
@@jeremygmail oh nah fr?
You are everywhere
My dream used to be a 1 TB storage server because when I started in IT in the late 70's the data center I worked at was one of only 13 TB data centers in the world. Now I can get that on a postage stamp.
Do you have your own now? I'm currently at around 20tb, but in the future I'd like to kick that up to at least 100.
@@samcolton943 I think it's around 46tb now
@@JoeHusosky nice! yeah, once you start adding more, it's too easy to keep wanting to add to it haha.
I just love the banter that goes on between you two; it makes the show so entertaining! 🤣
I deployed 30PB of capacity yesterday. 2 days to install and cable the storage. About 40 minutes to configure with Ansible. Sooooo cool
Ansible is beyond dope!
Watching Jake and Linus in videos is hilarious. The roasting between friends is super entertaining.
i love these kinds of videos where i cant understand a single thing they say, but i know if i continue to watch their vids for a few months and i will probably have enough knowledge to teach others
This project was awesome! And these two were so funny to watch.
Lol, I literally remember Linus saying like 4-5 years ago that he would never want a 20TB hard drive because if it fails in a raid it would take way too long to rebuild it.
Also why do they need the raw files to 10,000 videos in 8k? Like hes just a storage hoarder
I think that was before he started recording 8k video
@@Wicked_Carnifex like 30k worth of hard drives as well granted Seagate sponsored but still quite overboard
@@Wicked_Carnifex he actually talked about that in the last video. It’s a mix of ‚nice to have‘ and ‚it makes for good content‘. They get the drives for free, so why wouldn’t they do it
@@Wicked_Carnifex right? It's super stupid and counter productive.
Oh man, pickle Linus is such a flashback.
The old "livestream from a garage" days, so much nostalgia. I bet Edzel doesn't miss editing on an X79 ShuttlePC.
For some reason I also just remembered the stream where Linus had to explain how a thrown USB flashdrive shut down the forum for a few hours.
I can hardly believe the fact that I've seen *every single* episode of what we now call the WAN show. It was also the reason I made a Twitter account in the first place.
I can't wait for them to hire an actual IT person and they tell Linus & Jake everything was done wrong.
I find that hilarious - when someone buys “enterprise hardware” with much lower performance, capacity and reliability at a much higher price “for the support” which when it comes down to it , may or may not save your data
Till then we can all do it for them. It's amazing how, for a tech channel they can know so little about tech.
Like what? Please tell us.
@@tomorrow6 Any drive going into a Raid array needs to be an Enterprise drive as a standard drive will degrade and fail much faster, Not worth the risk when storing large amounts of data across multiple drives. 20 years of Experience in PC/Server builds I can assure you no one is paying for the support whatsoever!
@@spendy26 I’d suggest that they fail as quickly as consumer drives of similar capacity (or did before SSD’s) - Backblaze published drive failure statistics were a good source for non enterprise disk failure rates which didn’t differ too much from enterprise.
However - enterprise raid controllers with sufficient cpu and battery backed cache did allow for special drive setups for maximum fault tolerance at the cost of more wasted capacity. And of course SCSI drivers (including SAS) failed much more cleanly than consumer IDE drives
Plus the vendors did publish their own firmware updates based on faults experienced at other customers, especially if you paid for the extended support past four years. Enterprise software did offer predictive drive failure notification which allowed drives to be swapped out in raid sets online with minimal chance of data loss (albeit a slowdown as the raid array was rebuilt)
i just love the friendly banter between these 2 fellas :D
All the banter Linus has with his employees and guests make me realize that he’s the Conan of the tech world
Golden Comment !!
Lol, you get it
That's basically why I watch these videos lol. What the hell so I care about a petabyte server.
You nailed that.
@teamcoco Lets get Conan on LTT
I always love when Linus and Jake do videos together
Server updates are and always will be my favorite videos!
I'm just really glad Linus was practicing the 3-2-1 data backup rule so he minimized his data loss.
it's a pity that 3-2-1 backup strategy is falling out of favor for some years now.
As usual Linus is badly advised by its systems engineers/partners: nowadays the best/most effective backup strategies are 3-2-1-1-0 and 4-3-2
:-)
The essential data is backed up, such as whnnock server is backed up twice, once offsite at the lab and again in a much farther away backup server in a datacenter.
I love their banter now, Jakes turning into a great host
Did he gain weight?
@@Varde1234 probably. Too fuckin cold to go running this time of the year unless you’re a real believer
@@talltale9760 what a lame excuse
I'm loving the banter between Linus and Jake. Looks like this would have been pretty fun to make
Yup pRettY fUn
I like to imagine that in a few decades, people will be looking back at Petabyte server videos the way we currently look back at videos of the Saturn V computer and how the bits in it's memory also had to be put together by hand, like how they have to put in the drives into this larger case. Very cool thought
Love the fact that you have your Framework laptop and actually using it... thumbs up.... :)
I remember when the first 10 TB drives came on the market, and now 16 TB drives cost $280. Truly cool to see the rate of progress
Not really huge progress, just making smaller magnets on 20TB floppy disks
$280? Where? I could really use some, and the best sensible deal I've seen has been 14 TB Exos for 280€.
@@jubuttib They are 285 american bucks on amazon. But if you say in Euros it literally depends on where you live. In Germany they sell the 16TB version for 260€ at alternate but only let you buy one (the dumbest thing I've seen for an enterprise drive where nobody ever runs a single drive).
I love when Jake and Linus do a video together. Always fun to watch their banter
the sponsor of this video actually came in clutch, had bought new fans that i could not set up with razer chroma, so thanks lmg!
I love the Jake and Linus server videos. I have no idea how it works, but it's so enjoyable to watch
45 drives is an amazing company. I had a very specific request for a custom server chassis design and they put it together no problem. Great people over there!
Appreciate the love!
8:05 Linus is clearly handling the entire motherboard by holding the CPU cooler. Guess we know who bent the fins lol.
But isnt it how you supposed to hold it?
Probably the one who supposedly did a great job XD
8:34 "It's exactly how it should be right from the factory!"
"I'm not sure I believe that." -- wisest words ever.
So far, so good ... bravo !
what he said^
Jake casually calling his boss stupid at 12:55 🤣🤣
linus: "why would anyone buy a 20TB HDD"
also linus: buys 60x 20TB HDDs
Well the first statement was in regards to normal consumers. Linus is an enterprise consumer. There isn't much of a need for normal consumers to have 20TB at the moment. Maybe in 10-15 years file sizes might reach a point where 10s of terabytes make sense.
@@ragefacememeaholic5366 no, first statement was in regards to read/write speeds
Funny even though Linus' critique on R/W Speed vs. Data Density ratio of current HDDs (and thereby it results in compromises to reliability and other stuff) is actually valid (especially for normie consumers)... Most server users don't have much choice is the counter argument which he himself is ironically suffering now. I mean the two other alternative in the extreme is: 1) Pure SSD build (too expensive for 2PB, and an overkill performance for them), or 2) LTO Tapes (cheap and dense, but need a separate infrastructure for to be used effectively... unless they put Jake on 24/7 shifts to load tape back and forth).
Linus has a multimillion dollars business. He's not the one people are talking about when they make that statement.
Linus bought 0 hard drives. Seagate sent them in exchange for advertising. Good thing there was a failure, wink!
I wish I could work at a place like this sometimes. I love everything computers, even the crazy glitches that happen seemingly for no reason until one setting change fixes it.
Just looked at the datasheet for those drives, and it blows my mind to learn that they are CMR, not SMR! Seagate rates them at 285MB/s, which is bananas!
I love your server building videos!!! Bring on the NAS, the data, storage!
I remember the day when 10MB in an old XT computer was about the size of a shoebox and you would have to use programs to low-level format the drive occasionally to keep things "Lined up".
A company I worked for spent $1,000,000 for 1MB of RAM spread over 4, 7 foot by 19" cabinets. Whoa Dude!
We are living in the future!
Ngl this sounds caveman-like. When was this?
@@tanmay4217 One of these days you're going to say, "I remember when they came out with the 25 TB drives!" Some young punk is going to reply, "WHY? That's so small that you can't even put an operating system a drive that small. What did you use it for, still photos? LOLOLOL" When that happens...I want you to remember this day and your comment.
@@tanmay4217 A long time ago but there are many many of these "caveman-like" systems that keep the world running.
@@tanmay4217 Around 30-45 years ago i guess.
@@nobody7817
normal people don't use that much data but in linus case they use that much space for their work which takes a lot of space like you in the video
I love this project, its so outrageous for a regular PC user that might have 1 or 2 TB onboard, but I can't say I'm not disappointed. Its called PETAbyte project yet you only get 895TB of writable space. I know theres a difference between RAW and formatted storage, but I want to see a PB of usable storage dammit!
13:46 all these years of advancements and we don't have two tiny LEDs on those drives (one the "back" side of course, where they'd be visible). Have a power and a status LED, and you'd be all good to go
That's extra electronics not built for the process of the component doing its task and wastes power, especially for how many are in a server rack at a time, plus a power and status LED doesn't help as their are multiple power and status states of drive that no one could remember color code to just by memory..
Waiting for the next "We lost our data AGAIN video" for this server.......
Which won't take too long because... Seagate drives
LINUS you just need an IT guy 😂 OR AN IT TEAM
But they have a Linus
No they have the other 2 that are Linux tech proficient (mostly as result of last server setup fail) hopefully he is using z3 this time (nope still z2 with 15 disks per vdev) smart scans and zfs scub with all notifications enabled (he using truenas so not a roll your own setup he used last time)
Selecting each disk individually is not needed it can select all the disks automatically when creating the pool
All the IT people who would give anything to configure one petabyte watching LTT mismanage multiple petabytes. ;(
@@user-np7kr4wx7g not only does he not have enough experience and time to deal with servers constantly, but a great IT person generally has a different skill set and background/training than even the most knowledgeable consumer-tech enthusiasts. While we don't see the server-operations much I don't think anyone they have is fully qualified for such a position and those who are 90% there have a lot of other stuff happening.
Oh the irony that would be
This'll be able to hold like three or four games in 5 years time, pretty solid.
Seriously though, this is freaking amazing.
Cool stuff. I used to work in a NOC that dealt with a lot of drive repair. Cold & hot spares both have their uses and should be kept available.
More devices failing while you rebuild is what gets you so you do *not* want to waste time on having a replacement shipped if you can avoid it.
Few hundred bucks on an idle spare is way cheaper than data recovery or a catastrophic loss.
It also depends on economy of scale. You need a huge number of drives to make keeping your own stockpile of spares profitable compared to waiting a few days for each new drive.
Petabyte projects are my favorite videos. So much storage!
Perfect storage for taking a 1MB selfie every 3 seconds for ...100 YEARS! :/
That's a lot of Pickle Linuses... Linuses? Linusi? Linoos?
I did the math and that would take 1-2 more drive. Just wanted to let you know
Bruh pics take up 2-5 MB space these days.
encode as video to take advantage of compression across frames
@@Caesar512 If "octopus" becomes "octopi" then maybe the "us" is replaced with "i" when going from singular to plural, which would make it "Lini" :)
I enjoy that Linus just smiles and laughs when he messes up and called out. Gold.
My favorite video in a long while
i love how they always savage together, damn more of them
I agree 😂😂
Agree, in fact even the new mobo wife is displaced by the Jake Angry Browaifu
LET"S GO!!! Nice timing LTT team, I'm looking into building a small server rack with a NAS, switch, and a place to locate my modem and router.
Jake's tiny little giggles are everything
Love you're staff
Linus: "Andy likes it"
Jake: "You pay him to like it"
Linus: "I pay YOU to like it, just one of you is better at their job"
Funniest thing I've heard in ages
Jake has really come into his own. I really didn't like him at first. Now he's one of my favorites.
Jake dgaf. Linus is obviously not that great.
17:51 Pickle Linus
Them going EPIC is epic. I'm so blown. This server will be sick!!!
8:00 LOOOLL Great family Linus You've done well :)
This was a pretty fun video, didn't think a video about server could be fun, awesome work.
server videos are a W
Make sure to double check your ashift is set correctly on the pool! Can't change it without trashing the pool. Usually the auto detect works but I'm always paranoid the SSDs will throw it off.
Not sure what "ashift" is in this context, but the thought of a high end software system having terrible defaults is both scary and familiar as someone planning major server rework today.
Love these server/NAS vids
us too!
Linus is so passive aggressive in this scetch, I love it!
I agree with Linus, ssd smart readings are almost always inaccurate or no available. And sometimes the things just fail.
I actually had my 1tb Samsung 970 trip SMART and lock itself a couple of weeks ago with no warnings. After reading it in another PC it failed for extended high temps. It made the entire drive Read Only and I have no way to re-flash the SMART status to re-use the drive besides sending it to Samsung (which they may or may not do).
He keeps referring to what he had as "bit rot" it's not, it was mostly failed disks combined with a healthy dose of incompetence. I'm not going to say there was *no* bit rot, but it was probably pretty close to none, bit rot is slow and fairly rare, caused over long periods of time by cosmic rays and things, they will flip a bit here and there. What he had was varying degrees of disk failure. ZFS and enterprise/nas class storage devices do not try very hard to read problem areas, they just fail the disk and move on. If you get into a situation where you have an irrecoverable number of disks failed then that array needs to be taken offline, and the disks in an error state either sent to a data recovery house, or, start making an image yourself using things like ddrescue which will try quite hard to create a complete image of a semi functional disk. I've had multiple clients over the years with parity breaking levels of drive failure and yet I've never not been able to get the data back. It's probably too late for them now though if they onlined the disks and ran a scrub.
They need someone to do IT I think they basically just ignored it till they noticed a problem and then whoops lol.
I don't understand this either. Why they would actually turn on the drives and actually connect them to an array instead of backing them up with dd one by one is beyond me. That's one of the basics of how to recover from a broken disk.
@@thebaker8637 I'm sure they just don't know for the most part. Btw check out ddrescue, dd will generally fail on sketchy disks.
@@entelin Yeah, I’m not so much of a recovery expert, I never had to do it myself, by ‘dd’ I meant ‘disk cloning tool that copies raw device blocks vs file system level’, I’m sure there’s tools out there as you said in your main comment that are designed for this.
Bit rot isn't caused by cosmic rays. It's caused by the magnetic field on the metal disk platter decaying, which is a function of the material, size of the bit, and quantum mechanics. Bit rot is a lot more common as you go up in disk size. Which is why RAID consistency checks are a thing.
Super interesting as always :D
I’m in middle of building a cluster with 45Drives(great guys btw), they did actually build us a EPYC version, albeit not in the turbo config, just enhanced. Getting EPYC chips right now is a real pain though, so we mostly did Turbo Configs.
20 years from now: "they really needed all this to get only 1.2 petabytes?"
Well, diamond discs are a thing, an expensive thing.
When you go to some course, in the far future, you will get the slides on a usb 7 thumbstick with a capacity of 1 peta.
Welcome to the Exabyte
Welcome to Exabyte project I guess. :D
I bought a 30 drive Storinator cause of you guys. It’s been great. Went with 18TB drives. Put Unraid on it. Stuck with the Intel Xeon Gold 26-core though. 45 Drives that company is awesome to deal with. Thank you guys. Lovin the content as usual.
As an actual IT sys admin I am so, so sorry. You poor soul.
But hey, if it's just for your home Plex server and you're not actually trying to run a business off it, you'll probably be OK.
@@pufthemajicdragon Might want to explain why, im really curious why that would be bad?
@@pufthemajicdragon That's some Plex setup
@@ChakratosLP It depends on what you're doing with it. How much nuance do you want?
If you're building a home storage server that doesn't host any critical data - like a Plex server - this is fine. If you're running a multi-million dollar media, merchandise, and software company and storing business critical data, this is NOT fine.
This kind of setup has no redundancy and barely any fault tolerance. If more than two drives fail, which this LTT series demonstrates is entirely possible and even likely, you can lose literally everything. Nevermind if your backplane fails or your motherboard fails or you get a power spike that fries the whole thing.
LTT's previous same-as-this archive server failed, which is why we get this video in the first place. That failure means a half dozen of their staff had to spend weeks investigating, troubleshooting, attempting repairs, speccing a replacement, shopping, ordering, sorting, prepping for the build, performing the data migration, checking the data, rechecking the data, attempting to repair faulty data, scrubbing old data, and repurposing the old hardware. That's just the technical staff who will loose months of productivity fixing this problem, many of them working nights and weekends, ignoring their families and personal lives. That does not include the writers, videographers, gaffers, or editors who had to drop what they were doing to make this video. That does not include the accounting staff who has to budget all of that capital and labor expense. That does not include the lost productivity when editors have to twiddle their thumbs for weeks waiting for this recovery to finish so they can access the data they need. The cost for this failure can easily hit half a million dollars before you even count the cost of the hardware they just bought. And hell, we haven't even considered the emotional toll this is taking on everyone involved, including the families of the people LMG employs.
A proper storage server, or better, a SAN, is fully redundant, fully fault tolerant, highly available, and warrantied. It might cost $100k for the 2pb of storage, but it *just run * for at least 7 years. Every drive, every component, covered under a 4 hour onsite replacement warranty.
If your life is the only one that gets ruined when your home media server goes down, do what ya like, this kinda thing is fine (and certainly makes good ego-fodder).
But when you have a couple dozen people whose livelihoods depend upon your servers being reliable, you don't half-ass DIY your IT infrastructure and cross your fingers.
I'm so glad I stuck around long enough to catch Pickle Linus
I have no idea what half the stuff your saying is but I love watching lol
5:00 they love each other ^^
Dude!....20 years ago I was amazed at getting 20Gb HDs, only a few years earlier I was upgrading my home 386 pc to a 1GB HD...Wait 20 years and watch this vid again to see how amazed you were! :)
33 years ago I was amazed at getting 20MB HD's (and then a few years later using "stacker" compression to get 30MB)
Loved the random reference to ChrisFix :)
Please Please Please make more footage like this, it's actually interesting and makes me want to watch more. You could even create a new channel and incorporate all the LTT behind the scenes server stuff including the new house build ;-)
Getting hyped for the inevitable data loss video! Let's go!
Thanks for this video. I checked on my array for the first time in a while, and there was a supposedly dead drive. It's to be replaced and troubleshot later. Seriously, thank you.
Now we just need to see the offsite backup in case the building burns down!
This seems like a job I’d actually look forward to coming in
please more server stuff, I love it, I just build my own little Vault 2.0 with 72TB of usable storage
Less than 2 years from now, Linus will be doing another 'we sad' video because this new device was neglected into oblivion just like the last two. 😂
is a feature ... if there's an IT guy, nothing will break and that means lesser videos.
Also any rebuild of 1 20tb failed drive is going to take AGES :D
I mean, if it's going to be neglected--maybe do full mirror, those last just about forever. And investigate why so many drive failures--that's too many for a small (in total drive count) setup.
It's just content all the way down.
"neglected" is a strong word. We prefer 'gently misremembered'
😋
I bought one of the 18 TB Exos type drives a bit over a year ago for 500 €. It serves as the game drive of my new machine. So far I can't really complain about loading times. These Seagate data centre drives are a lot faster than consumer hard drives from 10 years ago. Even though you can get these drives for 300 € nowadays, I don't regret paying what I did.
Can't wait to see when you're both in your 70s working on the yottabyte server together.