All of our data is GONE!
ฝัง
- เผยแพร่เมื่อ 3 ม.ค. 2016
- We experienced a MASSIVE RAID failure on Whonnock server and lost a crapload of data. Can we get it back?
Rackspace has your Dedicated Environments covered. Head over to www.rackspace.com/ltt to learn more!
Data recovery done by www.WeRecoverData.com, thanks for the help!!
Pricing & discussion: linustechtips.com/main/topic/5...
Support us: linustechtips.com/main/topic/7...
Join our community forum: bit.ly/ZkLvE7
/ linustech
/ linustech
Intro Screen Music Credit:
Title: Laszlo - Supernova
Video Link: • [Electro] - Laszlo - S...
iTunes Download Link: itunes.apple.com/us/album/sup...
Artist Link: / laszlomusic
Outro Screen Music Credit: Approaching Nirvana - Sugar High / approachingnirvana
Sound effects provided by www.freesfx.co.uk/sfx/ - วิทยาศาสตร์และเทคโนโลยี
"How do you fix a server?"
*"Go tell Linus"*
this man's logic is genius
ah yes, genius.
he'll have plenty of tech tips for you
I know its horrible to say but that did make me giggle a bit. It's what I would say
Remember, You Only Liao Once
@@ixp8605 ayy
"We should physically print all of our videos, every frame... on paper..."
This comment didn't get enough credit in the video. +1 props.
Yepp....
In 8k resolution
*house burns down*
*500 billon houses burns down U KILLED EVERYONE EVEN ME WHY DID U DO THAT !!!!!!!!!!!!!!!!
Even the slow mo guys? 😂
You Only Liao Once hits different now because of his sign Live Laugh Liao
I like to think this that event implanted the idea into his brain
_Nice to see that I'm not the only one who thought this_
@@beefnoodlegaming same
Yeah 😂
+1
This still beats all reality shows I've ever seen on TV. The suspense is real!
My 1tb seagate hdd failed too, but I managed to move everything off before it died.. Hard drives give u warning before they die.
@@fynkozari9271 they do? does it say it in windows?
@@cloudyyy-._.-999 NO when the hard drive makes a ticking sound its dying. My HDD wouldnt let me boot, even when OS in a different drive. So I removed the seagate drive and its fixed.
@@fynkozari9271 I'm now convinced that Seagate HDDs are all PoS, a 2tb Seagate drive just crapped out on me a few days ago after using it for game storage for just a few months. Wasn't even my boot drive but I couldn't boot with it plugged in just like you.
@@cloudyyy-._.-999 The Drive will show "unhealthy" in Disk Management first. The Disk will still operational but you'll notice some file will be corrupted, sometimes load slowly or sometimes the drive is missing in windows explorer until you restart your pc. Happen to me first and i'm using it as normal for 1 month procrastinating to backup the drive and then kapuff 2 TB data is gone. Luckly its my junk drive.
I've been working in IT for 15 years and I can relate to the emotional rollercoaster he's experiencing.
Oh yes, I've been it in 3 years and I can relate.
Oh, how many times...
Same....had a co-worker pull two 'predicted failure' drives out of an array you CANNOT pull two drives out at once....he left right after replacing both drives saying one was taking a while to rebuild. I quickly put the puzzle together when people started coming to let me know they couldn't access data any longer. Erased our public share drive and domain controller, was there until 6am the next morning rebuilding the domain from scratch because the backups he was in charge of were missing. He claimed it was a virus and got promoted to manager after that.....luckily he didn't last much longer after that.
@@andymorris8521 man, that sucks.
Andy Morris Just reading your exp itself increased my bp.
you know shit goes down when linus calls for help
Very true
+HaoSs Linus always calls for help, because he's incompetent.
+John Smith Not necessarily incompetent, but when people panic they tend to forget things. But I must agree that all of this comes from the fact that they didn't think far enough, which indicates inexperience.
+ichihaifu When people have inexperience I generally wouldn't call them incompetent because that's largely unfair. I call Linus it due to the fact he has been told time and time again to perform tasks a certain way (by knowledgeable people) but never listens and ultimately fails. The networking and storage solutions have been a huge issue since moving to the new office and the biggest example of his incompetence, and in some ways negligent behaviour.
+John Smith 2.1 million subscribers and he is incompetent? huh?
"This is why it is a good idea to call tech support, because the second you call them, you will solve the problem on your own."
This definitely applies to me.
As someone in tech support i can tell you: you are not alone.
I get at least one call a day from people who are like: nvm figured it out. bye.
Or as soon as you walk into the IT Department at school your laptop works!
@@jettburmester1613 That doesn't really count. You just entered the IT Aura, a.k.a. the FIF (Fix-It Field).
@@achtsekundenfurz7876 Is this a "truer words did never dun spoke" thread?
or when they don't know how to solve the problem and then you figure it out yourself anyway
"NONE. of. it." Damn, there was some genuine venom in that sentence 👀
Thats the tone of voice that precedes someone getting beaten with that server Linus was working on if they said one more thing
Nunavut?
7:48 th-cam.com/video/74BzSTQCl_c/w-d-xo.html
I've been there, lmao
"This is the case."
"This is?"
"Yup, this is how it is."
"It is?
"Yeah."
"Is it some other way?"
"Nope, this is how it is."
"Are you-"
"Do not be afraid of the inevitable, friend. Life has led us to this point, and now we must play the hand we've been dealt. There is nothing that we can do but move forward."
@@OldManBOMBIN This sounds fake as hell but sure bud
Love how he calls tech support and he's like:
'Yeah this is Linus'
Most likely a small tech support outfit and he has to call them a LOT....
That's not really a good thing.
@@looneyburgmusic You probably missed completely the context of the video. It's definitely not easy setting up a RAID server, let alone diagnosing the problem and fixing it yourself. Linus is very much an IT professional, but he's not a server management expert although he is very knowledgeable. Your safest bet is definitely to call people who do this for a living, full-time, than to try and do everything 100% yourself (although Linus certainly did a lot on his own).
@@seanpe8474 you missed the point if looneys reply
@@gumbymofugga lol i never gave my opinion on what he said i just said he didnt understand what the other guy was saying
@@gumbymofugga lol i never gave my opinion on what he said i just said he didnt understand what the other guy was saying
"We'll have a better backup solution next time."
Says everyone after their first backup recovery disaster.
Robert Dascoli yeah, our base rule is one copy in your main server, one duplicated copy, and one off site server. As the wise old guy I used to work with said: Unless you have three copies, you don’t have a backup.
@@audigex wise words!
how about ANY backup? RAID is NOT A BACKUP.
@@audigex additionally, an untested backup is not to be treated as a backup
@@manuel0578 its bullshit I cant thumb up this comment more.
I watch this about once a year to keep my backup practices sharp.
Oh, same. Had a drive failure just this week and was reminded of this video.
That is so true! I learned from my mistakes!
Same here and this is a backup modernization year.
Sometimes, it doesn't matter, because ALL FIVE of your backups will fail all at once. That's what happened to me today.
Same
0:25 Taran: I'm actually doing some writing right now.
**Feet up on desk reading a book**
I think, he was searching Inspiration
When you’re moving files, you don’t need to do anything :D
Writing words in his head
wreading
I think it's writing some data to the server
All time favorite LTT line: “Reality TV ain’t got nothing on IT work”😂😂
Meanwhile on Keeping up with the Katrashians - Kim can’t find her Prada toilet paper and thinks kanye took it
Whenever I feel stressed I watch this...
I now feel more stressed.
Today, Jessica's processer isn't working and Jean's server isn't working
He really isn't joking though. It's really that stressful
Its the truth...
If your data doesn't exist in two seperate locations... it doesn't exist.
Not sure that's how it works bud.
@@avrona you must be fun at parties
Ah yes, I should listen to the creepy pasta that scared me as a little wee lad.
@@avrona facepalm 1000
@@avrona r/woosh
6 and a half years later and this is still the best LTT video, in my opinion.
Agreed
I watch this every time TH-cam recommends it 😂
under extreme pressure, they worked through it to come to a solution. it's a job very well done
Wonder where the guy is in the intro these days; iirc he was a major part of channel super fun.
@@LaSombraa his name is Nick and if I recall correctly he works for IFHT films which is a mountain biking channel. They're pretty cool actually
5 years now this is still my favorite video. Working in a data center the problems are real. People are waiting to work and you are left talking to someone over the phone praying they can help you with your issue. I think the only thing missing is since Linus is the owner he is saved from upper management calling down every 5 minutes for a status and "Do you know how much money we are loosing by minute?" speeches. (Also a lot of told you so's from us about converged and redundant infrastructure) :)
With this video WeRecoverData just received huge street cred
Yup, nice one you guys.
+Señor Tomas For me they lost their cred when I heard this: 16:30
+Josef Hornych I'm not an expert... are you sure that the drive letter changes were not due to Linus switching the raid cards around in his effort to get something working?
Linux does not have 'drive letters'. Only mounting points. Saying 'drive letters' in Linux doesn't mean anything.
+Josef Hornych One way you talk without end with no reason or you use terms the user may understand. How many times I had to explain but it didn't reach! It's just experience to use wrong terms to be understood, I have to do it regularly just to have a short sentence and not to explain everything. Especially when users are worried they want to know EVERY detail.
I know this video is old, but I was totally gripped. There should be a reality TV show around tech industry catastrophes.
WickedRibbon or like documentary stuff, those intense drama shows
there totally should! I would watch every episode
count me in!
Now just go to a producer and pitch your idea.
Because it's a great idea.
Isn't that basically The I.T. Crowed? Also, The Office, if it was a tech business?
"And we'll have a better backup scheme in the future, I promise"
Petabyte Server in 2022: "Yeah we need to talk Linus."
I work in IT and these types of videos remind me to be grateful that we migrated to the cloud for storage (and basically everything else except one domain controller, which we’ll be getting rid of soon in favor of full Azure AD).
I'm for sure a computer person, I almost cried with that recovery. Imagine a whole stressful week in your job and you finally got your data back. This is so satisfying.......
Some people watch soap operas or thriller movies.
We watch IT work.
Waluigi why arent you in smash
"IT work" isn't ever as exciting as this video, because real sysadmins wouldn't set up a monstrosity like this to begin with. It's guaranteed to break.
This looks about like 2 of the companies I have worked for and No they never did learn. Plus they always blamed someone else for the problem.
We may have a problem. 🤔😜
Some people juggle geese.
"This is why you should call tech support, once you call then you'll find the solution yourself"
Truer words have never been spoken.
Always fixes my wifi
Often, right at the moment, the tech answered your call.
"How can we help you ?"
"....uuuuhm ... nevermind ...."
Mostly, because , while ur on hold, you start thinking about the question they
are going to ask:
- Is the cable connected ?
- Is the power on ?
- Did you reboot ?
- Did you check the BIOS usb legacy devices ?
You just go through that checklist in your head.
Thank god for tech support phone numbers.
If I don't answer my phone at work and have to call back, 9 out of 10 times my colleague fixed it themselves.
this should be the golden rule of IT
This brings me back to a quote I heard somewhere before:
There are two kinds of people in this world. People who take backups, and people who’ve had a disk failure.
Nice one!
It was certainly one of the best swegway for werecoverdata ever. This is how it's done. Do your job properly for the right person and you'll be rewarded at some point
"NONE. OF. IT."
Damn, Linus can be intense when he wants to. Sure the music helps, but this could be a really nerdy thriller movie.
Having dealt with even some minimal it work at work, i can tell you that line hit me like the balrog in moria.
@@shutout951 if they had lost all that data, that's potentially months of work, raw footages, WIP projects, ideas, scripts, etc... down the drain
to say Linus were pissed off is an understatement
@@Codex_0613 oh absolutely. Like i said.
@@Codex_0613 months or even YEARS
I love the concept of nerdy thriller movie
Did u try turning ur server on and off again?
/facepalm
lol
RLLY?
+Pringle Lays Your joking right? Right? Please say Right!
Man you guys are fun at parties.
Alternative title:
Linus needs tech tips
1st Tip: First make shure you have back-up, or better yet backup of backup of your important files in the server (or at least, an incremental back-up)
@@typical_watcher4599 is it good practice to backup data on cloud?
@@alright7131 I think so, security experts recommended at least 2 backups, one in cloud and one offline. But of course, LTT have huge data and uploading always to the cloud for backup will always be pain in the end of large intestine.
It amazes me how much this video still enthrals and grips me despite the fact I’ve watched it dozens of times. Another desperate issue that requires dozens of man-hours to resolve but has a happy ending.
I know what the problem is... You never added RGB to the server
Filthy Hanzo Main.
Filthy Hanzo Main nahhh cuz u're a Hanzo main. stop playing Hanzo and throwing games in comp :p
Nah im genji main LoL
ohh gawd shimada weebs destroying others players competitive rank :/
hay he had some blue neons if that counts
It’s like the 5th time watching this now. Such good rewatchability value honestly. It’s like a mini-action movie
Same here, I come back every so often
@@mtftGames same here :D
I've probably seen it like 20 times
Same
Exactly my thoughts, just rewatched it as well
I'd like a follow up to this talking about A) why a raid 50 was used with 3 stripes. B) Why failed. C) and What you've moved on to D) why is the new platform better for peace of mind)
That raid/disk configuration really boggles my mind. Why would you configure it like that, it’s like creating raid and then creating multiple single point of failures for all data.
Educated guesses on a) and b) are "each card has limited bandwidth and striping together seemed like the easiest way" and "something was wrong with the motherboard and it messed up one of the RAID cards", respectively. I assume c and d are actually covered in the newer videos about NVMe-based Whonnock and their new backup scheme respectively.
@@MarkkuHanninen it's Linus, what else would you expect lmao
@@jonasdatlas4668 If you do setup like that, i'd say it's more on the lines "i don't understand what i'm doing" or "let's gamble a bit".
@@MarkkuHanninen I mean, I’m not going to dispute that. But especially in small businesses this sort of thing happens a lot, and I’ve seen far worse.
as a IT engineer myself, I know this roller coaster to well. I have had the pleasure of riding it more times than I would like to. So I still look back on this vid from time to time, just to keep me sharp knowing this ride could happen without a moments notice.
19:26 "We should physically print all of our videos, every frame on the video, on a paper!! " LOL
Mustafa Muhalab
You win.
stone is much more reliable than paper! I guess until you drop it... so one must have a flexible alternative as a backup as well. like wood
The real way to save videos would be to print the data on 5 slot punch cards.
No the real way is to transcribe every pixel onto a single moniter for every frame
You know shits hut the fan when Linus has to call tech support
Jabba had a fan?
12 Volt? 110 Volt?
Oh Yeah... Yeah that too.
Linus isn't a very deeply technical guy and he is to a large part emotions. Which means he doesn't think forward.
He didn't read up on the effect of designing a server that stripes three RAID-5. He did not plan ahead for the consequences of hardware failures or malicious software.
He likes things big and fast and lives for the moment. When things go wrong he is a trial-and-error guy.
He has the charisma and personality to do well with a tech channel presenting hardware. But he's the wrong guy to design or administrate server hardware. And that's the reason he thought three RAID controllers and parity meant "safe" and skipped backup until it was obvious the machine had issues.
yh hockey ultimate team the fan
Here in 2023 looking back and realizing this was my true calling. Here 7 years later as an IT Technician Level I and on-site technician working on business critical servers for multiple clients. Thanks LTT for showing that we, as IT specialist, can hire other people to do their jobs ;) Thanks guys for giving me crucial knowledge that I didnt know I would actually use!
I remember watching this video years ago and it almost entirely going over my head - I couldn't follow it at all.
Today figured I'd watch it again, cause it's been the 'recommended video' at the end of a few recent LTT videos, and I'm happy to say I easily followed the whole thing. Yay for learning!
"We should physically print every frame onto paper"
Classic Berkel
AresPhobos nice name
AresPhobos 500th like
AresPhobos I'd be cool honestly.
i miss him now
imagine a breeze mixing up the papers then having to reorganizing it
Linus: has whole server room.
Me: has two usb sticks.
Bitch, please I have an U L T R A 1 T B U S B T O T A L L Y N O T F A KE
I use my 5th gen iPod classic with iflash as external SSD
i have one
come at me
I have 6 32gb SD cards connected to an adapter
@@Yes----- I 48tb hdd + 2tb ssd
Both died last week 😭
"There's a thousand things I can think of and I'd rather be doing that"
I feel it.
5 years later and people are still coming back to this video, I've watched this video 5 times and it still manages to build suspence even though I know the ending!
7:40 Linus: "none of it" Cameraman: "some of it" Linus in CEO voice:"NONE OF IT"
I noticed that too. Gave me a nice laugh.
I imagined Gary Oldman ala Leon the Professional "EVERYONE!!!!"
We don't get to see Linus on CEO mode very often.
ya lol, but i guess i shouldn't laugh he looked ready to do his 3 Linus steps to freak out curse freak out and then cool down with some resolution
@@voidvev It's a really hard moment because the other guy doesn't understand why it wouldn't work, but he keeps asking and therefore making linus repeat that its fucked.
Hats off to Linus, CEO and front line IT support. Would've been easy to bark at staff to make this their problem, but instead he showed leadership and saw the problem through himself. 10/10 would watch again.
I wonder why he newer unplug PSU or use Antistatic 3m wristband
Or any What so ever prokortion against failiurs ..
Never made a PC or server with out my earth strap
@@kennethschultz6465 he's said before he often puts it on an ankle for serious builds in the past
@@kennethschultz6465 You yourself said factories (professional environments) strap their cords on the ankle. Why wouldn't Linus?
@@Sekaisien why woulden:t Linus !!!
Doe's Linus moove!!! Walk around ????
You know ther is a earth lead right !!!
So No he is not theatherd at the ankel!!!
And What would IT show and learn All the wiever's !!!
If IT was on the ankel and not on the wrist !!!
The tabel ain't Antistatic either !!!!
So why use ankel strap and not earthed tabe????
This is NOT ANTISTATIC ENVIORMENT AT ALL!!!!
@@kennethschultz6465 I have never seen the English language used in such a unique way.
The LTT team's production quality improves all the time, but the music in this one is a true master piece.
This is still my favorite soap opera episode of any show ever.
I felt like I was watching a scary movie...
So did I!!
+Adronomics This is scarier than any scary movie I've ever watched. Mainly because I've gone through something similar before, but with less important data.
+Adronomics this is way more intense!
+TheMoose2010 Yup. After multiple HD failures, I've got a networked nightly cloud backup for anything personally important and most of my large files are on my NAS. Nowadays, it's cheap to run a backup HD off your router with cloud storage for your important files.
+Adronomics Me too, it was very scary!
Did you try turning it off and on again?
+Miro Sturno lmao
+Miro Sturno like consoles press the power button to turn it off
not instructions clear, stuck into a dick fan
+Jayden Holzhauser IT crowd flashbacks.
+Jayden Holzhauser Well, it looks like it's turning itself off and on again...
Linus employees must be so glad to have a leader like him.
I've watched this 4 times this year. I love the drama and behind the scenes stuff! The whole team is just awesome imo
>Whonnock server broke
>Understandable have a nice day
HummusLord1337 oof
anyone called me?
Can I get a * B O N E L E S S C L A P*
understandable, not quite
Great day broke too
"...A thousand dollars in rental equipment"
Back then, not even Linus himself knew he would own that thousand dollar equipment himself
Recommended feed?
Something Disgusting took a trip down memory lane after that 3080, 8k video
@@DaWalkDude You know da-real shitz.
what equipment? I don't remember what he referring to
@@DaWalkDude Ah I see, I thought he was referring to some props or whatnot.
Yeah and they watercooled it no les.
I always forgot how so much have change, I miss Ivan.
Thanks man.
I have a tradition of rewatching this every year
Same
Me too
Lol
2022!
2023!
what a nice time re-watch this, can't wait for the new episode!
Dude, this video has such good re-watchability. This is like the third time I've watched this..
Fifth time for me, for once youtube has REAL drama
4ourth time for me.
Max Poma 1rst time..but Omg wanna watch this thing Consecutively
Mario S same here. linus is really awesome
I was just about to say the exact same thing! It`s like a LMG mini drama film!
Linus calling tech support, that's a first
Probably not.
R/wooosh
Lol!!!!! I will at a school and the some of the kids call tech support. Lol
@Money-Gucci-Bitches same here my friend. i am actually on hiatus from working since system administration work fucked my brain really really hard.
@Money-Gucci-Bitches D-.........for grammar and spelling.
I've probably seen this video 4 or 5 times over the last few years, and it still manages to stress me out each time I watch it.
Imagine being in the repair store and linus walks through the door with a faulty server
At least you guys had the balls to tell the world you used RAID 5.
is raid 5 bad?
+Choice777 yup
It's cool to move large files but you are risking loosing al your data, is like a house of cards.
Not true. Raid 5 uses parity so if you have one drive fail, you can swap it out and the data will be rebuilt on it.
eksadiss
RAID5 is dangerous because if an unrecoverable read error occurs on any drive during the rebuild, the entire array is lost. When rebuilding, every bit in every drive needs to be read to recalculate the parity. With modern hard drives being so large, UREs become much more likely.
Because of rebuilds being dangerous, RAID5 is the worst redundancy method in the bunch. Having a hot spare actually makes it worse since that automatically triggers a rebuild before you can take your most recent backup.
"pcieport...errors"
*tense music kicks in*
me : I DONT UNDERSTAND ANYTHING BUT THAT LOOKS VERY BAD IM SCARED
That last line is pretty much my parents thoughts every time i boot up my Linux machines 🤣
wait why is there boss music? "NONE OF IT"
That's from the kernel. I've not ever seen it spit errors continuously like that. Normally it kernel panics and the system halts
@@Nunya58294 on systems with failsafes you see this kind of stuff usually I know kali and tails I’ve seen this happen a lot because if it shuts down by anything other then the shut down button it deletes itself if you configure it to do so as per my understanding however I’ve never seen this happen with other distros and use cases though it’s kinda weird maybe it’s something to do with its own failsafes in server designed software
I was the thousand like
I know these videos are harder to produce, but I love these behind the scenes vlogs,
this is unironically better than any reality TV show
This is probably the MOST REAL, Behind the Scenes, video you've ever done. As a technician myself, there's a place you go in your head, a Focus that lets you SEE the process, commands, or schematics of what you are trying to accomplish.
Having your team of employees express a level of confidence in your ability to get it done really shows through. I'm a relatively new Subscriber to your channel, and happy that you are out there making these computer related videos. Great work. TT
Idc its four years ago, Im just realizing it now how awesome Linus is in this video. We see Linus doing what linus does, his usual thing.
But look at the context. Put it in your own terms. You work at said company. Crisis happens. Does the CEO drop his workload and fix the problem ENTIRELY himself? While keeping the staff calm and reassured that things will return to normal? Or the the CEO blame lower management and dump responsibility on them? More often than not, its the latter.
Any of Linus' fans who have been around since the NCIX days probably became geeks because of Linus. I built my first rig in 2009/2010, and was inspired that I could handle it thanks to Linus. Pretty much any of us always would say, working at LTT would be the dream job. Or at least a really cool job. But now its obvious it would be a great job that is personally fulfilling. WOrking for a guy like that, you know he cares about everyone the same. From his partner in crime, to the lowest end employee, the equivalent to the guy who wrangles the shopping carts, hes got your back the same as his main go-to guy, whoever that truly is.
Very well said!
So true. Most higher management don't react like Linus and take hold of the ship. Instead they cower and blame everyone below them. Love this guy. Honestly he seems like a good guy to work for and treats his ppl well
I feel like I'd want him as a friend... Like, a real freind xD
Nice try, Linus!
Im not saying I wish there were more videos like this due to the poor man's heart but it is a really interesting video.
Lesson 1: raid is not backups. Always have backups.
Lesson 2: never use raid 0 on anything that matters; it just increases the chance of failure. Use raid 10 or 60.
Lesson 3: on-site restore techs save a lot of time and headache :)
Lesson 4: number your drives and cables!
RAID5. not raid0
they seem to be using raid 50, multiple raid 5s under a raid 0. still better than plain old raid 0, but loss of two drives in one raid 5 element is deadly, so you still technically only have the ability to lose one drive before you need to panic. secondary to lesson 1 :)
@@andrejwalilko634 i would use raid for redundancy primarily and then speed. and still have off-site backup hehe.
@@andrejwalilko634 Why would you ever choose to do multiple raid 5s under a raid 0, wouldn't the opposite - multiple raid 0s over one raid 5 a much better option, or would it turn out to be slower for some reason.
EDIT: I guess It all comes down to hardware support.
@@MrAntarpreets the raid 0 on the outer layer gets you the speed of striping across multiple volumes, and the raid 5 on the inner layer provides the cost efficient redundancy. Raid 0 on the inner layer and 5 on the outer is not a standard organization. It would probably be slower, as there is only one checksum mechanism. Multiple checksum points on multiple raid5s lets you calculate more at once, so it should be faster.
i remember this when it went up, at the time i was making back ups on a monthly bases and only one back up. This video is what made me start making 2 back ups a week and having more then 2 back up of those back ups just in case. I know to well how much losing data like this can hurt and make you cry
This was my first ever LTT video
Then I saw the summer jobs series
And then I became a full on tech nerd
I can’t believe it’s been over 8 years
First video i ever saw was the whole room water cooling when they made videos out of that house.
19:34 "Yaaay for hiring other ppl to do our jobs" literally the most true words ever.
Hokeypokeyy ii
outsourcing
at 19:27 "We should physically print all of our videos frame by frame" Denis is awesome
+Henrich Winside Achberger yeah but denis is awesome
+Henrich Winside Achberger Nick in a nutshell
+Henrich Winside Achberger Maybe they should use PaperBak for backups. It lets you print (and then later scan!) your files. The papers look kind of like QR code and you can apparently store 0,5MB of data per paper :D
+SakariNy then you would need approx. 46,080,000 sheets of paper to backup whonnock server. 960x24x1000/0.5
+SakariNy Cool!
Got referred to this video at the end of your most recent upload concerning server errors/data loss, and man, remember when mineral oil submersion was going to be THE NEXT BIG INNOVATION? (4:50)
It was surprisingly nostalgic to see a fish-tank build while Linus was perusing customer service from that data recovery company.
This shows that even if you lose all hope , there is always that slither of hope still there somewhere. The resilience and suspense were insane
this episode is more intense than the star wars movie
True Story.
+caparzzo Yeah Like how you see Jaba the hut joining the dark side and being the new sith lord
Linus dies at the end of Pirates of The rings
I was expecting a couple of months before people who loved Star Wars started talking trash about it.
Cuz thats what teenagers seem to do these days.
Now Avatar is trash after it broke billions and billions on records. People from this generation are really sad.
Btw if that was not your intention then i apologize, it just brought me an issue very common lately.
+caparzzo snape kills dumbledore
i counted about 4 times my heart sank with his. who ever says IT is not intense has never experienced IT.
+shawn zylla or worked for IT, I'm with you there.
+shawn zylla I shadow the IT guy at the college I go to. There was a power surge one day and crashed the servers. We had to put a new power supply in one of them and the other had a messed up boot sequence. It took us about an hour to figure out/ fix the error.
+Dada blahblahblah
***** you still repair compoopers
+shawn zylla Well it really depends on what kind of IT service you are doing. If it's the basic "My computer is slow" that's nothing. If it's network or server admin services oh boy you are in a world of hell.
Ask Anthony.He knows what to do.
I forget if Anthony was working there then, but yeah, between Linus, Luke, Anthony and Jake, they would've had this down.
I got a big sad
@@thetalesofdaneandco evidently, no. The office was half or less than what it is now. I recall this happened no so long after they moved into the new office back then.
this brings me joy to see everything has been recovered
holy SHIT I felt less anxiety proposing to my wife.
+disastrix47 This comment needs more attention
Hah
+Ryan Struckman your TH-cam channel needs more attention :)
+disastrix47 Well I guess proposing a second time will do that.
+disastrix47 LTT, helping people propose to their fiances since 2016
Take the raid out and blow in it. Should work, I've had plenty of success using that method with my n64 and gameboy cartridges.
Rayman Ghafman XD
n64 and you have 64 likes. Coinsidence? Absolutely.
Rayman Ghafman idiot
'QUALITEE' poster.
beautiful
2024 anyone?
Me
For some reason TH-cam put this on an april fools list for LTT????
Linus I have been watching you since just after you and Fractal Josh organised the bunch of youtubers walk up a mountain and have a lan party.
So I remember this when it happened and it still gets me the cold sweats.
However You've never said afaik who the team were that helped you out?
I know Wendell consultancy company has been heavily involved with you since then, was it them?
yea
Absolutely
Oh yeah
Funny, this appeared on my for you after I watched Linus's newest video "This server can't loose data" which came out 8 years later
7:44 one of those moments you need people to remain silent so you don't kill anyone
Underrated comment of this video
If I was the cameraman I would have made some joke about Nunavut
@@Alexander-od6fi Jesus Christ man calm down
@@Alexander-od6fi nah it's content. If they all remained silent then the video would be boring as
@@Alexander-od6fi You have to realize that yes, he's interrupting and yes, it absolutely isn't helping, but they're making a video. The cameraman was trying to clarify, both for himself and the viewers. You're acting like he's shoving his face into the situation without permission but do you really think Linus would let him stay and record if he was actually causing harm? Linus is the boss, he's allowing him in specifically to document the recovery process so they can make a video of it later. I do tech support and coding so I know how stressful it can be to be constantly interrupted, but there's no need to be so negative about a video totally unrelated to you.
im glad i fix motorcycles and not I.T stuff.....holy shit thats stressful.
Andrew Jones ikr!
IT takes a different kind of cat to do, but we love it. Though there are days I'd rather ranch cattle xD!
Iv built a few rigs in the past for laughs and giggles to keep me out of the pub but that server is way beyond me for sure...awesome video and total respect to the guys.
for real, i work in IT and then i go home and work on motorcycles to chill out
Weird thing is. That server costs more than %90 of the motorcycles on the road.
I still think this was the best episode linus ever made, it was so exciting. But i doubt he would like to go through another heart attack like that.
Thank you guys for sharing this experience. It has been super educational as I have been recently dealing with a similar issue on a WAY smaller scale.
Edit: apparently this comment was a 'spoiler', so I've erased it. Seriously though guys, if you don't want to see the outcome of a video before you watch it then don't read the comments :P
it's youuuu
+DIY Perks Ah man, you spoiled it. :D
+DIY Perks Awesome! Love your channel man!
+DIY Perks spoiler alert
+DIY Perks I hope you've got a decent backup Matt....?
reality TV for computer nerds
The King Slayer lol yeah
Cool N boy Xbox Gamer I understand like 3% and I still like it
The King Slayer yes! this is amazing
That's because it isn't a "dramatic recreation"...it is actual reality.
Fuck You
this video is actually well shot and edited. maybe thats why I always come back an watch it again year after year
I'm so glad you got there in the end. I wish there was a service like that here. My brother used to do it for me, but he moved to Japan...
He handled this like the perfect boss
Doing it himself and keeping people calm
The difference between a boss and leader!
He wasnt calm broski. When he said "NONE OF IT." You could see how he had to cooldown for camera reasons. He wanted to explode (he did inside), but he didnt. THAT what makes a good LEADER.... and 5000 dollar intel sponsorships.
Bullshit. He edited out the temper tantrums
@@GregorioStyreco AMD*
Whenever I feel stressed I watch this...
*I now feel more stressed.*
First reply 😯🤣
@@phoenixfront Obama giving Obama a medal be like
@@michalthemichal3550 mccluer High School ma to do about 10 years ago video
@@walidfakhfakh3660 😂
I've seen this video a dozen times now and I still live it just as much as the first time
My heart was in my throat watching this video. As a small business owner myself; a techie but no tech support; I had a corrupted raid that caused me the same level of worry. 18-24 hours of no sleep to get a basic backup running as well as 4 days for data recovery.
In the end, i found that the IT company that had installed my 4 drive raid card had configured it as striped raid 0!!!! Not raid 5 as we had intended
Luckily, all data was recovered but I still have PTSD from that experience.
I have a bad experience with server data loss. One day we learned that our surge protector was busted, we learned it the hard way. It was a rainy day when a lightning strikes a nearby power transformer. The electric surge fried one of our servers. The server won't boot up, and when we hooked up the HDDs on another system, it won't recognize it.
We were a hospital, so that's mean patients record and billing, that was our most important data, and no, we don't have an off-site backup. I have to bring all the physical drive to a nearby data recovery techs, which is two islands over (this is Indonesia, we don't have that many good data recovery companies). Luckily it was Friday night, and the tech agree to work on Sunday (God bless his soul).
So I board the airplane on Saturday morning, drop the HDDs in the afternoon, and spent the rest of the day in my hotel room. I'm too nervous and stressed out to go out and look around town. I got a call on Sunday afternoon to come in and take my HDDs. They assure me that they've done everything that they can, and they were sure they got at least 90% of our data back. So I pack all my HDDs, my new HDD that contain all our rebuild data, and get on the first plane home.
After we hook up our new HDDs into our new system (my co-worker set it up while I'm away) we check it to find which of our data is missing. And it turned out, none of our data is missing. The tech manages to recover all of it.
That was the day when I put data recovery techs on top of my IT worker hierarchy.
Heru Muharman fun to read
Heru Muharman interesting
"surge protector was busted" it sucks af when the one thing we buy to protect us from that kind of situations, get damaged and mess our main component.s
gan, boleh tau nama jasa apa dan dimana? buat refrensi saya kalau tiba2 mengalami nasib yg sama. terimakasih.
Omg wtf i would die from that stress ..... Lost data from a hospital this doesnt sound good
Did you take the cartridge out and blow into it? That usually works for me.
now THAT was funny. this video is some weird mix between hardware porn and hardware horror, butr funny? not at ALL. :D
No dude, you're supposed to turn it on and off!
Did you set it to wumbo?
Try doing "POKE 59458, 62"
Everything about this video is so iconic. Linus' face on the thumbnail, the subtitles...legendary.
5 years later, and this is still one of my favorite LTT videos.
Oh my. I lost all of my data, school work, notes, family photos, all my youtube footage and videos, editing presets... basically everything I've ever done digitally was there, including most of my physical stuff I scanned in. Huge props to Seagate for helping me out getting my data back. I was fuuuuuuuuucked.
+ScrattleGG ATH Mx50s
+ScrattleGG
And that's why, even for personal use, I have online data storage with backup. Dirt cheap and everything important won't vanish when something stupid happens here.
+Jack Burton Parity certainly didn't help linus. It's not enough without multiple dedicated backups.
Why the F You were using RAID5? RAID6, RAIDZ2/3 and RAID10 are ONLY RAID's to protect such sensitive data
+Shangori Problem is some universities don't allow schoolwork to be put onto online data storage sites. I'm guessing as being data for schools is private/confidential and owned by the school, they are worried about people/governments can access these files on the online storage sites and they won't allow it for that reason.
7:49 "None of it."
Linus looked like he was about to crack the guy's head open just for breathing in the same building. This is why you always follow the 3-2-1 backup rule with critical data (3 copies, 2 on different devices and 1 off-site (cloud, rented server racks, co-location, etc.).
I said in a different comment thread, thats the tone that precedes Linus grabbing that server case (yes, drives included cause he thought they were all dead) and beating Taren severely if he had said one more thing after the third reiteration of " *_NONE_*_ of it_ "
@@KamenRiderRaiden and I wouldn't blame him. In a striped config losing any individual group means you're basically screwed, and he was pretty clear on the stuff they don't have backed up. This is a stressful situation, and if I was in his place I wouldn't be charitable towards people asking unnecessary questions, even if they seem reasonable from their perspective.
this felt like an action movie, i almost cried at the end this was a rollercoaster of emotions
I'm so happy you got everything back! Nice to see people that know better and are better than I have similar problems.
With so many comments I doubt you'll go back and read this but... I'm a technologist in a small town in Colorado. For the last 20 some years I've been a lonewolf and a few times been stuck in a server room for days after update crashes or corruption, it was very isolating and stressing. Seeing you deal with your server room was comforting and therapeutic, I've since watch tons of your videos and love the channel. Keep up what you do, it's very stressful and you're doing a great job. You rock!
I've always been accompanied by someone else in the server room all the time so that's beyond my imagination. Cursing or sobbing in a room with bunch of iron boxes must make me feel like shit..
@@IoriTatsuguchi that gives me a migraine, he is clearly someone who knows what he is doing. Someone else didn't listen and a train wreck that results in loss of years of life followed. Granted a perfect storm could have occurred, but I find the latter more likely.
I once witnessed someone destroy the company we were in by making a mistake restoring a HP Raid array on the Sunday. I was there for another reason (working back on some software), and he came in because of a HDD failure, we talked a bit, he told me he was stressed because the system was tricky, then boom, he screwed it up anyway. I don't think it was his fault, the system *was* extremely confusing.
When he realised, his face went white. He nearly broke down in tears, and he left. The next day the news was announced, 100% data loss. No backups because they thought the raid array was a backup. Ho ho. That's when I learned it wasn't. Luckily there was some files on many staff desktops (this was the late 90's, no laptops in common usage yet) and they company limped on for a while. I moved on to greener pastures, and he quite and became a pilot.
I've never used RAID since.
@@llaith2 raid when properly applied is a good safety net, how ever you hit the nail on the head, it's never a back up.
@YOURALLSHEEP Thanks for sharing this constructive and informative comment. I'll take it to heart and think on it for a while.
How much did that cost you to get restored?
That was probally 5-50k
Holy moly is this Jerry with such low likes?!
@@deadalpeca8099 That was 5 years ago
That was 5 in scale of hardness
The true cost was getting a better backup solution
This aged like fine wine. 7:28 face of pure terror.
"IT isn't stressful at all." - 2016, Linus probably
I can relate, not to this scale, but damn...
2016: All of our data is GONE!
2023: All of our channels are GONE!