Same....had a co-worker pull two 'predicted failure' drives out of an array you CANNOT pull two drives out at once....he left right after replacing both drives saying one was taking a while to rebuild. I quickly put the puzzle together when people started coming to let me know they couldn't access data any longer. Erased our public share drive and domain controller, was there until 6am the next morning rebuilding the domain from scratch because the backups he was in charge of were missing. He claimed it was a virus and got promoted to manager after that.....luckily he didn't last much longer after that.
@@looneyburgmusic You probably missed completely the context of the video. It's definitely not easy setting up a RAID server, let alone diagnosing the problem and fixing it yourself. Linus is very much an IT professional, but he's not a server management expert although he is very knowledgeable. Your safest bet is definitely to call people who do this for a living, full-time, than to try and do everything 100% yourself (although Linus certainly did a lot on his own).
"This is why it is a good idea to call tech support, because the second you call them, you will solve the problem on your own." This definitely applies to me.
@@cloudyyy-._.-999 NO when the hard drive makes a ticking sound its dying. My HDD wouldnt let me boot, even when OS in a different drive. So I removed the seagate drive and its fixed.
@@fynkozari9271 I'm now convinced that Seagate HDDs are all PoS, a 2tb Seagate drive just crapped out on me a few days ago after using it for game storage for just a few months. Wasn't even my boot drive but I couldn't boot with it plugged in just like you.
@@cloudyyy-._.-999 The Drive will show "unhealthy" in Disk Management first. The Disk will still operational but you'll notice some file will be corrupted, sometimes load slowly or sometimes the drive is missing in windows explorer until you restart your pc. Happen to me first and i'm using it as normal for 1 month procrastinating to backup the drive and then kapuff 2 TB data is gone. Luckly its my junk drive.
I've been there, lmao "This is the case." "This is?" "Yup, this is how it is." "It is? "Yeah." "Is it some other way?" "Nope, this is how it is." "Are you-" "Do not be afraid of the inevitable, friend. Life has led us to this point, and now we must play the hand we've been dealt. There is nothing that we can do but move forward."
"IT work" isn't ever as exciting as this video, because real sysadmins wouldn't set up a monstrosity like this to begin with. It's guaranteed to break.
Mostly, because , while ur on hold, you start thinking about the question they are going to ask: - Is the cable connected ? - Is the power on ? - Did you reboot ? - Did you check the BIOS usb legacy devices ? You just go through that checklist in your head. Thank god for tech support phone numbers.
Idc its four years ago, Im just realizing it now how awesome Linus is in this video. We see Linus doing what linus does, his usual thing. But look at the context. Put it in your own terms. You work at said company. Crisis happens. Does the CEO drop his workload and fix the problem ENTIRELY himself? While keeping the staff calm and reassured that things will return to normal? Or the the CEO blame lower management and dump responsibility on them? More often than not, its the latter. Any of Linus' fans who have been around since the NCIX days probably became geeks because of Linus. I built my first rig in 2009/2010, and was inspired that I could handle it thanks to Linus. Pretty much any of us always would say, working at LTT would be the dream job. Or at least a really cool job. But now its obvious it would be a great job that is personally fulfilling. WOrking for a guy like that, you know he cares about everyone the same. From his partner in crime, to the lowest end employee, the equivalent to the guy who wrangles the shopping carts, hes got your back the same as his main go-to guy, whoever that truly is.
So true. Most higher management don't react like Linus and take hold of the ship. Instead they cower and blame everyone below them. Love this guy. Honestly he seems like a good guy to work for and treats his ppl well
@@adame7277 Well, CEO is just three letters tbh. Some CEOs would not, but then they might have 3 meaningless PhDs in Economics and 300.000 employees around the world. (My daily ^^). A challenge to your perspectives here ladies n gents. :)
+John Smith Not necessarily incompetent, but when people panic they tend to forget things. But I must agree that all of this comes from the fact that they didn't think far enough, which indicates inexperience.
+ichihaifu When people have inexperience I generally wouldn't call them incompetent because that's largely unfair. I call Linus it due to the fact he has been told time and time again to perform tasks a certain way (by knowledgeable people) but never listens and ultimately fails. The networking and storage solutions have been a huge issue since moving to the new office and the biggest example of his incompetence, and in some ways negligent behaviour.
Robert Dascoli yeah, our base rule is one copy in your main server, one duplicated copy, and one off site server. As the wise old guy I used to work with said: Unless you have three copies, you don’t have a backup.
Lesson 1: raid is not backups. Always have backups. Lesson 2: never use raid 0 on anything that matters; it just increases the chance of failure. Use raid 10 or 60. Lesson 3: on-site restore techs save a lot of time and headache :) Lesson 4: number your drives and cables!
they seem to be using raid 50, multiple raid 5s under a raid 0. still better than plain old raid 0, but loss of two drives in one raid 5 element is deadly, so you still technically only have the ability to lose one drive before you need to panic. secondary to lesson 1 :)
@@andrejwalilko634 Why would you ever choose to do multiple raid 5s under a raid 0, wouldn't the opposite - multiple raid 0s over one raid 5 a much better option, or would it turn out to be slower for some reason. EDIT: I guess It all comes down to hardware support.
@@MrAntarpreets the raid 0 on the outer layer gets you the speed of striping across multiple volumes, and the raid 5 on the inner layer provides the cost efficient redundancy. Raid 0 on the inner layer and 5 on the outer is not a standard organization. It would probably be slower, as there is only one checksum mechanism. Multiple checksum points on multiple raid5s lets you calculate more at once, so it should be faster.
5 years now this is still my favorite video. Working in a data center the problems are real. People are waiting to work and you are left talking to someone over the phone praying they can help you with your issue. I think the only thing missing is since Linus is the owner he is saved from upper management calling down every 5 minutes for a status and "Do you know how much money we are loosing by minute?" speeches. (Also a lot of told you so's from us about converged and redundant infrastructure) :)
Hats off to Linus, CEO and front line IT support. Would've been easy to bark at staff to make this their problem, but instead he showed leadership and saw the problem through himself. 10/10 would watch again.
I wonder why he newer unplug PSU or use Antistatic 3m wristband Or any What so ever prokortion against failiurs .. Never made a PC or server with out my earth strap
@@Sekaisien why woulden:t Linus !!! Doe's Linus moove!!! Walk around ???? You know ther is a earth lead right !!! So No he is not theatherd at the ankel!!! And What would IT show and learn All the wiever's !!! If IT was on the ankel and not on the wrist !!! The tabel ain't Antistatic either !!!! So why use ankel strap and not earthed tabe???? This is NOT ANTISTATIC ENVIORMENT AT ALL!!!!
@@shutout951 if they had lost all that data, that's potentially months of work, raw footages, WIP projects, ideas, scripts, etc... down the drain to say Linus were pissed off is an understatement
With so many comments I doubt you'll go back and read this but... I'm a technologist in a small town in Colorado. For the last 20 some years I've been a lonewolf and a few times been stuck in a server room for days after update crashes or corruption, it was very isolating and stressing. Seeing you deal with your server room was comforting and therapeutic, I've since watch tons of your videos and love the channel. Keep up what you do, it's very stressful and you're doing a great job. You rock!
I've always been accompanied by someone else in the server room all the time so that's beyond my imagination. Cursing or sobbing in a room with bunch of iron boxes must make me feel like shit..
@@IoriTatsuguchi that gives me a migraine, he is clearly someone who knows what he is doing. Someone else didn't listen and a train wreck that results in loss of years of life followed. Granted a perfect storm could have occurred, but I find the latter more likely.
I once witnessed someone destroy the company we were in by making a mistake restoring a HP Raid array on the Sunday. I was there for another reason (working back on some software), and he came in because of a HDD failure, we talked a bit, he told me he was stressed because the system was tricky, then boom, he screwed it up anyway. I don't think it was his fault, the system *was* extremely confusing. When he realised, his face went white. He nearly broke down in tears, and he left. The next day the news was announced, 100% data loss. No backups because they thought the raid array was a backup. Ho ho. That's when I learned it wasn't. Luckily there was some files on many staff desktops (this was the late 90's, no laptops in common usage yet) and they company limped on for a while. I moved on to greener pastures, and he quite and became a pilot. I've never used RAID since.
+Josef Hornych I'm not an expert... are you sure that the drive letter changes were not due to Linus switching the raid cards around in his effort to get something working?
+Josef Hornych One way you talk without end with no reason or you use terms the user may understand. How many times I had to explain but it didn't reach! It's just experience to use wrong terms to be understood, I have to do it regularly just to have a short sentence and not to explain everything. Especially when users are worried they want to know EVERY detail.
It was certainly one of the best swegway for werecoverdata ever. This is how it's done. Do your job properly for the right person and you'll be rewarded at some point
Edit: apparently this comment was a 'spoiler', so I've erased it. Seriously though guys, if you don't want to see the outcome of a video before you watch it then don't read the comments :P
@@voidvev It's a really hard moment because the other guy doesn't understand why it wouldn't work, but he keeps asking and therefore making linus repeat that its fucked.
It amazes me how much this video still enthrals and grips me despite the fact I’ve watched it dozens of times. Another desperate issue that requires dozens of man-hours to resolve but has a happy ending.
He wasnt calm broski. When he said "NONE OF IT." You could see how he had to cooldown for camera reasons. He wanted to explode (he did inside), but he didnt. THAT what makes a good LEADER.... and 5000 dollar intel sponsorships.
Linus isn't a very deeply technical guy and he is to a large part emotions. Which means he doesn't think forward. He didn't read up on the effect of designing a server that stripes three RAID-5. He did not plan ahead for the consequences of hardware failures or malicious software. He likes things big and fast and lives for the moment. When things go wrong he is a trial-and-error guy. He has the charisma and personality to do well with a tech channel presenting hardware. But he's the wrong guy to design or administrate server hardware. And that's the reason he thought three RAID controllers and parity meant "safe" and skipped backup until it was obvious the machine had issues.
as a IT engineer myself, I know this roller coaster to well. I have had the pleasure of riding it more times than I would like to. So I still look back on this vid from time to time, just to keep me sharp knowing this ride could happen without a moments notice.
I have a bad experience with server data loss. One day we learned that our surge protector was busted, we learned it the hard way. It was a rainy day when a lightning strikes a nearby power transformer. The electric surge fried one of our servers. The server won't boot up, and when we hooked up the HDDs on another system, it won't recognize it. We were a hospital, so that's mean patients record and billing, that was our most important data, and no, we don't have an off-site backup. I have to bring all the physical drive to a nearby data recovery techs, which is two islands over (this is Indonesia, we don't have that many good data recovery companies). Luckily it was Friday night, and the tech agree to work on Sunday (God bless his soul). So I board the airplane on Saturday morning, drop the HDDs in the afternoon, and spent the rest of the day in my hotel room. I'm too nervous and stressed out to go out and look around town. I got a call on Sunday afternoon to come in and take my HDDs. They assure me that they've done everything that they can, and they were sure they got at least 90% of our data back. So I pack all my HDDs, my new HDD that contain all our rebuild data, and get on the first plane home. After we hook up our new HDDs into our new system (my co-worker set it up while I'm away) we check it to find which of our data is missing. And it turned out, none of our data is missing. The tech manages to recover all of it. That was the day when I put data recovery techs on top of my IT worker hierarchy.
"surge protector was busted" it sucks af when the one thing we buy to protect us from that kind of situations, get damaged and mess our main component.s
This brings me back to a quote I heard somewhere before: There are two kinds of people in this world. People who take backups, and people who’ve had a disk failure.
This video is full of feelings of terror, loss, regret and guilt (for not having setup a proper backup system beforehand). I can feel your pain and I'm very glad that you got everything back. I think this video is extremely useful to teach other people the value of backups and to show a way to potentially recover your data. Well done and thank you for this! I did not find it "funny" as you said but I was gripped by it from beginning to end.
Here in 2023 looking back and realizing this was my true calling. Here 7 years later as an IT Technician Level I and on-site technician working on business critical servers for multiple clients. Thanks LTT for showing that we, as IT specialist, can hire other people to do their jobs ;) Thanks guys for giving me crucial knowledge that I didnt know I would actually use!
hilarious to see people calling this a clickbait title. Maybe "We got it all back" will make the 23 minutes of suspense more enjoyable. And for sure, Reality TV aint got nothing on this. This is as real as it gets
7:49 "None of it." Linus looked like he was about to crack the guy's head open just for breathing in the same building. This is why you always follow the 3-2-1 backup rule with critical data (3 copies, 2 on different devices and 1 off-site (cloud, rented server racks, co-location, etc.).
I said in a different comment thread, thats the tone that precedes Linus grabbing that server case (yes, drives included cause he thought they were all dead) and beating Taren severely if he had said one more thing after the third reiteration of " *_NONE_*_ of it_ "
@@KamenRiderRaiden and I wouldn't blame him. In a striped config losing any individual group means you're basically screwed, and he was pretty clear on the stuff they don't have backed up. This is a stressful situation, and if I was in his place I wouldn't be charitable towards people asking unnecessary questions, even if they seem reasonable from their perspective.
I'd like a follow up to this talking about A) why a raid 50 was used with 3 stripes. B) Why failed. C) and What you've moved on to D) why is the new platform better for peace of mind)
3 ปีที่แล้ว +4
That raid/disk configuration really boggles my mind. Why would you configure it like that, it’s like creating raid and then creating multiple single point of failures for all data.
Educated guesses on a) and b) are "each card has limited bandwidth and striping together seemed like the easiest way" and "something was wrong with the motherboard and it messed up one of the RAID cards", respectively. I assume c and d are actually covered in the newer videos about NVMe-based Whonnock and their new backup scheme respectively.
@@Nunya58294 on systems with failsafes you see this kind of stuff usually I know kali and tails I’ve seen this happen a lot because if it shuts down by anything other then the shut down button it deletes itself if you configure it to do so as per my understanding however I’ve never seen this happen with other distros and use cases though it’s kinda weird maybe it’s something to do with its own failsafes in server designed software
@@Alexander-od6fi You have to realize that yes, he's interrupting and yes, it absolutely isn't helping, but they're making a video. The cameraman was trying to clarify, both for himself and the viewers. You're acting like he's shoving his face into the situation without permission but do you really think Linus would let him stay and record if he was actually causing harm? Linus is the boss, he's allowing him in specifically to document the recovery process so they can make a video of it later. I do tech support and coding so I know how stressful it can be to be constantly interrupted, but there's no need to be so negative about a video totally unrelated to you.
I remember watching this video years ago and it almost entirely going over my head - I couldn't follow it at all. Today figured I'd watch it again, cause it's been the 'recommended video' at the end of a few recent LTT videos, and I'm happy to say I easily followed the whole thing. Yay for learning!
This is probably the MOST REAL, Behind the Scenes, video you've ever done. As a technician myself, there's a place you go in your head, a Focus that lets you SEE the process, commands, or schematics of what you are trying to accomplish. Having your team of employees express a level of confidence in your ability to get it done really shows through. I'm a relatively new Subscriber to your channel, and happy that you are out there making these computer related videos. Great work. TT
@@alright7131 I think so, security experts recommended at least 2 backups, one in cloud and one offline. But of course, LTT have huge data and uploading always to the cloud for backup will always be pain in the end of large intestine.
I'm for sure a computer person, I almost cried with that recovery. Imagine a whole stressful week in your job and you finally got your data back. This is so satisfying.......
+shawn zylla I shadow the IT guy at the college I go to. There was a power surge one day and crashed the servers. We had to put a new power supply in one of them and the other had a messed up boot sequence. It took us about an hour to figure out/ fix the error.
+shawn zylla Well it really depends on what kind of IT service you are doing. If it's the basic "My computer is slow" that's nothing. If it's network or server admin services oh boy you are in a world of hell.
5 years later and people are still coming back to this video, I've watched this video 5 times and it still manages to build suspence even though I know the ending!
This felt like a reality tech tv-show episode. At the end I was expecting to hear a narrator say "Next time on Linus's servers, see how linus and the team tackles the problem of ..."
I had it when my phone's 64GB sd card failed.. Like I literally have everything in it with only 28gb of it Backed up. Thankfully , only my exfat table was corrupted , so I could recover it all ( took whole night) and the only thing I lost was 2 movies I downloaded on the day it corrupted ( which is nothing )..
reminds me of booting my first ever PC. psu switch (on/off) was flipped off thus not allowing the power button to turn the machine on. spent a solid 5 minutes in a frenzy of frozen panic before i checked the back just in case and found out the switch was off. flipped it to on, booted the thing and it started up. one of the biggest reliefs in my life to this date
This was incredibly suspenseful and gripping. Reality TV has NOTHING on this. As someone who has experienced data loss - I think that this is something that a LOT of people can relate to, especially given the kind of volume of data you're talking about. *yikes* You guys DESPERATELY need LTO tape backup. DESPERATELY!
Well, this doesn't work anymore, the storage needs exploded in the recent years because high bit depth video recording and 8K started to be a thing. Much better is a ZFS, which just transfers snapshots to a second location. It just needs to send the additional written data and some metadata since the last transfer and you can get older states from your storage server on both locations. If a transfer fails, it can be resumed or rolled back and you can do tripple redundancy, if you like.
With some more music, documentary-type interviews, some sci fi effects and a deep-voiced narration in background would have turn this into a Netflix blockbuster.
Except instead of Linus Sebastian, its Linus Torvalds, and the server was one holding the new version of Linux he was supposed to give to IBM for their new Z series mainframe...
Oh my. I lost all of my data, school work, notes, family photos, all my youtube footage and videos, editing presets... basically everything I've ever done digitally was there, including most of my physical stuff I scanned in. Huge props to Seagate for helping me out getting my data back. I was fuuuuuuuuucked.
+ScrattleGG And that's why, even for personal use, I have online data storage with backup. Dirt cheap and everything important won't vanish when something stupid happens here.
+Shangori Problem is some universities don't allow schoolwork to be put onto online data storage sites. I'm guessing as being data for schools is private/confidential and owned by the school, they are worried about people/governments can access these files on the online storage sites and they won't allow it for that reason.
+Henrich Winside Achberger Maybe they should use PaperBak for backups. It lets you print (and then later scan!) your files. The papers look kind of like QR code and you can apparently store 0,5MB of data per paper :D
I was expecting a couple of months before people who loved Star Wars started talking trash about it. Cuz thats what teenagers seem to do these days. Now Avatar is trash after it broke billions and billions on records. People from this generation are really sad. Btw if that was not your intention then i apologize, it just brought me an issue very common lately.
i remember this when it went up, at the time i was making back ups on a monthly bases and only one back up. This video is what made me start making 2 back ups a week and having more then 2 back up of those back ups just in case. I know to well how much losing data like this can hurt and make you cry
+Adronomics This is scarier than any scary movie I've ever watched. Mainly because I've gone through something similar before, but with less important data.
+TheMoose2010 Yup. After multiple HD failures, I've got a networked nightly cloud backup for anything personally important and most of my large files are on my NAS. Nowadays, it's cheap to run a backup HD off your router with cloud storage for your important files.
for some strange fucking reason I want to feel a tension like that because although I probably had similar amounts of stress for different things, I for some reason want that tension and especially the relieve after it.
You can do unlimited incremental backups to the cloud cheaply of all your data with unlimited history for about $60 a month. If you start early then 20Tb isn't that much, and with a Gbit connection it should take max two days to recover all of that. We normally just ask our provider to give us a daily bandwidth boost when we have to restore backups.
It's so funny to watch this years after it happend, look at how empty the server room was back then :'D Also seeing some old faces that don't work at LTT anymore makes me kinda sad :/
@@thetalesofdaneandco Well, we did see most of Dennis from the " you only liao once". Also, i will like to add that the amount of pain i have recieved from the Live Laugh Liao sign is far too great to thepoint of it becomes funny
I'm glad this video is still here. It's important for people to know that RAID = Speed and uptime (redundancy), NOT backup. RAID 5 has a major flaw with our "giant" hard drives today. I had a 4x2TB setup (software) and when one drive failed, it tried to recreate so many times, that another one failed before it could finish = all gone. No problem though, purchased new drives and restored 5TB from offsite backup (friends house). I actually replaced my 4x 2TB in RAID5 with 2x3TB in RAID0. It's cheaper and simpler. Of course if one drive fails, it's all gone, but doesn't matter because I don't need the uptime. But I also have weekly backup on a 2x3TB RAID0 NAS, and constantly synced cloud backup at all times. NEVER confuse RAID with backup.
Dihelson Mendonca Yes, unfortunately it looks like you get max 6-7MB/s. It’s fine for my use, as it syncs immediately and I don’t edit my files too often. The price is unbeatable though.
Sorry for coming back to you after 4 months. But I think the take away message here isn't "RAID5 stops working in 2009". One of his RAID cards corrupted due to a bad (consumer-grade) motherboard, so all the drives hanging at that card got kicked out. Only a RAID-1 using atleast 2 RAID cards could've survived such an event, as this is equal to 8 out of 24 harddrives disconnecting at once. Enterprises use multipath SAS to mitigate such events. To not overextend this post, here is my list of takeaway messages: - backup (3-2-1 scheme) - never use hardware RAID cards - use enterprise-quality gear instead of "gaming" motherboards for your business-critical data - don't mess around, if you don't have any clue and get help quickly (like Linus did) That beeing said, I think you shouldn't do RAID at all. As you said, RAID is for speed and uptime. As you mentioned you don't need uptime and modern HDDs saturating gigabit NICs, I don't see a reason to use RAID0 in your case. Instead of losing ALL data in case of a drive failure, you'd only lose data on that drive and recover it from backup.
GlennJacobsen You backup to a RAID0 NAS?!? That seems risky, either of those drives fail you've lost your backup. Well, I guess you also have your off-site backup so maybe it's not too bad. Years ago back in the 10GB-30GB drive size days, I had a drive fail in my desktop, then found out the USB external drive had also failed. I lost so many pictures and documents. So now I'm a bit overboard with backups. I use BRTFS RAID 1. It's not quite the same as traditional RAID 1 where every block is mirrored on both drives. It's a little more versatile, it can use 2+ drives of any size, it always ensures there are 2 copies of every block spread across the drives. I have two 1 TB drives, one 1.5TB, and one 3TB drive. When one drive fails, it does not have to rewrite everything to a new drive, but only needs to write blocks that have only one remaining copy. A second BRTFS RAID 1 array of one 4TB and one 5TB drive is used for my backup. A nice thing with BTRFS is I can bring up the second array, issue a BTRFS send command, and it quickly backup up only the files that changed since the last send. This makes backups faster because it does not have to do date compares and calculate checksums, it just compares the integrated file checksums. All my pictures and videos are also copied to my wife's desktop, as well as backed up to Google Photos and Flickr. My important documents are backup up to Google Drive. I don't backup my OS to offsite, with Linux I can reinstall from a USB live easily. I triple boot, Windows 10 and two different Linux distros anyway.
eksadiss RAID5 is dangerous because if an unrecoverable read error occurs on any drive during the rebuild, the entire array is lost. When rebuilding, every bit in every drive needs to be read to recalculate the parity. With modern hard drives being so large, UREs become much more likely. Because of rebuilds being dangerous, RAID5 is the worst redundancy method in the bunch. Having a hot spare actually makes it worse since that automatically triggers a rebuild before you can take your most recent backup.
You guys are idiots. I think there's more value in the knowledge gained from events like these than a systems administrator who would be paid a decent salary to manage such a minuet infrastructure, in the scheme of things. Considering actual outages like this will be so infrequent, as well as the fact that the client size is so limited, a full-time guy isn't needed. Insulting people for being creative and intuitive and thinking outside of the box with regards to finding solutions to problems makes me occasionally wonder what's wrong with the world. Remember, this wasn't their fault.
+TROLLFACE KNIGHT Exactly. That's the funny part. He'd essentially diagnose the same problem which wouldn't have been his fault in the first place (it wasn't LTT's fault) and would've undergone a similar data-recovery procedure.
+D-MMA Quite right. Everyone learns from things like this, so if anything i'm grateful to LTT for uploading this. So many people, even the most elite of sys admins experience issues like these. Seeing people actually experience and have to deal with it is so valuable, and we can all gain so much knowledge and understanding from it.
Fabian Miserus nouja mijn naam komt van een aflevering van spongebob van vroeger die ik mij nog half herriner. spongebob vangt een speciale paarse kwal en wil hem een naam geven, kan niet op een naam komen dus noemt hem geen naam en vond het bullshit dat je van google je naam moest invullen
Got referred to this video at the end of your most recent upload concerning server errors/data loss, and man, remember when mineral oil submersion was going to be THE NEXT BIG INNOVATION? (4:50) It was surprisingly nostalgic to see a fish-tank build while Linus was perusing customer service from that data recovery company.
3:01 what goes through the minds of the folks in charge of packaging will always confuse me. "Hey Jim this area over here seems a little bland, mind adding something there?" "Way ahead of ya Tim, I already added this completely unrelated face of an unattractive woman there."
+MeLikeBigBoom I sorta think they didn't even try to hire a graphic/design artist. But hey, what's more abstract and eye-popping than an oily western woman on the cover?
I work in IT and these types of videos remind me to be grateful that we migrated to the cloud for storage (and basically everything else except one domain controller, which we’ll be getting rid of soon in favor of full Azure AD).
@@Thomas-Almanza There is really not a practical reason to crash a huge working server with people currently working with it just for an advert, that and, just having the garbled data when it is "recovered"
Exactly. This whole thing was completely preventable because any competent person doing data management knows that everything should be backed up in several locations, otherwise the system should never be used.
@@MorzenMebs i get that quite often where I work. Folks are not even bothered about their backups (or lack of them) yet they are the first ones to moan when shit hits the fan and their whole business is one step away from going completly tits up. One major rule - serios about business? Then better be serious about your backup.
You missed the part where he was mid backup when this happened. The backup was being made before it broke... and while it broke. Yes, he could have done it sooner, but he did start before he had a problem.
Don't. 20 years ago it was a fantastic career choice. These days it'll send you to an early grave. For a start, your starting pay will be, equivalent for the time, 20% of the old days. The rest of the time will be holding things together with tape and string, because budgets have been turned down for the 3rd year straight. Lastly, a little saying I heard a few years ago: "Accounts departments love IT departments. Before there were IT departments, everyone hated Accounts. Now there is IT, everyone hates the IT department instead of Accounts."
setting up things can be fun, maintaining them is a depressing adventure. there are people w/ ptsd after a few years working with situations like the one presented in this video.. no exaggeration.
![BOWKYLION - วิงวอน (ex-change) [Official MV]](http://i.ytimg.com/vi/cLdnZWDaO-w/mqdefault.jpg)
"We should physically print all of our videos, every frame... on paper..."
This comment didn't get enough credit in the video. +1 props.
Yepp....
In 8k resolution
*house burns down*
*500 billon houses burns down U KILLED EVERYONE EVEN ME WHY DID U DO THAT !!!!!!!!!!!!!!!!
Even the slow mo guys? 😂
"How do you fix a server?"
*"Go tell Linus"*
this man's logic is genius
ah yes, genius.
he'll have plenty of tech tips for you
I know its horrible to say but that did make me giggle a bit. It's what I would say
Remember, You Only Liao Once
@@ixp8605 ayy
I've been working in IT for 15 years and I can relate to the emotional rollercoaster he's experiencing.
Oh yes, I've been it in 3 years and I can relate.
Oh, how many times...
Same....had a co-worker pull two 'predicted failure' drives out of an array you CANNOT pull two drives out at once....he left right after replacing both drives saying one was taking a while to rebuild. I quickly put the puzzle together when people started coming to let me know they couldn't access data any longer. Erased our public share drive and domain controller, was there until 6am the next morning rebuilding the domain from scratch because the backups he was in charge of were missing. He claimed it was a virus and got promoted to manager after that.....luckily he didn't last much longer after that.
@@AndyMorris_BAP man, that sucks.
Andy Morris Just reading your exp itself increased my bp.
You Only Liao Once hits different now because of his sign Live Laugh Liao
I like to think this that event implanted the idea into his brain
_Nice to see that I'm not the only one who thought this_
@@beefnoodlegaming same
Yeah 😂
+1
Love how he calls tech support and he's like:
'Yeah this is Linus'
Most likely a small tech support outfit and he has to call them a LOT....
That's not really a good thing.
@@looneyburgmusic You probably missed completely the context of the video. It's definitely not easy setting up a RAID server, let alone diagnosing the problem and fixing it yourself. Linus is very much an IT professional, but he's not a server management expert although he is very knowledgeable. Your safest bet is definitely to call people who do this for a living, full-time, than to try and do everything 100% yourself (although Linus certainly did a lot on his own).
@@seanpe8474 you missed the point if looneys reply
@@gumbymofugga lol i never gave my opinion on what he said i just said he didnt understand what the other guy was saying
@@gumbymofugga lol i never gave my opinion on what he said i just said he didnt understand what the other guy was saying
"This is why it is a good idea to call tech support, because the second you call them, you will solve the problem on your own."
This definitely applies to me.
As someone in tech support i can tell you: you are not alone.
I get at least one call a day from people who are like: nvm figured it out. bye.
Or as soon as you walk into the IT Department at school your laptop works!
@@jettburmester1613 That doesn't really count. You just entered the IT Aura, a.k.a. the FIF (Fix-It Field).
@@achtsekundenfurz7876 Is this a "truer words did never dun spoke" thread?
@@jettburmester1613 the fucking WHAT at school? Christ when did I get old
This still beats all reality shows I've ever seen on TV. The suspense is real!
My 1tb seagate hdd failed too, but I managed to move everything off before it died.. Hard drives give u warning before they die.
@@fynkozari9271 they do? does it say it in windows?
@@cloudyyy-._.-999 NO when the hard drive makes a ticking sound its dying. My HDD wouldnt let me boot, even when OS in a different drive. So I removed the seagate drive and its fixed.
@@fynkozari9271 I'm now convinced that Seagate HDDs are all PoS, a 2tb Seagate drive just crapped out on me a few days ago after using it for game storage for just a few months. Wasn't even my boot drive but I couldn't boot with it plugged in just like you.
@@cloudyyy-._.-999 The Drive will show "unhealthy" in Disk Management first. The Disk will still operational but you'll notice some file will be corrupted, sometimes load slowly or sometimes the drive is missing in windows explorer until you restart your pc. Happen to me first and i'm using it as normal for 1 month procrastinating to backup the drive and then kapuff 2 TB data is gone. Luckly its my junk drive.
"NONE. of. it." Damn, there was some genuine venom in that sentence 👀
Thats the tone of voice that precedes someone getting beaten with that server Linus was working on if they said one more thing
Nunavut?
7:48 th-cam.com/video/74BzSTQCl_c/w-d-xo.html
I've been there, lmao
"This is the case."
"This is?"
"Yup, this is how it is."
"It is?
"Yeah."
"Is it some other way?"
"Nope, this is how it is."
"Are you-"
"Do not be afraid of the inevitable, friend. Life has led us to this point, and now we must play the hand we've been dealt. There is nothing that we can do but move forward."
@@OldManBOMBIN This sounds fake as hell but sure bud
I know this video is old, but I was totally gripped. There should be a reality TV show around tech industry catastrophes.
WickedRibbon or like documentary stuff, those intense drama shows
there totally should! I would watch every episode
count me in!
Now just go to a producer and pitch your idea.
Because it's a great idea.
Isn't that basically The I.T. Crowed? Also, The Office, if it was a tech business?
Some people watch soap operas or thriller movies.
We watch IT work.
Waluigi why arent you in smash
"IT work" isn't ever as exciting as this video, because real sysadmins wouldn't set up a monstrosity like this to begin with. It's guaranteed to break.
This looks about like 2 of the companies I have worked for and No they never did learn. Plus they always blamed someone else for the problem.
We may have a problem. 🤔😜
Some people juggle geese.
"This is why you should call tech support, once you call then you'll find the solution yourself"
Truer words have never been spoken.
Always fixes my wifi
Often, right at the moment, the tech answered your call.
"How can we help you ?"
"....uuuuhm ... nevermind ...."
Mostly, because , while ur on hold, you start thinking about the question they
are going to ask:
- Is the cable connected ?
- Is the power on ?
- Did you reboot ?
- Did you check the BIOS usb legacy devices ?
You just go through that checklist in your head.
Thank god for tech support phone numbers.
If I don't answer my phone at work and have to call back, 9 out of 10 times my colleague fixed it themselves.
this should be the golden rule of IT
I watch this about once a year to keep my backup practices sharp.
Oh, same. Had a drive failure just this week and was reminded of this video.
That is so true! I learned from my mistakes!
Same here and this is a backup modernization year.
Same
@@milesfarberHOW
It’s like the 5th time watching this now. Such good rewatchability value honestly. It’s like a mini-action movie
Same here, I come back every so often
@@mtftGames same here :D
I've probably seen it like 20 times
Same
Exactly my thoughts, just rewatched it as well
Idc its four years ago, Im just realizing it now how awesome Linus is in this video. We see Linus doing what linus does, his usual thing.
But look at the context. Put it in your own terms. You work at said company. Crisis happens. Does the CEO drop his workload and fix the problem ENTIRELY himself? While keeping the staff calm and reassured that things will return to normal? Or the the CEO blame lower management and dump responsibility on them? More often than not, its the latter.
Any of Linus' fans who have been around since the NCIX days probably became geeks because of Linus. I built my first rig in 2009/2010, and was inspired that I could handle it thanks to Linus. Pretty much any of us always would say, working at LTT would be the dream job. Or at least a really cool job. But now its obvious it would be a great job that is personally fulfilling. WOrking for a guy like that, you know he cares about everyone the same. From his partner in crime, to the lowest end employee, the equivalent to the guy who wrangles the shopping carts, hes got your back the same as his main go-to guy, whoever that truly is.
Very well said!
So true. Most higher management don't react like Linus and take hold of the ship. Instead they cower and blame everyone below them. Love this guy. Honestly he seems like a good guy to work for and treats his ppl well
I feel like I'd want him as a friend... Like, a real freind xD
Nice try, Linus!
@@adame7277
Well, CEO is just three letters tbh. Some CEOs would not, but then they might have 3 meaningless PhDs in Economics and 300.000 employees around the world. (My daily ^^).
A challenge to your perspectives here ladies n gents. :)
you know shit goes down when linus calls for help
Very true
+HaoSs Linus always calls for help, because he's incompetent.
+John Smith Not necessarily incompetent, but when people panic they tend to forget things. But I must agree that all of this comes from the fact that they didn't think far enough, which indicates inexperience.
+ichihaifu When people have inexperience I generally wouldn't call them incompetent because that's largely unfair. I call Linus it due to the fact he has been told time and time again to perform tasks a certain way (by knowledgeable people) but never listens and ultimately fails. The networking and storage solutions have been a huge issue since moving to the new office and the biggest example of his incompetence, and in some ways negligent behaviour.
+John Smith 2.1 million subscribers and he is incompetent? huh?
6 and a half years later and this is still the best LTT video, in my opinion.
Agreed
I watch this every time TH-cam recommends it 😂
under extreme pressure, they worked through it to come to a solution. it's a job very well done
@@LaSombraa his name is Nick and if I recall correctly he works for IFHT films which is a mountain biking channel. They're pretty cool actually
Did u try turning ur server on and off again?
/facepalm
lol
RLLY?
+Pringle Lays Your joking right? Right? Please say Right!
Man you guys are fun at parties.
"We'll have a better backup solution next time."
Says everyone after their first backup recovery disaster.
Robert Dascoli yeah, our base rule is one copy in your main server, one duplicated copy, and one off site server. As the wise old guy I used to work with said: Unless you have three copies, you don’t have a backup.
@@audigex wise words!
how about ANY backup? RAID is NOT A BACKUP.
@@audigex additionally, an untested backup is not to be treated as a backup
@@manuel0578 its bullshit I cant thumb up this comment more.
Lesson 1: raid is not backups. Always have backups.
Lesson 2: never use raid 0 on anything that matters; it just increases the chance of failure. Use raid 10 or 60.
Lesson 3: on-site restore techs save a lot of time and headache :)
Lesson 4: number your drives and cables!
RAID5. not raid0
they seem to be using raid 50, multiple raid 5s under a raid 0. still better than plain old raid 0, but loss of two drives in one raid 5 element is deadly, so you still technically only have the ability to lose one drive before you need to panic. secondary to lesson 1 :)
@@andrejwalilko634 i would use raid for redundancy primarily and then speed. and still have off-site backup hehe.
@@andrejwalilko634 Why would you ever choose to do multiple raid 5s under a raid 0, wouldn't the opposite - multiple raid 0s over one raid 5 a much better option, or would it turn out to be slower for some reason.
EDIT: I guess It all comes down to hardware support.
@@MrAntarpreets the raid 0 on the outer layer gets you the speed of striping across multiple volumes, and the raid 5 on the inner layer provides the cost efficient redundancy. Raid 0 on the inner layer and 5 on the outer is not a standard organization. It would probably be slower, as there is only one checksum mechanism. Multiple checksum points on multiple raid5s lets you calculate more at once, so it should be faster.
5 years now this is still my favorite video. Working in a data center the problems are real. People are waiting to work and you are left talking to someone over the phone praying they can help you with your issue. I think the only thing missing is since Linus is the owner he is saved from upper management calling down every 5 minutes for a status and "Do you know how much money we are loosing by minute?" speeches. (Also a lot of told you so's from us about converged and redundant infrastructure) :)
Hats off to Linus, CEO and front line IT support. Would've been easy to bark at staff to make this their problem, but instead he showed leadership and saw the problem through himself. 10/10 would watch again.
I wonder why he newer unplug PSU or use Antistatic 3m wristband
Or any What so ever prokortion against failiurs ..
Never made a PC or server with out my earth strap
@@kennethschultz6465 he's said before he often puts it on an ankle for serious builds in the past
@@kennethschultz6465 You yourself said factories (professional environments) strap their cords on the ankle. Why wouldn't Linus?
@@Sekaisien why woulden:t Linus !!!
Doe's Linus moove!!! Walk around ????
You know ther is a earth lead right !!!
So No he is not theatherd at the ankel!!!
And What would IT show and learn All the wiever's !!!
If IT was on the ankel and not on the wrist !!!
The tabel ain't Antistatic either !!!!
So why use ankel strap and not earthed tabe????
This is NOT ANTISTATIC ENVIORMENT AT ALL!!!!
@@kennethschultz6465 I have never seen the English language used in such a unique way.
"NONE. OF. IT."
Damn, Linus can be intense when he wants to. Sure the music helps, but this could be a really nerdy thriller movie.
Having dealt with even some minimal it work at work, i can tell you that line hit me like the balrog in moria.
@@shutout951 if they had lost all that data, that's potentially months of work, raw footages, WIP projects, ideas, scripts, etc... down the drain
to say Linus were pissed off is an understatement
@@Codex_0613 oh absolutely. Like i said.
@@Codex_0613 months or even YEARS
I love the concept of nerdy thriller movie
All time favorite LTT line: “Reality TV ain’t got nothing on IT work”😂😂
Meanwhile on Keeping up with the Katrashians - Kim can’t find her Prada toilet paper and thinks kanye took it
Whenever I feel stressed I watch this...
I now feel more stressed.
Today, Jessica's processer isn't working and Jean's server isn't working
He really isn't joking though. It's really that stressful
Its the truth...
0:25 Taran: I'm actually doing some writing right now.
**Feet up on desk reading a book**
I think, he was searching Inspiration
When you’re moving files, you don’t need to do anything :D
Writing words in his head
wreading
I think it's writing some data to the server
I know what the problem is... You never added RGB to the server
Filthy Hanzo Main.
Filthy Hanzo Main nahhh cuz u're a Hanzo main. stop playing Hanzo and throwing games in comp :p
Nah im genji main LoL
ohh gawd shimada weebs destroying others players competitive rank :/
hay he had some blue neons if that counts
With so many comments I doubt you'll go back and read this but... I'm a technologist in a small town in Colorado. For the last 20 some years I've been a lonewolf and a few times been stuck in a server room for days after update crashes or corruption, it was very isolating and stressing. Seeing you deal with your server room was comforting and therapeutic, I've since watch tons of your videos and love the channel. Keep up what you do, it's very stressful and you're doing a great job. You rock!
I've always been accompanied by someone else in the server room all the time so that's beyond my imagination. Cursing or sobbing in a room with bunch of iron boxes must make me feel like shit..
@@IoriTatsuguchi that gives me a migraine, he is clearly someone who knows what he is doing. Someone else didn't listen and a train wreck that results in loss of years of life followed. Granted a perfect storm could have occurred, but I find the latter more likely.
I once witnessed someone destroy the company we were in by making a mistake restoring a HP Raid array on the Sunday. I was there for another reason (working back on some software), and he came in because of a HDD failure, we talked a bit, he told me he was stressed because the system was tricky, then boom, he screwed it up anyway. I don't think it was his fault, the system *was* extremely confusing.
When he realised, his face went white. He nearly broke down in tears, and he left. The next day the news was announced, 100% data loss. No backups because they thought the raid array was a backup. Ho ho. That's when I learned it wasn't. Luckily there was some files on many staff desktops (this was the late 90's, no laptops in common usage yet) and they company limped on for a while. I moved on to greener pastures, and he quite and became a pilot.
I've never used RAID since.
@@llaith2 raid when properly applied is a good safety net, how ever you hit the nail on the head, it's never a back up.
@YOURALLSHEEP Thanks for sharing this constructive and informative comment. I'll take it to heart and think on it for a while.
With this video WeRecoverData just received huge street cred
Yup, nice one you guys.
+Señor Tomas For me they lost their cred when I heard this: 16:30
+Josef Hornych I'm not an expert... are you sure that the drive letter changes were not due to Linus switching the raid cards around in his effort to get something working?
Linux does not have 'drive letters'. Only mounting points. Saying 'drive letters' in Linux doesn't mean anything.
+Josef Hornych One way you talk without end with no reason or you use terms the user may understand. How many times I had to explain but it didn't reach! It's just experience to use wrong terms to be understood, I have to do it regularly just to have a short sentence and not to explain everything. Especially when users are worried they want to know EVERY detail.
It was certainly one of the best swegway for werecoverdata ever. This is how it's done. Do your job properly for the right person and you'll be rewarded at some point
If your data doesn't exist in two seperate locations... it doesn't exist.
Not sure that's how it works bud.
@@avrona you must be fun at parties
Ah yes, I should listen to the creepy pasta that scared me as a little wee lad.
@@avrona facepalm 1000
@@avrona r/woosh
Edit: apparently this comment was a 'spoiler', so I've erased it. Seriously though guys, if you don't want to see the outcome of a video before you watch it then don't read the comments :P
it's youuuu
+DIY Perks Ah man, you spoiled it. :D
+DIY Perks Awesome! Love your channel man!
+DIY Perks spoiler alert
+DIY Perks I hope you've got a decent backup Matt....?
7:40 Linus: "none of it" Cameraman: "some of it" Linus in CEO voice:"NONE OF IT"
I noticed that too. Gave me a nice laugh.
I imagined Gary Oldman ala Leon the Professional "EVERYONE!!!!"
We don't get to see Linus on CEO mode very often.
ya lol, but i guess i shouldn't laugh he looked ready to do his 3 Linus steps to freak out curse freak out and then cool down with some resolution
@@voidvev It's a really hard moment because the other guy doesn't understand why it wouldn't work, but he keeps asking and therefore making linus repeat that its fucked.
It amazes me how much this video still enthrals and grips me despite the fact I’ve watched it dozens of times. Another desperate issue that requires dozens of man-hours to resolve but has a happy ending.
He handled this like the perfect boss
Doing it himself and keeping people calm
The difference between a boss and leader!
He wasnt calm broski. When he said "NONE OF IT." You could see how he had to cooldown for camera reasons. He wanted to explode (he did inside), but he didnt. THAT what makes a good LEADER.... and 5000 dollar intel sponsorships.
Bullshit. He edited out the temper tantrums
@@GregorioStyreco AMD*
You know shits hut the fan when Linus has to call tech support
Jabba had a fan?
12 Volt? 110 Volt?
Oh Yeah... Yeah that too.
Linus isn't a very deeply technical guy and he is to a large part emotions. Which means he doesn't think forward.
He didn't read up on the effect of designing a server that stripes three RAID-5. He did not plan ahead for the consequences of hardware failures or malicious software.
He likes things big and fast and lives for the moment. When things go wrong he is a trial-and-error guy.
He has the charisma and personality to do well with a tech channel presenting hardware. But he's the wrong guy to design or administrate server hardware. And that's the reason he thought three RAID controllers and parity meant "safe" and skipped backup until it was obvious the machine had issues.
yh hockey ultimate team the fan
Dude, this video has such good re-watchability. This is like the third time I've watched this..
4ourth time for me.
Max Poma 1rst time..but Omg wanna watch this thing Consecutively
Mario S same here. linus is really awesome
I was just about to say the exact same thing! It`s like a LMG mini drama film!
Mario S SAME. I've gone through a situation like Linus had before, much better when it's not happening to you.
as a IT engineer myself, I know this roller coaster to well. I have had the pleasure of riding it more times than I would like to. So I still look back on this vid from time to time, just to keep me sharp knowing this ride could happen without a moments notice.
I have a bad experience with server data loss. One day we learned that our surge protector was busted, we learned it the hard way. It was a rainy day when a lightning strikes a nearby power transformer. The electric surge fried one of our servers. The server won't boot up, and when we hooked up the HDDs on another system, it won't recognize it.
We were a hospital, so that's mean patients record and billing, that was our most important data, and no, we don't have an off-site backup. I have to bring all the physical drive to a nearby data recovery techs, which is two islands over (this is Indonesia, we don't have that many good data recovery companies). Luckily it was Friday night, and the tech agree to work on Sunday (God bless his soul).
So I board the airplane on Saturday morning, drop the HDDs in the afternoon, and spent the rest of the day in my hotel room. I'm too nervous and stressed out to go out and look around town. I got a call on Sunday afternoon to come in and take my HDDs. They assure me that they've done everything that they can, and they were sure they got at least 90% of our data back. So I pack all my HDDs, my new HDD that contain all our rebuild data, and get on the first plane home.
After we hook up our new HDDs into our new system (my co-worker set it up while I'm away) we check it to find which of our data is missing. And it turned out, none of our data is missing. The tech manages to recover all of it.
That was the day when I put data recovery techs on top of my IT worker hierarchy.
Heru Muharman fun to read
Heru Muharman interesting
"surge protector was busted" it sucks af when the one thing we buy to protect us from that kind of situations, get damaged and mess our main component.s
gan, boleh tau nama jasa apa dan dimana? buat refrensi saya kalau tiba2 mengalami nasib yg sama. terimakasih.
Omg wtf i would die from that stress ..... Lost data from a hospital this doesnt sound good
Linus calling tech support, that's a first
Probably not.
R/wooosh
Lol!!!!! I will at a school and the some of the kids call tech support. Lol
@Money-Gucci-Bitches same here my friend. i am actually on hiatus from working since system administration work fucked my brain really really hard.
@Money-Gucci-Bitches D-.........for grammar and spelling.
19:34 "Yaaay for hiring other ppl to do our jobs" literally the most true words ever.
Hokeypokeyy ii
outsourcing
This brings me back to a quote I heard somewhere before:
There are two kinds of people in this world. People who take backups, and people who’ve had a disk failure.
Nice one!
Did you try turning it off and on again?
+Miro Sturno lmao
+Miro Sturno like consoles press the power button to turn it off
not instructions clear, stuck into a dick fan
+Jayden Holzhauser IT crowd flashbacks.
+Jayden Holzhauser Well, it looks like it's turning itself off and on again...
This video is full of feelings of terror, loss, regret and guilt (for not having setup a proper backup system beforehand). I can feel your pain and I'm very glad that you got everything back. I think this video is extremely useful to teach other people the value of backups and to show a way to potentially recover your data. Well done and thank you for this! I did not find it "funny" as you said but I was gripped by it from beginning to end.
The lesson "do backup constantly" was taught by Microsoft to everyone with the last Windows 10 major update recently.
"...A thousand dollars in rental equipment"
Back then, not even Linus himself knew he would own that thousand dollar equipment himself
Recommended feed?
Something Disgusting took a trip down memory lane after that 3080, 8k video
@@DaWalkDude You know da-real shitz.
and drop 1000$ cpu's without batting an eye
Whenever I feel stressed I watch this...
I now feel more stressed.
Here in 2023 looking back and realizing this was my true calling. Here 7 years later as an IT Technician Level I and on-site technician working on business critical servers for multiple clients. Thanks LTT for showing that we, as IT specialist, can hire other people to do their jobs ;) Thanks guys for giving me crucial knowledge that I didnt know I would actually use!
How much did that cost you to get restored?
That was probally 5-50k
Holy moly is this Jerry with such low likes?!
@@deadalpeca8099 That was 5 years ago
That was 5 in scale of hardness
The true cost was getting a better backup solution
hilarious to see people calling this a clickbait title. Maybe "We got it all back" will make the 23 minutes of suspense more enjoyable. And for sure, Reality TV aint got nothing on this. This is as real as it gets
Well i mean, for one, theres no multiple takes to make sure they get the "spiciest" angle for some of the drama, so that adds to the authenticity
7:49 "None of it."
Linus looked like he was about to crack the guy's head open just for breathing in the same building. This is why you always follow the 3-2-1 backup rule with critical data (3 copies, 2 on different devices and 1 off-site (cloud, rented server racks, co-location, etc.).
I said in a different comment thread, thats the tone that precedes Linus grabbing that server case (yes, drives included cause he thought they were all dead) and beating Taren severely if he had said one more thing after the third reiteration of " *_NONE_*_ of it_ "
@@KamenRiderRaiden and I wouldn't blame him. In a striped config losing any individual group means you're basically screwed, and he was pretty clear on the stuff they don't have backed up. This is a stressful situation, and if I was in his place I wouldn't be charitable towards people asking unnecessary questions, even if they seem reasonable from their perspective.
I'd like a follow up to this talking about A) why a raid 50 was used with 3 stripes. B) Why failed. C) and What you've moved on to D) why is the new platform better for peace of mind)
That raid/disk configuration really boggles my mind. Why would you configure it like that, it’s like creating raid and then creating multiple single point of failures for all data.
Educated guesses on a) and b) are "each card has limited bandwidth and striping together seemed like the easiest way" and "something was wrong with the motherboard and it messed up one of the RAID cards", respectively. I assume c and d are actually covered in the newer videos about NVMe-based Whonnock and their new backup scheme respectively.
@ it's Linus, what else would you expect lmao
@@jdatlas4668 If you do setup like that, i'd say it's more on the lines "i don't understand what i'm doing" or "let's gamble a bit".
@ I mean, I’m not going to dispute that. But especially in small businesses this sort of thing happens a lot, and I’ve seen far worse.
Did you take the cartridge out and blow into it? That usually works for me.
now THAT was funny. this video is some weird mix between hardware porn and hardware horror, butr funny? not at ALL. :D
No dude, you're supposed to turn it on and off!
Did you set it to wumbo?
Try doing "POKE 59458, 62"
"pcieport...errors"
*tense music kicks in*
me : I DONT UNDERSTAND ANYTHING BUT THAT LOOKS VERY BAD IM SCARED
That last line is pretty much my parents thoughts every time i boot up my Linux machines 🤣
wait why is there boss music? "NONE OF IT"
That's from the kernel. I've not ever seen it spit errors continuously like that. Normally it kernel panics and the system halts
@@Nunya58294 on systems with failsafes you see this kind of stuff usually I know kali and tails I’ve seen this happen a lot because if it shuts down by anything other then the shut down button it deletes itself if you configure it to do so as per my understanding however I’ve never seen this happen with other distros and use cases though it’s kinda weird maybe it’s something to do with its own failsafes in server designed software
I was the thousand like
7:44 one of those moments you need people to remain silent so you don't kill anyone
Underrated comment of this video
If I was the cameraman I would have made some joke about Nunavut
@@Alexander-od6fi Jesus Christ man calm down
@@Alexander-od6fi nah it's content. If they all remained silent then the video would be boring as
@@Alexander-od6fi You have to realize that yes, he's interrupting and yes, it absolutely isn't helping, but they're making a video. The cameraman was trying to clarify, both for himself and the viewers. You're acting like he's shoving his face into the situation without permission but do you really think Linus would let him stay and record if he was actually causing harm? Linus is the boss, he's allowing him in specifically to document the recovery process so they can make a video of it later. I do tech support and coding so I know how stressful it can be to be constantly interrupted, but there's no need to be so negative about a video totally unrelated to you.
I remember watching this video years ago and it almost entirely going over my head - I couldn't follow it at all.
Today figured I'd watch it again, cause it's been the 'recommended video' at the end of a few recent LTT videos, and I'm happy to say I easily followed the whole thing. Yay for learning!
Whenever I feel stressed I watch this...
*I now feel more stressed.*
First reply 😯🤣
@@phoenixfront Obama giving Obama a medal be like
@@michalthemichal3550 mccluer High School ma to do about 10 years ago video
@@walidfakhfakh3660 😂
Take the raid out and blow in it. Should work, I've had plenty of success using that method with my n64 and gameboy cartridges.
Rayman Ghafman XD
n64 and you have 64 likes. Coinsidence? Absolutely.
Rayman Ghafman idiot
'QUALITEE' poster.
beautiful
"We should physically print every frame onto paper"
Classic Berkel
AresPhobos nice name
AresPhobos 500th like
AresPhobos I'd be cool honestly.
i miss him now
imagine a breeze mixing up the papers then having to reorganizing it
This is probably the MOST REAL, Behind the Scenes, video you've ever done. As a technician myself, there's a place you go in your head, a Focus that lets you SEE the process, commands, or schematics of what you are trying to accomplish.
Having your team of employees express a level of confidence in your ability to get it done really shows through. I'm a relatively new Subscriber to your channel, and happy that you are out there making these computer related videos. Great work. TT
>Whonnock server broke
>Understandable have a nice day
HummusLord1337 oof
anyone called me?
Can I get a * B O N E L E S S C L A P*
understandable, not quite
Great day broke too
I can't seem to stop coming back to this video. This video was so good.
Same
So true
Same! It's well edited and the carefully-placed tense music really kicks up the atmosphere and suspense at all the right times.
@@xplinux22 Agreed. ;)
This is my comfort youtube video. I watch at least twice a year.
19:26 "We should physically print all of our videos, every frame on the video, on a paper!! " LOL
Mustafa Muhalab
You win.
stone is much more reliable than paper! I guess until you drop it... so one must have a flexible alternative as a backup as well. like wood
The real way to save videos would be to print the data on 5 slot punch cards.
No the real way is to transcribe every pixel onto a single moniter for every frame
Alternative title:
Linus needs tech tips
1st Tip: First make shure you have back-up, or better yet backup of backup of your important files in the server (or at least, an incremental back-up)
@@typical_watcher4599 is it good practice to backup data on cloud?
@@alright7131 I think so, security experts recommended at least 2 backups, one in cloud and one offline. But of course, LTT have huge data and uploading always to the cloud for backup will always be pain in the end of large intestine.
I'm for sure a computer person, I almost cried with that recovery. Imagine a whole stressful week in your job and you finally got your data back. This is so satisfying.......
i counted about 4 times my heart sank with his. who ever says IT is not intense has never experienced IT.
+shawn zylla or worked for IT, I'm with you there.
+shawn zylla I shadow the IT guy at the college I go to. There was a power surge one day and crashed the servers. We had to put a new power supply in one of them and the other had a messed up boot sequence. It took us about an hour to figure out/ fix the error.
+Dada blahblahblah
***** you still repair compoopers
+shawn zylla Well it really depends on what kind of IT service you are doing. If it's the basic "My computer is slow" that's nothing. If it's network or server admin services oh boy you are in a world of hell.
Linus: has whole server room.
Me: has two usb sticks.
Bitch, please I have an U L T R A 1 T B U S B T O T A L L Y N O T F A KE
I use my 5th gen iPod classic with iflash as external SSD
i have one
come at me
I have 6 32gb SD cards connected to an adapter
@@Yes----- I 48tb hdd + 2tb ssd
Both died last week 😭
5 years later and people are still coming back to this video, I've watched this video 5 times and it still manages to build suspence even though I know the ending!
This felt like a reality tech tv-show episode. At the end I was expecting to hear a narrator say "Next time on Linus's servers, see how linus and the team tackles the problem of ..."
Accidentally releasing a development photo of the server
Linus's emotions matched mine when my first pc didn't boot up.
I went through exactly that yesterday! Stayed up till 02:00 to figure out I had used the wrong outputs in the power supply.
I had it when my phone's 64GB sd card failed.. Like I literally have everything in it with only 28gb of it Backed up. Thankfully , only my exfat table was corrupted , so I could recover it all ( took whole night) and the only thing I lost was 2 movies I downloaded on the day it corrupted ( which is nothing )..
hahah, it happened to me before, I rebuilt the whole thing to figure out that the power supply was off
reminds me of booting my first ever PC. psu switch (on/off) was flipped off thus not allowing the power button to turn the machine on. spent a solid 5 minutes in a frenzy of frozen panic before i checked the back just in case and found out the switch was off. flipped it to on, booted the thing and it started up.
one of the biggest reliefs in my life to this date
reminds me of when i fucked up my dads computer with viruses and staying up until like 3 am or so to get the computer working again. i was 5.
This was incredibly suspenseful and gripping. Reality TV has NOTHING on this.
As someone who has experienced data loss - I think that this is something that a LOT of people can relate to, especially given the kind of volume of data you're talking about.
*yikes*
You guys DESPERATELY need LTO tape backup. DESPERATELY!
@Asdew
Actually, I have. Several times. Great movie!
Well, this doesn't work anymore, the storage needs exploded in the recent years because high bit depth video recording and 8K started to be a thing.
Much better is a ZFS, which just transfers snapshots to a second location. It just needs to send the additional written data and some metadata since the last transfer and you can get older states from your storage server on both locations. If a transfer fails, it can be resumed or rolled back and you can do tripple redundancy, if you like.
You cant work off of tape back up, thats seriously a different need than what this server muti uses are for
The LTT team's production quality improves all the time, but the music in this one is a true master piece.
With some more music, documentary-type interviews, some sci fi effects and a deep-voiced narration in background would have turn this into a Netflix blockbuster.
Except instead of Linus Sebastian, its Linus Torvalds, and the server was one holding the new version of Linux he was supposed to give to IBM for their new Z series mainframe...
nn
nnn
nn
nn
Best horror movie on youtube.
Haha
how is this a horror movie?
The Gold Cat How is this not?
Oh my. I lost all of my data, school work, notes, family photos, all my youtube footage and videos, editing presets... basically everything I've ever done digitally was there, including most of my physical stuff I scanned in. Huge props to Seagate for helping me out getting my data back. I was fuuuuuuuuucked.
+ScrattleGG ATH Mx50s
+ScrattleGG
And that's why, even for personal use, I have online data storage with backup. Dirt cheap and everything important won't vanish when something stupid happens here.
+Jack Burton Parity certainly didn't help linus. It's not enough without multiple dedicated backups.
Why the F You were using RAID5? RAID6, RAIDZ2/3 and RAID10 are ONLY RAID's to protect such sensitive data
+Shangori Problem is some universities don't allow schoolwork to be put onto online data storage sites. I'm guessing as being data for schools is private/confidential and owned by the school, they are worried about people/governments can access these files on the online storage sites and they won't allow it for that reason.
Linus employees must be so glad to have a leader like him.
at 19:27 "We should physically print all of our videos frame by frame" Denis is awesome
+Henrich Winside Achberger yeah but denis is awesome
+Henrich Winside Achberger Nick in a nutshell
+Henrich Winside Achberger Maybe they should use PaperBak for backups. It lets you print (and then later scan!) your files. The papers look kind of like QR code and you can apparently store 0,5MB of data per paper :D
+SakariNy then you would need approx. 46,080,000 sheets of paper to backup whonnock server. 960x24x1000/0.5
+SakariNy Cool!
7:98 That “None of it” was said with so much genuine anger and frustration
I don't think that's how seconds work
@@mazzalnx its in metric
@@mazzalnx at least TH-cam's engineers where smart enough not to trust user input.
7:48 r/therewasanattempt
Because RAID0(RAID5(8 drives),RAID5(8 drives),RAID5(8 drives)) is quite risky, if one RAID5 fails, all the data is gone.
this episode is more intense than the star wars movie
True Story.
+caparzzo Yeah Like how you see Jaba the hut joining the dark side and being the new sith lord
Linus dies at the end of Pirates of The rings
I was expecting a couple of months before people who loved Star Wars started talking trash about it.
Cuz thats what teenagers seem to do these days.
Now Avatar is trash after it broke billions and billions on records. People from this generation are really sad.
Btw if that was not your intention then i apologize, it just brought me an issue very common lately.
+caparzzo snape kills dumbledore
i remember this when it went up, at the time i was making back ups on a monthly bases and only one back up. This video is what made me start making 2 back ups a week and having more then 2 back up of those back ups just in case. I know to well how much losing data like this can hurt and make you cry
I felt like I was watching a scary movie...
So did I!!
+Adronomics This is scarier than any scary movie I've ever watched. Mainly because I've gone through something similar before, but with less important data.
+Adronomics this is way more intense!
+TheMoose2010 Yup. After multiple HD failures, I've got a networked nightly cloud backup for anything personally important and most of my large files are on my NAS. Nowadays, it's cheap to run a backup HD off your router with cloud storage for your important files.
+Adronomics Me too, it was very scary!
Did you try turning it off and on again
Says ron and moss from it crowd :)
he did in the video multiple times
it was a joke...
r/woooooosh
XRNZ9 LMAO
The moment that Linus hit enter, my screen went black. And then I realized that I was updating my Graphics card drivers.
Francisco Amaro Loureiro he Sounds Like A Cop In NFS Most Wanted '05
did you have a mini heart attack when you saw the black screen?
lmfao lol
yep ive gotten scared as i forget im updating nvidia drivers...lol.
You know your a tech head when this was more dramatic and thrilling than normal Television.
Wymsic_ ...
This comment is so true!
Ya and triggered my PTSD from 25y of IT.... gone through this before.
Joe Prestera lol so funny
for some strange fucking reason I want to feel a tension like that because although I probably had similar amounts of stress for different things, I for some reason want that tension and especially the relieve after it.
This is why it is important to always backup your data.
IKR!!
20 TB of data is a lot though, and they have double back ups :)
They just moved. That was his backup.
Yeah, that's REALLY bad that they didn't have at least 2 backups. Different types and locations.
You can do unlimited incremental backups to the cloud cheaply of all your data with unlimited history for about $60 a month. If you start early then 20Tb isn't that much, and with a Gbit connection it should take max two days to recover all of that. We normally just ask our provider to give us a daily bandwidth boost when we have to restore backups.
It's so funny to watch this years after it happend, look at how empty the server room was back then :'D
Also seeing some old faces that don't work at LTT anymore makes me kinda sad :/
Does this asian looking guy still works there? (Idk most of them as i am not regularly watching him
@@remigiusznowak7277 Depends which one you are vaguely describing. Dennis does, Andy does, Jono does.
@@thetalesofdaneandco Well, we did see most of Dennis from the " you only liao once".
Also, i will like to add that the amount of pain i have recieved from the Live Laugh Liao sign is far too great to thepoint of it becomes funny
"There's a thousand things I can think of and I'd rather be doing that"
I feel it.
I'm glad this video is still here. It's important for people to know that RAID = Speed and uptime (redundancy), NOT backup.
RAID 5 has a major flaw with our "giant" hard drives today. I had a 4x2TB setup (software) and when one drive failed, it tried to recreate so many times, that another one failed before it could finish = all gone.
No problem though, purchased new drives and restored 5TB from offsite backup (friends house).
I actually replaced my 4x 2TB in RAID5 with 2x3TB in RAID0. It's cheaper and simpler. Of course if one drive fails, it's all gone, but doesn't matter because I don't need the uptime.
But I also have weekly backup on a 2x3TB RAID0 NAS, and constantly synced cloud backup at all times.
NEVER confuse RAID with backup.
Dihelson Mendonca JottaCloud. Approx $80 per year and unlimited storage/devices.
Dihelson Mendonca Yes, unfortunately it looks like you get max 6-7MB/s. It’s fine for my use, as it syncs immediately and I don’t edit my files too often. The price is unbeatable though.
Dihelson Mendonca Oh wow, that’s horrible!
6-7MB/s would be approx 50-60Mbps (even though it’s 1/1Gbit here.)
Sorry for coming back to you after 4 months. But I think the take away message here isn't "RAID5 stops working in 2009". One of his RAID cards corrupted due to a bad (consumer-grade) motherboard, so all the drives hanging at that card got kicked out. Only a RAID-1 using atleast 2 RAID cards could've survived such an event, as this is equal to 8 out of 24 harddrives disconnecting at once. Enterprises use multipath SAS to mitigate such events.
To not overextend this post, here is my list of takeaway messages:
- backup (3-2-1 scheme)
- never use hardware RAID cards
- use enterprise-quality gear instead of "gaming" motherboards for your business-critical data
- don't mess around, if you don't have any clue and get help quickly (like Linus did)
That beeing said, I think you shouldn't do RAID at all. As you said, RAID is for speed and uptime. As you mentioned you don't need uptime and modern HDDs saturating gigabit NICs, I don't see a reason to use RAID0 in your case. Instead of losing ALL data in case of a drive failure, you'd only lose data on that drive and recover it from backup.
GlennJacobsen You backup to a RAID0 NAS?!? That seems risky, either of those drives fail you've lost your backup. Well, I guess you also have your off-site backup so maybe it's not too bad.
Years ago back in the 10GB-30GB drive size days, I had a drive fail in my desktop, then found out the USB external drive had also failed. I lost so many pictures and documents. So now I'm a bit overboard with backups.
I use BRTFS RAID 1. It's not quite the same as traditional RAID 1 where every block is mirrored on both drives. It's a little more versatile, it can use 2+ drives of any size, it always ensures there are 2 copies of every block spread across the drives. I have two 1 TB drives, one 1.5TB, and one 3TB drive. When one drive fails, it does not have to rewrite everything to a new drive, but only needs to write blocks that have only one remaining copy. A second BRTFS RAID 1 array of one 4TB and one 5TB drive is used for my backup. A nice thing with BTRFS is I can bring up the second array, issue a BTRFS send command, and it quickly backup up only the files that changed since the last send. This makes backups faster because it does not have to do date compares and calculate checksums, it just compares the integrated file checksums. All my pictures and videos are also copied to my wife's desktop, as well as backed up to Google Photos and Flickr. My important documents are backup up to Google Drive. I don't backup my OS to offsite, with Linux I can reinstall from a USB live easily. I triple boot, Windows 10 and two different Linux distros anyway.
reality TV for computer nerds
The King Slayer lol yeah
Cool N boy Xbox Gamer I understand like 3% and I still like it
The King Slayer yes! this is amazing
That's because it isn't a "dramatic recreation"...it is actual reality.
Fuck You
At least you guys had the balls to tell the world you used RAID 5.
is raid 5 bad?
+Choice777 yup
It's cool to move large files but you are risking loosing al your data, is like a house of cards.
Not true. Raid 5 uses parity so if you have one drive fail, you can swap it out and the data will be rebuilt on it.
eksadiss
RAID5 is dangerous because if an unrecoverable read error occurs on any drive during the rebuild, the entire array is lost. When rebuilding, every bit in every drive needs to be read to recalculate the parity. With modern hard drives being so large, UREs become much more likely.
Because of rebuilds being dangerous, RAID5 is the worst redundancy method in the bunch. Having a hot spare actually makes it worse since that automatically triggers a rebuild before you can take your most recent backup.
I know these videos are harder to produce, but I love these behind the scenes vlogs,
this is unironically better than any reality TV show
It's pretty impressive how you guys manage to consistently fuck up everything.
You guys are idiots. I think there's more value in the knowledge gained from events like these than a systems administrator who would be paid a decent salary to manage such a minuet infrastructure, in the scheme of things. Considering actual outages like this will be so infrequent, as well as the fact that the client size is so limited, a full-time guy isn't needed. Insulting people for being creative and intuitive and thinking outside of the box with regards to finding solutions to problems makes me occasionally wonder what's wrong with the world. Remember, this wasn't their fault.
+Terrapin GD the fact that he can do that and get the system to work is amazing
what would a "real" server administrator do differently?
+TROLLFACE KNIGHT Exactly. That's the funny part. He'd essentially diagnose the same problem which wouldn't have been his fault in the first place (it wasn't LTT's fault) and would've undergone a similar data-recovery procedure.
+D-MMA Quite right. Everyone learns from things like this, so if anything i'm grateful to LTT for uploading this. So many people, even the most elite of sys admins experience issues like these. Seeing people actually experience and have to deal with it is so valuable, and we can all gain so much knowledge and understanding from it.
This film is more dramatic and nervekeeping than 90% of hollywood movies - even 3 years later xd
Ikr when the harddrives wernt showing up it was super tense
WORD. I'm still sweating.
👹
Exactly 😂
The music they use for each scene is very well placed and mood setting.
i realy expected linus half way into the video to say: stay tuned for part two of all our data is gone in the future!
+geen naam SAME HERE
+geen naam ooh je hebt wel een naam! en ik zal er achter komen ook!
kwallen hoeven geen naam te hebben
zelfs geen donkerpaarse kwallen
geen naam maar jij bent een gewone kwal in ik vid dat ze we namen nodig hebben!
Fabian Miserus
nouja mijn naam komt van een aflevering van spongebob van vroeger die ik mij nog half herriner. spongebob vangt een speciale paarse kwal en wil hem een naam geven, kan niet op een naam komen dus noemt hem geen naam
en vond het bullshit dat je van google je naam moest invullen
Got referred to this video at the end of your most recent upload concerning server errors/data loss, and man, remember when mineral oil submersion was going to be THE NEXT BIG INNOVATION? (4:50)
It was surprisingly nostalgic to see a fish-tank build while Linus was perusing customer service from that data recovery company.
"Wait we can recover some data!"
"really?!?! what did you recover?!?!"
"A RAID SHADOW LEGENDS SPONSOR? THATS ALL WE GOT BACK???????"
Raid:5 shadow legends
Nice transition
Can you not with that bullshit
this comment aged well
@@ZackHab LMAOOOOO
I know that "NONE OF IT..." look. I felt Linus so hard there; I felt the pain of this PHYSICALLY!
3:01 what goes through the minds of the folks in charge of packaging will always confuse me.
"Hey Jim this area over here seems a little bland, mind adding something there?"
"Way ahead of ya Tim, I already added this completely unrelated face of an unattractive woman there."
+MeLikeBigBoom I sorta think they didn't even try to hire a graphic/design artist. But hey, what's more abstract and eye-popping than an oily western woman on the cover?
+MeLikeBigBoom hey im one of your biggest fans
+Kaeden Wilson a 250mm [super?] bigboy from Antec? now thats a big fan !
Based on your comment i take it your a funny guy! Ill give your channel a shot! Thanks for giving me a laugh
+MeLikeBigBoom fun comment, but we need more videos about the upcoming unturned 4
"And we'll have a better backup scheme in the future, I promise"
Petabyte Server in 2022: "Yeah we need to talk Linus."
I work in IT and these types of videos remind me to be grateful that we migrated to the cloud for storage (and basically everything else except one domain controller, which we’ll be getting rid of soon in favor of full Azure AD).
Nothing about this was funny, devastation. Those recovery guys know their stuff!
Yeah looks almost like an advertisement for those guys...
@@Thomas-Almanza There is really not a practical reason to crash a huge working server with people currently working with it just for an advert, that and, just having the garbled data when it is "recovered"
What is the problem...
...just restore your backup...
...you DO have a backup, do you...?
Exactly. This whole thing was completely preventable because any competent person doing data management knows that everything should be backed up in several locations, otherwise the system should never be used.
@@MorzenMebs i get that quite often where I work. Folks are not even bothered about their backups (or lack of them) yet they are the first ones to moan when shit hits the fan and their whole business is one step away from going completly tits up. One major rule - serios about business? Then better be serious about your backup.
But... it was a RAID configuration... more than one drive won't fail simultaneously. You don't NEED a backup right? Right? Right...
He does it was the raid the motherboard failed
You missed the part where he was mid backup when this happened. The backup was being made before it broke... and while it broke. Yes, he could have done it sooner, but he did start before he had a problem.
Have you tried turning it off and on again?
Gotta do it 3 times.
How many times so you need to reboot? If he had Windows 9, he wouldn't have issues
Unless he installed the 9.1 update. Then good luck.
aguyandhiscomputer windows 9.1?
AndroidEight YT
You question windows 9.1 but not windows 9?
what a nice time re-watch this, can't wait for the new episode!
This was some of the most gripping video I've seen from a techtuber.
Makes me want to get into setting up networks and servers.
Don't. 20 years ago it was a fantastic career choice. These days it'll send you to an early grave. For a start, your starting pay will be, equivalent for the time, 20% of the old days. The rest of the time will be holding things together with tape and string, because budgets have been turned down for the 3rd year straight.
Lastly, a little saying I heard a few years ago: "Accounts departments love IT departments. Before there were IT departments, everyone hated Accounts. Now there is IT, everyone hates the IT department instead of Accounts."
I would say pure network administration or network engineering isnt a bad career choice.
setting up things can be fun, maintaining them is a depressing adventure.
there are people w/ ptsd after a few years working with situations like the one presented in this video.. no exaggeration.