We all know Linus' real endgame is having enough storage to finally upload his consciousness into a machine, then he can run LTT in perpetuity as a sentient AI that constantly drops database tables
It is always so great to see big name content creators/tech wizards like Jake Tech Tips and Anthony Media Group collaborate with young and upcoming screw driver enthusiasts like Linus.
This went smoothly compared to some jobs I've done. One rack was in a room so small that the 35kg servers had to be angled at more than 45 degrees just for the font of them to clear the wall and there was only space for one person to do so...from the side of the rack.
As a server engineer, every time I watch Linus and Jake work on servers I cry a little inside. Please, for the love of your HR department and your own health, please purchase a server lifter. You spend big money on lots of things, this will save your backs and potentially Linus' body if he keeps choosing to lie under servers... I also love the change management Normal Company: I need to raise a change with 5 business days lead time with install and back out plans, passed by Change Advisory Board and approved by change manager\service control manager. Notice sent out to all impacted users. LTT: YO GUYS IM CHANGING STUFF, GET OFF THE DISKS!!!
To be fair they did give advance notice, they just went the extra step of walking around to say its happening now instead of the usual "your fault for not listening, your fault its now gone"
If Linus keeps going around like this, there might be the event of him dropping a server on any part of himself. In the event of the part of him that is crushed being his head, the outcome maybe really bad and the endgame part a bit too real. Technically the way they are doing things, the event of any of Jake's extremities being in peril is not to be excluded from the equation. That would be as bad, and a bit of a liability for the company, too.
I shudder to think of what would happen to me if I was doing this on PRODUCTION servers just on a whim like how its depicted here. Now I get it, a smaller operation like this isn't going to have Lab/Dev/QA/Prod environments, but to go basically from the lab to production and just "hope and pray" it works, I cry along side you. I sent one of these videos to our OH&S guy a few years back, I honestly think he had a stroke because he just sat there staring at the screen mouth agape and twitching the whole time. All humour and joking aside, they REALLY need to have better safety practices, every one of these videos is just a worker's comp agent's worst nightmare.
That's what happens when you constantly choose a technology which is fundamentally not designed to be expanded (even if it with a lot of work can be but not designed for it). And discounting technology because "no one here knows how to set up and maintain it", and instead choose a different technology that ALSO clearly no one there knows how to set up and maintain and are learning bit by bit on the fly...
We only live physically once so we shouldn’t be afraid to do anything bro, i smoke weed on my TH-cam channel and i ate Mcdonald’s inside Walmart 😈😈, screw anyones opinion 🎯
you know, when I worked in Inventec, we had lifting thingies for carrying servers, so you could just adjust height and slide even full 150gk 4u server in and out of test rack by yourself without help, no matter if it was almost on ground level or 2m high (it wasn't in regular racks, it was more like store shelves with rails, so putting in and pulling out servers was easy) I bet logistics guys would love this thing
Might I recommend the acquisition of hydraulic table carts. Makes transferring heavy things like that from different heights much easier and safer for both the equipment and your staff.
Hope Wendell can do a video on how he restored some of Linus's bit-rotted files. I've never made this mistake but it would be interesting to see the repair.
@@LiLBitsDK bit-rot is when the bits that make up your files are changed / corrupted over time. If data is just stored on a hard drive, it will degrade and you can lose the data. ZFS has something called "scrubbing" which checks this and fixes issues before they become unrecoverable. I believe LTT did not have scrubbing setup or it was setup incorrectly so they were not protected against bit-rot
You might be interested in this video th-cam.com/video/tU0Wc6Kj6IQ/w-d-xo.html Wendell goes full computer janitor on recovering data from an old Enterprise SSD.
17:45 Linus be like: "Yeah let me just lay here under a sever with 60 drives in it that isn't mounted correctly and play with the mounting hardware. Not that it could just fall down and will splash my head."
honestly i love the server content more than the typical videos. because its legit just linus and jake spending a day nerding out and having a laugh while they do it. and this server stuff is far more interesting than another overpriced gaming pc
@@danieljonsson8095 as an enterprise IT Server Engineer, I cringe at everything Linus chooses for his server and network core solutions. It is obvious that Linus, nor anybody who works for him has ever worked in corporate/enterprise IT.
13:30 always love when Linus & Jake "playing" around with the server, comes the time to shut the server down and had to hesitantly tell the editors. But this episode comes with a twist lol
Having followed the new server videos for a while, I’m glad to see things get finalized. Although part of me wonders if Linus was partially inspired by the SSD-based server used for the new house.
Jake's job is so cool. He gets to design and build these amazing projects for Linus and spare no expense you normally would. With the house and everything he does at the office hes become my favorite face on LTT.
You really need a larger server room - something on the order of 5x larger. That one rack should be 3. You really need 5 feet in front and back of clear space for access, and then at least a half a rack for network appliances and patch panels. This would make working on the physical plant a whole lot easier.
I think the perfect backup for your archives would be an automated tape storage room Edit : yeah tape writing and accessing is slow but it is just a backup for a very dark day
Love the show and all you do! I gotta speak up here though. As a content creator and IT professional, managing this large of a redundant data set is going to grow in complexity as your data grows. Eventually, your internal staff will no longer be able to manage it on their own and you will need to hire a full time Tech just to monitor and maintain it. A tape archive system isn't as sexy and you're not going to have instant access to the raw content of your oldest videos [How often do you really need that?] but it's a hell of a lot cheaper and functional for decades. As you've seen in the past and as you will continue to see, having that instant access on a ginormous NAS costs a lot more than the hardware, eventually, it's going to cost you the salary of a dedicated Tech that does nothing else except manage that storage. Then again, you make bank at LTT so this might be something you're prepared to do but I'm already seeing the video title from LTT, 'Guys, I was wrong about the NAS, again, we lost everything!!!!...Maybe' video title in 2023 or 2024. 3.6 Petabytes isn't going to be nearly enough when you consider the steep ramp of camera tech and thus storage requirements.
You need a hydraulic lifting table (jack). It's normally used to lift stuff onto the bed of a pickup truck, but seems like a perfect fit here too. They even have wheels so you can cart heavy stuff around.
They seriously better get around to getting some of that before anyone of them seriously injures themselves. Being fun and goofy is cool until people get hurt.
Reminds me of every experience I've had doing projects in IT, especially new SAN / storage migrations. There's a million little things that are slightly off, with many tasks that have to be finished to deal with the last lingering issues before everything is "right". About what you said on the zones, most people don't use these kinds of setups, and there are always issues that come up. Only vendor appliances are truly headache-free (Nimble / Pure arrays). They are the reason most enterprises don't use these kinds of setups, appliances are vastly more easy to standup and their support is worth its weight in gold--they are the definitive experts so we don't have to be.
@@Braindead154 I'm not so sure about that. I'm a bit of a data hoarder myself and every time people suggest me to use cloud to store my data, I redo the math and it always ends up being cheaper to just store it locally, even at my scale (under 100TB). For example, using an unofficial glacier calculator, with 3.6PB of data, just to store (not retrieve) it for a year, it's giving me a quote of 175K USD.. And if you are actively retrieving data, you soon end up adding tens of thousands of dollars - as an example, the retrieval cost for 20TB costs 25K USD and requires a minimum of 4 hours. Now, compare to their new server - it's probably close to 200K in hardware (too lazy to do the parts list math myself), but they get so much free stuff from sponsors that I'd be surprised if it ended up costing more than 50K for the build (they got most of the drives for free, got the RAM for free, etc). And, maintained properly, the yearly cost of operation will be much lower than any cloud storage (it's just power plus the occasional hard drive replacement) and you get unlimited retrieval for no additional cost (and it's much faster than 4 hours for 20TB).
They aren’t storing 3.6PB of data, they are storing a fraction of that and using raid for redundancy. You don’t have to do any of that with S3, durability is just there and to a much higher level. Even if they are getting paid to build with the equipment they use, I still hope that they have some offsite data storage.
@@ddognine ... with dedicated support teams who watch the health of the arrays over time, regular hardware maintenance and operations documentation in a trio of purpose-built datacenters, with proper fire suppression and spread across a large enough geographic area that any one minor disaster doesn't lose your data, instead of someones spare time between script reviews and a closet.
Take some advice from a man that has had two back operations in that last year DO NOT LIFT LIKE THAT get help, wear a brace of use a telescoping cart or you will be sorry like I am. When I was your age and younger I did the same stuff and I am paying for it now BIG TIME?
LMG guys: I am curious... this will allow you to have access to all those projects.. how do you know which project file to grab what desired clip from? What is a typical list of keyords/tags you might use for a given project and how does your search for relevant video get achieved? Is there any kind of AI invloved in creating metadata for future searches or is it all manual? That might make for an interesting video.. at least I'd watch it. Not a lot of hardware to it.. but the whole workflow thing is interesting and worthy of a video.
@@joshjlmgproductions3313 sure but often a simple video title doesn't tell you all the things in a video, especially when you have a specific product or clip in mind that wasn't the highlight of the video...
Linus has got to be one of the only people to make a video building a new data server sound like it's going to be an epic tale that will one day be sung about. Also, he makes it all seem so easy but you can tell when they're talking about adding insulation and temps in the room, his hair at his forehead is soaking wet, meaning he's been sweating his ass off.
@@5dashes They intentionally (totally intentional, wasn't a mistake at all, we swear!) delete/corrupt all your data every 6 months to keep you on your toes and remind you that backups are important!
Linus: this Server, that needs four people to lift is not mounted correctly and only held in place by a piece not designed for heavy loads Also Linus: says that with his head directly beneath it
BTW bought the "Night" edition of the LTT water bottle, INCLUDING the sport lid. BEST F'ING PURCHASE I'VE ever made. I'm a Chef at an expensive restaurant. I have used and been through many of insolated water bottles. But, Legit they have the best one. I drink about 3 Liters of water just through my shift. Nothing has been so quick to open, SLAP DOWN, Wash hand and keep going. THANK YOU for making it easier for a Chef to cook and hyrdrate. I will Spread the word even to my non-tech friends.
Damn dude, as someone who's lost poetry and artwork that only existed in digital form, I feel your pain on whatever wasn't recovered. People say, you can create it again... And it's like no.. you can't.. Glad y'all got the majority recovered. You guys are living my dream, lol not counting TH-cam stuff. Stay awesome
One benefit in their case is that the majority of the content is actually still on YT and can be downloaded, just in significantly reduced quality and organization.
I love when Linus and Jake are doing server stuff :) Also I love how they have an order of magnitude more TB of SSD storage than the capacity of my HDD based NAS :D
Yeah not a fan of those. I get the thumbnails, YT makes the rules, and however annoying they are you get shanked if you don't play by them, but it gets annoying when it spans the first minute of the video.
I love LTT, but I'm genuinely considering unsubbing with the childish tiktoky style choices they're making these days :( I just really grates on my nerves for some reason
How does Linus not have a "proper" server room/rack by now? I've seen Cogen plants that have better server rooms. I get that he has absolutely insane SERVERS, but they'd probably last a lot longer with proper cooling, ventilation, spacing, etc.
@@shapshooter7769 he really doesn't though. That server room is way too small for the contents of that rack. If AC goes out or is underperforming, the smaller the room, the quick things burn out/up; hence the comment in the video about their UPS burning up in the past. Also, they should be insulated (both thermally and audibly) which they've kind of done once (based on past videos) but clearly not well since both sound and heat can leak out the vented door. I love Linus and especially all his server videos, I just figure by now he'd have a state of the art server room to match his BETTER than state of the art servers 🙂
It does look like where the janitor keeps his supplies after hours. LTT has enough square footage, I'd like to see something more 'bright tech clean-room' and less 'broom closet vibe' :) With how often they end up filming segments in there you'd think they'd want more space.
I agree, being able to walk in, have a few shelves for bits and bobs and being able to walk the whole way around the rack is night and day, probably not feasible due to the effort of terminating everything somewhere else though
Because of LTT I started to build my own NAS with Truenas Core and tried to Backup anything that I can't recover from the Cloud. It's not easy because you can buy only this much as a Teenager but soon I can store the Backups on another place, so that I much safer from getting a data loss than the average citizen.
@Ian Visser I only backup important document and photos but I don´t want to lose these, but the rest is in the cloud or not important enough so..... And I have not nearly this amount of raw data, must of the things on my PC are games that are already in the cloud.
"Leftover" 30 TB NVME Drives from Dream's build. Man the stuff that "falls off a truck" at LMG in any given week is prolly worth more than my house. . .
Hey great work on these, I love seeing you guys work on datacenter hardware/software solutions ! Next logical upgrade : geo replication on those storinator (now that you have multiple buildings)? I think trueNAS has some implementation of gluster Georeplication in the SCALE version, so that could be an interesting thing to try out !
@@Wooble57 Well, it depends. From what I remember, they're a few hundred meters apart (maybe 90-150m apart or smthg). It's definitely enough for fire to not spread across the two buildings, and it puts both storage cluster on different power circuit, which is probbly enough for a company the size of LMG.
By single point of failure, are you referring to the 'head' server? If that goes down, they will temporarily lose access to their data, but the data itself will still be safe, they'll just have to import the zpool on another server, same as they did on the new server in this video. So while there may be a single point of failure, I don't think it is inconsistent with the goal of avoiding losing data in the future.
@@Cyberguy42 A clean hardware failure is far from the worst case scenario. I've seen processors with incorrect AES s-boxes, silently mis-encrypting all data. I've seen memory bit flips lead to incorrect indirect jumps in the processor. Or power spikes (lightning) taking out multiple drives simultaneously. Or just simple software bugs. The list goes on.
@@jmickeyd53 I mean, you still need backups, yeah. I don't think they contested that? Doesn't Linus always try to throw at least one "RAID is not a backup!" into these videos?
@@jmickeyd53 For pure curiosity: How would your preferred setup look like? Because I don't see how two head servers will solve your mentioned issues from my basic infrastructure knowledge.
17:37 when that happens, get a hex wrench of the right size and L it up in there. If you don't have enough leverage, just get a hex driver with the right tube wrench and make yourself an extra lever. CAUTION, if the rack is 100% full, you risk bending the leg. I usually grab the offending corner and then push the rail in. ALSO, i can't believe you put the rack directly on the floor. WTFrak. No shock mounts?
@@marc0523 also depends on where you are locating i suppose, if its a building with any possibility of having substantial vibrations shock mounts would make sense.
Ok I'm gonna have an aneurysm from watching Linus laying under the server. Gah! Also, WHERE are your steel toes? Those sneakers, yea no. Maybe some basic safety precautions you guys? I mean at least they wore ear protection, sort of.
Haven't they been using 8k cameras for many projects for quite a while? Nothing like recording /storing stuff in 8K so that folks can watch it in 1080P (1/16th it's original res)
To reduce the weight of the JBOD, we usually remove the drives and power supplies before transporting/mounting to the server rack. Only install the drives and PSU back to it afterwards.
16:26 is the best part 😂. Two qualified IT specialists in that little Server room move out old servers in new ones in. Love it! Btw, very interesting video!
This was a great video, but I am very hopeful that when LTT labs becomes a functioning channel, you can literally do videos about all the nitty-gritty tech stuff and like how to use the operating system and how to set up everything in detail so we can learn how to do it.
Based on my understanding, the point of labs is to generate new information, to answer questions that haven't been reliably answered before. There are already plenty of tutorials on setting up and using a basic Truenas system, so IMO there's no need for videos about that. However, if there are questions without clear answers (for example, which nitty gritty settings are best for certain use cases), then videos from labs that answer those questions would be helpful.
Love to see a chart showing hardware redundancy etc. in case an entire array goes offline. When I was working, servers racks you couldn't easily stand on both sides to load stuff were hell. Glad you bolted it down too, had a server rack not bolted down at a clients site once and yeah. Finally, time to convert one of the bigger rooms into a full service server room with hvac, raised floor, etc XD
Just realized: JBOD has been in use for iPods too! You know, the old iPods with the clickwheels and such- they use microSDs as cheap hard drive replacements! JBOD is the connection they use, since it works well and doesn't need a fancy controller chip.
Just a thought, maybe try using MLV (mass loaded vinyl) to try and cut down on the noise escaping from the server room. Companies like Audimute sell it in rolls. Using something like mineral wool or acoustic foam isn't going to do much for stopping sound transmission.
Another great video as always - Thanks! One wishlist item is for more bleeping on some of the stronger language. No issue with the base words during the course of a vid but seems like there is more stronger language coming through over the past months. Again not a prude but some words such as gd do bother me. Thanks again and best for all the plans in the works.
I don't know if I'm right, but it feels like your editors all have slightly different styles that show up in the videos they edit. I really enjoy the variation.
I love how unassuming high end hardware gets. Those SAS cards look so much like some trash tier GPU. Green PCBs are like the beige boxes of circuit boards now
I think we forget sometimes that it's really only "enthusiasts" who care about the looks of anything inside their case (and not even all of them, either). Regular users never open their computers, and even if they do they don't expect anything "pretty". And businesses don't care either, do they? Both have no emotional connection to their hardware, and they don't "present" it to others or see it as decorative part of the room, either.
I work in IT with systems and other stuff that will crush any "enthusiast" nonsense. It's a completely different game where cost is no object and nobody gives two shits how things look. You're talking budgets where millions doesn't describe it, that's a budget error nobody cares about. A single rack costs millions. And you're talking multi acre data centers. The raw scale of it all would blow most peoples minds. Unless you work for a for real institution like Microsoft, Amazon, the military, facebook, you have no godly idea how stupid things get. It's all laughable. And while I don't directly do data center stuff now talking to people on headsets wandering around multi acre facilities while they have to use hear protection and wear winter gear is still a thing. And there are multiple facilities on that site. "We have the linux complex, the ibm AIX complex, the windows complex" and complex is multiple warehouse sized monstrosities. Walking into one is like walking into a jet engine that pumps cold air. Tons of the stuff is classified and even though you are cleared there are armed guards and you check your stuff at the enterance. Many ban smart devices completely as they can be used to trace movements, so leave in car or lose job.
We all know Linus' real endgame is having enough storage to finally upload his consciousness into a machine, then he can run LTT in perpetuity as a sentient AI that constantly drops segways to our sponsor.
50-100 years in the future the humans then will view these as being archaic and oversized. Data centers will probably be no bigger than the average desk computer today and a pc capable of high end gaming will be no bigger than a small tablet. At least that’s what I hope. But considering how little we’ve progressed in the grand scheme of things part of me has doubts.
We all know Linus' real endgame is having enough storage to finally upload his consciousness into a machine, then he can run LTT in perpetuity as a sentient AI that constantly drops database tables
inb4 his DROP TABLE permission is revoked :P
Next LTT video idea?
Portal style?
WHAT THE FUCK LOL THIS IS FUNNY
Ah I see what you did there😅
With the rate that Linus is going, he'll make a seperate channel talking about all his servers
Great, that's what I care about the most
I would subscribe to that instantly.
I would subscribe to that past century.
i’d binge watch it
“Linus Linux Tips”
It is always so great to see big name content creators/tech wizards like Jake Tech Tips and Anthony Media Group collaborate with young and upcoming screw driver enthusiasts like Linus.
Definitely one of the, if not THE, most underrated comment in this thread!
Well, you could say Linus has a talent to... Screw things up....
I love how professional these videos are… right up until they start installing things
Belive me, it's sometimes more near to reality how they do things then one would think. :D
Can confirm. It’s all good until someone opens the debug console, or grabs a screwdriver. Then it’s all asses and elbows
This went smoothly compared to some jobs I've done. One rack was in a room so small that the 35kg servers had to be angled at more than 45 degrees just for the font of them to clear the wall and there was only space for one person to do so...from the side of the rack.
As a server engineer, every time I watch Linus and Jake work on servers I cry a little inside.
Please, for the love of your HR department and your own health, please purchase a server lifter. You spend big money on lots of things, this will save your backs and potentially Linus' body if he keeps choosing to lie under servers...
I also love the change management
Normal Company: I need to raise a change with 5 business days lead time with install and back out plans, passed by Change Advisory Board and approved by change manager\service control manager. Notice sent out to all impacted users.
LTT: YO GUYS IM CHANGING STUFF, GET OFF THE DISKS!!!
Absolutely true and absolutely NECESSARY to start good business practices as his business is growing.
To be fair they did give advance notice, they just went the extra step of walking around to say its happening now instead of the usual "your fault for not listening, your fault its now gone"
If Linus keeps going around like this, there might be the event of him dropping a server on any part of himself. In the event of the part of him that is crushed being his head, the outcome maybe really bad and the endgame part a bit too real. Technically the way they are doing things, the event of any of Jake's extremities being in peril is not to be excluded from the equation. That would be as bad, and a bit of a liability for the company, too.
The same thought, i got, when i watched this video... why dont they buy some kind of forklift for handling the Servers in their rack 😓
I shudder to think of what would happen to me if I was doing this on PRODUCTION servers just on a whim like how its depicted here. Now I get it, a smaller operation like this isn't going to have Lab/Dev/QA/Prod environments, but to go basically from the lab to production and just "hope and pray" it works, I cry along side you.
I sent one of these videos to our OH&S guy a few years back, I honestly think he had a stroke because he just sat there staring at the screen mouth agape and twitching the whole time.
All humour and joking aside, they REALLY need to have better safety practices, every one of these videos is just a worker's comp agent's worst nightmare.
ZFS for the win! Great pick guys! This time we know your data is safe (unless someone drops a server)!
unless? - until.
@@Nalianna fr
Can't wait for the next "endgame" server next year.
We all know how this story goes 😂
That's what happens when you constantly choose a technology which is fundamentally not designed to be expanded (even if it with a lot of work can be but not designed for it). And discounting technology because "no one here knows how to set up and maintain it", and instead choose a different technology that ALSO clearly no one there knows how to set up and maintain and are learning bit by bit on the fly...
We only live physically once so we shouldn’t be afraid to do anything bro, i smoke weed on my TH-cam channel and i ate Mcdonald’s inside Walmart 😈😈, screw anyones opinion 🎯
I give them 9 months
He just needs a separate server for server videos
you know, when I worked in Inventec, we had lifting thingies for carrying servers, so you could just adjust height and slide even full 150gk 4u server in and out of test rack by yourself without help, no matter if it was almost on ground level or 2m high (it wasn't in regular racks, it was more like store shelves with rails, so putting in and pulling out servers was easy)
I bet logistics guys would love this thing
Yeah, I don't know why they don't have something like that - that would make it a ton easier and safer.
@@invisi1407 . Probably don’t mover servers that much.
They just need a lifting cart.
He likes Jake's help.
Yeah, any kind of rolling scissor jack.
I've used them for automotive stuff, like lifting transmissions into vehicles.
Might I recommend the acquisition of hydraulic table carts. Makes transferring heavy things like that from different heights much easier and safer for both the equipment and your staff.
Hope Wendell can do a video on how he restored some of Linus's bit-rotted files. I've never made this mistake but it would be interesting to see the repair.
bit rotted?
@@LiLBitsDK bit-rot is when the bits that make up your files are changed / corrupted over time.
If data is just stored on a hard drive, it will degrade and you can lose the data.
ZFS has something called "scrubbing" which checks this and fixes issues before they become unrecoverable.
I believe LTT did not have scrubbing setup or it was setup incorrectly so they were not protected against bit-rot
@@Steven-d4c it was not a bit rot issue...it was a totaly missconfigured setup...
You might be interested in this video th-cam.com/video/tU0Wc6Kj6IQ/w-d-xo.html Wendell goes full computer janitor on recovering data from an old Enterprise SSD.
he did mention it a few times . the server build series might be helpful as well. Wendell covers a lot of topics including software and backups.
17:45 Linus be like:
"Yeah let me just lay here under a sever with 60 drives in it that isn't mounted correctly and play with the mounting hardware. Not that it could just fall down and will splash my head."
Poor linus, He has to lay in a sewer just to make ends meet my condolonces 🙏
@@TheFalseShepphard Ikr poor linus and his very overpriced Sewer system
he would be just fine, yes it is heavy but he would still be fine
@@LiLBitsDK 60 drives would make a jbod weigh 210 lbs or so idk
@@LiLBitsDK actually in the video it says 290 lbs and that may be when empty
honestly i love the server content more than the typical videos. because its legit just linus and jake spending a day nerding out and having a laugh while they do it. and this server stuff is far more interesting than another overpriced gaming pc
As someone that manages enterprise IT equipment at an MSP, I love these videos about your backend infrastructure.
Don't use LTT as a source for good enterprise IT management... Just don't... You're going to be in for a rough time if you do.
16:45 For the others who enjoy Linus' backend
@@danieljonsson8095 It wouldn't be half as entertaining otherwise lol
@@HilbertXVI Certainly. I'm just saying, be well aware that LTT is an ENTERTAINMENT channel, not "good practices" channel.
@@danieljonsson8095 as an enterprise IT Server Engineer, I cringe at everything Linus chooses for his server and network core solutions.
It is obvious that Linus, nor anybody who works for him has ever worked in corporate/enterprise IT.
Love you too
@Jonathan Z they will. Wendell tease it on twitter.
13:30 always love when Linus & Jake "playing" around with the server, comes the time to shut the server down and had to hesitantly tell the editors.
But this episode comes with a twist lol
Having followed the new server videos for a while, I’m glad to see things get finalized. Although part of me wonders if Linus was partially inspired by the SSD-based server used for the new house.
Or the million dollar all SSD server that they made a video of and then completely forgot about.
@@jackhemsworth7515 its not optimal to run this many ssds, a million for a server is not cheap
@@jackhemsworth7515 they had to return the parts to the sponsor, it wasn't meant to be a stable thing
I wonder if this is just the way Linus deals with his own mortality, and the need for permanence in time
Even so, we all will eventually be forgotten, no matter what we do.
It’s similar to how i do it.
@bruh I can only guess that's the final point, to make a copy someday and for it to be tax deductible hehehehehe
Jake's job is so cool. He gets to design and build these amazing projects for Linus and spare no expense you normally would. With the house and everything he does at the office hes become my favorite face on LTT.
You really need a larger server room - something on the order of 5x larger. That one rack should be 3. You really need 5 feet in front and back of clear space for access, and then at least a half a rack for network appliances and patch panels. This would make working on the physical plant a whole lot easier.
I don't think they actually have enough need for that. It'd make sense if they were doing more than basically meming with the server room.
I love Jake. LTT just has such an amazing crew.
Ditto. He is clearly so knowledgable, continues to amaze me what he knows and the broad spectrum of things he knows and is capable of!
Yeah. He's like a super chatty LAN admin. You don't see those very often. It's fun.
Can’t forget the fire in the old old server rooms and how many you guys had it was crazy but did a very nice recovery on that one
I think the perfect backup for your archives would be an automated tape storage room
Edit : yeah tape writing and accessing is slow but it is just a backup for a very dark day
VHS or Beta?
@@donc-m4900 fish tape
As a backup yes, for reading 100% NO
900,000 feet of tape!
he would probably need LTO10. Even then... could you imagine trying to backup all that data he has?
I love how linus just casually lies directly under extremely heavy servers which are poorly held in place.
Love the show and all you do! I gotta speak up here though. As a content creator and IT professional, managing this large of a redundant data set is going to grow in complexity as your data grows. Eventually, your internal staff will no longer be able to manage it on their own and you will need to hire a full time Tech just to monitor and maintain it. A tape archive system isn't as sexy and you're not going to have instant access to the raw content of your oldest videos [How often do you really need that?] but it's a hell of a lot cheaper and functional for decades. As you've seen in the past and as you will continue to see, having that instant access on a ginormous NAS costs a lot more than the hardware, eventually, it's going to cost you the salary of a dedicated Tech that does nothing else except manage that storage. Then again, you make bank at LTT so this might be something you're prepared to do but I'm already seeing the video title from LTT, 'Guys, I was wrong about the NAS, again, we lost everything!!!!...Maybe' video title in 2023 or 2024. 3.6 Petabytes isn't going to be nearly enough when you consider the steep ramp of camera tech and thus storage requirements.
You need a hydraulic lifting table (jack). It's normally used to lift stuff onto the bed of a pickup truck, but seems like a perfect fit here too. They even have wheels so you can cart heavy stuff around.
They make ones specifically for loading racks.
Yep, can get them in a veritable sizes
Or use Jake and another guy instead...
See my comment earlier: battery powered server lift. I installed an entire datacenter by myself with it!
They seriously better get around to getting some of that before anyone of them seriously injures themselves. Being fun and goofy is cool until people get hurt.
Reminds me of every experience I've had doing projects in IT, especially new SAN / storage migrations. There's a million little things that are slightly off, with many tasks that have to be finished to deal with the last lingering issues before everything is "right". About what you said on the zones, most people don't use these kinds of setups, and there are always issues that come up. Only vendor appliances are truly headache-free (Nimble / Pure arrays). They are the reason most enterprises don't use these kinds of setups, appliances are vastly more easy to standup and their support is worth its weight in gold--they are the definitive experts so we don't have to be.
For their use case, they should just toss it in a Amazon S3 Glacier bucket and call it a day… would save him so much money and time.
@@Braindead154 I'm not so sure about that. I'm a bit of a data hoarder myself and every time people suggest me to use cloud to store my data, I redo the math and it always ends up being cheaper to just store it locally, even at my scale (under 100TB).
For example, using an unofficial glacier calculator, with 3.6PB of data, just to store (not retrieve) it for a year, it's giving me a quote of 175K USD.. And if you are actively retrieving data, you soon end up adding tens of thousands of dollars - as an example, the retrieval cost for 20TB costs 25K USD and requires a minimum of 4 hours.
Now, compare to their new server - it's probably close to 200K in hardware (too lazy to do the parts list math myself), but they get so much free stuff from sponsors that I'd be surprised if it ended up costing more than 50K for the build (they got most of the drives for free, got the RAM for free, etc). And, maintained properly, the yearly cost of operation will be much lower than any cloud storage (it's just power plus the occasional hard drive replacement) and you get unlimited retrieval for no additional cost (and it's much faster than 4 hours for 20TB).
@@Braindead154 There is no such thing as the cloud, just someone else's computer.
They aren’t storing 3.6PB of data, they are storing a fraction of that and using raid for redundancy. You don’t have to do any of that with S3, durability is just there and to a much higher level. Even if they are getting paid to build with the equipment they use, I still hope that they have some offsite data storage.
@@ddognine ... with dedicated support teams who watch the health of the arrays over time, regular hardware maintenance and operations documentation in a trio of purpose-built datacenters, with proper fire suppression and spread across a large enough geographic area that any one minor disaster doesn't lose your data, instead of someones spare time between script reviews and a closet.
Take some advice from a man that has had two back operations in that last year DO NOT LIFT LIKE THAT get help, wear a brace of use a telescoping cart or you will be sorry like I am. When I was your age and younger I did the same stuff and I am paying for it now BIG TIME?
LMG guys: I am curious... this will allow you to have access to all those projects.. how do you know which project file to grab what desired clip from? What is a typical list of keyords/tags you might use for a given project and how does your search for relevant video get achieved? Is there any kind of AI invloved in creating metadata for future searches or is it all manual? That might make for an interesting video.. at least I'd watch it. Not a lot of hardware to it.. but the whole workflow thing is interesting and worthy of a video.
They probably date their videos and name them the same name as the TH-cam upload.
@@joshjlmgproductions3313 or the internal name for the project, rather than the clickable title on TH-cam
they could also search for a video on youtube and then have more info about raw material, if there is some naming convection consistency
@@joshjlmgproductions3313 sure but often a simple video title doesn't tell you all the things in a video, especially when you have a specific product or clip in mind that wasn't the highlight of the video...
pretty sure this is only archival, not active projects, so they probably don't go back to it very often
I love that Jake knows he's in the untouchable zone, ordering food not only infront of the bossman but on video
Jake is Linus' adopted son, he is untouchable
And insists the editor leaves it in! :D
ive noticed an inversion of the early Jake/Linus dynamic, nowdays it feels like Jake is the adult and Linus is the child.
This is why Linus hires employees - so he is free to act like a child in camera.
Which is of course, content.
I absolutely love their dynamic, and I’m constantly impressed at Jake’s knowledge and skills.
Linus has got to be one of the only people to make a video building a new data server sound like it's going to be an epic tale that will one day be sung about. Also, he makes it all seem so easy but you can tell when they're talking about adding insulation and temps in the room, his hair at his forehead is soaking wet, meaning he's been sweating his ass off.
If there is ever a shortage of hard drives, we know Linus has bought them all.
I can't wait for Linus to create a new channel just to show off his backup servers. He could call it Linus Backup Tips?
More like "Linus Backup and Restore Tips"
As long as it's not "Linus backs his tip up", 'cause that's not exactly TH-cam material.
Linus Back Tips
Linus' Back That Thing Up, Just the Tip, and Only for a Second!
Linus server tips
I would finally be able to install my entire steam library with a setup like this.
3 games max
3 years later...
Linus: We bought another unit, and we're turning it into the LMG Data Center!
AWS competitor -- Linus Web Services
@@5dashes They intentionally (totally intentional, wasn't a mistake at all, we swear!) delete/corrupt all your data every 6 months to keep you on your toes and remind you that backups are important!
Clever way of showing “logistics” which has been mentioned many times but hasn’t been seen (by me) until now.
I loved that Segway during the segue.
18:14 I love how enthusiastic Linus promotes their screwdriver like "EASY!" while Jakes knee is almost getting crushed by the weight of the JBOD 😂
Then the sponsor is for a competitors screwdriver 🤣
8:30 - 9:40 I had a litle bit of ... the plant ... and this storage adventure scene really tripped me out. 10/10
25:27 MaN-TaH. MoN-TuH.
I love seeing the behind the scenes of LMG, it was hilarious seeing the logistics banter
Linus: this Server, that needs four people to lift is not mounted correctly and only held in place by a piece not designed for heavy loads
Also Linus: says that with his head directly beneath it
BTW bought the "Night" edition of the LTT water bottle, INCLUDING the sport lid. BEST F'ING PURCHASE I'VE ever made.
I'm a Chef at an expensive restaurant. I have used and been through many of insolated water bottles.
But, Legit they have the best one. I drink about 3 Liters of water just through my shift. Nothing has been so quick to open, SLAP DOWN, Wash hand and keep going.
THANK YOU for making it easier for a Chef to cook and hyrdrate. I will Spread the word even to my non-tech friends.
Damn dude, as someone who's lost poetry and artwork that only existed in digital form, I feel your pain on whatever wasn't recovered. People say, you can create it again... And it's like no.. you can't..
Glad y'all got the majority recovered. You guys are living my dream, lol not counting TH-cam stuff. Stay awesome
One benefit in their case is that the majority of the content is actually still on YT and can be downloaded, just in significantly reduced quality and organization.
Server and Moving Vlogs are still the best. Watching since 'whole room water cooling' 😁
I can not deal with the words hurled at my eyes in the first minute of this video. Please chill with the HYPER HYPE EDITING, thanks.
I love when Linus and Jake are doing server stuff :) Also I love how they have an order of magnitude more TB of SSD storage than the capacity of my HDD based NAS :D
LTT video editor: "How many flashy text fields do you want? How big shall they be?"
LTT marketing guy: "Yes."
Yeah it's gotten worse over the past few months, more clickbaity and seems to treat their audience as children or preteens
Yeah not a fan of those. I get the thumbnails, YT makes the rules, and however annoying they are you get shanked if you don't play by them, but it gets annoying when it spans the first minute of the video.
I love LTT, but I'm genuinely considering unsubbing with the childish tiktoky style choices they're making these days :(
I just really grates on my nerves for some reason
@@Shanghaimartin same
Go pay for Floatplane then.
18:50 "Breaking: CEO crushed under too much server"
I love the Server projects especially with AMD. They are epyc.
How does Linus not have a "proper" server room/rack by now? I've seen Cogen plants that have better server rooms. I get that he has absolutely insane SERVERS, but they'd probably last a lot longer with proper cooling, ventilation, spacing, etc.
But he does though.
it IS a proper server room, albeit small AND it is a professional rack... so what are you talking about?
@@shapshooter7769 he really doesn't though. That server room is way too small for the contents of that rack. If AC goes out or is underperforming, the smaller the room, the quick things burn out/up; hence the comment in the video about their UPS burning up in the past. Also, they should be insulated (both thermally and audibly) which they've kind of done once (based on past videos) but clearly not well since both sound and heat can leak out the vented door.
I love Linus and especially all his server videos, I just figure by now he'd have a state of the art server room to match his BETTER than state of the art servers 🙂
It does look like where the janitor keeps his supplies after hours. LTT has enough square footage, I'd like to see something more 'bright tech clean-room' and less 'broom closet vibe' :)
With how often they end up filming segments in there you'd think they'd want more space.
I agree, being able to walk in, have a few shelves for bits and bobs and being able to walk the whole way around the rack is night and day, probably not feasible due to the effort of terminating everything somewhere else though
Can I just say how much I LOVE videos with Jake and Linus, you can tell these 2 are bros and genuinely enjoy their time together
It’s your endgame… for now, you’re gonna have a new server by 2023 lol
If they do it right, this should be extendable by updating the "head" servers or adding more arrays.
Because of LTT I started to build my own NAS with Truenas Core and tried to Backup anything that I can't recover from the Cloud. It's not easy because you can buy only this much as a Teenager but soon I can store the Backups on another place, so that I much safer from getting a data loss than the average citizen.
@Ian Visser I only backup important document and photos but I don´t want to lose these, but the rest is in the cloud or not important enough so..... And I have not nearly this amount of raw data, must of the things on my PC are games that are already in the cloud.
It's nice to see the intro still being used. I've noticed some videos don't have it and I miss it.
Nice! Server content is my favorite part of the channel lately.
"Leftover" 30 TB NVME Drives from Dream's build. Man the stuff that "falls off a truck" at LMG in any given week is prolly worth more than my house. . .
most of it ends up at linuses house as well i reckon
"Endgame"
we all know a "bigger, better, and faster" one is coming in a few months for Lab 2.
“It’s LeviosAAAAH”
That reference did not go to waste. 11/10
STOP STOP you're doing it WRONG. It's levOOOOOsah not LeviosAHHH.
@@TheOrangeMelons I was actually thinking of Oney’s video… oops…
@@DarkSideKyp it's Levios-AUUHHHH
I actually heard that in Hermione's voice... :)
Oh stop it Linus!
Hey great work on these, I love seeing you guys work on datacenter hardware/software solutions ! Next logical upgrade : geo replication on those storinator (now that you have multiple buildings)? I think trueNAS has some implementation of gluster Georeplication in the SCALE version, so that could be an interesting thing to try out !
the two buildings he has are VERY close together. I don't think that's what your usually going for with geo replication.
@@Wooble57 Well, it depends. From what I remember, they're a few hundred meters apart (maybe 90-150m apart or smthg). It's definitely enough for fire to not spread across the two buildings, and it puts both storage cluster on different power circuit, which is probbly enough for a company the size of LMG.
That is the art of Linus (and Jake to be fair) topic was ok but they made it SO engaging and fun to watch!!
Imagine waking up in the morning and his new video ITS HERE
Me: wow
That transition a few seconds after 9:37 💯
I mean it's a pretty basic slide transition lol, nothing crazy tbh
Linus: We almost lost all of our data. Also Linus: We're going to build a new storage system with a single point of failure.
By single point of failure, are you referring to the 'head' server? If that goes down, they will temporarily lose access to their data, but the data itself will still be safe, they'll just have to import the zpool on another server, same as they did on the new server in this video. So while there may be a single point of failure, I don't think it is inconsistent with the goal of avoiding losing data in the future.
@@Cyberguy42 A clean hardware failure is far from the worst case scenario. I've seen processors with incorrect AES s-boxes, silently mis-encrypting all data. I've seen memory bit flips lead to incorrect indirect jumps in the processor. Or power spikes (lightning) taking out multiple drives simultaneously. Or just simple software bugs. The list goes on.
@@jmickeyd53 I mean, you still need backups, yeah. I don't think they contested that? Doesn't Linus always try to throw at least one "RAID is not a backup!" into these videos?
@@jmickeyd53 For pure curiosity: How would your preferred setup look like? Because I don't see how two head servers will solve your mentioned issues from my basic infrastructure knowledge.
@@ray89520 Ceph cluster would have been better immho.
270? Naw…. Go absolutely insane…1,200 hard drives at the maximum amount of storage per disk possible. THAT’S game over!
17:37 when that happens, get a hex wrench of the right size and L it up in there. If you don't have enough leverage, just get a hex driver with the right tube wrench and make yourself an extra lever. CAUTION, if the rack is 100% full, you risk bending the leg. I usually grab the offending corner and then push the rail in.
ALSO, i can't believe you put the rack directly on the floor. WTFrak. No shock mounts?
I've never seen shock mounts, is it possible you live in an earthquake zone and that's why you have them?
@@marc0523 also depends on where you are locating i suppose, if its a building with any possibility of having substantial vibrations shock mounts would make sense.
Only place I have seen racks with shock mounts is on ship installs.
It's not like Vancouver never gets earthquakes, either.
I've got that same JBOD enclosure at work full of SSDs, it's wild. That one shelf and the machine it's connected to are worth more than my life.
Ok I'm gonna have an aneurysm from watching Linus laying under the server. Gah!
Also, WHERE are your steel toes? Those sneakers, yea no. Maybe some basic safety precautions you guys? I mean at least they wore ear protection, sort of.
literally cannot the amount of space needed for these videos we realize now is absolutely insane
Haven't they been using 8k cameras for many projects for quite a while? Nothing like recording /storing stuff in 8K so that folks can watch it in 1080P (1/16th it's original res)
To reduce the weight of the JBOD, we usually remove the drives and power supplies before transporting/mounting to the server rack.
Only install the drives and PSU back to it afterwards.
5:49 Linus is getting "high" performances
Now you just need all those drives to be those Exobyte SSD drives for the largest SSD storage you could ever get
16:26 is the best part 😂. Two qualified IT specialists in that little Server room move out old servers in new ones in. Love it!
Btw, very interesting video!
qualified is a streach
Linus: "Hard drives are dead"
Also Linus: "I bought 270 hard drives"
I just hear Linus saying the Kylo Ren *MORE* meme 🤣
This was a great video, but I am very hopeful that when LTT labs becomes a functioning channel, you can literally do videos about all the nitty-gritty tech stuff and like how to use the operating system and how to set up everything in detail so we can learn how to do it.
Based on my understanding, the point of labs is to generate new information, to answer questions that haven't been reliably answered before. There are already plenty of tutorials on setting up and using a basic Truenas system, so IMO there's no need for videos about that. However, if there are questions without clear answers (for example, which nitty gritty settings are best for certain use cases), then videos from labs that answer those questions would be helpful.
Love to see a chart showing hardware redundancy etc. in case an entire array goes offline. When I was working, servers racks you couldn't easily stand on both sides to load stuff were hell. Glad you bolted it down too, had a server rack not bolted down at a clients site once and yeah. Finally, time to convert one of the bigger rooms into a full service server room with hvac, raised floor, etc XD
Just realized: JBOD has been in use for iPods too! You know, the old iPods with the clickwheels and such- they use microSDs as cheap hard drive replacements! JBOD is the connection they use, since it works well and doesn't need a fancy controller chip.
JBOD is not a "connection", it just refers to an enclosure with a number of disks passing each disk off to the OS or a controller.
Those iFlash quads aren't JBODs...
@@williamcampbell9859 Thanks, I had forgotten the term and just threw something in there 😂
Just a thought, maybe try using MLV (mass loaded vinyl) to try and cut down on the noise escaping from the server room. Companies like Audimute sell it in rolls. Using something like mineral wool or acoustic foam isn't going to do much for stopping sound transmission.
Linus used cavity wall insulation (recycled jeans) instead of actual sound dampening materials when he set up the room.
The issue with vinyl is that if it gets heated beyond a certain point it combusts
Another great video as always - Thanks! One wishlist item is for more bleeping on some of the stronger language. No issue with the base words during the course of a vid but seems like there is more stronger language coming through over the past months. Again not a prude but some words such as gd do bother me. Thanks again and best for all the plans in the works.
I just love how Linus is being all serious and Jake is more interested in when his burrito will arrive 🤣
I almost had a heart attack when linus ran in there with the smaller server
I have no idea what you're talking about, but I find all this strangely captivating.
Sometimes I think Linus' company is worryingly inefficient, but then I go to work
Then you look back at Linus and his team and go "You know what, those guys are doing a great job! I wish my company was more like theirs."
How many storage endgames will this man have, it feel like yesterday he made his last massive server
Server videos are always fun I don't understand anything they are saying but it is still fun to watch
At this rate, maybe Linus will make his own cloud storage services available to the public.
I don't know if I'm right, but it feels like your editors all have slightly different styles that show up in the videos they edit. I really enjoy the variation.
I'm a scientist at a multi-million dollar tech company and Linus has a better IT infrastructure than we do.
I'm a gamer!
@@djhokage1 I'm a data scientist at a cybersecurity company so its all IT but okay sure.
To be quite fair, they're shooting 8K/ 8K HDR content which are LARGE files + they're also a multi million dollar company hahah
@@valismind Yeah 1000p. Their data needs are beyond massive. This kind of content is half the reason I subscribe.
Which company
Does Wendell have a video on the data recovery process? I think that would be absolutely fascinating to watch.
Yes please!
I like the funny dynamic in this video, it's refreshing and actually makes it fun to watch
I love how unassuming high end hardware gets. Those SAS cards look so much like some trash tier GPU. Green PCBs are like the beige boxes of circuit boards now
I think we forget sometimes that it's really only "enthusiasts" who care about the looks of anything inside their case (and not even all of them, either). Regular users never open their computers, and even if they do they don't expect anything "pretty". And businesses don't care either, do they? Both have no emotional connection to their hardware, and they don't "present" it to others or see it as decorative part of the room, either.
I work in IT with systems and other stuff that will crush any "enthusiast" nonsense. It's a completely different game where cost is no object and nobody gives two shits how things look. You're talking budgets where millions doesn't describe it, that's a budget error nobody cares about. A single rack costs millions. And you're talking multi acre data centers. The raw scale of it all would blow most peoples minds.
Unless you work for a for real institution like Microsoft, Amazon, the military, facebook, you have no godly idea how stupid things get. It's all laughable. And while I don't directly do data center stuff now talking to people on headsets wandering around multi acre facilities while they have to use hear protection and wear winter gear is still a thing. And there are multiple facilities on that site. "We have the linux complex, the ibm AIX complex, the windows complex" and complex is multiple warehouse sized monstrosities. Walking into one is like walking into a jet engine that pumps cold air. Tons of the stuff is classified and even though you are cleared there are armed guards and you check your stuff at the enterance. Many ban smart devices completely as they can be used to trace movements, so leave in car or lose job.
Seeing Linus lie under like really heavy JBODs made my anxiety go through the roof. One (admittedly unlikely) slip and his chest would be crushed
i am beyond confused on how the entire layout of the servers work, but i still enjoy watching the videos on it all lol
WOAH THAT'S HUGE
That’s what she said 😏
Bruh
Random thought: I'm trying to picture Linus 40 years from now, will he still be making videos or will he pass the role onto his kids?
Undoubtedly there will be something on a New, New, (x25) Whonnock....*whatever the hell whonnock is supposed to mean anyway*...
How dare Jake not drink from an LTT water bottle! What a monster! 12:33
We all know Linus' real endgame is having enough storage to finally upload his consciousness into a machine, then he can run LTT in perpetuity as a sentient AI that constantly drops segways to our sponsor.
Amo gus
Real and True
Very
same
Amo Gus 😔
Based and sus
Man, you guys get the coolest toys to play with.
50-100 years in the future the humans then will view these as being archaic and oversized. Data centers will probably be no bigger than the average desk computer today and a pc capable of high end gaming will be no bigger than a small tablet. At least that’s what I hope. But considering how little we’ve progressed in the grand scheme of things part of me has doubts.
honestly if that is the truth those bit capacitors would be on quantum level or we moved on to DNA storage lol
1:05 *manta*
Manta
@@zetsu2816manta
LMG banter is the best. Couldnt imagine a better place to work!