"this is pretty dry technical stuff" honestly to me this is one of my favorite LTT videos in a while. It's always interesting to see real-world troubleshooting with this kind of hardware.
Ben Graham yeah I agree. I’d like to see way more of this sort of stuff but the channel is becoming more and more about just gaming and hardware related to consumer gaming. I’d like to see more infrastrure hardware and software videos.
There is a reason why so few high performance solutions use off the shelf parts and drivers. When developing solutions, the architect often has to dance around these kinds of limitations, sometimes even correcting bad drivers themselves.
"This is a lot of dry technical stuff" Yes, please make more! While the usual stuff is fun, the problem solving for real world deployment is very interesting. Maybe in 10 years or so I'll be using this hardware when it's obsoleted on the cheap. For now I'll keep running my 10+ year old opteron server.
@@Keyakina Yeah, I did the math a while ago and figured I have about $35,000 original MSRP worth of hardware in my rack. Spent about $1000 on the lot of it over the last 3 years.
@@wertin200 sand is used to make silicon which is used to make ICs/electronics, so they are saying that "teaching sand to think", that is making computers out of it, was a mistake because of these problems that are seemingly random at first.
@@tainicon4639 yeah you'd be surprised how many people actually enjoy learning and problem solving, rather than gawking at the next new Shiny toy shat out by Razer or Samsung.
"Teaching sand to think was a mistake" - Every electronic, software, systems, and network engineer ever. Incidentally i'd buy an LTT shirt with that quote on it.
It's not actually silicon that makes logic gates work, it's the doping impurities we introduce to it like boron and phosphorus that makes them semi conductors.
Subparanon facepalm, this quote took far too long to figure out ;-; I'm so blind sighted, But yes indeed, the impurities is what causes the difference in charges - thus allowing for manipulation (and thus we have a semi-conductor)
@@Subparanon Correct. Also there are multiple substrates you can use aside from crystalline silicon to dope into p & n types depending on the physical properties and electrical characteristics you desire. However "refining silicon into a pure unbroken crystal, slicing into wafers, cleaning, then doping using deposition & photo-lithography into a web of complex nano-meter sized features that from a microprocessor core, packaging with supporting micro-electronics into a housing, sold to a computer system integrator to be built with multiple other specialised silicon-based electronic circuit technology, to be programmed by low & high level software engineers so that computational tasks can be performed, written and read from non-volatile memory circuits" just isn't as catchy. But if they put *that* on an LTT shirt i'll buy it as well.
Wendell is like that hacker guy in any move who they call up in an emergency and ask about whatever super advanced computer related thing that has to be fixed to save the world from ending and he just answers like an encyclopedia. So far the world hasn't ended. Thank you Wendell.
"I thought it was as simple as 'OK let's load the final OS on this thing, chuck it in the server room we're ready to start editing off of it'" This is the single mentality that causes LTT's "enterprise" content to be so amazing.
I love the "do first, ask questions second" mentality. Yeah, they might have a ton of fuckups along the way but that's half the fun! WHOLE ROOM WATERCOOLING
I would said there are a small crowd of us that really enjoy this technical detail. It's fun and interesting topics. What you explained was low enough detail for the average computer user to understand. This is one of the reasons I'm following this channel. I'm a software engineer with a computer engineering degree. You keep me up to date with the latest news in the industry.
2001 Linus: “Oh my god!!! Intel came out with a 2 core processor!!! This is so cool!!!” 2020 Linus: “So 24 cores wasn’t enough. I’m gunna need 64.” AMD: “We got you fam.”
2001 Linus: Wait, so that super computer has 64 cores? WOW! 2050 Linus: This holowatch has 8192 cores. That may not sound like much, until you realize... You just put your wrist into your VR chair and you get an extra 8192 cores on top of the chair's processing power! Not too shabby for a simple little watch.
@@bradhaines3142 hey now, if we were idiots we wouldn't be watching ltt; i think you should really calm down with the name calling; i was just stating a fact
Ahahhaha im sorry but putting out unboxing videos and people building computers is only creative the first ten videos. After that it is just a job. You’re so cute though.
Did you take a look at reserving CPU core solely for kernel use as well as pinning IRQ to those specifics cores and played with numa nodes ? The key is to have NICs and Storage on the same CPU package (or numa node) to avoid cross memory traffic. Btw, you really should have a look at distributed filesystem (like Lustre FS, etc.)
+1 for distributed filesystem. Also screw RAID 5 for this. What kernel schedulers are you using? This whole thing is a very solved issue, so I'm chalking this all up to user error/lack of knowledge.
Amazing video. I used to do this type of build and troubleshooting for cloud systems in the past. The crazy things you run into when pushing limits of the technologies you are using can really teach you a lot.
"How do I always seem to find these weird edge case errors?" I dunno Linus, might have something do with going to the absolute balls to the wall bleeding edge of current technical computing capabilities like having "one of the fastest servers on the planet".
I laughed way too hard at his comment. At work when people would call in irate and complaining that they deleted something important "...being angry because the sand did exactly what the meat told it to"
IMO its more like heading into engineering with everyone at work wanting to be left alone, Then the server room is more like being in a jefferies tube adjacent to the core with wires going everywhere and a single wrong movement means something dies.
Me: they're still using the 24 core, I thought they had a 64 core? Linus:one thing we haven't mentioned, we've replaced the 24 for the 64, but went down to 32 bc 64 is overkill Since when has an ltt project not been overkill
@@nicolasa.bermellferrer8025 That's dumb. Last I checked Minecraft is still shit at utilizing a ton of cores and prefers high single thread performance. Basically a 64 core CPU would only be good for running a lot of servers but not at top tier performance. An overclocked 9900k would probably give the best performance for a Minecraft server.
PLEASE try some more things to optimize performance. In particular: - Switch between MSI and MSI-X IRQ Signalling if possible in the BIOS - Manually set CPU affinity for interrupts - Switch I/O Schedulers - Tune virtual memory
Designing storage arrays is very difficult. A good friend of mine was one of the original EMC developers, prior to being bought out by Dell, and he specializes in developing custom storage arrays for fringe case companies. I have great respect for Wendell, but after talking for many hours/days with my friend about storage design... especially flash based storage, it isn't something that you can easily configure with off the shelf hardware and not spend a fair amount of time writing you own code as well, assuming you are looking for the kind of IOPs and bandwidth that those workstations would be consuming.
@Caligula "We're getting a lot of CPU bottlenecks, here". Yes, Linus. That is why real enterprises get servers with a dedicated RAID card, and don't do software RAID.
@Caligula In the SMB market it is very hard to justify the cost of enterprise storage. There are mainstream solutions that attempt to bridge that gap, but thy aren't much better than DIY solutions. That is where the market for boutique storage designers come into play. The problem is most of these are not well known outside of Hollywood and other similar niche markets, so someone like Linus wouldn't necessarily have the connections to get a line on someone who could help him. I had a large state client who needed a storage solution for the platform we sold them. We sent vender recommendations, but they were stuck with HP as they had a purchase contract agreement with them. They could have used a $15-20k storage system from a mainstream provider, but as they had to purchase Enterprise level gear from HP, the cost was $220K+, which exceeded their budget.
@Caligula That's kind of the point of this channel, "DIY everything, regardless if it's a good idea". And it's way more entertaining than watching reviews of products that you don't even care for.
@Caligula Why would you play street fighter if is not on evo level then? Why would you practice any sport if you don't do at pro level then? Why would you cook something for youself if you can't reach a 5 fork cuisine level then? The fact you are watching these videos should make you realize why. It's entertaining.
@@MazeFrame PCIe Gen 3.0 x16 is enough bandwidth for 100GbE. PCIe Gen 4.0 has allowed 200GbE, which is available today. The current industry roadmaps point towards PCIe 5.0 CPUs coming online around 2022, which will bring 400GbE. PCIe 6 is slated for around ~~~2025, and that's going to bring 800GbE.
@@MazeFrame thats a switch though, designed for top of rack use. A 400GbE NIC would require a PCIe 4 x32 link at the server level. A 400GbE port at the switch would support 4x100 or 2x200 breakout cables to 100GbE or 200GbE NICs at the server. So yes, there are 400G top of rack and spine switches, but no 400GbE NICs at the server yet because the PCIe bus isn't fast enough until PCIe 5.
@@karthik11512 it's confusing if you're not in the industry, but there are 40GbE and 50GbE Ethernet specifications. They are similar in speed but are based on different specs. 40GbE is based on the same technology as 10GbE, and uses a quad-SFP+ (QSFP+) connector with 4x10GbE channels bonded as a single 40G link. 50GbE is a separate specification based on 25GbE. It is also related to 100GbE. A 25GbE Ethernet connection uses an SFP28 connector with a single 1x25 link. 50GbE uses a quad-SFP28 (QSFP28) connector with 2x25g links to form a 50Gb/s channel. 100GbE uses the same QSFP28 connector as 50GbE, but uses all four channels of that QSFP28 connector to yield a 4x25 which forms a single 100Gb/s link. With 200GbE now available, there are all kinds of new SFP56, QSFP56, OSFP, etc connectors... And 400GbE is around the corner around 2022 in the server space when the PCIe5 processors come around.
Cheap monitors wobble, and they summed up everything about it in two words. I burst out laughing when I read that. (There are a lot of Chinese earthquakes in my house. :( )
i'm mindblown by the fact of how this channel developed over the years. They deserve all of that youtube fame... most down to earth famous-youtuber i know
Cutting edge technical issues video, this was good. If you're watching this channel then probably you like learning and engineering. I enjoyed this because this is a new but old problem: where is your system bottleneck and will it affect what you do. Great to see Linus Tech Tips pushing technology to its limits and therefore helping to create new tech tips.
T A You shouldn’t learn from this video, its fundamentally flawed and inaccurate technical issues that are being addressed the wrong way. Its basically watching someone in his bedroom playing about trying to build a computer and ringing his mate when he gets stuck, the mate is also clueless but knows more technical words so make himself look like he knows what he’s doing, the public watching the video doesn’t know any better so just accepts what they are shown. Ultimately this video is only useful for entertainment purposes. Clearly, no one knows the design phase, maths behind the decisions and software development to ensure the correct platform is used the right way. As a professional organisation LTT needs reliable storage solutions and its nice to see them trying to solve their own issue, but they need better support than level 1 techs who should stick to home networking as this is clearly an issue for them and they have no idea what the problems are or how to solve them. Had the design phase been done correctly, there wouldn’t have been any issue in implementation and testing.
@@ChrisSmithy entrepreneurs go fast forward and don't sit down and work out all the exact details. They have an understanding of the problem, of a fix and they go fast at solving it. Their speed usually sees them right. Linus has solved his storage access problems, ran into other storage access problems, and he's used it to create interesting content. He's also saved himself money, got some learning, and taught us all as well. And he's making money on this! The guy is a genius, a likable genius. That place looks like a fabulous place to work. You make valid points about engineering and getting it right but he's running a very fast demanding business and he's going in the right direction most of the time, and that's enough to be very successful. We don't need perfect, we just need to be fast, good and cheap :-)
T A you don’t work with any entrepreneurs do you, that’s so not what happens in industry! I love the theory and the potential that someone somewhere is doing that, but I can assure you, even at r&d and prototyping stages it’s not done in fast forward! I think I can categorically say, no entrepreneur goes fast forward and puts a product into a production environment in this way, but not only that, this isn’t a product design in any way, its buying off the shelf kit and trying to make it work, lets not make it into something its not. Used to be the way a lot of years ago I agree at a design idea stage people would hammer things together to see what happens, but that would never be the finished product. Beside the point really anyway as this was a simple server deployment and not any kind of a product design. I must say though, it’s not a dig a Linus, I think he’s doing the right thing by bringing this to everyone to identify the issue that this is a specialist job and needs to be done by someone with such skills, but there should be a behind the scenes network with reliable data storage and this should just be for the public entertainment as can you imagine the affect it would have on the Iinus media group when things are bodged together like this. My criticism is connected to the support he receives, it’s substandard and not acceptable for a professional business. Obviously I’m a ltt fan, I wouldn’t watch otherwise, but I’m also a network design engineer and electronics design engineer by trade and watching Linus being advised badly for something business dependant really kills me. I work with storage engineers all the time and they regularly use the videos from ltt to make an example of the wrong ways to do things and I won’t even get started on level one techs or I’ll be going on forever. It’s no dig at Linus or the channel, but if you want something done correctly, use the right people and have more of an understanding first. He didn’t solve the network storage issues here at all, not sure if you watched this video, but it was bodged to work with absolutely no advantage considering the level of hardware used, no one even noticed the difference. So yeah, the video was great, Linus (and team) is a legend but what he taught us was don’t try to run before you can walk and do the design phase before committing to anything, oh and most importantly, make friends with someone who can support you properly through the project. In the land of the blind the one eyed man is king, the general public knows very little about IT, so when someone has a little bit of a clue they look awesome, this has happened here. No criticism of LTT or any of their team, they make amazing videos that I’ve always enjoyed ( even if the advertising kills me ) but this is a business after all and the hardware needs to back this up. Just to say as well, my initial comment was more due to you saying it was a cutting edge technical issues video, it wasn’t, it was a truly good guy trying to make something that should have been great, but was badly advised and there was no design to suggest it would even work at all. No consideration for PCIe lanes, functional available bandwidth, memory bandwidth limitations, caching, IOPS, technical functional requirements and most importantly no application layer considerations. Linus makes great videos, i stand by that and i will continue to recommend the channel to everyone, but the current trend of using other TH-camrs to help with projects that are just totally above their technical level is just showing why this is the limiting factor of TH-cam, at the end of the day, Linus has staff to pay, profits to be made and I have never worked with a business that would allow us to install an unproven system that could potentially carry mission critical data and I don’t think Linus should either as I’d prefer the channel was around for a long time. Support and outsourcing skills is critical when its not available in house, my main issue is that although this is an entertaining video, its not right that it exists, Linus and the LTT team deserve better.
Emanuel Acosta I think it needed to be said. Just a shame more people in the industry avoid commenting, or watching TH-cam. LTT is an excellent entertainment channel in my opinion, but people shouldn’t think that’s how things work in the real world. I’m still a massive fan of LTT but if you want technical and real life engineering skills it’s worth watching Louis Rossman, not only is he a very intelligent guy, but he’s got real skills with a thought process, it may be in the repair sector but it’s still valid and very educational.
I like seeing dry technical stuff in videos. I feel like I'm seeing through a rare window into the the arcane arts of computer subsystems and expensive server configurations which I will never have access to. It makes it feel accessible and like I can at least understand a bit of it. It also makes me want to tinker and upgrade.
13:41 He says "large, large, large" and they zoom in, I can't help but think the bulging codpiece on the Stormtrooper in the background was part of that joke...
This video was incredibly interesting, I didn’t understand all of it but I think it’s the most enjoyable video I’ve watched from LMG in a while! Also loved the line ‘by adding JUST 8 cores’ - I wish I had 8 cores to begin with! 😂
man I love these types of videos, i'm in the same boat as Linus when he said "this is pretty dry technical stuff, but I find this fascinating". But then again it is in part because of Linus that i switched my college major to computer science.
Can hit those for large files, for small files that's still a problem, which is exactly why raid configurations are needed. But yeah we need ddr5 and moving all the way to 16x memory configs and I hope that the next Epyc will release accordingly.
Faster than my PC... Slower than my friends "budget"build lol. Stay alive fren |-/ And Tyler and Jenna's baby is gonna be born this week. I don't believe the hype. Ok I'll stop now sorry
Guys I appear to have lost my left sock. Can you help me find it? It's the winter months and I really need it back.
Pinned this for you. Hopefully the community can come together to sort you out!
i found a right sock, guess thats not it tho...
Did you double check your drawer??
do you have tow right socks?
Did you check the North Pole?
"this is pretty dry technical stuff" honestly to me this is one of my favorite LTT videos in a while. It's always interesting to see real-world troubleshooting with this kind of hardware.
Ben Graham yeah I agree. I’d like to see way more of this sort of stuff but the channel is becoming more and more about just gaming and hardware related to consumer gaming. I’d like to see more infrastrure hardware and software videos.
There is a reason why so few high performance solutions use off the shelf parts and drivers. When developing solutions, the architect often has to dance around these kinds of limitations, sometimes even correcting bad drivers themselves.
yes totally agree server logs and hardware like this is always very interesting
I agree
Ben Graham yep me too.
"This is a lot of dry technical stuff" Yes, please make more! While the usual stuff is fun, the problem solving for real world deployment is very interesting.
Maybe in 10 years or so I'll be using this hardware when it's obsoleted on the cheap. For now I'll keep running my 10+ year old opteron server.
I agree. It was good info to know.
I agree. It was good info to know.
I agree. It was good info to know.
Yeah it's wierd to know that in 10-20 years this will be cheap enough for us plebs
@@Keyakina Yeah, I did the math a while ago and figured I have about $35,000 original MSRP worth of hardware in my rack. Spent about $1000 on the lot of it over the last 3 years.
"That's what the $17,000 battery backup is supposed to be for"
*One fire later*
"The server room is fireproof"
Haxalicious room not the servers lmao
"teaching sand to think was a mistake" - Wendell (Level1Tech)
Incredible
ctrl+f "sand" = 16
yeah... we get it!
Big Brain comment by Wendell right there... lol
And then Linus be like : " :( "
i dont get it
3:15 "this is pretty dry technical stuff for a lot of people, but I thought it was fascinating"
So did I!! Pls more technical videos.
yea please :D
I agree theses are the best videos that LTT make.
Should be 6th (or however many channels they have) channel on their portfolio... I'd probably sub to floatplane for daily videos like this.. :P
rkan2 metoo
That is why we all are here, hard core tech specs
"Teaching sand to think was a mistake" lol Wendell is great
Why is that sentence funny?
that would be a badass line for a movie about robot and AI
@@wertin200 sand is used to make silicon which is used to make ICs/electronics, so they are saying that "teaching sand to think", that is making computers out of it, was a mistake because of these problems that are seemingly random at first.
I actually enjoy this "dry" content, and would love to see more of this
Anthony Johnson this is a lot more interesting than all of the other stuff they do haha
@@tainicon4639 yeah you'd be surprised how many people actually enjoy learning and problem solving, rather than gawking at the next new Shiny toy shat out by Razer or Samsung.
Front Butt horrific typo on my part haha. I meant that i find this stuff interesting. Haha
@@tainicon4639 I like watching them Jerry rig watercooling into random devices for the lolz but I do miss these more technical videos
hansamekute.link/VmPbpRHHNil
The problem is simple: not enough RGB to kickstart the system.
LOL
Everybody thinks this a joke, but RGB actually stands for *rapid gaming boost*
You got it right!!!
@@akira8817 i tought it was rapid gigabyte bandwidth, on the server side.
yes, clearly needed more RGB
"A lewd of your wife to enjoy on your lunch break" 4:40
Now we know why Linus needs the fastest servers in the world
I thought that’s what I heard but I wasn’t sure! 😂😂😂😂
pussy
"Teaching sand to think was a mistake" - Every electronic, software, systems, and network engineer ever.
Incidentally i'd buy an LTT shirt with that quote on it.
It's not actually silicon that makes logic gates work, it's the doping impurities we introduce to it like boron and phosphorus that makes them semi conductors.
Subparanon facepalm, this quote took far too long to figure out ;-; I'm so blind sighted,
But yes indeed, the impurities is what causes the difference in charges - thus allowing for manipulation (and thus we have a semi-conductor)
@@Subparanon way to ruin the joke with real-world facts and stuff. While I appreciate the "more you know" moment, at the same time bleeeeeeeeeh
@@Subparanon Correct. Also there are multiple substrates you can use aside from crystalline silicon to dope into p & n types depending on the physical properties and electrical characteristics you desire.
However "refining silicon into a pure unbroken crystal, slicing into wafers, cleaning, then doping using deposition & photo-lithography into a web of complex nano-meter sized features that from a microprocessor core, packaging with supporting micro-electronics into a housing, sold to a computer system integrator to be built with multiple other specialised silicon-based electronic circuit technology, to be programmed by low & high level software engineers so that computational tasks can be performed, written and read from non-volatile memory circuits" just isn't as catchy.
But if they put *that* on an LTT shirt i'll buy it as well.
@@Subparanon You must be fun at parties.
Wendell is like that hacker guy in any move who they call up in an emergency and ask about whatever super advanced computer related thing that has to be fixed to save the world from ending and he just answers like an encyclopedia. So far the world hasn't ended. Thank you Wendell.
Wendell is a Deus Ex Machina.
The cisco ramon of linus tech tips
@@Yes-mx1js I lol'd
@@Yes-mx1js he is offsite anthony
"Just another 8 cores"
What a good few years it's been for tech. Thanks, Ryzen.
Ethan Gustavson yep, this was literally my favourite line from the video, we’ve come so far!
Imagine needing a 32 core cpu for a storage server and its not nearly near peak speed.
@@Sorestlor its because of that minecraft server
16:00
"teaching sand to think was a mistake".
~Wendell, 2020.
"teaching sand to think was a mistake". One sentence to rule them all.
69 likes
@@dnasu Dayum!
@@windwalkerrangerdm it got ruined but not by me
"teaching sand to do math was a mistake" is the real power behind the throne
Speaking as a professional editor, "it's not broken" is high praise.
"Adobe Premiere pro is not responding"
@@xpeng121 *sweaty palms*
@@xpeng121 My favourite is definitely "unexprected error".
"I thought it was as simple as 'OK let's load the final OS on this thing, chuck it in the server room we're ready to start editing off of it'"
This is the single mentality that causes LTT's "enterprise" content to be so amazing.
I love the "do first, ask questions second" mentality. Yeah, they might have a ton of fuckups along the way but that's half the fun! WHOLE ROOM WATERCOOLING
Making me wince every time he says "Windows Storage Spaces"
@@kaukospots makes me wince when people think what Linus does is what sysadmins do to 😰
When every fuck up is potential for more content I imagine there isn't to much concern about fucking it up the first time around.
I definitely don't view it as a bad thing in their case. He's done a lot of stuff I've dreamed about but ultimately decided was just too risky.
I would said there are a small crowd of us that really enjoy this technical detail. It's fun and interesting topics. What you explained was low enough detail for the average computer user to understand. This is one of the reasons I'm following this channel. I'm a software engineer with a computer engineering degree. You keep me up to date with the latest news in the industry.
2001 Linus: “Oh my god!!! Intel came out with a 2 core processor!!! This is so cool!!!”
2020 Linus: “So 24 cores wasn’t enough. I’m gunna need 64.”
AMD: “We got you fam.”
"For a price, Ugarte, for a price."
@@chooseymomschoose A surprisingly reasonable price.
2001 Linus: Wait, so that super computer has 64 cores? WOW!
2050 Linus: This holowatch has 8192 cores. That may not sound like much, until you realize... You just put your wrist into your VR chair and you get an extra 8192 cores on top of the chair's processing power! Not too shabby for a simple little watch.
32*
@@manictiger 2090 Linus: This computer has infinity petabytes of ram & storage! It is pretty good.
“A lewd of your wife to enjoy on your lunch break” do u wanna talk about anything Linus?
God’s Dead time stamp?
Yvonne just had an aneurysm
now we know what he needs such a huge server for. its all fucking lewds
Knowing him it's probably a groan worthy injoke.
Now what *kind* of groan is a good question.
it's actually spelled, "in lieu"
Thank you Edzel for being the only source of actually talking about if it's a good upgrade or not
"teaching sand to think was a mistake" is a line I will continue to use in the foreseeable future, thanks Wendell
Mystery yeah thought the same, very nice quote
We had to trp lightning inside of it as well :)
where did her say that?
"Teaching sand to think was a mistake" GIMME THIS SHIRT!!!!!!
I would definitely buy that
I dont understand the quote. Someone translate for my tiny brain
@@iwellbreastfed sand is made of silicon, that's the main component of chipa
Quote is not original, but I'm not even sure this is where the quote originated: twitter.com/mgattozzi/status/1042253511176544256
@@tiger.98 I thought Wendell was referencing Linus, not the chips. :P
" thats why we have the $30k battery bank"
* literally a few weeks later the battery bank blows up*
boi i know
this was a joke comment from 5 months ago that you found just now and commented on?
@@bradhaines3142 70 people disagree
@@bradhaines3142 hey now, if we were idiots we wouldn't be watching ltt; i think you should really calm down with the name calling; i was just stating a fact
**BRIAN THE ELECTRITION**
I like how every time Linus talks to the editors it's always "please go away... we have work to do"
i feel bad for them just chugging alone pushing videos out day in day out. gross job.
@@curt8806 oh yeah dude im sure they hate it man definitely worst job ever. No creative freedom just mindless slaves
Nathan Buckingham lol the most creative they get is adding millennial memes. I’m sure that gets old after their tenth video a day
Ahahhaha im sorry but putting out unboxing videos and people building computers is only creative the first ten videos. After that it is just a job. You’re so cute though.
@@curt8806 does that not just describe every job ever?
"Teaching sand to think was a mistake." - Wendell 2020 -
Someone please translate for my tiny brain
@@iwellbreastfed Computer chips are made out of Silicon. Silicon Dioxide happens to be the primary chemical make of of sand.
@@freednighthawk I thought that but wasn't sure it was what he meant. Thanks
I played mas effect, synthetic life has a place... Lol
@@freednighthawk woah. Thank you
"teaching sand to think was a mistake."
I want that on. a. shirt.
It is on the level1techs store
Did you take a look at reserving CPU core solely for kernel use as well as pinning IRQ to those specifics cores and played with numa nodes ? The key is to have NICs and Storage on the same CPU package (or numa node) to avoid cross memory traffic.
Btw, you really should have a look at distributed filesystem (like Lustre FS, etc.)
+1 for distributed filesystem. Also screw RAID 5 for this. What kernel schedulers are you using? This whole thing is a very solved issue, so I'm chalking this all up to user error/lack of knowledge.
My thought as well. On many core CPUs, you could fairly easily sacrifice one core specifically for this.
Please upvote this guy.
Yea, I'm not sure if NUMA affinity has anything to do with it but I think you are on the right track: user error.
At the end the day Wendell is truly a Level 1 Tech!
Linus: "I want you to work just not important work"...
Editor: "So just carry on with what I was doing?"
Amazing video. I used to do this type of build and troubleshooting for cloud systems in the past. The crazy things you run into when pushing limits of the technologies you are using can really teach you a lot.
"How do I always seem to find these weird edge case errors?" I dunno Linus, might have something do with going to the absolute balls to the wall bleeding edge of current technical computing capabilities like having "one of the fastest servers on the planet".
eh, this stuff really doesn't compare to enterprise server solutions.
@@Llama052 tell me more
I've nothing original to say, I'm just commenting to increase engagement because I really like these technical videos 👍
+1
Shoot. Why not.
You know you have good employees when they get annoyed at their boss for interrupting their work
"A lewd of your wife to enjoy on your lunch break." I guess now we know how Linus spends his lunch breaks.
Yeah that was kinda outta left field heh
Me writing software while listening to video when he said that: "wait, what?"
I wasn’t sure I heard that right. Guess I did. 😟🤭
I heard that as well, but is that really what he said?
What? Everyone enjoys lewds of their wife on their lunch breaks.
Right?
...
Guys?
"teaching sand to think was a mistake" - Wendell.
I think that is the highlight of highlights for all IT departments
I laughed way too hard at his comment. At work when people would call in irate and complaining that they deleted something important "...being angry because the sand did exactly what the meat told it to"
Marshall - Yep, typical PEBKAC's. 🙄
I adore these really deep technical videos with all your problem solving steps.
Why does it feel like when Linus walks into the editing room that he's stepping onto the bridge of the Enterprise? Captain Tyran
IMO its more like heading into engineering with everyone at work wanting to be left alone, Then the server room is more like being in a jefferies tube adjacent to the core with wires going everywhere and a single wrong movement means something dies.
And then one of the editors tell him: no, I won't participate xD
"A lewd of your wife to enjoy on your lunch break"
You had a good lunch today, didn't you Linus
@Punkmeister nah he said lewd
I love deep dive videos, these more technical ones like this. I really really do! keep them flowing whenever you have occasion
"Teaching sand to think was a mistake" 😂😂😂
@@Oakine Hey, thanks! Going alright, just testing some laptops haha, as you'd expect 😁
@@Oakine MAX-15/MAX-17
Quote of the day: "teaching sand to think was a mistake" - 16:01
Me: they're still using the 24 core, I thought they had a 64 core?
Linus:one thing we haven't mentioned, we've replaced the 24 for the 64, but went down to 32 bc 64 is overkill
Since when has an ltt project not been overkill
I suppose the 64 core cpu is for another project. Aka MC SERVER :v
@@nicolasa.bermellferrer8025 MC server isn't actually going to be run by them anymore
@@nicolasa.bermellferrer8025 That's dumb. Last I checked Minecraft is still shit at utilizing a ton of cores and prefers high single thread performance. Basically a 64 core CPU would only be good for running a lot of servers but not at top tier performance. An overclocked 9900k would probably give the best performance for a Minecraft server.
Uhhh Video editing is the backbone that produces these channels.... Why wouldn't you throw 64 cores at it if you have it and will improve times!?!?!
@@ZeroSpawn because it won't improve anything? At the speed the server is going at, they can scrub in full res which is more than enough.
This video was very interesting and informative, I love to see real work problem solving in very intensive network request on servers. Kudos !
4:40 "Let's say a.. a lewd of your wife to enjoy on your lunch break"
Damn, didn't know we eating RAW
Based linus
it's actually spelled, "in lieu"
So that's what Linus does during his lunch breaks.
@@Chaseticles No he says "a lewd".
I'm more amazed that no one not even Linus cracked a laugh or a smile when he said that lol
PLEASE try some more things to optimize performance. In particular:
- Switch between MSI and MSI-X IRQ Signalling if possible in the BIOS
- Manually set CPU affinity for interrupts
- Switch I/O Schedulers
- Tune virtual memory
Its just the rushed out AMD cpu.
@@draken5379 That's why I want to know which of these software settings affect the performance of that bottle-necked system...
I love videos like this. I know you thought it was dry, but seeing you go over content like this really makes my day.
dry technical stuff - The most coherent and enjoyable video in a while*
M.Streicher wait you’ve posted like 2 minutes after release, how can you know it’s enjoyable and coherent?
"A lewd of your wife to enjoy in your lunch.." - linus tech tips
had to pause when he said that...
@@user-el4su7tl6f same
brooooo
Id expect everyone to be commenting about this, but i guess im just immature.
guess we know what Linus does during his lunch break
"Teaching sand to think was a mistake"
lttstore.comThatThingPlease
Designing storage arrays is very difficult. A good friend of mine was one of the original EMC developers, prior to being bought out by Dell, and he specializes in developing custom storage arrays for fringe case companies. I have great respect for Wendell, but after talking for many hours/days with my friend about storage design... especially flash based storage, it isn't something that you can easily configure with off the shelf hardware and not spend a fair amount of time writing you own code as well, assuming you are looking for the kind of IOPs and bandwidth that those workstations would be consuming.
@Caligula "We're getting a lot of CPU bottlenecks, here". Yes, Linus. That is why real enterprises get servers with a dedicated RAID card, and don't do software RAID.
@Caligula This shit's entertaining though? To me at least. Building the whole thing, and even the issues that come along the way.
@Caligula In the SMB market it is very hard to justify the cost of enterprise storage. There are mainstream solutions that attempt to bridge that gap, but thy aren't much better than DIY solutions. That is where the market for boutique storage designers come into play. The problem is most of these are not well known outside of Hollywood and other similar niche markets, so someone like Linus wouldn't necessarily have the connections to get a line on someone who could help him.
I had a large state client who needed a storage solution for the platform we sold them. We sent vender recommendations, but they were stuck with HP as they had a purchase contract agreement with them. They could have used a $15-20k storage system from a mainstream provider, but as they had to purchase Enterprise level gear from HP, the cost was $220K+, which exceeded their budget.
@Caligula That's kind of the point of this channel, "DIY everything, regardless if it's a good idea". And it's way more entertaining than watching reviews of products that you don't even care for.
@Caligula Why would you play street fighter if is not on evo level then? Why would you practice any sport if you don't do at pro level then? Why would you cook something for youself if you can't reach a 5 fork cuisine level then?
The fact you are watching these videos should make you realize why. It's entertaining.
Wendell .. “tech support for tech support “
"Your favorite techtuber's favorite techtuber"
basicly
8:10 Oh no. Only 40Gbps connection? How will you survive?
400GB networking when?
@@MazeFrame PCIe Gen 3.0 x16 is enough bandwidth for 100GbE. PCIe Gen 4.0 has allowed 200GbE, which is available today. The current industry roadmaps point towards PCIe 5.0 CPUs coming online around 2022, which will bring 400GbE. PCIe 6 is slated for around ~~~2025, and that's going to bring 800GbE.
@@noname-gp6hk ethernet is not good enough. just switch to infiniband already!!!!!
@@noname-gp6hk Look at the Arista 7504R3 for example, 96 port 400Gb switch. Available since early 2019.
@@MazeFrame thats a switch though, designed for top of rack use. A 400GbE NIC would require a PCIe 4 x32 link at the server level.
A 400GbE port at the switch would support 4x100 or 2x200 breakout cables to 100GbE or 200GbE NICs at the server. So yes, there are 400G top of rack and spine switches, but no 400GbE NICs at the server yet because the PCIe bus isn't fast enough until PCIe 5.
The feeling when you are studying for IT BSc final exam and this helps you understand DMA, pooling and interrupts in a broader sense.
"There is a war between humans and firmware and everyone needs to pick a side" - Bryan Cantrill
"The problem with being faster than the speed of light is that you live in darkness"
16:01 'Teaching sand to think was a mistake' Lool
ive seen you comment on like 8 different vids
We need the TH-cam equivalent of the bat-signal for Wendell. Whenever someone is confused about server stuff we shine a giant W on a cloud.
What if it's foggy or clear skies? What if someone 3d prints a wendell signal and false flags?
Level1Techs logo.
@Khaffit :D
@@BGraves I feel like you just wrote a good basis for a new Batman film
@@LCTRgames yeah they should probably just text him
That end through....Bloopers is good, keep it ;) Linus and his script...Glad to know it was that through, figured he might know it all by hand.
"teaching sand to think was a mistake"
That is just a beautiful statement.
aws be like "its too big it wont fit" linux polling be like "is it in yet?"
"The drives are so quick nothing can keep up to its pace"
Kinda sounds like a line at the start of some weird sci-fi flick
"we only have a 40 gibabit network connection"
Remy Peacock that’s not internet its LAN bandwidth.
I’m buying a 50GB NIC soon for my server. 200 bucks.
freerice9595 I’ve never seen a 50gb NIC. Typically I see 1,10,25,40, and 100Gbps NICs out in the field. What NIC are you buying?
karthik11512 mellanox makes 50gbe.
Mellanox MCX416A-GCAT ConnectX-4 EN Network Interface Card 50GbE Dual-Port
@@karthik11512 it's confusing if you're not in the industry, but there are 40GbE and 50GbE Ethernet specifications. They are similar in speed but are based on different specs.
40GbE is based on the same technology as 10GbE, and uses a quad-SFP+ (QSFP+) connector with 4x10GbE channels bonded as a single 40G link.
50GbE is a separate specification based on 25GbE. It is also related to 100GbE. A 25GbE Ethernet connection uses an SFP28 connector with a single 1x25 link. 50GbE uses a quad-SFP28 (QSFP28) connector with 2x25g links to form a 50Gb/s channel. 100GbE uses the same QSFP28 connector as 50GbE, but uses all four channels of that QSFP28 connector to yield a 4x25 which forms a single 100Gb/s link.
With 200GbE now available, there are all kinds of new SFP56, QSFP56, OSFP, etc connectors... And 400GbE is around the corner around 2022 in the server space when the PCIe5 processors come around.
no name Legend. A perfect answer right there if ever I read one. Rare to see someone who actually knows what they are talking about.
Next on LMG: Making a Server With RAM Drives!
"We HAVE to put TEN car battery in it in order to make ALL the data alive!"
NEEDS MOAR DETITATED WAM!
Pepe Boi *explodes*
Marshall that’s to be expected
Saad Bin Haroon no
Linus: “That’s what the UPS is for”
UPS: time to go up in flames.
16:01 "Teaching sand to think was a mistake." -Wendell
Nice timestamp. Please explain the quote though I dont understand why everyone likes it.
@@iwellbreastfed Long story short, sand is used to make the silicon which in turn is used to make computer chips.
14:00 "[chinese earthquake]"
I don't even know why, but this made me laugh very hard
Cheap monitors wobble, and they summed up everything about it in two words. I burst out laughing when I read that. (There are a lot of Chinese earthquakes in my house. :( )
This was great news. We’ve had issues with Azure locking up inexplicably. I’m investigating if this could be the underlying issue.
"Teaching sand to think was a mistake" what a quote
"Teaching sand to think was a mistake." That needs to be on a tshirt.
0:21 I appreciate the camera tracking the editor did here to make sure Wendal was seamless.
Linus: "Just another eight cores"
Me:
I appreciate this struggle.
Holy shit... How old is that?
"teaching sand to think was a mistake" 10/10 would pay to see the comedy routine that included this!
LOVED the added footage at the end of the clip. 10/10 would watch more of those outtakes.
"That's what the 17k battery backup was for"
*buy LTT merch pops up in the corner*
I saw that too, I was thinking it was going to be a link to the video where they installed it... but they chose the merch plug instead
Hey, got to pay for more of that sweet, sweet battery backup...
Imagine having storage so fast, the Linux Kernel becomes a problem.
OMG THIS EXPLAINS ALL THE DIFFICULTY I WAS HAVING WITH MY QUAD NVME GEN4 ARRAY GAH THANK YOU
60% of the time, it works every time.
If you do it, and do it right, and dont fuck it up. It works... It just works...
The stuff I subscribed for is "dry technical stuff?" :(
It is a techy channel.
Everyone likes "dry stuff"!
I found it interesting, however not everyone is in to things like this.
i'm mindblown by the fact of how this channel developed over the years. They deserve all of that youtube fame... most down to earth famous-youtuber i know
Cutting edge technical issues video, this was good.
If you're watching this channel then probably you like learning and engineering.
I enjoyed this because this is a new but old problem: where is your system bottleneck and will it affect what you do.
Great to see Linus Tech Tips pushing technology to its limits and therefore helping to create new tech tips.
T A You shouldn’t learn from this video, its fundamentally flawed and inaccurate technical issues that are being addressed the wrong way. Its basically watching someone in his bedroom playing about trying to build a computer and ringing his mate when he gets stuck, the mate is also clueless but knows more technical words so make himself look like he knows what he’s doing, the public watching the video doesn’t know any better so just accepts what they are shown. Ultimately this video is only useful for entertainment purposes. Clearly, no one knows the design phase, maths behind the decisions and software development to ensure the correct platform is used the right way. As a professional organisation LTT needs reliable storage solutions and its nice to see them trying to solve their own issue, but they need better support than level 1 techs who should stick to home networking as this is clearly an issue for them and they have no idea what the problems are or how to solve them. Had the design phase been done correctly, there wouldn’t have been any issue in implementation and testing.
@@ChrisSmithy entrepreneurs go fast forward and don't sit down and work out all the exact details. They have an understanding of the problem, of a fix and they go fast at solving it. Their speed usually sees them right.
Linus has solved his storage access problems, ran into other storage access problems, and he's used it to create interesting content.
He's also saved himself money, got some learning, and taught us all as well. And he's making money on this!
The guy is a genius, a likable genius. That place looks like a fabulous place to work.
You make valid points about engineering and getting it right but he's running a very fast demanding business and he's going in the right direction most of the time, and that's enough to be very successful. We don't need perfect, we just need to be fast, good and cheap :-)
T A you don’t work with any entrepreneurs do you, that’s so not what happens in industry! I love the theory and the potential that someone somewhere is doing that, but I can assure you, even at r&d and prototyping stages it’s not done in fast forward! I think I can categorically say, no entrepreneur goes fast forward and puts a product into a production environment in this way, but not only that, this isn’t a product design in any way, its buying off the shelf kit and trying to make it work, lets not make it into something its not. Used to be the way a lot of years ago I agree at a design idea stage people would hammer things together to see what happens, but that would never be the finished product. Beside the point really anyway as this was a simple server deployment and not any kind of a product design.
I must say though, it’s not a dig a Linus, I think he’s doing the right thing by bringing this to everyone to identify the issue that this is a specialist job and needs to be done by someone with such skills, but there should be a behind the scenes network with reliable data storage and this should just be for the public entertainment as can you imagine the affect it would have on the Iinus media group when things are bodged together like this. My criticism is connected to the support he receives, it’s substandard and not acceptable for a professional business. Obviously I’m a ltt fan, I wouldn’t watch otherwise, but I’m also a network design engineer and electronics design engineer by trade and watching Linus being advised badly for something business dependant really kills me. I work with storage engineers all the time and they regularly use the videos from ltt to make an example of the wrong ways to do things and I won’t even get started on level one techs or I’ll be going on forever. It’s no dig at Linus or the channel, but if you want something done correctly, use the right people and have more of an understanding first.
He didn’t solve the network storage issues here at all, not sure if you watched this video, but it was bodged to work with absolutely no advantage considering the level of hardware used, no one even noticed the difference. So yeah, the video was great, Linus (and team) is a legend but what he taught us was don’t try to run before you can walk and do the design phase before committing to anything, oh and most importantly, make friends with someone who can support you properly through the project.
In the land of the blind the one eyed man is king, the general public knows very little about IT, so when someone has a little bit of a clue they look awesome, this has happened here. No criticism of LTT or any of their team, they make amazing videos that I’ve always enjoyed ( even if the advertising kills me ) but this is a business after all and the hardware needs to back this up. Just to say as well, my initial comment was more due to you saying it was a cutting edge technical issues video, it wasn’t, it was a truly good guy trying to make something that should have been great, but was badly advised and there was no design to suggest it would even work at all. No consideration for PCIe lanes, functional available bandwidth, memory bandwidth limitations, caching, IOPS, technical functional requirements and most importantly no application layer considerations.
Linus makes great videos, i stand by that and i will continue to recommend the channel to everyone, but the current trend of using other TH-camrs to help with projects that are just totally above their technical level is just showing why this is the limiting factor of TH-cam, at the end of the day, Linus has staff to pay, profits to be made and I have never worked with a business that would allow us to install an unproven system that could potentially carry mission critical data and I don’t think Linus should either as I’d prefer the channel was around for a long time. Support and outsourcing skills is critical when its not available in house, my main issue is that although this is an entertaining video, its not right that it exists, Linus and the LTT team deserve better.
@@ChrisSmithy Dude. Thank you so very much for your comments. They have been really informative. It's great to have the comment of a real engineer!
Emanuel Acosta I think it needed to be said. Just a shame more people in the industry avoid commenting, or watching TH-cam. LTT is an excellent entertainment channel in my opinion, but people shouldn’t think that’s how things work in the real world. I’m still a massive fan of LTT but if you want technical and real life engineering skills it’s worth watching Louis Rossman, not only is he a very intelligent guy, but he’s got real skills with a thought process, it may be in the repair sector but it’s still valid and very educational.
"a lewd of your wife to enjoy on your lunch break" 4:41
Who's wife...???
@@KyleChoularton I mean, doesn't really change anything
@@KyleChoularton I mean, that is a subset of the other one.
I like seeing dry technical stuff in videos. I feel like I'm seeing through a rare window into the the arcane arts of computer subsystems and expensive server configurations which I will never have access to. It makes it feel accessible and like I can at least understand a bit of it.
It also makes me want to tinker and upgrade.
Linus: "This is capable on paper of about a 100gb a second".......so hook it up to a printer.
*paper begins exiting the printer at .99 of the speed of light*
I've got a dot matrix. Let's get this party started.
printing minecraft bans book
13:41 He says "large, large, large" and they zoom in, I can't help but think the bulging codpiece on the Stormtrooper in the background was part of that joke...
This video was incredibly interesting, I didn’t understand all of it but I think it’s the most enjoyable video I’ve watched from LMG in a while! Also loved the line ‘by adding JUST 8 cores’ - I wish I had 8 cores to begin with! 😂
"Teaching sand how to think was a mistake"
LMFAO, That should be the quote of the century!
"Teaching sand to think was a mistake" --Wendell. Classic!
Andrew Fransen Does doped silicon think? I sincerely hope not!
16:01 _"teaching sand to think, was a mistake"_ 😂😂
00:20 I love when you guys use that clip of Wendell 🤣
man I love these types of videos, i'm in the same boat as Linus when he said "this is pretty dry technical stuff, but I find this fascinating". But then again it is in part because of Linus that i switched my college major to computer science.
I think the reason the servers are always getting so full is because someone at LTT has a folder with all the times Linus dropped something...
That sad face on the texts reminds me of a Windows blue screen :(
I like how linus is always afraid when entering the editing den like they're the real boss or something
"Teaching sand to think was a mistake" Wendell 2020 - I LOVE this line!
I miss Tarin on more videos. The one with iJustine reminded me how fun he is on camera.
They need to have him and Janice each do a Techlinked.
6:40 that is the most Linux answer I have ever heard...
My man just described linux in one sentence
Quote of the day: "teaching sand to think was a mistake" - 16:01
the server stuff is some of my favorite content from you guys.
"That's only 4 GBs a second!" haha yes Linus very slow
Well if you think about it a single good nvme drive can hit those speeds so its actualy quite bad
Can hit those for large files, for small files that's still a problem, which is exactly why raid configurations are needed. But yeah we need ddr5 and moving all the way to 16x memory configs and I hope that the next Epyc will release accordingly.
Faster than my PC... Slower than my friends "budget"build lol.
Stay alive fren |-/
And Tyler and Jenna's baby is gonna be born this week. I don't believe the hype.
Ok I'll stop now sorry
@@a.f.r.m...d.e.d.d.2898 Lol
Great stuff. I know these more technical videos are hard to sell, but you guys do a great job of them.
"Teaching sand to think was a mistake" best quote ever