My Server Flooded
ฝัง
- เผยแพร่เมื่อ 7 ก.ค. 2024
- lmg.gg/secretlablmgclips
Thanks to Secretlab for being the title sponsor of LMG Clips this year! Check out their ergonomic gaming chairs at the link above. Your back will thank you!
Linus’ server racks flooded.
Watch the full WAN Show: • Elon Musk Admitted I W...
► GET MERCH: lttstore.com
► GET EXCLUSIVE CONTENT ON FLOATPLANE: lmg.gg/lttfloatplane
► SPONSORS, AFFILIATES, AND PARTNERS: lmg.gg/partners
► OUR WAN PODCAST GEAR: lmg.gg/wanset
FOLLOW US ON SOCIAL
---------------------------------------------------
Twitter: / linustech
Facebook: / linustech
Instagram: / linustech
TikTok: / linustech
TikTok (LMG Clips): www.tiktok.com/@_lmgclips_
Twitch: / linustech - วิทยาศาสตร์และเทคโนโลยี
Didn’t know about his sister. Sorry to hear that Mr Sebastian
Damn. He had his sister on the show once on a build video. Is that her?
@@FrietjeOorlogno, he told it wasn't her
@@FrietjeOorlog No, he addressed it in one or two WAN's ago, that it wasn't the one in the video
yeah that hit me like a gut punch. My condolences, if you read this Linus
@@misterpuff797 thank god, the one in the show was so young, hope it wasn't premature tho
Sorry to hear about your sister.
Can someone explain?
@@DRSDavidSoft Linus had an unexpected death in his immediate family that he talked about a little in a previous WAN show, turns out it was his sister. He mentions it pretty quickly in this video.
@@TheSencatShit is that the sister we saw on video?
@@TheSencat People need to acknowledge why this is happening so frequently so it can finally STOP!
@@dakoderii4221 It's too late to stop it. At best, we can place the blame where it belongs.
My condolences Linus. I can't imagine the pain of losing a sibling.
May she rest in peace.
I'd watch that "resurrecting my water damaged UPS" video. It actually sounds super cool
Sounds more like a fire hazard, so super hot actually.
Something like a UPS you just dont risk, same goes for a PSU. Its safer, depending on why it died, to just get a new one. Water damage can make stuff do fucky things, and a UPS flowing quite a lot of power can still fail after some time even if its "resurrected"
There's not much to a UPS. If it's just him swapping internals from a donor there's no real risk as long as the chassis is completely dry.
@@renderedpixels4300 As long as you are competent you can repair a UPS, its functionally more like working as a mains electrician than a computer tech so you do need to know some things and the repair in most cases for anything now is replace larger assemblies - one FET died, replace the whole control board...
Multiple submarines have been lost due to water flooding into battery banks... I would not risk it. Just e-waste recycle it too be honest. If Navy folks cannot even figure out how to handle water inudation into battery banks then that UPS is a waste of time to try to salvage and dangerous.
I understand the whole point is for this to not happen to begin with but… there’s homeassistant compatible devices you can put on the floor to detect water leaks, then you can use that to fire off a critical emergency shut-down
I would like to see some kinda of device that can detect a change in flow or pressure, out side of a window, and fire off events
@@DarkAxi0m can’t remember what it’s called but there’s an LTT video about a device that keeps the entire water cooling loop under a bit of vacuum, so that if it has a leak the pressure will change and it can sound an alarm- but no water will get out because of the vacuum.
But this has giant, open reservoirs so that wouldn’t work
@@PentrossAquacomputer LeakShield
@@DarkAxi0m Or, a level detector in the tank itself. If it drops too low then somethings wrong and it needs to kill everything. As well as CPU temp monitoring if the pump fails
@@Pentross That's a bit over killed and would require the pump to be at the end of the loop etc.
Because they use a static flow rate pump, a single flow sensor on the return line would be enough. If the flow decrease even a little bit, you know either your pump is bad, or you have leak (be it small or big). You could complement that with a water leak sensor at the bottom of the rack and use them to turn of the pump in case of a problem.
Sorry for your loss Linus, and well done for having the restraint in that moment to not rip the server rack apart and smash it all into tiny pieces.
In 1988 I was working at IBM, in a brand new (unfinished) 56 level tower near Paris. And one day, we saw water falling from the ceiling, slowly at start, and becoming more and more like the Niagara’s fall. The building had 4 reservoirs of 30000l on the roof, for water usage in the building, and as a fire suppression, and one brand new valve broke, letting 10000 gallons of water, going down the 56 floors, devastating everything in its path. Fun time.The worst was the water getting more and more dirty at each floor, with the particles of ceilings tiles, and other crap.
that sounds like a spectacle
Why store underground and use pumps if you could just play the world's worst game of Jenga!
That is horrible
Where in this were you?
We had the same thing happen, but in an 8 story residential building and with the water heater. We were running all over with every bucket, bowl, and trashcan we could find trying to save musical instruments, tech, and furniture
sorry for your lost linus best wishes
"...I'm not saying it went well, I'm just saying that I'm not a f*cking idiot." -Linus 2024
Linus we watched the home build series.
maybe a voltage differential between 2 metallic parts in the same loop?
next time make shure every part is eighter grounged ot completly isolated.
Rip Linus sister I hope you're doing well Linus I know it's not easy to lose family that close.
Sorry for your sister first of all.
Your story really helped me today. Had a IT crisis today. Came back to a dead SSD after whole day of work and dealing with the gov for my ID. Saw this clip while frustrated. I got hope back and, following linus' precedence, left to get food instead.
Thanks for the motivation. Hopefully it'll work out for me too
Everything looks better on a full stomach
Manifesting everything working out Man U got this
Thanks for the likes! My SSD is completely dead and I had to reinstall my OS on a new one, but my backup setup worked and I got all the data back from the cloud. So everything worked out in the end
Air Cooler: I will overheat this cpu for fun.
Water Cooling: I will end this man whole career.
"I once used a flimsy intel stock cooler and now I think all air coolers suck." /s
@@xXRenaxChanXx technically,
@@xXRenaxChanXxthey suck and blow, its an air cooler thats how they work :^) Badumtss
reasonable AC will cause your CPU to throttle if fans stop but you still will be able to finish your work for the day, just a bit slower
if a pump stops... oh well
and if it leaks...
There's some universe where Linus just said F it and cooled directly with the pool water, and his entire basement filled with water.
The corroded connector sounds like it was caused by electrolysis induced by a grounding loop.
What if in the construction someone took a slightly identical connector made by a different material? It could be an human error
Al vs. copper
Have you tried putting it in rice?
He means well 😅
to summon Dennis?
The sister?
@@joaopedroalbernazoof
@@SignalJones Lmao
My condolences for your sister Linus, may she rest in peace
0:41 Damn.. Our condolences Linus.
In the aquarium hobby there is a sensor that sits on the floor and alerts you to water leaks.
That it the FIRST ff thing you do when doing shit with water.
Just knew about your sis, Linus. Rest in peace, and my soul and heart goes to your family
I'm always surprised at how resistant tech is to water. I'm still sticking to air cooling.
Yeah, I also got lucky with hardware and water: a few years back I was moving, my desktop was already outside to be loaded into a car, I did not notice it rained for a bit while I was inside finishing up. Once in the new apartment I connected everything and fired up the PC, which returned a black screen after a while I noticed water was coming out the bottom of the case, I my heart jumped, I let it dry for a day or so, tried again and everything was working just fine.
Yes, never had an air cooler kill a PC in almost 40 years of owning systems.
@@mastroitek Even if its not water the number of times I have had a issue with a piece of tech whether mine or my friends and I replaced it or set it aside intending to throw it away, check a warranty, evaluate possible fixes Etc only to get busy or forget about it and have it work perfectly fine once I got back around to dealing with it is kinda astonishing.
MY condolences go out to your whole family. I am sorry to hear that you lost a family member. It hurts so much, I can't imagine what you're feeling and totally respect you for coming in and still doing the things you do. Keep being a reason for me to have pride in the people of this city.
Oh, this brought back memories! I work for a large ISP. A few years back we were upgrading the incoming fiber to one of our contracted housing associations. This required drilling up and re-sealing the concrete floor of the server room, after which the association hired another contractor to sand and polish the floors. During this time, the existing hardware was still up and running.
When we came back the day after, we realized that the contractors hadn't covered the racks AT ALL. Every single system was CAKED in concrete dust, and they'd all self-terminated due to overheating. The entire room was covered in dust. We were less than pleased with the association, and told them that they would have to get the contractor to clean up their mess before we started to try to get everything up and running again.
Some of you just had a chill run up your spine, and yes, it is exactly as you feared. When we came back, they had literally pressure washed the entire room, including our racks. There was water dripping from the switches.
I still remember vividly how my boss got the call, stood up, and exclaimed incredulously - "They pressure washed the fucking racks.". I don't think I've ever seen a more apt example of someone having lost their faith in humanity...
They are 100% going to pay for damages
So sorry for your loss Linus, wishing you and your family some peace in these times❤
Maybe you need some sort of "drip tray" every few machines in the rack, which could divert the water away somehow. Like a rack shelf but with a fall towards one side and a drain hose away from the rack. This would isolate any failures to only a portion of the rack rather than losing a whole rack. If you also added a water detection into the tray then you'd know there was a fault. I didn't see anything with quick googling but I'd guess some 3D printing could produce something.
I think you could just make trays from sheet metal for that. Also yeah having flood sensors should be a no brainer as he's already running home assistant anyway, so all he really needs to do is to buy some.
I would add drain holes and tubes to all the systems which end in a single drip tray/container to limit them dripping on each other. Finally add a water sensor to the drip tray.
@@Iisakkiik Home assistant is a terrible way to deal with cutting power because you're relying on so much to work right; Best way to deal with it is to simply have a hardware water cutoff that cuts power, a close second is to use the tools that are likely built into the PDU's/batteries and have a graceful shutdown.
Dissolved solids in the water, and probably leaking current from the pump. I have zapped myself on a few occasions with hydroponic gardening and pumps leaking stray current into the reservoirs. Quickest way to find out put a bucket in the loop and stick your hand in there after a manicure if you can feel tingling on your cuticles its stray current.
Sorry to hear about your sister, thats heartbreaking
Im so sorry to hear about your sister. My condolences to you and your family.
I'm sorry to hear you lost your Sister Linus. I know it is poor consolation, but I'm sending my best wishes and condolences for your loss.
A few months back I had a water pipe break in my house right above my lounge room where all my electronics and my desk are and water ran through everything in there for hours before I came home to turn off the water, went through a similar state of mind when diagnosing everything but was also extremely surprised that damn near everything survived, even batteries(I watched em like a hawk), after letting everything dry for a week. modern tech is surprisingly durable
I do love when people forget that the IP rating isn't just about water. You tell them something is IP rated dustproof and they just look at you confused.
I also love how people don't understand that the IP rating isn't actually "proofing". Actual "proofing" rating is a different kind of "IP" rating.
My condolences and hope yall get the opportunity to properly mourn your lost
It all made sense when he mentioned EK for his water cooling
Sorry for your loss Linus
Instal moisture sensors throughout the rack.
Ouch man sorry to hear about that very hard day
My deepest condolences for your loss to you and the family, sorry for the server flood
I'm so sorry about your sister, I lost one of my brothers a few years ago and know how much it hurts. Losing a loved one isn't something you get over, it's something you learn to live with. My condolences and prayers to you and your family.
Sorry to hear about your loss
That's amazing most of the equipment survived. Way back when, my buddy had a 7990XT in a Mountain Mods Horizon case. He spilled a drink that ran off the edge of his desk right in to the open top, onto his 7990XT. I didn't see him online for a few months until he recounted this story to us. Lost the mobo and GPU to a cup of water.
Sorry to hear about your sister.
My condolences to you, the family and possibily her family and friends.
buy the thing, wanna see you fix it
You need to install humidity detectors, alarms, emergency shutdown valves etc
I’m soo sorry to hear about the loss of your sister my condolences to you and family
A video fixing this and having monitoring sensors. Bonus points for having a video camera so you can check up on it remotely.
pretty sure he already as a surveillance camera in his server room
Cameras only give you a view of what happened after. And may not work if the nvr recording it all is soaking wet
Lost a best friend and an uncle in the past, my condolences. Losing someone close to you never gets easy.
I wonder if there’s a way to add a mechanical shut off that runs from a flow rate monitor that triggers an emergency shut off when the flow rate spikes indicating a leak? Sounds like a Pii project.
Could do what factories do and put audio sensors that listen for out of place noises.
I'd say a pair of flow sensors before and after the rack, with logic that if there's less flow on the after than the before, you have a leak, and therefore shut down the pump and fire off a notification.
My condolences to you and your family about the loss of your sister.
sorry for your loss Linus!
Condolences on your sister. Losing family is never easy (I know I still have to properly deal with the death of my father over a decade ago). Best of luck to you and your family, immediate and extended and otherwise through this.
Have you considered reaching out to Jay (JayzTwoCents) about the water cooling issue? He may also have some suggestions including if you want to go with a different company rather than EK for whatever reason.
The company I work runs on a 2006 Dell Poweredge server, that got flooded, with a ton of mud, we pressure washed it, replaced the thermal paste, left to dry for a couple days on a room with the AC on the dry setting, and it's been working 24/7 ever since...
Pretty much all houses in North America will use 120v/240v two phase power, some apartment buildings and a lot of industrial sites will use 3-phase, where you can get 120v between neutral and a phase, or 208v between two of the 3 phases. So in theory appliances that use the ~200+ volt power could either get 208v or 240v, and there are usually fluctuations in the power grid so voltage can skew from those numbers as well.
So sorry to hear of the loss.
Fitting corroding and exploding is terrifying. Looking forward to update video because my water cooled server is now making me anxious lol
bro called xeon v3 ancient. My entire homelab runs on v3s and v4s
It is ancient.
Mine is also entirely on v2s and v3s.
Well, they're 10 years old
10 years is pretty ancient for tech
Much love to you and your fam. Sorry for your loss, Linus.
My deepest condolences. Never have I ever lost a family member as close as a sister, and I can't imagine how it might feel. I hope you have and will find strength in your loved ones in these trying times.
Sorry to hear thoughts go out to you & your family. That's sad
Brass and copper are still susceptible to galvanic corrosion. Galvanic corrosion happens when two dissimilar metals are in contact with each other AND have some sort of electrical current running through them. It can even happen to the same metal with different levels of impurity or different alloys.
Was the fitting grounded, or at least at the same voltage potential as the rest of the system? It doesn't take a whole lot of charge to get copper moving.
would love to see the ups rebuild video
Man, RIP to your sister and my condolences to you!
The first sentence is pretty much my experience with AMD as well. I only run a RX 6700 XT, but the eight months I have had with it so far were pretty uneventful. I had one minor bug that saw me roll back drivers to the last version and wait for the next one and that was it. Nothing I haven't experienced in the decade of Nvidia before.
Add a flow meter than alerts you when the debit is over a set value.
And as someone points out, it's probably an electrolysis because of a grounding short.
Check the other systems for possible grounding problem and electric tape them
My condolences Linus...
I'm sorry for your loss
Why is it stuff like this always happens after an awful day.
Sorry for your loss
I had a server room flood caused by one of the cassette ac's which had an electrical fault which triggered the fire sprinklers (because it's in an office building, you're not allowed to remove them). We immediately killed power to the server room, removed all the servers from the racks... Rinsed them under running water (because the insides were dusty before the flood), stored them on their sides to drain and hit them with heat guns and hair dryers. I think our survival rate was around 85 to 90% as well. With all of them going for another 4 or 5 years until they were upgraded and replaced during the next refresh.
What I do is use a heat exchanger for each machine to my cooling loop and use Koolance 705 non-conductive coolant. Highly recommend, been running like this for years.
first thing I always do is spray on another conformal coating on all PC parts pre install and after assembly , helps stop dust causing corrosion in humid area's and protects against any other liquid damage that might happen , amazed you got away with a flooded server rack , I normally use water to water exchanged loops so they liquids them selves are isolated per machine and the main loop just exchanges the heat from 1 block to another , sometimes impurities just get into the loops , I also nickle plate everything myself
sounds like a upgrade video, of adding in a fail safe. so if the water gets low in one of the tanks an alert goes out. if it gets lower it starts killing power to pumps and rails to keep a fire from happening.
I am sorry for your loss Linus, take as much time as you need.
My condolences Linus. May she rest in peace.
watch out on the lithium ups if water got into the bms it can cause a fire after a few charge cycles depending on how good the bms is once water gets in it can cause it to get stuck on losing the over charge and cell balance protection (why most ebike /scooter fires happen )
At my last job, an internal gutter broke. So under heavy rain, half of the roof water was piuring down... Through a switch!
There was an actual waterfall coming through the air holes in the front panels.
However, switch was still 100% working. We only noticed when the carpet in the office next to the rack room started being wet.
My condolences. Sorry for your loss
Thinking about an automatic fault detection. Something measuring the flow of water going into the loop and back out of the loop so you can measure the difference and if there is a difference, shut everything down.
Condolences on your sister Linus, greetings from 🇩🇰
We had a freak weather event here on Friday. Rain at a rate greater than 4 inches per hour. Overwhelmed my massive curtain drain for the first time in 20 years since installation. Water was 3 inches up the basement windows but thankfully didn't blow out the windows. Stood in 8 inches of water pumping out the basement window wells. My neighbor had his basement windows blown out and his finished basement flooded AND his pool full of muddy water. Have a whole pile of lithium batteries on the basement floor for the solar system - that was my first concern. Also have an unpopulated rack with servers on the floor waiting to be installed.
Sorry about your loss Linus, I wish you and your family well.
Thin sheet of plastic between machines can help direct the water off the machines below!
My condolences Linus
You can buy a flow rate measurement device and a couple solenoid valves, hook up the flow rate detector to all the valves with an independent system like an arduino, and if flow rate spikes, it can send out alarms and close the solenoid valves. Could isolate all the different water cooling loops in the machines, kill the power and keep this from happening again.
My condolences.
Damn Linus, super sad about your sister. I am very sorry for your loss. You talked about it as if it was common knowledge, you share so much about your life yet you didnt say anything about your sister, you could have had so many messages of support towards or family, so many prayers going towards them, you will have them now... I am so sorry
5:49 now thats what i call open loop water cooling
Sorry to hear about your loss, Linus.
I wish your family the best
❤
My condolences to you and your family in regards to your sister.
Would it be a good idea to rearrange the stuff in the rack so the water cooled systems are at the bottom, so if the next leak happens less systems take water?
Also how about water sensors?
One on the floor might be a late signal but better than none but maybe there is stuff you can put in or between the systems?
Linus I lost my little brother 2 years ago I know what you are going through this is something you will never get over the emotions will rush once in a while ( I’m tearing up thinking about it) I’m so sorry you have to deal with this.
Sorry for your loss and the incident.
i have no doubt the time she had was enriched with the moments you had together
Maybe look at 3d printed or alternative method but have like a shelf in between each system thats angled either forwards or backwards so any water leak won't hit a system below it?
Condolences to you and your family
Linus, I’m so sorry for your loss
Sorry for your loss Linus.
You need some sort of static pressure sensor, inlet vs outlet so if you see a dip in water pressure, the pump shuts down instantly. maybe even shut down the machines...
Really sorry about your sister. Very HAPPY about your Server survival
Condolences, Linus and family. Sorry to hear the passing of your sister. 😞
Sorry to hear about your sister, praying for your family.
Oh man, that's rough, sorry about your loss Linus.