My 48 Drives may have failed because of this..
ฝัง
- เผยแพร่เมื่อ 28 ต.ค. 2023
- Still working on the entire breakdown and diag of the inner power system.
🔥These links support my madness🔥
💲 NordVPN: www.bmbsucks.com
💲 Members-Only Punishment: bit.ly/bmbmembers
💯These are my other Socials💯
🔹Twitter: bit.ly/BMBTw
🔹Instagram: bit.ly/bmbinstagram
🔹Tik-Tok: bit.ly/bmbtock
🔹Discord: bit.ly/BMBDISCORD
✉️P.O. Box for Mail✉️
Byte My Bits
P.O. Box 77
Haysville, KS 67060 - วิทยาศาสตร์และเทคโนโลยี
If a server with any amount of important drives is doing this, unplug them first so you can diagnose without your drives potentially spinning up and down every time this thing shuts off and restarts. Please up vote this because its the first danged thing to do.
You are right of course, but having watched Jason for several years, he's not the sharpest knife in the drawer when it comes to this kind of stuff. Quite often, he is the maker of his own problems. I guess we'll have to wait and see.
This. It was such a pain seeing those drives being boot up, spun down, spin up, down, up down updownupdown SOOOO much times.
Do that a couple of hours and you will definitely shave some lifetime off your hdd motors...
How the drives weren't his first and only priority in such situation is mind boggling.
The overly complex design of everything you do constantly triggers me, yet I continue to come back and watch. I'm not sure why, but you have found something interesting to watch that is really hard to define. My best parallel is how people cannot take their eyes off a trainwreck. That being said, thanks for your content.
Yeah, as in, in a 4 or 5U chassis you have room for a giant air cooler, do you really need water cooling?
@@NateTheBrewer He wanted it hehe
Yes, he wanted it, but he still hasn't figured out that water and electronics never mix, and as I've said before, he's often the maker of his own problems. If you would just spend the day or two cleaning his shit up, and organizing his shit, he'd be in a far better place to properly troubleshoot this to begin with. He has the ability to do it, but he just won't do it.
@@Verminator151029 Definitely not the way I would handle things, but I'm entertained by his channel, so...
Go outside touch the grass, go get something to eat, go to the movie theater and watch a movie, then come back to the problem. Strip down the server, inspect, and test the core components, then go adding and testing as you put it back together. It could be something like the CPU needs to be reseated, you never know.
first.... pull the drives and test them in a different machine. See if they are recognized by the bios when plugged directly into the sata header.
Next, test your HBA in a different computer and verify it is recognized and accessible. plug in a drive to each of the ports on the HBA directly. Verify the new computer sees the drive.
It may still be a power issue, but pulling all of that and verifying you have not lost those will ease tension and stress while you find the issue.
Unplug the molex to 4 pin that is melted. I keep seeing it still plugged in while booting and it’s probably shorting 12v to ground.
I thought the melted fan header cable was coming from the motherboard for PWM signal to adjust for the CPU temps. Was it a molex to fan adapter?
@@Elemental-IT th-cam.com/video/r7KgxhRIppY/w-d-xo.html is the point in the video where he shows power in from molex to the fan hub/power splitter.
edit: you can still see it plugged in at 15:30 th-cam.com/video/r7KgxhRIppY/w-d-xo.html
I can only imagine that when he was in school his report cards read something like, "Jason has the ability to do anything, but he just needs to apply himself and think more before he jumps". Knowing that a connector is melting due to some problem and then just plugging it back in anyway without investigating it further is exactly what I'm talking about.
Good luck brother. Just give in and do a full dis-assembly and inspect everything meticulously one by one. Complicated redundancy leads to redundant headaches. Best of luck to you!
Why not take drive out and see if it shows up on a different computer.
i seriously doubt you shorted the drives... but it probably was not good with them spinning up and down every few seconds... I would take a random drive out and drive to connect it to another system to see if the drives are ok or not... at least see if you see it in the bios..
Yeah I said the same thing, idk if windows can read the files on the drives but the bios should indeed show the drive.
I had the same issue on 2 of my drives in Unraid a few days ago, after I powered off the machine, when Electricians came by. I was sad.. but turned it off and just pushed the cables in more.. and they randomly showed.
jeezz ive seen this before, the distrib board was messing up due to condensation... yes you read that right! plus you got a shock (or looked like it!) when you took the top off making me think something is shorting to the case!
Like others have said dismantle the server.
The data might be recovered depending on what filesystem you are using in UNRAID. I know for certain and I tested it. If using XFS, the data can be accessible on another Linux computer. The data will be scattered over the drives designated as data drives. However, if the drive is mounted as read-only and then brought over to an array in UNRAID, UNRAID will have to do a parity check.
I use BGears b-Blaster (also known as b-PWM) fans because of high CFM and high air pressure.
You may want to remove the PWM connector that feeds power for the fan bus board. Then in the place of that connector wire up at least 18 AWG wire or thicker wire to a Molex connector. The PWM connector is spec at 1 ampere per pin. Those high-speed fans will burn that connector.
A multimeter is good to use if you know the ATX voltage ranges. Sorry, I do not know them, so I suggest searching on Intel’s website.
Good Luck
Good luck man! Yikes I feel it in my soul.
I love the fact you forgot what hardware you have. Really enjoy watching you tinker with things... keep it up!
All these videos have that very specific crackhead server maintenance feel to them. Maybe it's the late nights, maybe it's frantic pace or the ever increasing complexity. All I know is that I don't want to see it, but can't look away.
I had a similar issue happen to me about a month ago. Like accidentally used a Corsair modular SATA power cable on an Asus power supply to connect some older Western Digital Red NAS drives (4TB and 6TB). Thankfully it was only three of them and the lower capacity ones and it was before I had any data on them but yea, what a gut punch. Hopefully you can get it back up and running and you have a Plex buddy you can stream from until you get things back up.
Check the water block is it hot? is the pump working? looks very similar to a problem I had... my pump stopped and the fans went full speed melting a connector like yours. if I left it off for a long time it would boot and then boot loop when the CPU gets to HOT
Multi-meter would come in handy to find out if you have power you need.
I wonder how cold the ambient needs to be for the inside of a busy server to be lower than dew point. Normally, whatever in the room is coldest will collect condensation first. That could contribute to an idea to drain water from the local atmosphere.
More than likely, the drives are good. Replace the back planes, swap out the power distro board, then boot without the storage drives to insure it will boot bare. Then, and only then put the drives back in line.
Yeah.... This is terrifying! I don't think the drives are fried, but hoping you can figure it out!
This is my nightmare. And I only have 5 drives (2 parity) in my server. Good luck man!
Wishing you a speedy and inexpensive *FULL* data recovery!
Teardown time approaches!
I know it sucks big time but that's the fun part of troubleshooting is figuring out WTF happened. Once you get this up and running probably throw in a couple of ESP32 environment sensors to warn you of rapid drop in temps and humidity which would lead to the condensation inside the servers.
Sounds like ya need a beer! I'm hoping you didn't loose the drives Jason!
Maybe I’m missing something but when you plugged in the consumer power supply I didn’t notice you actually turn on the computer. You turned on the switch for the power supply, but I didn’t see you short any pins on the mother board or press a power button on the case. But maybe you just didn’t show it, or maybe you did and I missed it.
I think the board is set to autostart after power loss. That's how all of my servers are set.
@@Elemental-IT I genuinely didn’t know that was an option on consumer motherboards!
When troubleshooting i keep it simple, unplug EVERYTHING apart from the essentials, so mobo, cpu fan and ram, then boot it, if it still DOES NOT boot then try doing a binary split on the RAM, if still no go then you are into PSU or CPU or Motherboard replacement, if it DOES boot then reconnect using binary split method, carry on doing this until you find the issue.
I love watching your content Jason, it gives a warm feeling about my well constructed server and makes me appreciate what i've got, i do hope you haven't fried your drives, fingers crossed for ya
Yup, he makes everything infinitely more difficult by not doing the obvious things first. He literally Works backwards from everyone else when trying to troubleshoot this.
@@Verminator151029 I fear that when he connected that secondary PSU for testing that is when the damage to the drives was done, although it could of been sooner of course, there was definitely some sort of electrical issue as shown by the slightly melted connector!. It is easy of course to say what to do after the fact and maybe had i been in this situation I would panic and do the wrong thing, 48 drives potentially fried though, that is terrifying!.
if I was Jason the next thing to do is take a drive out and connect it to a PC and see if you could read the data (no stripe), if so breath easier, if not...well you are into data recovery.
so from what i recall most motherboard and fan accessory headers and wiring, etc., is only rated for 1A or 12W max and anything beyond that can cause melting, burning, wrecking the motherboard, etc., and a lot of those big server fans will run 2A+ or 25W+ so regardless of all this I would NOT be powering those bigass fans from that lil tiny plug or that itty bitty fan controller....fans are one area where server/enterprise gear does NOT just swap in to consumer applications generally speaking...that said though I'd be checking the RAM since the mobo didn't pass the RAM test though the PSU could certainly cause that as well.....unhook all those fans, and test, remove RAM and test, unhook everything then start plugging in one thing at a time til it doesn't POST and then root cause that thing
P.S. I'm glad I'm not the only one with a rat's nest for wiring in my server rack :)
well... did you get it sorted?
It sounds like you need a power supply tester to ensure you have consistent voltage on all the rails and whatever method you are using for a hard drive controller may be at fault hence the lost of drives.
Exactly right, and this is the kind of stupid shit that I'm talking about that causes him more problems than anything else. I'm sure he must have a multimeter arpund there somewhere, but instead of using it first, he tries everything else under the sun and takes hours to do it, and still he doesn't know if the power supplies are providing the right power or if they're even the problem, but hey, why check that out first with a multimeter when you can reboot the system 80 times with all the drives connected and hope that nothing bad happens?
@Byte My Bits This is an example of what happens when you are the king of jank! More Jank more problems lol! I get it, you just want it to work! Well how about doing it right the first time to have it work extremely well? Jason, I love your channel 100% but can I ask from you this one question? Can you find it in your heart to just do it the right way every time? Just making it work, and jank should not be the outcome if you feel me. One love forever and always. FYI, Unraid/Raid is not a backup
That setup is way too complicated. Get a used Supermicro chassis, toss in your choice of mobo/cpu and hba and be done with it.
Don't buy consumer gear for a server... Supermicro boards are plentiful and cheap, and while you pay a premium for Noctua fans, they are worth it. I too had a consumer board fry a fan header on my gaming rig and the entire board... $500... gone.
Big oof, good luck bro!
Yeah im sure its been said by now. Dont use a splitter like that for those high amperage fans.
Heart breaking
Ouch 😢
Jason...
STOP BREAKING YO SHIT!!!
Always, Always check your Flux Capacitor first, Always! 🙂
Interesting.
PogChamp
I would've unplugged all the disks before I go through the process of elimination.
3:29 I think its mem sticks as you're not getting anything on the screen but I'm not there I can't really say but i have had this happen to my server and changed the mem sticks. 13:20 well not mem wow crazy cant saw what it is crazy hope you cant find the problem.
Use gear rated for servers if your running 24/7
Hi Jason I think you should rename this server if you keep it Cathulu.
id be leaning towards a motherboad/ cpu problem, as said by others unplug all your drives and try and boot just mobo.
you are unorganized and in state of panic from all appearances - you need to have better backups and redundancy but also the ability to just walk calmly away from the chaos for an extended period of time - diversify your compute
spend 10's of thousands more dollars to have another 500TB of storage.
backup and redundancy only help with data. The hardware costs if all of those drives are cooked is easily 5 figures. Not to mention the time investment in getting your main server set up and working correctly. Either you don't have a server of this size / complexity, or you have all the moneys and times..... because my main server (which is just as expensive and complex - if not more) IS completely backed up and I have felt this same panic on more than one occasion. I have 6 separate servers backing up my main server... but they are backup for a reason.
@@Elemental-IT go big or go home to your hell hole or go smaller - if you can't maintain what you have that is a problem and you need to address it but continue to try and grow later - with a bit more manageability and availability - a mistake happened but it may have been a good thing if you can learn and grow from it. This is a really good lesson and case study for small biz or anybody that has mission critical data - always have 2-3 nas, don't rely on just 1 - everybody need to have high availability and failover and redundancy and also to test it regularly #allocation of resources #mission critical data #systems #backups #tabletop exercises
I'm just here thinking, maybe don't do so hazardous DIY things with a server that holds this much important data to you. But really, good luck, let's hope it's just a controller or backplane or something simple and not the drives. The next steps are just to test each thing one at a time to rule out what causes the problem.
What a mess !!!
pull melted cable IMMEADIATLY
You clearly drain too much power. you have to rework your power delivery. Could be the ram too.
Ok, check your backplanes.
If you use truenas, ZFS will probably save your butt on the data side of things, and you can export your config on a second computer and plug the drives on it.
Its unraid, take your USB stick put it in anyother computer take one drive from Loki and plug it into that PC. 5 minutes you would know.
i'd wager your HBA got shot. not responsible for the bootloop of course, but likely responsible for half and all drives not showing up.
He´s crashing because you speak about him like " it", you´ve hurt his feelings...so he killed himself.
Amateur hour
45 bay sucks just buy 4 , 24 bay supermicros and run multiple servers.
You don't even need to do that. They make enclosures that hold a crap load of drives. And if one wants to stick with the drives only accessible from the front, one can just use them as JBOD enclosures.
But I agree. If one wants to run stuff 24/7, they need to stop using gamer stuff and pony up for stuff that was designed to run 24/7. He has so much stuff lying around unused that he could sell off to just build something that will last years, instead of reconfiguring every few months with extra bits.
my 24 bay unraid with 2 disk parity , so 22 drives are at 408TB
my other unraid 24 bay is at 350TB
my true nas scale will soon be filled with 18s and will have 3 vdevs with 2 disk parity
all using supermicro 24 bay and they are solid except one bad sas cable that caused issues but was stock and old.
All the back planes and cases are easily swapped, just buy a few racks and a some consumer ups and you are done.
@@timramich
Haha codes. Bloody youngsters can't count beeps.
This sort of meltymelt on a crimp connector is quite familiar to me, it's not water. It is surface oxide, but see it happens at elevated temperature as well. Some time the connector is just barely warm but it's already degrading, and then you have a runaway process where it'll just keep heating more and keep getting worse. It can go from pretty OK seeming to an utter fire hazard in weeks. What's also not helping is that the heat presses the punched sheet structure of the female connector apart, as the heat expands, which is another contributor to connector conductivity dropping.
Gold plated connectors can actually prevent this degradation mode.
Do keep in mind that the integrated controllers in the fans sometimes fail short. Need to go start yeeting stuff. That unfortunately goes for everything, everything can fail.
First, S**k it Nate.