TrueNAS 12: Replacing Failed Drives
ฝัง
- เผยแพร่เมื่อ 8 ก.ย. 2024
- Connecting With Us
---------------------------------------------------
+ Hire Us For A Project: lawrencesystem...
+ Tom Twitter 🐦 / tomlawrencetech
+ Our Web Site www.lawrencesy...
+ Our Forums forums.lawrenc...
+ Instagram / lawrencesystems
+ Facebook / lawrencesystems
+ GitHub github.com/law...
+ Discord / discord
Lawrence Systems Shirts and Swag
---------------------------------------------------
►👕 lawrence.video...
AFFILIATES & REFERRAL LINKS
---------------------------------------------------
Amazon Affiliate Store
🛒 www.amazon.com...
UniFi Affiliate Link
🛒 store.ui.com?a_aid=LTS
All Of Our Affiliates that help us out and can get you discounts!
🛒 lawrencesystem...
Gear we use on Kit
🛒 kit.co/lawrenc...
Use OfferCode LTSERVICES to get 5% off your order at
🛒 lawrence.video...
Digital Ocean Offer Code
🛒 m.do.co/c/85de...
HostiFi UniFi Cloud Hosting Service
🛒 hostifi.net/?v...
Protect you privacy with a VPN from Private Internet Access
🛒 www.privateint...
Patreon
💰 / lawrencesystems
#TrueNAS #NAS
![POLYCAT - ข้อความรูปยิ้ม [Music Video]](http://i.ytimg.com/vi/lBhIgPGCgbI/mqdefault.jpg)
FreeNAS ZFS VDEV Pool Design Explained: RAIDZ RAIDZ2 RAIDZ3 Capacity, Integrity, and Performance.
th-cam.com/video/-AnkHc7N0zM/w-d-xo.html
How to Build & Extend FreeNAS ZFS Pools with VDEVs / Adding Drives to an Existing FreeNAS
th-cam.com/video/76qnBjZF65g/w-d-xo.html
Extending a NAS on ZFS isn't a ZFS issue it's a TrueNAS issue. Extending the RAID on unRAID which uses ZFS is quite easy.
Worth noting: if you replace all disks progressively with larger disks, you can expand the array to the extra space once the array is stable on the larger disks.
thank you for this comment
one at a time like unraid or can you do multiples and let the parity drive recreate the data?
@@cosmickatamari one at a time only
Does it expand it automatically once you've systematically replaced all the disks one at a time? Thanks
Thanks TOM. Keep up with the TrueNAS 12 Core Video Series! We the community TRULY appreciate it.
whenever I have an issue with TrueNas I can always find what I need with Lawrence Systems TH-cam videos... Thank Tom!
Thomas Lawrence helping out the TrueNAS Community one video at a time!
Tom, I really enjoy your videos on TrueNAS. You might tell users to label the back of their drives with the serial number. As a hobbyist who's used TrueNAS a long time, this is essential if you mount the drives close together!
Also worth noting (for future readers) is that if you originally imported your ZFS pool from a non-Truenas/Freenas system and you are replacing like for like you may need to do the drive replacement manually from the shell. The reason is the GUI "replace drive" code always creates a swap partition on the first few drives and then tries to create a ZFS data partition. If your pool was created on another system it probably wont have these swap partitons, so if you are replacing a failed drive with the same sized drive it will abort with "not enough space" after creating the swap.
I know this is a fairly old video - but you helped me recover a ZFS2 array out of a failed system *with* a bad drive out of the 4 - and now it's recovering because I knew that the "unavailable" drive (which was no longer installed) had to be "replace" with the new one installed in it's place. Top job!
Maybe I missed it, but I didn't hear you mention that you can replace the drives in a vdev by larger ones if you do it one-by-one. After that, you can easily extend the filesystem. This solves some problems for people with no spare SATA ports that need to upgrade their storage capacity.
Thank you for your videos. I am a complete Noob working with TrueNAS, I had one of my HD power connector caught on fire. It was a cheap power connector. I have never had on in the past 30 years do this however, there is a first for everything. So the fire damaged the drive and had to be replaced. You made it very easy for me to replace the disk, and get back online. Thanks
These kind of videos are really useful, even if the process is pretty straightforward. Taking the edge off the panic ;)
Absolutely agree!
Useful thanks! FreeNAS box failed with Seagate Ironwolf 4TB. While waiting for a replacement, a second disc had issues! Rebuilding the first disc quickly before replacing a second! Thanks!
wow, this is much more simple than I thought it would be
Thanks Tom I'm new to truenas,My experimental HP N54L system indicated a problem and after a bit of head scratching ,kludging my way through to find the failing drive I couldn't work out how to replace the pool drive. This explanation really worked for me much appreciated, onwards and upwards.
Literally the same day I have a degraded drive on my TrueNas. Thank you 🙏.
Lil ironic for a lad named backup plan
Nice informative tutorial. To echo what others have said here, thank you, and please keep this really invaluable series alive.
note that you can tell zfs to replace the missing drive automatically with a drive that is connected to the same location that the missing drive was in, but this option is not enabled by default.
also, that you can modify vdevs directly (add/remove/attach/detach) for mirror/stripe vdev only pools, but not any pool with a raidz vdev of any type.
Two for one! Awesome! And just in time! I had a problem where all my disks show healthy and smart shows no errors but every time I kept getting small checksum errors on ALL drives at the same time (and the same count), even after I switched the HAB. It was so frustrating for the past few days that I just nuked the pool I had and redid everything from scratch. Im not 100 certain my problem went away. Im waiting on new SAS cables to arrive today.
Thanks heaps ! couldn't be more precise and elaborative :) awesome video
so helpful im having problem with a drive that is lil older then year old and just got new one to replace it but will be sending it in for a replacement hopefully WD honor's the 3 year warranty. Wasn't sure how to do it
Thanks! not to many videos about this subject!!
Another great video Tom! Thanks for always sharing with us!💖👌👍😎JP
thanks for this video - its a pretty easy procedure when you know how. thank you for showing me how!
just finished setting up replication tasks for my (ULTRA JANK) FreeNAS lab so boi does this video give me confidence...
Thank you. All the info i needed :)
This is EXACTLY what I was looking for. Thank You
amazing video. Useful information, no bs.
Thank you for the video! Have one question. If there are lots of data in the pool, it may take quite a while for the re-silvering process to complete. What's the best way to deal with possible power outage especially at home? Basically, can the re-silvering process be interrupted (due to power outage) and resumed later? Thank you.
Yes, If disrupted it will resume
Great video
Great video! Thank you!
will it default to the lowest size drive in the pool, or would you have to manually make it bigger? like if you replaced a 320gb drive with the 1tb like you said, then eventually all the drives got replaced by 1tb or larger, would the pool become bigger, or still be stuck at 320gb ?
Thank you for this video. I am attempting to run TrueNAS on an older HP system that uses a raid controller. We needed to change the controller to HBA mode in order to see the 8 drives in TrueNAS. However, when I run the 'loose cable' scenario by pulling out the drive, when I plug the drive back in it doesn't go back online, I cannot 'online' the drive using the GUI, and I cannot 'replace' the drive either.
If I put in a new drive it sees the new drive and allows the replacement of the drive.
Not sure if there is some sort of default setting that needs to be changed, or if I need to re-format the drive before putting it back in, or if there is a manual way to dismount the drive. We are using 1 zfs pool.
If it says one drive failed, then how doI know which one to take out?
Really funny turn of events. About one hour after I watched this video my ssd acting as cache gave up. I replaced it and move to UI with replace command and fail :)
I have done this thru shell before you have to dd /dev/zero to begining of disk and do replace then, but was lazy and just removed the cache and add it again from the "Add Vdevs to pool". TrueNAS Core 12.0-U1.1
Just replaced a 10TB drive that failed (seagate sent a replacement asap, thanks seagate) and your instructions took the panic of doing it wrong out. Drive is replaced and will be scanning for....a while.
I have built everything recomended and all pretty good trunas and Pfsense
I am enjoying you detail in removing and adding drives in this video, I had a quick question on what is the host you are using to run TtrueNAS . This looks like it might be a SuperMicro? I am looking to build a small TrueNAS server and might want to get a SFF server for that. Thanks Jim
Thanks for the video
The little Atom that could.
Hasn't died yet!
@@LAWRENCESYSTEMS mine is working fine as pfsense, ran as a shoutcast server for two years before that.
I have had issues with "Force" not always working and had to resort to a Erase Track Zero utility to replace the drive.
My drive supplier puts a logo label on them (covering the serials) which it makes finding the failed drive a nightmare... I have a system running Plex & Nextcloud. Drive da11p2 shows errors. I have a 14 bay system... I could just plug a new drive into bay 12 & substitute it for da11p2. But still wonder if there is a way using the lights to ID the drive that is replaced? Using a DL380eG8
What is the link to the video that explains more about the different raids and ZFS I'm newer to the nas deal so most of my file systems have been ntfs and I use cloning systems, but now looking for larger amounts of space so I need to build something that is redundant. Just trying to learn how to properly make myself a good solid home nas and learn how to properly back up and restore it. ( Great video though man ) :) ps o found your pinned video
Muchas gracias
2:16 - Very odd, so I have a brand new WD Red NAS Drive (4TB), that does exactly this once it was moved to a new slot, ( installed nvme which disabled one onboard sata controller)
Installed into an pci sata expansion card, but now experiencing the same thing, drive will come and go after a reboot, it is a brand new drive, it doesn't have any sign of corruption. (every file is intact & no smart errors)
However truenas seems to have the sata device detach if it goes without use for a long period of time.
For instance, if i boot up my truenas VM (prox mox vm with passthrough) i can begin a file transfer almost immediately once started, no hitches, no errors.
But once the drive is not in use it is detached from truenas it seems.
nice video. Thanks.
But I have qustion, I have 8 HDD running RAIDZ2, I found 2 or 3 drive has badblocks, can I replace 2 or 3 drive at same time?
I had a question about replacing a failed/failing drive but the drives in my home built TrueNas Scale are not hot swapable, how do I replace a failed drive?
Do I need to shutdown the system, replace the failed drive and select replace from the trueNas Scale menus?
Good video
How about showing what happens when the motherboard or the OS drive fails?
Got a video on that here th-cam.com/video/_DmMpETyBsY/w-d-xo.html
OK one question. every so often I will get a degraded state that one drive has failed. I can restart and everything thing is back to normal. I either have a cable that is not as good as it should be or I do have a drive that is you know getting ready to croak. I have the smart turned on ect, But I never get what drive caused the degraded state. because I never get to catch witch drive went off line in the first place. were can you go to see a log of what drives or drive could be causing the degraded state so I can go to the source of the troubled drive and investigate. further.
Can you have two m.2 drives via pci-e adapter in raid 1 as your boot pool? It would really save up on those precious sata ports.
Hi! Nice video! How exactly does that work if the pool is encrypted? I’ve read that you have to renew the key while resilvering? After resilvering? Before a reboot? Otherwise you might loose access to the pool? Last time I checked the FreeNAS/TrueNAS documentation I found that the instructions were not 100% clear. Does anyone know the exact steps? Did they change from old FreeNAS encryption to new ZFS native encryption?
So, if the four drives are all 320GB, one fails and is replaced with a 500GB drive, Vdev and data is rebuild, all is good!
The week after a 2nd drive fails, same thing happens, replaced with a 500GB drive, rebuild, all good again! Then the 3rd drive, and finally the 4th drive!
Now they're all equal size again. Would that mean the Vdev could be expanded to include the 4 x 180GB of otherwise wasted space?
Hey thanks for sharing! I was trying the same thing without sucess, only difference between your and my setup is, that my pool uses the new native encryption. is there a different methode for encrypted pools? would be really nice if you can help.
no difference with encrypted pool.
Hi? I was trying exactly (almost) the same what you did: replace a disk. Actually the original disk was start counting some errors. I took another exactly the same disk. Everything was deleted from that "new" disk and killed any partition. So, raw empty disk attached. The problem was that "Replace" command didn't work since there was no option to choose from at "Member disk" position. I saw no other option then restart whole box. Disks syncing was done fast during the boot, so everything ended OK.
Do you have idea what may cause that difference? (Probably motherboard bios specs)
This is experimental and educational box just for exercise those operations. Installed is th latest TrueNAS 12./8U
Tom. Have you ever tried using a USB HDD in a FreeNas system? Say as an internal backup target, or otherwise?
USB drives are notoriously flaky, but they will show up (assuming they work at all) like any other drive.
Is truenas suitable for enterprise high load system? like 1000 clients streaming movies?
yes
i have a 5 disk TrueNas 12 that consist of 4x2TB and 1x4TB in a raidz2
i now have 4 additional 4TB drives and would like to swap the 2TB's with them.
Since raaidz2 has a failure resilience for 2 drives, can i replace 2 drives at a time between resilvers?
Or is it better to still do it one drive at a time? if so, why?
Either way works fine. But you have a higher risk to loose all your data in case you replace two drives in one go because if a third drive fails during resilvering all data is gone forever.
From TrueNAS documentation:
"When considering the number of disks to use per vdev, consider the size of the disks and the amount of time required for resilvering, which is the process of rebuilding the vdev. The larger the size of the vdev, the longer the resilvering time. When replacing a disk in a RAIDZ, it is possible that another disk will fail before the resilvering process completes. If the number of failed disks exceeds the number allowed per vdev for the type of RAIDZ, the data in the pool will be lost. For this reason, RAIDZ1 is not recommended for drives over 1 TiB in size."
how do you replace a drive in cli
Maybe I'm missing something but how do you identify which physical drive is the one going bad.
Check the serial numbers.
At 5:12 by which clue did you assume that it was ada0?
PS:Was ada0 the new disk or the previous one?
New edit: Ok under RaidZ2 it labels disks as ada0,ada1,ada4,ada5 but why this weird sequence? From the example it seems that
ada0 remains the same for the old and new drive but causes ZFS to name it ada0 since it is cable independent?
ada0 represents the physical location but as I said in the video, it's not directly relevant to ZFS.
"which clue did you assume that it was ada0?"
if you go under Storage > Disks, the table there includes both the disk id and the serial number which you can match with whatever is printed on the physical drive when you look at it.
Checksum errors are almost always cable or contact issues.
Yup and I have seen it often when people virtualize it.
Can you do a clearos firewall review?
I don't use it so not likely
I run FreeNAS 12 as a VM under ProMox with the LSA SAS 9211-8i controller in "pass thru" mode. Can you elaborate or otherwise guide thru the same process to replace a failed drive in this scenario PLEASE?
I don't ever recommend virtualization of TrueNAS but if you are passing the controller through I am not sure what the difference would be in the process.
@@LAWRENCESYSTEMS Thanks for the reply! I have pretty much come to the same conclusion but my "superiors" insist on it! I tried it before and came into problems because of the difficulty in the replacement drives showing up as they need to have "pass thru" entries in the config file. Not too difficult but not trivial either. So the best info you can prove is "why you do not recommend visualizing TrueNAS". Thanks for anything you can provide.
@@LAWRENCESYSTEMS Hello again. I discovered at least one reason NOT to run TN under ProxMox. It is because you cannot mount or otherwise import a drive that has any zfs partitions. So while you can go through the process of "adding a (raw) physical drive to ProxMox" and carry on to create a RAID configuration using zfs, you cannot import a drive that already has a zfs partition. This effectively means you cannot use a three drive collection to use the replacing method to keep an offline copy of a RAID 0 set. I have done this as an easy way to keep a reliable backup by rotating in / out a third drive and just re-slivering the imported drive, which should have minimal data deltas. Sorry if this is out of scope for this post.
If I replace all the drives one by one with a larger capacity, can I expand the storage then?
I have not tried but I think that works
doing this poses but suddenly the pool now shows as offline and EXPORT/DISCONNECT. What the hell happend and how do I recover from this. there ar 4 good drives with data.
so how about if I want to replace stripe disk?🤔
Nope
I have a question, if I take out a failed disk and format it on my mac, then do a first aid. mac says the drive is fine. Is there a better way to inspect a hdd?
Test it under load by doing lots of reads and writes
@@LAWRENCESYSTEMS what would be the best program to test that with?
In Linux there is a tool called FIO
Is there a way to know which drive actually failed? If you have 24+ drives on the server, how does one know which drive failed?
By looking at the serial numbers on the drives compared to the serial number of the one that failed.
@@LAWRENCESYSTEMS oh thank you. Do you typically make a note of every serial number before hand so u can track their location?
For many drives they are already printed on the drive opposite the power & sata connector side.
What if my boot drive failed ??
Only thing missing is how to identify a bad drive physically to remove
if you go to Storage > Disks you can match up the disk id with the serial numbers. note which failed id has what serial number, shut down the system, and start pulling drives till you find the offender(s).
Re-Silvering ?? what is that?
the process of remirroring or rebuilding a RAID drive
@@LAWRENCESYSTEMS I have never heard that term before. Rebuilding yes, but not re-silvering.
Hopefully it's not 'drives'. Nothing worse then having to replace 'drives'. One drive at a time is pain enough.
slow down in talking
You can adjust youtube video speed to .5x.
I forgot to set the failed disk to OFFLINE on FreeNAS-11.2-U8. I just shutdown the system then put the new disk on it. Now the /dev/gptid/db0bf12d-d94a-11e9-98d1-40b076912f18 shows UNAVAIL. Can I directly choose REPLACE to ada3 like you shown in this video?
U sir are a hero! Thanks for sharing all the great knowledge..