Mastering ZFS: Easily Auto Snapshot & Replicate a ZFS Dataset - Part A
ฝัง
- เผยแพร่เมื่อ 22 ส.ค. 2023
- Effortlessly set up automatic snapshots and replicate a ZFS dataset on your Unraid server with the provided script. This not only facilitates quick autosnapshot configurations but also allows you to replicate (or backup) your data to another dataset, be it locally or over a network or the internet. If you don't have another ZFS location for backups, don't worry. The script can manage that as well. Simply toggle to rsync replication, and your dataset will be snapshot-taken and incrementally replicated using rsync to any desired location
Please, if you can and want to support the channel and donate you can do so by Paypal here goo.gl/dw6MLW or check my patreon page / spaceinvaderone
Bitcoin donations 3LMxDzcwPdjXQmmeBPzfvUgjYFDTqDAgQF
Snapshot and Replication script
github.com/SpaceinvaderOne/Un...
Beep speaker
Amazon USA - amzn.to/3swqv6Y
Amazon UK - amzn.to/45qFrlR
----------------------------------------------------------------------------------------------------------------
Need a VPN?
PIA is popular with Unraid users as its easy to setup with various vpn download containers - www.privateinternetaccess.com...
Torguard is also an excellent VPN again with both openvpn and wireguard protocls supported.
Get 50% off for life using code spaceinvaderone torguard.net/aff.php?aff=6005
----------------------------------------------------------------------------------------------------------------
Need a cheap windows 10 license for around $10
consogame.com/software/window...
----------------------------------------------------------------------------------------------------------------
Need to buy something from amazon? Then please use my link to help the channel :)
USA - amzn.to/3kCikfU
UK - amzn.to/2UsYb1f
USA link - USB HDD Docking station amzn.to/3v754WG
UK Link - USB HDD Docking station amzn.to/3hLenYp
HighPoint RocketStor 6414S amzn.to/3fiXv9s USA
Mini SAS 26-Pin SFF-8088 Male to Mini SAS 26-Pin SFF-8088
amzn.to/2V4x9kT USA
amzn.to/3xfkxEl UK
----------------------------------------------------------------------------------------------------------------
A big thankyou to Limetechnology for all the great work that they put into always improving Unraid OS. - วิทยาศาสตร์และเทคโนโลยี
Thank you so much for this and all of your hard work. It is truly appreciated!
You are so welcome!
Yes!!! I've been looking forward to this video.
Thank you for your anticipation and patience! I wanted to ensure the script was refined and polished to perfection before releasing the video. I truly hope you find it worth the wait and enjoy the content! 🚀
I just love your videos sir! Easy to follow, very structured and to the point - each and every one. I have just followed your latest series on ZFS and have updated my cache pools to ZFS, installed and scheduled your scripts and even gotten replication to work against an RPI I have in a corner in the garage. Works perfectly! :) I do have one question: If one, for any particular reason, should loose the disk pool that holds the snapshots how does one recover from the off-synced snapshots? It might be clear as a day to others, but not to me. :) Thanks again.
Mirrored app cache drives, replicated to the array including snapshots thanks to this video, and backup up to the cloud. If I somehow STILL lose my appdata, something really has gone wrong, haha!
Definetly. If you you loose your appdata now you were just never meant to have it lol ;)
With the replicated snapshots and the backup to the cloud, do I still need to mirror my cache drives? Or can I get the extra space back?
Hi @icenode , how did you solve "and backup up to the cloud"?
@@user-vk9in7qg4b I went with Backblaze B2 for inexpensive online storage and the Duplicacy Docker to handle the synching, encrypting and restoring. Been working well for me for a year now.
what if you want to snapshot multiple datasets in the same source pool? comma separate them?
Thank you for finishing this multipart ZFS series for this version of UnRaid. The one week anticipated time-frame for its release came and went long ago.Your videos explain a lot, are well done and appreciated. I watched this video can see why it was delayed and realize how much work went into the creation. I can now set up the new server which has more drive slots and I can max out my drive limit. I used to do Microsoft Tech Support before it moved to India and am still learning new things. Excuse my impatience but if I was a food item my Sell By Date expired years ago and my Expiration Date is faded badly enough that it can't be read.
Very useful video can't wait trying ZFS on my UNRAID.
Thanks! Your videos are incredibly helpful and I appreciate all the time and effort you put into it
I've converted all my array disks to ZFS. Does it make sense to now convert folders on the array to datasets? Does it make sense to convert especially important array folders, all array folders /shares, or none at all?
And if it does make sense to do so, can I use your dataset creation script to do it?
@@905jayEd goes through the difference between folders and datasets and the benefits of datasets in his recent video ZFS Essentials: Auto-Converting Folders to Datasets on Unraid: th-cam.com/video/pLFaDnTVpuM/w-d-xo.html
If you're a beginner with ZFS (as I am) it's worth having a watch through the whole ZFS playlist: th-cam.com/play/PL6MCtOroZNDCr7TgKEgYDD_5WPV75M2If.html
Thankyou so much for your tip :)
There it is! Awesome. Been refreshing your channel constantly watching for this video!!
Thanks for your patience :)
How did you get the "Main" tab separated into tabs for Array Devices, Pool Devices, etc? I like this so much better than having to scroll up & down.
Go to Settings->Display Settings and change Page view to tabbed.
Great video Ed! Thank you for always sharing with us!💖👍😎JP
My pleasure!!
@@SpaceinvaderOne 😎
Awesome tutorial! Thank you!
You're very welcome!
Thanks for this great series. I finally came around setting everything up and it works like a dream. The V2 of the Auto Conversion script is also really great, especially the skip when the destination is already a dataset.
I have some questions:
- I´ve set up one script per dataset i want to snapshot and replicate. Multiple datasets pers script are not supported or did i miss something?
- What is the best strategy for offsite backups? I have a second unraid server 500km away and would like to replicate to it too. Do i just set up every script again, but with the remote server? What source pool do i choose in this case (the one on my cache again or the one in the array)? If i have 15 datasets to replicate, that would be 30 scripts then xD.
Maybe it's pretty self-evident but could you possibly do a quick restore from the backups? single container, multiple containers and the whole appdata folder.
I agree. Maybe it is just as simple as copying everything from disk13 into your new - empty cache pool under the same names but it would be nice to mention how it would be done.
@@laov6843 Same here. I added a new 1tb NVME drive to my ZFS cache and it wants to format both drives despite being set to mirror. I can either run off the original NVME or format both and restore snapshots, but the options don't make it clear on what is doing what.
I'm in same boat here. I'd wish to see how you use the snapshots which are stored in the array. Do these need to be moved back to original disk/share or can they be used straight from backup share?
@@x1mega from what I can tell, you should be able to use them from the backup share. I've had some successfully restore and some not, so there is still something I'm missing.
I am in the same boat. want to swap cache drives. but shouldnt you just be able to copy the data from the backup folder and start docker?
For some reason it isn't deleting the snapshots on my backup drive. I have 13 snapshots on my cache drive with the appdata. While I have 24 on the backup drive.
I'm having the same issue, I would be interested if you found a fix or not.
@@Jleonard3314 I created a PR on the github to fix it. You have to also add the flag --delete-target-snapshots to syncoid to delete the snapshots
Same issue! Would be great if someone could share a fix for it.
@@olitrolli8695 The fix has already been added to the GitHub
I'm not sure if this is related, but I think once you run the script for the first time, changing the retention parameters in the script doesn't have any effect. This line I think says that if a sanoid config already exists then don't create a new one. Well one would exist the first time you run it and then never change again no matter what you change the retention variables to in the script. I had to manually edit the sanoid.conf file.
# check if a configuration file has already been created from a previous run, if so exit the function
if [ -f "${sanoid_config_complete_path}sanoid.conf" ]; then
return
fi
New intro is lit 🔥🔥🔥 🕺🏼
Hey thanks. Glad you like it. Thanks for watching :)
@@SpaceinvaderOne please upload more videos....
Awsome Video thank you!
I have 3 Questions for you.
1) If I want to snapshot all of my shares is there a way to add that to the script or do i have to create a duplicate script for every other shares?
2) What happens if the drive has no more free space will it auto delete older snapshots or fail to create new snapshots?
3) how many hdd's would you need for say a 84tb snapshot backup? I have 6 free 14tb drives (mostly empty + a 14tb parity), and 14tb for zfs backups) but id like to have everthing snapshotted. Is it a 1:1 ratio? would i need 6 14tb zfs backups?
Hi Spaceinvader! Awesome video! Seems a couple of people including myself are experiencing that snapshots in the destination folder aren't being deleted to mirror the source 1:1. Any idea on a fix for this? syncoid_mode="strict-mirror" so it should work 😕
fantastic video! thank you
Thanks Ed, You are an ace😊
Your welcome. Thanks for your kind comment :)
@@SpaceinvaderOne seriously. I really don't know what I would do without you :)
I do love that you choose Raw on github, when there is a copy button right next to it that will place the contents into your clipboard. That being said, thank you for all your ZFS videos!
I have followed your tutorial setting up an appdata backup from my ssd to my ZFS array disk and it's working great. I have now decided to encrypt all my disks.
I have watched your tutorials how to use mover and unbalance to encrypt disks one by one, but could you perhaps make a video of what to watch out for when doing this with ZFS Disks (in terms of snapshots and datasets)?
Excellent video. So at the end the snapshot said 268k. Is that the actual space used to for each snapshot?
Excellent video as always - how do you have your main page in unraid grouped into Array, Cache, etc?
settings-display set-page view
Thanks so much for all your videos, you are such a help to the unraid community. Regarding the ZFS Master plugin do you or anyone else here know anything about the issue where all ZFS pool drives are constantly written to when this plugin is installed and the web UI is left on the main tab? I had installed it after watching the previous videos and uninstalling it was the only thing that fixed the problem, has that issue been fixed or does using the tabbed view help mitigate it?
Just set automatic refresh interval to 0 and it won’t constantly ping/refresh the drives. You can still manually refresh.
Thank you so much for another great video. Every time i watch a tutorial from you i always pick up great stuff besides what the tutorial is about.
I'm not that skilled so i like many backup fallbacks. I have a few questions:
- "mbuffer not available on source [local machine] - sync will continue without source buffering." i guess this is because i'm writing appdata from and SSD disk to a mekanic one?
- Now i have appdata in two places and on both i have the ability to revert a snapshot. I don't get where i should do what and in witch scenario?
- I have an GDrive mounted that i can put stuff i want to secure in the cloud. Should i run a rsync from the appdata on the SSD cache or from the mekanic zfs drive?
Thank you again for your very easy to follow tutorials.
/Peter (Sweden)
Where do you get your thumbnail graphics from? i.e the deathstar with the UnRaid logo on etc?
While I was waiting for this video, I managed to setup more or less the same thing with znapzend and it has been working fine so far :)
But with this method, you want to create 1 script for each dataset you want to snapshot/backup?
And if you want hourly snapshots I guess you need to set the script to run on the same interval as you want to run hourly snapshots?
But learned now that you can have the main page tabbed :O
yes each location requires a separate script and schedule
Thank you for the great video. I hope people understand the time and the effort needed for all the content you do. I did everything from the video then did some tests and everything is working. But i don't understand why my zfs disk on the array is always spin up even i exclude it from all of the shares? It is used only for the zfs replication. And is it possible do make the replication on an Unassigned Devices to prevent the array to spin up?
Just to clarify, source/dest datasets can only be top level(?) For example appdata and not appdata/nextcloud. When I tried to backup specific programs in appdata, the backup would fail
I still hope that unraid can realize these automatic backup functions in gui like truenas
what plugin is giving you sub tabs under the main tab?
Great video/ Great script / Great job!
Did you ever figure it out? I have been hunting for the plugin too
Just found it :)
Go to Settings->Display Settings and change Page view to tabbed.
@SpaceinvaderOne: 1st thanks for your great work!
You imply dataset replication to remote servers.
In addition to local replication from cache to an array disk, I would also like to replicate the datasets to my backup server.
How does this work? I assume, I have to somehow mounte the remote ZFS dataset to my server, but how?
Thank you!!
Question, can this do recursive? I have quite a few datasets, and setting up a user script for each one was a pain, and now I've found an issue in that if one is running when the next starts it throws an error. So yeah '-r' option, if so where?
This guide is great! I had set it 8 months ago and didn't look back. Now I'm seeing daily backups since then and I've got 6k snapshots. Any advice on troubleshooting this? I didn't change the default retention settings.
How on earth did you get tabs on your Unraid main area 13:49 looks so much better than Unraid’s traditional “here’s your 30 disks”
It is in the display settings.
Specifically Settings->Display Settings->Page View
tabbed view
Cheers, this is new to me but when you have made a backup to another zfs single disk how do you restore a snap shot from that backup single xfs disk if something goes wrong with the main zfs array? Backups are kind of pointless if you are left scratching your head wonder what do I do now to restore.
Awesome video Ed, thank you! Everything is working great except the script doesn't seem to be pruning the snapshots in the destination. I have 6 hourly snapshots set and those get pruned on the source dataset but not on the destination. What am I missing?
Noticed the same issue- Have you figured out a solution?
@@tunatron3280 Same problem here, I have 13 daily snapshots instead of 7. I looked through the script and there doesn't seem to be anything in there about pruning the data other than in the source.
I think something is missing either in my setup or the video.
I'm having the same issue. Snapshots aren't getting pruned on the destination.
@@thebaldconvict there's a pull request on his github now that fixes this
@@JByteTX there's a pull request on his github now that fixes this
Sir .. please upload more videos , update the old content according to latest version of unraid its really helpful and we all thank you ❤
Yes! Thank you
You are so welcome!
i made a cache pool with 2 1TB NVME as a ZFS Mirror. Question now that i have: How to replace one NVME in case of failure. Just replace the Drive and Unraid will rebuild the mirror itself?
Thank you so much for this awesome video. Makes me feel better about my cache drives data being saved since they're not protected. I am having an issue though. I started with snapshoting my appdata and replicating fine. My issue is when doing the same for my domains and same for my nextcloud folder. Any chance you can give me a clue into why? Here is the error shown in the log.
*update - my issue is corrected. I believe I was having issues with my drive so I wiped and reformatted. All working now. Thank you anyway.
Maybe a stupid question, but I saw your main tab is decided up in sub tabs. How to set that up?
25:23 how is everything a child dataset of appdata? do I have to create that for individual containers?
Great video! This simplifies my backups a lot! Anyone know if I use Syncthing to sync my local zfs replication to an offsite server, does all the hard linking and everything still work? I know I could use rsync in the Part B video, however, I have Syncthing already in place and wondering if anyone has does the zfs replication over syncthing? Thanks
Thanks!
I have set this up a while ago but after upgrading my ZFS Cache disk from 240GB to 4TB I would like to recover the ZFS datasets that I backed up.
How can I perform a restore from the snapshots?
Hello from greece!! i would like to point the need for shadowcopy implemented in shares, or how to do it, i tried some configs but nothing seems to work with these snaps!!!!
What is the best method for restoring a snapshot from the backup drive?
Will ZFS master still be needed when Unraid completed the ZFS roll out?
Wow! 😍
Can you do a video on replicating ZFS snapshots to unraid from another system like truenas ? very good content, keep up the good work.
Am I missing something for restoring a snapshot from the array disk? If I use rollback, I get a message saying it's successful, but nothing is restored to the cache drive. I guess I could copy all of the files out of the backup drive and move them to cache, but it's a little inconvenient.
The only way I can get anything to restore is if I roll back a snapshot on the cache drive itself.
Edit: maybe I'm going about it wrong, I was thinking of restoring snapshot from my array to cache, but everything is replicated in the share, I can move from backup to cache. Either way, I'll probably give it a whirl since I'm adding another NVME drive and my current + new will have to be formatted (guessing because of ZFS+encryption on current).
Yes just tried to test my backup snapshots on my external ZFS drive and get the same result as yourself. (successfully rolled back but data is not restored) Would love to know how to make use of these snapshots otherwise this is all pretty pointless
How do you got tabs on the main page? Makes it the loading faster because my main page is terribly slow.
settings-display set-page view
Isn't dangerous to snapshot database files or open files in general? A part of the content is in ram so it gets lost. Corruption can happen so I think you need to stop the containers or dump contents anyway.
Should I stop Using appdata backup pluging after completing this?
Destination isn't cleaning up old snaps I have 2 weeks worth now, it's working fine on source
Same problem here!
Anyone know how I can replicate a zfs data set to another encrypted zfs data set? Everything works great until the target is locked
Love the videos, thank you for all of your hard work. Would love to see how you did the “Plex” zfs pool, does it refer back to your Unraid server media folder??
Thanks! For the response to my email
I like unRaid because I can have different size drives. If I convert to ZFS, will I need to have all the drives the same size?
Not in the array, but it does not make sense to have all drives zfs in array. Its messy when the datasets gets split over different drives.
@@39zackthe ONLY advantage of keeping all disks in array as zfs is snapshots ... The other benefits of zfs are none ... So it's better to keep one or 2 disk as zfs in array .. 2nd thing is if all disks are kept in zfs than when we access data from one drive or write to that disk all disk will spin up .. consuming energy ...
Its better to make a separate zfs pool and use and keep 1 or 2 disk in zfs in array rather than all
@@drmetroyt yup, thats how I run it now after trying zfs for all array drives. Sure, snapshots are nice, but the restore got messy when a dataset is split over multiply drives and you ether need to restore all for all drives or figure out where the data you want to restore is/was
@@drmetroyt forgot to mention. The one drive I have in array that is zfs is also removed from global shares so nothing gets written to it by unraid.
I use it purely for zfs replication backup and its in the array so it gets added to the parity :-)
There it is! Awesome. Been refreshing your channel constantly watching for this video
thanks for your patience.I hope you find it useful :)