I mentioned in the video that BTRFS was a requirement for SHR, and I want to clarify that this isn't correct and it is possible to use EXT4 with SHR. However, BTRFS is the recommended file system, and it provides a number of benefits, most relevant being data scrub capability. Apologies for the erroneous information there, and although its not central to the video I want to make sure the information is as complete and correct as possible. Thank you all for watching!
That diagram was excellent. I made assumptions about how SHR worked, but it wasn't until I tried to make sense of your diagram that I realized my assumptions were wrong. Great job explaining!
Thank you! When I was first trying to understand why the Synology tool was giving the capacities it was, I was also confused. So I wanted to try and make a resource that explained why. Glad I succeeded.
Thanks for the explanation for beginners. I bought a 1522+ and 2 drives with the intention of adding several months down the line. Im glad to know that i can easily do this and increase the storage exponentially.
Thanks for giving me the heads-up on SHR! It sounds a lot like Drobo (although I used the old BeyondRAID calculator via Wayback Machine and found Drobo often created 10%-33% more usable capacity, e.g. look at 10+10+10+6+6 - there could be good or bad reasons for this). Your channel is amazing - I'll give 10x more if you make content about what we've lost with the demise of BeyondRAID and where we can find a modern substitute. I'm talking about a DAS with direct mounting native filesystems (as if it's one big disk on USB or Thunderbolt or internal), and where mixed sizes and changing sizes are normal, not just a fallback. I'm sure if Drobo didn't start dying 5 years ago they would have supported APFS and SSD speeds by now.
Thank you @whophd ! It's really appreciated. I have not used Drobo, before Synology had used Thecus but I don't know how Drobo organised its disks. More NAS providers now are providing flexible options for disk arrays, and I think the implementations vary a little. I will take a look at BeyondRAID vs unraid and trueNAS.
@@sometechguy Thanks so much! I feel lonely dying on the hill of "direct connect only" but I just don't want to put up with accessing all my files over SMB, plus having to manage yet another IP device on my LAN.
Really enjoyed learning about SHR here from you. Thanks. I too am coming over from my beloved Drobo 5 bay drive. I'm looking strongly at Synology SHR as a replacement. With Drobo I could mix and match any size TB drive from any manufacturer and Drobo would work it out. If the smallest drive got filled Drobo would notify me to replace it with a larger drive. I'd pop that smaller drive out, say a 4TB drive, and put a larger one in, say an 8TB drive. Drobo would let me do that live without stopping what I was doing. I guess I'm hoping that sort of hot-swap of a smaller drive for a larger drive is something that Synology SHR can do too. Thanks, again.
I believe Drobo implemented it a little differently, but the goal was the same, to give the flexibility to use and replace disks with different sizes for dynamic growth of the array. Which is a capability I love. Others have also implemented similar now also, including Terramsster’s TRAID. Glad this was helpful and appreciate the comment and feedback.
I couldn't make heads or tails out of the Synology Raid Calculator. I'm running SHR2. The concept of SHR didn't make sense when the calculator was showing "unused space" for drives bigger than those installed. This is the video I should have seen before I purchased the expansion unit with the same drive sizes as the other 4 and changed from SHR to SHR2.
Thank you for watching, and for commenting. I appreciate that. I had the same experience, and the stats the tool provided for SHR-2 were particularly confusing until you get under the hood on how it works. So I am glad this helped make it clearer. 😁
Fantastic video and a great explainer. I was replacing 6TB drives with 14TB drives and thought all the extra capacity of the new drives would go wasted until I replaced all the smaller drives. I was pleasantly surprised to see the reality was as you described in the 9 minute mark. One question I have is, when you eventually replace all the smaller drives, doe the separate RAID arrays combine into a single volume?
No, they won’t. And I believe the reason is that to do this, all the volumes would need to be deleted and recreated and there would be nowhere to store the data during that process. But also, I am not sure there is a pressing benefit. It may look ‘neater’, but probably doesn’t make much difference from a data management standpoint. But of course, just my assessment. I don’t know what design decisions and compromises were made.
The only video on TH-cam about how shr really works . How did you figure this out ? Synology documentation is complicated or unavailable :) Thanks for your video :)
A lot of poking about. 😄 Thank you also for watching. In TOS 6, the current beta software for Terramaster they added TRAID+ which looks like it should be a replica of this, but need to have a poke at this also.
You can use SHR on any Synology device, the caveat though is that you must create it on a device that supports it, then you can migrate the drives to the units that don't support it and it will work as normal from there.
Thanks for sharing this, its good to know. I wonder if Synology would support that if an issue arose. Seems like a legitimate workflow to move a disk array when updating hardware, if less common than new NAS+disks and data migration. Out of interest, which device did it get migrated to and is it a larger disk count than you would find in a supported device post migration? Was growing the SHR array possible in the unsupported unit?
@@sometechguy I am not sure if Synology would "support" the configuration, as they really don't want SHR being used in an enterprise level device, but aside from not being able to create an SHR, once you move any SHR drive in to the unit as a new configuration and import it, it operates as any other unit and you can do everything you do on a device that allows you to create the SHR, I believe that all they did to "prevent" SHR from being used on enterprise devices is to make it so you can't create it, but importing existing works just fine.
My NAS has 12 x 4TB drives. After being in the enclosure for 7 years, I had two disks fail in the array, both within a week or two of each other; before I noticed that the drive had failed. It was not a Synology NAS, but it was configured for RAID 6. At times I had considered rebuilding it as RAID5, but I really didn't need the space.. The point is, multiple drive failures can and do happen.
Thank you, I really appreciate the comment and encouragement. Yes, I am looking at the Intel version of this and plan to get something together. Hopefully soon!
Will depend on what software you mean. But placing a failed array into another Synology NAS will work, and as SHR uses standard Linux tooling, you can also recover it in a non Synology device, but this requires some know how, and there are guides available. Any raid recovery will need enough data to be intact though, so will depend how many disks were lost and the consistency of the file system etc. This is why having regular data scrubs is important.
@@sometechguy Doing a quick internet search I have found RAID recovery software to include 1.diskinternals, 2. Stellar Data Recovery, 3.reclaime, etc... I am sure there are others. None of them say SHR but they do have like RAID5 and RAID6 which is close to SHR. I did have data loss in a RAID system a few years ago and think its because the drives may have been SMR (Shingled magnetic recording ) hard drives which the manufacturer hide from the public. It was not listed anywhere on the drives or even in the specs. I wish Synology did not call it Data scrubbing because that names makes me think they are going to scrub(delete) the data on the drive. They should instead call it like drive scan and repair or something like that.
@@ThrowtheJ.DownTheWell , sorry for the slow reply on this, I missed it originally. SHR actually uses the MD implementation of RAID under the hood, which is the linux standard. For the file system, it depends on the configuration, but Ext4 is possible, with BTRFS being the recommended system. Again, these are linux standard, so it would take some knowledge but the array should be recoverable in a device other that Synology as long as the right things are in place. I would say though, it would require some knowledge and wouldn't work 'automagically'. As for 'scrubbing', this language is inherited and didn't come from Synology. Its the usual work to describe a check and repair of files in systems that support it, notably BTRFS (used by Synology) and XFS, a very popular highly robust file system also available for *nix based systems. So Synology just went with the industry nomenclature there I think. But you are correct that Synology could have used a more abstract and 'user friendly' naming in the UI. They would likely have had criticism for doing that also though, for 'hiding' the function etc.
Looking forward to seeing your follow up video as I am not sure if I should enable SHR-2 on my 1821+ if I am going to us iSCSI am I going to lose any performance?
Hi, and thanks! I am not sure if the filesystem choice has any meaingful impact on iSCSI performance, and its a good question. I have used iSCSI on one of my NASs, but not for a heavy duty workload, but I gather the Synology iSCSI implementation has historic performance issues, and NFS would likely perform considerably better. And I don't believe this has a direct linkage to the underlying filesystem, more the iSCSI implementation itself. Maybe separating the iSCSI onto separate interfaces, and using MPIO can help, but it sounds like a broader problem than that. I would look into that issue before choosing iSCSI on Synology. Overall, I have seen numbers of around 1% performance hit, but I have not benchmarked them as yet.
My big question as a home user is what is the minimum drive NAS I'd need to get a full benefit from SHR? By the graphics I'd say a 5-disk setup. But would similar benefits appear when starting with one 10GB and one 14GB, and then adding another 14GB? I guess I should have asked the question as, where do the benefits begin?
You can't convert a traditional RAID into SHR, without recreating the volume. So its best to start with it from the beginning. But separating the benefits of SHR and BTRFS, as they are distinct (BTRFS gives you data scrubbing and snapshots etc), SHR really provides the most significant benefits once you want to use disks of different sizes, which gives you flexibility in the short term, and the ability to grow the array later cost effectively. So if you start with, as an example, 2 x 10TB disks. With RAID you would continue to add 10TB disks (you can add larger but you can't use the additional capacity). With SHR, you can start to add 14TB disks, or 18TB etc as the price point makes sense. And once you have enough of the larger disks (2 with SHR, 4 disks with SHR-2) then you start to get the benefit. With RAID, you would need to go back and replace all disks to get that benefit. So, I would start with whatever disks you plan to start with, lets say 2. And then you can add disks that make the most sense in the future based on price per/TB and get the benefits of that. Once your NAS is full, or the disks start to age or fail, you can replace those disks with larger disks and start to get increased capacity as you replace. And to address your example directly. If you chose SHR and added a 10TB and 14TB disk, you would get 10TB of usable space. (10TB Parity with 4 unused), the same as with RAID1. Once you add the second 14TB disk, with SHR you would get 10+10+4=24Tb (with 14TB used for protection), where as RAID 5 would only give you 10+10=20TB (with 10TB for protection and 8Tb unused). Hope that is clear, and helpful.
@@sometechguy yes, it's clear. Thank you! This will be my first NAS, so there's no conversion issue... I'll just start with SHR. I'm well-familiar with Btrfs as I've been using it in all of my Linux machines the last three years or so. To get my feet wet I think I'm going to use one of my several Linux laptops with TrueNAS to start. My data is all sentimental-type stuff... Family photos, documents, etc. No media production files or anything that requires fast transfer speeds.
If you start for example with one 12tb drive , the next drive must be same capacity or higher ? Or can you have a lower capacity for instance 8gb? Another question, what happends with the unused space? Is it lost or can something be donde with it? Thanks
To add a drive to an existing SHR (or a traditional RAID), it has to be at least as large as the smallest existing drive, this is because this drive (the 12Tb in your example) will be the size of the first logical disk. Any new drives have to at least have a 12Tb available space to be added to that. On the second question, the space isn't usable initially, and this was the 'red' space shown in the graphics. But as you add other drives, it will become usable. For SHR1 you will need at least 2 drives of that size (Shown at 6:55 in the video), and for SHR2, at least 4 because SHR2 needs two parity partitions.
Can you recover data from an SHR in the event of equipment damage to the synology but undamaged hard drives via some kind of RAID software recovery program?
You can, as it just uses standard BTRFS and Linux tooling. However, I would say it needs significant know how, or maybe a lot of research if you don't have it. I had a NAS fail and I took the disks out and put them in the warranty replacement and it just ran. But restoring in a non-Synology box would take some work. Definitely, for any disk or NAS or other store of important data, do not neglect backups. Failures are not at all common, but they can happen. Thanks for the question.
I assume Synology has a hardware problem or a software problem with SHR if the limit on the number of hard drives is exceeded. The reason may be reasonable. Standard RAIDs have existed and been available for a much longer time. In the days of Amiga computers, remember SCSI hard drive validation started if the power went out unexpectedly. Nothing could be written to the SCSI hard drive if validation was in progress. After validation, the hard drive worked normally again. A protection like this could help with RAID systems, but many people could get nervous waiting for a complete check and repair.
of what I understand, SHR will result in uneven performance? I mean, let's say a 10GB file is written on "last" two disks (in your graphic), read/write speeds will be limited to those two disks, instead of combined speed of 8 disks? or am I getting it wrong?
My expectation would be that you are correct. If it's SHR-2 then the data has to be on at least 4 disks. If its SHR, it has to be on at least 2 disks. But in both cases, the more disks you have the more spread the data will be and there could be performance changes. But as both SHR1 and 2 with different sized disks may produce different logical RAIDs over different numbers of disks, then there is the potential for variance. But for smaller NAS (home and SMB), this likely isn't going to cause any measurable issue as its often the network bandwidth that is the bottleneck anyway. But this is an interesting point, and it could even be one of the reasons they don't offer this no the enterprise products.
Thanks a lot for your helpful videos, you helped me chose which device to buy (DS923+). One question: If I add 4x 4TB drives with SHR, how is it posible that a single 4TB drive is used to "backup" the rest of my data (12TB)? Or did I misunderstand? Thanks in advance!
Hi, and thank you for the comment. The 4Tb drive doesn't keep a copy of all the data from the other 3, rather there is 'parity data' that uses the capacity of one drive, though that data is spread around and not just kept on 1 drive. Any 3 of them remaining drives hold enough information to fill in the blanks if a disk is lost, or some data on that drives becomes corrupted. If you want to dig a bit further into how this actually works, I do have a video (th-cam.com/video/2Dovoc9LP34/w-d-xo.html) that explains RAID parity. And SHR uses these mechanics under the covers, along with some other things. Glad to be of help, and enjoy your new NAS!
@@sometechguy That's a great explanation, I really appreciate it! Thanks a lot and even if one more doesnt really matter, you just got a new subscriber 😌
If I just purchased a Synology NAS and only have one 16TB Iron Wolf drive…what should I do with regards to RAID? Do I need more disks in order to get the Synology 418 NAS to work on arrival?
With 1 disk, you can’t get any fault tolerances. e.g. you won’t be protected against a disk failure, which may be obvious. So this means you can’t configure traditional RAID, as you need two disks to do that, of which one would just be used as failure protection for the other. But you can configure the unit for SHR with one disk, again no fault tolerance to protect from a failure but you can add a disk after you start to do that. When and if depends on your data and how much you need to protect against loss. The advantage of SHR here is you can then add other disks later, get your fault tolerance and grow your storage. If you want to visualise this, you could try and Synology disk calculator linked in the comments.
@@sometechguy AWESOME! I should have watched your full video before asking July question lol. I have seen many videos but none of them was as clear as yours. Thanks again.
thanks for your video. I just installed my synology to Raid 5 untill i saw your video. And it has 35 Tb data in it. I am curious is there anyway convert my raid 5 to SHR without losing my data ? or i have to carry all my data to somewhere else and start from scratch ? I have DS1621+ thank you
Synology supports converting between certain RAID types, and moving from SHR1 to SHR2. However, you can't move from traditional RAID to SHR in a non-disruptive manner. Unfortunately, if you want to move to SHR, you would need to move the data off, start a new clear array as SHR, and then move it back. Or of course restore from an existing backup. But check its integrity first. So sorry, this isn't the answer you were probably hoping for.
Thank you, appreciate the positive feedback! I have some content coming comparing the under the hood of BTRFS vs EXT4 on Synology, and I also want to do some other comparisons, including ZFS, which is a really interesting topic.
I watched this video months ago.. and when I bought new HDDs for my NAS, I switched to SHR. After using it for 2 months.... I switched back to Raid 5. Raid 5 gives me 20% more storage area and seems faster in transfering files.
I can't see why that would be the case, unless you are comparing SHR-2 to RAID 5. Is that possible? At the minimum SHR-1 should provide the same storage as RAID5, SHR-2 would provide at the minimum the same capacity as RAID-6. But if you are comparing a 5 disk array with RAID5 vs SHR-2, you may well see what you see. On the performance side, there could be a slight overhead with using BTRFS over Ext4, but if you are using the same filesystem to compare RAID5 to SHR2 then again, this is likely because SHR2 will use RAID6 under the hood and RAID6 is going to be a little slower than RAID5 due to the additional parity calculations. But I would guess it isn't really large and in many cases, the bottlenecks would somewhere else, such as the network. Maybe check your disks into the RAID calculator at www.synology.com/en-uk/support/RAID_calculator and see how RAID5 compares to SHR1 and SHR2 and see if this is the issue.
0:51 to say that it is "especially suitable for users who are new to RAID technology" is, IMHO, no way implying that it's not for advanced or experienced users. That's nonsense. I have managed my own RAID on other machines and it's not a big deal. But, SHR is brilliant in its ability to mix different sized HDD's and that has nothing to do with experience it's just a great way to create flexibility in choosing your drives, now and in the future.
I think you are agreeing with me here, that SHR for sure is not only for new or ‘inexperienced’ users, and the point here is that it’s a great choice for many applications. But my take away at least from the Synology article, is that it implies it’s primarily for newer users. But maybe that’s just my reading.
I use BTRFS on all my Synology devices. It offers a number of advantages over Ext4, both for functionality (snapshots) but also data integrity (data scrubs), which Ext4 doesn’t.
I've been intrigued by RAID in the past but never actually used it. A number of reasons: If the raid hardware, e.g. motherboard or NAS fails, would the array be picked up by a replacement or would all data be lost? I don't actually know the answer to this one but it has put me off. If a drive in a raid array fails then rebuilding puts the remaining drives under strain, and it can take a long time. Lack of portability, having a separate external drive allows you to take a copy somewhere while a copy remains at home. Common cause failure, using similar drives and running them the same amount could increase the chances of all drives failing at a similar time. Having an external drive occasionally used to backup mitigates this. For always on NAS, running costs are higher than a cold backup. Cold backups are also resilient to data ransoming. For these reasons I typically have a large drive in my main computer and an external drive that I do a backup to every month or so. Important work in progress I save on Dropbox. I'm fortunate that my data collection has always been smaller than the largest drives on the market so I've only needed two drives rather than a higher multiple of two. Just my PoV.
It will depend on data volume, but RAID should never be an alternative to a backup, it’s an extra real time mechanism for data protection. For example, if you backup to an external disk, there will be a delta between your live data and your backup, which could be a day, or many months, depending on how rigorous the backup regime is. For portability, it depends but you should usually find moving an array between similar hardware should be fine. I once had a mainboard failure on a Synology NAS and the unit was replaced, the disks went into the new unit and the entire thing started like nothing has happened, NAS OS, all volumes etc. it may often be recoverable on other hardware if the software is the same also. But recovering SHR on another device than a Synology is more complex than a basic RAID config. And yes, replacing a failed disk and recovering an array is a high stress event for other drives. It’s totally fine for them, but if any of them are also verging on failure, it can kill them and I have seen this happen. RAID 6 or SHR2 on Synology reduces that risk, but before I do a RAID recovery, I check all my backups first. If you don’t have the high capacity requirements, what you are doing is good. Depending on the criticality of data, another backup kept elsewhere is also good. Maybe not for everything, but the things that really cannot be replaced. Thanks for watching and for the comment, great questions!
I'm more partial to ZFS as I've been using it for years but not a fan of its lack of expandability apart from adding entire V-devs (Mirrored or Z) which can create their own problems.
ZFS is a great file system, but this isn't supported on Synology. And as you say, I believe it doesn't provide the same expandability options. Synology do a nice job of abstracting this away so it doesn't have to be managed by the user.
I believe unraid has a similar implementation though it may be a little different. But of course this video wasn't about unraid. And Synology has also done this for a while, it just isn't always well understood, especially how using different size disks affects availability capacity.
yeah, I'm using unraid now for about a month or so I think it's way easier to understand it's as simple as: 1- your parity disk needs to be same size or bigger than the bigger disk in your array 2- the amount of parity disks you add are equal do the amount of disks that can fail without losing any data.@@sometechguy
It's my list to have a deep dig into unraid, it looks really interesting. So thanks for bringing it up. I assume the concept of a 'parity disk' isn't really one physical disk, but a logical construct. RAID 4 was similar to RAID5, but rather than having parity spread in blocks across the entire array, it had all the parity on a single 'parity disk', but this caused performance and drive load issues, as every time anything is changed on a disk, the parity needs updating and this means a single parity disk becomes a bottleneck.
Honestly i would stay away from it. I understand the point of putting storage to use but if you hold valuable data worth protecting and not stuff you can easily replace i wouldn't want them on a jbod across 3 raid-5's on the same failure domain (disk), that's just playing with fire
Wow, what madness. If you use you RAID system extensively, performance could be atrocious if and when you start accessing data from the same disk simultaneously and which will stress your hard disks. Home users should keep things simple. I hate to see a rebuild of a 8 to 12 disk RAID 5/ 6 when you have a disk failure.
I don't recommend RAID at home unless it's RAID 0 for performance with your backups of course. I would just create a sync, it's way easier to manage. The performance of anything other than 0 is pretty bad, and rebuilding the array is pretty painful.
@@xlion it's even faster than rebuilding RAID. Less data to gather and copy around. If the NAS is a 5 bay, I would buy 4 of the same capacity for RAID 0, daily use, and a bigger one for backups. Then you can use whatever tool you want to sync the data, you can use rsync which is int the control panel, or one of the many ones they have in the market. This way you also don't lose access and have fast speed access to your data
Maybe it would have been worth watching, and this video th-cam.com/video/aLoajg9yFxg/w-d-xo.html goes deeper into what is really happening under the hood. SHR has BTRFS as a prerequisite, and it is because it uses its file system, its Copy on Write mechanism for write integrity and its scrubbing capability for ongoing protection against bitrot. So while its true that it uses MD for RAID management and data protection, it is built on BTRFS for many other capabilities, with custom integration with MD for data parity management.
@@sometechguy raid or shr, and ext4 or btrfs, are independent choices, one does not depend on the other. In particular, you can have btrfs on raid, it does not require (or based on) shr. That is my point - did you mean something else by 'based on' ? "SHR has BTRFS as a prerequisite" is just wrong.
@paulbarnett1461, I take your point. BTRFS is the recommended filesystem choice for SHR, but you are correct it isn't _only_ available choice for BTRFS, which I suggested in the video. So thank you. But to be clear, I had not suggested RAID was not available on BTRFS or that the choice was EXT4/RAID vs BTRFS/SHR. This video was more focused on the benefits of SHR over traditional RAID, and why it's a good choice. BTRFS also provides other benefits missing from EXT4, in this context especially, the ability to perform data scrubs to check file integrity. And this is why its recommended. I appreciate you pointing this out however, so thank you.
@@sometechguy I think we mostly agree, but not in some details. btrfs does not depend on shr, not even for data scrubbing. you can (and should) do scrubbing, and therefore btrfs, even on raid (I do this) And because btrfs does not require shr for anything it can do (at least, not in DSM), it becomes not relevant to a discussion of raid vs shr. If it had been me, I'd have done two videos, one on raid vs shr, and a separate one on ext4 vs btrfs. I watched this and your other videos, and enjoyed them (apart from the 'btrfs requires shr' idea) - and I agree with all your recommendations: use shr use btrfs, & enable checksums (and maybe compression, but not for VMs) scrub regularly Suppose you had a large (say 100tb) raid pool, with a large (but not that large, say 50tb) volume. I claim you can add another 50tb volume as btrfs with no outage. (and I've done this (with smaller numbers)). You are implying that you'd have to backup all existing data, delete and recreate the pool (as shr), recreate the original volume (now as btrfs), restore the original data, and only then create the 2nd volume as btrfs. I don't intend to argue this further. :-)
I mentioned in the video that BTRFS was a requirement for SHR, and I want to clarify that this isn't correct and it is possible to use EXT4 with SHR. However, BTRFS is the recommended file system, and it provides a number of benefits, most relevant being data scrub capability. Apologies for the erroneous information there, and although its not central to the video I want to make sure the information is as complete and correct as possible. Thank you all for watching!
That diagram was excellent. I made assumptions about how SHR worked, but it wasn't until I tried to make sense of your diagram that I realized my assumptions were wrong. Great job explaining!
Thank you! When I was first trying to understand why the Synology tool was giving the capacities it was, I was also confused. So I wanted to try and make a resource that explained why. Glad I succeeded.
Thanks, this was so much help especially the graphics that explained it clearly.
Thank you, and especially for the generous Super thanks. Its appreciated. 🙌
Thanks for the explanation for beginners. I bought a 1522+ and 2 drives with the intention of adding several months down the line. Im glad to know that i can easily do this and increase the storage exponentially.
Thanks for giving me the heads-up on SHR! It sounds a lot like Drobo (although I used the old BeyondRAID calculator via Wayback Machine and found Drobo often created 10%-33% more usable capacity, e.g. look at 10+10+10+6+6 - there could be good or bad reasons for this). Your channel is amazing - I'll give 10x more if you make content about what we've lost with the demise of BeyondRAID and where we can find a modern substitute. I'm talking about a DAS with direct mounting native filesystems (as if it's one big disk on USB or Thunderbolt or internal), and where mixed sizes and changing sizes are normal, not just a fallback. I'm sure if Drobo didn't start dying 5 years ago they would have supported APFS and SSD speeds by now.
Thank you @whophd ! It's really appreciated. I have not used Drobo, before Synology had used Thecus but I don't know how Drobo organised its disks. More NAS providers now are providing flexible options for disk arrays, and I think the implementations vary a little. I will take a look at BeyondRAID vs unraid and trueNAS.
@@sometechguy Thanks so much! I feel lonely dying on the hill of "direct connect only" but I just don't want to put up with accessing all my files over SMB, plus having to manage yet another IP device on my LAN.
Thanks for this - I never understood how SHR worked (though I knew that it did) - very helpful!
You’re welcome, thanks for the kind comment. 👍
Thank you so much the graph and your explanation so much easier to understand!!
You're very welcome! Thanks for commenting.
Really enjoyed learning about SHR here from you. Thanks. I too am coming over from my beloved Drobo 5 bay drive. I'm looking strongly at Synology SHR as a replacement. With Drobo I could mix and match any size TB drive from any manufacturer and Drobo would work it out. If the smallest drive got filled Drobo would notify me to replace it with a larger drive. I'd pop that smaller drive out, say a 4TB drive, and put a larger one in, say an 8TB drive. Drobo would let me do that live without stopping what I was doing. I guess I'm hoping that sort of hot-swap of a smaller drive for a larger drive is something that Synology SHR can do too. Thanks, again.
I believe Drobo implemented it a little differently, but the goal was the same, to give the flexibility to use and replace disks with different sizes for dynamic growth of the array. Which is a capability I love. Others have also implemented similar now also, including Terramsster’s TRAID.
Glad this was helpful and appreciate the comment and feedback.
I couldn't make heads or tails out of the Synology Raid Calculator. I'm running SHR2. The concept of SHR didn't make sense when the calculator was showing "unused space" for drives bigger than those installed. This is the video I should have seen before I purchased the expansion unit with the same drive sizes as the other 4 and changed from SHR to SHR2.
Thank you for watching, and for commenting. I appreciate that. I had the same experience, and the stats the tool provided for SHR-2 were particularly confusing until you get under the hood on how it works. So I am glad this helped make it clearer. 😁
Very elegant and helpful explanation. Thank you.
Thank you 👍
Fantastic video and a great explainer. I was replacing 6TB drives with 14TB drives and thought all the extra capacity of the new drives would go wasted until I replaced all the smaller drives. I was pleasantly surprised to see the reality was as you described in the 9 minute mark. One question I have is, when you eventually replace all the smaller drives, doe the separate RAID arrays combine into a single volume?
No, they won’t. And I believe the reason is that to do this, all the volumes would need to be deleted and recreated and there would be nowhere to store the data during that process. But also, I am not sure there is a pressing benefit. It may look ‘neater’, but probably doesn’t make much difference from a data management standpoint.
But of course, just my assessment. I don’t know what design decisions and compromises were made.
The only video on TH-cam about how shr really works . How did you figure this out ? Synology documentation is complicated or unavailable :) Thanks for your video :)
A lot of poking about. 😄
Thank you also for watching. In TOS 6, the current beta software for Terramaster they added TRAID+ which looks like it should be a replica of this, but need to have a poke at this also.
You can use SHR on any Synology device, the caveat though is that you must create it on a device that supports it, then you can migrate the drives to the units that don't support it and it will work as normal from there.
Thanks for sharing this, its good to know.
I wonder if Synology would support that if an issue arose. Seems like a legitimate workflow to move a disk array when updating hardware, if less common than new NAS+disks and data migration.
Out of interest, which device did it get migrated to and is it a larger disk count than you would find in a supported device post migration? Was growing the SHR array possible in the unsupported unit?
@@sometechguy I am not sure if Synology would "support" the configuration, as they really don't want SHR being used in an enterprise level device, but aside from not being able to create an SHR, once you move any SHR drive in to the unit as a new configuration and import it, it operates as any other unit and you can do everything you do on a device that allows you to create the SHR, I believe that all they did to "prevent" SHR from being used on enterprise devices is to make it so you can't create it, but importing existing works just fine.
My NAS has 12 x 4TB drives. After being in the enclosure for 7 years, I had two disks fail in the array, both within a week or two of each other; before I noticed that the drive had failed. It was not a Synology NAS, but it was configured for RAID 6. At times I had considered rebuilding it as RAID5, but I really didn't need the space.. The point is, multiple drive failures can and do happen.
Your work is amazing mate. dont stop. Also I would love to see a video about the intel LGA 1700 chipset comparison like you did on AMD
Thank you, I really appreciate the comment and encouragement. Yes, I am looking at the Intel version of this and plan to get something together. Hopefully soon!
Can RAID recovery software work with SHR in case of hardware failure?
Will depend on what software you mean. But placing a failed array into another Synology NAS will work, and as SHR uses standard Linux tooling, you can also recover it in a non Synology device, but this requires some know how, and there are guides available.
Any raid recovery will need enough data to be intact though, so will depend how many disks were lost and the consistency of the file system etc. This is why having regular data scrubs is important.
@@sometechguy Doing a quick internet search I have found RAID recovery software to include 1.diskinternals, 2. Stellar Data Recovery, 3.reclaime, etc... I am sure there are others. None of them say SHR but they do have like RAID5 and RAID6 which is close to SHR. I did have data loss in a RAID system a few years ago and think its because the drives may have been SMR (Shingled magnetic recording ) hard drives which the manufacturer hide from the public. It was not listed anywhere on the drives or even in the specs.
I wish Synology did not call it Data scrubbing because that names makes me think they are going to scrub(delete) the data on the drive. They should instead call it like drive scan and repair or something like that.
@@ThrowtheJ.DownTheWell , sorry for the slow reply on this, I missed it originally.
SHR actually uses the MD implementation of RAID under the hood, which is the linux standard. For the file system, it depends on the configuration, but Ext4 is possible, with BTRFS being the recommended system. Again, these are linux standard, so it would take some knowledge but the array should be recoverable in a device other that Synology as long as the right things are in place. I would say though, it would require some knowledge and wouldn't work 'automagically'.
As for 'scrubbing', this language is inherited and didn't come from Synology. Its the usual work to describe a check and repair of files in systems that support it, notably BTRFS (used by Synology) and XFS, a very popular highly robust file system also available for *nix based systems. So Synology just went with the industry nomenclature there I think.
But you are correct that Synology could have used a more abstract and 'user friendly' naming in the UI. They would likely have had criticism for doing that also though, for 'hiding' the function etc.
Looking forward to seeing your follow up video as I am not sure if I should enable SHR-2 on my 1821+ if I am going to us iSCSI am I going to lose any performance?
Hi, and thanks!
I am not sure if the filesystem choice has any meaingful impact on iSCSI performance, and its a good question. I have used iSCSI on one of my NASs, but not for a heavy duty workload, but I gather the Synology iSCSI implementation has historic performance issues, and NFS would likely perform considerably better. And I don't believe this has a direct linkage to the underlying filesystem, more the iSCSI implementation itself. Maybe separating the iSCSI onto separate interfaces, and using MPIO can help, but it sounds like a broader problem than that. I would look into that issue before choosing iSCSI on Synology.
Overall, I have seen numbers of around 1% performance hit, but I have not benchmarked them as yet.
Truly great video!
My big question as a home user is what is the minimum drive NAS I'd need to get a full benefit from SHR? By the graphics I'd say a 5-disk setup. But would similar benefits appear when starting with one 10GB and one 14GB, and then adding another 14GB? I guess I should have asked the question as, where do the benefits begin?
You can't convert a traditional RAID into SHR, without recreating the volume. So its best to start with it from the beginning. But separating the benefits of SHR and BTRFS, as they are distinct (BTRFS gives you data scrubbing and snapshots etc), SHR really provides the most significant benefits once you want to use disks of different sizes, which gives you flexibility in the short term, and the ability to grow the array later cost effectively.
So if you start with, as an example, 2 x 10TB disks. With RAID you would continue to add 10TB disks (you can add larger but you can't use the additional capacity). With SHR, you can start to add 14TB disks, or 18TB etc as the price point makes sense. And once you have enough of the larger disks (2 with SHR, 4 disks with SHR-2) then you start to get the benefit. With RAID, you would need to go back and replace all disks to get that benefit.
So, I would start with whatever disks you plan to start with, lets say 2. And then you can add disks that make the most sense in the future based on price per/TB and get the benefits of that. Once your NAS is full, or the disks start to age or fail, you can replace those disks with larger disks and start to get increased capacity as you replace.
And to address your example directly. If you chose SHR and added a 10TB and 14TB disk, you would get 10TB of usable space. (10TB Parity with 4 unused), the same as with RAID1. Once you add the second 14TB disk, with SHR you would get 10+10+4=24Tb (with 14TB used for protection), where as RAID 5 would only give you 10+10=20TB (with 10TB for protection and 8Tb unused).
Hope that is clear, and helpful.
@@sometechguy yes, it's clear. Thank you! This will be my first NAS, so there's no conversion issue... I'll just start with SHR. I'm well-familiar with Btrfs as I've been using it in all of my Linux machines the last three years or so. To get my feet wet I think I'm going to use one of my several Linux laptops with TrueNAS to start. My data is all sentimental-type stuff... Family photos, documents, etc. No media production files or anything that requires fast transfer speeds.
If you start for example with one 12tb drive , the next drive must be same capacity or higher ? Or can you have a lower capacity for instance 8gb? Another question, what happends with the unused space? Is it lost or can something be donde with it? Thanks
To add a drive to an existing SHR (or a traditional RAID), it has to be at least as large as the smallest existing drive, this is because this drive (the 12Tb in your example) will be the size of the first logical disk. Any new drives have to at least have a 12Tb available space to be added to that.
On the second question, the space isn't usable initially, and this was the 'red' space shown in the graphics. But as you add other drives, it will become usable. For SHR1 you will need at least 2 drives of that size (Shown at 6:55 in the video), and for SHR2, at least 4 because SHR2 needs two parity partitions.
Can you recover data from an SHR in the event of equipment damage to the synology but undamaged hard drives via some kind of RAID software recovery program?
You can, as it just uses standard BTRFS and Linux tooling. However, I would say it needs significant know how, or maybe a lot of research if you don't have it. I had a NAS fail and I took the disks out and put them in the warranty replacement and it just ran. But restoring in a non-Synology box would take some work.
Definitely, for any disk or NAS or other store of important data, do not neglect backups. Failures are not at all common, but they can happen.
Thanks for the question.
I assume Synology has a hardware problem or a software problem with SHR if the limit on the number of hard drives is exceeded. The reason may be reasonable. Standard RAIDs have existed and been available for a much longer time. In the days of Amiga computers, remember SCSI hard drive validation started if the power went out unexpectedly. Nothing could be written to the SCSI hard drive if validation was in progress. After validation, the hard drive worked normally again. A protection like this could help with RAID systems, but many people could get nervous waiting for a complete check and repair.
of what I understand, SHR will result in uneven performance? I mean, let's say a 10GB file is written on "last" two disks (in your graphic), read/write speeds will be limited to those two disks, instead of combined speed of 8 disks? or am I getting it wrong?
My expectation would be that you are correct. If it's SHR-2 then the data has to be on at least 4 disks. If its SHR, it has to be on at least 2 disks. But in both cases, the more disks you have the more spread the data will be and there could be performance changes. But as both SHR1 and 2 with different sized disks may produce different logical RAIDs over different numbers of disks, then there is the potential for variance.
But for smaller NAS (home and SMB), this likely isn't going to cause any measurable issue as its often the network bandwidth that is the bottleneck anyway. But this is an interesting point, and it could even be one of the reasons they don't offer this no the enterprise products.
Thanks a lot for your helpful videos, you helped me chose which device to buy (DS923+). One question: If I add 4x 4TB drives with SHR, how is it posible that a single 4TB drive is used to "backup" the rest of my data (12TB)? Or did I misunderstand? Thanks in advance!
Hi, and thank you for the comment.
The 4Tb drive doesn't keep a copy of all the data from the other 3, rather there is 'parity data' that uses the capacity of one drive, though that data is spread around and not just kept on 1 drive. Any 3 of them remaining drives hold enough information to fill in the blanks if a disk is lost, or some data on that drives becomes corrupted.
If you want to dig a bit further into how this actually works, I do have a video (th-cam.com/video/2Dovoc9LP34/w-d-xo.html) that explains RAID parity. And SHR uses these mechanics under the covers, along with some other things.
Glad to be of help, and enjoy your new NAS!
@@sometechguy That's a great explanation, I really appreciate it! Thanks a lot and even if one more doesnt really matter, you just got a new subscriber 😌
It all helps, thank you!
If I just purchased a Synology NAS and only have one 16TB Iron Wolf drive…what should I do with regards to RAID? Do I need more disks in order to get the Synology 418 NAS to work on arrival?
With 1 disk, you can’t get any fault tolerances. e.g. you won’t be protected against a disk failure, which may be obvious. So this means you can’t configure traditional RAID, as you need two disks to do that, of which one would just be used as failure protection for the other.
But you can configure the unit for SHR with one disk, again no fault tolerance to protect from a failure but you can add a disk after you start to do that. When and if depends on your data and how much you need to protect against loss.
The advantage of SHR here is you can then add other disks later, get your fault tolerance and grow your storage. If you want to visualise this, you could try and Synology disk calculator linked in the comments.
@@sometechguy AWESOME! I should have watched your full video before asking July question lol. I have seen many videos but none of them was as clear as yours. Thanks again.
thanks for your video. I just installed my synology to Raid 5 untill i saw your video. And it has 35 Tb data in it. I am curious is there anyway convert my raid 5 to SHR without losing my data ? or i have to carry all my data to somewhere else and start from scratch ? I have DS1621+ thank you
Synology supports converting between certain RAID types, and moving from SHR1 to SHR2. However, you can't move from traditional RAID to SHR in a non-disruptive manner. Unfortunately, if you want to move to SHR, you would need to move the data off, start a new clear array as SHR, and then move it back. Or of course restore from an existing backup. But check its integrity first.
So sorry, this isn't the answer you were probably hoping for.
@@sometechguy Thank you so much for your respond I will do that and definately i will switch to SHR
wow your work is amazing, that diagram was superb!! love it so much, can you do SHR vs ZFS ? synology vs truenas
Thank you, appreciate the positive feedback! I have some content coming comparing the under the hood of BTRFS vs EXT4 on Synology, and I also want to do some other comparisons, including ZFS, which is a really interesting topic.
so SHR allows for smooth replacement of grandparents? x)
Strange imagery, but very helpful video.
I watched this video months ago.. and when I bought new HDDs for my NAS, I switched to SHR. After using it for 2 months.... I switched back to Raid 5. Raid 5 gives me 20% more storage area and seems faster in transfering files.
I can't see why that would be the case, unless you are comparing SHR-2 to RAID 5. Is that possible?
At the minimum SHR-1 should provide the same storage as RAID5, SHR-2 would provide at the minimum the same capacity as RAID-6. But if you are comparing a 5 disk array with RAID5 vs SHR-2, you may well see what you see.
On the performance side, there could be a slight overhead with using BTRFS over Ext4, but if you are using the same filesystem to compare RAID5 to SHR2 then again, this is likely because SHR2 will use RAID6 under the hood and RAID6 is going to be a little slower than RAID5 due to the additional parity calculations. But I would guess it isn't really large and in many cases, the bottlenecks would somewhere else, such as the network.
Maybe check your disks into the RAID calculator at www.synology.com/en-uk/support/RAID_calculator and see how RAID5 compares to SHR1 and SHR2 and see if this is the issue.
Wonderful presentation and excellent detail! Sub and Like added!
Thank you so much 👍
Do we still see single array volume. Actually SHR creates three raid array...
Yes, it’s still presented as a single volume. So you can’t really see what is happening under the covers.
0:51 to say that it is "especially suitable for users who are new to RAID technology" is, IMHO, no way implying that it's not for advanced or experienced users. That's nonsense. I have managed my own RAID on other machines and it's not a big deal. But, SHR is brilliant in its ability to mix different sized HDD's and that has nothing to do with experience it's just a great way to create flexibility in choosing your drives, now and in the future.
I think you are agreeing with me here, that SHR for sure is not only for new or ‘inexperienced’ users, and the point here is that it’s a great choice for many applications. But my take away at least from the Synology article, is that it implies it’s primarily for newer users. But maybe that’s just my reading.
Do you use BTRFS vs EXT4?
I use BTRFS on all my Synology devices. It offers a number of advantages over Ext4, both for functionality (snapshots) but also data integrity (data scrubs), which Ext4 doesn’t.
Awesome this is super helpful
Thank you!
only bad thing about Synology is the 108TB max limit
I've been intrigued by RAID in the past but never actually used it. A number of reasons:
If the raid hardware, e.g. motherboard or NAS fails, would the array be picked up by a replacement or would all data be lost? I don't actually know the answer to this one but it has put me off.
If a drive in a raid array fails then rebuilding puts the remaining drives under strain, and it can take a long time.
Lack of portability, having a separate external drive allows you to take a copy somewhere while a copy remains at home.
Common cause failure, using similar drives and running them the same amount could increase the chances of all drives failing at a similar time. Having an external drive occasionally used to backup mitigates this.
For always on NAS, running costs are higher than a cold backup.
Cold backups are also resilient to data ransoming.
For these reasons I typically have a large drive in my main computer and an external drive that I do a backup to every month or so. Important work in progress I save on Dropbox. I'm fortunate that my data collection has always been smaller than the largest drives on the market so I've only needed two drives rather than a higher multiple of two. Just my PoV.
It will depend on data volume, but RAID should never be an alternative to a backup, it’s an extra real time mechanism for data protection. For example, if you backup to an external disk, there will be a delta between your live data and your backup, which could be a day, or many months, depending on how rigorous the backup regime is.
For portability, it depends but you should usually find moving an array between similar hardware should be fine. I once had a mainboard failure on a Synology NAS and the unit was replaced, the disks went into the new unit and the entire thing started like nothing has happened, NAS OS, all volumes etc. it may often be recoverable on other hardware if the software is the same also. But recovering SHR on another device than a Synology is more complex than a basic RAID config.
And yes, replacing a failed disk and recovering an array is a high stress event for other drives. It’s totally fine for them, but if any of them are also verging on failure, it can kill them and I have seen this happen. RAID 6 or SHR2 on Synology reduces that risk, but before I do a RAID recovery, I check all my backups first.
If you don’t have the high capacity requirements, what you are doing is good. Depending on the criticality of data, another backup kept elsewhere is also good. Maybe not for everything, but the things that really cannot be replaced.
Thanks for watching and for the comment, great questions!
Why is the audio so low?
No one else has given that feedback, and it works fine if I watch it. Maybe an issue with your playback?
@@sometechguyPossibly... But other videos are normal volume.
I'm more partial to ZFS as I've been using it for years but not a fan of its lack of expandability apart from adding entire V-devs (Mirrored or Z) which can create their own problems.
ZFS is a great file system, but this isn't supported on Synology. And as you say, I believe it doesn't provide the same expandability options. Synology do a nice job of abstracting this away so it doesn't have to be managed by the user.
Isn't this similar to what unraid does for a while now? - I mean pratically
I believe unraid has a similar implementation though it may be a little different. But of course this video wasn't about unraid. And Synology has also done this for a while, it just isn't always well understood, especially how using different size disks affects availability capacity.
yeah, I'm using unraid now for about a month or so I think it's way easier to understand it's as simple as:
1- your parity disk needs to be same size or bigger than the bigger disk in your array
2- the amount of parity disks you add are equal do the amount of disks that can fail without losing any data.@@sometechguy
It's my list to have a deep dig into unraid, it looks really interesting. So thanks for bringing it up.
I assume the concept of a 'parity disk' isn't really one physical disk, but a logical construct. RAID 4 was similar to RAID5, but rather than having parity spread in blocks across the entire array, it had all the parity on a single 'parity disk', but this caused performance and drive load issues, as every time anything is changed on a disk, the parity needs updating and this means a single parity disk becomes a bottleneck.
9:59 bro showed an old sick person to illustrate a dying hdd 😭😭💀
They are actors and no one was harmed in the making of this video. 🫣
Isn't this what Drobo's have been doing for years?
The stock footage cut-ins re health and legal are priceless
Sure! Here’s a positive comment for you:
"I truly believe that every small act of kindness can crea🙂🙂
Honestly i would stay away from it. I understand the point of putting storage to use but if you hold valuable data worth protecting and not stuff you can easily replace i wouldn't want them on a jbod across 3 raid-5's on the same failure domain (disk), that's just playing with fire
If it can do BTRS then it can run TrueNAS.
Wow, what madness. If you use you RAID system extensively, performance could be atrocious if and when you start accessing data from the same disk simultaneously and which will stress your hard disks. Home users should keep things simple. I hate to see a rebuild of a 8 to 12 disk RAID 5/ 6 when you have a disk failure.
I don't recommend RAID at home unless it's RAID 0 for performance with your backups of course. I would just create a sync, it's way easier to manage. The performance of anything other than 0 is pretty bad, and rebuilding the array is pretty painful.
Don't ignore time cost when you need to rebuild the system, everyone should use at least RAID 1 or SHR
@@xlion it's even faster than rebuilding RAID. Less data to gather and copy around. If the NAS is a 5 bay, I would buy 4 of the same capacity for RAID 0, daily use, and a bigger one for backups. Then you can use whatever tool you want to sync the data, you can use rsync which is int the control panel, or one of the many ones they have in the market. This way you also don't lose access and have fast speed access to your data
*Synology?! You want me to have a cult be responsible for all of my data?!*
I am sure there is a reference to a cult called SHR that I am missing here. 😌
when you said that SHR was based on BTRFS, right at the beginning, I stopped watching. No it is not.
Maybe it would have been worth watching, and this video th-cam.com/video/aLoajg9yFxg/w-d-xo.html goes deeper into what is really happening under the hood.
SHR has BTRFS as a prerequisite, and it is because it uses its file system, its Copy on Write mechanism for write integrity and its scrubbing capability for ongoing protection against bitrot. So while its true that it uses MD for RAID management and data protection, it is built on BTRFS for many other capabilities, with custom integration with MD for data parity management.
Would you mind sharing what's the correct way of putting it if he is inaccurate?
@@sometechguy raid or shr, and ext4 or btrfs, are independent choices, one does not depend on the other. In particular, you can have btrfs on raid, it does not require (or based on) shr. That is my point - did you mean something else by 'based on' ? "SHR has BTRFS as a prerequisite" is just wrong.
@paulbarnett1461, I take your point. BTRFS is the recommended filesystem choice for SHR, but you are correct it isn't _only_ available choice for BTRFS, which I suggested in the video. So thank you. But to be clear, I had not suggested RAID was not available on BTRFS or that the choice was EXT4/RAID vs BTRFS/SHR.
This video was more focused on the benefits of SHR over traditional RAID, and why it's a good choice. BTRFS also provides other benefits missing from EXT4, in this context especially, the ability to perform data scrubs to check file integrity. And this is why its recommended.
I appreciate you pointing this out however, so thank you.
@@sometechguy I think we mostly agree, but not in some details. btrfs does not depend on shr, not even for data scrubbing. you can (and should) do scrubbing, and therefore btrfs, even on raid (I do this)
And because btrfs does not require shr for anything it can do (at least, not in DSM), it becomes not relevant to a discussion of raid vs shr.
If it had been me, I'd have done two videos, one on raid vs shr, and a separate one on ext4 vs btrfs.
I watched this and your other videos, and enjoyed them (apart from the 'btrfs requires shr' idea) - and I agree with all your recommendations:
use shr
use btrfs, & enable checksums (and maybe compression, but not for VMs)
scrub regularly
Suppose you had a large (say 100tb) raid pool, with a large (but not that large, say 50tb) volume. I claim you can add another 50tb volume as btrfs with no outage. (and I've done this (with smaller numbers)). You are implying that you'd have to backup all existing data, delete and recreate the pool (as shr), recreate the original volume (now as btrfs), restore the original data, and only then create the 2nd volume as btrfs. I don't intend to argue this further. :-)
OMG thank you so much. You really helped this NAS noob 😭
Pleasure to help, glad you found it useful. 👍