Unraid is an amazing solution for a home server thanks to its scalability, since you don't have to buy lots of drives at the beginning like you have to with ZFS. Imagine buying 6 10TB drives filling them to 50% after 3 years. It means you are using drives and degrading them even when you are not using the space on them. With Unraid, you can expand at any time by adding drives one by one. And now you can create ZFS pools too if you really want.
Yes there are points that will be in a dedicated UnRAID video here soon and these are among them. Spindown on disks, I got ZFS much faster, growth of odd sized drives, and more will be included in that.
"It means you are using drives and degrading them even when you are not using the space on them." not sure what that is supposed to mean, regardless of what kind of ZFS pool you create you are using all the drives in the pool unless they are dedicated spares. If you mean not filling all of the space on individual drives on a per drive basis that pretty much boils down to the difference in design philosophy, reading and writing lots of data to a single disk is slow. non zfs Unraid gets around this with NVME cache but that process is somewhat clunky and not really an option for the intended use of ZFS. While Truenas isn't as simple as non zfs Unraid for expansion you can add vdev's so you can start with as little as 2 or 3 drives and expand 2/3 drives at a time. So your 6 10tb drive example you can start with either a 2 drive mirror or a 3 drive raidz1 and expand when you need to. Yes you 'lose' more disk space than a single parity drive but you could also lose more disks before you lose any data and in most situations it will be faster.
I don't think its that bad with TrueNAS, really you should be starting with 4 drives plus anyway on any NAS platform so you have redundancy. If you want to start off small and expand with your data this is how I would do it on TrueNAS. Start off with a ZPool with one VDEV, RaidZ2 with 4 drives, so you have a fault tolerance of two drives. Lets say this is 4 x 6TB = 12TB of usable space, a fantastic start to a home lab or movie storage server. Now lets say you fiil up that initial vdev of 4 x 6TB RaidZ2 drives and you want to add more drives, all you need to do is add another 4 drives (of whatever size you want, e.g 4x 10TB) and then add that to the pool so it matches the parity. this will give a zpool of 12TB+20TB total 34TB, physically you will see the zpool expand and front end wise everyone would be none the wiser that they are using multiple groups of RaidZ as whatever share you have it will still show as one. In the end you maintain fantastic fault tolerance of your data, to the point were you would need to have more than 2 disks to fail per group to lose data, and speeds as before will be great too.
I think the general idea is that you have to add whole vdevs instead of the flexibility of adding individual drives. If you're configuring a first nas on a budget, the flexibility of adding another single drive because it's on sale (low $/TB), then saving up to eventually add another one is valuable. Honestly, it's a considerable selling point for fund-limited individuals that still want DIY solutions and maybe can't afford to stack on $400-$1000 jumps, but still want to increase capacity/reliability. The ability to move from 3:1 data:parity up to a 4:2 configuration is a valid use case for those that may only be able to afford 1-2 drive jumps of different sizes, and adding additional space/parity being worthwhile justifications to rebuild the array without losing data. I believe this is possible with unraid, but not with ZFS pools. It seems like Truenas is better when configuring a larger setup at the start or larger incremental jumps when upgrading, compared to the piecemeal (albeit performance reduced) unraid format. Please let me know if I'm incorrect, as I'm looking to configure a video editing nas and am struggling with the upfront cost/increment jump or truenas, but am reluctant to take the performance hit of unraid.
@@KrysRevampsI think the idea was that most people don't want to have to buy 4 drives at a time to expand their storage. My Unraid server uses 16TB HDDs. Replacing/adding 4 at a time would be over $1200 vs replacing 1 for ~$310 and can still lose 2 drives before data loss. ZFS is cool, but unnecessarily costly for most home and prosumer users.
For a share which resides on a pool only, you can enable “permit exclusive shares” (see global shares settings). With this setting enabled, the share will be accessed directly and bypassing fuse, this is specifically done to take advantage of zfs speeds. See also the release notes of Unraid 6.12.0 which gives more information.
Was going to say this. FUSE is highly likely to be the issue here. Dont know if it would match truenas but I'd imagine it will vastly improve the unraid performance anyway.
There may be a setting I missed in UnRAID to avoid hitting FUSE on the pool share that I missed. I'm checking on it but they need to call that out in the UI at the location your creating the share or the pool from if this is so. Having it in a separate screen in settings isn't ideal.
@@DigitalSpaceport Yeah I agree if you're having to hunt for essential settings then that's not intuitive. This is unfortunate especially since it's not free...
I asked on the unraid forum - they suggested using exclusive shares. There are mentions of 200%+ improvement in ZFS performance thru doing so on the unraid forum.
I think I found the issue. Unraid shares, that go through the user mounts has a high overhead and limits the transfer speed (for me 600-700mbs), if I set one of the pools to direct share, I can get up to 1.1gb/s transfer speeds. If you redo your test with direct access, you should see better speeds I think the overhead, is because SMB is not truly using more than 1 core (one of mine was pegged), but when I did direct share I was able to use multiple cores
What version of smb is unraid using? I remember the newer version allow multi cpu's to make the transfer faster or something maybe it was a command line I remember back in the day I had a command for smb to make faster, maybe unraid doesn't have the newer version or the command line thing isn't in there?
From my experience Unraid zfs has sync by default on and TrueNAS by default off (async) use "zfs set sync=disabled poolname" and you get the TrueNAS transfers "speed" or very close :)
I'm suspecting that and record size. 1M does have a huge impact on writes for lg files and I'm assuming UnRaid rightly so doesn't set this as it's wasteful if space.
For me both TrueNAS and UNRAID are strictly to be used for file shares I do not really care about other features they provide, TrueNAS is a clear winner here hands down, its probably the default settings it ships with compared to UNRAID, Proxmox is still the best way to go for anything virtualization related. What I would like to see you compare this to is plain old Windows share on a raid0 array and see how that performs.
Did you ever get this working? I'm able to saturate my 10g ethernet using Unraid & windows. Using raid 0 U.2 SSD's and a RAIDZ2 with 8 14TB HC530's. Surely some setting not right. Have you tried bypassing FUSE by going directly to the pool at /mnt/pool vs /mnt/user/pool ? I'm trying it right now, through the FUSE system and SMB and I'm bouncing between 900MB-1100MB/s
My Unraid (Epyc 7502P, 96gb ram, 3.8tb NVME Enterprise drives in the main array an SAS SSD array, plus an NVME array and no matter what settings I try, I cannot get past 700mb/s transfer speeds to unraid (to/from windows machine) If I use Krusader, I can get 1.2gb/s transfer rates between arrays. My Truenas has 10ea 12gb SAS Spinners, with 4ea 1tb 660p cache drives and it hits similar speeds that you are seeing. I wish we could determine, what is causing the slow down. Also since FIO is no longer available, it is hard to see if Unraid is truly using the arrays correctly
Keep in mind that when transferring data stored in bytes, it’s transmitted over the wire in bits. Eight bits to a byte. To get 2 gigabytes per second of data transfer, that’s equivalent to approximately 16 gigabits per second on the wire. If you are using TCP instead of UDP (best practice for data integrity) this significantly increases frame processing overhead at these speeds, even with jumbo frames.
Definitely don't think you missed anything. FreeNas now Truenas has been all in on ZFS for a long long time and they have gotten their OS down to an enterprise grade science. I would be interested in seeing how they perform when configured ideally for their individual use cases eg Unraid + SSD caching tiers vs Truenas pool of mirrors and plenty of RAM no L2ARC/ZIL since they're super use case specific and Truenas recommends throwing memory at it until it doesn't help anymore.
It’s Fuse. You can confirm by comparing write speeds to a single disk share vs. a user share (basically with and without Fuse in the equation). I know that won’t help in this example, but you’d be surprised of the speed difference if you try it. I still stick with Unraid though just for the overall feature set. It performs well enough for me, and ZFS is a welcome improvement over BTRFS.
Something's fishy. Mine has ZFS with optane's acting as metadata devices up to 512KB. On 10Gb network and transfers over NFS i am getting over 800MBps speeds (up and down) for large files without any special tweaks.
Didnt they say in the notes that by default its only 1/8th of your ram that is being utilized by Unraid ZFS? That should be more like 1/2 and can be modified in the config.
I have had multiple issues, issues I think most users can encounter on first setup, however I am doing a dedicated video just on UnRAID and ZFS setup and all the settings to get the best tune for high speed share transfers. Many of the issues I have had are my own fault for setting it up without fully checking into their release notes.
From what I understand, UNRAID doesn't utilize RAM cache, unlike TrueNAS, which is why TrueNAS inherently performs faster in file transfers. UNRAID can certainly enhance its performance by configuring file transfers via SMB to initially use the cache pool SSD (Nvme) and then move to the array (HDD). I'm exploring ways to increase the available RAM for the cache, and I've come across some suggestions involving the command "sysctl vm.dirty_ratio=20." This command sets a percentage of RAM for managing dirty cache in the kernel. Upon applying it, there is a significant improvement in transfer speeds. If anyone has a better understanding of how to optimize file transfer in UNRAID by adjusting RAM settings, that would be great (I'm unsure if this method is valid).
I think ZFS on unraid is just in its infancy and not really what the OS was built for. Unlike Truenas which was built for ZFS and has a very good reputation on being a good enterprise NAS OS. I switched to Truenas Core about 2 years ago from Unraid. And performance is solid.
Yes there are also several items missing from the offering on UnRaid that you really need to have a good performing ZFS pool. How they handle these, in addition to how they handle the main array moving into new releases will be interesting. For my data stores of important data, I'm TN also.
You said that you and enable SMB multi-channel? But doesn't the box only have a single NIC? For ZFS did you try to disabling synchronous writes? Record size? Block sizes?
Yes it did only have a single nic, but I have tried TrueNAS in the past with dual 40gbit nics and SMB but it didnt work for me then either. Someone told me my windows machine needs some special settings to get it to properly function with the 40Gbit and 10Gbit cards in it. 1M size in TrueNAS. Not sure how to set that in UnRAID yet. I to think I had sync enabled. There will be more in depth testing as folks told me some more settings in UnRAID to adjust plus I will be pulling in a bare metal machine also.
@DigitalSpaceport SMB multi requires the two connections, or more, to be the same speed. I ran into this trying to get a 2.5g and 1g to work with my PC with the same two speeds. It only used 2.5g. took some research. But before I got the 2.5g I ran two 1g on each side and got 2g. So I know it works when same speed.
truenas Scale should also take a place here. But dude you cant compare proxmox to truenas. Proxmox is a VM solution and truenas is a storage solution with vm capability. But as I am using freenas/truenas over a decade now I will not change at all.
Yeah I doubt I will do a Proxmox VS Unraid/Truenas video ever and I dont use VMware/hyperV or Citrix. Proxmox 8 on recent AMD chips is SOOO good though. Loving it
To get the best performance over SMB, you need to configure multi channel. Not just enable it, but actually get both the Windows and TrueNAS side configured to use multiple connections. That pool should be able to push even harder.
I am planning an all SSD Unraid server with LSI 9207-8i SAS HBA. Does UnRaid ZFS support Auto-TRIM? I've heard that LSI drivers are a hit or miss with UnRaid... What controller would you recommend for an all SSD server?
if you are going all SSD, then the question is how many SSDs, do you already have the SSDs, what kind of a systems and what size do you want to hit? I would recommend a PCIe M.2 adapter card and get like 4 NVMe in if you are building it all new and that will be smoking fast. The 9207-8i is OK for an internal HBA if you already have the SSDs.
@@DigitalSpaceport"PCIe M.2 adapter card and get like 4 NVMe" . Don't forget to mention that in this case the card should either has its on PCI-E arbitrator chip (these are really expensive cards) OR the motherboard should support PCI-E Bifurcation (which is rare to see unfortunately)
When I first set up my Unraid server with a 10gb NIC I was super disappointed with the transfer performance. There's pretty much no advantage to have any NIC over 1gb. The FUSE system just can't take advantage of the faster NIC's
I would like to know, how often happens that a disk fails according to the experience of the community, so that a raid 0 would fail. According to my experience I had since 30 years only one HDD which failed and that was not in a raid. Therefore, I am using in my unraid home server 4 x 8TB U.2 NVme in raid 0, honestly not because of the speed, rather than because of the space. But I must mention, that all of my data in unraid is being synchronized immediately with a qnap nas and daily backups are being made on a second qnap NAS since many years in order that I dont lose data (only the data in ram will be lost in worst case). The con is in such setup, to rebuild up all the pools and data again, which will take a lot of effort. I wanted just mention, if you secure your data accordingly, you dont really need to avaoid a raid 0 in a home lab environment.
I have 2 PiB of space running on Raid0 spans that are 12 wide. The write performance is awesome and there is very little waste. TrueNAS ZFS with 1MB record size and like you mentioned, appropriate backups mitigate for lack of redundancy in any primary array at the cost of time.
Did you configure how unraid fills the disks? If I remember it correctly unraid fills the disk with the most space at a time. I don't know if that applys to raid0 but if it does it should in theory mean that you only get the speed of one disk as it only write/read on/of one disk.
@@geo2160 yes, but but if you put the drives in for example raidz1 or 2 you can have the system always write to the drie with the most space. As i said i'm not sure it applies to raid0 but in both cases the disks is shown as 1 pool.
Unraid array is slow. I am really dumbfounded by the slow ZFS pool. I just did some testing and I got what I expected from 7SSDs. going to have to look into this more now. Thanks For the video
I would have tried to create a test file on the the Unraid server array/share, something as simple as 'dd if=/dev/urandom of=test.bin bs=1M count=8192' which would establish a baseline to establish how fast Unraid can write directly to the array/share without the network layers being involved. I'm sure there are actual Unraid apps that probably do something similar, in a much more sophisticated manner. The bottom line is to start testing on the host, not over the network.
Curious how much you're seeing as a bottleneck in protocol with smb being tcp vs. something like nfs and udp. Also, might make sure it's using smb multichannel too, should be supported with samba, but never know with windows talking to something NOT windows. Shame these don't support DPU storage offload/acceleration yet to manage volumes on the dpu nic(s) like nvidia bluefields directly.
NFS is very fast, but more single core speed throttled. There are A LOT of tunables around NFS that can get you very high performance but not sure that would be useful for a video? DPU is indeed the future.
@@calhta I understand this. The TrueNAS representative even mentions this later in the video. Yes the data starts flushing to the pool but like I said, until you overflow the ARC you are not really benchmarking the pool. You're just dumping the data into ARC. If your pool writes at 500MB/s and you have 64GB of RAM/ARC, with a 1000MB/s link. You have to wait for (64,000/(1000-500))s 128 seconds is the minimum you should wait before you're hitting the pool as a bandwidth limit in that scenario.
I use a RAID0 array on my primary PC for some game installs via Steam, but yeah, I'd never use it in my NAS. RAIDz1 is the perfect price to security sweet spot for my use case.
@@DigitalSpaceport I had been debating it, but I uninstall games so rarely that it would be wasted space (I splurged for 4.5TB of all SSD storage in my gaming PC). You can still do it, and it is useful in some use cases. For example, your gaming PC storage is limited and you want to play a wide variety of games, or you have several users all on one internet connection downloading games.
Yeah I had Jumbo enabled but I will be retesting UnRAID again here as I got some more great suggestions in and I think FUSE was impacting my results a lot.
To be honest, I am finding myself partial to just installing Debian 12 along with BTRFS and OpenZFS and setting up my own NFS and Samba server. Unraid, TrueNAS, OpenMediaVault, CasaOS are all examples of how computers make things easier while also making things also more difficult. For instance, TrueNAS Scale does not allow for DHCP on more than 1 network card and an LACP LAG is not so easy as their simple gui instructions are broken. OMV was a learning curve with the permissions. Unraid seems to have all these extra tweaks you have to make to get decent speeds. CasaOS seems to be nothing more than a very basic web interface that requires web design knowledge just so you can modify the look of to include moving widgets around.
So basically, Unraid if you don't really need performance and need have different disk capacity. And TrueNAS for performance with the same disk capacity. Got it
I built 2 large NAS’ and moved off Unraid to TrueNAS after 6 months. It’s fine if you have a few drives, but it’s not performant enough, the stability is not ready for prime time, and the support sucks. Unraid as a company have major bugs, even ones related to the core data management functionality that they don’t bother fixing. I’d not trust my data with Unraid again.
@@DigitalSpaceport There are too many issues to list. Poor performance is expected given its design, but the reliability and stability is the biggest issue for me. When the servers get busy the UI freezes, safe shutdowns from the shell are unreliable, if you have too many browsers open connecting to UNRAID UI it crashes, and on and on. One time, I had data that was being migrated, was taking up space, but wasn’t there in any other way. Like a black hole. When the forum moderators sent me to log a bug ticket, Unraid didn’t even want me to reproduce it. They said they’d just “monitor the thread”, but the thread was concluded as a bug. They’re not a serious company. It’s cool how you can easily expand, and even if you lost a drive while you had lost your 2 parity drives, you’d still retain the rest of your data, opposed to losing everything like in TrueNAS, but it’s a bit Mickey Mouse. The container support is better than TrueNAS, but I’d rather Proxmox over both of them.
Check if the network mtu is the same with the rest of your network because i saw for whatever reason your unraid had 9000 mtu on it's network card. check if 1500 mtu on your unraid resolves your issue.
The folks at Zimaboard are doing a marketing push right now but didnt meet my needs for doing a dedicated video, but the OS itself might be interesting to look into more.
from someone who wasted years of life, don't ever touch truenas. Go straight to unraid or windows server; or just use good old usb hard drives. Its basically like comparing a few transistors laying around to a fully working apple computer.
Unraid tho you have the cost factor. Unraid no doubt is plug and play as far as grouping drives and getting going but you have to pay for that privillage. TrueNas is free and tho the UI etc. is more akward IT'S FREE. Major downside for TrueNas tho atleast in my case is i have Marvell ACQ107 10Gbe network cards and tho it recognises them the speeds are not consistant and usually on the low side.
‘And’ - the 5th most common word in the English language. Following ‘the’, ‘be’, ‘to’ and ‘of’. Not the case in this video however, where nearly every sentence starts with ‘and’! :D Some interesting stats: 75 ‘and’s in total, in a little over 16 minutes. That’s an average of 1 ‘and’ every 12 seconds, or 4.69 ‘and’s per minute or about every 5th word. And that’s just a lot of ‘and’s :D And it’s annoying to listen to. And if you think about it, you don’t naturally speak this way. And if you write a script and stick to it, you CAN make a good video that isn’t annoying to watch. And I’m sure you understand by now ;)
@DigitalSpaceport I am, actually. AND I appreciate the vote of confidence. Thank you :) AND I enjoy quality content from many creators. I just couldn't continue watching this. Shame that. I'm quite certain the information I was looking for is in there, somewhere :| If you don't appreciate and learn from constructive criticism, you won't win at life, or YT. My delivery may be... different. But the intent is genuine ;) Also, that 'also' is superfluous. Your statement; 'I bet you are a lot of fun in real life' was sufficient :'D
@@painfull73 It's not what your saying, it's your way of saying it. I'm good with criticism, much less so with your tone of condescending ******* superiority. Thanks for the life advice 😆, I've filed it appropriately.
Unraid is an amazing solution for a home server thanks to its scalability, since you don't have to buy lots of drives at the beginning like you have to with ZFS. Imagine buying 6 10TB drives filling them to 50% after 3 years. It means you are using drives and degrading them even when you are not using the space on them. With Unraid, you can expand at any time by adding drives one by one. And now you can create ZFS pools too if you really want.
Yes there are points that will be in a dedicated UnRAID video here soon and these are among them. Spindown on disks, I got ZFS much faster, growth of odd sized drives, and more will be included in that.
"It means you are using drives and degrading them even when you are not using the space on them."
not sure what that is supposed to mean, regardless of what kind of ZFS pool you create you are using all the drives in the pool unless they are dedicated spares. If you mean not filling all of the space on individual drives on a per drive basis that pretty much boils down to the difference in design philosophy, reading and writing lots of data to a single disk is slow. non zfs Unraid gets around this with NVME cache but that process is somewhat clunky and not really an option for the intended use of ZFS.
While Truenas isn't as simple as non zfs Unraid for expansion you can add vdev's so you can start with as little as 2 or 3 drives and expand 2/3 drives at a time. So your 6 10tb drive example you can start with either a 2 drive mirror or a 3 drive raidz1 and expand when you need to. Yes you 'lose' more disk space than a single parity drive but you could also lose more disks before you lose any data and in most situations it will be faster.
I don't think its that bad with TrueNAS, really you should be starting with 4 drives plus anyway on any NAS platform so you have redundancy.
If you want to start off small and expand with your data this is how I would do it on TrueNAS. Start off with a ZPool with one VDEV, RaidZ2 with 4 drives, so you have a fault tolerance of two drives. Lets say this is 4 x 6TB = 12TB of usable space, a fantastic start to a home lab or movie storage server.
Now lets say you fiil up that initial vdev of 4 x 6TB RaidZ2 drives and you want to add more drives, all you need to do is add another 4 drives (of whatever size you want, e.g 4x 10TB) and then add that to the pool so it matches the parity. this will give a zpool of 12TB+20TB total 34TB, physically you will see the zpool expand and front end wise everyone would be none the wiser that they are using multiple groups of RaidZ as whatever share you have it will still show as one.
In the end you maintain fantastic fault tolerance of your data, to the point were you would need to have more than 2 disks to fail per group to lose data, and speeds as before will be great too.
I think the general idea is that you have to add whole vdevs instead of the flexibility of adding individual drives. If you're configuring a first nas on a budget, the flexibility of adding another single drive because it's on sale (low $/TB), then saving up to eventually add another one is valuable. Honestly, it's a considerable selling point for fund-limited individuals that still want DIY solutions and maybe can't afford to stack on $400-$1000 jumps, but still want to increase capacity/reliability.
The ability to move from 3:1 data:parity up to a 4:2 configuration is a valid use case for those that may only be able to afford 1-2 drive jumps of different sizes, and adding additional space/parity being worthwhile justifications to rebuild the array without losing data. I believe this is possible with unraid, but not with ZFS pools. It seems like Truenas is better when configuring a larger setup at the start or larger incremental jumps when upgrading, compared to the piecemeal (albeit performance reduced) unraid format.
Please let me know if I'm incorrect, as I'm looking to configure a video editing nas and am struggling with the upfront cost/increment jump or truenas, but am reluctant to take the performance hit of unraid.
@@KrysRevampsI think the idea was that most people don't want to have to buy 4 drives at a time to expand their storage. My Unraid server uses 16TB HDDs. Replacing/adding 4 at a time would be over $1200 vs replacing 1 for ~$310 and can still lose 2 drives before data loss. ZFS is cool, but unnecessarily costly for most home and prosumer users.
For a share which resides on a pool only, you can enable “permit exclusive shares” (see global shares settings). With this setting enabled, the share will be accessed directly and bypassing fuse, this is specifically done to take advantage of zfs speeds. See also the release notes of Unraid 6.12.0 which gives more information.
Was going to say this. FUSE is highly likely to be the issue here. Dont know if it would match truenas but I'd imagine it will vastly improve the unraid performance anyway.
Oh, they moved it to a setting for the release. In the later RC versions I remember it being on by default
Nice video! I stand by my last comment on the other video! #TeamTN! All speed an no BS!
There may be a setting I missed in UnRAID to avoid hitting FUSE on the pool share that I missed. I'm checking on it but they need to call that out in the UI at the location your creating the share or the pool from if this is so. Having it in a separate screen in settings isn't ideal.
@@DigitalSpaceport Yeah I agree if you're having to hunt for essential settings then that's not intuitive. This is unfortunate especially since it's not free...
I asked on the unraid forum - they suggested using exclusive shares. There are mentions of 200%+ improvement in ZFS performance thru doing so on the unraid forum.
yes I am doing yet another followup to see if we cant get the MAX speeds out of unraid
I can top that: by 400%. Went from flaky 30 Mb/s to constant perfect 121 Mb/s on 1 Gbit. FUSE has terrible influence on performance it appears
I think I found the issue. Unraid shares, that go through the user mounts has a high overhead and limits the transfer speed (for me 600-700mbs), if I set one of the pools to direct share, I can get up to 1.1gb/s transfer speeds. If you redo your test with direct access, you should see better speeds
I think the overhead, is because SMB is not truly using more than 1 core (one of mine was pegged), but when I did direct share I was able to use multiple cores
What version of smb is unraid using? I remember the newer version allow multi cpu's to make the transfer faster or something maybe it was a command line I remember back in the day I had a command for smb to make faster, maybe unraid doesn't have the newer version or the command line thing isn't in there?
From my experience Unraid zfs has sync by default on and TrueNAS by default off (async) use "zfs set sync=disabled poolname" and you get the TrueNAS transfers "speed" or very close :)
I'm suspecting that and record size. 1M does have a huge impact on writes for lg files and I'm assuming UnRaid rightly so doesn't set this as it's wasteful if space.
For me both TrueNAS and UNRAID are strictly to be used for file shares I do not really care about other features they provide, TrueNAS is a clear winner here hands down, its probably the default settings it ships with compared to UNRAID, Proxmox is still the best way to go for anything virtualization related. What I would like to see you compare this to is plain old Windows share on a raid0 array and see how that performs.
I do have a server 2016 hummm this is a good idea.
@@DigitalSpaceport You can always get that 180 day trial for Microsoft :D
Did you ever get this working? I'm able to saturate my 10g ethernet using Unraid & windows. Using raid 0 U.2 SSD's and a RAIDZ2 with 8 14TB HC530's.
Surely some setting not right.
Have you tried bypassing FUSE by going directly to the pool at /mnt/pool vs /mnt/user/pool ?
I'm trying it right now, through the FUSE system and SMB and I'm bouncing between 900MB-1100MB/s
Yes and you are spot on about FUSE and I had some other issues. Dedicated UnRAID video on ZFS high performance in works
My Unraid (Epyc 7502P, 96gb ram, 3.8tb NVME Enterprise drives in the main array an SAS SSD array, plus an NVME array and no matter what settings I try, I cannot get past 700mb/s transfer speeds to unraid (to/from windows machine)
If I use Krusader, I can get 1.2gb/s transfer rates between arrays.
My Truenas has 10ea 12gb SAS Spinners, with 4ea 1tb 660p cache drives and it hits similar speeds that you are seeing.
I wish we could determine, what is causing the slow down.
Also since FIO is no longer available, it is hard to see if Unraid is truly using the arrays correctly
Keep in mind that when transferring data stored in bytes, it’s transmitted over the wire in bits. Eight bits to a byte.
To get 2 gigabytes per second of data transfer, that’s equivalent to approximately 16 gigabits per second on the wire. If you are using TCP instead of UDP (best practice for data integrity) this significantly increases frame processing overhead at these speeds, even with jumbo frames.
Definitely don't think you missed anything. FreeNas now Truenas has been all in on ZFS for a long long time and they have gotten their OS down to an enterprise grade science. I would be interested in seeing how they perform when configured ideally for their individual use cases eg Unraid + SSD caching tiers vs Truenas pool of mirrors and plenty of RAM no L2ARC/ZIL since they're super use case specific and Truenas recommends throwing memory at it until it doesn't help anymore.
I feel like I missed so much lol. These topics are dense but heck....just make more videos ammiright?
What's a Turnaz?
It's what we run in the south for our nfz and iscuzziez
It’s Fuse. You can confirm by comparing write speeds to a single disk share vs. a user share (basically with and without Fuse in the equation). I know that won’t help in this example, but you’d be surprised of the speed difference if you try it. I still stick with Unraid though just for the overall feature set. It performs well enough for me, and ZFS is a welcome improvement over BTRFS.
Agreed and now tested and confirmed. BTRFS has taken forever to not fix several very problematic issues 😑
Have considered putting the question in the unsaid community? It would be interesting to see what they come up with or not.
Something's fishy. Mine has ZFS with optane's acting as metadata devices up to 512KB. On 10Gb network and transfers over NFS i am getting over 800MBps speeds (up and down) for large files without any special tweaks.
Didnt they say in the notes that by default its only 1/8th of your ram that is being utilized by Unraid ZFS? That should be more like 1/2 and can be modified in the config.
I have had multiple issues, issues I think most users can encounter on first setup, however I am doing a dedicated video just on UnRAID and ZFS setup and all the settings to get the best tune for high speed share transfers. Many of the issues I have had are my own fault for setting it up without fully checking into their release notes.
From what I understand, UNRAID doesn't utilize RAM cache, unlike TrueNAS, which is why TrueNAS inherently performs faster in file transfers. UNRAID can certainly enhance its performance by configuring file transfers via SMB to initially use the cache pool SSD (Nvme) and then move to the array (HDD). I'm exploring ways to increase the available RAM for the cache, and I've come across some suggestions involving the command "sysctl vm.dirty_ratio=20." This command sets a percentage of RAM for managing dirty cache in the kernel. Upon applying it, there is a significant improvement in transfer speeds. If anyone has a better understanding of how to optimize file transfer in UNRAID by adjusting RAM settings, that would be great (I'm unsure if this method is valid).
I am struggling with truenas core not being able to transfer over 10Gbit/s despite having a 100Gbit/s nics
What processor are you running? NFS or SMB? Is it capping on a single thread at 100% when transferring?
@@DigitalSpaceport a pair of epyc 7642's. Using SMB and it does not seem to be capping out the single thread on the cpu.
try NFS and use the tune info from here fasterdata.es.net/host-tuning/linux/100g-tuning/
I think ZFS on unraid is just in its infancy and not really what the OS was built for. Unlike Truenas which was built for ZFS and has a very good reputation on being a good enterprise NAS OS.
I switched to Truenas Core about 2 years ago from Unraid. And performance is solid.
Yes there are also several items missing from the offering on UnRaid that you really need to have a good performing ZFS pool. How they handle these, in addition to how they handle the main array moving into new releases will be interesting. For my data stores of important data, I'm TN also.
You said that you and enable SMB multi-channel?
But doesn't the box only have a single NIC?
For ZFS did you try to disabling synchronous writes?
Record size? Block sizes?
Yes it did only have a single nic, but I have tried TrueNAS in the past with dual 40gbit nics and SMB but it didnt work for me then either. Someone told me my windows machine needs some special settings to get it to properly function with the 40Gbit and 10Gbit cards in it. 1M size in TrueNAS. Not sure how to set that in UnRAID yet. I to think I had sync enabled. There will be more in depth testing as folks told me some more settings in UnRAID to adjust plus I will be pulling in a bare metal machine also.
@DigitalSpaceport SMB multi requires the two connections, or more, to be the same speed. I ran into this trying to get a 2.5g and 1g to work with my PC with the same two speeds. It only used 2.5g. took some research.
But before I got the 2.5g I ran two 1g on each side and got 2g. So I know it works when same speed.
Is UnRAID skipping the RAM altogether as cache for its pools?
it seems like it is but there is a whole video dedicated just to UnRAID for this all.
truenas Scale should also take a place here. But dude you cant compare proxmox to truenas. Proxmox is a VM solution and truenas is a storage solution with vm capability. But as I am using freenas/truenas over a decade now I will not change at all.
Yeah I doubt I will do a Proxmox VS Unraid/Truenas video ever and I dont use VMware/hyperV or Citrix. Proxmox 8 on recent AMD chips is SOOO good though. Loving it
To get the best performance over SMB, you need to configure multi channel. Not just enable it, but actually get both the Windows and TrueNAS side configured to use multiple connections.
That pool should be able to push even harder.
Can windows 11 Pro run SMB MC? I want faster 😃
@@DigitalSpaceport Yes, even 10 supports it
Point the shared directly to the drives and not the user folder
I am planning an all SSD Unraid server with LSI 9207-8i SAS HBA. Does UnRaid ZFS support Auto-TRIM? I've heard that LSI drivers are a hit or miss with UnRaid... What controller would you recommend for an all SSD server?
if you are going all SSD, then the question is how many SSDs, do you already have the SSDs, what kind of a systems and what size do you want to hit? I would recommend a PCIe M.2 adapter card and get like 4 NVMe in if you are building it all new and that will be smoking fast. The 9207-8i is OK for an internal HBA if you already have the SSDs.
@@DigitalSpaceport Does UnRaid ZFS support Auto-TRIM with LSI 9207-8i?
@@DigitalSpaceport"PCIe M.2 adapter card and get like 4 NVMe" . Don't forget to mention that in this case the card should either has its on PCI-E arbitrator chip (these are really expensive cards) OR the motherboard should support PCI-E Bifurcation (which is rare to see unfortunately)
When I first set up my Unraid server with a 10gb NIC I was super disappointed with the transfer performance. There's pretty much no advantage to have any NIC over 1gb. The FUSE system just can't take advantage of the faster NIC's
I would like to know, how often happens that a disk fails according to the experience of the community, so that a raid 0 would fail. According to my experience I had since 30 years only one HDD which failed and that was not in a raid. Therefore, I am using in my unraid home server 4 x 8TB U.2 NVme in raid 0, honestly not because of the speed, rather than because of the space. But I must mention, that all of my data in unraid is being synchronized immediately with a qnap nas and daily backups are being made on a second qnap NAS since many years in order that I dont lose data (only the data in ram will be lost in worst case). The con is in such setup, to rebuild up all the pools and data again, which will take a lot of effort. I wanted just mention, if you secure your data accordingly, you dont really need to avaoid a raid 0 in a home lab environment.
I have 2 PiB of space running on Raid0 spans that are 12 wide. The write performance is awesome and there is very little waste. TrueNAS ZFS with 1MB record size and like you mentioned, appropriate backups mitigate for lack of redundancy in any primary array at the cost of time.
@DigitalSpaceport thank you very much for the info
Did you configure how unraid fills the disks? If I remember it correctly unraid fills the disk with the most space at a time. I don't know if that applys to raid0 but if it does it should in theory mean that you only get the speed of one disk as it only write/read on/of one disk.
@@geo2160 yes, but but if you put the drives in for example raidz1 or 2 you can have the system always write to the drie with the most space. As i said i'm not sure it applies to raid0 but in both cases the disks is shown as 1 pool.
Only if you use unRaid's array, that's just JBOD with parity drives. ZFS uses all the drives more or less equally.
Unraid array is slow. I am really dumbfounded by the slow ZFS pool. I just did some testing and I got what I expected from 7SSDs. going to have to look into this more now. Thanks For the video
I would have tried to create a test file on the the Unraid server array/share, something as simple as 'dd if=/dev/urandom of=test.bin bs=1M count=8192' which would establish a baseline to establish how fast Unraid can write directly to the array/share without the network layers being involved. I'm sure there are actual Unraid apps that probably do something similar, in a much more sophisticated manner. The bottom line is to start testing on the host, not over the network.
Okay great point! I am about half way done with a dedicated UnRAID video and will capture that stat. Might use FIO however.
Curious how much you're seeing as a bottleneck in protocol with smb being tcp vs. something like nfs and udp. Also, might make sure it's using smb multichannel too, should be supported with samba, but never know with windows talking to something NOT windows.
Shame these don't support DPU storage offload/acceleration yet to manage volumes on the dpu nic(s) like nvidia bluefields directly.
NFS is very fast, but more single core speed throttled. There are A LOT of tunables around NFS that can get you very high performance but not sure that would be useful for a video? DPU is indeed the future.
Youre benchmarking the memory unless you transfer enough to fill the memory and start writing directly to disk.
Not really, after about 6 seconds its already starting to write to disk.
@@calhta I understand this. The TrueNAS representative even mentions this later in the video.
Yes the data starts flushing to the pool but like I said, until you overflow the ARC you are not really benchmarking the pool. You're just dumping the data into ARC.
If your pool writes at 500MB/s and you have 64GB of RAM/ARC, with a 1000MB/s link. You have to wait for (64,000/(1000-500))s
128 seconds is the minimum you should wait before you're hitting the pool as a bandwidth limit in that scenario.
I use a RAID0 array on my primary PC for some game installs via Steam, but yeah, I'd never use it in my NAS. RAIDz1 is the perfect price to security sweet spot for my use case.
Do you have a steam cache? Is that still a thing? I have been thinking of running one off my UnRAID box.
@@DigitalSpaceport I had been debating it, but I uninstall games so rarely that it would be wasted space (I splurged for 4.5TB of all SSD storage in my gaming PC). You can still do it, and it is useful in some use cases. For example, your gaming PC storage is limited and you want to play a wide variety of games, or you have several users all on one internet connection downloading games.
Windows active protection kills performance… disable and retry to see if that’s it
You should have to benchmark pool performance and network performance separately to figure out where is the bottleneck.
You should have done these tests with NFS not SMB. You would have had significant gains in performance
Jumbo frames? I get 1000+ from unraid over 10gbe
Yeah I had Jumbo enabled but I will be retesting UnRAID again here as I got some more great suggestions in and I think FUSE was impacting my results a lot.
ZFS uses your ram as a read write cache, this might be why you get better read speeds using truenas
he is using zfs on both unraid and freenas
ZFS uses RAM as read cache, provided data is cached there already
My unraid server has a 2TB nvme cashe disk. I can transfer with full 10Gbit/s speed to it.
To be honest, I am finding myself partial to just installing Debian 12 along with BTRFS and OpenZFS and setting up my own NFS and Samba server. Unraid, TrueNAS, OpenMediaVault, CasaOS are all examples of how computers make things easier while also making things also more difficult. For instance, TrueNAS Scale does not allow for DHCP on more than 1 network card and an LACP LAG is not so easy as their simple gui instructions are broken. OMV was a learning curve with the permissions. Unraid seems to have all these extra tweaks you have to make to get decent speeds. CasaOS seems to be nothing more than a very basic web interface that requires web design knowledge just so you can modify the look of to include moving widgets around.
So basically, Unraid if you don't really need performance and need have different disk capacity. And TrueNAS for performance with the same disk capacity. Got it
I would also factor in "fun" and UnRAID leads there imo also.
"So basically, Unraid if you don't really need performance..." don't forget that you have to pay for license
I built 2 large NAS’ and moved off Unraid to TrueNAS after 6 months. It’s fine if you have a few drives, but it’s not performant enough, the stability is not ready for prime time, and the support sucks. Unraid as a company have major bugs, even ones related to the core data management functionality that they don’t bother fixing. I’d not trust my data with Unraid again.
I need to revisit UnRaid on some drives and check this. I've had performance issues also, which is kinda expected. Anything specific you noted?
@@DigitalSpaceport There are too many issues to list. Poor performance is expected given its design, but the reliability and stability is the biggest issue for me. When the servers get busy the UI freezes, safe shutdowns from the shell are unreliable, if you have too many browsers open connecting to UNRAID UI it crashes, and on and on. One time, I had data that was being migrated, was taking up space, but wasn’t there in any other way. Like a black hole. When the forum moderators sent me to log a bug ticket, Unraid didn’t even want me to reproduce it. They said they’d just “monitor the thread”, but the thread was concluded as a bug. They’re not a serious company. It’s cool how you can easily expand, and even if you lost a drive while you had lost your 2 parity drives, you’d still retain the rest of your data, opposed to losing everything like in TrueNAS, but it’s a bit Mickey Mouse. The container support is better than TrueNAS, but I’d rather Proxmox over both of them.
Check if the network mtu is the same with the rest of your network because i saw for whatever reason your unraid had 9000 mtu on it's network card. check if 1500 mtu on your unraid resolves your issue.
Unraid VS CasaOS
The folks at Zimaboard are doing a marketing push right now but didnt meet my needs for doing a dedicated video, but the OS itself might be interesting to look into more.
from someone who wasted years of life, don't ever touch truenas. Go straight to unraid or windows server; or just use good old usb hard drives. Its basically like comparing a few transistors laying around to a fully working apple computer.
Care to explain?
Unraid tho you have the cost factor. Unraid no doubt is plug and play as far as grouping drives and getting going but you have to pay for that privillage. TrueNas is free and tho the UI etc. is more akward IT'S FREE. Major downside for TrueNas tho atleast in my case is i have Marvell ACQ107 10Gbe network cards and tho it recognises them the speeds are not consistant and usually on the low side.
‘And’ - the 5th most common word in the English language. Following ‘the’, ‘be’, ‘to’ and ‘of’.
Not the case in this video however, where nearly every sentence starts with ‘and’! :D
Some interesting stats:
75 ‘and’s in total, in a little over 16 minutes. That’s an average of 1 ‘and’ every 12 seconds, or 4.69 ‘and’s per minute or about every 5th word.
And that’s just a lot of ‘and’s :D And it’s annoying to listen to. And if you think about it, you don’t naturally speak this way. And if you write a script and stick to it, you CAN make a good video that isn’t annoying to watch.
And I’m sure you understand by now ;)
I bet you are a lot of fun in real life also
@DigitalSpaceport I am, actually. AND I appreciate the vote of confidence. Thank you :) AND I enjoy quality content from many creators. I just couldn't continue watching this. Shame that. I'm quite certain the information I was looking for is in there, somewhere :|
If you don't appreciate and learn from constructive criticism, you won't win at life, or YT. My delivery may be... different. But the intent is genuine ;)
Also, that 'also' is superfluous. Your statement; 'I bet you are a lot of fun in real life' was sufficient :'D
@@painfull73 It's not what your saying, it's your way of saying it. I'm good with criticism, much less so with your tone of condescending ******* superiority. Thanks for the life advice 😆, I've filed it appropriately.
@@DigitalSpaceport You're welcome. Wish you all the best mate :)