@@Jims-GarageI much prefer using LXC for sharing Proxmox storage, then all other LXCs on the host can use it directly, rather than resharing. Plex is a natural use case here.
LXC requires additional shenanigans if you have ZFS and want to use Docker. By default it will use VFS driver which not only is slow but doesn't support copy on write so Dockers layers instead of megabytes will take gigabytes of disk. That's why you have to manually format zfs subvolumes to something else (xfs, ext4) and keep their naming properly if you want to preserve ability to do backups from webui. I've used to create LXC, create additional zfs subvolume, format it to xfs, mount it as a second drive to LXC, copy /var/lib/docker to it and replace original /var/lib/docker with symbolic link to docker folder on that drive.
I’ve got a dedicated NAS, so this won’t necessarily be my use-case, but it does clear up some confusion I had on the mapping of drives. Excellent tutorial. I’m going to use this as the basis for a FOG storage instance. You may want to check that out project out, as it’s a pretty cool way to swap out environments for non virtualized hardware. I use it to swap out hypervisors for testing on different hardware. It’s a no-frills, but extremely useful project.
Perhaps do a follow-up to this by adding additional users with their own "folder" and permissions. Is there a way to hide specific folders from other users? Also, connecting to it using an FQDN for those users outside the network.
This is a great topic, Jim. Have you thought about cockpit and zfs-manager module to manage snapshot, replications, etc? Potentially, samba shares also. I think it might be easier and safer in the long run. Great work on your homelab series. Clear and complete instructions make it a pleasure to follow your videos every time.
As always, well explained and demonstrated. I'm not sure the VM + Docker is really necessary, unless you just really want to run SMB in the virtualized environment instead of directly on the Proxmox host. I've been running CIFS shares directly on my Proxmox host for a couple years now, specifically to share OS.iso image files for easy cataloguing and sharing of all my various flavors of Linux and Windows, and the connectivity works the same as this example. Either way, the data is shared out transparently to the target, it's just a different method to get there. Thanks for the video!
Thanks, I do mention that you can directly serve from the host and an LXC, but I prefer to let Proxmox do just Proxmox, and I like the simple repeatability of Docker. But definitely, if it works it works
@@Jims-GarageThe problem is, this way you're virtualizing a SCSI controller (that's virtio for you), that is presented with a ZFS dataset created for the VM, and then you're also putting ext4 on top of that. Each read/write operation has to update things in ext4, go through the virtualized SCSI controller, and write to underlying ZFS. And then Docker and Portainer on top of that... honestly, it would be easier to just apt install samba and avoid yet another abstraction layer. Even better, you can install cockpit with the plugin from 45drives, so you can manage your Samba config from a web UI tuned for working with a storage server, basically the same as 45Drives Storinator units - instead of faffing about with environment variables in docker-compose (which may not expose functionality of samba that you need further down the line). A smarter way to go about it would be to either pass in a HBA (so the VM can access drives directly), or if you don't have one, you could do a raw block device passthrough to the VM and set up ZFS inside the VM. That way you could create new datasets, use snapshots and other ZFS features from inside the VM. Perhaps having different shares with different settings for things like compression, deduplication, atime and so on.
@@chrisa.1740 I have the same thoughts. For few years I used VM, but now transformed all my services to LXCs containers which are at least 5 times more efficient and less power hungry in comparison to VMS... Finally all what I ran on strong PC currently run on small NUC which use 20 Watts instead of 100...
Great video, I actually do it via LXC container. What would be cool would be a ISCSI target on the same VM and having it mounted in windows, some apps dont play well with files on a samba share. Have greate weekedn, cant wait for the nexit video :) By the way, getting a Minis Forum MS01 delivered in the next week or so for dedicated firewall, cant decide if I wanna play with Sophos or OPNSense, can I get a demo of Sophos to trial for a month or so?
I'm using a LXC container to make my samba server and passing the full ZFS pool , added webmin to setup the share points. Being on a LXC I just give it own directory and use a mount point to it , no passing it the full drive . I got a pair of sandisk 960gb SSDs from Maplin and not had a real problem with them , I guess you have a story about yours. Most of my SSDs are samsung 870-QVOs 6 x 4TB total plus 2 crucial mx500 4TB drives .
Any reasons why you chose webmin over cockpit to manage the shares? I've been investigating web frontends for server management. Webmin is the grandaddy but not clear (to me, at least) how capable the interface is with modern server technologies. Redhat still seems to be actively extending and promoting cockpit. 45Drives has re-written the cockpit zfs module (Houston). Looks very capable.
I don't think you can expand raidz yet. It should be implemented on open zfs 2.3. Expanding mirrors you're tied to the smallest capacity drive within the same vdev but usually you expand mirrors pool by adding mirror vdevs to the pool so you can have mixed drives capacity vdev within the same pool
Mirrored disks is my preferred setup in a homelab environment. ZFS lets you add drives to the mirror (more redundancy) or add more mirrors to the pool (more storage space). You can even do the reverse and remove drives. In my opinion the most efficient solution for the homelabber.
@@chrisg394 I do the same, even when raidz expansion will be a thing it won't allow for different drive size while using all the capacity and downsizing either.
Hello Jim!, I finally found this guide about SMB Share. On my previous setup, I was struggling about file permissions. Upon creating a ZFS pool, I mount pointed some folders to different lxc containers. I have this the same setup now on my SMB Share and I can't modify the files that was created by other lxc containers on that specific folder. Am I missing some permission setup to be able to edit all the files on the share? Thanks!
When you add the share to the container, it's thin provisioned on the zpool, right? i.e., from your example of the 32GB NAS bit assigned to the smb container, if there's only 5GB put in that share via the container, only 5GB is used in the zpool? And you can use that single pool for multiple containers yeah?
Thanks for the video. Just one question, why to bother with docker at all and not to use Truenas core or OMV? As soon as the data protected with ZFS on the host, you can attach a disk to TrueNas/OMV VM and assign a singe disk to pool.
Thanks. I've shown how to do that in previous videos. In this video I mention that docker has advantages in being simpler if you don't need all that TrueNAS has to offer. Plus you don't need an HBA etc. AFAIK, the documents recommend that TrueNAS directly controls the disks, it cannot do that in the way you have described.
@@Jims-Garage Thanks for the answer. One more question, ZFS is using RAM for caching (ARC) on the host and in LInux by default it equals to half of the RAM. How Proxmox react on the fact that almost all memory is used? Would it deflate RAM on the VMs with balooning=1 flag in order to free more memory?
That is a bad idea. ZFS assumes to have direct access to hardware. If this condition is not met and something bad happens, there is a strongly increased risk to loose data.
Hey Jim. This video kept me wondering how did you manage to connect to this SMB container without exposing 465 port in compose file. But then i have noticed network mode: host xD
No, traditionally the compose file tells how the container should be deployed. You normally supplement that with a .env or config file to configure the actual application (sometimes you use env variables in compose). A good example of this is using a separate file for password credentials.
The setup shown here has a number of disadvantages when compared to a "native" Samba setup on the Proxmox host itself: 1. You have to allocate the space for the shared disk statically, thus rendering the potential to expand the ZFS pool later on partly useless. 2. You stack one filesystem (ext4) on another (ZFS), meaning higher CPU usage than using ZFS directly. 3. You lose the possibility to access snapshots via Samba, because ext4 does not offer it. Instead, I would just install Samba on the docker host, plus the package zfs-auto-snapshot. There are lots of instructions out there showing how you can expose the snapshots to Samba clients. This way, you can even distribute the ZFS pool capacity to your liking. And also, you can expose the share via NFS as well.
"don't buy these - they're terrible" I disagree - I remove the labels using a hairdryer then keep them in a box to use as arse roll in the next pandemic. Haven't found a use for the SSD parts yet though so they just go to recycling.
i noticed in the "Use Proxmox Cloud-Init " video you did not check "discard" when setting up your SSD or NVME drive. You only checked SSD emulation. But in this video you did. Why is that?
I've covered raidz in a previous video and outlined how to expand. You can expand a raidz but not in the same way. You're limited by the size of the smallest drive
@@Jims-GarageYes, but no. It is absolutely not recommended unless you want to risk losing your entire RAID. In my opinion, this is something you really shouldn't advise to amateurs, who make up the majority of your audience. I would never take such a risk, given the high probability of hardware failure. Most people attempting this kind of operation likely do it on HDDs that have already run for thousands of hours and are probably at the maximum for RAID Z1.
@@blender_wiki I agree, which is why I've extensively discussed and shown how to operate a 321 backup solution. That should remove the risk. There's also the point of personal responsibility, people are shown how to do something, they need to test and be sure to understand it before doing it (it's the 'jump of a cliff' scenario).
@@misterc3835 this should help you understand everything, also check out my previous TrueNAS videos. www.truenas.com/community/resources/introduction-to-zfs.111/download
@@74357175 anything using an LXC shares the host's kernel. Yes there's logical segmentation but if it fails your device is basically compromised. However, if it's just internal stuff with no exposure to the internet etc it should be fine for a homelab. All depends on how comfortable you feel. As someone discussing tech and recommending things I feel obliged to be more risk averse.
So the shared smb storage is a virtual (qcow2 ??) disk on an underlying zfs dataset managed by the proxmox host. Seems like extra overhead and potential fragmentation from the sparse nature of qcow2 on zfs, but my knowledge on that topic is limited. Why couldn't you create a zvol block device in the NAS pool on the proxmox host and pass that to the ubuntu VM running docker. I'm guessing the proxmox web gui doesn't handle creating and managing zvols (yet), but that step could be done on the command line. Aside from a different block device, the steps on the VM and docker container should be the same.
@@Jims-Garage Among the many youtube videos on zfs and homelab, I really don't see much (or any) discussion on using zvols. Maybe the lack of gui support. Looks like an underutilized zfs feature.
Recommending RAIDZ over mirrors as a general advice is wrong. In fact for most use-cases other than archiving mirrors are the better choice, because they deliver more IOPS. RAIDZ delivers more net capacity, but of what value is that if your VMs perform slowly.
Thanks. I'm not proposing people use it for VMs, I explicitly show using nvme for VMs, and in this video it's purely an SMB share which isn't suitable for VMs. I've covered this topic in previous videos.
BTW, 24% wearout? Is that at all concerning to you? Have you discussed wearout info in Proxmox? I have a SSD that's reporting 14% wearout and I've been getting nervous. Seeing your 24% makes me wonder if that concern is warrented.
To me it means worry in 76% ha. It's not as if it'll stop working after that, just that it's a good idea to replace. All data is backed up, plus at current rate it's another 4 years
How the heck did you create a zfs pool and suddenly it's in a NAS under the datacenter??? I just created a ZFS pool and it is NOT in a NAS. I don't even see that as an option.
"Don't get these. They're terrible." There. That could be your first short. Your long-form videos are nearly always must-see for me.
Haha, true 😂 I might do that if I get time. Thanks for the feedback
You don't want to mount drives using the /dev nomenclature. Always use the UUID as those are immutable.
Yeah! I told him too 🤔🤨😳😲😉
true, was just a victim of the /dev when mixing up with the storage on xcp-ng, hence I second that xd
Thank you James, Tutorials are constantly getting better in every way, keep it up 👍
Appreciate the feedback and support 👍
The other option worth trying would be a turnkey file server from LXC templates library.
Yes, that's a good option. I'm yet to cover turnkey.
@@Jims-GarageI much prefer using LXC for sharing Proxmox storage, then all other LXCs on the host can use it directly, rather than resharing. Plex is a natural use case here.
LXC requires additional shenanigans if you have ZFS and want to use Docker. By default it will use VFS driver which not only is slow but doesn't support copy on write so Dockers layers instead of megabytes will take gigabytes of disk. That's why you have to manually format zfs subvolumes to something else (xfs, ext4) and keep their naming properly if you want to preserve ability to do backups from webui. I've used to create LXC, create additional zfs subvolume, format it to xfs, mount it as a second drive to LXC, copy /var/lib/docker to it and replace original /var/lib/docker with symbolic link to docker folder on that drive.
I’ve got a dedicated NAS, so this won’t necessarily be my use-case, but it does clear up some confusion I had on the mapping of drives. Excellent tutorial. I’m going to use this as the basis for a FOG storage instance. You may want to check that out project out, as it’s a pretty cool way to swap out environments for non virtualized hardware. I use it to swap out hypervisors for testing on different hardware. It’s a no-frills, but extremely useful project.
Thanks, glad it helped. I'll take a look at that.
Great idea for a low power mini server 👍
I think so too!
Perhaps do a follow-up to this by adding additional users with their own "folder" and permissions. Is there a way to hide specific folders from other users? Also, connecting to it using an FQDN for those users outside the network.
This is a great topic, Jim.
Have you thought about cockpit and zfs-manager module to manage snapshot, replications, etc? Potentially, samba shares also.
I think it might be easier and safer in the long run.
Great work on your homelab series. Clear and complete instructions make it a pleasure to follow your videos every time.
Great suggestion! I'll take a look at it
As soon as the new ZEN 5 release (getting a 9950X), I'm going to move from UnRaid for my NAS needs into Proxmox. Will be back to this video in July ;)
Awesome, that'll be a monster machine
excellent explanation and video!
Thanks, Marc. Have a great weekend
As always, well explained and demonstrated.
I'm not sure the VM + Docker is really necessary, unless you just really want to run SMB in the virtualized environment instead of directly on the Proxmox host. I've been running CIFS shares directly on my Proxmox host for a couple years now, specifically to share OS.iso image files for easy cataloguing and sharing of all my various flavors of Linux and Windows, and the connectivity works the same as this example. Either way, the data is shared out transparently to the target, it's just a different method to get there. Thanks for the video!
Thanks, I do mention that you can directly serve from the host and an LXC, but I prefer to let Proxmox do just Proxmox, and I like the simple repeatability of Docker. But definitely, if it works it works
Entirely fair, I just wanted to point out it was also possible without using either LXC or VM.
@@Jims-GarageThe problem is, this way you're virtualizing a SCSI controller (that's virtio for you), that is presented with a ZFS dataset created for the VM, and then you're also putting ext4 on top of that. Each read/write operation has to update things in ext4, go through the virtualized SCSI controller, and write to underlying ZFS.
And then Docker and Portainer on top of that... honestly, it would be easier to just apt install samba and avoid yet another abstraction layer. Even better, you can install cockpit with the plugin from 45drives, so you can manage your Samba config from a web UI tuned for working with a storage server, basically the same as 45Drives Storinator units - instead of faffing about with environment variables in docker-compose (which may not expose functionality of samba that you need further down the line).
A smarter way to go about it would be to either pass in a HBA (so the VM can access drives directly), or if you don't have one, you could do a raw block device passthrough to the VM and set up ZFS inside the VM. That way you could create new datasets, use snapshots and other ZFS features from inside the VM. Perhaps having different shares with different settings for things like compression, deduplication, atime and so on.
@@chrisa.1740 I have the same thoughts. For few years I used VM, but now transformed all my services to LXCs containers which are at least 5 times more efficient and less power hungry in comparison to VMS... Finally all what I ran on strong PC currently run on small NUC which use 20 Watts instead of 100...
I'm doing that the similar way but using Open Media Vault VM instead of docker container you're using
I'm thinking about spinning it up on a K3s cluster :)
Yes, that would be possible. There could be better options though depending on what you're trying to achieve.
Thanks!
Thanks for the donation, that's very kind
Wonderful video, I would like to know in 6:21, if that would be better to enable Cache section with such as "write back" function?
Great video, I actually do it via LXC container. What would be cool would be a ISCSI target on the same VM and having it mounted in windows, some apps dont play well with files on a samba share. Have greate weekedn, cant wait for the nexit video :) By the way, getting a Minis Forum MS01 delivered in the next week or so for dedicated firewall, cant decide if I wanna play with Sophos or OPNSense, can I get a demo of Sophos to trial for a month or so?
Thanks. Sophos XG is free for home use. Limit is 4 cores and 6GB RAM which is more than enough for a homelab.
enjoy the MS01 they look sweet
I'm using a LXC container to make my samba server and passing the full ZFS pool , added webmin to setup the share points.
Being on a LXC I just give it own directory and use a mount point to it , no passing it the full drive .
I got a pair of sandisk 960gb SSDs from Maplin and not had a real problem with them , I guess you have a story about yours.
Most of my SSDs are samsung 870-QVOs 6 x 4TB total plus 2 crucial mx500 4TB drives .
Any reasons why you chose webmin over cockpit to manage the shares? I've been investigating web frontends for server management. Webmin is the grandaddy but not clear (to me, at least) how capable the interface is with modern server technologies. Redhat still seems to be actively extending and promoting cockpit. 45Drives has re-written the cockpit zfs module (Houston). Looks very capable.
I don't think you can expand raidz yet. It should be implemented on open zfs 2.3.
Expanding mirrors you're tied to the smallest capacity drive within the same vdev but usually you expand mirrors pool by adding mirror vdevs to the pool so you can have mixed drives capacity vdev within the same pool
Good to know. Yes, believe you can expand a mirror into raidz10
I confirm, you can't expand raidz, yet
Mirrored disks is my preferred setup in a homelab environment. ZFS lets you add drives to the mirror (more redundancy) or add more mirrors to the pool (more storage space). You can even do the reverse and remove drives. In my opinion the most efficient solution for the homelabber.
@@chrisg394 I do the same, even when raidz expansion will be a thing it won't allow for different drive size while using all the capacity and downsizing either.
Hello Jim!, I finally found this guide about SMB Share. On my previous setup, I was struggling about file permissions. Upon creating a ZFS pool, I mount pointed some folders to different lxc containers. I have this the same setup now on my SMB Share and I can't modify the files that was created by other lxc containers on that specific folder. Am I missing some permission setup to be able to edit all the files on the share? Thanks!
I think I'll stick to virtualizing trueNAS on my server. Proxmox will be installed on the machine with all the drives anyway.
When you add the share to the container, it's thin provisioned on the zpool, right? i.e., from your example of the 32GB NAS bit assigned to the smb container, if there's only 5GB put in that share via the container, only 5GB is used in the zpool? And you can use that single pool for multiple containers yeah?
Thanks for the video. Just one question, why to bother with docker at all and not to use Truenas core or OMV? As soon as the data protected with ZFS on the host, you can attach a disk to TrueNas/OMV VM and assign a singe disk to pool.
Thanks. I've shown how to do that in previous videos. In this video I mention that docker has advantages in being simpler if you don't need all that TrueNAS has to offer. Plus you don't need an HBA etc. AFAIK, the documents recommend that TrueNAS directly controls the disks, it cannot do that in the way you have described.
@@Jims-Garage Thanks for the answer. One more question, ZFS is using RAM for caching (ARC) on the host and in LInux by default it equals to half of the RAM. How Proxmox react on the fact that almost all memory is used? Would it deflate RAM on the VMs with balooning=1 flag in order to free more memory?
@@MenkarX yes, it probably would. You can amend the amount of ram it uses though by amending the defaults (I do that for my boot pool)
That is a bad idea. ZFS assumes to have direct access to hardware. If this condition is not met and something bad happens, there is a strongly increased risk to loose data.
@@christophjahn6678 yeah, I like my data tight too.
Hey Jim. This video kept me wondering how did you manage to connect to this SMB container without exposing 465 port in compose file. But then i have noticed network mode: host xD
Haha, yes. I probably should have called that out explicitly.
Nice tutorial. Can somebody explain what is with that config.yml file? Isn't docker compose file suppose to have all the settings?
No, traditionally the compose file tells how the container should be deployed. You normally supplement that with a .env or config file to configure the actual application (sometimes you use env variables in compose). A good example of this is using a separate file for password credentials.
The setup shown here has a number of disadvantages when compared to a "native" Samba setup on the Proxmox host itself:
1. You have to allocate the space for the shared disk statically, thus rendering the potential to expand the ZFS pool later on partly useless.
2. You stack one filesystem (ext4) on another (ZFS), meaning higher CPU usage than using ZFS directly.
3. You lose the possibility to access snapshots via Samba, because ext4 does not offer it.
Instead, I would just install Samba on the docker host, plus the package zfs-auto-snapshot. There are lots of instructions out there showing how you can expose the snapshots to Samba clients.
This way, you can even distribute the ZFS pool capacity to your liking. And also, you can expose the share via NFS as well.
You must have some really long arms.
I do, it's really frustrating, I usually have to sit in the adjacent room just to operate the keyboard 😢
"don't buy these - they're terrible"
I disagree - I remove the labels using a hairdryer then keep them in a box to use as arse roll in the next pandemic. Haven't found a use for the SSD parts yet though so they just go to recycling.
Haha 😂 that's likely more performant
i noticed in the "Use Proxmox Cloud-Init " video you did not check "discard" when setting up your SSD or NVME drive. You only checked SSD emulation. But in this video you did. Why is that?
I simply forgot 😔 discard should be clicked to take advantage of trim
Is it possible to spin down the hard disk while not in use, assuming I am using 3.5” non SSD? Read somewhere about this hd-parm in terminal.
4:44 It sounds like you are saying raidz is expandable. I think it's important to explain the limitations of that to new users of zfs.
I've covered raidz in a previous video and outlined how to expand. You can expand a raidz but not in the same way. You're limited by the size of the smallest drive
@@Jims-GarageYes, but no. It is absolutely not recommended unless you want to risk losing your entire RAID. In my opinion, this is something you really shouldn't advise to amateurs, who make up the majority of your audience. I would never take such a risk, given the high probability of hardware failure. Most people attempting this kind of operation likely do it on HDDs that have already run for thousands of hours and are probably at the maximum for RAID Z1.
Can you explain or point to some more info?
@@blender_wiki I agree, which is why I've extensively discussed and shown how to operate a 321 backup solution. That should remove the risk. There's also the point of personal responsibility, people are shown how to do something, they need to test and be sure to understand it before doing it (it's the 'jump of a cliff' scenario).
@@misterc3835 this should help you understand everything, also check out my previous TrueNAS videos. www.truenas.com/community/resources/introduction-to-zfs.111/download
Question: Why not run Docker in an LXC container rather than a full-on VM?
Mainly for security. More secure as it doesn't share the host's kernel and there are only minimal overheads.
@@Jims-GarageI see! Is that not also an argument not to use LXC containers, since then you share the kernel with the host?
@@74357175 imo yes. There's nothing wrong per se, they work exactly as intended. I simply prefer the extra safety layer.
@@Jims-Garage got it. Do you think the same way about LXC+turnkey (as suggested elsewhere in the comments) or only about Docker?
@@74357175 anything using an LXC shares the host's kernel. Yes there's logical segmentation but if it fails your device is basically compromised. However, if it's just internal stuff with no exposure to the internet etc it should be fine for a homelab. All depends on how comfortable you feel. As someone discussing tech and recommending things I feel obliged to be more risk averse.
So the shared smb storage is a virtual (qcow2 ??) disk on an underlying zfs dataset managed by the proxmox host. Seems like extra overhead and potential fragmentation from the sparse nature of qcow2 on zfs, but my knowledge on that topic is limited. Why couldn't you create a zvol block device in the NAS pool on the proxmox host and pass that to the ubuntu VM running docker. I'm guessing the proxmox web gui doesn't handle creating and managing zvols (yet), but that step could be done on the command line. Aside from a different block device, the steps on the VM and docker container should be the same.
Both of those are valid alternatives. I'm keen to test the performance impacts. I may update based on what I find.
@@Jims-Garage Among the many youtube videos on zfs and homelab, I really don't see much (or any) discussion on using zvols. Maybe the lack of gui support. Looks like an underutilized zfs feature.
I got lost and gave up
Recommending RAIDZ over mirrors as a general advice is wrong. In fact for most use-cases other than archiving mirrors are the better choice, because they deliver more IOPS. RAIDZ delivers more net capacity, but of what value is that if your VMs perform slowly.
Thanks. I'm not proposing people use it for VMs, I explicitly show using nvme for VMs, and in this video it's purely an SMB share which isn't suitable for VMs. I've covered this topic in previous videos.
Can’t you just install samba without any docker? What is the point of docker here?
@@aijokker you can. Docker simply means you have a portable, instantly deployable configuration to speed things up.
BTW, 24% wearout? Is that at all concerning to you? Have you discussed wearout info in Proxmox? I have a SSD that's reporting 14% wearout and I've been getting nervous. Seeing your 24% makes me wonder if that concern is warrented.
To me it means worry in 76% ha. It's not as if it'll stop working after that, just that it's a good idea to replace. All data is backed up, plus at current rate it's another 4 years
You don't interpret that to mean ~240GB or the 1TB are now no longer usable? Or at the very least, no longer reliable?
How the heck did you create a zfs pool and suddenly it's in a NAS under the datacenter??? I just created a ZFS pool and it is NOT in a NAS. I don't even see that as an option.
At 10:48 you mention that you will link a container that will do the same thing with NFS - am I missing it, or did you forget to add it?
There no shortage of ways to do this.
Agreed, this is one of them.