This is kinda blowing my mind. I havent built a pc since 2011 and havent really paid close attention to the advancements. Things have changed soooooo much!
While passthrough of disk from proxmox to the TrueNas VM works, it is highly discouraged. As you stated correctly, any abstraction between the disk and ZFS can cause problems. Even the loss of a pool. Therefore, one should consider always passing through the full HBA where the disks are attached to. That is considered best practice for virtualizing TrueNAS.
I took most of your parts but decided to build it with a jonsbo n3. I dont regret, its a really neat NAS with the option for 8 3.5 inch disks + 2 2.5 inch SSDs and the option for one additional nvme. Also added my sfp+ card of course for some speed :D
Am always proud of my fellow swedes. Fractal Design Node 304 (10 year old design) with 3 mirrored pairs is "my precious" storing all my data. Let's look if Terra holds at least half of this time :)
I love when some of my favorite creators decide to make content that happens to be relevant to me, all at the same time! I'm currently planning a DIY NAS inside a Jonsbo N3 case, which I've already ordered. For the platform, I'm look at using an ASRock IMB-X1231 motherboard paired with an Intel Core i5-14500 (probably). My main reason for this combo is ECC support (I know, it's a polarizing topic lol). I'm still on the fence about installing TrueNAS bare-metal or as a VM inside Proxmox. I want to give Frigate a try using a Coral USB Edge TPU and I'm worried the extra layer is going to make things complicated. Great content as always, Tim! 😁
Thank you! Yes, you can pass anything through to a VM. I run ECC in my servers but when it comes to consumer stuff, I don't bother even if it's providing a server role 😅
I love this application. I am looking to build something similar, although hampered by my low level of software skills. I am hoping to re-use my large Lian-Li tower with fanless PSU and CPU for the perfect homelab - and more space for drives! I am more confident having seen this video. Thanks for the effort :D
I've read on the forums, that you should pass through the whole HBA card/sata controller and not the disk by itself. Apparently you lose some SMART functions that are required for ZFS to detect errors and do its self-healing. I've never actually tested this, so just speculating here.
Thanks! Yes, always HBA for sure if you can. I had to do individual disks though because I was using the same controller. 2 for TrueNAS and 2 for Proxmox VMs.
@@TechnoTimTinkers Tim do you have any concern about losing data because you passed through only the drives. If you read anything on the TrueNAS forums they insist this is a formula for disaster. Yet, I have done this exact same thing on my testing machine for a while now and never had a problem. Interested in hearing your point of view on this. Thanks.
I was renovating a house but because of different reasons I was forced to temporary move into an appartment where I cant have my rack (well, I can't hide it enywhere, wife is not keen on having it in the living room). So this is just perfect timing for a build like this. I will be repurposing Define R5 case from my gaming pc for this build, it gives me a lot of space and is also very quiet. But I'm thinking of using Terra for the new gaming PC.
I really hope Fractal ends up making a Terra+ that can fit at least four 3.5" HDDs. It's such a wonderful design and it'd be perfect for a smaller NAS if it just a had a bit more room.
Nice build. Been debating building something akin. But would love to include space for 6 + 3.5s maybe 2 2.5 or NVME and room for a 3090. Do all my NAS, AI, plex, etc. on one box and even maybe run a gaming VM on it in down time. i9-9900k still so with that would need room for AIO. Less I upgrade the CPU and explore more cooling options. But for your use case this is not bad at all.
Thanks! I totally know what you mean! Once you start thinking of all of the use cases you start adding more and more. I had to think about this build as if I were 99.9% of the home users who just wanted something good looking, quiet, low power, and would never fill almost 30TB or storage 😂
Wouldn't it have been better to pass your SATA controller directly to your Truenas guest, rather than the individual drives? I have a similar build with both the SSD and HDD pools on the Truenas guest which are shared back to the Proxmox host via NFS as needed. Alternatively, throw a 2nd little SATA controller in the unused PCIe slot for a dedicated controller for the Truenas guest.
How did you manage to get the drives passed directly to truenas. I was under the impression that proxmox always had a layer that hides the bare metal drive and that the only way to directly pass it is with a sad controller. This has been a big pain for me personally given my hardware setup. Would love to know how you do it
If you don’t want to get a separate HBA card, as long as there is a SATA controller that has its own IOMMU group, you can just pass that through to the TrueNAS VM and it should allow TrueNAS to get direct control of the hard drives just like bare metal installations. I have been able to do that on a X99 machine, which has two SATA controllers and one of which has its own IOMMU group. Everything works perfectly in TrueNAS core, including all the disk info, SMART info etc. the only issue I found was the VM will lose control of the SATA controller after resuming from sleep and so far I haven’t find a solution to it. I eventually went back to importing the ZFS array to Proxmox itself and it works perfectly with sleep which allows me to use this machine as a on-demand NAS storage and avoiding the high idle power consumption of X99.
I have also tried what’s shown in this video to pass through the disks to the VM and it seemed to work. Everything appears normal except for one thing that is I cannot get the correct disk info, such as serial number, temperature and SMART info, which makes me feel shaky about it and went the other way around.
@@cyberbass4 o wow nice. Thanks for that. Might give it a try. Unfortunately my case can’t use an hba card because of the custom backplane of the dell workstation
@@Ezza232499 Yeah. Hope you are lucky enough to have such a SATA controller on your motherboard which can save you a lot of trouble. It is actually easier than passing through individual disks and all ZFS related stuff works perfectly, except for the sleep part.
Few things to get off my chest 1. even turning those hdds on a side and using screws from perforated top panel would have been a better option. im imagining you still plan on cleaning living room and need to move that case at some point. you will end up with hdd failures sooner rather than later. 2. living room nas? with dual noise hdds? please try and give good advice to the average Joe that wants to build his first nas
Hi, I wonder why not just create a debian LXC with cockpit and avoid all the truenas overhead and let proxmox manage everything, I have tried both personally and I love truenas but it is true that LXC with samba is much lighter and runs faster.
Hi there. First of all congratulations for your video. I'm a newbie on this stuff, especially on linux environments, and after tried a few months a working units with Truenas Scale, it seemed too complex to get it working also as a media center using Emby, and also a few other things like qbitorrent, etc. due to the annoying permissions sets... So you triggered me a desire to have a try on this combo Proxmox + Truenas Scale, but i miss some skills on that part of getting the whole hhd drives pass-through available to Truenas. I'm aware that this wasn't the purpose of your video, but can you please help me out? Or maybe have a dedicated video for that? 🙏 Thanks in advance
I had the same wishes to build in the Terra for my NAS. I love the aesthetic of the Terra, but I will need expandability. My main server has been in a Corsair ATX tower running Proxmox and TrueNAS Scale virtualized. I wanted to separate everything. I went with the Jonsbo N3 and TrueNAS Scale Bare Metal. Jonsbo N3 was the same price as the Terra with expandability for the future. I really didn't want to drop 1k on a 8 bay Synology even though it would have been much less work. The Jonsbo N3 will be solely a NAS and now I'm continuing a separate build for my main server. I went with a Jonsbo C6 for the main server. I enjoy the idea of having everything separate.
@5:10 had same idea for another build but these "simple" peices were being sold on Amazon Canada for $94 for 6 three that idea out the window until i got a buddy with a 3D printer to make a dozen for $10.
Does your Mini ITX motherboard support PCIe Bifurcation? If so, you could make a 'Ultra Fast' storage option using the PCIe 4x4x4x4 Option with (4) NVMe M.2 SSDs.
Great video, I like the build. Do you have any concern to a virtualizing TrueNas? I have a standalone TrueNas server and then I have another server that I use for virtualization, I went with XCP-NG as my hypervisor. I said it works perfectly fine, but I wonder if I could get a little bit more bang for my buck by consolidating down to one server?
How does setting up the boot drives work? I understand you have two mirrored SSDs, is Proxmox installed on these?
2 หลายเดือนก่อน
How are the Exos disks doing soundwise please? I am building a NAS which will be right next to me and was looking at these disks, but I am afraid they will be quite loud (especially when not idle).
Sadly they have effectively discontinued TrueNas Core, it's still going to get security updates for sometime, but that's it, The Register did a good article on this back in march.
One issue is that your TrueNAS VM can't really utilize Proxmox's extra/unused RAM for the ARC cache. You have to choose a static allocation ratio. Why not just host the ZFS volume(s) from Proxmox directly? For the TrueNAS GUI?
Good question, not sure. It's a great looking case too! I would just look at the specs and see how much wdith you have next to the PCIe card slot and then see if there is enough depth and width. It's tough to know for sure which is why I went out on a limb when I built this.
@@TechnoTimTinkers I appreciate it. New to servers and home labs. Trying to figure out storage for my photos as a photographer and dad. As well as a media collector
If your mounting a network share from a TrueNAS VM to the host proxmox. Be aware that you will probably have to manually start and LXC or VM that uses it. As even if you start the TrueNAS VM first. The share won't be ready in time, and the container/VM won't start. How I get around this is running the containers from a VM using autofs or creating the volumes from NFS shares, and forcing (in my case docker) to wait for the mount before starting.
And I've just built a nas with a ryzen 670e motherboard with a 9700x , a really bad idea after undervolting it by 0.1 volts and a maximum 300mv extra undervolting on PBO I can get it down to 50 Watts idle when set to 45 Watts tdp , making my 6000 ram run at full speed using expo uses extra about 10 Watts . that is with the igpu only and one ssd.
you don't need to build out an 800 dollar machine to do this though add some ram to a 9th gen coffee lake box for half the price and have room to expand and add more drives while using more power for a year or two until you build something better - how most people should and will do it - why be extravagant in this economy
I also have coupled Proxmox and TrueNAS together and just love the benefits (console, VM snapshots, backups). Especially migrating went flawlessly. However, to have disk spindown and SMART I bought a PCIe SATA controller and passed that one through. Only issue I had was that I had to tell Proxmox to NOT try to import the TrueNAS rpool on bootup.
7:00 🤣
This is kinda blowing my mind. I havent built a pc since 2011 and havent really paid close attention to the advancements. Things have changed soooooo much!
While passthrough of disk from proxmox to the TrueNas VM works, it is highly discouraged. As you stated correctly, any abstraction between the disk and ZFS can cause problems. Even the loss of a pool. Therefore, one should consider always passing through the full HBA where the disks are attached to. That is considered best practice for virtualizing TrueNAS.
lol, if you need to expand add another one and have the storage as a Ceph clusters :)
I took most of your parts but decided to build it with a jonsbo n3. I dont regret, its a really neat NAS with the option for 8 3.5 inch disks + 2 2.5 inch SSDs and the option for one additional nvme. Also added my sfp+ card of course for some speed :D
Am always proud of my fellow swedes. Fractal Design Node 304 (10 year old design) with 3 mirrored pairs is "my precious" storing all my data. Let's look if Terra holds at least half of this time :)
I do wish they would make a NAS in this same style! I'm here for it!
I'm about to build my first NAS, using that same case. Parts coming in by the day, can't wait!
the 304 is so nice. 6 HDDs is a sweet spot for me
@@therick0996 can take up to 8hdds, with some 3d printing
The older define revisions and node 804 are fantastic too.
I love when some of my favorite creators decide to make content that happens to be relevant to me, all at the same time! I'm currently planning a DIY NAS inside a Jonsbo N3 case, which I've already ordered. For the platform, I'm look at using an ASRock IMB-X1231 motherboard paired with an Intel Core i5-14500 (probably). My main reason for this combo is ECC support (I know, it's a polarizing topic lol). I'm still on the fence about installing TrueNAS bare-metal or as a VM inside Proxmox. I want to give Frigate a try using a Coral USB Edge TPU and I'm worried the extra layer is going to make things complicated.
Great content as always, Tim! 😁
You can pas though that coral to a vm, so it wont be an issue
Thank you! Yes, you can pass anything through to a VM. I run ECC in my servers but when it comes to consumer stuff, I don't bother even if it's providing a server role 😅
Where can you find the ASRock IMB-X1231?
I love this application. I am looking to build something similar, although hampered by my low level of software skills. I am hoping to re-use my large Lian-Li tower with fanless PSU and CPU for the perfect homelab - and more space for drives!
I am more confident having seen this video. Thanks for the effort :D
This is exactly the build I want to do! I can see space for a couple more 3.5 drives too 👀
I've read on the forums, that you should pass through the whole HBA card/sata controller and not the disk by itself. Apparently you lose some SMART functions that are required for ZFS to detect errors and do its self-healing. I've never actually tested this, so just speculating here.
Thanks! Yes, always HBA for sure if you can. I had to do individual disks though because I was using the same controller. 2 for TrueNAS and 2 for Proxmox VMs.
@@TechnoTimTinkers Tim do you have any concern about losing data because you passed through only the drives. If you read anything on the TrueNAS forums they insist this is a formula for disaster. Yet, I have done this exact same thing on my testing machine for a while now and never had a problem. Interested in hearing your point of view on this. Thanks.
I was renovating a house but because of different reasons I was forced to temporary move into an appartment where I cant have my rack (well, I can't hide it enywhere, wife is not keen on having it in the living room). So this is just perfect timing for a build like this. I will be repurposing Define R5 case from my gaming pc for this build, it gives me a lot of space and is also very quiet. But I'm thinking of using Terra for the new gaming PC.
We need a deep dive on Nano please, sweet pup!
He's such a good boy
An ARC A310 seems like a perfect card for that! And a 3d printer 😂
Jonsbo N3 - nice bit of kit
I really hope Fractal ends up making a Terra+ that can fit at least four 3.5" HDDs. It's such a wonderful design and it'd be perfect for a smaller NAS if it just a had a bit more room.
I hope so too!
Nice build. Been debating building something akin. But would love to include space for 6 + 3.5s maybe 2 2.5 or NVME and room for a 3090. Do all my NAS, AI, plex, etc. on one box and even maybe run a gaming VM on it in down time. i9-9900k still so with that would need room for AIO. Less I upgrade the CPU and explore more cooling options. But for your use case this is not bad at all.
Thanks! I totally know what you mean! Once you start thinking of all of the use cases you start adding more and more. I had to think about this build as if I were 99.9% of the home users who just wanted something good looking, quiet, low power, and would never fill almost 30TB or storage 😂
Hey, great tutorial! What are you using the TrueNAS for? If all of your VMs are running on SSD
Wouldn't it have been better to pass your SATA controller directly to your Truenas guest, rather than the individual drives?
I have a similar build with both the SSD and HDD pools on the Truenas guest which are shared back to the Proxmox host via NFS as needed.
Alternatively, throw a 2nd little SATA controller in the unused PCIe slot for a dedicated controller for the Truenas guest.
How did you manage to get the drives passed directly to truenas. I was under the impression that proxmox always had a layer that hides the bare metal drive and that the only way to directly pass it is with a sad controller. This has been a big pain for me personally given my hardware setup. Would love to know how you do it
If you don’t want to get a separate HBA card, as long as there is a SATA controller that has its own IOMMU group, you can just pass that through to the TrueNAS VM and it should allow TrueNAS to get direct control of the hard drives just like bare metal installations. I have been able to do that on a X99 machine, which has two SATA controllers and one of which has its own IOMMU group. Everything works perfectly in TrueNAS core, including all the disk info, SMART info etc. the only issue I found was the VM will lose control of the SATA controller after resuming from sleep and so far I haven’t find a solution to it. I eventually went back to importing the ZFS array to Proxmox itself and it works perfectly with sleep which allows me to use this machine as a on-demand NAS storage and avoiding the high idle power consumption of X99.
I have also tried what’s shown in this video to pass through the disks to the VM and it seemed to work. Everything appears normal except for one thing that is I cannot get the correct disk info, such as serial number, temperature and SMART info, which makes me feel shaky about it and went the other way around.
@@cyberbass4 o wow nice. Thanks for that. Might give it a try. Unfortunately my case can’t use an hba card because of the custom backplane of the dell workstation
@@Ezza232499 Yeah. Hope you are lucky enough to have such a SATA controller on your motherboard which can save you a lot of trouble. It is actually easier than passing through individual disks and all ZFS related stuff works perfectly, except for the sleep part.
Few things to get off my chest
1. even turning those hdds on a side and using screws from perforated top panel would have been a better option. im imagining you still plan on cleaning living room and need to move that case at some point. you will end up with hdd failures sooner rather than later.
2. living room nas? with dual noise hdds?
please try and give good advice to the average Joe that wants to build his first nas
Great video Tim, as always! Can this setup be placed in remote location like parent's house and accessed just like on my network ?
Have you considered making a DIY JBOD?
Hi, I wonder why not just create a debian LXC with cockpit and avoid all the truenas overhead and let proxmox manage everything, I have tried both personally and I love truenas but it is true that LXC with samba is much lighter and runs faster.
Hi there. First of all congratulations for your video. I'm a newbie on this stuff, especially on linux environments, and after tried a few months a working units with Truenas Scale, it seemed too complex to get it working also as a media center using Emby, and also a few other things like qbitorrent, etc. due to the annoying permissions sets... So you triggered me a desire to have a try on this combo Proxmox + Truenas Scale, but i miss some skills on that part of getting the whole hhd drives pass-through available to Truenas. I'm aware that this wasn't the purpose of your video, but can you please help me out? Or maybe have a dedicated video for that? 🙏
Thanks in advance
I wonder if having the HDDs not being vertical would be an issue on the long run, because of physics? (Talking here about 10 years later or so)
More than $1.200 us for a simple NAS? It’s not logic 🤷♂️
Hobby I think 😊
I guess you didnt get it
I had the same wishes to build in the Terra for my NAS. I love the aesthetic of the Terra, but I will need expandability. My main server has been in a Corsair ATX tower running Proxmox and TrueNAS Scale virtualized. I wanted to separate everything. I went with the Jonsbo N3 and TrueNAS Scale Bare Metal. Jonsbo N3 was the same price as the Terra with expandability for the future. I really didn't want to drop 1k on a 8 bay Synology even though it would have been much less work. The Jonsbo N3 will be solely a NAS and now I'm continuing a separate build for my main server. I went with a Jonsbo C6 for the main server. I enjoy the idea of having everything separate.
@5:10 had same idea for another build but these "simple" peices were being sold on Amazon Canada for $94 for 6 three that idea out the window until i got a buddy with a 3D printer to make a dozen for $10.
Don't you need proper ECC memory for ZFS? I thought standard memory was risky?
Does your Mini ITX motherboard support PCIe Bifurcation? If so, you could make a 'Ultra Fast' storage option using the PCIe 4x4x4x4 Option with (4) NVMe M.2 SSDs.
why does Truenas GUI says it's ECC mode?
Love the TShirt!
Thanks for the shirt!!!
Great video, I like the build. Do you have any concern to a virtualizing TrueNas? I have a standalone TrueNas server and then I have another server that I use for virtualization, I went with XCP-NG as my hypervisor. I said it works perfectly fine, but I wonder if I could get a little bit more bang for my buck by consolidating down to one server?
I virtualized TrueNAS for years. I had no issues doing so as long as you pass the controller or individual drives though.
is intel 13gen issue fixed ?
How does setting up the boot drives work? I understand you have two mirrored SSDs, is Proxmox installed on these?
How are the Exos disks doing soundwise please? I am building a NAS which will be right next to me and was looking at these disks, but I am afraid they will be quite loud (especially when not idle).
These are pretty quiet if you use some kind of sound dampening like rubber feet or bushings.
interesting experiment anyway that case it's too much small to be used as a NAS. You don't have any space left to mount other hdd.
Why do you choose TrueNAS Scale instead of TrueNAS Core if you want to do virtualization and containers via proxmox on a single node host?
Sadly they have effectively discontinued TrueNas Core, it's still going to get security updates for sometime, but that's it, The Register did a good article on this back in march.
@@h.b.5577 Thanks for that information. I ended up using TrueNAS Scale as vm after I tried the TrueNAS lxc unsuccessfully..
Is the power consumption with the hdd's running or spun down?
Is it safe to have zfs without ecc memory ?
One issue is that your TrueNAS VM can't really utilize Proxmox's extra/unused RAM for the ARC cache. You have to choose a static allocation ratio. Why not just host the ZFS volume(s) from Proxmox directly? For the TrueNAS GUI?
How much clearance do you have on the CPU cooler side? Could you fit a little taller cooler. Would your same config work in notch 2?
14th gen intel. I see you like to live dangerously.
I promise I updated the firmware and microcode 3 times 😅
two editional fans in my livingroom is not an option for me
Thank you! Do you think you could build this in the Era 2 and fit 4 3.5 hdd inside?
Good question, not sure. It's a great looking case too! I would just look at the specs and see how much wdith you have next to the PCIe card slot and then see if there is enough depth and width. It's tough to know for sure which is why I went out on a limb when I built this.
@@TechnoTimTinkers I appreciate it. New to servers and home labs. Trying to figure out storage for my photos as a photographer and dad. As well as a media collector
Got an ncase m2 a few weeks ago, planning a similar build for the next gen intel processors. Time to ditch the raspberry pi.
It's absurd the price you spent to build that, considering the NAS you can get for less than half of that.
how big should a boot ssd be
Whats the best way to share the TrueNAS VM storage with the different LCX containers like Proxmox/Jellyfin in this situation? A share in TrueNAS?
That's the route I would go. Create an NFS share and map it.
If your mounting a network share from a TrueNAS VM to the host proxmox. Be aware that you will probably have to manually start and LXC or VM that uses it. As even if you start the TrueNAS VM first. The share won't be ready in time, and the container/VM won't start. How I get around this is running the containers from a VM using autofs or creating the volumes from NFS shares, and forcing (in my case docker) to wait for the mount before starting.
No ECC for a nas is kinda risky ngl
am I dumb? How are you using so many vms with a processor only having 4 cores?
CPU over subscription. Pretty safe to do. Memory over subscription on the other hand is not good.
Nice.
no ecc :(
And I've just built a nas with a ryzen 670e motherboard with a 9700x , a really bad idea after undervolting it by 0.1 volts and a maximum 300mv extra undervolting on PBO I can get it down to 50 Watts idle when set to 45 Watts tdp , making my 6000 ram run at full speed using expo uses extra about 10 Watts .
that is with the igpu only and one ssd.
you don't need to build out an 800 dollar machine to do this though add some ram to a 9th gen coffee lake box for half the price and have room to expand and add more drives while using more power for a year or two until you build something better - how most people should and will do it - why be extravagant in this economy
Yup. My main Proxmox node I built this year is an Asrock Rack mb and 9700.
Yeah as the conclusion at the end of the video, the case is the limiting factor plus the motherboard. Bad choice.
Why such an expensive motherboard, when using the cheapest CPU? Or did you get a good deal on it or something?
It was on sale but I went this route because I wanted DDR5 and Mini ITX.
Using truenas in proxmox without passing disk controller is a bad idea
are we trending back to blacked out sff systems again? no rgb 4 life
Hopefully!
@@wojtek-33 It was fun while it lasted, back to RGB! 😅
you gotta noc TUA and black out that thang
Sweet case. $1,281.40 if anyone is wondering.
Jonsbo N1
For my Linux ISO ..... 🤣🤣🤣👍
I think it’s time for you to buy a 3d printer , design custom brackets for hdd’s
I also have coupled Proxmox and TrueNAS together and just love the benefits (console, VM snapshots, backups). Especially migrating went flawlessly. However, to have disk spindown and SMART I bought a PCIe SATA controller and passed that one through. Only issue I had was that I had to tell Proxmox to NOT try to import the TrueNAS rpool on bootup.
You stole my thermal paste 😠😠
I will say it. Unraid runs on a usb stick. Then you can use the NVME drive for VM's -- FAST
Unraid also cost money. Fine when you have it to spare. Some of us do not.
Why True Nas on Proxmox, It's Or nont And. 2 systems do the same, TrueNas is user friendly, proxmox not...
Or just get a Jonsbo n2….
Light on the tech content. Heavy on the shill. Not a channel for me.