Sorry about the mistake by saying 5.25" drives! While researching and testing, I was trying to figure out how many drives I could fit in the Corsair's 5.25" bays and somehow that got into my script. 🤦♂ In the spirit of mixing things up, let me know what you've mixed up before!
Please try Unraid. It is very different in ways I'd like you to show people. I finished an ITX build on Wednesday and by now ( Friday) I have an entire arr stack with multiple instances of certain containers running. Even while being pretty busy at work.
4:40: x16 will only run as x8 and x8 runs as x4 if you are using a feature that shares the same lane, x16 and NVMe Gen4 slot for example. It's important to know your mobo limitations like how many PCIe and which features shares the same lanes.
I went bear metal on my home lab server for a while, because it could do everything I wanted. However, things changed and now I’ve reinstalled everything on proxmox. The overhead is low and you have the flexibility anything in the future. So I would just install a hyper visor of you’re choosing.
@@redhonu Black Metal Bears go hard, but them Polar Metal Bears really be flexxing. Almost as much as the Metal Spirit Bears. "Jamie, Pull up the video of the Metal Bears in a mosh pit"
I’ve run for years an unRaid server which had a w10 VM with GPU and NVME passthrough that I used for playing games and the rest of the system was used for docker stuff: Plex, arr suite, homeassistant and a large etc. Just make sure you have a Renesas chip based USB PCIe card passed to the VM so you can plug and unplug peripherals without freezing the VM.
I moved to a single giant server build a while back from my own giant rack with dell power edge servers. But it was to much power usage. For most use cases in a single server build. I found unraid to be the best base OS for me. 5 years later. Still rock solid and never had any major issues. Kind of on autopilot and it just works.
I have gotten an 80 mm fan, some double sided tape and attached it to the case for the airflow for an HBA card before. Just get the stongest tape not the least expensive. Mine has been working fine for 4 year now.
15:18 When racking servers by myself, I've found that there's usually holes in both the sliding part of the rail in the rack, and the stationary part of the rail (the portion that attaches to the rack itself). Every rail is different, so it always takes some experimentation, but I've found that I can put a spare screw/toothpick/pointy-thing through both holes so that sliding part of the rail doesn't push back while I get things lined up. I do this for both sides but sticking out different amounts so I can line things up one side at a time. Just be sure that the holes you pick in the rail can be reached from the front of the rack.
Oof! What the heck was I thinking when I wrote this. I think this snuck into my brain because I was playing around with my old Corsair case and was trying to figure out how many drives I could fit in the 5.25 drive bays 🤦♂
@5:04 I don't know cpu lane calculation is as simple as you explain. you should probably check motherboard info, how many lanes are used for internals like lan, usb, ...
5:14 look at the block diagram instead, it may not what extra features are disabled on the motherboard with fully populated PCIE lanes. On my system it will trade between m.2 slots and 4x PCIE connections.
Also, thanks for the info on the Sliger cases. You just made me spend more money, they look great and are made in the USA for a reasonable price. My Threadripper platform is getting a new home. :)
I rackmounted my PC a week ago. The chassis was less than (the equivalent of) £100, including rails, and fits my 3 chunky radiators in too. My office is so much cooler and incredibly quiet now.
Great Video! Just be sure to look at your motherboard spec when concerned about PCIE lane availability. Motherboards also come with a controller with additional PCIE lanes. Basically, the processor's PCIE count is only half the story.
I don't think this is the ideal project to try Unraid on, but you really need to try it if you haven't. I always have at least a couple of servers in my rack and love changing things out, but the Unraid box is always there
Same here. 😁👍 Taking a play from my corporate sys admin world. Separate storage and compute boxes. Going to build a TrueNAS Scale box as the central storage for the entire homelab. Then use the 2nd unRAID lic to build a fresh compute server. Both will have 10Gbe until i can swap for fiber.
Hey Tim, love your videos. Fellow homelabber here, but I can’t stress enough as a homelabber you need to RTFM on the motherboard and CPU. In particular look at the block diagram in the doco. Super micro docs are always good and show how the lanes are used as some of those 48 lanes have to service usb, SATA, IPMI etc… Also, the motherboard silk screen will also say 8x in 16 etc to tell you its only 8 lanes on a physical 16 slot. Don’t put that GPU in the first 16 slot if you want those juicy 16 lanes.
Don’t go with the EVO 870’s, I made the same mistake (They are consumer drives and wear out quickly!) I replaced all of them with the Samsung SM883’s (ZFS pool)
I would go with Unraid for a build like this as it's easy to use and have a lot of features and if you really want to tinker, it's running on linux and have a terminal for any custom tinkering :)
I'm curious how your SSD pool works out. I have a Truenas Scale setup with 6 2TB Samsung EVO's and I've struggled to get decent performance. Literally it's less than a single disk. I tried Stiped 2 x (3 raidz1), 3 mirrored vdevs and raidz1 across 5 of them. Hopefully you can do a follow up video on your SSD pool setup and final performance numbers.
Consumer models performance will drop I believe because of the drives cache, PRO models address this. And enterprise SSD's will be far superior. At the end of the day, you get what you pay for...
Hey Tim, great build can't wait to see how that works out, Always looking for projects like this. I ended up with the iStarUSA D-410-DE36 case that allows for 36 drives a while back for the hotswap trays.Paired it with NORCO RPC-4224 4U with uses 24 3.5in drives. Purpose was to run Flash NAS and backup to spinning disk NAS. If you end up making 3d printed drive holders for this project, I would love see that adventure.
You mentioned this will not be taking over your NAS role. What about your firewall? I've seen some folks virtualize pfSense or vyos, but I don't think I would be comfortable hosting a firewall/ids/vpn on the same machine in case the virtualization solution has a security issue.
Sliger offers watercooling options for the Threadripper/Epyc line of CPUs in that chassis. I'm not aware of any constraints that would stop you from using random AIOs as long as they aren't too thick for your GPU clearances. There is a build someone did in that case with a thick 360MM cooler and a 3090ti.
creating a dedicated VM with the GPU is kind of limiting IMO, I find that using the GPU in lxc containers is the best way, you can have multiple containers use it at the same time and even monitor its usage from the host os
@@WhyDoesNothingWork it can be done even with some consumer cards. Those are not super expensive. Some more expensive pro ones can be found on the service have market. Intel cards will perhaps allow this as well.
@@BoraHorzaGobuchul if your talking about sr-iov only the commercial Intel cards do it and the chances of finding a consumer cars that does it is like winning the lotto
@@WhyDoesNothingWork there's a number of consumer Nvidia cards that work, but it does require rtfm and research, not something I'd love to do now. Then again, at a younger age I'd've already done that just for the lulz...
I just finished building a server around a EPYC 8024P and a Gigabyte ME03-CE1 motherboard. I'm contrast to this build though I don't have a GPU yet and I need it to double as a NAS, too. 96 PCIe lanes is plenty, room for growth (CPU, RAM, PSU, GPUs). I used SuperMicro SuperChassis case though (no rack).
I can’t remember landing the plane in Top Gun on the NES. Well done ! 🎉 As for the OS, I would vote for Proxmox or a regular distribution like Debian, Ubuntu or a RHEL clone. You could use Cockpit if you want to spin up VMs on them.
You mention the AMD in the script that has more PCI express lanes but in the video showed 20,16 then said "same number" but actually the Ryzen series have Native PCIe Lanes (Total/Usable) 28 , 24.
@Tim, it would actually make better sense to space your drives apart for better cooling and less restricted airflow in certain areas of the case. Since you have 6 drives and 3 fans, that's easy. Just space them where you have 2 drives per fan. The drives will run cooler, which will make them last longer and sustain the top speed longer, and they won't interfere as much with the airflow.
 I need advice to make a quick decision  I got Dell Precision T7920 Tower, 2 X Silver 4110 2.1GHz which am planning to use as a NAS drive, is it over kill or I should get some cheap PC to use it for that? thanks in advance. 
Tim, great video! Have you considered exploring Proxmox on this server and demonstrating GPU passthrough? It would be valuable to see which remote software is optimal for accessing VMs, and perhaps even conduct a gaming test. Looking forward to your future content! Greetings from Bosnia :)
I'm about to rebuild my server and I'm actually really curious what OS you're gonna run because it may influence my decision. I really didn't enjoy truenas because setting up apps wasn't intuitive at all. Curious about unraid but not thrilled about the price. Proxmox seems logical but I wanna be able to add more storage later on (Ideally) into the same pool. Cool build! Will be looking forward to the part 2!
If you do find a way to convert those 3.5" bays to 5.25", Wendell has shown some interesting enclosures that adapt 5.25" bay to 2.5" or nvme flash backplanes
Love the videos Tim, but why not just go EYPC? I got a 7601 cheap on ebay 32 core 64 threads has 128 pcie lanes and an ASrock rack motherboard with dual 10 gig built in?
"Try" and that's it! Don't get me wrong. Unraid is good but... ZFS isn't officially support, only community plugin & it's running in userspace! so your SSD's & 10Gbe are going to waste... So many great open source OS'es to choose from... if money doesn't mean anything to you, by all means
@@kbng02 ZFS on Unraid is officially supported since 6.12, released in june 2023. ZFS on Linux exists only as a kernel-level module. So what are you even talking about, even with community plugin it was running the same kernel module.
With the price of electricity I like to keep the 24 hour a day wattage to a minimum , about 50-60 including a rtx2060 super . I have a Ryzen 5700g with 4 x 4 tb SSDs and 64gb ram , but no ecc memory and only 20 lines .
I plan to build a server with the 5700g and 64 GB RAM as well. Do you have any numbers of the power usage without the video card? I thought about and suggest that it will be around 25-30W but numbers are hard to find online (atleast for me). Would appreciate it if you could help :)
@@pesfreak18 switch all the power boost modes off and which 2 x 4tb SDDs and 2 x nvme 2tb drive , for a small nas on deskmini x300 just set up as a samba server 15 watts according to my power monitor plug on idle. But my other system on a B550 motherboard , with the gpu and about 10% cpu all the time about 50-60 watts , but I think I must be the only person who is running a 5700g water cooled with a 360mm cooler , but it was a remove my x570 motherboard with a 3950x because I could not get it below 120 watts on idle . 16 core 32 treads was a little over kill for what I wanted my home server to do , in fact I think the 5700g is over kill.
@@peteradshead2383 15 watts seems very efficent. Water cooling is a little bit overkiil for me too but it sounds like a cool project. The Ryzen is maybe overkill but I want to experiment with gameservers and maybe AI in the future. I think I can utilize the CPU well enough with these tasks.
Sorry I'm new to all this and have been watching countless videos on server builds. What was that long green thing called that all your SSDs go into? Never seen that many hookups on one. Nice.
I'd like to cast my vote for a baremetal setup. I have a Proxmox server myself and decided to turn an old ITX box into a minecraft+calibre server, chose to do baremetal Debian (Crafty Controller setup was problematic for me on Ubuntu). Kind of wondering what all I can run on this, wouldn't mind some inspiration from whatever you come up with. My little setup can't handle much, but I'm dreaming of upgrades to do more in the future.
I'm contemplating similar build, and here are my observations: for proper local AI selfhosting, having Mac Studio with 64GB unified memory is MUCH more suitable than RTX 3090 which is limited to "only" 24GB vram, that said nVidia gpus utilizing CUDA are often faster than Apple's neural engine, all reasonably-priced Xeons are limited to PCIe 3.0 which is not futureproof at all, and already bit of a bottleneck for current gpus, any strong gpu takes lot of space and covers most PCIe slots on the motherboard, so it's rather difficult to decide on PCIe expansion layout, preferably gpu on "lowest" x16 slot, which then needs bigger than ATX case, that said some local AI tools can utilize multiple gpus or even multiple computers, so there's a delicate (cost/efficiency) balance between running let's say 4 gpus in one rig, 4 computers connected via 100Gbe together, latest Windows Server or plain Windows Pro, with WSL (Subsystem for Linux) seems best OS for widest range of local AI applications - arguably better manageable via Proxmox but with undesired performance loss, to sum it up, one all-in-one server doesn't seem that efficient, depending on what to run on it
I would like to see you give unraid a shot since you haven't yet, selfishly I am curious what your thoughts about it on a setup like this. I started with a single PC like this a decaid ago running Windows Server 12 r2 and Hyper-v. Over the years I upgraded that server and eventually replaced it with a much smaller proxmox cluster. I still have that machine and I am not ready to let it go yet. I have thought about unraid or truenas.
Hi. I would like to know your opinion on how long the SSDs would last compared to regular HDDs on a more intensive writing/reading application. Let's say rtorrent with 2-3TB of data. I have a dedicated HDD just for this purpose that is passed directly to the VM and I am reluctant to change it for an SSD.
Thanks, the way I look at it is that all drives have a lifespan, some fail faster than others, and others out live their lifespan. Since I look at all drives through the same lens (whether SSD or spinners) I just make sure that they have a long warranty and have one spare just in case. These Samsung drives have 5 year warranty and out of the 30+ EVO/Pro Samsung drives I have bought over the years, only 2 failed and Samsung replaced them within a week. For this reason, I typically go with consumer grade.
When you were talking about the power supply with those giant connectors you've never seen before, you really proved to me how old I actually am, so thank for that. LOL. Looking forward to seeing how this turns out.
Built a homlab server as well - ran Unraid for a while which was in the end a patchwork of tools for functions that a server system should already have integrated. And running from a USB stick is not something I want to rely on. So I decided to run Proxmox which is the most flexible system in my eyes. It's lightweight, can run VMs or LXCs, passthrough hardware, it's reliable and definitely the more professional choice.
It's basically only the config files stored on the USB stick. Unraid will load its config into RAM and run from there. For reference my Unraid system has 19,877 reads and 7341 writes to the USB stick and it's been in operation for 2 years with a lot of changes. Should my USB stick fail, I can just download the automated backup to a new USB stick, recover my license and I'm up and running again.
So apparently we are on the same tech wavelength and I just now noticed it, lol. I too just built the a few servers to do all the things. I ended up going with an Epyc system for all 128 of its PCI lanes dumped into a HL15. Another machine made use of the SilverStone RM41-506 4U chassis. I needed the 5.25" bays for the tape drive and an Icy Dock 4x 2.5" ssd hot swap cage, but the GPU just fits :)
I've been eying the dual epyc setups on ebay to eventually migrate everything to. I have absolutely no need for all 128 lanes or cores/threads, but it'll be all I'd ever need and then some.
@@DrDipsh1t Wow yea dual Epyc would be Epic 😂 I am actually making pretty good use of the PCI lanes on my 7282 16c/32t rig. A 16x PCIe card that holds 4x NVMEs, 16 port HBA, Radian RMS-200 Edge Card and a dual port 10gig SPF+ nic. Still have 2 more NVME m.2 slots, 2 OCuLink ports as well as 2 mini sas connectors on the motherboard I could populate. Its an Epyc8d from AsRock Rack board.
If you didn't have the hardware already, I would've recommend with a Threadripper system because it offers more PCIe lanes. (That's the direction that I'm heading down, except that I'll likely end up with something like an 8U chassis and then using PCIe risers/extensions so that it won't cover the rest of the slots. My other option will be a Supermicro 4U or 5U GPU server, but those are limited to dual-slot-wide GPUs only, which means that my 3090s will be blocking some of the other slots.)
I had the same idea a few months ago (all encompassing build with a lot of PCI), and I ended up going with a barebones Dell Precision T5820. I threw in an Intel Xeon W-2140B and some ECC RAM. Don't sleep on repurposing used workstation hardware!
Love my Sliger Cases. Have the CX4712 for my NAS and CX4150a for my desktop. I will probably buy a CX4200a for a gpu upgrade though. 3090FTW BARLEY fits with a low-profile radiator. It just fits a credit card between them.
Isn't that CPU a real limitation for an 'everything build'? Can't one just get a high end AM5 AMD processor, or are they really that limited in PCIe lanes? Maybe for AI, need a GPU, SSDs and 10gig NIC
Interesting. I am literally building a AMD Threadripper 7960x using a Sliger 4170i case with the ASetek 836SA AIO cooler. Just waiting on the case to be delivered to start the build.
Tiny subtitling correction at 2:05. I’m pretty sure you just left a gap in speech, rather than starting a new sentence. Generally though, your subtitles are very good. Thank you for taking the time to do them.
If you look at the schematics again iot says it right there which Slots are which for example the first x16 Slot says CPU SLOT6 PCI-E 3.0 x( (in x16) so that is an x8 in a x16 connector which is connected to the CPU and the last x8 says PCH Slot1 PCI-E 3.0 x4 (in x8) so it's a PCI-E via the PCH and only has 4 Lanes. So in any case the GPU only will run on 8 Lanes on any of the x16 connector slots as both are only x8(in x16).
Thank you! Yes, I actually edited out the part where I explained that, I probably should have left it! When I did the math you can see I adjusted for it (8 x 6 = 48, and the last was 4 in an 8)
I noticed you were counting your PCIe lanes. Depending of the board manufacturer and models, onboard components can also take up PCIe lanes as well. Such as NVMe, SATA, wireless, USB 3. Very common on consumer boards. Especially if they have NVMe slots onboard. Some boards have the option to disable onboard components to free up some lanes 👍🏽 this information is good to know if you want to convert your old gaming rig or workstation into a server and need full lane speed support on your expansion slots.
11:12 Fans are not installed correctly on the CPU heatsink. Firstly, do they even blow towards the back of the case? (and they should). If so, they need to be shifted one slot down (as looking on the video frame), i.e. the one closer to the front of the case doesn't need to suck air via fins, but would push the freely-collected air into them.
Wonder why noone considers the AMD EPYC 8004 series (zen4c, Siena) for homelab use. I'd go for them in a heartbeat, but I 'need' QuickSync/Arc (iGPU) so I'll have to wait until Intel finally catches up with their equivalent Sierra Forest lineup 😔
There are companies that offer 3D printing services; if you are not going to be printing stuff regularly, could be a good idead to look into one of those services. Great video, love your content.
So tired of them artificially holding back pcie lanes and also holding back on bifurcation of pcie lanes on consumer cpu's and motherboards. For the 2.5" drives- print out a bar that goes across the top of the drives and long enough to touch both sides of the case. There would be small ridges printed into the bar at each drive location to keep them in place. That way you don't block air flow. And you could easily lift off if you need access to drives.
Sorry about the mistake by saying 5.25" drives! While researching and testing, I was trying to figure out how many drives I could fit in the Corsair's 5.25" bays and somehow that got into my script. 🤦♂ In the spirit of mixing things up, let me know what you've mixed up before!
Please try Unraid. It is very different in ways I'd like you to show people. I finished an ITX build on Wednesday and by now (
Friday) I have an entire arr stack with multiple instances of certain containers running. Even while being pretty busy at work.
i was about to comment on that xD yeah i mixed stuff up too i cant come up with anything rn thou
great video btw
Gave up on floppy disks along ago
leaving mistakes in is a surefire way to drive engagement lol ppl love to tell you when you're wrong hahaha
I hate typing "disk" in front of someone at work and accidentally typing "dick"
Hey Tim, I’m hard of hearing, but I just want to say thank you for your time and effort into adding subtitle😊
No problem! I try my best everywhere, even on websites with A11Y!
I, also, hate using 5.25" hard drives. Such a pain ;)
I know, the Quantum Bigfoot, it's a nightmare, it's even not have the ultraDMA mode.....
I had 5.25 on my mind because I was trying to see how many drives I could fit in the Corsair case's 5.25 bays when writing this🙃
@@TechnoTim how many can you fit?
Better than using 8” floppies
@@yuan.pingchen3056 LOL I remember those I only had one in my time... I also remember MFM/Winchester drives from the days of AT/XT days.
4:40: x16 will only run as x8 and x8 runs as x4 if you are using a feature that shares the same lane, x16 and NVMe Gen4 slot for example.
It's important to know your mobo limitations like how many PCIe and which features shares the same lanes.
I went bear metal on my home lab server for a while, because it could do everything I wanted. However, things changed and now I’ve reinstalled everything on proxmox. The overhead is low and you have the flexibility anything in the future. So I would just install a hyper visor of you’re choosing.
Grizzly, Black or Brown ?
@@Marsh.x Sorry i didn't clarify, black bear metal of course.
@@redhonu Black Metal Bears go hard, but them Polar Metal Bears really be flexxing. Almost as much as the Metal Spirit Bears.
"Jamie, Pull up the video of the Metal Bears in a mosh pit"
I’ve run for years an unRaid server which had a w10 VM with GPU and NVME passthrough that I used for playing games and the rest of the system was used for docker stuff: Plex, arr suite, homeassistant and a large etc.
Just make sure you have a Renesas chip based USB PCIe card passed to the VM so you can plug and unplug peripherals without freezing the VM.
Your public library might have a 3D printer if you don't want to purchase one. I use my library's printer often.
I moved to a single giant server build a while back from my own giant rack with dell power edge servers. But it was to much power usage. For most use cases in a single server build. I found unraid to be the best base OS for me. 5 years later. Still rock solid and never had any major issues. Kind of on autopilot and it just works.
Been running unRAID for 🤔 3+... I've lost count. 😆 The UI makes it's super easy to manage everything.
200 Watt idle. Where I live (Netherlands) that's about 500€/year.
ouch. typical USA kwh price is .15 USD about 262 USD a year for 200 watt / year
You can thank the failed sanctions on Russia for that.
Why I have an Off-grid solar system running my homelab now. My homelab can push 500watts when processing. Guiltless home-labbing!
For an ultimate all-in-one homelab server, a hypervisor whiteout even thinking. One solution (nearly) fits all.
I have gotten an 80 mm fan, some double sided tape and attached it to the case for the airflow for an HBA card before. Just get the stongest tape not the least expensive. Mine has been working fine for 4 year now.
15:18 When racking servers by myself, I've found that there's usually holes in both the sliding part of the rail in the rack, and the stationary part of the rail (the portion that attaches to the rack itself). Every rail is different, so it always takes some experimentation, but I've found that I can put a spare screw/toothpick/pointy-thing through both holes so that sliding part of the rail doesn't push back while I get things lined up. I do this for both sides but sticking out different amounts so I can line things up one side at a time. Just be sure that the holes you pick in the rail can be reached from the front of the rack.
5 1/4 DRIVES?? They're 3.5 inch!
Oof! What the heck was I thinking when I wrote this. I think this snuck into my brain because I was playing around with my old Corsair case and was trying to figure out how many drives I could fit in the 5.25 drive bays 🤦♂
@TechnoTim we've all been there, lol!
@@TechnoTim I think I still have some old SCSI or maybe RLL 5.25" HDDs. Just in case you need one that has a sum total of 32Mb. (megabytes).
He was using Quantum Bigfoot drives ;)
Beat me to it!
@5:04 I don't know cpu lane calculation is as simple as you explain. you should probably check motherboard info, how many lanes are used for internals like lan, usb, ...
5:14 look at the block diagram instead, it may not what extra features are disabled on the motherboard with fully populated PCIE lanes.
On my system it will trade between m.2 slots and 4x PCIE connections.
Proxmox for the win!
No reason why, I just love it.
Also, thanks for the info on the Sliger cases. You just made me spend more money, they look great and are made in the USA for a reasonable price. My Threadripper platform is getting a new home. :)
Proxmox or xcpng are what I'd go with. It's nice to not have projects competing for ports or service configs.
I rackmounted my PC a week ago. The chassis was less than (the equivalent of) £100, including rails, and fits my 3 chunky radiators in too. My office is so much cooler and incredibly quiet now.
Great Video!
Just be sure to look at your motherboard spec when concerned about PCIE lane availability. Motherboards also come with a controller with additional PCIE lanes.
Basically, the processor's PCIE count is only half the story.
Probably too late to the conversation but recommend sticking with Proxmox but using SRIOV to pass through part of the GPU to multiple machines
I don't think this is the ideal project to try Unraid on, but you really need to try it if you haven't. I always have at least a couple of servers in my rack and love changing things out, but the Unraid box is always there
Same here. 😁👍
Taking a play from my corporate sys admin world. Separate storage and compute boxes.
Going to build a TrueNAS Scale box as the central storage for the entire homelab. Then use the 2nd unRAID lic to build a fresh compute server. Both will have 10Gbe until i can swap for fiber.
Switch to a fiber SFP+ transceiver and it will run much cooler. The RJ45 copper ones run really hot and use a lot of power.
DACs are a nice alternative as well.
I have a dual 10 gig E card in my unRAID and noticed first hand hot toasty it is. Now I wish I didn't give away the SFP card 🤦🏻♂️
3:12 5.25" drives? Don't you mean 3.5" drives?
A lot of multi-drive bays still occupy 5.25" docks.
@@PhotonHerald Yes but this isnt a dock, these are just the drives.
Great video. Some of your inflections remind me of LGR. I was chuckling.
Hey Tim, love your videos. Fellow homelabber here, but I can’t stress enough as a homelabber you need to RTFM on the motherboard and CPU. In particular look at the block diagram in the doco. Super micro docs are always good and show how the lanes are used as some of those 48 lanes have to service usb, SATA, IPMI etc… Also, the motherboard silk screen will also say 8x in 16 etc to tell you its only 8 lanes on a physical 16 slot. Don’t put that GPU in the first 16 slot if you want those juicy 16 lanes.
@@simbozoni thank you!
Strap a fan to that hba. It needs airflow. Look up the cfm needed for it and youll see why, or touch the heatsink after running under load for a while
Perhaps not the best idea to touch it, particularly when it's installed like that where it's bound to get pretty hot
Don’t go with the EVO 870’s, I made the same mistake (They are consumer drives and wear out quickly!) I replaced all of them with the Samsung SM883’s (ZFS pool)
I would go with Unraid for a build like this as it's easy to use and have a lot of features and if you really want to tinker, it's running on linux and have a terminal for any custom tinkering :)
I'm curious how your SSD pool works out. I have a Truenas Scale setup with 6 2TB Samsung EVO's and I've struggled to get decent performance. Literally it's less than a single disk. I tried Stiped 2 x (3 raidz1), 3 mirrored vdevs and raidz1 across 5 of them.
Hopefully you can do a follow up video on your SSD pool setup and final performance numbers.
Consumer models performance will drop I believe because of the drives cache, PRO models address this. And enterprise SSD's will be far superior. At the end of the day, you get what you pay for...
Hey Tim, great build can't wait to see how that works out, Always looking for projects like this. I ended up with the iStarUSA D-410-DE36 case that allows for 36 drives a while back for the hotswap trays.Paired it with NORCO RPC-4224 4U with uses 24 3.5in drives. Purpose was to run Flash NAS and backup to spinning disk NAS. If you end up making 3d printed drive holders for this project, I would love see that adventure.
You mentioned this will not be taking over your NAS role. What about your firewall? I've seen some folks virtualize pfSense or vyos, but I don't think I would be comfortable hosting a firewall/ids/vpn on the same machine in case the virtualization solution has a security issue.
The Silverstone RM52 is fairly roomy, rackable, and supports dual 360 rads.
Sliger offers watercooling options for the Threadripper/Epyc line of CPUs in that chassis. I'm not aware of any constraints that would stop you from using random AIOs as long as they aren't too thick for your GPU clearances. There is a build someone did in that case with a thick 360MM cooler and a 3090ti.
There are lots of makerspace’s that have 3D printers. Some public libraries as well.
creating a dedicated VM with the GPU is kind of limiting IMO, I find that using the GPU in lxc containers is the best way, you can have multiple containers use it at the same time and even monitor its usage from the host os
Or split the card, but that requires finesse and doesn't work with all cards
@@BoraHorzaGobuchul very few cards suppprt thay and generally they're expensive and require licenses
@@WhyDoesNothingWork it can be done even with some consumer cards. Those are not super expensive. Some more expensive pro ones can be found on the service have market.
Intel cards will perhaps allow this as well.
@@BoraHorzaGobuchul if your talking about sr-iov only the commercial Intel cards do it and the chances of finding a consumer cars that does it is like winning the lotto
@@WhyDoesNothingWork there's a number of consumer Nvidia cards that work, but it does require rtfm and research, not something I'd love to do now. Then again, at a younger age I'd've already done that just for the lulz...
Question, is that power supply able to handle the power spikes 3090s are infamous for? Genuine question
I just finished building a server around a EPYC 8024P and a Gigabyte ME03-CE1 motherboard. I'm contrast to this build though I don't have a GPU yet and I need it to double as a NAS, too. 96 PCIe lanes is plenty, room for growth (CPU, RAM, PSU, GPUs).
I used SuperMicro SuperChassis case though (no rack).
What about your server in the DC?
Can you still buy 5.25" spinning rust drives?
I can’t remember landing the plane in Top Gun on the NES. Well done ! 🎉 As for the OS, I would vote for Proxmox or a regular distribution like Debian, Ubuntu or a RHEL clone. You could use Cockpit if you want to spin up VMs on them.
You mention the AMD in the script that has more PCI express lanes but in the video showed 20,16 then said "same number" but actually the Ryzen series have Native PCIe Lanes (Total/Usable)
28 , 24.
@Tim, it would actually make better sense to space your drives apart for better cooling and less restricted airflow in certain areas of the case. Since you have 6 drives and 3 fans, that's easy. Just space them where you have 2 drives per fan. The drives will run cooler, which will make them last longer and sustain the top speed longer, and they won't interfere as much with the airflow.
 I need advice to make a quick decision  I got Dell Precision T7920 Tower, 2 X Silver 4110 2.1GHz which am planning to use as a NAS drive, is it over kill or I should get some cheap PC to use it for that? thanks in advance. 
Hey! Quick question, what rack rails do you use? great video!
Yes! Thanks for the change in content
Tim, great video! Have you considered exploring Proxmox on this server and demonstrating GPU passthrough? It would be valuable to see which remote software is optimal for accessing VMs, and perhaps even conduct a gaming test. Looking forward to your future content! Greetings from Bosnia :)
I'm about to rebuild my server and I'm actually really curious what OS you're gonna run because it may influence my decision. I really didn't enjoy truenas because setting up apps wasn't intuitive at all. Curious about unraid but not thrilled about the price. Proxmox seems logical but I wanna be able to add more storage later on (Ideally) into the same pool. Cool build! Will be looking forward to the part 2!
Hi Tim, Quick question
What are you using for your NAS currently?
If you do find a way to convert those 3.5" bays to 5.25", Wendell has shown some interesting enclosures that adapt 5.25" bay to 2.5" or nvme flash backplanes
Love the videos Tim, but why not just go EYPC? I got a 7601 cheap on ebay 32 core 64 threads has 128 pcie lanes and an ASrock rack motherboard with dual 10 gig built in?
Try out Unraid. It's imho the best OS for a single system. file shares, VMs, passthrough, containers and ZFS support
"Try" and that's it! Don't get me wrong. Unraid is good but...
ZFS isn't officially support, only community plugin & it's running in userspace! so your SSD's & 10Gbe are going to waste...
So many great open source OS'es to choose from... if money doesn't mean anything to you, by all means
@@kbng02 ZFS on Unraid is officially supported since 6.12, released in june 2023.
ZFS on Linux exists only as a kernel-level module. So what are you even talking about, even with community plugin it was running the same kernel module.
@@kbng02 ZFS is officially supported since last year, and ZFS always runs in kernelspace, you are out for lunch
Have you not considered Threadripper at all?
With the price of electricity I like to keep the 24 hour a day wattage to a minimum , about 50-60 including a rtx2060 super .
I have a Ryzen 5700g with 4 x 4 tb SSDs and 64gb ram , but no ecc memory and only 20 lines .
I plan to build a server with the 5700g and 64 GB RAM as well. Do you have any numbers of the power usage without the video card? I thought about and suggest that it will be around 25-30W but numbers are hard to find online (atleast for me). Would appreciate it if you could help :)
@@pesfreak18 switch all the power boost modes off and which 2 x 4tb SDDs and 2 x nvme 2tb drive , for a small nas on deskmini x300 just set up as a samba server 15 watts according to my power monitor plug on idle.
But my other system on a B550 motherboard , with the gpu and about 10% cpu all the time about 50-60 watts , but I think I must be the only person who is running a 5700g water cooled with a 360mm cooler , but it was a remove my x570 motherboard with a 3950x because I could not get it below 120 watts on idle .
16 core 32 treads was a little over kill for what I wanted my home server to do , in fact I think the 5700g is over kill.
@@peteradshead2383 15 watts seems very efficent. Water cooling is a little bit overkiil for me too but it sounds like a cool project. The Ryzen is maybe overkill but I want to experiment with gameservers and maybe AI in the future. I think I can utilize the CPU well enough with these tasks.
Can I please have the stl files for the ssd cage you showed 9:07, or atleast guide me to the closest model for personal use
Sorry I'm new to all this and have been watching countless videos on server builds.
What was that long green thing called that all your SSDs go into? Never seen that many hookups on one. Nice.
Welp I asked ChatGPT and then I took a SS of the image from the video and showed it to ChatGPT. It says i'ts a Backplane. I freak'n love technology!
So why are u preffering 48 pcie 3.0 lanes over 20-24 pcie 5.0 lanes?
I'd like to cast my vote for a baremetal setup. I have a Proxmox server myself and decided to turn an old ITX box into a minecraft+calibre server, chose to do baremetal Debian (Crafty Controller setup was problematic for me on Ubuntu). Kind of wondering what all I can run on this, wouldn't mind some inspiration from whatever you come up with. My little setup can't handle much, but I'm dreaming of upgrades to do more in the future.
Can you share the exact hw specs of this build?
100% add this as a worker node to your K8s cluster! No need to have a different management layer!
K8S ALL THE THINGS!!! 🤘
1:35 "GTX" 3090???
Is there a link for the Storinator case?
I'm contemplating similar build, and here are my observations:
for proper local AI selfhosting, having Mac Studio with 64GB unified memory is MUCH more suitable than RTX 3090 which is limited to "only" 24GB vram,
that said nVidia gpus utilizing CUDA are often faster than Apple's neural engine,
all reasonably-priced Xeons are limited to PCIe 3.0 which is not futureproof at all, and already bit of a bottleneck for current gpus,
any strong gpu takes lot of space and covers most PCIe slots on the motherboard, so it's rather difficult to decide on PCIe expansion layout, preferably gpu on "lowest" x16 slot, which then needs bigger than ATX case,
that said some local AI tools can utilize multiple gpus or even multiple computers, so there's a delicate (cost/efficiency) balance between running let's say 4 gpus in one rig, 4 computers connected via 100Gbe together,
latest Windows Server or plain Windows Pro, with WSL (Subsystem for Linux) seems best OS for widest range of local AI applications - arguably better manageable via Proxmox but with undesired performance loss,
to sum it up, one all-in-one server doesn't seem that efficient, depending on what to run on it
I would like to see you give unraid a shot since you haven't yet, selfishly I am curious what your thoughts about it on a setup like this. I started with a single PC like this a decaid ago running Windows Server 12 r2 and Hyper-v. Over the years I upgraded that server and eventually replaced it with a much smaller proxmox cluster. I still have that machine and I am not ready to let it go yet. I have thought about unraid or truenas.
How about Nutanix for your Hypervisor?
Hi. I would like to know your opinion on how long the SSDs would last compared to regular HDDs on a more intensive writing/reading application. Let's say rtorrent with 2-3TB of data. I have a dedicated HDD just for this purpose that is passed directly to the VM and I am reluctant to change it for an SSD.
Thanks, the way I look at it is that all drives have a lifespan, some fail faster than others, and others out live their lifespan. Since I look at all drives through the same lens (whether SSD or spinners) I just make sure that they have a long warranty and have one spare just in case. These Samsung drives have 5 year warranty and out of the 30+ EVO/Pro Samsung drives I have bought over the years, only 2 failed and Samsung replaced them within a week. For this reason, I typically go with consumer grade.
what patchcables are you using? they look fancy
So where did we end up on power usage?
When you were talking about the power supply with those giant connectors you've never seen before, you really proved to me how old I actually am, so thank for that. LOL. Looking forward to seeing how this turns out.
Built a homlab server as well - ran Unraid for a while which was in the end a patchwork of tools for functions that a server system should already have integrated. And running from a USB stick is not something I want to rely on.
So I decided to run Proxmox which is the most flexible system in my eyes. It's lightweight, can run VMs or LXCs, passthrough hardware, it's reliable and definitely the more professional choice.
It's basically only the config files stored on the USB stick.
Unraid will load its config into RAM and run from there.
For reference my Unraid system has 19,877 reads and 7341 writes to the USB stick and it's been in operation for 2 years with a lot of changes.
Should my USB stick fail, I can just download the automated backup to a new USB stick, recover my license and I'm up and running again.
@@Hansen999 Thanks for the information! I know the story. Used it for three years-
So apparently we are on the same tech wavelength and I just now noticed it, lol. I too just built the a few servers to do all the things. I ended up going with an Epyc system for all 128 of its PCI lanes dumped into a HL15. Another machine made use of the SilverStone RM41-506 4U chassis. I needed the 5.25" bays for the tape drive and an Icy Dock 4x 2.5" ssd hot swap cage, but the GPU just fits :)
I've been eying the dual epyc setups on ebay to eventually migrate everything to. I have absolutely no need for all 128 lanes or cores/threads, but it'll be all I'd ever need and then some.
@@DrDipsh1t Wow yea dual Epyc would be Epic 😂 I am actually making pretty good use of the PCI lanes on my 7282 16c/32t rig. A 16x PCIe card that holds 4x NVMEs, 16 port HBA, Radian RMS-200 Edge Card and a dual port 10gig SPF+ nic. Still have 2 more NVME m.2 slots, 2 OCuLink ports as well as 2 mini sas connectors on the motherboard I could populate. Its an Epyc8d from AsRock Rack board.
If you didn't have the hardware already, I would've recommend with a Threadripper system because it offers more PCIe lanes.
(That's the direction that I'm heading down, except that I'll likely end up with something like an 8U chassis and then using PCIe risers/extensions so that it won't cover the rest of the slots. My other option will be a Supermicro 4U or 5U GPU server, but those are limited to dual-slot-wide GPUs only, which means that my 3090s will be blocking some of the other slots.)
I'm excited for the software part! What would you say are the disadvantages of proxmox in a build like this? Doesn't that give you more flexibility?
maybe try Zimaos its for the Zima cube but also works on generic x86. it is like casos but better with Vm and docker/container
I had the same idea a few months ago (all encompassing build with a lot of PCI), and I ended up going with a barebones Dell Precision T5820. I threw in an Intel Xeon W-2140B and some ECC RAM. Don't sleep on repurposing used workstation hardware!
Love my Sliger Cases. Have the CX4712 for my NAS and CX4150a for my desktop. I will probably buy a CX4200a for a gpu upgrade though. 3090FTW BARLEY fits with a low-profile radiator. It just fits a credit card between them.
i would go baremetal and install cockpit and cockpit-machines for the VMs with lxc img
Isn't that CPU a real limitation for an 'everything build'? Can't one just get a high end AM5 AMD processor, or are they really that limited in PCIe lanes?
Maybe for AI, need a GPU, SSDs and 10gig NIC
Interesting. I am literally building a AMD Threadripper 7960x using a Sliger 4170i case with the ASetek 836SA AIO cooler. Just waiting on the case to be delivered to start the build.
im tempted to threadripper my next all in one homelab build. but this is a good and cheaper alternative I feel
Tiny subtitling correction at 2:05. I’m pretty sure you just left a gap in speech, rather than starting a new sentence.
Generally though, your subtitles are very good. Thank you for taking the time to do them.
Thank you! All fixed!
Do a Harvester install, opensource kubernetes based IAC for containers and VMs with immutable nodes.
The last PCIe slot on that motherboard is served by the PCH and it’s a 4x, so it doesn’t “count” for the CPU lanes total. ;)
Why not use amd epyc cpu?
getting a long pcie riser cable and making full use of the CPU you have seems like the best move imo
If you look at the schematics again iot says it right there which Slots are which for example the first x16 Slot says CPU SLOT6 PCI-E 3.0 x( (in x16) so that is an x8 in a x16 connector which is connected to the CPU and the last x8 says PCH Slot1 PCI-E 3.0 x4 (in x8) so it's a PCI-E via the PCH and only has 4 Lanes. So in any case the GPU only will run on 8 Lanes on any of the x16 connector slots as both are only x8(in x16).
Thank you! Yes, I actually edited out the part where I explained that, I probably should have left it! When I did the math you can see I adjusted for it (8 x 6 = 48, and the last was 4 in an 8)
lol I love him talking about the specs and what he wants and i just want a raspberry pi for Christmas
I noticed you were counting your PCIe lanes. Depending of the board manufacturer and models, onboard components can also take up PCIe lanes as well. Such as NVMe, SATA, wireless, USB 3. Very common on consumer boards. Especially if they have NVMe slots onboard. Some boards have the option to disable onboard components to free up some lanes 👍🏽 this information is good to know if you want to convert your old gaming rig or workstation into a server and need full lane speed support on your expansion slots.
Thank you!
Thanks Tim.
11:12 Fans are not installed correctly on the CPU heatsink. Firstly, do they even blow towards the back of the case? (and they should). If so, they need to be shifted one slot down (as looking on the video frame), i.e. the one closer to the front of the case doesn't need to suck air via fins, but would push the freely-collected air into them.
They are, this is a pull configuration, and yes, it does exhaust out of the back
Wonder why noone considers the AMD EPYC 8004 series (zen4c, Siena) for homelab use. I'd go for them in a heartbeat, but I 'need' QuickSync/Arc (iGPU) so I'll have to wait until Intel finally catches up with their equivalent Sierra Forest lineup 😔
You can find Arc380 cards VERY cheap everywhere.
@@ickyendeavors4179 Dedicated GPU is overkill for my use case. Also I'm in camp Europe, so low power, low noise and compactness are priorities 🤷
@@koloblicin4599 Understandable. Obvivously, the more VMs. you are spinning up the more it changes what you need.
There are companies that offer 3D printing services; if you are not going to be printing stuff regularly, could be a good idead to look into one of those services.
Great video, love your content.
That bottom PCIe slot uses lanes from the chipset, as it’s marked PCH, not CPU.
Custom watercooling would give you back some more lanes. But i am maybe a bit to much in love with wc 😂
This describes EXACTLY what I want to do (in the intro anyways). Perfect workout watch.
Did I miss it? What Supermicro mobo are you using?
I'm also interested
Yes, you did. Picture of the board and the schematic both are of a X11SPL-F
@@shalak001Supermicro X11SPL-F
15:30 that's what server lifts are for 😂
So tired of them artificially holding back pcie lanes and also holding back on bifurcation of pcie lanes on consumer cpu's and motherboards.
For the 2.5" drives- print out a bar that goes across the top of the drives and long enough to touch both sides of the case. There would be small ridges printed into the bar at each drive location to keep them in place. That way you don't block air flow. And you could easily lift off if you need access to drives.
you really should take a look on a distro such as nixos, declarative config and such... Kinda like ansible... but better in my opinion