Building a 100% SSD Home Server - Is an Intel Xeon D a good option?
ฝัง
- เผยแพร่เมื่อ 2 ส.ค. 2024
- In this video I build a new home server with fully SSD based storage using the Logic Case SC-23400-2 from Server Case. I also try something a bit different by using an Intel Xeon D-2143IT motherboard from AsRock Rack removed from a second hand Datto appliance that I purchased from eBay. In terms of software, I considered several different options including UNRAID, TrueNAS and Proxmox VE however I ultimately settled building a custom system using SnapRAID and MergerFS running on Debian to give me an easily expandable filesystem with good storage density and full filesystem scrub and repair capabilities.
Buy the Logic Case SC-23400-2: www.servercase.co.uk/shop/ser...
Find similar Xeon D based servers (Affiliate):
- UK - ebay.us/GvqVz6
- US - ebay.us/wgYtPX
Buy the IPC-5100 Plus CCTV & Network Tester on AliExpress (Affiliate): geni.us/8kCNqG
Other parts used (Affiliate) UK:
- Hot Swap Drive Backplane: amzn.to/3TBEqks
- Crucial MX500 SSD: amzn.to/3yZDLS6
- MiniSAS Breakout Cable: amzn.to/3VO2ym8
- 4 way SATA Cable: amzn.to/3z0Bp5n
- Noctua 40mm Fan: amzn.to/3yWGuvC
- Seasonic 350w ES2 PSU: uk.rs-online.com/web/p/pc-pow...
Other parts used (Affiliate) USA:
- Crucial MX500 SSD: amzn.to/3F1swMM
- MiniSAS Breakout Cable: amzn.to/3ToMipR
- 4 way SATA Cable: amzn.to/3F41h4s
- Noctua 40mm Fan: amzn.to/3CNBdaS
www.camerongray.me/
/ camerongray1515
Chapters:
00:00 - Introduction
03:03 - Looking at the Case
14:29 - Where is the motherboard and CPU?
29:20 - This thing is basically new?
32:20 - Tour of the motherboard
34:53 - SSD Choice
35:29 - Software Setup Explanation
43:37 - PSU Selection
46:29 - How are we installing the SSDs?
50:33 - A cheap M.2 SATA controller - Do we need it?
54:13 - Keeping everything cool
56:50 - Building it!
59:33 - Tour of the completed build
1:04:26 - First power up
1:08:55 - Cheap SATA controller problems
1:09:54 - Software configuration & Performance
1:12:55 - Noise & Power Consumption
1:15:26 - Mounting it in the Rack
1:16:50 - Conclusion
AFFILIATE LINKS NOTICE:
Product links under this video marked “(Affiliate)” are affiliate links where I may receive a small commission on qualifying sales. Affiliate programs that I am a member of include, but are not limited to: Amazon Associates, eBay Partner Network and AliExpress Affiliates.
As an Amazon Associate I earn from qualifying purchases.
Purchasing through these affiliate links will not cost you any more money, however the commission earned significantly helps fund the production of videos on my channel. - วิทยาศาสตร์และเทคโนโลยี
![NONT TANONT - ดอกไม้ที่รอฝน (spring) - Reimagined [Live Session]](http://i.ytimg.com/vi/nY2PsFZFEOg/mqdefault.jpg)
Buy the Logic Case SC-23400-2: www.servercase.co.uk/shop/server-cases/rackmount/2u-chassis/2u-short-atx-chassis---low-profile-pci-card-support-sc-23400-2/
Find similar Xeon D based servers (Affiliate):
- UK - ebay.us/GvqVz6
- US - ebay.us/wgYtPX
Buy the IPC-5100 Plus CCTV & Network Tester on AliExpress (Affiliate): geni.us/8kCNqG
Other parts used (Affiliate) UK:
- Hot Swap Drive Backplane: amzn.to/3TBEqks
- Crucial MX500 SSD: amzn.to/3yZDLS6
- MiniSAS Breakout Cable: amzn.to/3VO2ym8
- 4 way SATA Cable: amzn.to/3z0Bp5n
- Noctua 40mm Fan: amzn.to/3yWGuvC
- Seasonic 350w ES2 PSU: uk.rs-online.com/web/p/pc-power-supplies/2132358
Other parts used (Affiliate) USA:
- Crucial MX500 SSD: amzn.to/3F1swMM
- MiniSAS Breakout Cable: amzn.to/3ToMipR
- 4 way SATA Cable: amzn.to/3F41h4s
- Noctua 40mm Fan: amzn.to/3CNBdaS
all that in 10 months.....so a fuckload longer than liz truss was in office....
Now do the same video in English
@MrDecide1 different people have different accents, you're more than welcome to find another video that you can understand better
thanks for sharing this video, can you also share what is that ´´mini display´´ you use in this video? thanks
I made a video showing it here: th-cam.com/video/DZQSkFl4yIM/w-d-xo.html (there's links to it in the description) - essentially it's sold as a "CCTV Tester" but it has loads of extra features including network cable testing and tracing (including being able to identify the location of a break) and can also operate as an HDMI or VGA monitor. It can even record the video from the HDMI and VGA inputs which is how I captured the video of the BIOS shown in this video!
Of all the 3U and 4U Logic Case chassis that we bought (about 10+), around 40%+ of the fans failed in less than 2 years. Although in fairness ours did not have that metal grill bit, so perhaps the ones you have in your chassis will be more reliable.
Hi Cameron. Started looking at your channel when you reviewed the Hisense Oled tv from Costco. You review made me hit the buy now on Costco site as there were limited supplies and I had been hovering for weeks, so thanks I find you other content to be great viewing. I've been footering with PC's for 30years 63 now, never too old to learn. After working on mainly Dell non consumer products I find that it can be useful to remove all exterior plastic and covers if possible. Thank again from the west side of the M8
I have been using mergerfs and snapraid for years now on my server, been super happy with them.
Another great video Cameron, thanks for putting the time into making it. It's nice that this is a fairly low power device given the state of the energy market in the UK right now.
Power consumption was definitely a large part of my reason for moving to solid state storage. I feel bad for people who have built home labs with huge amounts of ex-enterprise hardware - those huge dual CPU servers are super cheap to buy but the current prices can make running them 24/7 totally unaffordable! When it comes to setting up my old server to hold backups, I'm planning on setting it up to spend most of the time powered off and will have it set to automatically boot up when backups are scheduled. Previously running a couple of machines like that 24/7 would have hardly been an issue but nowadays it's around £10-20/mo per machine!
@@camerongray1515 RaidOwl put a video up in the last few days showing his rackmount homelab - I noticed on his UPS it was drawing about 400w idle, that's about 100 quid a month in the UK before putting any load on it. He lives in Texas, I guess electricity is cheap there.
Yeah, pretty sure power is a lot cheaper in the US, especially with this current crisis since it's relatively europe-centric due to the reliance on russian gas. Additionally people also probably have different priorities. For some people running a homelab is worth the power cost whereas for me, despite what it may look like, my server and network are a very minor thing in my day to day life so they're definitely not worth £100/mo to me, I have much more fun stuff I'd rather spend that on!
@@camerongray1515 indeed mate 100 quid is a good night out ha ha
My thoughts exactly! 😅
Great build! Thanks for mentioning the cheaper alternative to IcyDock for the drive caddy, as well as the aftermarket fan replacement for it!
"If it had said 'made in the UK,' I would expect it to catch fire." LOL
The IT Crowd fire extinguisher sketch immediately comes to mind any time I see something is made in the UK. th-cam.com/video/1EBfxjSFAxQ/w-d-xo.html
Nar only lasted a month if it was from UK
Cool build out! I've been wanting to go 100% SSD via TrueNAS for my media server, but I'd want to replicate capacities that I already have for uniformity... I currently run 8 4TB NAS SATA HDDs, and they work fine, but I'd love it to be silent as the server sits in my bedroom.. I"m used to the HDD chatter/hum, but without it would be amazing too. Cheers!
Nice work Cameron!
You've got a stupidly good deal there mate with that datto server :) Sadly I don't expect to find anything remotely this advantageous on eBay anytime soon :D:D Good find!! You didn't mention the case by the way, all the stuff you got is already a superb deal, plus you get to keep the chassis to use for something else too with the brand new PSU already in there...
Interesting board. Lots of 10GB, but as you say would prefer SFP+.
Awesome video, as always!
Great solution for the CPU cooling 👍 Thanks for the idea.
(42:22) I can relate! I have a custom NAS and I run regular Debian on it, but only so I could have full control over the software and administer it over VNC and X windows instead of through a Web interface made to look like some desktop environment. For the storage, I just use BTRFS, but didn't have the time to set up some of its more advanced features like subvolumes and snapshots. I found BTRFS RAID to be faster than the hardware RAID on the dedicated SATA controller on the previous mainboard, plus when that chip suddenly cut out in the middle of large file transfers, using software RAID saved me because I could just replace the whole mainboard. My setup originally ran Debian 9, which I upgraded to Debian 10 due to some long-standing bugs, and I plan to upgrade it to Debian 12 once that's released.
(42:33) I do run VMs on Debian like the first option you mentioned. It's not as intuitive as I think you mentioned, but I do it that way as I don't run VMs all the time on my NAS, only launching them when needed for specific workloads that involve multiple apps. If I want to run a specific service all the time that's a single app, I'd instead look into running it in a Docker container.
(28:22) That reminds me of when I bought a used Mac mini (late 2006 model), and instead of the original 60 GB hard drive with the Apple logo on the label, it had a 320 GB Seagate hard drive that was clearly taken from some HP laptop from 2010 because it had a HP part number sticker on it.
Great video! I only wish I could get those rails in the States or from Logic-Case if they ever become available.
those long shatner esque pauses keeps making me think my video is freezing lmao this gives me some good ideas planning on building a server soon
Join me…..as I…..build a server. Kaaaaaaaaaahhhhhhn!
Deja vu! Recently built the exact same server, shallow depth, multiple expansion card options including full height/length. The D-2143IT AsRock board is great, but the Type C KX/KR mezzanine cards do not have enough bandwidth to run at full 10GB, so on the lookout for a Type A/B full PCIe based NIC instead.
Interesting to hear that about the mezzanine card. Hopefully it'll be fine for me since all I'd really need is ~5Gb/s since this machine will max out at the speed of a single SSD. Once I get some sort of 10GbE switching I'll definitely be stress testing the NICs on this!
@@camerongray1515 Indeed, it caught me out, I couldn't work out why the network was running slower than it should be when performance testing. Many hours later I discovered some specs in a datasheet somewhere that confirmed the limitation. It runs at around 7.5Gbps so still adequate.
excellent, exhaustive upload
thank you
Excellent video, really comprehensive, thanks Cameron! Sadly, those Fantec caddies seem to now be discontinued...
Yeah, unfortunately they seem to just have very limited stock - they've previously sold out then come back again. I bought them a good while before doing the build when they suddenly got stock back. It's a relatively generic item made by "Genesys Group" so you may find it sold under other names - I actually found the Fantec one I bought by doing a Google reverse image search! Server Case now sell the 8 bay version though so that could be a good option with increased density as long as you're okay with being limited to 7mm thick drives.
Great video, thanks for sharing. It gave me an awful lot of de-ja-vu as a fellow UK homelabber that's also recently built a pair of 2U short-depth servers. I'm also considering a shift to custom MergeFS/SnapRAID set up although it would be running under Proxmox and would be a move from UnRAID. I'm interested to see how you're set up turns out long term.
P.s. I wish I'd seen your video before going out and buying a full size ATX power supply with a side-mounted 120mm fan for a 2U case! And also that excellent eBay server deal. The same seller has a similar machine up for sale and it's super tempting.
Do it! Proxmox is Debian. I dumped esxi and the complexity of disk controller passthough years ago for Proxmox w/samba,webmin, and zfs installed. It makes for a great home NAS with the bonus of an easy to use hypervisor.
Are you active on Level1forums ? I feel like I read a similar comment in one of the discussions on ahomelab thread recently there.
@@tanmaypanadi1414 Hi.. Nope, just a lurker now and then.. But I've been running "the forbidden router" in my setup long before Wendel forbade it. lol.
I was very tempted to do this - install Proxmox bare metal then setup SnapRAID, MergerFS, SMB.etc on Proxmox itself. I was just a bit worried about this potentially breaking with Proxmox updates since as far as I'm aware it's something not officially supported by Proxmox. Although this may just be more of a fear to me coming from a mostly XCP-ng/XenServer background which is much more locked down and requires a lot of care to ensure that anything installed on the underlying OS (mostly monitoring agents in my case) can survive updates. Might be something I'd try in the future but would be interesting if Proxmox would provide information on whether installing Debian packages on the underlying OS is something they take into consideration when testing updates.
Just stumbled on your channel, good stuff! Keep up the good work… and not sure why everyone has problems understanding you, i set you to 2x speed and it still makes sense lol
What model datto appliance did you use for this?
I had two of these (from ex clients) - I "melded" the 2 into one (have a spare MB and PSU) including the fans and memory (using the original case) and installed UNRAID. Used the existing 3x 6TB drives (One for parity) one 4TB SSD (from another Datto device) and 2x 2TB NVME drives (one on an addin card). Been running for a few months on the original PSU just fine. Dockers include - PIhole, Shinobi (With 3 cameras), Sonar, Radar, FTP server, Windows VM etc. Hasn't skipped a beat. This is my second UNRAID server in 3 years. Note - using original case, 2x10gig NICs and PSU.
Hi. Great video. Watched it several times. Got one of these boards based on this review but did not got lucky with an optane and a decent ssd. Anyway touching you tweaked the power settings in bios. I did the same but still iddles in the 55-60w - eye popping. Do you mind sharing your settings in bios and bios version. Looks very hungry to me still.
I had my suspicions that it was a relatively new server when I saw the UKCA conformance label on the RAM sticks…confirmed by the PSU!
awesome r&d, thank you :) seems shame just to use it as just a *home* nas, might source similar config to replace my aging i7-3632QM 16gb laptop currently running proxmox with pfense/airsonic/squeezeserver/meshcentral/win10/omv/homeassistant/jellyfin VM's (before it conks out)
hello, have you replaced the old laptop server? I had a config like that with an i3-3120m that ran for 2 years before failing (I'm not sure, it failed when i went abroad for 6 months, i still am abroad).
Regarding the front panel LEDs and switches on the casing, how did you figure out the connection to the motherboard without the manual?
Been there, tried that, and I bet you'll be on ZFS within a few months. Build your array with 3x matching drives in RAIDZ1 and stripe those groups-of-3 together. You can expand 3 drives at a time.
BTW, that m.2 SATA card you have is Apple-silicon compatible. Toss it into a cheap Thunderbold NVMe ensclosure (I have a Wavlink UTE02) and you can use it with an M1/M2 Mac . Works with ZFS for OSX. Create a pool with the right flags and you can move the drive set between Linux and macOS. I won't guarantee any kind of reliability but it is fun to play with.
Thanks for the heads-up on those Datto machines. Multiple sellers seem to have them and all are shipping from the UK. I wonder what happened.
Out of interest, what sort of issues did you have with your setup? Don't get me wrong, I love ZFS and use it extensively in other environments, however for this it just doesn't have the ease of expansion. As you said, I'd have to expand it using 3 drives at a time which would only give single parity. For double parity I could use RAIDZ2 but this would require me to expand with even more drives at a time. Then since parity is stored at the vdev level, I'd lose more capacity to parity data than I would with my current setup using a pair of dedicated parity disks. You also then end up with a tricky situation where you'd get better storage density by using larger vdevs (you'd get more usable capacity from 9 drives in a single RAIDZ1 than you would with 3x RAIDZ1 vdevs with 3 drives each) so it then comes down to trying to balance initial investment vs storage efficiency. I tried out Windows Storage Spaces and rejected it for this reason too - the storage efficiency is defined by the number of drives that you had when you initially created the volume, you can expand it in the future but that efficiency is fixed. It also makes swapping drives for larger ones in the future tricky since you'd need to upgrade an entire vdev at a time. By contrast with this setup, I will always just have two parity disks and can expand 1 drive at a time. If I want to start fitting larger drives, all I need to do is upgrade the parity disks to match. Then if I run out of drive bays and want to replace one of the 1TB drives with a larger one, assuming the parity disks are large enough, I can just upgrade the capacity of drives one at a time and immediately see the storage improvement. Once RAIDZ expansion reaches mainstream ZFS I'll definitely be giving it a go since it should allow single drive expansion however it will be interesting to see what happens in terms of storage efficiency as the vdev is expanded. I suspect you'll be limited to identical sized drives still but this would definitely make ZFS a much more viable option.
As for the M.2 SATA controller I was considering trying exactly that, coincidentally I also have a Wavlink UTE02 that I'm no longer using. However, it would only really be just out of curiosity since I can still return the SATA controller to Amazon which is probably more sensible since I don't really have a use for a Thunderbolt SATA adapter.
@@camerongray1515 Now that you mention it, I think I got the UTE02 on your recommendation. After much abuse I had to 3d print a bracket to stop the cable from wiggling but otherwise still works great.
It sounds like that M.2 board is overheating or is otherwise faulty anyway. I bought mine to try with a Pi CM4 and threw it into the UTE02 just to see if it would work with macOS. Few things do.
As for SnapRAID/MergerFS, it has been a long time. One problem is that it was slower than I expected. I was using it with spinning hard drives, though. The other problem was that even though I configured it to spread data out (and replicas were spread out) it always wrote new data to one drive. This may have been addressed. Documentation was frustrating.
I mostly love ZFS for the features, and a lot of my use is on single drives or laptop partitions. Sending the latest hourly snapshot of my work folder to the backup server. ZFS root and docker support on Ubuntu. Plugging a 4-bay USB enclosure into a machine (Linux or Mac) and all the NFS shares coming up with no extra config.
I don't mean to sound like a fanboy. They are all just tools. By all means, use whatever storage setup works best for you.
Oh, I fully get the appeal of ZFS, part of me not using it for this was also just so I could try something different since literally every other storage system I deal with uses ZFS in some way or another (Bare ZFS on Linux, Bare ZFS on FreeBSD, TrueNAS, ZFS on Linux + GlusterFS). In terms of performance, MergerFS is definitely slower since you'll only get the performance of a single drive vs ZFS where your data is striped. This is less of an issue with SSDs since they are already so fast but it also comes down to what you're using it for. If you're actively working on large files stored on the NAS then something with striping would definitely be beneficial, however here the machine is basically just an archive and I work on most stuff off of my local SSD in my laptop - the only files I edit live off of the NAS are things like word documents and spreadsheets. Then as for your issue with it not spreading files out, I definitely agree that the documentation is pretty lacking. With MergerFS, you need to specify a "policy" on how it spreads out data. As standard it uses epmfs "Existing path most free space" which *I think* tries to group certain directories onto certain drives. This would make it easier to find files on the underlying drives. For my use case, I opted for mfs "Most free space" which will replicate the directory structure onto all the drives and then split files between them creating each file on the drive with the most free space at the time of writing the file. This is important for me since the vast majority of the storage space used on my machine is my "TH-cam Edited" directory.
Not sure if it would work for every type of storage, but when I migrated my bare metal server that used a zfs pool for it's data storage to a proxmox vm, I found that you could pass the entire drive over to a vm rather than having to pass the controller over. I'm not sure if there are performance penalties either but it works.
Can you use a 2tb ssd drive on a xeon w5590 dual processor
I got an old server for free years ago, an LGA 1356 based server with a pair of e5-2470's, the board did quit working on me about 8 months ago with both CPU's, after a full tear down, I found a cracked solder joint on a component responsible for the 2nd CPU's power, I used a cheap solder gun and fixed it, been running non stop, as a plex & jellyfin server, pihole server, and a NAS.
I do have a box I keep my spare server parts in, I got an old FX 8320e system in there just in case my server does go south or if I want to expand on my network in new ways in the future, no real use for Proxmox yet, but I have been thinking about it.
"It's a bit messy, nothing that can be done" referring to the cable management.
Me looking at the best cable management I've ever seen for a server
what was the model number of the Datto server you bought to get your MB and parts for this build?
I am now searching for a used system for a powerful but low power NAS/VM server. I haven't been able to find a smiliary system on Ebay in the USA with those FUJI MiniATX MB's. Do you have a reccomendation for how old of a Xeon or Core CPU we should buy and which to avoid? Currently my search for XEON is 2018 or newer and for CORE it is 8th Gen or newer. THey must have the two virtualization features, a TDP of 50w max, and at least a 2.2Ghz base clock. The CORE processors also must have on board GPU so I don't waist a PCI slot on video. Does that make sense? I am curious as to what the criteria is for excluding the older models.
one thing you might like like is the write life of the mx drives they wear out really quick with zfs on them and all the log files. i always try and have a spinny to deal with the swaps to improve the write life.
It'll be interesting to see for sure, does look like the MX drives have lower rated write endurance than other drives, however I've yet to have an issue on any that I've had in production for years which see relatively high write usage on a pair of servers that regularly replicate data to each other on a nightly basis. In this build I'm not too worried, sure the boot volume is on ZFS but it shouldn't really see much activity and with 64gb of RAM, I doubt I'll see much, if any, swapping. The rest of the storage drives will see very little write activity since they work basically like a long term archive - once a file is written to them, it'll likely stay there forever. It's likely I'll have upgraded from this server long before the drive's write endurance becomes an issue.
@@camerongray1515 typically 3 years is what i see on my proxmox install, thats with 32gb of ram and an i7 ( i wanted the lower power stepping you mentioned on the cpu ) its been fine i used open media vault as the nas controller vm with Hba passed through i was just shocked that the mx drives burn so fast however on price to performance you cannot beat them
is it possible to make a nas with a xeon/epyc using a noctua cooler, so it would be quiet?
I can't see why not, you'd just need to find a cooler that has suitable mounting for your CPU, enough cooling capability to handle it and a suitable form factor to fit in the case that you're using. The reason I can't do that here is because the Xeon D CPUs use their own cooler mounting pattern and there aren't many third party coolers that are compatible with it.
Is there any suggested cheaper alternative to that mobo+cpu?
Is it worth buying the datto rack and use in the PC case? I would like to use most of components such as mobo, power supply and etc
I think I'll build one of these !
really nice case just a shame they don't use 80mm fans now it is 2U high
Great video and very informative. As a personal project I’m looking to build my first NAS. It’s a steep learning curve but I intend to use this not only for media storage but as a Plex/Jellyfin server as well as storing home CCTV potentially running Blue Iris. If I could get my hands on this server would it be an ideal for this purpose or would there be some limitations? I’m not sure if this Xenon CPU supports transcoding. I’ve done some reading and would like to utilise CPU transcoding at the highest 4K HDR (10-bit HEVC) for Plex/Jellyfin and think this Xenon falls short as it’s Skylake. I think I would need the next generation. Do you have any similar recommendations to this build perhaps?
I'm not overly familiar with Plex type stuff but it's my understanding that you'd want a GPU that explicitly has h.265/h.264 acceleration hardware. This is generally built into the iGPU on Intel's desktop CPUs and some Xeon E chips (known as Intel QuickSync) but since chips such as this Xeon D are designed for servers without intense graphical requirements (this particular machine using the Aspeed BMC as the GPU) they tend not to have integrated GPUs. For your needs you'd probably be better off looking at a regular desktop Intel chip or a Xeon E that includes an iGPU, or look into adding in a discrete GPU for transcoding.
This got me wondering what you have running on your servers!
This is mostly just a NAS. In terms of other stuff running on it, it'll host my home automation system (Node-RED, Home Assistant, MQTT Broker, Grafana, InfluxDB) and a local DNS resolver, nothing much more than that.
Curious to know how the SSDs would do in a 24/7/365 environment as they are consumer drives and not built to handle constant use.
It'll be interesting to see however I'm not worried, sure they're powered on 24/7 however they sit idle the vast majority of the time, only being used when I read/write files or when performing SnapRAID scrubs so it's likely that they'll actually see less wear than if they were used in a frequently used PC. They're also actively cooled whereas in most PC/laptop situations they wouldn't be. Even the boot drives which run a few VMs don't see particularly high usage, just regular OS filesystem access and small writes to logs and a time series influxdb database containing home automation data. I've used MX drives in much more intensive server environments and they've been running fine for years without fail.
Good video. ty
what is the name of the small monitor tester? does anyone know? and where can I buy it?
It's a "CCTV Tester" from AliExpress which is also great to use as a small VGA or HDMI monitor and for testing cables. I made a video about it (including links) here: th-cam.com/video/DZQSkFl4yIM/w-d-xo.html
About a month or so ago i purchased a SATA extension card that plugs into a PCIe gen 3 x1 slot (because i needed to connect more than 4 SATA devices into a PC that only has 4 SATA ports on the motherboard, and the only spare interface was a PCIe x1 slot (i have no idea what generation the PCIe slots are on that PC)
I currently have 2 devices connected to that card (a 4TB 5400RPM HDD , and a DVD-RW drive), the card in question has 8 ports
There are currently 3 devices connected directly to the SATA ports on the motherboard, a 2TB SATA SSD, which i am using as the OS drive, a 1TB SSD, used for data, and a BD-RE drive (the motherboard does not have any M.2 slots, so i cannot use an M.2 SSD)
(I am thinking of maybe getting another HDD for putting the VHDs of VMs on, just so i am not wearing out the main SSD in that PC by constantly deleting VMs and creating new ones)
The HDD has read speeds of about 180MB/s, and write speeds of about 170MB/s, although it fluctuates (the fluctuations could be caused by the drive itself though, not the SATA card)
And you have seriously tempted me to put together my own home server (I currently have an “off the shelf” NAS, with 1 HDD in it (4TB))
How did you connect the wireless keyboard to the server and it worked at first boot w/o configuration?
It's a wireless keyboard but it uses a USB dongle rather than bluetooth so doesn't require any configuration.
You could also 3dprint a brackwt to offset the powersupply in whixhever directtion is required for air movement.
I bought the two different types (pcie/sata) of that 5-port M.2 adapter from aliexpress and none of them worked in any machine I tried. It just does not get detected.
Yeah, I suspect they're maybe a bit temperamental, on my board they are detected in the OS but not in the BIOS so it worked to add storage drives to the OS but couldn't be used as a boot device. Treating this experience as evidence that my initial suspicion was correct and I'll stick to higher end HBAs in the future
Is there a full hight expansion adapter in the box they ship or did they not update the pictures as it's not there on the site?
Yeah, I think the photos on the website are possibly outdated as it came already on the case I received. If you absolutely need that expansion bay it may be worth dropping them an email before ordering to confirm that it'll be on the case you receive.
@@camerongray1515 thanks il give them a message, plan is to condense my servers if possible as is stupid running a 4 u case for one graphics card
as always love the videos and projects you do
Hello Cameron. Great video. I was planning in building something with how swappable hard drives. I had a bad experience with failing HD, so nowadays I rather something with hot swappable drives. The cases that I have seen online are awfully expensive. Are you aware of any place I could shop for rack mount cases with opening for 4 or more bay? Ebay just is not cutting.
Just found you.... Now subscribed.... Thanks for the video
haha no way, I got the same deal 2 or so years back ebay liquidation company, a 1U and a 3U datto. bloody amazing value
Nice!
"If that had said 'Made in the UK 🇬🇧', I'd expect it to catch fire" 😂😂 - I had just been thinking the same thing as you said it!!
I have an older Xeon D in a supermicro mini server. Not rack mount unfortunately, I have it on a shelf. That has no real storage just boots vmware and I use a synology that has ssd cache as the NFS datastore.
Since Proxmox is based on Debian anyway, I have my server machine just running Samba directly on the host. I don't actually have that many VMs running on the thing very often.
Is this shot on the Panasonic HC-X1500? It's very good quality
Yep, it's a great camera! Not the best at low light due to the small sensor but with enough light it gives amazing results!
@@camerongray1515 Thank you!
I keep seeing more and more TH-camrs saying that three pin fans are not speed controlled. They definitely are. You cannot get them to spin as low as pwm is the whole reason PWM was added. With three pin fans you modulate the voltage and if you drop too low, the fan just won’t spin, but with pwm fans you pulse the full 12 V so you can get them to spin as slow as you want
Of course you can control 3 pin fans by varying the voltage, but this requires the motherboard to support this which is generally pretty rare and definitely not supported with the motherboard I'm using here. Motherboards offering voltage control for fans definitely used to be more common but nowadays they mostly seem to have transitioned to PWM control only so it's generally a much safer bet to simply recommend 4 pin fans for situations where speed control is important. My comments about the fans in this video was basically a throwaway comment based on the fact that I knew the motherboard I was going to be installing only supported controlling the speed of 4 pin fans.
SSD storage never seems to last as long as regular spinning HDs in my experience.. Most of my failed drives are Solid State, Magnetic Drives seem to last almost forever or until they become obsolete before they fail. SSD can't be beaten for performance though, so it's Swings and Roundabouts. Best to backup the array to the cloud, then it make sense to invest in a build like this. This seems a relatively cost effective build and quiet for a home based solution. Synology make a nice small 2.5" NAS that works with SSDs and that would be even cheaper still.
Strangely my experience has been the opposite having never had an SSD fail but having to deal with hard drive failures on a monthly basis. Although I suppose it's also a game of numbers as I deal with significantly more HDDs than SSDs. Ultimately it just comes down to having a good backup and integrity check system in place. When I build any sort of system, I design it expecting every single component to fail at some point.
@@camerongray1515 Thanks for the reply, it was an interesting project to watch.
I can say from personal use, I've had several hdds fail but yet to see an SSD fail. Consumer drives mostly, non-raid notebook/PC scenario, some working 24/7.
The tbw is relatively low but ssds are often at a slight disadvantage as they are often quite full with little free space for housekeeping. Sample size not too big with number of drives in systems I manage about two dozens at the moment.
Unraid unsettles me a little bit too 😂 Thanks for sharing your knowledge on it - loved the "software setup explanation." I'm about to use Synology Hybrid Raid (SHR) on one of their models that has it for similar flexibility, I think it's just something to do with Linux LVM and/or mdadm but of course their implementation is proprietary. And it seems like they are locking things down more and more with the "use Synology drives or else" on certain models.
How much did you spend in all?
alternative build for smb is simple - 6/7th gen refurb or 10th gen and nvme - also since nvme scales so well you can add nodes, have high availability- nvme and faster network is way but price/perf is a moving target - going all ssd is still a good step #price parity
NVMe would have been ideal but the price difference is currently astronomical. In the future it'll definitely become much more realistic but for this build it was out of the question in terms of budget. It's not just the drives, the motherboards/CPUs with enough PCIe lanes and the required, breakout cards and backplanes also massively increase the cost when it comes to NVMe. SATA/SAS can still go a very long way in terms of HA and clustering.
I guess power consumption will much high?
as for cooling you could have took the server fans out of the server and used the air flow guid to cool the CPU. You pretty much took a 1u and made a 2u and that air flow guide would have worked on the new case. Just had to mount the fans.
From a cooling perspective that would have worked well however the original fans would have been far too loud, this machine sits in a rack in my office next to my desk so it needs to be as quiet as possible.
@@camerongray1515 how many 4 pin headers di the mother board have for fans. The server I have I have like 8 of them I replaced the old ones with noctua PWM fans and its crazy quit, the only time I hear them is if my office gets a little warm. My network switch is way louder LOL. I tried quite fans in those and well they ran hotter so I kept the noisy ones.
It has 5 4pin headers however even with them, the original fans would be too loud. Those noctua ones are great for many situations however I personally wouldn't trust them to move enough air to keep a CPU cool without having them running at a relatively high speed. With this build, since I have a 2U case I can't really see a reason to use 40mm fans to cool the CPU when a practically silent 70mm fan can fit.
Hi - Just subbed....couldl you tell me where you got that small external screen from please?
It's one of these CCTV testers from AliExpress: bit.ly/3gquJan (Affiliate). I made a full video about it here: th-cam.com/video/DZQSkFl4yIM/w-d-xo.html
@@camerongray1515 ah - brilliant = thanks very much sir. !
I've been given a HP DL380 G7 server and thinking of using the chassis for a server build. However, I do have a PC case that I might do first. What I'm after is a quiet and low powered option for a server.
To be honest, if you're looking to custom build you'd really need to use a regular PC case or if you want something rackmount, you'd need a rack case that's designed for standard motherboards like I used here. Prebuilt systems such as that HP use completely proprietary form factors so you won't be able to fit any other motherboards short of absolutely butchering the case and cutting it to pieces. If quietness and low power consumption are key, you'd definitely want to go for a decently sized PC case with space for 120mm (or larger) fans since these can run at slow speeds and still move enough air. When you get into the realm of rack cases you'll begin to need smaller fans which have to run louder at higher speeds to move sufficient air to cool the system.
@ngray1515 I do have a full size PC case that I can butcher or a PC case I got for $15 Australian for a small server build as well. I just need to make more money to make it happen 🙂
What about Water Cooling the server like what Linus Sebastian did on one of his servers.
quite a nice case, shame they didn't use the case size to use 80mm fans instead of those tiny loud ones...
why not use nfs instead of smb?
Same question SMB seems so inefficient
This server can't do trans coding for something like plex and you would have to get an gpu correct?
That is correct, for that sort of thing you'd either need a GPU or more likely, a CPU that has video transcoding hardware such as most of Intel or AMD's desktop chips rather than a server chip such as a Xeon.
@@camerongray1515 Does: that mean in Intel CPU with onbaord grahpics has to have "Intel Quick Sync Video" to do transcodeing on a nas or plex server without a GPU? Does AMD have something similiar. That way I can chose a CPU based on those specs and not use a PCI
Sorry for hijack this video comments. I am building a server too with fujitsu tx1310. Saw your post about it years back. Any chance you still remember what front panel pins for the power on off?
I never ended up using the motherboard in that build, just the RAM and CPU. However I still had the remains of the TX1310 sitting around so I've just taken a look. When facing the board with the edge containing the front panel connectors towards you (so that the words "Front Panel" are the correct way up) the two leftmost pins in the top row are the power LED and the two leftmost pins in the bottom row are the HDD LED. The power switch is connected to the two pins to the right of the power LED pins (so, top row, skip the leftmost two pins, then use the next two pins). Hopefully this helps.
@@camerongray1515 thanks for the info
Nice little server that. Very satisfying. Those dinky little hot-swap bays are cute.
When you said you weren't satisfied with 60 watts but then you tweaked the BIOS settings, I thought you were setting the stage to declare a massive improvement. After that, "48 watts" sounded ridiculously unremarkable!
Personally I would not put up with that much power consumption just to have some storage on standby. It would make a 170% increase to our house's idle power draw! But I suppose "low power" is always relative, and quite subjective.
I wasn't particularly clear but 60 watts was just at idle without any storage devices or fans, 48w was with all SSDs powered up, fans running and network connected which would all add to the power consumed. This is also significantly lower than my previous hard drive based server. Realistically, 48w under the current already ridiculous energy costs is £12/mo which for me is worth it to provide a large amount of always on storage. I need a way to access storage away from home and host other home automation type services on the machine that need to be running 24/7, it also allows cloud backups to run overnight. With the sort of stuff I do, I really need some sort of always available network storage, local storage on a single machine isn't really suitable and regular cloud storage wouldn't work well for regularly transferring 50gb+ video files.
@@camerongray1515 Ah, well that's different then; a big improvement. I thought you were comparing apples to apples.
I didn't mean to suggest that you built the wrong server, only that it would be the wrong server for me!
An unspoken subtext here was that I have a vague intention of building a low-power (both in terms of performance and power consumption) home server at some point, and up until your wattage reveal I hadn't bothered to stop and consider that a Xeon D platform might not necessarily fall into that category. (Because I was only watching the video for entertainment, not for research!)
Yeah, Xeon D is an interesting product line, some of them (model numbers starting with a 1) are specially low power chips whereas there is a separate range (models starting with a 2) which are very similar to the full size Xeon Scalable chips just combined into an SoC. For lower power consumption you'd be best to focus on the lower power Xeon Ds, Xeon Es or Atoms.
@@camerongray1515 what's your average power draw under usual to high load?
@Airbag888 As in the server or my entire home consumption? Server idles around 48w which is where it will set the vast majority of the time. When performing syncs or filesystem scrubs it spikes up to around 80-90w but this is only for a few hours a week maximum. As for my overall home consumption - I don't currently have any sort of realtime monitoring of this so don't really know (it's something I'm planning on building soon as a project) but it'll be relatively low - I use gas for central heating and hot water, all lighting is LED and most electronic items I use are relatively modern and power efficient.
Everyone should have a home RACK ! :-)
Thanks Cameron, very interesting. What rack is that?
It's a ZPAS SJB (zpasgroup.co.uk/floor-standing-cabinets-and-open-racks/4-sjb-19-network-cabinet.html ). I bought it many years ago (2013 I think) from ZPAS through eBay although they don't seem to list them there anymore so not sure where you can order them from - may need to drop them an email.
@@camerongray1515 Thanks, is that the 600mm or 800mm depth? I'm wondering if the case you used would be feasible in just a 600mm deep rack.
Yeah, it's the 600mm one, although if you have the space, 800mm would open up a tonne of more options. With the 600mm you're basically limited to self built servers or prebuilts without removable disks. When I bought it I was a student living in shared flats with limited space, nowadays I wish I had a slightly deeper one
I'm probably biased as a big Unraid user it's far more than just a storage system...and I agree if you just want basically a NAS there are better options out there.
Yeah, I just feel like it tries to be a bit too much. I feel like if someone took TrueNAS, added in Unraid's filesystem and added in file integrity scrubbing it would be perfect for me. I'd honestly pay so much money for such an application!
Synology's Hybrid RAID does look good - as far as I'm aware underneath it's just standard LVM and Linux MD RAID to slice up drives into smaller chunks then creating RAID arrays from them to maximise the amount of usable space on mismatched drives. So it's not really the same as UnRAID/my setup here which works at the file level but it's still an interesting way of providing a better level of RAID expansion over many other solutions. Unfortunately I don't really want to tie myself into a proprietary NAS solution and going beyond the basic desktop form factor Synology devices gets very expensive. They're definitely great products for a lot of people though!
SATA SSDs typically do not get that hot to require fan cooling even in enclosures like these. (I ripped out the fans from my IcyDocks)
Also the future you is likely to get mad at you for zip-tying everything to everything. :)
I was very tempted to do the same, I was just a bit concerned about all of the drives packed closely together where they'll all see large amounts of simultaneous activity when performing filesystem scrubs. I have the noctua fans running but at a very low speed, just enough to provide slight airflow through the drive bays while being almost completely silent.
Given the sticker on the Datto PSU suggests the system isn't even a year old, wonder if it was built for a client but then never used, either that or it fell off the back of a lorry!
I suspect it's never really been used, the Optane drive only shows 46 power on hours and that was after I'd been using it for a while and the time it'll have been powered on for while it was being erased by the recycler I bought it from!
@@camerongray1515 See, that would make me slightly suspicious of it, it seems almost too cheap for what is essentially a brand new system!
The only thing I could think of is it maybe went to a start up that went bust, and was liquidated before it even went into service.
Other than that, I can't see a genuine reason for it to be sent to a recyclers!
That's true although I'm not too worried - at that sort of age there would have almost certainly been some sort of warranty/service contract in place so if it had been faulty it would have been unlikely to have been recycled. The seller also had 6 of this exact spec in the one listing then several other Datto appliances so I suspect they've received a job lot of them from somewhere. Looks like the Datto Siris 5 came out earlier this year which looks to be based on Dell hardware (the machine I have was a Siris 4) so it may just be a case that the new model came out and this was some left over stock or machines that had been kept around as spares that were never needed.
The plot thickens - I was just getting rid of the box that it was shipped to me in and realised that there were original shipping labels on the box that showed a shipment from Datto USA to Datto Europe in Reading however other labels on the box showed a shipment of Datto branded antennas, not the server itself. I doubt that a company selling such expensive products would send a device to a customer in a second hand shipping box so this makes me think it possibly never left Datto and they ended up offloading a bunch of stock to the recycling company in some leftover boxes, possibly due to them being the previous generation. Additionally, the recycling company looks to only be around 12 miles from the address of Datto Europe.
Wow, very nice find! Thank you @camerongray1515 for the video! Now I’m tempted to buy one of those servers and reusing the MB/CPU. Do you know if it supports bifurcation on the PCIe slots? Would be great to be able to buy a cheap 4x NVMe card to it.👍
i had made a shroud for this to mount the fan with a 3d printer
Have one of these boards too, however the impi password is locked :(
I can't remember the exact process but I was able to reset the IPMI through the BIOS setup which reset the password to default which I think was admin/admin. On other machines I've also been able to reset the IPMI password from within the OS using ipmitool.
@@camerongray1515 The supermicro IPMI tool doesn't work on these boards. The ipmi firmware is locked. The password you are re-setting in the bios is just the bios admin password, nothing to do with the IPMI..
ipmitool is a UNIX application: github.com/ipmitool/ipmitool so it's not SuperMicro specific, I haven't tried it on this particular board but I'm pretty sure I've used it on Dell and HP machines in the past. As for the BIOS, I'm not at home right now so can't check, but it definitely wasn't the BIOS password that I reset, there was an option somewhere to reset the IPMI configuration and this included the password.
@@camerongray1515 ipmi tool works in dos or Linux. It won't work on this board, it won't even see the ipmi. The password for the ipmi reset isn't in the bios eitherm
Weird, I just got home and was able to test both methods with my board and they both work fine. For the BIOS it's under Server Mgmt > BMC Tools > Load BMC Default Settings. This resets the username and password to admin/admin. Then for ipmitool, I installed it from the Debian repos (not sure what the DOS version you mentioned is) and was able to list users with "ipmitool user list 1" then change the password for a user with "ipmitool user set password 2 temppass" where 2 is the ID of the user retrieved from the user list command and "temppass" is the new password. My BMC firmware version is 1.10.00 so I suppose different firmware may have issues or your BMC may be faulty in some way but these methods definitely worked for me.
I would love to get this server I'm at Nigeria
Very nice gear ..but I dont understand the need for a fancy server for only 7GB of storage
I presume you mean "7TB" - it's not just about the raw storage. It means that I can have all of my data stored centrally so I can access it from multiple different devices and because it runs 24/7, it can automatically run backups overnight. I primarily work on laptops where adding that amount of storage internally would be impractical, and splitting data across a bunch of external hard drives gets cluttered.
nice build just 1 little tiny problem: how to put a 4090 in it ?
Whoaaa dude, slow down. I'm sure most people have a really hard time understanding what you're saying. Articulate ffs. This is a super useful and interesting video and I'm genuinely interested, yet it's nerving to not be able to understand half of what you say :( Sorry for the rant.
Xeon E5 series chips support registered ECC and are a lot more powerful than Xeon d but Xeon e3 are not and only support unregistered ecc
Yeah, the Xeon E chips are based off of the desktop chips and the Xeon D is based off of the E5 (and latterly Xeon Scalable) chips. The chip shown here is based on the Skylake Xeon Scalable chips. Performance wise there are almost two different ranges of Xeon D, the models starting with a 1 are lower power parts with relatively low performance but low power consumption. The models beginning with a 2 such as the one I used are higher power parts with performance comparable to the traditional Xeon chips but as an SoC architecture. The chip here performs similarly to some of the Xeon Silver/Bronze chips.
@@camerongray1515 yeh the Xeon D Chips are great ! Agree totally take a look at the HPE moonshot… that’s something else lol
we have in our company such server who never seen service. the reason is the proprietary system with a lack of security and documentation. so there are sitting around until someone has the patine to make this thing work now it should suppose to work in the existing infrastructure but in the end it is way cheaper buying a server that cost more and is set up in couple minutes because every one knows the system as buying a cheaper option. the problem is the shopping department they question you why you need a 2.000€ if you could buy a 1.700€ server that does the same. but they dont see the work that is needed to get this new unknown system work. Eventually they see the point because you spent that much time on it to get it work they had better authorized you to buy the expensive server as they cheaper one they come up with. but this is normal daily nonsense on that companies burning money
i commented on this the second it came out
You should feel very accomplished in life.
@@NicMG what?
@@Aeroguy_09 they said that you should feel very accomplished in life.
@@RoddyDev he's a guy called Nick, he's not a "they".
@RWL2012 "they" is perfectly valid as a pronoun to describe a singular person. It's something that I've started doing personally in a lot of professional communication where I do not necessarily know someone's gender identity.
UBUNTU Server latest version might be the way to go.
Can you please explain in detail what you mean?
@@be-kind00 UBUNTU Server Latest version is UBUNTU for servers is headless i.e no GUI but you can add whatever packages you need for your application.
Three pin fans ARE speed controlled
It entirely comes down to the motherboard, while some can control the speed of 3 pin fans by regulating the voltage, it's not universally supported (including with this motherboard here). You're always much better off using 4 pin fans for proper PWM speed control.
Those are OG speed controlled fans, pwm was invented so you can go slower since at about 6-7ish volts they stop spinning.
Every mb should have a dc/pwm setting since like 2005
P = Port sata Port.
now update your bios!
Raid 0 for the win
RAID without a controller, are you sure?
For the boot drives? Software RAID has been a perfectly viable option for years now, just look at the popularity of ZFS nowadays - it's essentially software RAID. Definitely not worth the cost and power consumption of a hardware RAID controller for something as simple as a RAID 1 mirror of two drives. The only time I really would use hardware RAID nowadays would be on VM hosts running a traditional hypervisor such as XCP-ng where the hypervisor doesn't officially support software RAID. However that trend is already starting to shift with hypervisors such as Proxmox supporting ZFS natively.
Anybody else have to watch this at 75% speed to be able to understand him? No? Just me?
I think you get used to it over time! Initially I was struggling with the accent too, but these days I actually like watching these videos on 125% speed because of their sheer length ...
I do try to slow down but it's just something I find surprisingly hard. It's almost as hard as I find it to produce videos that don't end up stupidly long! This one in particular ended up over 1h45m long before I went through and cut bits of it down and reordered some of it.
No problem here either. As per the first reply I also have these run at 125% speed to get through them quicker and gave no issue following along. Maybe it's what you are use to 😀
Thanks for the video, but I find you talk a bit too much and the video is a bit too long. But thanks for this setup
I agree this is a very long video, hence the provided chapter timestamps. I find it surprisingly difficult to make a concise video without scripting it, and I don't really have an interesting in scripting my content. I find it's also a tradeoff between making a short video, and providing all of the information people may need. If I miss out a piece of information, I can almost guarantee that I'll get comments with people irrationally angry as to why I did something a certain way. I'll often see such comments posted and then later get deleted, clearly after I have explained why I did something a certain way later in the video.
It was a good deal. BUT. When you talk about what it is "worth", you have to know that the acutal price that is payed by a company consists of likely 50% for guarantees and service for the usual 3-5 years of runtime. And thats what you as a dude buying on ebay do not get. So saying it is "worth" a grand means it is worth 500 without the guarantees.
Edit: I'm just saying that because i am often astonished when people put their gear on ebay for 80% of the new price and i'm like: yo wtf, half of the product is missing, the guarantee and service part.
Bro you've gone full geek mode. Somehow I've ended up watching this after a few vinos and compared to your earlier stuff it's like you've completed a maths and computer science degree since then.
Do normally enjoy your vids but this is brain damage for me 😅
intel is a thing from the past. please do a video on AMD.
I'm totally impartial between platforms and use whatever is best for a given situation (over the years I've used both Intel and AMD for various builds). With this build I strongly considered AMD however there was no AMD option that I could come up with that was comparable in terms of features and performance for anywhere close to the price of this. While I'd love to do videos on all sorts of different platforms, it's just not financially viable. With the exception of the case, everything in this build came out of my own pocket (well over £1000) and it's highly unlikely that the revenue from this video will come anywhere close to covering my costs. I can only produce videos like this because I wanted to build a new server anyway. That said, if you can suggest a similarly performing AMD platform where the CPU, Motherboard, RAM, 10GbE NIC and storage controller would cost under £450 then I'm all ears.
also, please slow down.
Trust me, I try, it's just something I find extremely difficult to do. The only other option would be for me to script everything however this would astronomically increase the time to produce each video to the point where it may not be viable.
@@camerongray1515 A quick tongue, a quick mind :--)
I'll go with that as the excuse! At least you don't see all the outtakes where my mouth has clearly outpaced my brain and absolute nonsense ends up coming out!
Speed seems absolutely fine to me! 👍
It pronounced WA-T W-ILL TH-EN HA-PEN not "uht El en appen" please.... explain yourself sir