How efficient can I build the 100% Arm NAS?
ฝัง
- เผยแพร่เมื่อ 31 พ.ค. 2024
- This build goes all-out for efficiency, with a 32-core Arm64 CPU on a brand new motherboard.
HUGE thanks to ASRock Rack, 45Homelab, Ampere, and Noctua for helping me build this system.
Parts list (some links are affiliate links):
- (Motherboard) ASRock Rack ALTRAD8UD-1L2T Motherboard specs: reurl.cc/67jk0V
- (Motherboard) ASRock Rack ALTRAD8UD-1L2T on NewEgg: reurl.cc/qrnXNp
- (Case) 45Homelab HL15 + backplane + PSU: store.45homelab.com/configure...
- (PSU) Corsair RM750e: amzn.to/3OyDQ79
- (RAM) 4x Samsung 16GB 1Rx4 ECC RDIMM M393A2K40DB3-CWE PC25600: amzn.to/49lCtkb
- (NVMe) Kioxia XG8 2TB NVMe SSD: amzn.to/3Uzag5d
- (CPU) Ampere Altra Q32-17: amperecomputing.com/briefs/am...
- (SSDs) 4x Samsung 8TB 870 QVO 2.5" SATA: amzn.to/3OylbZk
- (HDDs) 6x Seagate EXOS 20TB SATA HDD: amzn.to/3OA2CDM
- (HBA) Broadcom MegaRAID 9405W-16i: amzn.to/3srcZOh
- (Power) Comeap PCIe to 8-pin CPU adapter: amzn.to/3Sx52o4
- (Cooler) Noctua NH-D9 AMP-4926 4U: noctua.at/en/nh-d9-amp-4926-4u
- (Case Fans) 6x Coolerguys CG12025M12B2-3Y 120mmx25mm: amzn.to/42tTwhD
Other things mentioned in the video:
- 3D Printed 2.5 to 3.5" SSD to HDD adapter and other HL15 parts: www.printables.com/@45Drives_...
- M3 x 4mm pan-head screws: amzn.to/49wBe1K
- Techno Tim's HL15 review: • EVERYTHING You Should ...
- Raid Owl's HL15 review: • The best home server c...
- Pigeon vs Internet speed test: • A homing pigeon is fas...
- 45Homelab Forum: forum.45homelab.com
- ServeTheHome review of ASRock Rack motherboard: www.servethehome.com/asrock-r...
Support me on Patreon: / geerlingguy
Sponsor me on GitHub: github.com/sponsors/geerlingguy
Merch: redshirtjeff.com
2nd Channel: / geerlingengineering
Contents:
00:00 - A unique Arm64 HL15 build
01:12 - Parts & Specs
04:13 - Motherboard assembly
08:27 - Thermal paste application
08:44 - A prototype fan
10:20 - 45Homelab's HL15 chassis
11:44 - Will it fit?
14:17 - How hard is it to press power?
16:52 - It's not quiet
17:51 - Hard Drives
19:02 - SSDs and 3D printed adapters
19:40 - First boot! (And a RAM issue)
23:14 - Linux install
25:27 - Idle power draw (higher than I hoped)
26:10 - Rack it up!
29:17 - Patch cable's too short (oops) - วิทยาศาสตร์และเทคโนโลยี
So its not a cake?
The cake was a lie.
@@JeffGeerling about to say the same thing 🤣
@@JeffGeerlingThe cake was always a lie.
@@JeffGeerlingOkay, so the pivot to baking isn't happening yet. Got it.
no thats extra....
I like how you don’t pretend the video isn’t sponsored because no cash changed hands - integrity move!
It's best to be blatant about the vendor relationship; then you can decide how much you'd like to trust my conclusions!
i dont do things based on how much money involved
this guy tried to sell linus screwdrivers! maybe its because you dont know what integrity means!?
??? I paid for all the screwdrivers used in my LTT screwdriver review videos - and never recommended them over cheaper alternatives@@bent540
Or are you implying I tried to sell Linus some of my screwdrivers? That... is definitely not true. I like my screwdrivers, don't want to sell them to Linus.
Hey Jeff, It's Mark (met you at the creator summit) So short story on that power switch.. the batch of switches we had in production were not consistently terminated by our supplier. I ended up making a batch of power switch testers for our production guys to use once I found out about this. I'm pretty sure that your unit was one of the first off the line, so it looks like it got through before we knew about the supplier issue. Wouldn't be something we'd see on a full build, as they would have used the power switch mounted to the rear panel to test the system, but since your unit didn't include a mobo they would have used a drop in test bench motherboard. (which is a motherboard mounted to a sliding plate with its own power switch that we install in the back of the unit with enough IO options to accomodate all cabling possibilities offered).
They have to leave the rear panel off when testing like this, the same rear panel that houses your untested power switch. 😅
Good to know, and glad to meet you virtually again :)
Small problems like these are non-issues, as long as one is able to easily find a description of a working solution and can readily apply it oneself 🙂❤
Much worse is problems (in other computers or software) that takes days of tearing one's hair out, not being able to find anything conclusive to solve the matter.
As a homelabber, I would love to see more 3D printed items in a homelab environment. Lets face it, Homelabbing is about cramming enterprise tech into your house. Seems like 3D printing fits right in there.
For sure!
Having purchased several 3D printed little adapters over the years from eBay for various enterprise tech that I've used in creative ways, I concur. I just don't need another hobby...
This is my dream NAS. When this stuff starts showing up on eBay in a few years I'm going to be so happy.
yeah those 20TB hdd
@@jacquesb5248 Honestly from a practicality standpoint, 8 or 10 TB for a SATA drive is probably the max I'd go with (accounting for price + rebuild time for a full disk).
20TB+ is really only best for maximizing storage per lb/volume!
@@JeffGeerling at these sizes the risk of another drive to also break while rebuilding is just a bit too great
and while you have backups for that, it's really annoying if you need to actually go and start from there again...
Why is it your dream NAS?
@@JeffGeerling true, i have 4 6TB for 15,6TB and have 2TB left so bigger is going to work for me. data hoarder
Jeff violated the first rule of system building, never close the case for your first boot, it's bad juju. ;)
Actually another great video, love seeing unusual builds
BTW Rocky is my favorite distro, I have several machines running it. Highly recommend!
And then he locks it up again after swapping the RAM! No, no, no. Never seal until boot succeeds.
@@BrBill That's just how he rolls 😎
The older I get, the more a appreciate the value of a workshop trolley when it comes to moving awkward and / or heavy kit around.
Pulling the mag drives before you rack it also helps
On the switch, you usually have NO - Normally Open, NC - Normally Closed, and C is actually Common, ie a changeover switch.
That makes more sense since Normally Closed would be Closed, normally, heh
AsrockRack also makes similar motherboards but with an AM4 socket, which I'm using in my own setup. Most or all of the desktop-class, non-APU AM4 CPUs support unregistered ECC. Turn on Eco mode in the BIOS and you've got a powerful, quiet, and efficient homelab/virtualization/storage server using a CPU recycled from your last-gen gaming rig.
Running one of their Intel Core boards with a 12600K and ECC. It's a great system.
they have also AM5 boards
@grizant
Eco mode does fantastic work when it comes to load power consumption, but what about idle? Is it acceptable on ASRock Rack setup?
Ryzens idle quite high, they don't properly support C6.@@antagonista8122
I used to have an ARM NAS! It was called an Acorn Archimedes 😀 ran RISC OS in ROM and the network card had drivers in it’s ROM!
Blast from the past!
How many hard disk did it hosted and for which capacity ?
Never expected to get a refresher on what can go wrong with a pwrswtch today, but glad to see I am not the only person cursed with learning this the cursed way.
The worst power switch issue I've seen is the computer just randomly shutting down or rebooting. The switch was faulty and sometimes stayed half open and would randomly click based on wind currents or movement of the shelf
Hey Jeff, that 4 pin connector is for, standby, power on, power good and ground. It tells the power supply to turn on and makes sure everything is okay.
to reduce noise on my 15AV, installed noctua fans, and those spacers that offset the fans, and connected them to an aquacomputer fan controller
The Noctua NA-FC1 could also do the job or not? Should be a bit cheaper then the aquacomputer solution..
@@theboundless7102 the Octo is expensive for sure, its nice in that all 8 fans inside the case (3 front, 3 mid, one back, one downblowing on PCIe) all fit in a centralized location. The other advantage is that they can be tuned from the aquasuite software, without having to turn a dial like on the noctua. Have it sitting right next to me, the thermals are great and can't hear it.
@@Dr_b_ you don't need to turn the dial for the Noctua. The dial is used for two things mainly:
- give it control for when you don't have a PWM signal
- reducing an incoming PWM signal on dials amount.
You can leave the dial at 100% and have it pass through the PWM signal entirely or, if you prefer to, you could reduce the incoming PWM signal, too (which is for the times you cannot set a fan curve or it's still too loud, I suppose?).
At least it states it in the manual and I can confirm it as I have one running here (though, be it in manual mode, tried it on my main though with PWM which worked, too) and planning another one for my upcoming IT cabinet type of thingy.
There are many tech people out there, but I love your videos and your style. Honest, calm and professional. Thank you for the video!
I'd be interested to see what you can accomplish once everything is all set up and you start playing with C-states, etc. to get the power usage down. Being a mere pleb I have been using Chinese x99 motherboards and cheap Xeons for my recent builds and have managed to do pretty well all things considered as I have only, so far, stood up a "do everything" server that serves as my NAS and home services. A dedicated low power secondary backup target as well as off-site cold storage is in the plans. Just not sure which direction i'm going to go. The new Arm stuff looks like fun but just too spendy for my use case. So either going to go with another Xeon or maybe an N5100 series cpu.
The coffee mug on the case is triggering me!😅 My Dad had a Heath kit TV die on him when a vase of flowers on it spilled.
Heh. In this case it was an empty little prop. I had it sitting around because I planned on recording with my Dad on set for a Geerling Engineering video-figured it was a fun little thing to have sitting around in this video :)
Likewise I killed the family TV after sitting a glass of milk on top and spilling it down the back. This was the 70s and my family had saved up for a long time just to buy this TV, and I killed it. We didn't have a working TV again for years. Years. TVs cost a fortune in those days.
Definitely curious about your 3D printing adventures!
Great experiment Jeff! So much fun geeking out with you. And here's a huge bonus - your video dropped 2 mins after I finished watching todays' ExplainingComputers video. Happy Sunday 😁
It's a great pairing!
@@JeffGeerling Like Wine & Cheese!!! 😜😆
Very cool project. Awesome to see this system up and running.
Something to note with Asrock IPMI, at least on the board I have, is that you have to manually configure which temperature sensor(s) correspond to which fan(s), as well as setting the fan curve.
You also have to make a mapping from temperature to fan speed, or I think you can set it in a mode where it's like a feedback control that steps the fan speed up or down.
love the full setup video, its very informational and shows the whole process of building and racking a server.
Awesome as always! Looking forward to the performance and testing!
27:24 I felt this moment in my entire being, and the "sometimes that happens" was the perfect response.
takes 30 minutes to build. takes 45 to remove from packing lol
one of the cooler HL15 builds I have seen, very nice keep it up!
RAID is not a backup, but those ZFS snapshots are super handy for day to day restores.
True!
Also superhandy for the day to day revert you latest system software upgrade if it all fails.
I recently discovered that I have been doing daily snapshots of the volume containing my virtual machine's hard drives, for 3 years, and forgot about it.
I didn't notice it until I went to migrate the volume. It didn't impact performance.
@@edwardallenthree When you forget you took a snapshot of a VM years ago in VMWare and have been running on it since. >.
FREAKING AWESOME video Jeff !! Nice work !!
I have to present a project at the end of my apprenticeship. An ARM based NAS is exactly what I was looking for! Great idea! Thanks!👍
I'm so excited with this setup! The presentation is top-class as always :)
Cant believe we're in an era where arm cpus are socket able
This is the first video of yours I have watch with you in the new space. Studio looks good and functional! Looks like the hard work is paying off.
Thanks! And it is, I have been able to get a lot more done, and finally seeing the effects of it in being able to do more than one project at a time now. Plus my wife is very happy having a sewing room at home now haha
Nice to see you have all the same issues any of could have! Well, except for dropping screws into the gubbins - maybe I'm clumsy
I have several Asrock Rack boards. All have been very solid and super stable. They are not cheap but if 24/7 is what you need.
awesome reviews! Love your ARM64 content - especially now when Adlink and Asrock delivers products that are useful!
Took the courage to actually invest in a 128core adlink after your latest video!
Can inform you that my 128Gig ECC LR DIMMs didnt work, sent them back and populated it with 64gig ECC R DIMMs instead and they work like a charm.
Looking forward for your next Ampere video!
Good job on the Video, Your edits are getting so very good! I can't wait to see how Arm Cpu's are going to change as time goes on. It'll be cool to see laptops using them as fully as they do with Intel and AMD cpu's.
Although I am not an Apple fan, their M series shows that RISC / ARM with emulation can perform as well as x86-64 for modern laptops & desktops. I wish there were more powerful & efficient options in the PC hardware ecosystem.
I didn’t know you had a mug. Gotta get that. Good product placement ❤
Really cool! I can't wait to see more ARM servers, especially ones that could be used in the homelab!
Very interesting project Jeff. And important too, efficiency is both mathematically beautiful and a necessity for the progress and survival of humanity.
9:20 The screw tightening order should be in a "star pattern", not just "round the clock" (or in this case "against the clock"). It stabilizes the pressure load and position of the cooler, making it a downward movement instead of a skewed movement. I learned that from a mechanic, for mounting engine tops.
Bonus points if:
* Make three "rounds":
1. make the screws just get a grip
2. screw them so they touch the bottom position
3. tighten
Nice... Clear and useful as always! Tks for sharing this.
This was a epic episode! Always enjoye someone building a server
Nice video Jeff, thanks for sharing this bleeding-edge stuff. I just did a NAS build using Epyc Milan using an AMD Epyc 7313P + ASRock Rack RomeD8-2T + 8x32GB DDR4 3200 RDIMMs. Drive config is 16 x Micron 1100 2TB SATA SSDs arranged in two RAIDZ1 VDEVs. I'm seeing nearly identical idle power draw as what you are which is surprising given ARM VS x64, with it going up to just under 200W with multiple VMs running in addition to the TrueNAS Scale. Love that 45HomeLab chassis, but too rich for my blood.
I was wondering how it would compare with a similarly specced Epyc setup. So about the same? I have an ASRock Rack EPYC3451D4I2-2T running TrueNAS Scale, will have to get some power measurements on it.
@@justinclacherty9181To be fair my comparison isn't 100% apples to apples to Jeff's here because I'm using only SATA SSDs and he has spinning rust. But yeah I don't think the ARM power efficiency claims are enough here to justify the boutique nature of an ARM Server for a HomeLab. I'll be able to source at least functional (if not exact) replacement parts for my build for years - I'm highly suspect that will be true for the ARM Server.
Looks fun! Although Im watching every video without having any experience, so you are learning me thanks!
Definitely would love to see some 3D printing shenanigans on the channel! It's always awesome to see the unique ways that people utilize additive manufacturing in their professions and hobbies.
Awesome video sound was top notch as well
OMG I was right when I said Ampere ARM on your image when you asked what we think you'd be doing with this chassis. Wow.
Great video, I really want to build and use such a server, but when I saw 158W in idle, I decided I don't want to (can't afford it at the moment anyway), because electricity here is 40¢ per kWh I believe.
It should be lower when the hard drives spin down.
CPU should be 6 to 12 W at idle.
That is a big honkin home lab NAS, wow.
Thanks for sharing this build with us.
So cool, looking forward for part 2 :)
So happy you didn't go with an off the shelf nas software. Truenas is cool but I love the idea of just using Linux
IIRC the 2 in 3-2-1 backup strategy means two different types of media. So I'm hoping for a video on a tape robot in the near future. :D
Does Amazon's robotic tape library count? My 3rd backup is in Amazon Glacier Deep Archive :)
Nice video. Can’t wait for the next one!
Nice video. Will have a look and see what these run for in uk
great build.. can wait to see the set up!!! Also excited for the new Dr. Who season?
Awesome build Jeff. Not gonna lie, very jealous of that rig. 🤤
Your new studio looks so fancy
150w with 32 cores and so many pcie lanes is awesome for virtualization, excited to see where ARM goes in the next few years.
You can already do that now with EPYC.
Perfect for me! Hater of arm since the early beginning into larger things
Very cool use of ARM
"Hello there!" 😂 Always looking forward your Linux/Servers videos. Keep em up!
Would love more of your take on 3d printing.
Hiya Jeff! Feel free to expand on 3D printing any time you like! I've always appreciated how the 3D print community has rallied around the SBC industry. When I started printing, my first REAL project after cutting my teeth on test models was to print a RazPi3 case that would mount to my printer and run OctoPi. Of course I could have bought something, but being able to conjure a practical and functional piece of computing hardware out of thin air was one of the primary things I enjoyed about 3D printing. I'm really interested to see what you use to print. Personally, I have a a Monoprice, a couple of Creality's Ender3-family printers, a Prusa MK3s+, and a Creality K1 Max. And also, basically an entire wall rack of every interesting type and color of filament I could get my hands on (I got addicted to buying cool plastics for a bit). I am just getting back into adult-level RC cars, so it'll be really fun to start designing some of my own parts. Eventually I'd like to 3D print buck-molds of popular cars as well as my own designs and vacuform shells that will fit the chassis that I'll be racing around! I can't wait!!!
This is amazing content! Arm is the future.
Love your rocky shirt as well Jeff :)
Nice build. I would love to have one of those 45 drives cases. My major problem is having short rack space as I am using a network rack in the top of my closet, so I am currently cutting a few inches off the front of my case. 😀
Cool build. I look forward to hearing how it performed over time. If you used encryption, CPU loads and such
Great video Jeff. Love sth. I want to build a nas w/ solidigm P5336 61T drives when I win the lotto.
That would be... yeah I think I'd have to sell my house haha
Hey Jeff, good work as always. You look healthy, hope you are well!
I can't wait for part 2!
I recently built out my new storage pool in a HL15... Crammed 12 15Tb Mach.2 drives in there along with some nvme drives. Luckily I didn't have the power switch issue you did but for some reason the power light on the switch doesn't want to work... Given that it's on the back I just stopped caring though... hehe.
Your case seems louder than mine was. Not sure if that's the mic picking up a bigger frequency range or something else. Overall it looks like you're putting together a solid build... I'll be interested to see the benchmarking for storage and network throughput when you're fully setup....
You’re not made of money, but these parts aren’t cheap! Jeff always gets to play with the cool toys. Thanks for sharing the setup! Hope you have a happy and healthy 2024
I really like my mini-ITX nas I mostly put together from stuff I have lying around and some drives, but this kind of thing is what I actually want to play around with.
Excellent video!😊
I got a few of those Logitech K400 keyboards too. Truly a great keyboard for bench testing! Definitely a favourite on my benches.
I love the way the first rail was waiting for Jeff to say "it doesn't come out" before it decided to come out. Typical.
The adaptor cable is for Power Good (Gray), 5V Standby (Purple), PS-On (Green) and Ground (Black). It needs this to provide standby power to the board and to signal the PSU to turn on. I had to mod the cables that came with my Supermicro M11-SDV and figure these out.
Great information, thanks 👍
Love your videos! Hehe, that's totally me - always using screwdrivers as knives.
Huuu, 150+ Watts in idle... for Test, or if you have paying customers, yes... but probably not for fun!
Oh sweet they used the Super micro rails, nice choice to not have to design a whole new thing. Looking forward to seeing more ARM server stuff coming.
For switches, NC = normally closed, NO = normally open, C = common. It has to do with the default state of the circuit when wired: NO means an open circuit which the switch closes, NC means a closed circuit which the switch opens. C is the common which completes the circuit either way. Since power switches in computers only need to briefly toggle, it should always be wired NO so when the switch is pressed it closes the loop. It knows the circuit change due to a sensing voltage, usually around 3-5 volts of direct current (VDC) but quite low amperage.
thanks very good video looking forward to chapter 2
Hey Jeff, any idea when you will be releasing the next part to this video? Keep up the amazing work!
On the power switch:
NO is Normally Open.
NC is Normally Closed.
And "C" is Common.
There is continuity between NC and C when the switch is not pressed, between NO and C when the switch is pressed.
It's pretty common that these switches get wired incorrectly.
What a lovely chassis, it looks like a music computer!
I like that you are using ARM for your NAS. I just recently got acquainted with Rocky Linux for a test I am doing at work. I am glad it is available so that non-enterprise users can use it as a direct replacement for RHEL and CentOS. I look forward to see how you complete the setup. Just an FYI, I run TrueNAS Core on my Dell R720 with 4x Seagate Exos 18TB and 4x Seagate SAS 3TB drive with a dual 10G SFP+ NIC and the built-in quad NIC and it only draws 140w on average. I an running dual xeon 8c/16t procs and 128GB RAM. I thought newer hardware and ARM would be less power draw?
Great video!
I just love your sense of humor Jeff :)
I love this board and chip. Keeping an eye out for when they start to hit the secondary market. Also I was hoping that stupid orange overprice bit driver would have gone away by now.
Wow looks like it will work good
I was just working on a PC without an antistatic strap and simultaneously watching Jeff working on a PC without an antistatic strap. I thought "that's going to catch up to us!" Looks like it might have for Jeff but I got lucky! 😂
i always love your content
Nice build. I hope one day a standard mini-itx or m-atx MOBO Arm64 based CPU
Great video. I just picked up a used, massive, old server with 14 drive bays, 2 power supplies, and lots of space for tinkering. Unfortunately it is too deep for my rack, and I am scared of the power draw. I'll have to hook up my kilowatt and see how it compares to your new setup, Jeff. :)
Be prepared for anything! I've seen a few old systems power and sucking down like 300-600W :O
A lot of them were also never tuned for power saving at all-and some platforms don't have much you can tweak either :/
Honestly, having used both the 45Drives chassis, as well as the Rosewill. The 45Drive chassis is definitely worth the price tag. I have a bunch of the Rosewill cases in my lab. They are extremely flexible, and well thought out pieces of hardware for the price point. So if you just don't have the money for a 45Drives system, they're a great second choice. Rosewill has a set of components that allows you to easily change out the front of your chassis.
Very cool to see this type of product, very curious how it performs compared to epyc server. Want to see how it compares in power usage for an all core work load too
And here I was trying to figure out how to get my intel setup under 17 W idle.
Honestly for a typical *home* NAS, this machine would be waaaay overkill :D
Also, one thing I've found with Ampere's chips is they are most efficient for medium to mostly-utilized workloads. If you run it idle most of the time, you can probably get a lower power budget with other CPUs.
whats 17 vs 35 watts? honestly you waste and throw away more food than what you'll ever save in power costs.
Correct me if I missed something, but the idle was approx 150 W (let's say 100 without the drives). That's still more than a 200$ difference in electricity costs where I live.
But as Jeff said, this is a server for entirely different use case than a small home NAS.
And sorry if the comment came out wrong. It was more about me being silly about a few watts than about comparing server grade hw to small home nas with a plex server.
Yes i meant 17 watts vs. 35 watts of another system, not jeffs. Of course going from 350 to 17w idle is an entirely different story.
@@JeffGeerlingthat's something very few understand. just because it's arm, doesn't mean it will draw less power than x86 in all cases.
I'm crediting this video for my massive ARM stock gains yesterday.
Would love to see more 3d printing content!
Looks nice. Finally. Manual and features look really good. It also looks better than the WS system we saw, that had gimped memory channels, and price is right (the 10G version is little overpriced, but still good). Also it has enough power and VRM to support all CPU models I think. The boards are coming in. There is one review on a newegg.
For the file server, virtualization host, and even a router, this is fantastic.
I hope they also do work on E-ATX version, with maybe a bit more space between PCIe slots, for a GPU or two. Then I would totally use it as my desktop. But even then, I think it is just about right, I need 1 GPU, and 1 NIC (25/100Gbps). Which with big GPU, will basically use all slots. But then storage can be done using OCulink so that should be fine too.
Those look like Supermicro rails! FWIW, my 15 drive (rust) old E5 32t Proxmox system runs about 155-170w.
They are!
22:41: "Hello there!"... sold!
Finally......!!!! My dream project...!!!! A DIY ARM NAS....!!!!🤩🥳
great video.