My biggest server build yet! 88TB TrueNAS CORE Build.
ฝัง
- เผยแพร่เมื่อ 18 ก.ค. 2020
- What do you do when your primary NAS is 85% full? You build a new server of course! Just in time for the release of FreeNAS / TrueNAS Core 12.0, I've got an all new NAS and 88TB of disks ready for some sweet ZFS goodness.
But first... What am I drinking??
This is a pretty rare beer, as it's normally only available on draft, and not widely distributed. It's the Prince Charles Wee Heavy Scotch Ale from Santiam Brewing (Salem, OR). Clocking in at 9.2% and 25 IBU, this is the PERFECT beer to just sit down, read your favorite book, and enjoy for two hours. It's sweet, but not syrupy. Almost reminds me of a really good brandy in how you want to drink it. HIGHLY recommended.
Links to items below may be affiliate links for which I may be compensated
Check out the parts from today's build on Amazon, eBay, or AliExpress!
Gigabyte GA-7PESH2 Motherboard (Not available, but look at these alternatives): amzn.to/30reKNx amzn.to/32Cje6K ali.ski/t-7P7r
Intel Xeon E5-2650 (8C/16T): ebay.to/3jkrVc0
Intel Xeon E5-2643 (4C/8T): ebay.to/3fPPOpK
Samsung DDR3 ECC Memory: amzn.to/3fEIFbw ebay.to/2Clsu4G ali.ski/HkZAs
HGST He8 8TB SAS Drive: amzn.to/2CSpkF7
Seagate 3TB Constellation ES.2: ebay.to/3hjkwrE
SuperMicro 846 4U 24-Bay Chassis: ebay.to/32ycrer
Sedna 2x 2.5" SATA PCIe Controller: amzn.to/3hfkyAz
Silicon Power 256GB NVMe PCIe 3x4: amzn.to/3fLt0qP
Find the parts I recommend on my Amazon store: www.amazon.com/shop/craftcomp...
Follow me on Twitter @CraftComputing
Support me on Patreon and get access to my exclusive Discord server. Chat with myself and the other hosts on Talking Heads all week long. / craftcomputing
Music:
Bass Vibes by Kevin MacLeod
Link: incompetech.filmmusic.io/song...
Opportunity Walks by Kevin MacLeod
Link: incompetech.filmmusic.io/song...
License: creativecommons.org/licenses/b... - วิทยาศาสตร์และเทคโนโลยี
aw, this is cute.
Awww.... mine still works.
@@CraftComputing touche!
lol
Craft Computing yeah because it’s new
@@jj-icejoe6642 New how? The board is used and the last generation...
@@CraftComputing OUCH!!!! LMAO!!!! That was an awesome BURN right there!!!
Fantastic #TrueNASbuild! As always, this is awesome..!
Great video, Jeff!
I love how Jeff's "Needs" and "Use" are incredibly aligned with my own. "I'm the only one with a 10 GB connection anyway". LOL YUP!
Same almost exact thing happened to me with this case, learned a lot about the backplane and the non existent buzzer, then found out it was an issue with the dual PSU, glad you kept this in the video.
"Hardware documentation: None of the information that you need, all of the information that you don't!"
I knew that beep as soon as you turned it on. I've heard it too many times to forget. XD
Welcome to the disk array shelf club!
Happy to be here. Which way to the bar?
@@CraftComputing 🍻
@@CraftComputing Clank 'em and drank 'em!
Havent watched for a little while, but I'm back, so thanks linus for reminding me about this cracking channel. Love your work
I learned a lot about computer hardware types that I didn't know was out there on this build. Great Video Thanks!
I think i will build a Nas eventually thanks for the Tips Jeff keep up the goodwork mate!
Oh man, sweet build. That SSD to PCIE card is something ive never seen before
Thank you Jeff for the battle against the beep, I enjoy this type of content as a budding data hoarder. I'm definitely interested in the migration of data off your current FreeNAS box. My current CPU is not powerful enough to stream plex remotely so I'm thinking about an upgrade that will let me share the horde.
I knew what your problem was the instant I heard the beep and it was confirmed when I heard you say it posted thanks to Jason with Byte My Biits trouble with either Zeus or Loki, can't remember which one it was. Great video, thanks Jeff.
As soon as you powered that beast on and I heard the long consistent beep I shouted out "Power Supply!". 10 years building these things gives you that experience, I guess.
plus usually there's no green light on the back of the PSU
I have the exact same penguin compute chasis! Have enjoyed it for the last 1-2 years and done a lot with it to quiten it. I love the PCIe to 2x SATA card, will have to get that. The beeping is a nuisance, found that if it’s populated with both PSUs and only one is plugged in it beeps regardless. None of my PSUs were dead, it was just pissed that both weren’t plugged in haha. Enjoying the content, keep it up!
Loved your video. As soon as I heard that beep I knew it was you power supply. I have a supermicro as well and had the same problem.
Awesome timing! I'm buying parts now for a 70TB TrueNAS build off of a NUC ;)
That 2.5" SSD to SATA thing looks awesome for storage. I might just have to grab one in the future
Nope...... Get a NVME !!!!
@@paulchambers8033 Not every motherboard/BIOS can boot from an NVMe and booting some OS's from, would be a total waste. Like unRAID, ESXi, and perhaps others. Also you may be at your upper limit for PCIe lanes. Many reasons, but this thing is just cool. (And yeah, I have 4 NVMe's in an Asus holder, but sometimes SATA is a good thing too)
@@bobcarpenter1551 Agreed !!! I think the card is amazing !!! But £50 !!!!!! For that price I would much rather Get an NVME and Sata PCIE card ( I have one )... boot from a 120Gb Sata And use the NVME...... Personally I've wanted that card for years now..... Just annoying it becomes available after a newer technology becomes available... I only pay £5 more for a 1tb nvme drive instead of a Sata........... We are also at the start of a weird technological time....When the latest games console blows past £1500 PC's something's wrong.... DDR4 is now cheaper than DDR3, Those two xeon processors he used can be outpaced by a single Ryzen ! Now is not the time to be spending anything on any system!!! Changes are coming !
@@paulchambers8033 Sadly, I went down a rabbit hole, I have discovered that they also have quad ones too, with and without RAID controllers! I completely agree, the Threadripper Pro's are exciting too. 128 PCIe lanes!
Those hard drives are not SATA drives they are scuzzy drive that don't have SATA ports of them . Yeah that you have to have scuzzy controller to use them
Great content as usual.
As soon as I heard the beep I started to chuckle, then that face you made was icing. 😂. We've all been there.
Even more fun when you enter a large server room, and somewhere in the noise you hear a faint constant beep. And then the search begins... where does that beep come from? XD
@@GerbenWijnja Most enterprise organizations would have a server/network monitoring setup to alert them when a server is having a problem or is down. Essential in a large Datacenter environment. Label all equipment and you dispatch the tech to exactly the right server. No guesswork should be involved. Too noisy to just rely on trying to hear which server is screaming for attention.
Geoff, if I ever make it to the US mainland, I'm visiting!! Geek speak with the TrueNAS & Proxmox box talk, plus beer, awesome beer!!
Oh my, the music reminds me of 1978 at a hotel piano bar ... it does bring back interesting memories ...
I put a 4U 24 bay SuperMicro chasis in my server rack too but I couldn't stand the noise so I pulled the P/S and its cage and put in a 700 watt SFX. I did the same with a 3U case too, in fact all my SuperMicro cases I've similarly modified. Thanks for the tip on the SAS drives! I've got that gigabyte board too and given the price on Amazon I'll pull my ATA controller and use the onboard SAS with those drives.
I love PCIe cards, especially stuff like that one that mounts two 2.5" drives. There's a quad one available too.
I did not know this existed! This is super nice!
These cards are especially popular for use in older Mac Pros (2006-2012) as they have no native way to mount 2.5" drives and this way you can have 6 drives instead of 4.
Icy Dock makes some pretty dense devices. They have an 8 bay SATA enclosure that fits in a standard optical bay.
For a moment I had to think what a Scotch ale actually was, even though I'm Scottish & live in Scotland..... here we just call it heavy :)
Beast mode! And here I am just trying to setup a basic FreeNas or possibly going with Synology.
Good job
@Craft Computing.... Is there a second, final to this build video? I'm not finding it..... Your explanations are the most clear and concise I have come across! Thanks!
Out fricken standing on the NAS ! I wonder how many thousands less that you spent on this project vs what Jason spent on Loki .... LOL Love watching Jason as well.
I recognized that beep immediately. My issue was only having one power cord. Had to remove the second PSU until I found another cord. It took me longer than I care to admit to figure it out.
I also recognized it right away from starting to rehab a supermicro server recently. I have both PSUs and they both work but I was only using one. The annoying part was fixing that only to be confronted by another beep issue that was hard to distinguish from the fan whine without hearing protection.
Love this kinda content :D
I have been trying to find a PCI(e) to 2.5" SATA adapter with integrated power for months to add extra SSDs to the ole r710. I almost fell off my chair when i saw you whip one out. Good lad.
They make a 4x drive adapter too!: amzn.to/3hhHRtu
And one with RAID onboard: amzn.to/32BMc70
Why not just get a PCIE SSD .. I don't get it?
Yes! New server builds!
Nice Geoff, like it
Nice server!
fwiw, the "Art of Server" channel is pretty expert at flashing these controllers.
When you’re ready for your next NAS upgrade, I’ve got a pair of Sun X4400 series servers that could use a good home. 48x 3.5” drive bays in a 4U enclosure.
Them supermico full system storage pods are actually very nice (loads of used and new ones on ebay, some complete with 72 to 96TB of disks already installed for under £3700 or complete system no hdds for £1000 with motherboard, cpu, ram and raid controller)
Never seen these systems before
I got to say your B Role music taste is like a fine wine. :)
You should look into the special vdev. It really speeds up ZFS much more than cache in my experience. You would need to mirror it though so that would be another NVME drive to fit in.
Very nice. It is quite nice to see that another lab enthusiast think alike. I pretty much have comparable setup (dual 2643 v1 for computing, sc847+ dual socket it mode e5-2620 for freenas).
Did you manage to get near 10gbit/sec throughput?
I tried using FreeNAS for a year, then I switched to Unraid and it was the best decision I ever made. I've lost data on multiple occasions while using FreeNAS in that one year of use. After 4 years of continuous use with Unraid I haven't had a single issue to this day. To each his own, but after my experience with FreeNAS I wouldn't trust it as far as I can throw an 88TB server. I know the ZFS format is suppose to be more reliable when it comes to things like sudden power outages without a battery backup and XFS gives you better I/O R/W rates. However, every loss of data I had with ZFS FreeNAS software was due to a sudden loss of power. I've lost power on multiple occasions using Unraid's XFS format and not a single thing has been lost. I also found that adding in new HDD's to expand my drive space was way simpler in Unraid over FreeNAS. Sorry for the long drawn out diatribe, but I'm really curious as to why you would use FreeNAS over Unraid?
Unraid is far more better
Hey I am in Salem, Or as well. Hi, neighbor!
Lol the best bit was you lifting the case... And it was empty!
Nice kit there.
Like the SATA PCI.
It's 70lb unloaded. Even more now that it's full of drives.
@@CraftComputing they are beasts and then you loaded it. I noticed that it didn't get moved far.
You should have someone else helping you move it as you'll "do a injury".
Ive some around the 90lbs mark.
Are you going to try out SAN?.
The difference is that you don't have to make server's and it costs less*
I decided to experiment and go with Unraid with ZFS installed rather than freenas. Mostly due to GPU passthrough and the simple drop-in docker support. I have a similar build but I'm running dual xeon e5 2678 V3 from China at 100 USD a pop along with 128GB RAM & 8*10TB Toshiba enterprise grade HDDs. Also threw in an Asus quad nvme card with 4x1TB NVME SSDs - still have absolutely no clue what ill do with these SSDs apart from using them as temporary scratchdisks for various future projects...
I love you server tank yoy sooooo much
Jeff, I've read and watched everything you can imagine about deploying a TruNAS with 24 bays chassis, and your videos are the best so far, fantastic! I'll deploy a TrueNAS 24 bays in an 846 chassis, but I really need 10GB NIC cards and SAS drives. I've planned a lab test with a near customer. They have files in use for a design app running in 20 client workstations, and I want to test a better performance using your video tutorial. What do you suggest in terms of motherboard and processor Xeon E5 for 96TB SAS? I don't want to deploy an expensive solution, but I need to build a 2nd Supermicro 846 to back up those files. They should have the same processor/memory and capabilities? Thank you, and congrats on your videos!!
Hey @Craft Computing ! What was that clear plastic cover you had on top of the motherboard? I'm currently moving my NAS to a new 2U rackmount chassis, and one thing I'd like to do is keep it a bit cleaner than my previous build, and something like that would come in incredibly handy, provided of course I can find one suitable for my new case. Thanks!
I'm at about 80% on my current NAS as well. trying to decide if I want to bump up from 4tb drives to 8tb drives or jump all the way to 10tb. I think what i'm most interested in is that little NVMe to PCIe adapter. That would certainly solve some inconveniences for my setup. I'm currently testing moving from basically just a hardware raid-6 and running my server OS on bare metal to running Proxmox on the box and running the server OS, as well as a few other things in VM.
Great quality control… shipping a new product with broken redundant psu is actually a very good idea :)
This chassis was 8 years old on eBay....
@@CraftComputing you didn’t mention that…. I thought it was new a bad in the same time
This channel is causing me to drink more.
Comment to show support.
Amazing video dude! what about using Ryzen for truenas? (I was watching the lASRock Rack X470D4U) or is not recommendable due any driver or compatibility issue?
Couple things,
Did you point five added card is extremely handy as I bought one a couple years ago. Mine however caused CRC issues. So just watch out for that.
And to my knowledge, that supermicro case is SAS 2 or 6GB/s. Are the discs as well, did you upgrade the backplane, for that speed good enough for you?
You need powerd for cpu boost to work in freenas/truenas. also enable c-states in bios!
Slowly building. A nas system. And like learning from your videos
Hi Really enjoyed watching this build, I have a similar chassis CSE-846E16-R1200B, The fans are a little noisy, do you have any suggestions for quieter fans that can be used?
that boot up would be satisfying to watch when there were no beeps
those fans, well sounds satisfying
This video is playing while I building my own and there is a beer on my desk
hi, those 2U coolers whit those fans on, what type are they and do they make loads of noice?
they are not listed in the "server specs" you have on amazon/ebay.
i am after some coolers and im not shure if i should go with normal pc coolers or server graded ones.
Nice!
What's the power consumption at idle for that motherboard/chassis combo? And what about with your 8TB disks?
Did you ever get rid of that Sedna card? I believe its a 1x Gen2 interface which is nowhere near fast enough to support two SATA SSDs.
I bought one a while ago for my NAS but it just wasn't good enough.
You mentioned sending up data to the cloud. What type of cloud solution are you using?
I didn't hear this mentioned, but those PSUs will alarm if seated in the PSU back plane without power. Should also have a RED LED lit if failed or power loss. There are also a "SQ" (super quite?) version of the PSU which are considerably quiter.
God, I want beer
I am not regretting my 36 bay Super Micro FreeNAS build. If I get the point where I need more drive slots I will probably seek professional mental help. Even though, adding one or more of the Super Micro 45 drive only cases is now super simple and all the cases would match. Boom.
Have you had any drive failures from your HGST and Seagate drives?
What you plan to use those and servers. The storage are way too much for consumer level using
I'm actually looking at rebuilding my current server. Are there any SATA drives you recommend?
You are killing me with the beer description. There is no way I would be able to get that beer in New England.
Salem Oregan, good beer+Jazz= Witches Brew...Miles Davis
Your parts list is missing those network cards. Can you recommend models that are reliable and cheaply available?
How is the cache drive's endurance? I've heard it's rough on nand based drives. That's how some websites like camelcamelcamel got offline for days.
I know you know this but have you thought about swapping out the fans for 120mm in the front and 80mm for the 2 in the back. It made my supermicro so much quicker. Just food for thought.
Hi, that motherboard (and the alternatives) are impossible to find. Can you list available boards that would work with your parts list? Also, I did see on Ebay some complete Supermicro 846 systems including original motherboard and cpu's. Can you flash that board's card to IT mode?
Great troubleshooting when it wasn't obvious. Except for the enterprise admins in the comments who immediately recognised the steady beep. Still prefer ESXi hosting a virtual FreeNAS with the SAS controller directly passed-through - I know you could have done that too. Not ready to jump FreeNAS trains to TrueNAS Core just yet, will wait to hear if there are any horror stories after upgrading. Interesting mounting of 2.5" SSDs directly on a card. The price differential with NVMe is so small now that there doesn't seem to be any point in purchasing new SATA SSDs (2.5" or SATA M.2)
I'm a former enterprise admin, and I know a steady beep is a component failure. Checking documentation, the only buzzer listed in the chassis is for the backplane.
And both power supplies were plugged in ;-)
Did you ever post a followup about the RAID controller issue?
Nice! I have 8 of those 3TB disks myself. I wanted to add more and have some spares, they are dirt cheap, but nobody is shipping to Argentina because of the human malware. Are you planning on adding more SSDs to try Fusion Pools or Metadata on Flash?
I am planning on testing out mixed VDEVs with SSD and HDDs. Stay tuned.
TrueNAS Will Core It Build Work
Don't dead open inside
@@CraftComputing this some sort of inside language or something? please explain
@@jrok96 See for thumbnail enlightenment!
How tf are you verified with 5 subscribers...
The rules used to be different. There was a time when changing your account to your real name got you verified.
Did you end up flashing your onboard SAS controller or installing a seperate card?
Just curious - what are you into that you need this kind of power at your house? I mean if you're Linus sure, or maybe running a torrent hosting site lol.
Man, that thing powers up like a Ferrari before the beginning of an endurance race :O
Only this one won't catch fire!
nice build. I also just finish my TrueNAS VM build still using 1G hard drive. Uhh :-)
TrueNAS will core it build work
I love to have me a truenas supermicro 24 bay, but they are expensive and on top of that I can´t build anything that complicated electronics. However I would like one, but with support for consumer sata3 drives
Did you try to connect both to power at the same time?
Jeff where can I find the yellow drive sleds?
The 10gbe HP nic is missing in the parts listing. Which model is it?
Why did the cooling fans on the cpu heatsinks get removed?
so, im new to all this, and was reading the suggested hardward on the TrueNAS forums. I liked the idea of dual CPUs... but the docs on the forums suggested there wont be any benefit. So.... i thought Id ask someone who has done it. What is the benefit of DUAL cpus?
nice! How much did you spent in total on this build?
I was wondering why I hadn't watched this video and promptly realized that I put it on and fell asleep to it.
I fell asleep so quickly and hard that obnoxious beeping didn't even remotely wake me.
So I'm little by mentally building my dream rackmount NAS. Its a supermicro sc847 one of the version,but I cant find anywhere if its just to mount a supermicro x11 or h11 motherboard in there. I also am looking to buy a 36 bay server, or you reckon that 24 bay is enough. I was kinda thinking of 24 bay of HDD's and 10-12 bay SSD raid array with a dual 10Gbit connection. I also want to use enterprise drives,but I also want like 14 TB + drives so I'm still undecided after the insane problems with SMR and CMR drives that both Seagate and WD is lying about.
have a beer mate!
Don't mind if I do
Does this supermicro server not have IPMI ? That should have shown the PSU failure pretty quickly
What do you use for your cloud storage?
I did a similar build myself last year.
CSE-847 (36 bay variant like yours)
Dual E5-2620's (waiting for 2643 v2's to come down in price)
192GB ECC Memory
36X 3TB HGST SAS Drives
9211-8I's (one for front 24, one for the back 12)
Chelsio 420 10GB adapter
One pool with mirror'd devs (like RAID10) for my virtualization.
Cheers!
Also forgot to add: 2X 512GB NVMe for L2ARC and 1X 256GB NVMe for ZIL.
Great looking build! Out of curiosity, why use a second NIC vs the on-board 10GbE of the 7PESH2? SFP+ vs copper preference?
My switch has SFP+, and this + a DAC is cheaper than an SFP+ RJ45 module for the switch.
Craft Computing Makes sense! (And thanks for the reply! The first I’ve ever received from a content creator!)
sas backplanes work with sas and sata drives right? sata backplanes only with sata not with sas?
That is correct.