Server Rack Tour 2021 - Epyc Rome and Ice Lake, OH MY!
ฝัง
- เผยแพร่เมื่อ 3 พ.ย. 2021
- Check out the Viper V765 Mechanical Gaming Keyboard here, and thanks to Viper Gaming for sponsoring today's video! amzn.to/3mH1CAc
What is all the equipment in my server rack, and what do I use it all for? It's time for 'What's in my Server Rack - 2021 Edition'!
But first... What am I drinking???
A seasonal favorite from Rogue Ales and Spirits (Newport, OR), it's the Dead & Dead. Rogue takes their classic Dead Guy Ale, and ages it with Dead Guy Whiskey Barrel Chips. A delicious way to celebrate Halloween.
Links to items below may be affiliate links for which I may be compensated
- Network -
UniFi UDM-Pro: amzn.to/2RV7gPg
UniFi G3 FLEX Camera: amzn.to/2EsGFpr
UniFi G4 PRO Camera: amzn.to/32W9RhU
UniFi USW-24-POE Gen2: amzn.to/3iZcCEK
Mikrotik CRS317-1G-16S+ on Amazon: amzn.to/3p5k5Gp
Mikrotik Desktop 5-port 10Gb Switch: amzn.to/2TvVezP
Mikrotik CRS328-24P-4S+ 24-Port PoE + 10Gb Switch: amzn.to/3fHpGjf
Ubiquiti UniFi UDM-Pro Router/Firewall: amzn.to/3fA3Spm
10Gb DAC cables: amzn.to/3wLrWeJ
HP NC523SFP Dual 10-Gigabit PCIe adapter: amzn.to/3fCjEjB
Asus XG-C100F 10Gbps PCIe 3.0 Network Card: amzn.to/3g1Ulqj
- HP DL80 Gen9 -
(Barebones) 2U: ebay.to/3ebduG4
HP Gen8 - Gen10 3.5" Drive Trays: amzn.to/3egChsf
Xeon E5-2660v3 - ebay.to/3i8JmMG
Xeon E5-2678v3 - ebay.to/3eh1yTb
- Supermicro Ultra Servers -
www.supermicro.com/en/product...
- Hyve Zeus 1U Servers -
Hyve Zeus Dual 2011-0 1U Barebones: ebay.to/2JxBFCI
Hyve Zeus Rails: ebay.to/3qxBJCU
Xeon E5-2650 v2: ebay.to/3lVJUWk
nVidia Quadro P400: ebay.to/2LyaiJg
nVidia Quadro P1000: ebay.to/3rIwf8n
HP Dual 10Gb SFP+ Card: ebay.to/3fpjmLk
1U PCIe Riser (the right one...): ebay.to/35TJLxG
- Chenbro NR12000 -
(Barebones) 1U 12-Bay: ebay.to/3s6X6eM
Chenbro NR12000 Rails: ebay.to/2O4NLF1
Hard Drive Rubber Grommets: amzn.to/35RPQKU
Drive Mount Screws (6-32 3/8" Truss Head): amzn.to/35Vn2kE
HGST 6TB SATA: amzn.to/3r32v57
PCIe 1U Riser: ebay.to/35TJLxG
- Storage Server -
Supermicro 846 Chassis: ebay.to/3pAilW1
Supermicro H11SSL-i SP3 Motherboard: amzn.to/3xtSkKG ebay.us/N73DGi
32GB DDR4-2666 ECC RDIMM (x8) - amzn.to/3zDledg ebay.us/rZwRv8
8TB HGST He8 Helium SAS: amzn.to/2X7Xb7C
Seagate FureCUDA 1TB NVMe: amzn.to/3iOOIij
Seagate IronWolf 110 1.92TB SATA SSD: amzn.to/3cNdKKG
EVGA Supernova T2 1600W: amzn.to/2U9cjjb
Noctua NH-U12S TR4-SP3: amzn.to/3q1lZby
Fractal Design Define 7: amzn.to/3iQJeUe
AMD Epyc 7601: amzn.to/3iKYNg5 ebay.us/D9ioE9
- Cloud Gaming Server -
AMD Epyc 7742: amzn.to/3hYoKIe
AsRock RACK ROMED8-2T Motherboard (7002/7003 Support): amzn.to/3xdxoal
256GB (8x32GB) DDR4 ECC-REG 2666: amzn.to/2TxxVpA
be quiet! Dark Power 12 1500W 80+ Titanium: amzn.to/3kTp0de
Asus Hyper m.2 x16 V2: amzn.to/3xdxtLb
Noctua NH-U12S TR4-SP3: amzn.to/3eOOftu
Noctua NH-U9 TR4-SP3: amzn.to/3eRAhHt
InWin R400N 4U Server Chassis: amzn.to/3BFYUjQ
- UPS Unit -
APC Smart-UPS RM SMT1500RM2U: amzn.to/3CIqc9G
- Air Conditioning -
Tripp Lite SRCOOOL7KRM on Amazon: amzn.to/34SYN6I
Duct Kit for SRCOOL7KRM: amzn.to/32L6Wrg
I've got merch, and you can get it too!
craftcomputing.store
Follow me on Twitter @CraftComputing
Support me on Patreon or Floatplane and get access to my exclusive Discord server. Chat with myself and the other hosts on Talking Heads all week long.
/ craftcomputing
www.floatplane.com/channel/Cr... - วิทยาศาสตร์และเทคโนโลยี
![[UNCUT] ย้อนฟัง 2 บิ๊กตำรวจปะทะกัน "บิ๊กโจ๊ก VS รองแต้ม" ฟ้องเรียก 10 ล้าน!! I คนดังนั่งเคลียร์](http://i.ytimg.com/vi/vNpTtVjdcBo/mqdefault.jpg)
For a professional beer review channel, you have a hell of a network setup!!
Nice rack! Also, home assistant is a great way to take control of all of your home devices.
Home assistant is a must for sure! The addons and default features will be all you ever need!
"I want self hosted home automation!"
Home Assistant? I've been runnig it on an airgapped system for a few months now. Been a non issue
Came down here to say this. TONS of videos on youtube. Heck, even Linus is going to run it in his new house.
I was thinking the same lol It's amazing how people don't find this stuff. Home Assistant has changed my life! Self-hosted, private, feature rich, amazing community. What else could you ask for?
I also use Home Assistant with ZWave, Ubiquiti, etc. I have redundant Raspberry Pi 4Bs plus I run it in ProxMox so that I have a local copy on each floor of the house (I have a rack on each floor that has local control and communicates with the others). Home Assistant also automatically controls all of my air cleaners, dehumidifiers, HVAC, exhaust fans, etc. based on environmental measurements all over the house and outside temps/air quality.
Yeah I can think of a few off the top of my head, I am setting up an open hab system right now for controlling some lighting in my bedroom, an open sprinkler controller, yard lighting, my home theater system, and my hifi stereo system.
HIGHLY recommend home assistant, w/ zigbee
"Letting everyone know you are better at typing than they are" CRACKED ME UP! 🤣
I'm truly amazed that your not running Home Assistant on all that. Not going to get any better for self-hosted home automation.
Same I think his argument is many of the smart things you can buy require the cloud
Lol when he started talking about not being able to run a home automation solution locally, I was like bruh... You gotta check out home assistant
@@testthisfordecficiencies name them?
@@testthisfordecficiencies What are they?
I've had that Docker installed on UnRAID for months now but haven't set it up. The updates to it are frequent and huge for some reason.
you might still want to consider going with two UPS units when you do decide to upgrade; having A/B redundancy and splitting the load between the two is a great way to not only bring the individual unit load down, but in the event of a failure, PM, battery swap, etc., you have one unit that can remain up and handle the full rack on its own. Plus, if you already have to run a 240 circuit, running a second at the same time is better than having to run it after you get everything buttoned up. that tripplite unit is frickin sick though! i didnt realize they had rack mountable AC units like that. will definitely be looking into one of those.
"rails are worth their weight in gold"
yea sadly sellers for said rails think the same and they are hidiusly expensive for what they are in the end
it depends. sometimes just the inner rail, other time just the outer
Shipping costs are a factor too. Dimensional weight is a pain.
Thanks for the tour. Lots of cool systems, and lots of cleaning to do
Thanks for the demo and walk thru, have a great day
Great as always.
Thank you for this! Glad I found you
I’ve just happened upon your channel after searching for home computer/server lab setups. Of course I have subscribed! I’ve worked in IT for the past 13 years and am finally setting up my own computing lab for fun and learning new IT skills/certifications. Looking forward to viewing more of your content. 😀
great setup
Blooper end.. loved it
This is amazing. What you could do is grab all the metrics from the entire rack from temps, cpu, memory, networks, whatever and be able to track it and monitor it on a single grafana dashboard. Probably can even host the grafana services on your systems. Again amazing. I do this for my current job.
I see a fair few comments about using the vented air to heat the house during winter, but that's the time of year when less such air would be created anyway. Also, as it is, the air would be rather dry, laced with heaven knows what vapours from all the machinery and probably static prone, not something one would want to blow straight into a home. Perhaps a heat exchanger could do something useful, I don't know.
If I am ever in OR, one stop would be to have a meetup and beer with Jeff! Love the content!
I can vouch for the Mikrotik equipment.
Last company I worked they had a router board and a switch from them and at the first glance I, who always had used Cisco/Netgear/other enterprise brands dismissed as bad.
Yes their management software on the PC is archaic and somewhat confusing, with a somewhat step learning curve.
But in terms of hardware for the price/performance bracket, heck they are some powerful little machines.
Yes has you said, if you start messing with L3 the cracks start to appear, but again as you said, for L2 that is most of the normal needs of a small to medium business, they just rip and tear.
L3 and cracks? Laughs at CCR1036-8G-2S+EM at work! I mean who excepts gigabit+ routing to be good with a SWITCH?!
I don't know most of what I just heard, but I love hearing it. This stuff is awesome. Here I am with my NAS made from an old Dell Optiplex and feeling happy that I just bought a single 14TB hard drive. Maybe someday I will have a valid excuse to do any of this home networking stuff.
This is literally how my Media Server started. ~10 year old gaming PC that was collecting dust, figured I'd get Unraid and play around with it. 4 years later I'm still using it, and have 32TB of storage, plex server, and other fun stuff.
Very informative and entertaining. Didn't realize you had that much processing power.
2:35 look at you with your fancy 50 up! Spectrum doesn't go past 40 for love or money 😭
Oh, I pay handsomely for upgrading from 20 Mbit to 50 Mbit....
Me just vibing wondering if I should go from $55 a month for 100 by 100 to $70 a month for 500 by 500 🤔
Kinda crazy to think that I pay $85 a month for gigabit fiber up and down in central Iowa. Ya'll need to move out here lol
Rogers won't give higher then 30 up on their gig plans and Bell is taking their sweet time finishing FTTH in this building
@@computersales And my bill for 1000/40 is $150/month 😵
Awesome Rack! Thanks for the tour!!! Cheers!!
😯 props for PRTG!!! 😁👍👍👍
my birthday could not get any better :)
Love me some Dead 'n Dead.. It's amazing, especially if you hit one of their restaurants and get it right off the tap.
Great tour! Cant wait to se the video on the supermicro server.
Nice tour Jeff!
I'm living my dream of a full fledged server rack home lab through Jeff!
My home lab just got rehoused in a Meshify 2. I'll be adding a rx580 4GB for some VM usage shortly, and hope to add more/larger HDDs for my plex and file file server. Proxmox FTW!
This is impressive Jeff!
Nice setup! I would love to see a future video from you on how you then have come up with a dust filter solution😉
The only time we can say "NICE RACK!" without offending someone.
I had to specifically add "Nice rack" to my comment section whitelist, just for videos like this 😀
@@CraftComputing Seriously?
@@SpookyLurker I believe it. It's very easy to imagine TH-cam automatically holding comments that say "nice rack" for review because so many scumbags would be posting it inappropriately in the comments of any video with a woman in it.
LOL
@@SpookyLurker what did he said lol
Great video as always. Just one observation - everything needs a good clean
You need to setup positive pressure filtered air in your garage to get those dust levels down! a few hvac hepa filters and two of the same blowers your using for heat extraction would do the trick, nice little weekend project and youtube video content!
One year and a half already? Wow! Hope you keep growing up!
Home Assistant is what I use. I can run it off of a Raspberry Pi computer. It is easy to set up and maintain. I also use Linkind switches that I flash with Tasmota to keep everything local. I totally understand the need of having local control over my local devices. Awesome rack, I tip my hat to you.
hey man, i live in the netherlands and we have this thing around christmas time called "surprise". its basically where you make a present for someone else of the group you do it with (chosen at random). i have my dad and im making a GIANT beer bottle and i wanted to ask you what beers you whould reccomend. my dad drinks a lot of special Belgian beers. i would like to get some weird beers and i have already chosen a few like a beer with a taste of a cake thing we eat here a lot, "caramel fudge stout", and "christmas ale". thanks already!!!
Great video. Would have loved for you to talk about the power usage in watts and what is costs to run the setup per month.
Nice video!!! Related to Mikrotik RouterOS v7 is in release candidate and this version enables L3 HW offloading so the issue on L3 routing will be addressed and fixed. You may want to explore a video testing this new version!
be careful with that feature ;) it still is a bit buggy (even on 7.6 as of date writing this comment)
@@drumaddict89 for me has been working great!! Stable and no issues at all
@@kirksteinklauber260 glad to heat that. on which hardware you are using it? cheers
@@drumaddict89 I have deploy it on 5 x CRS326-24G, 1 x CRS312 and 2 x CRS328-24P and all running fine and with lower CPU usage after l3hw-offlod was enabled
@@kirksteinklauber260 well thank you sir, i need to check that one out on my CRS326 then.
nice setup and you even share power draw noice
You can host home assistant and buy tasmota flashed (or flash your own) devices to integrate and be completely independent from the cloud
Yo I work in IT been at a couple notable companies in RTP in NC and just straight up understanding the jargon you are saying is awesome. Not to mention the amount of effort and tears goes into a full rack like that. A large kudos from me (not that you need it necessarily) but I wish I could manage to do something like that but HOA might have something to say.
Looking forward to the Supermicro review! The only thing missing is a PDU. With the right PDU you can measure/log input Voltage/Amps, per outlet output V/A, temperature/humidity sensors and the ability to remotely turn off/on outlets of anything in your rack. I have Zabbix monitoring and remotely rebooting a power meter monitoring box via a PDU when it stops responding to API calls (it's old, no longer supported and no direct access). Cheers!
What PDUs do you have in mind?
Big Fan of the channel, bit jealous of your setup ;) Got something similiar going on in my home, but what I'm struggling with is cable-managemet so I can actually "use" my sliding rails right... Maybe follow up video like a "how to cable manage in a rack"? Great Video, love the content!
Dead Guy + whiskey barrel chips? Time to figure out where to find some in Salem, then. Great tour!
As for your KVM solution, I used to have mine in the rack but over time I wanted to have side surfaces for my notes and other needs so I decided to move it to a full desktop surface 10 feet away from my dual racks. (refer to my home lab videos...) It was a great move for me. I can see my notes and so on and have room for space for anything I want vs. a tight format in my rack. BTW you could move this KVM mode back to your desk if you want. Just saying...
Hey Jeff, long time watcher of your TH-cam channel. When you got to the USW and mentioned having issues trying to figure out an anti-cloud home automation solution I just felt like I had to reach out personally to say that you can pick my brain about home assistant. I have over 100 devices in my house, from bulbs to switches to sensors to TVs and much more, all with automations galore. I would love nothing more than to give back and help you out if you need any guidance. Oh, and I have a dedicated beer fridge in my basement … Cheers 🍻
Home Assistant with a Zigbee / Zwave usb stick would be good. Also Home Assistant Amber would be good with a cm4.
Agreed. Can't wait to check out Amber!
@@JeffGeerling holy smokes! I I love watching your videos. Amber seems like a win in my books.
Great video! You should have mentioned that the UDM pro also has a poe version now. It’s worth mentioning since it’s basically and all in one router /switch. Also that while the UDM pro does have a single hard, they make a dedicated nvr with raid if that is what you need. 4 bay or 7 bay… home user/small bis only probably have like 3-5 cameras at best
The best comes last, heck of a computing Power :o
Awesome
I’m surprised you still use the UDM Pro. I stopped using anything Ubiquiti related due to the constant bugs and issues. Switched to Untangle which is absolutely fantastic, switches and APs are now Engenius (also working out great).
Of course you can put your backup server into another room, on another power storage and different flood-level (in your same home)
Amazing & impressive ! Take care of dust, specially in the bottom of the rack :-)
if you have an APC netshelter you should look at the ACF400, a replacement back door for netshelters with fans to force/exhaust hot air out attached ducting
Cheers, I’ve been following since day one and what an upgrade you’ve made over time. Awesome lab. A good project for the future would be powering everrything with solar power. (Given that it is feasible in your region) That would massively drop your energy bill, and given that you are planning on installing a new apc it makes sense to future proof it.
I'm highly considering solar as part of an upgrade this year.
I'm definitely interested in when you go to 240V!
Should have been in NBA with that height! LOL thanks for the server tour it's been very good.
FYI Layer 3 Acceleration for CRS 3x devices is coming in version 7 of ROS which is currently in beta. Check out the release notes for version 7.1 beta 6. The current verison is 7.1 RC6
I feel your pain on the Comcast front. I'm using a T1 line from GTT for upload and public-facing stuff because it was the only way I could get more than 5 static IP addresses. I have a /27 that's basically maxed out.
The UDM Pro SE is on the early access store and will have 8 POE++ ports and an internal SSD
I just started a Threadripper Pro build in a CSE-836 to replace my current 2 x E5-2630L v2 CSE-846 UnRAID server.
I have the case and motherboard now but am going to wait on the CPU and RAM until CES to see if the 3rd gen Threadripper Pro is released. If it's not I'll get a 3955WX. It should be a HUGE improvement.
I also just grabbed a L1Techs KVM and it's awesome!
21:43 Guac-a-molly, the little known dip that makes you trip.
That dusty hat makes my OCD go crazy :DD
Looking much better man! Great video. You are getting there... Now you just need a high power bill to pay each month like I do... lol
The chenbro does have a sas controller. Hdd 1-8 I believe are all sas. I have 5 SAS drives in mine BUT you have to use a sata to sas adapter and you won't be able to fit 12 drives with adapters but it does have SAS controller. They might have smaller adapters out there though.
Great video. One question: Can you redirect the air conditioned exhaust to help to heat your house in the winter ?
Great video man!!
I know the udm you have doesn't have PoE but it is worth noting they do make one with four PoE ports on it now.
I believe Lutron caseta dose use a hub for their proprietary clear connect but its its dependent on the internet.
Odd that I was wondering what had changed on your rack and you just now post this.
Don't forget that your PBX provides the "cloud" LED system used to signify to your wife and kids that this is YOUR OWN CLOUD SERVER :D
Look at all the dust on those servers.
might want to clean some of those intakes
I want the beer.
and I thought I had alot to dust just in my room lol Man clean that out. You gonna mess up your servers!
UDM-Pro SE will be coming out soon with 8 Gigabit POE ports with a few extra nice to have features including. Keep an eye out.
With the same 1 Gbps switch backplane as the original. Those ports will still be useless in many cases...
@@bartoszkazmierczak7249 Right, but it should work much better for an all in one NVR for home use. Esspecially if you're just connecting 1080p cameras to the 8 port POE switch.
11:30 you know why rack rails are worth their weight in gold? because they cost about as much as if they were made of the stuff.
Unifi have a unvr and a unvr pro now for the cameras for more drive bays.
oh boy
I mean, you "could" back up to S3 in AWS with a snowball, but that is overkill. I just like thinking out of the box, great setup though!!!!
Same CRS328-24P-4S+😁
DUDE I love the videos ! and the rack and sever play stuff , have u made any videos about unraid ?
"Guacamale" 😂
I started experimenting with a PI security camera with PoE hat and an ELP camera with night vision. it may be an option at least for the cameras.
Great overview. I must admit that I'm a bit nervous that you don't have an off-site backup. But then ISP in USA are complete ass
so is most of the rest of our infrastructure. that's why so many of us build our own stuff!
🔥🔥🔥🔥
@Craft Computing You have a great rack! As a fellow 6’5 person, when you grabbed a chair and sat next to your rack, I felt that.
I see there's quite a bit of dust buildup on some of the lower devices. Do you have any processes or procedures for physical cleaning of your systems?
For your consideration, when upgrading your UPS, consider upgrading to something like the Tripp Lite SU2200RTXLCDN. It has a 2200VA/1800W rating (not quite double what you have now). It is a double conversion UPS rather than a line interactive which will provide more consistent clean power. And, the best part is the option for expandable run time. While you could add on run time with their OEM rack mount battery packs, since you are short on rack space, and to keep some green in your pocket... go DIY and build your own external battery bank. I just recently did this myself with a handful of marine deep cycle batteries, welding cable, and the same Anderson SB175 connector that Tripp Lite uses for their own battery packs. All said I spent less than $1k on the battery pack and got roughly 3 times the run time per dollar compared to OEM. If you mirrored the exact same build that I did, with your current 1kW load, you would get about 3 hours of run time. Feel free to give me a poke if you have any questions about how to build it. I'm happy to lend a hand.
Links? Pictures? Video(s)?
@Scrat sorry man. I tried to reply with Amazon links to all the major parts I purchased, but TH-cam bots think it's spam or something and remove the post (I tried reposting it 3 different times). Hopefully you can figure out most of the cable, lugs, etc... perhaps the part I had to do the most research for was the Anderson SB175 connector Beyond that, the next nugget of wisdom to pass along is that the Tripp Lite UPS I have (as well as the one I recommended) doesn't talk to the battery bank... it's just a box full of batteries. So, it has no idea how big or how many of them there are daisy chained in the system. In the UPS software interface you just have to plug in which OEM banks you have connected and it understands the capacity of each one to calculate your run time. That said, you need to do some math and figure out how much capacity your DIY bank has with respect to the OEM banks and just put in the appropriate number of OEM banks to match your DIY bank capacity.
You should upgrade your Comcast modem to the Nighthawk CM2000 since you have the UDM Pro. You can then take advantage of the new 1.4-1.5 Gbps downloads when plugged into the 10Gbps port on the UDM Pro … yes it will negotiate with a 2.5Gbps port on the Nighthawk CM2000. Uploads still stink at 50mbps but the downloads are great.
theres a reason why alot hate comcast it makes me glad im in a area not serviced by them and has the option for google fiber but i in reality dont have a backup for my server currently as i dont have the funds and also a place to store a server that backs up my media/backup server
You mention that the AC unit only runs about 8 months out of the year. I'm curious what the coldest temps are in your garage during the remaining 3-4 months? I've been considering moving my computer equipment to the garage, but it can frequently drop down to below freezing (and occasionally below 0F) in the winter where I live, so I'm a little worried about that
Your electric bill must set you back a pretty penny every bill cycle .... Epic rack equipment , and thank you for the tour .
$0.074/kWh ;-)
Jeff just curious was there a typical load on your rack when you were filming? I was just curious not a huge amount of head room left on a typical outlet there let alone your ups. So can your cloud gaming setup up run in that rack or do you have to run it somewhere else?
don't know if it's been said yet but....
"Nice Rack!"
Good video. By pumping hot air of that room/garage, where is your inlet to compensate it? I guess hot&humid air from the outside?
I use a modem with a 2.5gbe port and the UDMP has a SFP+ module in WAN2 that will negotiate to 2.5gb while presenting to the UDMP as 10gb.
Works well, when the UDMP is working well, that is. :/
As someone who uses a lot of APC UPS units. I recently discovered that HPE UPS has a much better user interface.
RTX voice/nvidia broadcast is trully magic in removing unwanted noise from a mic
It'd love to get all of my desktops off of the floor, stick them into a 1 or 2u case and rack mount them to free up some space.
Have seen Linus's PC closet? That's kinda what I want too, even though I like seeing my good-looking PCs on my desk.
Could you compare the Cost of the Power Consumption aswell of yearly upgrades to servers aviable from hosters?