Those pcie switches for accommodating more NVMe drives were a nice idea. They should be doing that more often in other areas so that we can add as many pcie devices as we want :) Thanks Patrick for showing this to us.
Phoronix posted an article yesterday about DDR5 memory channel scaling on EPYC 9004 series, it might be useful for anyone to look at side by side with this video. In the Phoronix benchmark results 10 channels seems to be about the sweet spot vs DDR5 cost, with that said if you're building a system of this scale the cost of DDR5 DIMMs isn't a big issue. 😅
120kW + potential 44u rack density. Man i wished i still designed and built datacenters, would love a crack at that challenge, and 10 years ago or so, 40Kw would have been an impressive feat
I've actually started seeing the 96c parts showing up for well under MSRP even for individual unit cpu sales. They are still pricy but actually the prices aren't too bad all things considered.
Otoh look for the per core performance. If you can afford the licensing and you have the scale, these are the platforms to get. Personally i look forward for 9554 performance figures, as these should be enterprise licensing kings for a while
Soo close... I was looking forward to diving into more accessible CXL. Interesting to see how far ahead of commercial software (or at least licensing models) that hardware is getting.
Hello Patrick, I always appreciat your passion about hardwares. Do we have a chance to add test softwares that Corona Render Benchmark and V-Ray Render Benchmark? Those are the engines that use all the cores properly. Not just because I use them all the time😁
The USB on the mainboard is for booting Vmware ESXi into a vSphere management environment. I've installed a lot of IBM servers running ESXi and that's pretty much how it's done now.
Unfortunately, VMware has given guidance not to install to USB anymore due to USB drive endurance. These days the M.2 installation is much more common because they are higher quality drives and very low cost.
@@ServeTheHomeVideo ahh was thinking how could one make a server for home be absolutely insane always thought of you could get a Epyc CPU with SRIOV you could easily make the best setup Media/VM though it'd be insanely expensive but you wouldn't need to upgrade for many years and can avoid the limitations by conventional server cpus. Very cool to see these servers getting more and more modular shows industry really wants to break the old proprietary shackles and actually design with purpose. Thank you for always showing these innovative products.
@STH, no need for a direct answer. I am just hoping this serves as food for thought. Maybe another video... First, as an IT guy, I think about failures all day. How is the warranty and onsite support? With Dell, I can call 24/7 and speak to a tech (with the appropriate warranty options chosen) and get a tech within 4 hours with new parts. Is that type of support available with the newer servers that you are showing us? Also, when ASUS makes a server like this, what use case did they envision for this? I would like to see an example of the workload this handles. Are these always parts of dedicated clusters of 50 nodes where a single server failure is just a blip on a performance monitor in the datacenter or is this meant to handle a load by itself at a mom and pop, maybe doing graphics encoding or some workload only running on this single node? Again, no real answer needed or expected down here, I would just enjoy hearing answers to those while making new reviews. Thanks again for the amazing content!
I have a bit of a different thought on support. I think ASUS generally assumes that smaller organizations will use resellers that will provide that on-site service capability. I was on the board of a ~$300M/yr reseller of IT gear including servers from smaller vendors and that was common in the industry. Dell also charges a LOT for their service. After, a few servers it is much cheaper to buy extra nodes than to get on-site service so long as you have an environment where you can effectively utilize hot/ cold spares. Usually warranty and service are based on a percentage of list price and that is the pricing mechanism that makes buying extra boxes very cost effective even at a smaller cluster size. On the workload side, these are more of general purpose servers so they could handle GPUs to do graphics encoding or be in clusters. They are not heavily optimized for GPUs or video encoding though. Usually we talk about the specifics for servers that are more focused on an application rather than this ASUS server that is more general.
I cant waity for Zan5, or whichever generation gets rid of the L3 and maybe the L2 on the compute die, and then slaps the cache directly on top of the core, you could have an IBM like cache layout with a segmentable combined L2/L3, stacked on top/under the core(whichever is hotter will need to go on top) Imagine 16 cores, 168MB L2/L3 on a single chiplet.
a beast - team it up with a huge array of spinning rust and new apu to really unleash potential - this will have a long lifetime and even smb will use this in a few years #power10 #arm #consolidated
The major Disappointment I have about this server, there is no QSFP or QSFP+ on board for that kind of CPU power, ram and NVMe speeds. What was asus thinking putting 2 ethernet jacks on the back? You can find HP Proliant's and Dell Poweredge's with onboard QSFP ports some even have 4 on board.
Hi Friends! Nice, one thing I hate about Asus is their trays are crap feeling. I have to say I was quite disappointed in them on our 7443P builds. They really should move to making their trays tool-less as well.
@@ServeTheHomeVideo Oh nice to hear! Glad they stepped it up. The ones we currently use don't and feel super cheap. They looked the same from the front. :)
Another advantage of choosing the 32 core CPUs is the chiplets can double up IO bandwidth by using two instead of one IO die connections. I'd like to (but do not) know if this can be done with half of the cores on the 64 core CPUs?? Will these dual-socket boards support mix-and-match of Genoa/Genoa-X/Bergamo/Siena CPUs? ...I can see several use-cases for this hybrid approach... e.g. Bergamo CPU is running server processes for clients while the second Genoa-X CPU runs the dedicated database layer, all on a single dual-socket server unit, a perfect use-case too for using a CXL memory as shared memory. p.s. What speed difference between 1 dimm per channel and 2-per?
I wonder if the ASM iKVM on this thing is just as trash as on the WRX80 SAGE, where CDROM emulation is completely busted when loading ISO from the microSD, or when the web UI randomly logs you out in the middle of a remote KVM session.
The auto-logout one is a setting that many vendors (not just ASUS) have. Usually that is easy to fix by just changing the option to a longer duration since many times it is default to 15-30 min. We have many machines with 4+ hour logouts now.
@@ServeTheHomeVideo TY. I'll have a look around for the option for that. Still gutted that microSD ISO emulation is broken (yet somehow ISO over NFS is fast). It's slower than a real DVD drive lol.
Holly crap that's a nice machine... Just in time for my homelab :P Regarding this, I'm going with only dual 64 core and 1tb mem, I think that's enugh for few VMs and containers for homelab VM host + nas, VPN, DNS, fw/router + media server :PPP Keep dreaming, but would be nice anyway
Patrick XSL/XCL are still being traded and the market standardized on them. Epyc has taken a small chunk AMD being supply constrained. The majority of the general data center compute / enterprise market is waiting to see what come next? Would you agree? The question then is are Sapphire Rapides and Genoa too complex for the vast majority of the general server market beyond virtualized containers on the applications coding challenge? mb
Any word on Genoa motherboards suitable for 'desktop" workstations - single or dual CPU? ASROCK have a "mini" board, but it's rather cut down compared to 'regular' motherboards.
It is tough to fit Genoa into desktops just because of how wide SP5 plus 12 DIMMs are. We have the ASRock Rack board and will have a review on the main site the last week or so of this month.
@@ServeTheHomeVideo 😎, thank you, 👍. I did find one company (there may be others) that offered 'tower' workstation cases specifically designed to take (some) vertically mounted rack 'boards, but can't find it now. They did have some Gen' 3 EPYC systems as well as XEON. Supermicro did some for the XEON, in single or dual format, with a neat liquid cooled option, so maybe they're looking at genoa options there? They're all outside my current budget, and certainly massively over-kill, but neat seeing what's out there for the folks that can actually use these machines.
Hehe "fans there to make you feel good ?!!!". Jesus, these data centre fans sound like a jet engine at full blast!. They needed phenomenal airflow across the slim channels designed for cooling. Decades of data centre work in both IT and Telecom premises means I needed earplugs if I had to work for hours troubleshooting a particular server. Plus thick clothing as well!.
Watching this as someone who's not in the industry but is a tech enthusiast. I see everyone here mentioning Microsoft and Oracle licensing costs for servers being crazy because of per core licensing. Is there not another solution? Why does everyone need to use this software?
Because lots of legacy software were written with those solutions in mind as there was no easy alternatives back then. Right now such solutions exist and both vendors doing their best to maintain income in shrinking market
How are the NVMe SSDs managed by the switch boards? If each is connected through x16 PCIe 5.0, its like having 32 PCIe 4.0 lanes. If im not mistaken there arent U.2 PCIe 5.0 drives out there. So can the NVMe SSDs be used with their actual bandwidth?
Nvm, so 16 NVMe drives can be used in that configuration. Both switch boards are connected with PCIe 5.0 x16 each which totals around 64 PCIe 4.0 lanes in bandwidth. So each U.2 PCIe 4.0 NVMe of those 16 slots can be used with their actual bandwidth .. Nice :)
We got ~3% higher performance at 400W cTDP on the 96C parts. Mentioned it in the main site article but those numbers were not done when the video was recorded.
Remember, the STH main site is ~10x the size of the TH-cam channel at this point. The reason we do not do more video reviews of the servers we review on the main site is that the audience is much smaller for us. Right now TH-cam is a subset of what we do on the website.
Hey Patrick! I think I saw you at Costco on Friday LOL. I regret not saying hello! maybe next time. Great videos!
Ha! I was at Costco in Austin yesterday. You should certainly have said hi!
Those pcie switches for accommodating more NVMe drives were a nice idea. They should be doing that more often in other areas so that we can add as many pcie devices as we want :)
Thanks Patrick for showing this to us.
I like your high enegy product reviews + really amazing detail and quick technical understanding. Thank you once again STH
Thank you! I feel like this was much slower than the ones I did for Supermicro X13 which felt like lightning-round 3-5 min videos.
Phoronix posted an article yesterday about DDR5 memory channel scaling on EPYC 9004 series, it might be useful for anyone to look at side by side with this video.
In the Phoronix benchmark results 10 channels seems to be about the sweet spot vs DDR5 cost, with that said if you're building a system of this scale the cost of DDR5 DIMMs isn't a big issue. 😅
Even when costs are no bar, 10 channels seems to be producing close to top results. Using all DIMMS would only be to increase memory size.
120kW + potential 44u rack density.
Man i wished i still designed and built datacenters, would love a crack at that challenge, and 10 years ago or so, 40Kw would have been an impressive feat
What a beast. Servers are so huge nowadays in terms of performance. This level only makes sense for huge players and cloud
I've actually started seeing the 96c parts showing up for well under MSRP even for individual unit cpu sales. They are still pricy but actually the prices aren't too bad all things considered.
😊
@@robertojosedossantos9771 ෛටඓඓනෛඓඓඓෛඓඓරෛඓරෛඓඓඓඓඓෆෛෛෛඓඓඓඓඓඓඓඓඓ නම් බහ
. ඓඓඓ
imagine the m$ or oracle license cost.....
So much. We only showed down to the 32C 9374F's because those are the smallest we have.
Otoh look for the per core performance. If you can afford the licensing and you have the scale, these are the platforms to get. Personally i look forward for 9554 performance figures, as these should be enterprise licensing kings for a while
$40K per core for Oracle EE database.
@@oscarcharliezulu JESUS FUCK
@@oscarcharliezulu lol, Oracle is a joke
Soo close... I was looking forward to diving into more accessible CXL. Interesting to see how far ahead of commercial software (or at least licensing models) that hardware is getting.
We are going to do that. Just need Intel to launch too, although there is a quirk there (more in Tuesday's SPR launch video)
I really like those PCIe switches. I actually have some on my AMD desktop motherboard which supports 3 NVMe drives on a B550. They are very neat.
Hello Patrick,
I always appreciat your passion about hardwares.
Do we have a chance to add test softwares that Corona Render Benchmark and V-Ray Render Benchmark?
Those are the engines that use all the cores properly.
Not just because I use them all the time😁
We were thinking of adding v-ray later this year and can look at corona.
The USB on the mainboard is for booting Vmware ESXi into a vSphere management environment. I've installed a lot of IBM servers running ESXi and that's pretty much how it's done now.
Unfortunately, VMware has given guidance not to install to USB anymore due to USB drive endurance. These days the M.2 installation is much more common because they are higher quality drives and very low cost.
@@ServeTheHomeVideo correct, plus most usb drives make esxi perform like poop
Now I wish Genoa would release a 12-16 core cpu for a slick True Nas build with VM on side.
Actually, I think these are too big for traditional TrueNAS. For TrueNAS Scale, with hyper-converged they are more interesting.
@@ServeTheHomeVideo ahh was thinking how could one make a server for home be absolutely insane always thought of you could get a Epyc CPU with SRIOV you could easily make the best setup Media/VM though it'd be insanely expensive but you wouldn't need to upgrade for many years and can avoid the limitations by conventional server cpus.
Very cool to see these servers getting more and more modular shows industry really wants to break the old proprietary shackles and actually design with purpose.
Thank you for always showing these innovative products.
Wait for Siena
@STH, no need for a direct answer. I am just hoping this serves as food for thought. Maybe another video...
First, as an IT guy, I think about failures all day. How is the warranty and onsite support? With Dell, I can call 24/7 and speak to a tech (with the appropriate warranty options chosen) and get a tech within 4 hours with new parts. Is that type of support available with the newer servers that you are showing us?
Also, when ASUS makes a server like this, what use case did they envision for this?
I would like to see an example of the workload this handles. Are these always parts of dedicated clusters of 50 nodes where a single server failure is just a blip on a performance monitor in the datacenter or is this meant to handle a load by itself at a mom and pop, maybe doing graphics encoding or some workload only running on this single node?
Again, no real answer needed or expected down here, I would just enjoy hearing answers to those while making new reviews.
Thanks again for the amazing content!
I have a bit of a different thought on support. I think ASUS generally assumes that smaller organizations will use resellers that will provide that on-site service capability. I was on the board of a ~$300M/yr reseller of IT gear including servers from smaller vendors and that was common in the industry. Dell also charges a LOT for their service. After, a few servers it is much cheaper to buy extra nodes than to get on-site service so long as you have an environment where you can effectively utilize hot/ cold spares. Usually warranty and service are based on a percentage of list price and that is the pricing mechanism that makes buying extra boxes very cost effective even at a smaller cluster size.
On the workload side, these are more of general purpose servers so they could handle GPUs to do graphics encoding or be in clusters. They are not heavily optimized for GPUs or video encoding though. Usually we talk about the specifics for servers that are more focused on an application rather than this ASUS server that is more general.
HA HA HA The ad that played first was about a couple getting engaged!
I cant waity for Zan5, or whichever generation gets rid of the L3 and maybe the L2 on the compute die, and then slaps the cache directly on top of the core, you could have an IBM like cache layout with a segmentable combined L2/L3, stacked on top/under the core(whichever is hotter will need to go on top)
Imagine 16 cores, 168MB L2/L3 on a single chiplet.
i love doing 7z b on a new server with more cores and seeing the big number wooooo
This will be a the ultimate render-server for any media company.
Dedicated OOB pcie lanes is a great feature! They don't need to be fast, so don't waste the 5.0 IO
a beast - team it up with a huge array of spinning rust and new apu to really unleash potential - this will have a long lifetime and even smb will use this in a few years #power10 #arm #consolidated
The major Disappointment I have about this server, there is no QSFP or QSFP+ on board for that kind of CPU power, ram and NVMe speeds. What was asus thinking putting 2 ethernet jacks on the back? You can find HP Proliant's and Dell Poweredge's with onboard QSFP ports some even have 4 on board.
Hi Friends!
Nice, one thing I hate about Asus is their trays are crap feeling. I have to say I was quite disappointed in them on our 7443P builds. They really should move to making their trays tool-less as well.
These are tool-less drive trays.
@@ServeTheHomeVideo Oh nice to hear! Glad they stepped it up. The ones we currently use don't and feel super cheap. They looked the same from the front. :)
Another advantage of choosing the 32 core CPUs is the chiplets can double up IO bandwidth by using two instead of one IO die connections.
I'd like to (but do not) know if this can be done with half of the cores on the 64 core CPUs??
Will these dual-socket boards support mix-and-match of Genoa/Genoa-X/Bergamo/Siena CPUs?
...I can see several use-cases for this hybrid approach... e.g. Bergamo CPU is running server processes for clients while the second Genoa-X CPU runs the dedicated database layer, all on a single dual-socket server unit, a perfect use-case too for using a CXL memory as shared memory.
p.s. What speed difference between 1 dimm per channel and 2-per?
Awesome. How much would something like this cost?
@@_-Karl-_ wauw, you hurt my feelings
@@_-Karl-_ thank you
The v-cache ones will 1gb+ cache will be interesting
Ah yes perfect for my TrueNAS server!
its cool, just don't make me buy per-core licensing for that thing
Are SuperMicro going to be releasing H13 building blocks?
I wonder if the ASM iKVM on this thing is just as trash as on the WRX80 SAGE, where CDROM emulation is completely busted when loading ISO from the microSD, or when the web UI randomly logs you out in the middle of a remote KVM session.
The auto-logout one is a setting that many vendors (not just ASUS) have. Usually that is easy to fix by just changing the option to a longer duration since many times it is default to 15-30 min. We have many machines with 4+ hour logouts now.
@@ServeTheHomeVideo TY. I'll have a look around for the option for that. Still gutted that microSD ISO emulation is broken (yet somehow ISO over NFS is fast). It's slower than a real DVD drive lol.
Holly crap that's a nice machine... Just in time for my homelab :P Regarding this, I'm going with only dual 64 core and 1tb mem, I think that's enugh for few VMs and containers for homelab VM host + nas, VPN, DNS, fw/router + media server :PPP Keep dreaming, but would be nice anyway
Patrick XSL/XCL are still being traded and the market standardized on them. Epyc has taken a small chunk AMD being supply constrained. The majority of the general data center compute / enterprise market is waiting to see what come next? Would you agree? The question then is are Sapphire Rapides and Genoa too complex for the vast majority of the general server market beyond virtualized containers on the applications coding challenge? mb
and storage servers . . . mb
Certainly a challenge is how large the platforms are but also how expensive simply making systems with DDR5 and lots of PCIe Gen5 is.
Any word on Genoa motherboards suitable for 'desktop" workstations - single or dual CPU?
ASROCK have a "mini" board, but it's rather cut down compared to 'regular' motherboards.
It is tough to fit Genoa into desktops just because of how wide SP5 plus 12 DIMMs are. We have the ASRock Rack board and will have a review on the main site the last week or so of this month.
@@ServeTheHomeVideo
😎, thank you, 👍.
I did find one company (there may be others) that offered 'tower' workstation cases specifically designed to take (some) vertically mounted rack 'boards, but can't find it now. They did have some Gen' 3 EPYC systems as well as XEON.
Supermicro did some for the XEON, in single or dual format, with a neat liquid cooled option, so maybe they're looking at genoa options there?
They're all outside my current budget, and certainly massively over-kill, but neat seeing what's out there for the folks that can actually use these machines.
2×2600W Gaspower-ed 😉
이정도면 지뢰찾기 풀옵션해도 잘 돌아가겠다. 함 써보게 보내줘바요 돈은 그대들이 지불 하시고,
sounds expensive, I want one lol
336 watts only for the four fans 😀
No !
Hehe "fans there to make you feel good ?!!!". Jesus, these data centre fans sound like a jet engine at full blast!. They needed phenomenal airflow across the slim channels designed for cooling. Decades of data centre work in both IT and Telecom premises means I needed earplugs if I had to work for hours troubleshooting a particular server. Plus thick clothing as well!.
😊Taiwan 🇹🇼No1 😉
Watching this as someone who's not in the industry but is a tech enthusiast. I see everyone here mentioning Microsoft and Oracle licensing costs for servers being crazy because of per core licensing. Is there not another solution? Why does everyone need to use this software?
Because lots of legacy software were written with those solutions in mind as there was no easy alternatives back then. Right now such solutions exist and both vendors doing their best to maintain income in shrinking market
@@zbigniewmalec4816 thanks for the info
Sure I have one of those downstairs
7A fans. i can hear them from here
But can it run Crysis?
😍😍😍
Can we see Cinebench and AIDA64 memory speeds?
How are the NVMe SSDs managed by the switch boards? If each is connected through x16 PCIe 5.0, its like having 32 PCIe 4.0 lanes. If im not mistaken there arent U.2 PCIe 5.0 drives out there. So can the NVMe SSDs be used with their actual bandwidth?
Nvm, so 16 NVMe drives can be used in that configuration. Both switch boards are connected with PCIe 5.0 x16 each which totals around 64 PCIe 4.0 lanes in bandwidth. So each U.2 PCIe 4.0 NVMe of those 16 slots can be used with their actual bandwidth .. Nice :)
Full nerdgasm!!!
still kinda wanna see the r23 number tho
why will any server need 384 threads? for what
If you lift that thing fully loaded you're going to need a mega pint
This is not too bad. The GPU and 60-100 drive storage servers are much heavier.
and now the fan noise
1.2kW not fully loaded in a 2U. Nice.
How do the individual CPUs communicate between each other to spawn new threads?
🤖🥩🍞🥗🍰👍I wish U would tell us about the chipset's also! pure genious! why go over the mountain, when we can visit the lake!
No chipset here.
Could I use a server as a basis for a DAW?
These things sound a bit like a large sinousidal oscillator at C8 at full volume - you really don't want one sitting in the same room as you!
@@capability-snob that’s clear, I’d put it somewhere away from my mixing desk, in my cellar probably
You probably could. But does it make any sense is another question.
5W per DDR slot?! Isn't it too much?
Nope
When will AMD take over Intel in server marketshare?
Can those passive heatsinks cool 400W apiece without thermal throttling?
The CFM of those fans should be high enough. Also, the air intake temp should be cold enough thanks to the AC.
We got ~3% higher performance at 400W cTDP on the 96C parts. Mentioned it in the main site article but those numbers were not done when the video was recorded.
Licensing is killing innovation...
i will hear the noise of this server...
im never going to approve purchase on asus hardware, just no way
I bet you can compile Chromium in 3 hours xd
No RAM installation montage! SCANDAL!
I feel like if this guy buys these himself to give us reviews , his TH-cam revenue is not enough to cover the costs.
Remember, the STH main site is ~10x the size of the TH-cam channel at this point. The reason we do not do more video reviews of the servers we review on the main site is that the audience is much smaller for us. Right now TH-cam is a subset of what we do on the website.
frist
die frist ist abgelaufen