wasn't his reason just he didn't have a system board for Eypc but did for Xeon for the Storage server so he used the Xeon system board. Now to keep thigs consistent they only use Xeons for the storage servers it seems.
@@yumri4 Gigabyte engineering sample of a motherboard likely to blame for their [buttload of PCI-E] drive disconnect/reconnect issues adding latency, I don't know to what extent Liquid hooked them up [referenced in their last server video]. I'm not sure why they [LTT] just didn't buy one [that wasn't an engineering sample] if Gigabytes AMD offering was as advertised, fully featured, and cheaper than an Intel board.
@@RNGwhydoihavetoregis from what I understand that engineering sample was essentially release hardware, it was just marked as an engineering sample, and they weren't going to go out and buy an entire new server if the one they already had was essentially the same thing.
I think including stories about bios bugs you've had to solve to get so much nvme accepted into one system would make the video cool as well. The adventures you go on are interesting to many viewers.
Interface may be 2x the speed on PCIe 5, but more to the story than that. Will need both NAND drives and NVMe controllers that can handle 2x the load as well, which is not likely to exist at launch when Zen 4 is released. But yeah, in a couple years 15m IOPS is going to seem trivial on high end datacenter equipment.
@@tommihommi1 that’s unlikely, vendors are still qualifying and that could take another year or so… given the wafer shortage and Covind that’s going to push it further plus the extra validation effort needed.
I posted some M.2 raid results dealing with a similar issue in the level one tech forms a few years ago. Six m.2 drives spilt over threadripper in numa mode. Memory speed used to impact the results significantly when it was tied to infiniband interconnects on early threadripper
Well he did get close to the IOPs of the system RAM. I think he said if you account for overhead he did get there. As Eypc and Threadripper Pro are basically the same chip i doubt any difference from 1 Eypc CPU and this but a little one from 2 Eypc CPUs and this. A little one as he is maxing out how many I/O ports he can use.
I would 100% recommend EVGA, had to RMA a 780 after a driver update from Nvidia bricked my card. Had been using it for 2+ years and they sent me a 970 replacement card which had better performance and more ram (well) lol
Hi Wendell, and thanks for another great video! Liked & subscribed. I’ll bet I speak for many when I say we’re dying to see a video about using the P5800X as the system drive for a Windows 10 Pro system 🙂
Running a BOINC setup for Einstein@home or milkyway@home on your Threadripper Pro, or your future dual EPYC Milan, to stress test it, at least in some ways.
Why not do a partner video and get 7-8 Honey badgers and fill them up with Samsung drives? That ought to smash the record. Or better yet, do this on a dual socket Epyc system and increase the number of NVMe drives even more. 20 million should be easily attainable on such a system and may even be more ‘real world’ than ‘mad scientist’ if it was configured as a massive NVMe NAS server.
Great video lots of useful unto for me to digest here! You should do some Chia plotting tests with that build and show us the results, would be really interesting to see how many plots that could pump out.
Fantastic achievement 👏👍 I initially thought oh yea he's gonna bottleneck either on kernel limitations or on the Infinity Fabric with such bandwidths I was wrong ! But in a satisfying way .. This is great How exceptional can the AMD/Linux combo be ?
I'd really like to see more of how the testing process works honestly. I keep hearing the results of these tests, but could you go more in depth on the testing process. What programs are used, how many different programs running at the same time, etc. I find that really interesting.
When I was working at a Start-up called WebTV back in 1996 we had a Network of Sun Microsystems servers and some NetApp appliances and filers. I remember the engineers talking about how many I/Ops the the Netapp appliances could perform with there large disk platers. I don’t remember the exact number but it was less than 5 i/ops. I am sure your Threadripper build can beat it. But we had a Nation wide network of over 3000 WebTV users all on this Netapp appliance database.
I want to see how close to this I can get with my x299 system. I mean, nowhere near, but since 3.0 hardware is cheaper, and I have less lanes and cores and channels, it'll be cheaper to max out.
Would it help to setup scheduling to maximize the time between requests per device using mirrored drives? Then do a round robbin to maximize the time until you request the next block from each drive (adjusting for core and device latency) giving it time to complete the request without flooding it. Another thought is, the threadripper chipset has 8 PCI lanes as well. Might it help to use those in addition to the IO die to offload some load from the IO die? They will be slower but if you can get the timings adjusted, increased performance might more than make up for it since you would basically have 2 IO dies. When you look at the latency timing per core in the threadripper there is a big difference between the closest and furthest core, so you can save yourself all polling requests that fall within that latency because the original request hasn't even gotten there. You could also monitor request times to completion and start polling slow and increase it as it becomes more likely to be done. This should reduce the load on the cores by quite a bit without any loss in performance.
Watching this has reminded me of the weeks of pain trying to get AMD RAIDXpert and Windows 10 to let me boot from a NVME RAID 0 array. Spoiler alert, I'm booting from a single drive and the NVME array is the D drive because of Windows.
What was the name of the tool you used at 10:41? LS Toppo? I can totally see myself having a pcie config optimization problem in the near future. Thanks a ton!
How much overhead will be created by PCIe bifurcation hardware is someone will try to super-optimize the available PCIe lanes? I would imagine that PCIe4 drives are not using 100% of the provided lanes for them 100% of the time
"There is nowhere else in the solar system as suitable as the Earth, so it is not clear if we would survive if the Earth was made unfit for habitation. To ensure our long-term survival, we need to reach the stars. That will take much longer. Let's hope we can last until then."
We do have them the issue is QA is bad for PCIe 4 riser cables. I think it is due to the speed of PCIe 4 is above what their testing program can measure so it looks the same as PCIe 3. On paper the cable should be the same but in reality it isn't. I think that is in part why the riser cable issue isn't getting fixed as quickly as we would want it to. To this try convincing someone that that 1/32 trillionth of a second is important. In computers it is as that is the difference from PCIe 3 and PCIe 4 for data send or receive. You know it is getting insane when you start talking about trillonths of a second as something important but it is made for workstations and servers that seconds can add up to days.
Is there any 7 GPU waterblock arrangement for threadripper pro such that i can fill all 7 pcie4.0 x16 slots with 7 RX 6900 XTs for a render farm and can still have decent temps without any throttling?
Those are dual slot cards isn't it?. You would need to install half of them outside by using riser cables and external enclosures, also adding external PSUs. Theoretically possible but sounds too expensive and complicated to maintain. Better use two systems,, if you pay for 7 top class GPUs the cost of the second system is peanuts (as long as each one have a theradeipper with half number of cores and half RAM)
pro is a whole other series of threadripper chips aimed at professionals. Its a totally different socket too. AMD will probly be putting out both regular and pro in the future.
@@tommihommi1 Thanks. I have a 3960 and the only real slight annoyance is I have to restart it every couple of weeks. I figured ECC might help that but if its not supported then not really an option.
I only imagine an all in one CPU with internal storage/memory been the perfect scenario for max IPS. In this small scenario I also imagine a phone having good chances to beat more complex systems on this task.
The problem with using IOPS as a measurement for performance is it doesn't represent the real world. Yes, I can buy an all flash SAN that can do 100 million IOPS but the second I enable RAID the performance tanks. Unless you have a completely reproducible workload that you just need to crunch and discard we need to protect our data, even in RAID 10 there would be an impact. Can you imagine kicking off a 10TB workload that takes 5 days to process in RAID 0 then losing an SSD, that's the express lane to unemployment.
Remember when Linus couldn't build a storage server with Epyc? Pepperidge farms remembers
wasn't his reason just he didn't have a system board for Eypc but did for Xeon for the Storage server so he used the Xeon system board. Now to keep thigs consistent they only use Xeons for the storage servers it seems.
@@yumri4 Gigabyte engineering sample of a motherboard likely to blame for their [buttload of PCI-E] drive disconnect/reconnect issues adding latency, I don't know to what extent Liquid hooked them up [referenced in their last server video].
I'm not sure why they [LTT] just didn't buy one [that wasn't an engineering sample] if Gigabytes AMD offering was as advertised, fully featured, and cheaper than an Intel board.
@@RNGwhydoihavetoregis from what I understand that engineering sample was essentially release hardware, it was just marked as an engineering sample, and they weren't going to go out and buy an entire new server if the one they already had was essentially the same thing.
Wendell and Linus are not the same
I think including stories about bios bugs you've had to solve to get so much nvme accepted into one system would make the video cool as well. The adventures you go on are interesting to many viewers.
Next year with PCIe 5/CXL: DOUBLE ALL THE NUMBERS!
Interface may be 2x the speed on PCIe 5, but more to the story than that. Will need both NAND drives and NVMe controllers that can handle 2x the load as well, which is not likely to exist at launch when Zen 4 is released.
But yeah, in a couple years 15m IOPS is going to seem trivial on high end datacenter equipment.
@@benjaminlynch9958 Yep, a P5800X on PCIe 3.0 would have higher IOPS than any NAND based PCIe 4.0 SSD.
pci-5 is still a few years aways, the validation and tools arnt even finished yet.
@@kleanthisgroutides7100 the first servers will be on the market in 6 months
@@tommihommi1 that’s unlikely, vendors are still qualifying and that could take another year or so… given the wafer shortage and Covind that’s going to push it further plus the extra validation effort needed.
love you're work thanks for all the time you give the you tube community my home lab is almost off the ground learning more every day thanks again
I posted some M.2 raid results dealing with a similar issue in the level one tech forms a few years ago. Six m.2 drives spilt over threadripper in numa mode. Memory speed used to impact the results significantly when it was tied to infiniband interconnects on early threadripper
I suggest connecting the deguneranker to the electro-clavinator, then connect both to the gonkulator. Your system will perform way better.
Thanks for the lstopo tip. The output doesn't look as magnificent on my everyman's system, but it's still good to be able to see what's going on.
You're GO for mad science, more of this please!
That's a lot of IOPS.
And a fascinating technical discussion as always! Even though I'm not nearly intelligent enough to follow.
Love your videos Sir. Keep up the good content.
Tell me your farming Chia, without telling me your farming Chia.
-how many plots you can do a day Wendell?
-yes
I don't understand it all, but I'm a simple man: I see mad science, I press like
okay, mad scientist stuff kinda nutty
_engagement_
Well he did get close to the IOPs of the system RAM. I think he said if you account for overhead he did get there. As Eypc and Threadripper Pro are basically the same chip i doubt any difference from 1 Eypc CPU and this but a little one from 2 Eypc CPUs and this. A little one as he is maxing out how many I/O ports he can use.
I would 100% recommend EVGA, had to RMA a 780 after a driver update from Nvidia bricked my card. Had been using it for 2+ years and they sent me a 970 replacement card which had better performance and more ram (well) lol
More of such hardware videos please :)
How long to get into 1 billion IOPS? 10 years? 20 years?
Absolutely love mad scientist level 1 and u don't even gotta ask about the napkin math level 1 lol
I see what you did there Editor Amber!
Support mad science? Well I'm sold
Wendell, you really need a monocle and fluffy white cat.. 🤣🤣
Hi Wendell, and thanks for another great video! Liked & subscribed. I’ll bet I speak for many when I say we’re dying to see a video about using the P5800X as the system drive for a Windows 10 Pro system 🙂
When shopblt ships me one I'll tell you how it goes. Have a 905p right now, looking for the upgrade!
Your using 4 generations of the best server drives and roiling your eyes.😎you’re in a deserved position 👍
I enjoy your content
@04:04 - that's the ASUS 4 x NVMe card I bought after watching one of your other videos...
@14:40 yes, some are a bit sketchy...
Pssst.. Wendel.. we're at this camera! ;)
Running a BOINC setup for Einstein@home or milkyway@home on your Threadripper Pro, or your future dual EPYC Milan, to stress test it, at least in some ways.
Why not do a partner video and get 7-8 Honey badgers and fill them up with Samsung drives? That ought to smash the record. Or better yet, do this on a dual socket Epyc system and increase the number of NVMe drives even more. 20 million should be easily attainable on such a system and may even be more ‘real world’ than ‘mad scientist’ if it was configured as a massive NVMe NAS server.
spoiler: the epyc version of this video is SUPER NEAT :D
@@Level1Techs noice
/me sees all those PCIe lanes and weeps in anguish.
yeah, not just that, but rom bar space and.. things like interrupts.. all those things would make the whole pcie gpu on rpi easier.... something :D
@@Level1Techs haha, I'd take any one of those things; you get 'em all!
YEP
Great video lots of useful unto for me to digest here! You should do some Chia plotting tests with that build and show us the results, would be really interesting to see how many plots that could pump out.
I love these mad science videos
Is this your new Chia farming rig?
I was thinking the exact same thing...
Chia wasnt even a thing when I started this project!
not after cryptocalypse lol
@@Level1Techs Then you should start to test chia. Would love to see big numbers ;)
@@niklasp.5847 Yeah it would be fun, der8auer was at 1w for maybe something
Liqid needs to send you a Badger Den like they did Linus 🤣
I might be the lone crazy out here, but I wish in 2021 we could do a L1T open house.
I've been in those queues for pretty much since release. No email yet.
Fantastic achievement 👏👍
I initially thought oh yea he's gonna bottleneck either on kernel limitations or on the Infinity Fabric with such bandwidths
I was wrong ! But in a satisfying way .. This is great
How exceptional can the AMD/Linux combo be ?
I'd really like to see more of how the testing process works honestly. I keep hearing the results of these tests, but could you go more in depth on the testing process. What programs are used, how many different programs running at the same time, etc. I find that really interesting.
Oh he wasn't kidding about the world record thing...
When I was working at a Start-up called WebTV back in 1996 we had a Network of Sun Microsystems servers and some NetApp appliances and filers. I remember the engineers talking about how many I/Ops the the Netapp appliances could perform with there large disk platers.
I don’t remember the exact number but it was less than 5 i/ops. I am sure your Threadripper build can beat it.
But we had a Nation wide network of over 3000 WebTV users all on this Netapp appliance database.
Marvell is bringing something new to the PC game. The Marvell Bravera SC5 offers 2M IOPS and 14GBps in a PCIe Gen5 SSD
"But......we can do it" them very words gotten me into some trouble over the years
Engagement challe.... 9:00 THE TIDE PODS MAKE A REAPPEARANCE !
Wendell you need to send me some of that hardware before i drown in my own drool
uhm, I think the wrong snippet was cut into 12:56, since it still shows 8 million iops. 13:40 shows the narrated 13 million though.
@17:40 is that 'hollywood' or 'wallstreet' top left :-)
Insane
twenty MILLION iops *evil pinky*
Absolute mad lad!
Boiler Snake Merch!
4:50 it's called milchmädchenrechnung xD
Jawoll!
More mad science videos!!!
Could you point me to the command you used with FIO? I would like to load test the machine I am building for my Rippled Nodes and Validator. :)
When are we going to get HBM2e integrated with the CPU?
joy
Can you talk about DPDK? Would that help in this case?
I want to see how close to this I can get with my x299 system. I mean, nowhere near, but since 3.0 hardware is cheaper, and I have less lanes and cores and channels, it'll be cheaper to max out.
Would it help to setup scheduling to maximize the time between requests per device using mirrored drives? Then do a round robbin to maximize the time until you request the next block from each drive (adjusting for core and device latency) giving it time to complete the request without flooding it.
Another thought is, the threadripper chipset has 8 PCI lanes as well. Might it help to use those in addition to the IO die to offload some load from the IO die? They will be slower but if you can get the timings adjusted, increased performance might more than make up for it since you would basically have 2 IO dies.
When you look at the latency timing per core in the threadripper there is a big difference between the closest and furthest core, so you can save yourself all polling requests that fall within that latency because the original request hasn't even gotten there. You could also monitor request times to completion and start polling slow and increase it as it becomes more likely to be done. This should reduce the load on the cores by quite a bit without any loss in performance.
Watching this has reminded me of the weeks of pain trying to get AMD RAIDXpert and Windows 10 to let me boot from a NVME RAID 0 array. Spoiler alert, I'm booting from a single drive and the NVME array is the D drive because of Windows.
fiber channel controler?
How about ddr4 disk drive....?
Would having fewer cores make a difference?
I would like see a crystal diskmark run or better still how long does it take to copy 40 Gib of 4Kib files.
What was the name of the tool you used at 10:41? LS Toppo? I can totally see myself having a pcie config optimization problem in the near future. Thanks a ton!
How do I hook up the P5800X to the Threadripper board ?
That p5800x is soooo expensive for penniless noobs
How much overhead will be created by PCIe bifurcation hardware is someone will try to super-optimize the available PCIe lanes?
I would imagine that PCIe4 drives are not using 100% of the provided lanes for them 100% of the time
"There is nowhere else in the solar system as suitable as the Earth, so it is not clear if we would survive if the Earth was made unfit for habitation. To ensure our long-term survival, we need to reach the stars. That will take much longer. Let's hope we can last until then."
p5800x review when?
It would be faster if you had the RGBs...
Wendell, are you planning on doing a standalone review of the p5800X?
its almost ready! its so fast!
@@Level1Techs nice!
@@Level1Techs Can't wait! Waiting for shopblt to ship me mine.
What's the GNOME extension at 16:50?
You might want to change out that 16x RTX video card and save some lanes by using a cheap 1x video card.
Why not use the aspeed 2500 onboard vga then?
raid 0?
How long until we can reasonably expect true PCIe 4.0 riser cables?
We do have them the issue is QA is bad for PCIe 4 riser cables. I think it is due to the speed of PCIe 4 is above what their testing program can measure so it looks the same as PCIe 3. On paper the cable should be the same but in reality it isn't. I think that is in part why the riser cable issue isn't getting fixed as quickly as we would want it to. To this try convincing someone that that 1/32 trillionth of a second is important. In computers it is as that is the difference from PCIe 3 and PCIe 4 for data send or receive.
You know it is getting insane when you start talking about trillonths of a second as something important but it is made for workstations and servers that seconds can add up to days.
oems are buying all that can be made now. the big missing piece is QA stuff. Some 'solutions' I've got use redrivers with dip switches. Not pretty.
Is there any 7 GPU waterblock arrangement for threadripper pro such that i can fill all 7 pcie4.0 x16 slots with 7 RX 6900 XTs for a render farm and can still have decent temps without any throttling?
Those are dual slot cards isn't it?. You would need to install half of them outside by using riser cables and external enclosures, also adding external PSUs. Theoretically possible but sounds too expensive and complicated to maintain. Better use two systems,, if you pay for 7 top class GPUs the cost of the second system is peanuts (as long as each one have a theradeipper with half number of cores and half RAM)
iop bedrock @14:20
4:39 Lisa Su is going to be *very disappointed* in you. Y U not have her photo next to Jensen Huang???
My Floppies do not scale linear ...
Is threadripper pro the next threadripper? its confusing to me. I thought a new series of trx-40 chips was coming out?
pro is a whole other series of threadripper chips aimed at professionals. Its a totally different socket too. AMD will probly be putting out both regular and pro in the future.
@@chainingsolid Do you know if regular threadripper has full support for ECC memory?
Threadripper Pro is Epyc with a chipset and higher power limit.
Non-Pro threadripper doesn't support regECC.
@@tommihommi1 Thanks. I have a 3960 and the only real slight annoyance is I have to restart it every couple of weeks. I figured ECC might help that but if its not supported then not really an option.
@@skarscalp8157 ECC on UDIMMs is supported, just not regECC and LRDIMMs.
Numa Numa?
I only imagine an all in one CPU with internal storage/memory been the perfect scenario for max IPS.
In this small scenario I also imagine a phone having good chances to beat more complex systems on this task.
So in a nutshell SSD are going to be graded like Ram Dimm's.
More Optane...
So... how would (or could) you use this monstrosity of I/O to feed something besides itself?
Now put FreeBSD on it and show what you can do with ZFS on it
That was ONLY A 3090 BOX - hahaha
adata xpg gammix s70 .......blazing drive ..........somebody please go indepth on this........
I really want to see a threadripper pro system with 7 water cooled 3090’s since they can fit into single slot
The beatings will continue until the IOPS improve.
Hi Wendell I sent you a couple of messages on patreon a couple of weeks ago, I know you get busy any chance of a reply please
EVGA B-STOCK products do not qualify for ELITE Membership
Oh, okay then.
boooo
The problem with using IOPS as a measurement for performance is it doesn't represent the real world. Yes, I can buy an all flash SAN that can do 100 million IOPS but the second I enable RAID the performance tanks. Unless you have a completely reproducible workload that you just need to crunch and discard we need to protect our data, even in RAID 10 there would be an impact. Can you imagine kicking off a 10TB workload that takes 5 days to process in RAID 0 then losing an SSD, that's the express lane to unemployment.
Ok how about you get an office chair company to make an L1 gaming chair
Ryan get the Rogue Assault Bike you been putting it off for too long and it won't break straight away like the cheaper ones!
Imagine the number of chia plots you can make per day...
I might cheat a little and throw an FPGA in there.
Can you stop advertising asrock now?
They're shady and hated by those in the know and you folks deserve better.
And that's why the IT guy shouldn't be allowed to play with things
Why are tech guys always overweight? At least this one seems to have a shower more often than monthly. It's something.
Just get 16or32 Gigabyte 7GB/s AORUS 7000 Premise M.2 nVME's. They are triple stacked with fins and pipes and will Never Throttle. RAID zero'em and 😃.
Throughput is good. but need iops :)
The craziest thing is in about five years you’ll be able to pick one of these up for $250 and they’ll be as common as cockroaches.