ฝัง
- เผยแพร่เมื่อ 9 มิ.ย. 2024
- 100G Networking Part 1 - • 100 Gig Networking in ...
Mikrotik 100G Switch - amzn.to/4d5pFkG
Connectx5 - amzn.to/3Wau6om
-------------------------------------------------------------------------------------------
🛒 Amazon Shop - www.amazon.com/shop/raidowl
👕 Merch - www.raidowlstore.com
🔥 Check out today's best deals from Newegg: howl.me/clshD8fv8xj
-------------------------------------------------------------------------------------------
Join the Discord: / discord
Become a Channel Member!
/ @raidowl
Support the channel on:
Patreon - / raidowl
Discord - bit.ly/3J53xYs
Paypal - bit.ly/3Fcrs5V
My Hardware:
Intel 13900k - amzn.to/3Z6CGSY
Samsung 980 2TB - amzn.to/3myEa85
Logitech G513 - amzn.to/3sPS6yv
Logitech G703 - shop-links.co/cgVV8GQizYq
WD Ultrastar 12TB - amzn.to/3EvOPXc
My Studio Equipment:
Sony FX3 - shop-links.co/cgVV8HHF3mX / amzn.to/3qq4Jxl
Sony 24mm 1.4 GM -
Tascam DR-40x Audio Recorder - shop-links.co/cgVV8G3Xt0e
Rode NTG4+ Mic - amzn.to/3JuElLs
Atmos NinjaV - amzn.to/3Hi0ue1
Godox SL150 Light - amzn.to/3Es0Qg3
links.hostowl.net/
0:00 Intro
0:51 Fixing my issues from before
1:08 PCIe lanes issue
2:09 Fixing the slow file transfer speeds
3:46 Testing our speeds...we need more
4:33 Now that's fast
5:12 Real world file transfer speeds
5:39 My thoughts on the results
6:45 Conclusion - วิทยาศาสตร์และเทคโนโลยี
100gbps... I could move my meme folder in... well still minutes but less minutes 😂
So thats how they call porn these days? "Meme"?
@@jesus2621 whoa, that's disrespectful... I don't need to download porn 🤣
@@SpoonHurler you may not need to download it but you still have to store it
Cant wait for Part 3 🫡
My gigabit connection is feeling unloved right now.
My 100 Megabit switch is feeling unloved
Hey Raid Owl I have to say this- I LOVE your videos. You explain the issues you run into extreme well and I really appreciate how you walk through and troubleshoot your issues. I think it's awesome that you listen to the feedback of viewers and react accordingly. The last video you had on this everyone said "time to upgrade your hardware!" and you saw that, thought about it, and did it your way while explaining why you made the decisions you did. You're extremely relatable- you do things for the fun of experimenting with it and explain your mistakes and/or lack of knowledge with humility. We all benefit and learn with you because of it, thank you. So often you'll see content creators who curate their content to show them in the best light- the smarted most capable people who never make mistakes and never show the issues they encounter even if there were millions. Of course there isn't anything wrong with presenting concise content like that- a 10 hour video of troubleshooting wouldn't be all that entertaining- but I feel you personally have a fantastic mix of explaining issues while still being concise.
Just want to say again I really appreciate how you create your content. I personally have a home lab setup very similar to yours. I am considering the pros and cons of the same exact switch you bought. I am curious about the same levels of performance that you exactly are trying to achieve. I'm gaining a lot from your work and I just have to say again- appreciate you so much! Keep it up!
BTW if you upgrade from your MikroTik switch help me out and sell me yours! haha. Joking but maybe not really lol.
Also REALLY funny comedy in this video. You had me laughing as the Threadripper slowly faded in. Amazing. I would LOVE to see you do this workstation build as that's also something I've been curious about doing. Let's see it!
Best of luck and looking forward to more videos!
This is as fast as me clicking the notification of a Raid Owl video
Nothing is that fast
@@RaidOwl Neural network for the win!
Wait… My 10Gb router and switches with optical links are not the fastest ones anymore ? Dang I feel so low
Time to upgrade...you NEED to
Kind of crazy to think that 800gb is starting to roll out when so many home users are running 1/2.5 and maybe 10.
Same here
Great stuff, great storytelling, love it!!
If you’re looking for more pcie lanes for a more reasonable price consider an Epyc system with a gen 2 or 3 processor. The motherboards start around $500. I managed to get a 7352 24c for $200 on eBay. Then you can get all the gen 4 lanes you could need! Alex from the Self Hosted Podcast has a good video on this.
I actually mentioned something similar in the part 1 video, this seems like a great place to try out Epyc Siena if you can actuaklly find one of the motherboards in stock.
NICE!!!!! Love it man !!
would be interesting to deepdive into the Truenas performace stuff, disable compression etc, windows server is such a license hell
Thanks Bro... always solid
I am glad you are upgrading your setup. It is kinda weird how for the longest time we have always been told that you don't need faster than gigabit because your WAN connection is a lot slower than that anyway. Now I have 10G internet at my home and am actually needing to upgrade my home network because most of the devices on the network don't actually support that. I kinda feel like your home LAN should be at least 2-3x what your wan connection is which means I need to seriously do some upgrades. And do I need those speeds? Sure don't but I do try and utilize it as best as I can. I have a tor node setup and I have my own nas and server setup to account for as well. I am really hopeful that now that Gigabit and faster speeds are becoming more common that we will finally move on from the idea that gigabit is fast enough. I would love to see SMPTE 2110 proliferate for virtual systems and I do think that it is getting easier and easier to just run your own homelab.
youre Vids are the best
Hello Raid Owl! Nice follow-up video. You mentioned the use of FEC cl74 on the 25gbps uplink in part 1. Is FEC set on the qsfp ports? FEC reserves part of the bandwidth for error correction. Cl74 allows 74% of 100gbps to be used by the end device. You might be able to get a bit more throughput if cl91 works or if you can turn FEC off. Would recommend researching how FEC works.
I have 74 set on the 25g ports and 91 on the 100. Thanks for the info!
@@RaidOwl - Cool. Cl91 seems like a decent middle ground if people want FEC enabled.
One. Point. Twenty-one JIGGABITS!
I have a feeling I'm going to see part three of this saga,... and.. with a happy ending....lol
great timing, I am currently upgrading to 100G (/25G) as well 🙂
Just bought a CRS510 which will hopefully arrive soon, my Server has a 100G NIC and my Workstation is currently at 40G...
I already benchmarked it using a DAC cable and found out that my CPU and RAM are limiting the speed to ~26Gbit/s, but it is still good enough for my use case (I will have other clients and servers using 10/25G connections)
Depends on whether you're using ethernet protocol or Infiniband.
Really cool
the outro allways hilarious
0:00 "I have a problem..." finding a shirt that didn't come from the bottom of the pile of (clean?) laundry. Jkjkjk, I'm just jealous that I'm not your "full 100G connection with RDMA that actually gets 100G speeds." Fun video. Nice work, thanks!
great video, did you try to transfer files with robocopy (mulitreaded file copy standaard in windows program)?
Nope! I’ll try that though
I had a similar issue w/McSpecial fw'd nics for my e810s. I have the -cqda2, which is 100G total. You'd want to get the -2cqda2 to be able to use both ports at 100G. After my little failure w/a couple Dell cards, claimed to be Intel, but had to return because nothing I tried would get the fw to update. Seems even these would be cheaper than actual Nvidia ConnectX-6 dx cards.
Ah cool we tried in VM and with RAM disks. I was curious about those. In the end, it's often more about available lanes/lane speed, storage speeds, and even sometimes CPU power. Also files are WEIRD. If there are bunch of little files instead of one big one, we can lose out even on a high speed connection with CONSTANT fluctuations.
I am curious if those higher speeds you found can be used to run back up pushes from an active server. Maybe in a full server environment, you can at least increase the rate at which you can back up your configurations.
Always a fan of what’s coming from RaidOwl!
Awesome video! What happens if you copy 5 files at the same time? Also I’m going to get you one of my cards so you can see if this is a drive os queuing issue.
Yeah part 3 of this video is going to involve trying different file transfer processes for sure 💪🏼
When running SSDs/NVMEs in a pool under Windows you need to remember to run through a lot of hoops to get TRIM to work.
I found that out the hard way my self.
"I can fix anything. I just pull out my checkbook"-- Raid Owl
Very true
Seeing those speeds is crazy to me, meanwhile I am happy with my 2.5Gb transfer between my desktop and my server.
i just migrated 80TB of data abandoned NTFS in favor of TrueNas ZFS, but... smb and nfs(better than smb) is just unsable for any amount of high IO application. too much overhead, the only workaround is iscsi.... but puts me back to NTFS... are there any workaround to utilize rdma /smb direct with truenas zfs pools?
0:01 sure you do.
Have you tested just the direct connection, without the second port connected to the switch?
This was direct connection
Here's the thing, the main goal of 100gbps is not to get the best transfer speeds possible, the goal is to help with things like network congestion. It's about getting non-blocking throughput when you have thousands of clients connected on the network at the same time. The ultra high-speeds are mostly meant for the network. Your analysis is correct that you will always be dealing with overhead, and this overhead only seems to increase as network speeds go up. Using RDMA with SMB Direct is basically the best possible scenario in terms of overhead, as you already described. What it is useful for though, is being able to handle ceph, live VM migrations, internet traffic and file transfers at the same time and having it all run smoothly. If you want to achieve this properly you are going to have to seriously rethink your network topology though to create a true fabric infrastructure, since you want to make sure your traffic is non-blocking.
Yeah VM storage and Ceph is the end goal
"Here's the thing, the main goal of 100gbps is not to get the best transfer speeds possible, the goal is to help with things like network congestion."
That REALLY depends on your use case.
For 100 Gbps Infiniband, it's actually both the bandwidth (bandwidth * channels) as much as it is reducing the point-to-point latency (which is inversely proportional to bandwidth for a given clock speed) as much as possible.
There's a reason why Infiniband ruled (and still rules) the Top500 supercomputers as the system interconnect method.
"What it is useful for though, is being able to handle ceph, live VM migrations"
Depends on whether you're talking about Ceph synchronisation or getting data into and/or out of a Ceph OSD.
For live VM migrations, if the VMs (and/or CTs) live on shared storage anyways (which it ABSOLUTELY SHOULD), then there is no need to actually move the data around (because it is already sitting on shared storage). Therefore; IF it is already sitting on shared storage, then the only thing that you're migrating is the state, which can be completed in seconds even with GbE. (I do that right now with my three N95 Mini PCs.)
"since you want to make sure your traffic is non-blocking."
Not a critical requirement.
You CAN get very high data transfer rates even if you have blocking traffic.
You could join me on the crazy train of “Let’s use AMD EPYC as a workstation”. A 16c/32t 73f3 around $1k + motherboard for $600 + 128GB DDR4 ECC 3200 for another $300… $50-80 for a tower cooler and you’re all good for less than just the 7965WX w/ cash to spare. I essentially have a 5950x3D w/ 8 channel Memory, it’s stupid and fun and I love it.
SMB is such a frustration.. you have bonding two Inc’s , your simple smb method (nice to learn new tricks). But why after getting so much working well is the file transfer speed still Middle Ages?
No jumbo frame mentions?!
Smb and samba are a real bottleneck for file transfert in my small setup also.
Money solves all problems 😊
interesting vid, just a guess:
1) you might be limited by CPU actually, because there's tons of processing handling networking, storage, and the transfer process itself
maybe you'll need to configure CPU affinity to dedicate specific CPU cores to specific tasks ? (it's very common practice on workstations)
2) you might be limited by Windows Explorer doing the file transfer
maybe try something else like Total Commander, or Free File Sync ?
When using tc make sure the setting is too use own copy method vs the windows one iirc
....Or just use massive size local nvme and then automaticly syncthing backup over while you are not using computer...
Thanks for confirming!! i went through a similar journey. I switched from 10G to 100G and had all running with x8 and x16.
It was a Rabbitbhole to get to at least 3GB/s with SMB while i expected lime you to have 7 to 10GB/s.
It was so complex and I wonder,, are we the only ones??? There is almost no good info out there and I think we are at the forefront of Homelab.
Because honestly this isn't how 100G is meant to be used, but that doesnt mean it's not fun to tinker with anyway.
@@nadtz i don't think its really the network. The 100G network does it's job. its the file protocols that are not optimized for this use case.
But if you want to build a Live Storage for Processing... it's just hard.
@@jensunger6560 400GbE and 800GbE (and infiniband) require PCIE 5/6. It's the whole reason PCIE 6 and 7 have been ratified so quickly. 100GbE is certainly fine for a lot of things but AI/ML, HPC and hyperscalers are already using 400 and NVidia at the least already has 800 capable equipment.
@@jensunger6560 The file protocols aren't optimized for the use case because like I said it's not how 100G is generally used. You would typically have 25G devices connecting to 100G switches. It's the whole reason 100G can be broken out to 4x 25G connections in addition to 100G > 100G.
RoCE requires not only nic support but switch support as well to make 100 gigs possible as switch will make proper QoS and pass packets faster. On some conferences some ppl claimed that dpdk is also need but I am not sure if this is True... Also if you are using RoCEv1 instead of RoCEv2 you should get slighty faster speeds.
Intel with their iWARP make claims that iWARP needs only nic support without switch but never had chance to test it out.
100 Gig ?? I mean love it don't get me wrong ...but w the hella are you planning to do with 100 Gb??😳 and such a big switch too !? You definetely turned me curious
Idk move my meme folder back and forth
SMB is always gonna be slow. One of the reasons why I use NFS with TrueNAS Core.
Only the 24 core? C'mon... in for a penny, in for a... lot more pennys
So why do you approach this from a 1 to 1 prospective? Why are you not testing with multiple devices transferring data as is intended?
Find out next time on Dragon Ball Z
It would be interesting to see how fast a WireGuard connection would be on a setup like that!
Part 3: NFS 😂
While grabbing that Threadripper 79xx series will do you well, you might also find you don't need to go that far. One of the big problems of the Z790 chipset is that even in comparison to Desktop AMD, your number of PCI-E lanes available is limited. Desktop AMD (Ryzen) 44 total PCIE vs. Intel (36) using Z790 vs 670E. But more beneficial in many use cases is that on many readily available Threadripper 5xxx series and 39xx series, you have more than enough PCI-E to push through a solid 100Gb. That said, I don't like the 87Gb/s. I'm not sure what is on both ends, but you absolutely should get better than that. There is definitely something not hitting target if a benchmark to real world isn't a payoff.
I mean yeah it’s more for fun cuz I’ve always wanted to build a threadripper system.
@@RaidOwl Then do it. I have several running; in my homelab using a 5975wx.. I just can't justify grabbing a 7xxx upgrade right now for lab use, especially since I picked up the 5975wx at near free ($250 for the rack it was in at liquidation). which came with the Asrock Rack WRX80D8-2T. Been very happy with all slots being PCIE-4x16. That said, if you are willing to wait until the fall, and you want to total max your bandwidth, look at the new Threadripper and grab a Highpoint 1608A for an 8x NVME AIC card. Should be a monster. Add 8 PCI5 drives.. damn. I have one client who I could see doing that. Right now, though, I have enough room for everything I need.
@@icky7835 Is there a liquidation sale notifications signup that you'd be Ok sharing? That sounds incredible.
maybe you should test with SMB Multichannel
I think that SMB Direct uses multichannel by default
@@RaidOwl for multichanneling you need two paths between client and server.
tried that as well. didn't help to saturate 50G or 100G
When you streaming again buddy
Hmmm maybe next week
@@RaidOwl can’t wait
Swap out Windows Storage Pools with StableBit Drivepool/Scanner
It will make for a fun test but not sure it will be faster
Maybe robocopy is something to look into for file transfer
But honey, your Plex videos will stream so much faster now!
So using Adaptec 8 Series raid controllers... My off the cuff speed estimations are 100MB per drive, 15 drives R6 x8 PCIe.... depend on file size/type/content slower (50G of PDFs suck) or faster (80GB video).... on average I can move 30TB
i think you need to goto 100g bonded from ws to nas then the bottleneck will be nvme? you cards are dual port already from the looks - go for it and slay the performance bottlenecks once and for all - people are genuinely interested and this trend is not going away - plenty of people will want ws/nas ultra fast connections - maybe you can sync to a dual nas somehow too - have it sync in the background over 56 g or something
👍🏻👍🏻👍🏻👍🏻👍🏻
I wonder if you dumped Windows and SMB how much faster it would just be without dumping money into jumping through hoops.
For me, I get that you increased your speed. But what is the purpose of having such high bandwidths at home? I'm confused. I use a lot of bandwidth, and 10G would be more than enough.
Nothing
@@RaidOwl Not that I am against playing with it and figuring it out, but I could never use 100G at home. Yes, I could do a backup of my system to a network location, but I also just do diffs throughout the week; and, it's there almost instantly without sending 250GB over the network. But, again, I love your content. I just didn't know if you were figuring it out for work or something like that.
Nah just trying stuff out and learning. I think 25g is probably the max anyone needs in a home lab. 10G is the sweet spot for 99% of people.
@@RaidOwl And here lies the rub. Really need to be a network engineer AND programmer to figure this stuff out.
Come on, spend the money!😊
What is your firewall? Ie in your network dies 100gbs goes through it? Layer 3 asics may also help another, especially compared to the weak mikrotik. Gotta play wierd games to not be limited by thier stuff. However w layer 3 switch, not having authoritative dhcp may be an issue, but setting up proper acls will tear your hair out. Wait....sorry no offense
You know you don't exactly need gen4 x8 for a gpu right?
Not many options though
Great video 👍
Really helpful insights into 100Gbs.
Am here for the sequels.
Except now you is t have ecc memory.
What are still waiting... bite the bullet and buy the threadripper system.
Soon bby
Oh the wives. They really have absolute;t no clue what we’re up to in our homelands or if the latest delivery is a $1,000 or a $10 part….except when new servers arrive. Then they know we’re getting a little speedy. I usually monitor its process and then ask her to go to the grocery store and find some apparel the kids need for camping or hiking and ask her to pick it up while I’m taking the server up and racking it. Mwahahahaha
And then tell the wife when she asks is this new? "Oh no, it's been here for awhile, don't you remember?"
@@Darkk6969 You must have a very observant wife! Mine twigs it when I need her nails to peel the adhesive off the drive labels from the label maker. 😁
$600 for a NIC
damn
I am, by no means, an expert in this field, so the information I'm about to provide is just theoretical based on other information I see. Your real-world speed might be bottled neck by the processing power of your CPU... *Went to look up the LTT video where I saw this information. Watched the whole video and came back to this. Hit play, forgetting the whole reason that I went to look at the LTT video, and the next words that came out of your mouth are about getting a thread ripper. Oh yeah... I have a half typed comment that I haven't submitted yet...*
LOL
Before your wife introduces your nether regions to a 5 iron (realistically she's probably trying to save money so cheaper to ruin a goodwill wooden driver) consider an Epyc Milan used enterprise setup or a used Epyc Genoa setup but you'll have to get her a Gucci bag first with Genoa.
With Milan You get 128 PCI-E lanes PER CPU installed, PCIE 4.0, and a lot of boards have 7 PCIE 4.0 X16 slots. You get access to ECC DDR4 3200 and up so mountains of memory is cheap. Core counts top out at 64 for Milan which is fine it only boosts to nearly 4GHz all core on a 64 core proc psshhh slow right? Many of the boards support bifurcation so you can run a whole bunch of U.2 HBAs / PCIE switches for truly face melting storage speeds if you really want to go there. Or you can be pedestrian and throw in 3 8 port SAS24 HBAs with a total of 24 1.6TB 12G SAS SSDs? Threadripper is cool, undoubtedly. But there is something about looking at a flashy corvette that only costs $130,000 from a $1.6M excavator that builds car factories which only cost you 61K at a gov auction and earns your company a corvette every time you book a job. Take a look at Milan.
Spend an extra 2k and still not be able to do basic stuff. Homelab stuff is getting very frustrating. Nas is supposed to be fast.
Crying on 1gbit ethernet 😢😢😢😢😢😢😢😢😢😢😢😢😢
WHERES THE THREADRIPPER AT BRETT WHERES THE THREADRIPPER.
YOU KNOW YOU HAVE BAD SPENDING HABITS BUT YOU HAVE THE INSnANITY TO DO IT. FOR THE TH-cam HOMIES BRETT FOR THE HOMIES
But why didn't you want to spend 6k on a Threadripper system...You're already too far gone with the 100GbE stuff. Also, a better option would be a second hand older Epyc system instead of a brand new Threadripper one.
Great video I love all the acronyms. SMB1.0 for ever!!!!!!
"If you are still watching you are SMB 1.0", so old, slow, underperforming and extremely insecure?
Jokes aside this whole thing is very interesting and in a decade or two my Homelab might actually be built out with those speeds (I just got 10G-Base-T for my NAS and a desktop switch)
Stop handling peripheral cards like a noob. You gonna leave cheeto stains on the copper leads!! 😂😂😂😂
Yeah makes them taste better
@@RaidOwl 😂😂😂😂😂
Why u so greedy?😆😆😆😆😆😆
you must have unlimited monetary resources for you to constantly keep buying this highly expensive equipment.. It's cute.. But not really relevant to most of the homelabers assuming that is one of your main target audiences.
You think I'm cute?!?!?!
It does help homelabbers spend less knowing the issues they may or may not run into.