Talking About Mellanox 100g
ฝัง
- เผยแพร่เมื่อ 29 ธ.ค. 2021
- **********************************
Thanks for watching our videos! If you want more, check us out online at the following places:
+ Website: level1techs.com/
+ Forums: forum.level1techs.com/
+ Store: store.level1techs.com/
+ Patreon: / level1
+ L1 Twitter: / level1techs
+ L1 Facebook: / level1techs
+ Wendell Twitter: / tekwendell
+ Ryan Twitter: / pgpryan
+ Krista Twitter: / kreestuh
+ Business Inquiries/Brand Integrations: Queries@level1techs.com
IMPORTANT Any email lacking “level1techs.com” should be ignored and immediately reported to Queries@level1techs.com. - วิทยาศาสตร์และเทคโนโลยี
![เปิดบ้าน เปิด Supercar อั๋นโอกิ เคยเอา rolls Royce เเลกพระ องค์ละ 100 ล้าน l [Nickynachat]](http://i.ytimg.com/vi/FeGwJwXauZ0/mqdefault.jpg)
People are afraid of 5g, and here you are handling 100g without any protective head gear!
But I like the warm buzzy feeling it gives me!
Well, 5g is survivable for some time for some people, not great, but not terrible either.
However 100G and once you cross 9g it starts to get rather dicey, it all depends for how long you decide to be subjected to all those gs...
Just stay away from the bathroom scales when you are subjected to them, because no diet can fix the number you see at that point.
@@CMDRSweeper LOL! You beat me to it!!! Good one!
Its only active once injected intravenously.
Hahaha, "protective head gear"!
Astronauts don't even pull 100G at liftoff! Wendell ftw!
ONIE is love, ONIE is life. Commodity switches are the future.
Word!
Not really, they are used mosty by hyperscalers so aws, azure, google cloud but still there isn't replacement for typical edge/core router (not layer 3 switches). ATT is trying to use that open computing model with DriveNets but it isn't really that popular, may end up using more power, space then chassis devices.
No
I love how cheap network hardware can get. You can pick up 40GB Mellanox switches under 200 and the NICs under 100. Great fun!
That would be awesome, and less than the cost of 10 gig hardware
I picked up a supermicro blade system with 4 blades each with 2x 40g Mellanox boards in them for $600 and then added the switch to it. Instant home cloud. Though at 1400w for the power supply it costs quite a bit to keep it running. lol
@@kenzieduckmoo It's kind of an odd situation. Everyone and their dog is looking for a faster than gigabit network and are laser focused on 10G. This has made the price of 10G stuff kinda high as used cards and equipment is drying up. But as nobody is really looking at 40G and used gear is getting more available it's almost at the point there 40G is just as cheap or cheaper.
I almost want to take a stab at the InfiniBand stuff. 16 Port 56Gb switches for $100 shipped. I just have no idea how to interface an InfiniBand network with Ethernet without spending stupid amounts on a "gateway"
I am starting to understand why I watch all these videos. They are just outside of my capacity to understand, but I just feel smarter listening to Wendell (I really hope I spelled your name correctly) talk about things!
Good news, looks like connection speed is slightly beating Moore's Law.
Based on the speed difference between the two you'd expect the old modem to have been made in 1964
Bad news(?), 10mbit ethernet was available in the early 80s. The comparison with the modem is unfair.
Did I just watch a kid in a candy store video?
Heads up, 400G is not 4x 100G, its 8x 50G. The switch to 400G changed the physical port from a QSFP28 (rated for 4x 25g) to QSFP-DD (rated for 8x 50G). Along with this physical change, the signal modulation was also changed from NRZ Modulation to PAM4 Modulation. You can buy a 400G optic that will combine 2x channels per lane for 4x 100G transport/breakout. But the originating channels are still 8x 50G
I thought that 400 Gbps was QSFP56, no?
@@ewenchan1239 QSFP56 is 200G (4x 50G lanes). You can run 400G with QSFP-DD (8x50G), OSFP (8x 50G) or QSFP112 (4x 100G).
Basically it all boils down to the number of physical lanes available to the hardware, and how fast the electronics can drive each lane.
FYI these “lanes” are electrical and directly connect to the ASICs. Pluggable optics only convert the electrical signaling into lasers and then back into electrical signaling.
If you want some fun nightime reading, pull up some MSA documents for each type and they will describe the physical standard of each optic type
For 400G: SR8, DR4, FR4, LR4 and ZR4 are all defined!
@@StupidFoxInSpace
Ahhh...okay.
My bad.
I mean QSFP112. (I hadn't read the Mellanox ConnectX7 documentation until now.)
But to Wendall's point though, you CAN get 400G using 4x100G if it is a QSFP112, which means that what Wendall said isn't necessarily incorrect, with respect to your comments above:
"Heads up, 400G is not 4x 100G, its 8x 50G."
and
"You can run 400G...QSFP112 (4x 100G)."
@@ewenchan1239 At the time of my original comment, QSFP112 had not been ratified and products using QSFP112 did not exist yet :) Apologies I didn’t clarify that in my response last night!
@@StupidFoxInSpace
Gotcha.
I wasn't sure about the timeline, but it looks like that the first public release of the QSFP-DD/QSFP-DD800/QSFP112 came out on May 20th, 2021 in Rev. 6.0 of the multi-source agreement, so your comment wasn't too far off from when it was published, and I would have to imagine that it was being worked on, behind-the-scenes.
They don't make em beep-boop like they used to though.
do I have to pay royalties for reusing the Mt. Stupid quote? That one deserves a T-shirt for sure.
I love this content, keep it comming!
From a Network Engineer point of view I would change the 100G port distribution, 2x100G for the inter-switch link and 1x100G for the Storage per switch, why? usually on this type of deployment MLAG is used and if that single 100G inter-switch goes down it will wreak havoc where the best scenario will be that a whole switch will shutdown all of its ports
400G is only available on QSFP-DD ports (which is actually 8x50G lanes, not 4x100). AFAIK, 100Gb single lane SERDES aren't commonly available, yet.
You also have OSFP ports that have much better cooling and have flexibility to do more, the number of lanes are getting a bit more complicated, as of today we are seeing as you said 50G lanes in the ASIC but when you get to the actual physical on the optics we see a gearbox that will transform the 8x50G to 4x100G and that's the most common implementation, so as always it depends :)
nice video, good editing, great content. thanks
Wendell, long time viewer since teksyndicate days and still a noob here. I am trying to dabble in making a clustered SAN lab but my head is spinning understanding Mellanox through iSCI, iSER, RDMA, Infiniband, iWarp. Would it be too much to create a video explaining the lay of the land?
If you're doing a SAN, you're going to be dealing with either fibre channel or iSCSI.
I don't even have 10gb cards to play with and Wendel has 100gb cards to play with :O
You can get 10/40gbe cards for under $30 atm.
I'd say the bigger cost depends on your setup.
If you have the devices close enough you can get cheap DAC cables but if you are doing distance you've got to deal with either Fiber or the high price of 10Gbase-T.
@@LanceThumping I think the problem is getting a 10g switch that you can use with you Ethernet wired house
@@LanceThumping Can you link a few of those 30$ cards? I looking to buy some.
@@SilenceGProd What's the problem? Switches are available, price may be problem.
@@SilenceGProd
MikroTik switches with several 10G ports is fairly inexpensive.
great scott! I have found my plateau! so, a great world 4 software processing!
I'm looking forward to replacing my 1GbE switch in the next year or so, but for now I'm fine with a single 10Gb link in my home network between my server & desktop. The two machines are only a couple meters apart so I got a pair of X520-DA1 cards from China (knockoffs, not genuine Intel) and connected those with a cheap DAC cable. A proper switch will be a nicer setup, although considering my main storage is spinning rust in raid6, I may not be in a rush to go beyond 10G. The switch I want is something like: 4 x SFP+, 8 x 2.5GbE (4 w/ PoE). I've been leaning toward Mikrotik but I'll consider Mellanox too.
This channel is so cool.
I'm glad you mentioned how slow the cloud can be, comparatively, even if it's on great hardware. Also cloud equipment is shared, with bandwidth split between customers.
One thing I have not tried is the "dedicated" cloud hardware options, when you pay extra for bare metal equipment. I wonder if you can also get dedicated links between all the hardware, since different machines host CPU, drives, etc. Back when I did cloud things, it just chugged (compared to a high-speed desktop) due to the shared resource model. I remember several blog posts about "cpu steal" where another tenant could steal cpu time and your VM would be slow. There were also several blog posts comparing speed between cloud providers, some with better Iops, etc.
Maybe this is largely solved now, I've been out of this space for a few years.
i think your proposal has been an option for quite a while. if you look at a host like OVH, they have a vRack. so you rent multiple servers from them, but then they can be placed in a virtual private network. i don't know the specifics of the backbones between racks/floors etc in their datacenters, but they push some heavy data.... so i don't think it would ever feel like those network connections are bottlenecking you beyond actual enterprise grade technology limitation anyways.
Firstly, your host needs to be in a rack where the Top Of Rack Switch is not oversubscribed. But if you get bare metal in an appropriately divided TOR, seems you would have full fat access to the network via sfp/qsfp depending on the host type
The problem with Mellanox 40 and 100Gbps NIC is actually availability. I got some quotes recently, and it is 2 months waiting list, plus need to sign no-resale agreement.
I'm old enough to remember my dad being elated to get 2400 baud running on his hobby system. Sometimes. It was hard to tell on the amber monitor.
Looking forward to that openshift video
so cool... iplay around with some 10gbit X3 still not easy to use all the bandwith on a standard NAS, even with ssd.
These numbers all sound incredible for a homelaber like me, but can imagine busy servers needing it. Love the videos :)
With ESOS, you could create your own DIY storage array and use it to present block-level storage to your devices to hold things like Steam libraries so you don't need to migrate an internal drive with every PC upgrade, you just change the path in the array config. 40Gb cards are becoming very inexpensive, though you do need to make sure that you get one that isn't Infiniband or is configurable as a NIC or an Infiniband HBA. I saw some for as little as $70 with AoC. Pretty cool how cheap high-speed networking equipment is getting these days. Hopefully 100Gb stuff drops this fast.
Servers are one thing, but the true consumers of newest networking gear are telco companies :P
I'm holding out for 1Tbps
I came across this random piece of hardware while browsing the web and decided to youtube it. Asking myself, who is this relevant to? I am now in a whole new rabbit hole. So cool even tho I don't understand much yet.
what are those beautiful panel screens in the background open to firefox?
Ahm, about the acoustic coupler is it possible you were talking about 300 baud, not bit/s?
Happy new year.
I'm curious if the 100G/40G cards behave like a single 10GB/s transfer or if they behave like a link aggregation of 4 * 25G/10G?
I'm looking for the fastest access for a single desktop to a single server for single file copies and access. So it's really 25gbe or 40gbe that are the two specs that are interesting and in budget.
That's not how LAG works.
No, a 100G link formed by 4*25G channels doesn't behave like a link aggregation of 4 links of 25G, so in your use case the 40G link will be better, but good luck reaching does speeds on a single TCP session data transfer, for large and fast data transfer is much better to use multiple streams
I understand that at the ends of the DAC cable there is an EEPROM and it says who I am. Question - does the NIC card ports reading the eeprom compare it to the eeprom on the other end of the cable?
I connected 2 DAC cables through a docking board (QSFP28 QSFP28) and can't get a connection between NIC cards
I don't understand what's wrong.
Did that open shift video ever come out? (mentioned toward end of video)
Just a small question... @ 6:07 th-cam.com/video/lAk89Id-5RU/w-d-xo.html it was supposed to be 12x SFP28 which is 12x 25GbE and it got mixed up ?
Used these cards for some time; most Mellanoxs are a good card for their custom SoC stack and ToE with DDPK, not sure why would you use them for run-of-the-mill setup!
LAN party over acoustic modem anyone?
Modem Wars anyone?
I had a dedicated server running on my LAN party days with two channels of 28.8k glory. But we didn't need to patch games or play videos over WAN, so bandwidth was ok for most things
Linus is drooling right now.
Yeah and harassing the shit out of his product manager to get this in their hands.
More TrueNAS SCALE video please. I am baning my head on 40Gb ConnectX-3 without RDMA.
9:00 Link is in fact not in the description ;)
Which server chassis model is shown in this video?
Is that a 300 baud modem from the 80s or from the 70s? :)
Heads up. Mellanox RDMA is not standards compliant. It doesn't work with any other vendors RoCE. It breaks the key tenant of Ethernet... standards, interoperability, and open.
This looks awesome! Anyone got a recommendation for a 24 port 1Gb switch with at least 4x 10GB ports that won’t break the bank?
there is a way to have a good bus connection between (4x NVIDIA 3080 TI) and (8x AMD 6700XT) using this technology to use in AI training ?
Hey Wendel, I take offense to that tomato fruit comment in that image. Tomatoes are totally fruit, and why is that useful to use as a classification? They ripen, even after detached to the plant, you can use them to make new tomato plants, it's awesome to learn about biology (you would not want to take wrong or the genitalia and leg of a person so why a plant?)
I have no idea what you are saying. I am a Desktop PC builder. This enterprise kit is so far beyond me. I used to connect to the MF via a 300 baud acoustic modem. Those were the days. I would love a series on the components of an enterprise solution. But keep it at a level so Krista's dog can understand it.
Cheers
So I have 3 servers, 2 are data storage servers exceeding 1250MB/s read/write, and the last server is for running VMs and pfSense. The VM server has a dual 100GbE NIC, with one connection to each server...any reason a switch would be better?
With JUST that setup, a swtich would give you some configuration options and outside access to the ressources aswell, but ig they only need to communicate together thats fine.
One storage server could in theory go over the pfsense to the other storage server. But if you think about scaling it up more and more, you should opt in for a switch, since just one pc dying could mean everything behind it is not acessible anymore. Ringbus stuff for example. Or just having way too much overhead since someone is talking to one server trough a third one etc.
@@TV4ELP makes sense. it's a homelab, it'll never grow more than this; I've moved my entire environment (my gaming/workstation computers) into the VM server, so there are no additional server I could ever need.
Dell was pushing 25Gb minimum connectivity four years ago. Now they're working toward having 100Gb as their new minimum standard. Given how their storage arrays are designed to function, with clusters of storage nodes aggregating high-performance NVMe flash RAID arrays, the array itself should be more than capable of providing multiple 100+Gbps data streams, sustained. I mean, the nodes in the clusters themselves are interconnected using no fewer than a pair of 200Gb infiniband connectors each, if not more. With the introduction of NVMe packetization for network transport protocols, the floodgates have really opened for high-speed networking well in excess of 10Gb.
If i look around, that switch cost around 3000-7000€ piece, depending if new or refurbished. All this onie switches are expensive as hell.
Wasn't the modem CPS and not bits?
We called them acoustic couplers
Can someone please educate me. 25gb are hundreds of dollars online. But eBay has a bunch of "40gb" cards for under $100. I see a bunch of postings for Mellanox MCX314A-BCCT for cheap. Is there some sort of catch?
it's a scam, I'd avoid ebay like the plague.
Cries in 1Gbps networking with 10Mbps internet (on good days)
small g, we don't wanna look like we're showing off
I would love to get that Dell switch...but not for $3000 off ebay....oh man scalpers are even in enterprise stuff...
That's bloody cheap.
Wendell, review the NVIDIA CONNECTX-7 that's 400Gb/s!!! That should be fun xD
But what about cx-6 cards?
I thought "100g" was 100 gramms. To warn the users how heavy the card is ;-)
The connectx-4 is obsolete with esxi8
nvidia bought Mellanox March 11, 2019
100g fiber isn't ethernet? It is unless you are talking about fiber channel, just like 10g, 40g or 25g SFP ports are just ethernet.
I believe Mellanox Connect-X cards use their own protocol, not ethernet.
Or they can switch between protocols? Not sure
@@popcorny007 What kind of protocol?
They have connect-x cards listed as ethernet, what would you connect them with? Infiniband is under seperate category, along with fibers channel you don't plug them into ethernet switches that were shown here.
Not sure about the newer gen cards or if the naming is still the same after the Nvidia buyout but connect-x3 cards could do Infiniband, Ethernet or both.
Base connect-x3 cards were Infiniband only
connect-x3 EN were Ethernet only
connect-x3 VPI were able to do both
Not sure what Wendel meant when he said that or if he just miss spoke because the switch he was talking about is an Ethernet switch.
I think he just Missspoke, it is typical to call Cat5/6/7 with RJ45 connectors Ethernet cables so I think what he meant to say that is not RJ, ethernet doesn't really care about the actual cable as it is a Layer 2 protocol
I think I'm halfway between Mount Stupid and the dip that is after Mount Stupid.
We only need 333,333,333 or so of those phone shoes to be equal. parrel is the future
And here I sit. Thought those two fibers dangling under my ceiling running 10Gig were fast...
ok completely unrelated but that BGM is basically if the mother series had to do "The Lick" in band class
All my Linux and BSD ISOs distributed to my close friends and family quickly in my network of fun?
It blows me away sometimes how persistent work has been using 95% 1Gbit.
Wendel your fired !!! Joking
I hope I found this video before. I was fighting with my dell switches to get some of the ports from 25 Gbits to 10 Gbits and I wasn't able until a friend told me about the 4 ports groups...
Nope 👎
Nobody gives a shit racist. Go away.
@@Stopinvadingmyhardware Can you elaborate your comment? Are you calling me racist?
@@javierthewish Made sure you were paying attention invaders
@@Stopinvadingmyhardware I live in Germany is it getting more invader/racist under your very confuse understanding of reality? Don't worry mister or miss fake account. There is still hope for you, there are professionals of Psychology out there that can be a very good help for your problem. Take care.
@@javierthewish
That’s why
QSFP112 Please.
Til, 2.5x higher bandwidth is orders of magnitude. :-p
Thank God I am an idiot and I can just think I am right all the time.
Pretty sure the DBA I have been working with thinks I am.
This makes my 5 port 10/100 BaseT unmanaged switch feel inadequate. :(
I’m so glad Wendell stopped correcting himself “mellanox I mean nvidia”
I thought he was talking about some drug, 100g doses
Who stold my first modem? Mom, hang up the phone, you kicked me off CompuServ... Little did I know then how far we would come. Now I understand why my dad was so amazed with personal computers like the C64. It was miles beyond a typewriter. Of course, there are very few places that have a 100Gb connection to the world, but like everything else, eventually someone will ask you why you're still stuck on slow af 100Gb to your house when its barely fast enough to surf the metaverse and certainly not in full 5 sense mode. Please bury me before we have metaverse fart jokes that you can smell!
5mbit connection via wire and maybe 15mbit with mobile. Hybrid router gud stuff
Dammnit, stop calling it linIx, its LinUx, there is a God damned U there in the end not and i.
Happy New year btw.
About chips shortage... 😅😂😂😂😂
7:26
One hundred billion vs 300. Needless to say, these aren't the spartans you're looking for...
Lol I can barely make use of 10gig as-is
this video would have saved me a day if I had it 5 months ago ;)
My first mofdem was a 300 bit per second one... I swear I could type faster . Jajaja
It's still almost impossible for single server to deliver more than 10 Gbps application layer throughput.
Over 10 Gbps is over kill for application server such as web server, database, etc.
Nobody wants to write an operating system? Then why does routerOS, HP Comware, etc. exist? Microsoft has an allegedly FOSS switch software named Sonic but MSFT software is always a trap. These Dell switches are cool but Nvidia now owns Cumulus and will not be supporting any Broadcom-based chipsets. These Dell switches are Broadcom chip devices.
In 40 years time someone will be making a video holding up a 100gbit card and laughing at it while they show their 1 petabit NIC. In 400 years time there will be no concept of data speeds as we know it. They will have NICs hardwired into their brains.
You watch, with all the r/selfhosting going on everywhere. Pretty soon you're going to see server racks and blade servers sold to end users for their IOT and micro services running at home. People will start buying up solar and house batteries for powering these beasts (And Texans for other reasons) Sys-admin skills will become part of the school curriculum, and the humble home desktop PC disappears as every house becomes it's own VPS, perhaps with redundant backup to family in other cities..
Were you drinking something? Home Desktop PCs already dissapeared, leaving laptops, tablets and smartphones, most people won't be running additional hardware maybe beside USB hdd/ssd for additional storage.
Lol, what are you smoking? most people barely know how to log in to their isp modem admin web gui
Drools ~~~~~~~~~~~ wags wags wags wags wags wags wags wags wags
Heh I remember b4 300 baud... and when 300 came out it was blistering!