I don't even care what the subject of this video was. It was so well presented, with so much clarity and insight. Thank you for putting this out there. I'm kidding, I really do care about high speed networking.
Just published part 3 of 10Gb Home Network - Build A Switch. In this video we wrap the 10Gb Home Network series with a really fun project to build a very powerful and usable home brew 10Gb network switch. All the magic is embedded in the network operating system. I invested easily over 100 hours of designing, building, troubleshooting, and testing in this project (just part 3) so I stand by the results. Now you can build one using my recipe. I hope you enjoy the end result!
+Narwaro Awesome! Thanks for the kudos. I completely agree with you about needing to upload more regularly. I've been trying to think of ways to reduce the time it takes to make a video. On top of that I'm a one man operation and I have a full-time job. So it's extremely difficult to knock videos out on a routine basis, as this could be a full time job easily. I'm going to figure something out and strike a balance. That's my goal. Good luck with your project. That's how great ideas are born!
I know how that is like, unfortunately. Only put the amount of time in to TH-cam that you can afford! (I have actually been into networking quite a lot recently because Im redoing my server room and all my cables with fibre and CAT7, and I am also one of the first ones to get fibre at work, fk yeah)
I tried building a DIY 10gb Network Switch a year ago, and everywhere I went people said it wouldn't work. I got it working in a dell PowerEdge 2950 but it would only do like 500MB in one card and out another. I ended up buying a 10gb switch (crazy expensive). Glad you figured it out. Now you have to build a raid0 fileserver with ssd's. I get 2000MB read and write on mine. Loading games over 10Gb is faster than my 2 ssd raid0 on my gaming machine because over the network the game loads straight into RAM. =)
+Levi Blaney that's really interesting. Thanks for sharing Levi. Now that the 10Gb infrastructure is in place, I'm kicking around a few ideas on how to leverage it with fast storage in a fun way. Out of curiosity, what OS are you using for your RAID0 setup?
iTechStorm Try not to laugh... I'm using windows 10. This is because I built the "Ultimate" HTPC. It's used as a media center, and to game in the living room and to be my 10Gb file server. I was going to use my PE2950 but it's just not the right setup for a 10Gb file server. A Enterprise Server with PCIx 3.0 is extremely expensive but a consumer motherboard with two 16x and one 8x is only $200 or so. With the R9 290, the H700 raid card and the Melanox 10gb NIC I needed all the fast slots I could get. I still have a server cabinet full of servers for VM's and stuff, and the PE2950 has 12TB's raid10 for backups and VM's.
iTechStorm I would really like to see you make a RAM disk on two different machines and copy a file like a 4gb or 2gb movie back and fourth. I really don't think the switch you built will hold up. Mine couldn't in my tests. The issue is SMB has more overhead then just raw IP packets. This tends to translate into a much greater latency when going through the PCI bus, this causes throttling and lower throughput. With all the testing I did. I was only able to get 600-750Mb's. Still much better then 125MB's though. I bought a 10Gb switch because I already had invested in the Raid0 SSD file server.
OMG I LOVE YOU! Am having a new home built, and decided that I wanted to upgrade to 10GigE and start putting everything on NAS. My god is this helpful!
+Jennifer Royal awesome!! Checkout my other video for the network closet setup. I did the all my cable runs after my new home was framed. I'd recommend no less than Cat6 cable. If you can afford Cat7 that is a better option for future proofing. I purchased a 500 foot spool of bulk media cable which was a dream to run. 2x Cat5e and 2x RG6 bundled into a single cable. Then faceplates are easy. 2x2 faceplate with 4 keystone jacks. One of my favorite upgrades to my house.
Wow this was very useful for me, thank you very much for this very well explained video. I would like to know if on this switch I can create vlans. I'm going to build one of these and would like to use vlans in my vmware projects. It is possible ? Thanks in advance for your help.
I was waiting for this since P2 release, and put the plans to buy a 10g switch on hold until i see what comes in P3.. Certainly i wasn't disappointed, great effort and very organized and informative video. i will give it a try since i already purchased 4 Mellanox ConnectX-2 cards that i will use in a similar build. I would like to thank you for the dedicated effort and wish that you will keep posting such quality materials when ever it becomes available. Regards,
+Essam Mohsin thanks for your kind words. I'm glad I could help with your decision. One thing I forgot to mention in the video is that most 10Gb switches are very noisy. That can cause problems for a lot of people. One more thing to consider I guess. Good luck with the build. I had a lot of fun with this project, and I'm glad I could share in a meaningful way.
Awsome 10G guide series. I am currently upgrading my home-labs on 10G with new FreeNAS as my VM storage. I was debating what to about the 10G switch since they are sooo expense - VyOS looks like its gonna do great. Ubiquity runs a fork of Vayata on their Edge Switches and Routers and they run superb. Good job and thanks for all the info provided in the description!
This was a project I truly believed in, and I wanted it to reflect in my work. I easily invested over 100 hours of designing, building, troubleshooting, and testing. I truly appreciate that everyone is enjoying it! Thanks for the feedback.
Really great series. Well made! Really liked the way you explained everything and why you made certain choices. Vyos really reminds me of working on Juniper network gear. Thanks!
Until seeing this video, I would never have thought of this as an option. I have two "segments" in my home network. My entertainment room which has mine and my wife's desktops plus our entertainment center. And out in our living room is our router along with the NAS and a virtualization system. I want to connect the entertainment room to the router using 10Gb or faster. And I'd been trying to find hardware to give me what I want, but now that I've seen this video, I know how to proceed. And with this option I can start small with getting the wide bandwidth (10Gb or better) set up first and then add everything else, likely including putting 10Gb Ethernet cards in mine and my wife's systems. I've seen switches that have Gigabit connections with a faster uplink (typically SFP) but that always seemed a little limiting since each client would be limited to only Gigabit. This idea of a custom switch, though, as you said, allows you to meet specific requirements, and change things as the requirements change. I definitely have more research to do regarding the available options. I've considered 10GbE and Fibre Channel for the link between the living room and the entertainment room depending on what I can find for what price. And I still plan to do 10GbE in the NAS -- which will likely be the first move since it's backward compatible to GbE. But overall it means two switches: one for the entertainment room to the living room, then another main switch in the living room to include the router (provided by my ISP, so I can't change it out or I would with a pfSense box). Out of curiosity, have you tried combining wireless into your custom switch using a USB or PCI-Express Wi-Fi adapter? If so, how easy was that to set up?
+brandishwar excellent! I'm happy to hear you are getting a plan together for your 10Gb upgrade. I love hearing about different configurations and how they are used. Unfortunately I have not tried connecting a wifi device to the switch yet. But I'm certain I remember seeing wifi functionality built into VyOS, so I would say your chances are pretty good. If you get a chance to test it out before I do, make sure you post back with an update! I would love to know how it works out.
+Thomas White I've heard you can find a cheaper Quanta at times. But aren't those switches noisy? I actually forgot to mention this switch build is quiet in comparison.
iTechStorm They can be pretty noisy. Mine isn't horrible but I have heard some that are like jets. I wish mine was more quite but that's something I can get over for the price.
I've been using a steam library over SMB on 1G network from FreeNAS - most games are fine and only certain ones hang for 20 seconds when loading. I imagine you'd have no problem using 10G network.
+Donald Coleman thanks for the feedback! Glad you are enjoying the content. I wouldn't put so much effort into my videos if I didn't believe they were going to resonate with people.
Awesome idea, I was actually toying with it myself but was thinking of RouterOS from Mikrotik, since I'm more of a Mikrotik fan. I've used Ubiquiti in the past though and they are coming with some awesome products! Their new EdgeSwitch 16 XG would be a competitor (around 600$) for anyone who is trying to build this from scratch I think. But if you have some of the parts already (like an older PC as you mentioned) this is really a killer way to build a cheap 10Gbit switch! The video and tutorial is very clear, great work! :D
With that motherboard, memory, NICs combination and considering the power draw, you can buy a 16 port 10g netgear switch. I've seen them on ebay for $1300.00. I use VYOS extensively in our company of 300+ servers, and it is rock solid.
+jkljkljkljkljkljkl I don't disagree with what you said. But at the same time I never said you had to use the same hardware as I did. I mention in the series you should size your hardware to your own requirements. Some people could even use hardware they have sitting around. At the end of the day, it's just an alternative to typical vendor solutions. That's all.
Hiiii, this is honestly the best tutorial I've ever seen. such a comprehensive and detailed video. thank you sooo much. however, I'm sad that this video was 6 years ago, hopefully you are doing well and nothing happened.
Very nice! I just bought Quanta LB6M for 230.00 from unix suplus. CRAZY CHEAP! 24 ports of 10Gb goodness. If you don't need that many ports there are a ton of Dell Force10 switches out there for under 100.00 like S50-10-GE-48T-AC which has 4 10Gbit and 48 1Gbit ports. I needed 8 ports of 10Gbit or i would have considered something like this. Also, if you are on Windows 2012 or better and want maximum throughput on SMB shares you will need a card that supports SMB Direct/Multichannel via RDMA the mellanox connectx-2 cards don't have this but I haven't found it to be a huge bottleneck. Lastly, the cost of cables is kind of crazy too. For long runs I'm using fiber and sfp transevers which for long runs is WAY cheaper than trying to get a DAC cable. for short stuff DAC is the way to go.
You sure the the Force10 S50 switches support 10Gb via the front 4 ports though? I thought those were only SFP ports (not SFP+). I thought only the rear expansion modules supported 10Gb (via XFP modules) and only for switch stacking.
It's funny how 4 years later that server hardware he used in the video is probably more expensive than buying a Ubiquity XG 16 switch. It is not as flexible for sure, but more compact and damn that nice UI :D
My question would be is it really worth it to build your own compare to an already built switch? I'm looking at it between the price point of a homelab and a datacenter. it's awesome to build your own network switch and all but not sure other then a costly of cards as well as motherboard with a lot of lanes on it.
Can you comment on the cost aspect? What parts did you select and how do they compare to a currently available commercial switches? For example I know that capable of the shelf switch could cost around $1000.
Well, it's never easy to throw out one price fits all comments. So on one end if you already have a system laying around (doesn't need to be anything special), and you can pickup a dual port card on E-Bay for $50... that's your cost. On the other side of the equation if you have to build a system from scratch, your price tag is going to be a lot higher. The best part is that your Network Operating System is no cost. Beautiful!
Great job on the video! I DON'T think you sound like a text-to-speech engine. LOL I definitely like the options you are presenting here. I like the fact that the home builder or average techie can stick it to the companies which, up until recently, held the monopoly on this kind of networking.
+Tony Carballo maybe I'll include a text to speech segment in the next video, lol. Well 10Gb is certainly overdue for the power-user segment, especially when you look at what has been going on with data transfer rates internal to PC/game systems.
One issue. Putting together this hardware does approch the cost of purchasing a Netgear 8 or 12 port 10Gbe Switch... Tho it does have more versatility in terms of using copper/fiber plus 40Gbe cards could also be used I suppose..
Exactly. Although I want to stress the versatility is also in using different combinations of hardware (cheaper). I never stressed the use of these particular components in the video. My point was, you get what you pay for. Thanks for the feedback!
Netgear sucks! I would never buy a Netgear for home, heck even if someone gave me a brand new 10GB Netgear, I would sell it on eBay and use the cash to buy the parts to make the switch in this video. Also, you get the feature of layer 3 in the switch build here, which is very helpful if you have a wifi network and you want to route to a DMZ. For what it's worth if you were to buy a cisco or brocade version of what he is building here; go ahead and take out your checkbook and write out something close to 5K and then we can talk cost and if this build is worth it.
The 10Gb hardware has also dropped quite a bit since he made these videos. Since your comment, they have even dropped further! It is now cheap enough for the average consumer to consider buying 10Gb Ethernet for normal home use! (May 2018)
LogicProHacks recently had a data centre job that involved managed services for some of the customers. Interspersed amongst them were a few who had chosen Netgear prosafe 10g switches for the storage network of their virtualisation setups. At first I was mortified as I’d previously had less than stellar experiences with people using Netgear equipment in production. In the end though we had almost no issues with them at all. The key with them seems to be the same as when you come across Mikrotiks, Ubiquiti, and other lower tier gear, keep them patched and keep the config simple. I won’t vouch for any other of Netgear’s product line, but the 10g prosafes were good enough for a home or soho deployment. The actual biggest pain in the arse with them is they are web managed only so you are reduced to clicking stuff in a GUI like some sort of chump.
I subscribed to the channel so I would get notified when this went up. Thank you so much for your effort on this! Great Work! The illustration looks like a SupperMicro system board and you said that you would have more information below, but I don't see any additional info.
Hi there. great video! Only issue i have is im trying to do this on a desktop based i5 PC. The only option in the bios that is remotely similar to those advanced options you specified is enabling Virtualization technology. Are those options really necessary? You sounded full of conviction so im assuming they are important... but how important is my question? lol.
Great video! Thanks! Wondering why you didn’t virtualize the Vyos install via hypervisor. This way your system is not just dedicated for 10g switching but can be used for other purposes. Finally, your rational for doing a custom 10g switch was to save money on enterprise 10g switch but then you end up dedicating your whole system (super micro mb, 4x10g nics etc) for switching. May be I missed something or just misunderstood your part 3 video. Regardless, the idea & the presentation is great & useful. Thanks again.
I didn't see about how to set up the card's configuration. Perhaps I just missed it. It was late when I was viewing this. I had been fighting a consistent no-carrier issue. I finally went in and changed the config setting in my dual-port Qlogic card from SH (what ever in the heck that is) to NL-partitioned. after a reboot (required because the firmware hung during the "save" command, which made me a bit nervous) and BING - now we have carrier and the first MAC on each port now have an IP addresses assigned by the DHCP on the PFsense box. Now each port has 4 sub-ports and each sub-port has it's own MAC address. Sadly, CentOS 8 still marks all the sub-ports other than the first as unavailable. Although I haven't looked yet, I'm betting it will be the dreaded no-carrier again. It appears I will still have to set up routes from each system to the other 4 as even after getting an IP, I am unable to ping a neighbor or the gateway on that network. So not really working yet, but at least some progress was made. I had swapped SFP+ adapters on both ends of the fiber and also changed the fiber itself from one set to another (OM3, LC-LC, 3 meter).
Great series, thanks! Ever since I bought a NAS (yes, a very simple out-of-the-box NAS with a 1Gb port) I have been wondering: now that home (out-of-the box) NAS solutions become more and more common, why is there not more focus on improving home network speed - and NAS ethernet cards for that matter - so that we can really go towards centralized storage? Of course, I quickly found out that 10Gb equipment is rediculously expensive. Then again, this is no doubt partly caused by the fact that the market is pretty much enterprise-only. So, as long as the manufacturers don't make the switch (pun intended :P) towards 10Gb networking, at least we have your great explainations of how to get it up and running 'on a budget' :) I do have two questions though. First, I love to play around with software and hardware, but I also know I'm not a security expert. So, I'm wondering, is there anything we should know in terms of keeping our home network secure with a home-brow switch like this? I mean, strictly speaking, I think it's routers that are 'responsible' for ensuring safety - and there would still be a router between such a 10Gb home network and the internet. But then again, you also mention firewalls and stuff when talking about the Linux distros. In your opinion, should we pay any attention to security when it comes to building a home-brew switch, and if so, in what aspects? My second question relates to speed. Sure, a network has awesome throughput rates, and you would definitely want to do some low-level testing like you did to see if everything is set up correctly. However, I also saw somebody else asking the question whether he/she could host a Steam library on his/her NAS and run it from there. Now, I understand that you have not tested this particular thing, but to really test if 10Gb network could 'solve' the fast, centralized storage problem, one would also need low latency. Have you done any latency tests? Or, maybe even better, some storage benchmarks that mimic real-life use? I would be extremely interested in seeing some of those results, but as you mention, building a network like this is not just a couple of hours (nor a couple of $$ for that matter, even with the self-buid :P).
Thanks for your comment. I'll do my best to answer as concisely as possible. In your first paragraph you mentioned the storage centralization trend and why it's important for the home network infrastructure to keep up. You are exactly right, and that is largely what motivated me to make this series. I felt there was a significant void in this segment, so I offered a possible solution. To answer your first question security should always be a concern with every device you configure. There is always going to be a balance you have to strike between functionality and security controls. Using best practices is always recommended, but not always easy to obtain. Everyone's configuration will be a little different and suffer different vulnerabilities. So the idea is to minimize your attack surface, but turning off unused serviced, inherently blocking all traffic and only allowing valid traffic through rules. Use a strong/proven boundary protection device and avoid using it for anything else (minimize attack surface). In the case of the home-brew switch, don't enable anything you don't need, use VLANS to segment different types of traffic (mgmt lan, storage traffic, media traffic, guest traffic, etc.), use strong passwords, disable default accounts, and patch when vulnerabilities are identified. For your second question latency testing was mostly out of scope for this series. But I understand why you are interested. Consider the context of "gaming network latency" versus "block storage network latency". When talking about gaming latency, a 120ms RTT (Round Trip Time) would not be favorable at all. While a 40ms RTT would be considered a pretty good network latency for gaming over a cable modem (WAN). But when it comes to block storage network latency every professional will tell you those times are a total deal-breaker. Centralized (network attached storage) is not quite as fast as locally attached storage (ALL THINGS BEING EQUAL), but these differences are measured in sub-milliseconds. And that is why I use the "context" example. I mentioned DAC (Direct Attach Copper SFP+) is faster than 10GBase-T from a latency perspective in my first video, and that is why you see more people using it for enterprise storage. So we are talking about microseconds versus milliseconds here. So what I'm trying to say in a long confusing way is that for the purposes of gaming don't confuse the two. A home user would never notice the difference. Just focus on building/buying a decent performing NAS, use a 10Gb connection, and have a workstation that is capable of handling the heavy 10Gb traffic stream. Does that answer your question :D
Yes, that indeed answers my question, thanks so much for your elaborate answers! I was by the way thinking about block storage network latency - the Steam library may have been a somewhat unfortunate and confusing example. I merely chose this example because loading a game, or a level, is a use case involving loading many small files, rather than one large file as would be the case for multimedia applications. To justify storing game installations on a centralized storage, I would at least require that loading times would not increase tóó much compared to internal storage - hence my question :) Regarding the security: I think I understand your answer, but as soon as you mention 'use VLANS to segment different types of traffic', I find out why for me, a home-brew switch might be a bridge too far. I can image why it would be wise to separate traffic like that, but would not have thought of this myself, nor would I know how to implement that. I guess I'll keep hoping that manufacturers will see the light and start producing 10Gb for consumers as well... And if it takes too long, I may give it a go myself after all. At least, I have a good guide now, thanks again! :)
+Caspar v. L. A few more parting words from your follow-up. It is expensive and requires a degree of planing to pull off a central storage solution that is larger and faster than local storage. Don't let my security example scare you away. None of that is required to build a healthy 10Gb switch. Just an example of paranoid security suggestions.
I'd say that most home users who want to delve into 10Gb stuff could easily afford a gigabit switch that has options to add a few 10Gb ports. I opted for the Dell 6224 with a 2-port SFP+ card that plugs into the back of it. It has another slot to add another card. I can have 24x 1Gb copper ports, or 20x 1Gb copper and 4x 1Gb SFP ports on just the front of the thing. Since 2016, switch prices have gotten saner. It is a shame that the majority of 10Gb switches out there are still 10GBase-T.
Well, it's really difficult to answer your question directly but generally speaking my intent was for people to use a system they had sitting on a shelf that could accommodate a few PCIe cards. Purchase the 10Gb cards for $18-$50 each (ebay), and grab a Direct Attach Copper cable for $18. If you have to purchase system components your costs are going to go up from there obviously. Everyone's cost is going to be different. I used a system I purchased for a SAN build, but just about anything with decent power and bandwidth will work.
iTechStorm v0.62 I misspoke earlier. It isn't keyboard but Window > Translation > Remote character set on the screen PuTTY presents when connecting. The default says something like ISO8859 (Latin, Western) or something similar. When I changed this setting to UTF-8 when logging into Debian via SSH, the characters I was getting on the screen that represented lines were replaced with actual lines. Be sure to select UTF-8 for linux distributions in the PuTTY configuration settings before connection and you're golden. :)
Hi folks great video. After watched this video I have made a VYOS switch with my tiny HP T5740 PC. I have installed one dual port intel 10G fiber card. I have connected it by fiber. Can you tell me how how to see optical power reading on this ?
+yggdrasil viking there really wasn't much to see. Just network cards stacked next to each other, but I did show that in the video. I wanted to keep the video focused and on track.
this entire series talks about 10gb ethernet - not fiber channel. cheap 4gb fc cards like the garbage you can grab for 1$ a piece from emulex are not ethernetcards. they are designed to work in a san enviroment, not an ethernet enviroment.
iTechStorm you do know that FC as in Fibre Channel is a completely different protocol compared to ethernet. 4gb fc hbas cant be used for ethernet over a network. they are made to connect to fc devices or in a fabric enviroment of a sanswitch/san
there are converters that you can buy. I pulled these out of my old netapp san and i have a 4gb switch that i dont use. I like the idea and the possibility of building a switch. thats all
you do know of RFC2625, as in IP over Fibre Channel (IPoFC) right? RFC2625 has been in place since June 1999 too. Never say never Zishy. www.ietf.org/rfc/rfc2625.txt "FC is standardized under American National Standard for Information Systems of the National Committee for Information Technology Standards (NCITS) and has specified a number of documents describing its protocols, operations, and services."
One of the most useful and thoroughly explained videos out there on Vyos. I was able to build a virtualized switch in ESXi. It was a little more involved setting the MTU of each vSwitch, port group, and VM, but was not far from what you were accomplishing in the video. However, I was not able to find out how to install iperf on Vyos. Any tips? wget keeps giving http/ftp errors despite there being a clear internet connection.
nice video! but would it work with your bog standard consumer grade motherboard +cpu combo? without all the fancy IOAT and DCA...etc options in BIOS? Sounds like a nice project to undertake if possible.
+Steve Lee really good question! My approach was aimed at achieving an "optimal" configuration. I wanted to show it was possible to reach those speeds at home. You can certainly achieve a great result even without the optimal tech (IOAT, DCA, etc). Everyone's mileage will be different because the performance is highly dependent on the type of hardware you are using and the configuration. If you can get 2-4 Gbps, that's already a massive improvement over 1 Gbps. So beauty is in the eye of the beholder :)
Really good and detailed information. It'll be quite a jump from my Edgerouter X attempting this. I'm guessing combining this with a SG-2220 outbound to a modem is ideal. I hope you update the NAS/SAN build, as a new one can exploit this network (plus Xeon D and Atom integrated boards make the builds much less complicated).
Thanks Quick! I have an Edgerouter X too, and you'll be pleased to know they pretty much ran with Vyatta-Core and forked a slightly different direction from VyOS. I actually enjoy using VyOS a lot more since it's easier to use from the command line with features like abbreviated commands. Anyway, that's a really good question about the SG-2220, since I haven't seen where anyone else has tested it with VyOS. It probably works great if it's using standard chip sets. And on the topic of the NAS/SAN build, I agree the Xeon/MB/Mem should work quite nicely with a 10Gb connection XD
I just want to add at 9:44, this is only true for PCIe NICs. It does not blink for onboard NIC and you cannot re-arrange the onboard NIC if you have multiple ones.
This is a great video. Thanks! I might be trying this out. Big question: Can I get respectable performance from cheaper hardware with low port counts?... like if I dug through my computer graveyard and put 2 or 3 of these $99 ASUS copper cards inside?
you should set mtu to bridge too. logically bridge is in front of NIC but in reality it is in middle You receive on NIC, which is sent to bridge and then it traverses bridge, then nic and back to bridge and that is on both sending and receiving. Actually it is even more complicated having low mtu on bridge may severe high mtu connections remember mtu is not part of IP (biggest IP packet is 65535, period) mtu is my for EtherNet frames (including MAC - source and destination and other bytes I never can remember meaning of 0x0800 - total 14 bytes)
great video, and great work, i am installing Vyos too and it seems it has no support for my Mellanox NICs do you have any idea or litterature , on driver install solutions / scripts thanks a lot !
Question: How does IP assignment work with bridging all devices? Can you still utilize your router's DHCP function to assign IP addresses to devices connected through the bridged connections? Does this automatically obtain IP's or do you need to assign them seperately? Can you manually assign IP's as well? Like you I attempted to connected a 10gig server and Work station via PFsense and I got sub par results in transfer speeds... surprisingly even worse than 1gig. I'd really like to make an affordable 10gig switch that I can expand in the future.
Thanks for your question Eric. Just like any other conventional switch the bridged ports will pass a DHCP request through just fine. In my demo, I assigned a static IP address to the VyOS switch itself. This was for the purpose of easy management, but normally a basic/unmanaged switch would not have an IP assigned. I'm pretty sure you could configure the VyOS interface to request an IP dynamically too. So you should be just fine.
There are several reasons for this. When companies change out their older servers for newer models they typically sell to the highest bidder so they don’t have to deal with parting things out. The winning bidder then sells everything salvageable on eBay. These SFP+ cards are very common in server hardware and get replaced by newer models just like everything else out there.
Thanks for the reply, that makes sense. Obviously this is a 'how long is a piece of string' kind of question but are the 2nd hand cards prone to failing?
If i buy a 10gb network card, can i use spf dac and aoc sr cables? Fiber optic cables can be used along the dac Cooper cables? Or Just dac cables works with network Card, do i need something to run aoc fiber optic cables, can i use bot dac and aoc cables on the network card?
You should be able to use any of those methods you mention. It’s important to check for compatibility of the SFP module to your NIC or brand of switch. Keep that in mind when selecting your medium.
Dude, this is like a $1000 system. Do you have something that might be just a bit cheaper like using some stuff around the house or left over parts, how about using an old i3 4150 or something like that? Like if I had the money to buy this I wouldn't be looking to build one bro.
Is it better to use x86 and x86-64 processors for this kind of stuff or would it better to use OEM switch ASICs? Also, do you have to build a bridge in order for the ports to be able to communicate with each other? Are bridges the same as link aggregation?
Keep in mind this is just a DIY project that allows you to use commodity hardware you might have laying around. ASICs are better for specialized functions like packet processing of course. You need to bridge ports so they can communicate at layer 2. You could route at layer 3 without the need to bridge, but that is beyond the scope of this video. No, bridges are not the same as link aggregation.
@@Itechstorm So, there are a few reasons why I asked the questions I asked: I've recently acquired 4x EDR Infiniband cards (dual port VPI Mellanox ConnectX-4 100 Gbps cards) and also a 36-port 4x EDR IB (all QSFP28 ports) externally managed switch as well. My NAS systems have dual 10 GbE ports (SFP+) and then I have, of course, "conventional" RJ45 CAT5/6 gigabit ethernet switches as well in my home. So the idea was I wasn't sure I really wanted to get breakout cables to be able to step between the speeds (and admittedly, I'm not even entirely sure whether VyOS would work with the Mellanox Infiniband driver, although with it being Debian based, it might be able to work) or if I would build something like this and this type of a system would be the kind of custom switch that would be able to handle the different step downs. For example, a single 4x EDR IB connection can step down to 10x 10 GbE connection or I would make the NAS systems (which run on 10 GbE SFP+) to be made available and accessible to my systems that are running the 4x EDR IB interconnect via the IB switch. Apparently Mellanox has differentiated their products such that the 100 GbE switches are separate from their IB switches and that the ports on the IB switch AREN'T VPI ports such that you can configure them to run in either IB mode or ETH mode like you can with their cards (shame). And I didn't want to have to pay for an entirely different 36-port 100 GbE switch either (or any of their smaller offerings) just to be able to run 100 GbE as well. (Although interestingly enough, cost per port, works out close to be about the same.) Qnap makes a 10G/1G 12-port switch. Mellanox makes their 100GbE switches. And I wanted to be able to step up/down between the speeds (and also my existing gigabit ethernet network as well), so I didn't know how well a x86 or x86-64 processor would be able to handle this type of a switching workload compared to the switch ASICs found in an actually Mellanox 100 GbE switch (for example). (Of course, nothing would be able to do a multi-step up/down like this, but that's part of the question.) And then, as you mentioned, there are other considerations as well. For me to be able to switch between 100 Gbps IB to 10 GbE to 1 GbE (and back), the IB card alone takes up a PCI Express v3.0 x16 slot and like you said, the 10 GbE cards take up at least a PCIe x8 (and then having enough PCIe lanes on the CPU to support this). And last but not least, is the consideration in terms of the physical space that an actual computer/switch will take up. Yes, it will be able to do this more "unique" multi-speed step up/down (which no commercial switch really offers), but it does take up more room in the rack than a commercial OEM switch takes. Don't get me wrong. This is interesting and I am definitely interested in possibly building something like this because it will help solve my somewhat rather unique situation of needing to step up/down several levels like that. But at the same time, there are also all of the other considerations/factors as well. This is very neat and it's certainly providing some food for thought. Thank you.
I don't know if I just missed it but an important thing to check is that your CPU has enough PCI-E lanes to run all the cards at the correct bandwith. Thank you for this very informative video
problem I had when I built my 10 gb network is that it defaulted only to xfinity ipnetwork & would not add A second nic - the second 10 gb only worked when the xfinity was off the lan - is this normal
your Mellanox cards? great 22 queues? or just 22 interrupts? I shall check how this does chelsio and then check Mellanox, unfortunately my server have no more pcie ports left (LSI SAS 9211-8i, nVidia Quadro NVS 290 PCI-e 1x, Intel PRO 1000 ET quad port, LSI MegaRAID 9260-8i, Intel PRO 10G XF SR) this runs on AMD Phenom II X6 1100T and network does not hit CPU much, encryption and RAID does
I put the switch together on Saturday using four, four port Intel cards and everything appears to be working great. My 10Gb cards should be here before next weekend. Do you know if there is a support forum for VyOS where I can ask questions? I looked for one on the site, but what I found doesn't appear to be directed at users, more at developers.
That's fantastic! I'm glad you are having fun with your build. That makes it all worth while. I know VyOS just released their newly updated website here: vyos.io/ You might try Reddit/VyOS too: www.reddit.com/r/vyos/
I watched the three parts and my main question is why not use cat6a/cat7 8p8c instead of DAC? It's way cheaper and I think it's better for budget systems because if you start adding switches, routers, firewalls and stuff, 8p8c hardware is cheaper
You can use any media you prefer. In the third video I talked about using the custom switch as a media converter as one possibility. But yes, you can 10GBase-T interfaces. At the time I made the video the DAC cards were dramatically cheaper. A lot of the 10GBase-T cards are still very expensive.
I don't even care what the subject of this video was. It was so well presented, with so much clarity and insight. Thank you for putting this out there. I'm kidding, I really do care about high speed networking.
Watching this in 2023 and it's so useful. Thank you so much for uploading this!!
Just published part 3 of 10Gb Home Network - Build A Switch. In this video we wrap the 10Gb Home Network series with a really fun project to build a very powerful and usable home brew 10Gb network switch. All the magic is embedded in the network operating system. I invested easily over 100 hours of designing, building, troubleshooting, and testing in this project (just part 3) so I stand by the results. Now you can build one using my recipe. I hope you enjoy the end result!
Speaking of the "recipe", would you put up a hardware list? What system board and network cards did you use?
Check P1 and P2 of the series.
I watched those and I don't remember any mention of the system board that he used in the switch. After all, this is the video that covers the switch.
Oh you're right my bad, I thought you were talking about the NIC's I misinterpreted what you asked.
Hey guys, I just updated the video details fold. Tons of stuff for your browsing pleasure!
I wanted to thank you for your awesome 10Gb series, i was able to make my own 10GBe switch for my unraid server using all 3 of your videos.
Congratulations on your build! Hopefully you had fun and learned a few things along the way.
You need to upload more regularly!! Very enjoyable. You inspired me to start a project like this as a VM on my storage/VM server.
+Narwaro Awesome! Thanks for the kudos. I completely agree with you about needing to upload more regularly. I've been trying to think of ways to reduce the time it takes to make a video. On top of that I'm a one man operation and I have a full-time job. So it's extremely difficult to knock videos out on a routine basis, as this could be a full time job easily. I'm going to figure something out and strike a balance. That's my goal. Good luck with your project. That's how great ideas are born!
I know how that is like, unfortunately. Only put the amount of time in to TH-cam that you can afford!
(I have actually been into networking quite a lot recently because Im redoing my server room and all my cables with fibre and CAT7, and I am also one of the first ones to get fibre at work, fk yeah)
+Narwaro good times!! It is always fun to be involved with those projects. So much to learn! So little time.
As a physicist and programmer I have the honour to be allowed to play around with hardware and call it 'work' :'D
+Narwaro good excuses are priceless!
i really want you to do another one of these just for testing see if there is any better software or see if it will even really work
I tried building a DIY 10gb Network Switch a year ago, and everywhere I went people said it wouldn't work. I got it working in a dell PowerEdge 2950 but it would only do like 500MB in one card and out another. I ended up buying a 10gb switch (crazy expensive). Glad you figured it out. Now you have to build a raid0 fileserver with ssd's. I get 2000MB read and write on mine. Loading games over 10Gb is faster than my 2 ssd raid0 on my gaming machine because over the network the game loads straight into RAM. =)
+Levi Blaney that's really interesting. Thanks for sharing Levi. Now that the 10Gb infrastructure is in place, I'm kicking around a few ideas on how to leverage it with fast storage in a fun way. Out of curiosity, what OS are you using for your RAID0 setup?
iTechStorm Try not to laugh... I'm using windows 10. This is because I built the "Ultimate" HTPC. It's used as a media center, and to game in the living room and to be my 10Gb file server. I was going to use my PE2950 but it's just not the right setup for a 10Gb file server. A Enterprise Server with PCIx 3.0 is extremely expensive but a consumer motherboard with two 16x and one 8x is only $200 or so. With the R9 290, the H700 raid card and the Melanox 10gb NIC I needed all the fast slots I could get. I still have a server cabinet full of servers for VM's and stuff, and the PE2950 has 12TB's raid10 for backups and VM's.
iTechStorm Pictures from a year ago when I first started, everything is much different today.
goo.gl/photos/g97xo2vgwGdDxxYQ8
edit: correct link
iTechStorm I would really like to see you make a RAM disk on two different machines and copy a file like a 4gb or 2gb movie back and fourth. I really don't think the switch you built will hold up. Mine couldn't in my tests. The issue is SMB has more overhead then just raw IP packets. This tends to translate into a much greater latency when going through the PCI bus, this causes throttling and lower throughput. With all the testing I did. I was only able to get 600-750Mb's. Still much better then 125MB's though. I bought a 10Gb switch because I already had invested in the Raid0 SSD file server.
Lol. Love the picture with the cat! Cats power the internet you know...
OMG I LOVE YOU!
Am having a new home built, and decided that I wanted to upgrade to 10GigE and start putting everything on NAS.
My god is this helpful!
+Jennifer Royal awesome!! Checkout my other video for the network closet setup. I did the all my cable runs after my new home was framed. I'd recommend no less than Cat6 cable. If you can afford Cat7 that is a better option for future proofing. I purchased a 500 foot spool of bulk media cable which was a dream to run. 2x Cat5e and 2x RG6 bundled into a single cable. Then faceplates are easy. 2x2 faceplate with 4 keystone jacks. One of my favorite upgrades to my house.
The most informative video I have ever seen brother, you rocked it
That bios is intense with the specialized payload settings and all the other cool server stuff. Wish mine was like that.
+mrlithium69 some day your bios will grow up to be big and strong... some day
Wow this was very useful for me, thank you very much for this very well explained video. I would like to know if on this switch I can create vlans. I'm going to build one of these and would like to use vlans in my vmware projects. It is possible ? Thanks in advance for your help.
Absolutely! Here’s the link to get you started:
support.vyos.io/en/kb/articles/vlan-sub-interfaces-802-1q-2
I want more videos like this, they're the best!
That was awesome! Really learned a-lot! Thank you for this well detailed video!
Thank you so much for this excellent series ! I am learning a lot and can't wait to put it into practice.
I was waiting for this since P2 release, and put the plans to buy a 10g switch on hold until i see what comes in P3.. Certainly i wasn't disappointed, great effort and very organized and informative video. i will give it a try since i already purchased 4 Mellanox ConnectX-2 cards that i will use in a similar build.
I would like to thank you for the dedicated effort and wish that you will keep posting such quality materials when ever it becomes available.
Regards,
+Essam Mohsin thanks for your kind words. I'm glad I could help with your decision. One thing I forgot to mention in the video is that most 10Gb switches are very noisy. That can cause problems for a lot of people. One more thing to consider I guess. Good luck with the build. I had a lot of fun with this project, and I'm glad I could share in a meaningful way.
Awsome 10G guide series. I am currently upgrading my home-labs on 10G with new FreeNAS as my VM storage. I was debating what to about the 10G switch since they are sooo expense - VyOS looks like its gonna do great. Ubiquity runs a fork of Vayata on their Edge Switches and Routers and they run superb. Good job and thanks for all the info provided in the description!
Very well done. Looks like you put a lot of tie into the video. Thankyou
*time*
This was a project I truly believed in, and I wanted it to reflect in my work. I easily invested over 100 hours of designing, building, troubleshooting, and testing. I truly appreciate that everyone is enjoying it! Thanks for the feedback.
iTechStorm Absolutely!
Really great series. Well made! Really liked the way you explained everything and why you made certain choices.
Vyos really reminds me of working on Juniper network gear.
Thanks!
Until seeing this video, I would never have thought of this as an option. I have two "segments" in my home network. My entertainment room which has mine and my wife's desktops plus our entertainment center. And out in our living room is our router along with the NAS and a virtualization system. I want to connect the entertainment room to the router using 10Gb or faster. And I'd been trying to find hardware to give me what I want, but now that I've seen this video, I know how to proceed.
And with this option I can start small with getting the wide bandwidth (10Gb or better) set up first and then add everything else, likely including putting 10Gb Ethernet cards in mine and my wife's systems. I've seen switches that have Gigabit connections with a faster uplink (typically SFP) but that always seemed a little limiting since each client would be limited to only Gigabit. This idea of a custom switch, though, as you said, allows you to meet specific requirements, and change things as the requirements change.
I definitely have more research to do regarding the available options. I've considered 10GbE and Fibre Channel for the link between the living room and the entertainment room depending on what I can find for what price. And I still plan to do 10GbE in the NAS -- which will likely be the first move since it's backward compatible to GbE. But overall it means two switches: one for the entertainment room to the living room, then another main switch in the living room to include the router (provided by my ISP, so I can't change it out or I would with a pfSense box).
Out of curiosity, have you tried combining wireless into your custom switch using a USB or PCI-Express Wi-Fi adapter? If so, how easy was that to set up?
+brandishwar excellent! I'm happy to hear you are getting a plan together for your 10Gb upgrade. I love hearing about different configurations and how they are used. Unfortunately I have not tried connecting a wifi device to the switch yet. But I'm certain I remember seeing wifi functionality built into VyOS, so I would say your chances are pretty good. If you get a chance to test it out before I do, make sure you post back with an update! I would love to know how it works out.
Great video! I just picked up a quanta 25 port 10gb switch off ebay for like $350. Works great!
what? i paid 5,000 for mine
The Quanta LB6M's are plentiful on ebay for around $350-$400!
+Thomas White I've heard you can find a cheaper Quanta at times. But aren't those switches noisy? I actually forgot to mention this switch build is quiet in comparison.
iTechStorm They can be pretty noisy. Mine isn't horrible but I have heard some that are like jets. I wish mine was more quite but that's something I can get over for the price.
+Thomas White good to know, thanks.
This guy has his shit together. Excellent video series!
+thegreatga lol, thanks! I try...
Amazing series. Thank you. Now to go earn some money for my new setup.
If I had this setup. Could I have my Steam library on my server and run the games from there? Seeing as the speed is so great?
+Luke Eaton that's great to hear you found the series so useful. Thanks for the feedback!
+Luke Eaton I couldn't say for certain. Sounds like a good project for you to test in concept though!
;) Will definitely do that. Maybe around 6 months till I can complete it. I will report back and let you know.
I've been using a steam library over SMB on 1G network from FreeNAS - most games are fine and only certain ones hang for 20 seconds when loading. I imagine you'd have no problem using 10G network.
Amazing! I watched all your 10Gb videos! Thank you!
+Wei Ming Liang you are very welcome. Thank you for the comment.
THANK YOU for all that you put into these. Very good detail 👍🏼👍🏼
+Donald Coleman thanks for the feedback! Glad you are enjoying the content. I wouldn't put so much effort into my videos if I didn't believe they were going to resonate with people.
Awesome idea, I was actually toying with it myself but was thinking of RouterOS from Mikrotik, since I'm more of a Mikrotik fan. I've used Ubiquiti in the past though and they are coming with some awesome products! Their new EdgeSwitch 16 XG would be a competitor (around 600$) for anyone who is trying to build this from scratch I think.
But if you have some of the parts already (like an older PC as you mentioned) this is really a killer way to build a cheap 10Gbit switch! The video and tutorial is very clear, great work! :D
+InterTech ya I agree the Ubiquiti product line is really promising. I think we are going to see some really good products roll out their doors.
With that motherboard, memory, NICs combination and considering the power draw, you can buy a 16 port 10g netgear switch. I've seen them on ebay for $1300.00. I use VYOS extensively in our company of 300+ servers, and it is rock solid.
+jkljkljkljkljkljkl I don't disagree with what you said. But at the same time I never said you had to use the same hardware as I did. I mention in the series you should size your hardware to your own requirements. Some people could even use hardware they have sitting around. At the end of the day, it's just an alternative to typical vendor solutions. That's all.
Hiiii,
this is honestly the best tutorial I've ever seen. such a comprehensive and detailed video. thank you sooo much.
however, I'm sad that this video was 6 years ago, hopefully you are doing well and nothing happened.
This Video makes me so hyped for also committing a build :D
Paul Wolf 5 months later did you build it? I just converted a legacy Citrix Netscaler in a 10Gb switch
Very nice!
I just bought Quanta LB6M for 230.00 from unix suplus. CRAZY CHEAP! 24 ports of 10Gb goodness.
If you don't need that many ports there are a ton of Dell Force10 switches out there for under 100.00 like S50-10-GE-48T-AC which has 4 10Gbit and 48 1Gbit ports.
I needed 8 ports of 10Gbit or i would have considered something like this.
Also, if you are on Windows 2012 or better and want maximum throughput on SMB shares you will need a card that supports SMB Direct/Multichannel via RDMA the mellanox connectx-2 cards don't have this but I haven't found it to be a huge bottleneck.
Lastly, the cost of cables is kind of crazy too. For long runs I'm using fiber and sfp transevers which for long runs is WAY cheaper than trying to get a DAC cable. for short stuff DAC is the way to go.
You sure the the Force10 S50 switches support 10Gb via the front 4 ports though? I thought those were only SFP ports (not SFP+). I thought only the rear expansion modules supported 10Gb (via XFP modules) and only for switch stacking.
Yikes!! That's 130 watts while idling.
Thats actually an idea I had too when 've seen that you can get dual sfp+ fro just 25€ each
It's funny how 4 years later that server hardware he used in the video is probably more expensive than buying a Ubiquity XG 16 switch. It is not as flexible for sure, but more compact and damn that nice UI :D
My question would be is it really worth it to build your own compare to an already built switch? I'm looking at it between the price point of a homelab and a datacenter. it's awesome to build your own network switch and all but not sure other then a costly of cards as well as motherboard with a lot of lanes on it.
How does IPFire stack up against VyOS, I wonder? It is also a GNU/Linux router OS. VyOS is currently command line only, which does not appeal to me.
Why no more videos for 2 years?? Pls do part 4, on NAS 10Gbe home /office network. Pls.
Just the info i was looking for.. subbed 👍
This is so helpful! Thank you very much for the clear instructions!
Thanks. Nice video and excellent understandable diction and pronunciation. Why don't you use just bridge tools in some centos or like?
Can you comment on the cost aspect? What parts did you select and how do they compare to a currently available commercial switches? For example I know that capable of the shelf switch could cost around $1000.
Well, it's never easy to throw out one price fits all comments. So on one end if you already have a system laying around (doesn't need to be anything special), and you can pickup a dual port card on E-Bay for $50... that's your cost. On the other side of the equation if you have to build a system from scratch, your price tag is going to be a lot higher. The best part is that your Network Operating System is no cost. Beautiful!
Literally a much better trilogy than twilight
Great job on the video! I DON'T think you sound like a text-to-speech engine. LOL I definitely like the options you are presenting here. I like the fact that the home builder or average techie can stick it to the companies which, up until recently, held the monopoly on this kind of networking.
+Tony Carballo maybe I'll include a text to speech segment in the next video, lol. Well 10Gb is certainly overdue for the power-user segment, especially when you look at what has been going on with data transfer rates internal to PC/game systems.
I'm starting research into how to build a multi purpose machine that is both a nas and switch.
Do you have any ideas that could help?
You are a god among us
Thanks man. Great job on the video. I really love it.
Excellent!
One issue. Putting together this hardware does approch the cost of purchasing a Netgear 8 or 12 port 10Gbe Switch...
Tho it does have more versatility in terms of using copper/fiber plus 40Gbe cards could also be used I suppose..
Exactly. Although I want to stress the versatility is also in using different combinations of hardware (cheaper). I never stressed the use of these particular components in the video. My point was, you get what you pay for. Thanks for the feedback!
Netgear sucks! I would never buy a Netgear for home, heck even if someone gave me a brand new 10GB Netgear, I would sell it on eBay and use the cash to buy the parts to make the switch in this video. Also, you get the feature of layer 3 in the switch build here, which is very helpful if you have a wifi network and you want to route to a DMZ. For what it's worth if you were to buy a cisco or brocade version of what he is building here; go ahead and take out your checkbook and write out something close to 5K and then we can talk cost and if this build is worth it.
The 10Gb hardware has also dropped quite a bit since he made these videos. Since your comment, they have even dropped further! It is now cheap enough for the average consumer to consider buying 10Gb Ethernet for normal home use! (May 2018)
LogicProHacks recently had a data centre job that involved managed services for some of the customers. Interspersed amongst them were a few who had chosen Netgear prosafe 10g switches for the storage network of their virtualisation setups. At first I was mortified as I’d previously had less than stellar experiences with people using Netgear equipment in production. In the end though we had almost no issues with them at all. The key with them seems to be the same as when you come across Mikrotiks, Ubiquiti, and other lower tier gear, keep them patched and keep the config simple. I won’t vouch for any other of Netgear’s product line, but the 10g prosafes were good enough for a home or soho deployment. The actual biggest pain in the arse with them is they are web managed only so you are reduced to clicking stuff in a GUI like some sort of chump.
@@denshi-oji494 do you have any examples of affordable 10gbe Ethernet switches with more than 2 X 10gbe ports?
I subscribed to the channel so I would get notified when this went up. Thank you so much for your effort on this!
Great Work!
The illustration looks like a SupperMicro system board and you said that you would have more information below, but I don't see any additional info.
I just posted a ton of info to the video description. Check it out! You are correct though, SuperMicro board.
Hi there. great video! Only issue i have is im trying to do this on a desktop based i5 PC. The only option in the bios that is remotely similar to those advanced options you specified is enabling Virtualization technology. Are those options really necessary? You sounded full of conviction so im assuming they are important... but how important is my question? lol.
Great video! Thanks! Wondering why you didn’t virtualize the Vyos install via hypervisor. This way your system is not just dedicated for 10g switching but can be used for other purposes. Finally, your rational for doing a custom 10g switch was to save money on enterprise 10g switch but then you end up dedicating your whole system (super micro mb, 4x10g nics etc) for switching. May be I missed something or just misunderstood your part 3 video. Regardless, the idea & the presentation is great & useful. Thanks again.
I didn't see about how to set up the card's configuration. Perhaps I just missed it. It was late when I was viewing this. I had been fighting a consistent no-carrier issue. I finally went in and changed the config setting in my dual-port Qlogic card from SH (what ever in the heck that is) to NL-partitioned. after a reboot (required because the firmware hung during the "save" command, which made me a bit nervous) and BING - now we have carrier and the first MAC on each port now have an IP addresses assigned by the DHCP on the PFsense box. Now each port has 4 sub-ports and each sub-port has it's own MAC address. Sadly, CentOS 8 still marks all the sub-ports other than the first as unavailable. Although I haven't looked yet, I'm betting it will be the dreaded no-carrier again. It appears I will still have to set up routes from each system to the other 4 as even after getting an IP, I am unable to ping a neighbor or the gateway on that network. So not really working yet, but at least some progress was made. I had swapped SFP+ adapters on both ends of the fiber and also changed the fiber itself from one set to another (OM3, LC-LC, 3 meter).
GREAT SERIES. WELL DONE
Great series, thanks! Ever since I bought a NAS (yes, a very simple out-of-the-box NAS with a 1Gb port) I have been wondering: now that home (out-of-the box) NAS solutions become more and more common, why is there not more focus on improving home network speed - and NAS ethernet cards for that matter - so that we can really go towards centralized storage? Of course, I quickly found out that 10Gb equipment is rediculously expensive. Then again, this is no doubt partly caused by the fact that the market is pretty much enterprise-only.
So, as long as the manufacturers don't make the switch (pun intended :P) towards 10Gb networking, at least we have your great explainations of how to get it up and running 'on a budget' :)
I do have two questions though. First, I love to play around with software and hardware, but I also know I'm not a security expert. So, I'm wondering, is there anything we should know in terms of keeping our home network secure with a home-brow switch like this? I mean, strictly speaking, I think it's routers that are 'responsible' for ensuring safety - and there would still be a router between such a 10Gb home network and the internet. But then again, you also mention firewalls and stuff when talking about the Linux distros. In your opinion, should we pay any attention to security when it comes to building a home-brew switch, and if so, in what aspects?
My second question relates to speed. Sure, a network has awesome throughput rates, and you would definitely want to do some low-level testing like you did to see if everything is set up correctly. However, I also saw somebody else asking the question whether he/she could host a Steam library on his/her NAS and run it from there. Now, I understand that you have not tested this particular thing, but to really test if 10Gb network could 'solve' the fast, centralized storage problem, one would also need low latency. Have you done any latency tests? Or, maybe even better, some storage benchmarks that mimic real-life use? I would be extremely interested in seeing some of those results, but as you mention, building a network like this is not just a couple of hours (nor a couple of $$ for that matter, even with the self-buid :P).
Thanks for your comment. I'll do my best to answer as concisely as possible. In your first paragraph you mentioned the storage centralization trend and why it's important for the home network infrastructure to keep up. You are exactly right, and that is largely what motivated me to make this series. I felt there was a significant void in this segment, so I offered a possible solution.
To answer your first question security should always be a concern with every device you configure. There is always going to be a balance you have to strike between functionality and security controls. Using best practices is always recommended, but not always easy to obtain. Everyone's configuration will be a little different and suffer different vulnerabilities. So the idea is to minimize your attack surface, but turning off unused serviced, inherently blocking all traffic and only allowing valid traffic through rules. Use a strong/proven boundary protection device and avoid using it for anything else (minimize attack surface). In the case of the home-brew switch, don't enable anything you don't need, use VLANS to segment different types of traffic (mgmt lan, storage traffic, media traffic, guest traffic, etc.), use strong passwords, disable default accounts, and patch when vulnerabilities are identified.
For your second question latency testing was mostly out of scope for this series. But I understand why you are interested. Consider the context of "gaming network latency" versus "block storage network latency". When talking about gaming latency, a 120ms RTT (Round Trip Time) would not be favorable at all. While a 40ms RTT would be considered a pretty good network latency for gaming over a cable modem (WAN). But when it comes to block storage network latency every professional will tell you those times are a total deal-breaker. Centralized (network attached storage) is not quite as fast as locally attached storage (ALL THINGS BEING EQUAL), but these differences are measured in sub-milliseconds. And that is why I use the "context" example. I mentioned DAC (Direct Attach Copper SFP+) is faster than 10GBase-T from a latency perspective in my first video, and that is why you see more people using it for enterprise storage. So we are talking about microseconds versus milliseconds here. So what I'm trying to say in a long confusing way is that for the purposes of gaming don't confuse the two. A home user would never notice the difference. Just focus on building/buying a decent performing NAS, use a 10Gb connection, and have a workstation that is capable of handling the heavy 10Gb traffic stream.
Does that answer your question :D
Yes, that indeed answers my question, thanks so much for your elaborate answers!
I was by the way thinking about block storage network latency - the Steam library may have been a somewhat unfortunate and confusing example. I merely chose this example because loading a game, or a level, is a use case involving loading many small files, rather than one large file as would be the case for multimedia applications. To justify storing game installations on a centralized storage, I would at least require that loading times would not increase tóó much compared to internal storage - hence my question :)
Regarding the security: I think I understand your answer, but as soon as you mention 'use VLANS to segment different types of traffic', I find out why for me, a home-brew switch might be a bridge too far. I can image why it would be wise to separate traffic like that, but would not have thought of this myself, nor would I know how to implement that.
I guess I'll keep hoping that manufacturers will see the light and start producing 10Gb for consumers as well... And if it takes too long, I may give it a go myself after all. At least, I have a good guide now, thanks again! :)
+Caspar v. L. A few more parting words from your follow-up. It is expensive and requires a degree of planing to pull off a central storage solution that is larger and faster than local storage.
Don't let my security example scare you away. None of that is required to build a healthy 10Gb switch. Just an example of paranoid security suggestions.
I'd say that most home users who want to delve into 10Gb stuff could easily afford a gigabit switch that has options to add a few 10Gb ports. I opted for the Dell 6224 with a 2-port SFP+ card that plugs into the back of it. It has another slot to add another card. I can have 24x 1Gb copper ports, or 20x 1Gb copper and 4x 1Gb SFP ports on just the front of the thing. Since 2016, switch prices have gotten saner. It is a shame that the majority of 10Gb switches out there are still 10GBase-T.
Very interesting and well done video. How much did the end result cost, and what would a comparable "off the shelf" switch cost?
Well, it's really difficult to answer your question directly but generally speaking my intent was for people to use a system they had sitting on a shelf that could accommodate a few PCIe cards. Purchase the 10Gb cards for $18-$50 each (ebay), and grab a Direct Attach Copper cable for $18. If you have to purchase system components your costs are going to go up from there obviously. Everyone's cost is going to be different. I used a system I purchased for a SAN build, but just about anything with decent power and bandwidth will work.
In putty, be sure to use UTF-8 for the keyboard set in the connection. Lines will draw instead of characters as lines ;)
+themaconeau interesting... never seen that happen before. What version of Putty are you using?
iTechStorm v0.62
I misspoke earlier. It isn't keyboard but Window > Translation > Remote character set on the screen PuTTY presents when connecting.
The default says something like ISO8859 (Latin, Western) or something similar. When I changed this setting to UTF-8 when logging into Debian via SSH, the characters I was getting on the screen that represented lines were replaced with actual lines.
Be sure to select UTF-8 for linux distributions in the PuTTY configuration settings before connection and you're golden. :)
+themaconeau thanks for the update
Hi folks great video. After watched this video I have made a VYOS switch with my tiny HP T5740 PC. I have installed one dual port intel 10G fiber card. I have connected it by fiber. Can you tell me how how to see optical power reading on this ?
Great video. I wish you would have included some video / pictures of the actual build though!
+yggdrasil viking there really wasn't much to see. Just network cards stacked next to each other, but I did show that in the video. I wanted to keep the video focused and on track.
Nice work - Does VyOS support 10Gb over USB 3.0? With card like Mailiya PCI-E to USB 3.0 5-Port PCI could you get 5 ports per PCI-E slot?
I love this video. I've got a bunch of 4gb fiber nics that I'm going to try this with. 10g is still to expensive for me
this entire series talks about 10gb ethernet - not fiber channel. cheap 4gb fc cards like the garbage you can grab for 1$ a piece from emulex are not ethernetcards. they are designed to work in a san enviroment, not an ethernet enviroment.
+Lmn Plays should work just fine with the fiber NICs. As long as the cards are detected and drivers available. Let me know how it turns out!
iTechStorm you do know that FC as in Fibre Channel is a completely different protocol compared to ethernet. 4gb fc hbas cant be used for ethernet over a network. they are made to connect to fc devices or in a fabric enviroment of a sanswitch/san
there are converters that you can buy. I pulled these out of my old netapp san and i have a 4gb switch that i dont use. I like the idea and the possibility of building a switch. thats all
you do know of RFC2625, as in IP over Fibre Channel (IPoFC) right? RFC2625 has been in place since June 1999 too. Never say never Zishy.
www.ietf.org/rfc/rfc2625.txt
"FC is standardized under American National Standard for Information Systems of the National Committee for Information Technology Standards (NCITS) and has specified a number of documents describing its protocols, operations, and services."
Freaking amazing build. Thank you for going into such great detail.
glad you enjoyed it! I'm a big fan of detail myself...
Awesome content, awesome presentation. Thx for everything!
Awesome video series! Thank you
+davester32 thanks! Hopefully you learned something useful.
One of the most useful and thoroughly explained videos out there on Vyos. I was able to build a virtualized switch in ESXi. It was a little more involved setting the MTU of each vSwitch, port group, and VM, but was not far from what you were accomplishing in the video. However, I was not able to find out how to install iperf on Vyos. Any tips? wget keeps giving http/ftp errors despite there being a clear internet connection.
IPerf is already installed on the vyos device. I just checked in version 1.1.8. You should be able to run it from the cmd line.
man iperf
iperf -s
Wow, that was quick. I swear I tried it before and it didn't show it was installed! :P Well, that fixed my issue
Happy iperfing trails! Outstanding utility! Checkout “bmon” if you haven’t seen that one yet either.
Loved the video series on this topic....
I had fun with the series myself. I'm glad you enjoyed it too!
nice video! but would it work with your bog standard consumer grade motherboard +cpu combo? without all the fancy IOAT and DCA...etc options in BIOS? Sounds like a nice project to undertake if possible.
+Steve Lee really good question! My approach was aimed at achieving an "optimal" configuration. I wanted to show it was possible to reach those speeds at home. You can certainly achieve a great result even without the optimal tech (IOAT, DCA, etc). Everyone's mileage will be different because the performance is highly dependent on the type of hardware you are using and the configuration. If you can get 2-4 Gbps, that's already a massive improvement over 1 Gbps. So beauty is in the eye of the beholder :)
This was a good guide, thank you for making this!
+Kad1nat it was my pleasure. I'm glad everyone is enjoying it so much.
Really good and detailed information. It'll be quite a jump from my Edgerouter X attempting this. I'm guessing combining this with a SG-2220 outbound to a modem is ideal. I hope you update the NAS/SAN build, as a new one can exploit this network (plus Xeon D and Atom integrated boards make the builds much less complicated).
Thanks Quick! I have an Edgerouter X too, and you'll be pleased to know they pretty much ran with Vyatta-Core and forked a slightly different direction from VyOS. I actually enjoy using VyOS a lot more since it's easier to use from the command line with features like abbreviated commands. Anyway, that's a really good question about the SG-2220, since I haven't seen where anyone else has tested it with VyOS. It probably works great if it's using standard chip sets. And on the topic of the NAS/SAN build, I agree the Xeon/MB/Mem should work quite nicely with a 10Gb connection XD
Just wait for the new 10Gb EdgeRouter!
+Thomas White sounds interesting
You could do this with for example 25 or even 100 Gbe cards ?
I just want to add at 9:44, this is only true for PCIe NICs. It does not blink for onboard NIC and you cannot re-arrange the onboard NIC if you have multiple ones.
This is a great video. Thanks! I might be trying this out. Big question: Can I get respectable performance from cheaper hardware with low port counts?... like if I dug through my computer graveyard and put 2 or 3 of these $99 ASUS copper cards inside?
It’s worth a shot. You may not even need the full 10Gb performance anyway. Even if you only get 4Gb that’s 4x faster than 1Gb.
great video! keep up the good work
Great series! Thank you
I really enjoyed making the series. Thanks for the comment!
whew, all that Java is giving me cold sweats.
+Thorsummoner0 mmm, coffee, yes please.
Aand now I'm going to try out Vyos. I like that CLI menu.
Truth. . Low mem usage for many years
you should set mtu to bridge too.
logically bridge is in front of NIC but in reality it is in middle
You receive on NIC, which is sent to bridge and then it traverses bridge, then nic and back to bridge and that is on both sending and receiving.
Actually it is even more complicated
having low mtu on bridge may severe high mtu connections
remember mtu is not part of IP (biggest IP packet is 65535, period) mtu is my for EtherNet frames (including MAC - source and destination and other bytes I never can remember meaning of 0x0800 - total 14 bytes)
At about 9:22 I noticed it's a version of OpenBSD which is really good from what I've heard about it
great video, and great work, i am installing Vyos too and it seems it has no support for my Mellanox NICs do you have any idea or litterature , on driver install solutions / scripts thanks a lot !
I don’t have any specific docs on what you are asking for, but you might check the forms to see if anyone else has a workaround.
Nice work!
+VyOS thanks! But I would be remiss if I didn't thank you guys for the amazing job you've done with VyOS!!! Excited for when version 1.2 drops.
Could this all be done in a vm using pci pass-through?
Question: How does IP assignment work with bridging all devices? Can you still utilize your router's DHCP function to assign IP addresses to devices connected through the bridged connections? Does this automatically obtain IP's or do you need to assign them seperately? Can you manually assign IP's as well?
Like you I attempted to connected a 10gig server and Work station via PFsense and I got sub par results in transfer speeds... surprisingly even worse than 1gig. I'd really like to make an affordable 10gig switch that I can expand in the future.
Thanks for your question Eric. Just like any other conventional switch the bridged ports will pass a DHCP request through just fine. In my demo, I assigned a static IP address to the VyOS switch itself. This was for the purpose of easy management, but normally a basic/unmanaged switch would not have an IP assigned. I'm pretty sure you could configure the VyOS interface to request an IP dynamically too. So you should be just fine.
Wow. Thank you. So much information.
Hi, thanks for the video. I was wondering, why are there so many sfp+ cards going for so cheap on ebay? Why do people replace them if they still work?
There are several reasons for this. When companies change out their older servers for newer models they typically sell to the highest bidder so they don’t have to deal with parting things out. The winning bidder then sells everything salvageable on eBay. These SFP+ cards are very common in server hardware and get replaced by newer models just like everything else out there.
Thanks for the reply, that makes sense. Obviously this is a 'how long is a piece of string' kind of question but are the 2nd hand cards prone to failing?
I would say it’s pretty standard for 1 in 10 cards to fail weather they are brand new or used. That has been my experience.
If i buy a 10gb network card, can i use spf dac and aoc sr cables? Fiber optic cables can be used along the dac Cooper cables? Or Just dac cables works with network Card, do i need something to run aoc fiber optic cables, can i use bot dac and aoc cables on the network card?
You should be able to use any of those methods you mention. It’s important to check for compatibility of the SFP module to your NIC or brand of switch. Keep that in mind when selecting your medium.
what MTU would i need for a: HP InfiniBand 10GB/40GB
awesome stuff, thanks for the video.
+Thohan79 tech is fun, it should be shared
Dude, this is like a $1000 system.
Do you have something that might be just a bit cheaper like using some stuff around the house or left over parts, how about using an old i3 4150 or something like that? Like if I had the money to buy this I wouldn't be looking to build one bro.
Can you please create a small demo of google data centre network at home using switches desktop computer rispbberi pi router, network cards?
Is it better to use x86 and x86-64 processors for this kind of stuff or would it better to use OEM switch ASICs?
Also, do you have to build a bridge in order for the ports to be able to communicate with each other?
Are bridges the same as link aggregation?
Keep in mind this is just a DIY project that allows you to use commodity hardware you might have laying around.
ASICs are better for specialized functions like packet processing of course.
You need to bridge ports so they can communicate at layer 2. You could route at layer 3 without the need to bridge, but that is beyond the scope of this video.
No, bridges are not the same as link aggregation.
@@Itechstorm
So, there are a few reasons why I asked the questions I asked:
I've recently acquired 4x EDR Infiniband cards (dual port VPI Mellanox ConnectX-4 100 Gbps cards) and also a 36-port 4x EDR IB (all QSFP28 ports) externally managed switch as well. My NAS systems have dual 10 GbE ports (SFP+) and then I have, of course, "conventional" RJ45 CAT5/6 gigabit ethernet switches as well in my home.
So the idea was I wasn't sure I really wanted to get breakout cables to be able to step between the speeds (and admittedly, I'm not even entirely sure whether VyOS would work with the Mellanox Infiniband driver, although with it being Debian based, it might be able to work) or if I would build something like this and this type of a system would be the kind of custom switch that would be able to handle the different step downs. For example, a single 4x EDR IB connection can step down to 10x 10 GbE connection or I would make the NAS systems (which run on 10 GbE SFP+) to be made available and accessible to my systems that are running the 4x EDR IB interconnect via the IB switch.
Apparently Mellanox has differentiated their products such that the 100 GbE switches are separate from their IB switches and that the ports on the IB switch AREN'T VPI ports such that you can configure them to run in either IB mode or ETH mode like you can with their cards (shame).
And I didn't want to have to pay for an entirely different 36-port 100 GbE switch either (or any of their smaller offerings) just to be able to run 100 GbE as well. (Although interestingly enough, cost per port, works out close to be about the same.)
Qnap makes a 10G/1G 12-port switch. Mellanox makes their 100GbE switches. And I wanted to be able to step up/down between the speeds (and also my existing gigabit ethernet network as well), so I didn't know how well a x86 or x86-64 processor would be able to handle this type of a switching workload compared to the switch ASICs found in an actually Mellanox 100 GbE switch (for example). (Of course, nothing would be able to do a multi-step up/down like this, but that's part of the question.)
And then, as you mentioned, there are other considerations as well. For me to be able to switch between 100 Gbps IB to 10 GbE to 1 GbE (and back), the IB card alone takes up a PCI Express v3.0 x16 slot and like you said, the 10 GbE cards take up at least a PCIe x8 (and then having enough PCIe lanes on the CPU to support this). And last but not least, is the consideration in terms of the physical space that an actual computer/switch will take up. Yes, it will be able to do this more "unique" multi-speed step up/down (which no commercial switch really offers), but it does take up more room in the rack than a commercial OEM switch takes.
Don't get me wrong.
This is interesting and I am definitely interested in possibly building something like this because it will help solve my somewhat rather unique situation of needing to step up/down several levels like that. But at the same time, there are also all of the other considerations/factors as well.
This is very neat and it's certainly providing some food for thought.
Thank you.
@@Itechstorm
Does VyOS support link aggregation?
Thank you for this!!!
I don't know if I just missed it but an important thing to check is that your CPU has enough PCI-E lanes to run all the cards at the correct bandwith.
Thank you for this very informative video
He addresses that early in the video, moron.
problem I had when I built my 10 gb network is that it defaulted only to xfinity ipnetwork & would not add A second nic - the second 10 gb only worked when the xfinity was off the lan - is this normal
I really enjoy the video thanks!
why not install a ESXI server that has Vswitch ? the WebGui is more effective to operate .
your Mellanox cards?
great
22 queues? or just 22 interrupts?
I shall check how this does chelsio and then check Mellanox, unfortunately my server have no more pcie ports left (LSI SAS 9211-8i, nVidia Quadro NVS 290 PCI-e 1x, Intel PRO 1000 ET quad port, LSI MegaRAID 9260-8i, Intel PRO 10G XF SR)
this runs on AMD Phenom II X6 1100T and network does not hit CPU much, encryption and RAID does
I put the switch together on Saturday using four, four port Intel cards and everything appears to be working great. My 10Gb cards should be here before next weekend. Do you know if there is a support forum for VyOS where I can ask questions?
I looked for one on the site, but what I found doesn't appear to be directed at users, more at developers.
That's fantastic! I'm glad you are having fun with your build. That makes it all worth while. I know VyOS just released their newly updated website here: vyos.io/ You might try Reddit/VyOS too: www.reddit.com/r/vyos/
Curious.. you mentioned early that it was based on Debian, but then at 9:34 I see its running BSD.
Awesome video dude!
+JBloomfield1990 excellent! I try to keep my viewers happy...
Great video!
I watched the three parts and my main question is why not use cat6a/cat7 8p8c instead of DAC? It's way cheaper and I think it's better for budget systems because if you start adding switches, routers, firewalls and stuff, 8p8c hardware is cheaper
You can use any media you prefer. In the third video I talked about using the custom switch as a media converter as one possibility. But yes, you can 10GBase-T interfaces. At the time I made the video the DAC cards were dramatically cheaper. A lot of the 10GBase-T cards are still very expensive.
Brilliant video! Thanks! :)
+Emma Stott you bet Emma. Glad you enjoyed it!
Would it be possible to virtualize this if you had the extra CPU power and pcie lanes?