Hi! Along with Clint, who posted down below, we're the network admins for PDXLAN, so can provide some basic info on this. Suricata doesn't do the port isolation, it only monitors and detects the P2P traffic, the 'heavy lifting' is done with software called Packetfence, which also handles the captive portal. Packetfence uses the info from Suricata, along with the attendee's login information to map them to their specific switch port, then shuts the port, moves it to the 'naughty users' vlan, then unshuts the port, so they get the p2p warning and remediation page. Packetfence is very capable, and very frustrating software, but it's the best cheap solution we've come up with so far. The discussion has been had amongst staff to develop our own software that does the same thing as what we use Packetfence for, but it hasn't really gotten past the discussion stage, since Packetfence works, even if it's not perfectly ideal for our use.
I am networking staff for an event very similar to PDXLAN, although of about half the size. The architecture of PDXLAN is pretty similar to the network I'm currently operating. You do hit a few points on the head of the nail when it comes to bandwidth distribution and network quality, and there's a few others I wanted to chime in about as well. The network I administer operates at Layer 3 in the core, Layer 2 at the edge, with each "section" of the LAN being on its own Layer 3 Segment. Generally speaking, this has pros and cons. The biggest Pro to this is, in the event something goes wrong with a subnet, such as a jabbering NIC in a PC, someone creating a Broadcast or Multicast storm which isn't caught correctly by the network switches, a network switch fault, or in the event of say, a malware attack, problems with the network stay isolated to just that Layer 3 segment. Instead of having 600 people experience problems, maybe 40 people experience a problem. It's much less stress on networking staff when something goes wrong because of a client, and you get far less tickets about "lag spikes" in games. There have been numerous instances where, once you start pushing into 200+ person territory, the attachment of say, NVIDIA Shields or game consoles to the network, or the amount of Steam Server Discovery / SMB / Bonjour discovery traffic, starts to overwhelm the NICs in computers with multicast and broadcast traffic which must be discarded. You don't always get the choice of using hardware which can perform Multicast snooping or Storm limiting, either... at least not without cost (Juniper... grrrr). The drawback to using small subnets is extra administrative overhead, so organization is key. But the other drawback is, you lose the ability to do peer-to-peer discovery when a friend in one section wants to host a game server on their PC. Some games only treat the local subnet as "LAN" and anything else is considered "Internet" so, that can make joining your friend's game a little less easy. Source games are usually well behaved. Other games, not so much. But this is quickly becoming a moot point as many popular games today ONLY support the use of Online servers provided by the publisher. The other thing the network I operate does, is on-network caching for Steam, Origin, Blizzard, Windows Update etc. Using DNS hints, intercepts and spoofing, clients when downloading game content are directed to the on-network cache. The on-network cache performs Man-in-the-middle proxying of content for common gaming downloads and traffic, and this saves TERABYTES of traffic each event from hitting the Internet connection. So if say, someone downloads Call of Duty which is 140GB, and another friend decides to download it too, the Internet connection only eats that 140GB download once. The next download is handled on-network. It's also been extremely helpful when VALVe releases CS:Go updates mid-event, or there's a big Blizzard update. It takes only one person downloading the update to cache it for everyone else, and as I said, literal Terabytes of bandwidth are saved. You can use actual server hardware or, even Intel NUCs loaded up with large SSDs and Linux. 10Gbps Internet from a venue gets to be pretty pricy sometimes, so whatever you can do to save on bandwidth costs is key. I generally work with a 1Gbps connection, and bandwidth shaping/queue adjustments in Mikrotik or whatever you end up using for an Internet router is key. I also find that having less hardware in service helps with reliability. Virtualizing the UniFi Controller for example, is something else I do if I am using UniFi APs. If it breaks, you restore from a known snapshot in seconds. For Torrenting, definitely all of the above. Copyright concerns, bandwidth concerns, but the other thing you forgot to mention is NAT Exhaustion. Most of these LANs aren't using IPv6, and they may not have multiple WAN IPs to pick from. Torrents consume a lot of connections and when you have hundreds or thousands of people trying to pipe through a single connection, at some point torrenting will overwhelm the number of connections which can be tracked by upstream equipment (some ISPs like AT&T have BUGGY CPE), or placed into the NAT. Mikrotik does P2P filtering and L7 however, compared to Suricata, it's nowhere near as good and it has many drawbacks. For example, P2P clients using UDP or encryption will thwart it, and blocking UDP ports has many, many consequences for applications like Discord, games, etc.
> the other drawback is, you lose the ability to do peer-to-peer discovery when a friend in one section wants to host a game server on their PC. Some games only treat the local subnet as "LAN" and anything else is considered "Internet" so, that can make joining your friend's game a little less easy This is the main reason we have kept things to a single broadcast domain, there are still a number of classic titles commonly played within our community and at our events. In fact we intend to run some of these as "throwback" tournaments for our 20th anniversary event next year. Instead of breaking this experience we instead do some targeted broadcast blocking to make sure we are limiting some of the more prolific broadcast spam from influencing the entire network, without killing all broadcast traffic. We have found that there is a pretty short list of major offenders that we aggressively don't care about and that, when blocked at the switch level (max of 24 users), will bring network-wide broadcast traffic down to a manageable level. Blocking MDNS hasn't been on the table yet, though the exceptionally noisy Oculus client will be what eventually drives us to this. SMB broadcast traffic also isn't huge by itself, even with 1000 clients. Our main consideration, that I think most people miss, with game caching is the resources necessary to compete with a 10g internet pipe. You need some serious iops and storage to be as fast or faster with more than a handful of users. It is definitely something we could do, we have in the past when our connection was more limited, but the barrier to actually improving our experience with a cache is more expensive than tossing a couple consumer-grade SSDs in a box.
Interesting insights into the technical details of a network admin for Game Center’s very cool! I would love to know how the man in the middle caching proxy works to save on downloading updates!
8:30 I'm a network engineer at an ISP. I recently worked on the circuit for a well known streamer. While on stream I was curious so I looked up their current utilization. It was less than 5mbps. Gaming and even streaming doesn't use a ton of bandwidth, you just want lower latency.
The last lan I was at in Philly with 500 people had someone torrent and comcast threatened to turn off the 10g internet within a few hours of it occurring. The exact scenario you outlined.
Yup. With games these days needing to be on an internet connection, this is very scary - the idea you could have your connection shut off at the event. Very scary for the event owner :)
@@TannerBugatti it's got limits though. Even with L7 inspection rules, any torrent client encrypting traffic or using UDP will skirt right around it. Going any harder starts to break games and Discord, and torrent clients can simply masquerade the traffic once again with what works. Usually you would use an IDS solution like Suricata which can adaptively look at traffic and whack it. More premium solutions like what you find in a Juniper or Cisco will scan the entire payloads of traffic rather than the first 20KB like Mikrotik would. You usually combine that approach along with DNS filtering to kill DNS over HTTPS, DNS over TLS, and NAT Rewrite rules to redirect standard DNS traffic to something you have control over, where known trackers/etc are blocked. Suricata can also whack known DHT IPs and the IPs of trackers. You also disable the ability to use UPnP so those torrent clients aren't likely to punch holes into the network and start seeding. But yeah, it'd be fairly easy to block P2P traffic with that Dream Machine they have too, although I wouldn't trust that at all to route the LAN.
I'm a college student currently studying for my CCNA and recently discovered and am planning on going to PDXLan this year. This is pretty fascinating stuff! I felt like I could follow along this video well enough but it seems like there's so much more under the hood. I'll be checking out the rest of your channel now!
Hi! I'm with PDXLAN's staff. Be sure to keep an eye out for volunteer signups. This will give you a chance to look behind the scenes and ask the other on staff on how all of this is set up.
No NAT. Going IPv6 removes all of the issues with NAT traversal for peer to peer connections and that whole set of headaches for everyone. Also no issues trying to go from the public wifi to LAN net which are each behind their own NAT, since every v6 address is globally addressable. There are also advantages on the operator side that aren't as visible to participants like: - You can lease less public IPv4 space since most CDN traffic will be v6 and won't consume public IP/port pairs - Less load on the router since most CDN traffic will be v6 and won't go through NAT, which means it's more likely to hardware offload as well depending on the router (this particular router does *not* have L3 offload though) - Uses multicast instead of broadcast for network management which can improve airtime use if you have a large broadcast domain combined with a wireless network, although mdns is usually a way bigger problem than DHCPv4 and friends - Providing modern service instead of legacy-only service to users
@@CrosstalkSolutions Because you build a Network not a Museum. Its not a question of if this network needs it. Its a network, so it by default needs it. And as apalrd mentioned p2p connectivity which is especially important for many games.
I would run 10gb branches to the desktop switchs and put 1gig links for end users with egress and rate limits in place to prevent a single user from bringing down the bandwidth for a table.
I was there.. nice walkthrough. I was hoping to say hi when I heard you were there, as I used your videos to help set up my UDM Pro.. Keep up the good work!!
Regarding the question you received on why only 1Gbps to the distribution switches, I think the driver for that question comes down to a generational gap. The last LAN party I attended was in the early 2000's, where it was truly a party to play multiplayer games and share files over the LAN, not a means for everyone to play disparate games across the WAN while in the same room.
Matt here, event founder. It's actually simpler than that -we aren't playing old games. It's more than most games today need a internet connection to work. We simply don't "need" 10gb to the table for gaming. Sure downloading a game *could* be faster if we allowed 10gb to the table switch, but it comes with the risks then that one single person could saturate the internet. There's also a massive cost-saving here by not going to 10gb distribution - we're a not-for-profit event, we don't have the budget for fancy toys used just 8 days a year :)
@@Vectality Do you pay for the 10Gbps bandwidth or the traffic as well? IMHO you could allow downloads (updates, patches) IF the overall bandwidth does not go too high as to impact latency
@@Vectality There are some pretty cheap ubiquity optical distribution switches that'd allow you a cost effective 10Gbps fibre connection to each switch. I'm in the process of upgrading the network at work and i'll have 2 x 25Gbps aggregated to the 2 hubs from main comms room and 2 x 10Gbps aggregated to each client switch.
The admin told me that he considered doing a cache server (and has done them in the past), but wanted to test maxing out the available bandwidth without one.
@@CrosstalkSolutions I'd second this - We've never run into the problem yet, and when / if we do, we'll adapt. Perhaps the most intense bandwidth we've ever had was when we gave all the gamers in the room Battlefield 4. All of them needed to download the game. We 100% cached that :)
Hi, @@Vectality Would you be able to share about the packetfence+sucircata integration and how do you think and configure the packetfence to be used on those events?
excellent setup for keeping it simple!!! kiss - letting it manage itself by limiting switch uplink speeds and keeping dhcp scope very large!!!! and port mirroring for content "inspection" prevents the content "filtering" from bottlenecking the main setup
Having all the edge switches and clients in the same L2 domain might be easy. But a lot can go wrong. Such as broadcast storms and network loops. You want some e segmentation, I would make every edge switch a /24. Only a few extra commands.
Excellent insights! Always wanted a more in-depth tour of a big LAN event. I spoke to some net admins at DH Dallas and the setup was comporable. However, I've seen photos from older dreamhacks with networks that look like a Cisco Live demo rack!
the 1 gig limitation is their choice..however running a gamecache server pre seeded would have IMO been a better option. force all traffic through that gaming cache and it's controlled well that way..then you could do 10 gig to each switch. Linux tech tips actually demonstrated this and he was able to share his 2 gig link amongst a 50 person lan party he hoswted using a game caching server..:)
@crosstalk - At the very end you mentioned the 1gb links between switches and explained why you thought this made sense. The problem is that 1-2 computers in one row could cause the interswitch link to be saturated. Packets would be dropped between switches and could easily impact gamers in the same row. The least complex ways prevent this is to either (1) set all end user ports to 100mb OR (2) use 10gb connections between switches. Another option is to use LACP to mitigate this if there are sufficient distribution switches. Unless there is a compelling reason to provide more than 100mb bandwidth or ability to get new access switches with 10gb uplinks, it seems that 100mb access ports are the way to go.
3:15 never do big L2 domains, I'd give every row of 48P Switches a /24 and route it appropriately at the Router. If there's a need you can enable HW flow offloading on the mikrotik and do it on any switch itself. Those I'd give another subnet.
Please do a in-depth video of the Suricata setup and rules setup with full details preventing the Torrenting. I would like to implement a similar setup in my own house.
You'd think a huge event like that would be running a lancache server. By having that it would also justify having greater than 1g links back to the core switch.
We haven't had the need actually. The bandwidth we get from Comcast has been enough for our setup. Most of our attendees come with their games pre-installed, and the ones that don't get *up to* 10% of the bandwidth to install it. We've used our own custom LAN-Cache for games that we've given to all attendees at the event, such as Battlefield 4 - we absolutely cached that. Really, we haven't had the need for a LAN Cache, the network has been rock solid without. Actual gaming just doesn't take that much bandwidth.
I'd like to see the more in depth setup on that torrent blocking setup. Would have also been cool if they were using pfsense, or the like, for the main router.
We used to use pfsense, years ago. When we bumped up to a 10gbit internet connection we had to move away as it couldn't sustain 10gbit at line rate. I personally use opnsense at home.
@@ClintWhiteside odd, maybe that was with an older version. I am running 10g right now with no problems. maybe it's because have plus? Glad you got something to work though! :D
@@neosmith80 The key phrase is "line-rate" - a lot of the traffic we really care about is the very small packets games tend to exchange. In our testing (years ago) pfsense could easily handle close to 10gbit w/ large packets but fell apart rather quickly as you started decreasing the packet size. This is easy to test w/ iperf2 (which is multi-threaded). IIRC this was a FreeBSD kernel limitation, there are solutions (do processing in userspace) but I think Netgate turned that into their TNSR product instead of improving pfsense, though I no longer pay any attention to Netgate/pfsense so my information is likely out of date.
More advanced bandwidth management is needed - IF someone needs a windows or game update done he/she will be limited for no reason even if there are e.g. 9 Gbps available. Bandwidth should be used up till the latency becomes too bad for good gameplay, and with fiber that usually is in the 90%+ range
I won't lie, I had to chuckle that every answer he gave boiled down to, "you aren't a network administrator so you don't understand this is why we do it."
Hm I disagree with their choice of 1g uplinks from access to core switches. I would've done 10g links then on the firewall I would guarantee each device a minimum amount of bandwidth. This way people can share files between each other AND users have the ability to download much faster if needed without jeopardizing everyone else's guaranteed minimum speeds.
Agreed. I would either do 10gb between switches, or for even better QoS on a tight budget just set all edge port speeds to 100mb. At least one other commenter here (kinsel) said that's what they do at their event.
The core switches and the table switches are only on 1Gbit because their internet line isn't fast enough anyway, but if PDX ever wants to use LanCache, they would need to upgrade their entire network for this to work properly, well, maybe not the core switch, but still a large portion of switches and cables.
Yeah. Good video! When I game, my internet usage can be measured well below 1 Mb. You don’t need 500Gb service at that event. You just need to set it up in a smart way so everyone gets a piece of the internet pie!!
One interesting thing to me those looked like Cisco 5K and fexes (but didn’t see the switch at the table). That technology the sat switches are dumb, and don’t even do local switching. Every packet must go up to the distribution (Cisco 5k I assume). So that may really push a need for 10G at the tables, but obviously the team seems to monitor it, and if not a problem I agree, keep it simple!
Good eye. We only use a handful of FEXes and they typically have a pretty wide link back to the Nexus. This allows us to, for larger events, place the FEXes more centrally, run fiber back to the big noisy Nexus and servers, and reduce the length of our worst-case ethernet runs.
i'd not want nexus / fex at the tables for noise reasons. rather 4 tp-link 24 ports with 2/4x10g up instead. for bargain core I'd take two used mellanox msx1024 in mlag...
I'm running a /22 in my home for my main network. My other vlans for Iot/guest/private is a /24. I have a protect vlan (/28) for my protect cams and NVR that only talk to each other, NO internet. My NVR, using 10gb sfp+ for protect vlan and using the rj45 port for IoT vlan internet connection. Isolating my cameras
By limiting switch uplink speeds from table to core and keeping DHCP pool very large; guess you are keeping with kiss method. Port mirroring for content mitigation to prevent the content filter from limiting the CORE is moving in the right direction. At this point if an event is not running a simple lancache on the LAN it just brings up questions. 10 gig to the table switch is rather low cost these days but I understand events have overhead. But saying because you can doesn't mean you should is a no go that is not the answer to the PDXLAN gig limit to table they just don't have the switches that can do it. If it is the same table switches they have had for the last 6 years then of course there is a limit. With that said if they upgraded this year then everything I just said means nothing. That is inside baseball for another day. The PDXLAN folks are super awesome. Have had fun working with them in the past. _________________________________ TH-cam comments no one cares.
Exactly my thoughts... so many useless bottlenecks. One gigabit link to 48 users connected via gigabit?? must be so damn overloaded when someone decide to download some game (specialy when group sitting next to each other decide to play something)... No wonder they were unable to utilize 10gbit internet. I do smaller events for 60 people, with 10gig uplinks between switches and local lancache, we download about 3TB of data from internet (half of it from lancache!). You can also easily solve traffic issues directly in mikrotik, using simple queues and traffic prioritization...
right, why not just run the cable to the switch... they had enough cable to do that. seems kinda silly to me! Plus, and just my 2c, i have never had a good experience with rj45 couplers.
Watched whole video and like it. You explained it well but that brings so much questions... I dont hear anything about caching server of any like Steam or something. Without it 10 Gbps can be saturated easily :) I have multiple 10 Gbps connections at my home and do share them with couple friends in my neighborhood... I happens couple of times to download a single game at same time with my friend to play and we can achieve near 200-300 MB/s for each of us trough single connection... Yes i have multiple but it is fine... Imagine what will happen if there are 20, 30 or 50 peopla who want to download particular game at the same time :) So that about 1 Gbit for 24 people is unnapropriate as well ;) There are so much things they can do better and they are even easier for doing than what they did like suricata, VLAN, etc. :)
If you're available in January check out LANWar in Louisville, KY. They've been hosting large LAN parties for over 20 years and have had over 1,000 gamers at many events.
That WiFi setup sucks badly. Port mirrored traffic is monitoring only, not filtering. 1GbE backhaul from table switches is bad planning, especially when the core switch is 10GbE, I know what you said Chris but no redundancy and sounds like a cover for the before mentioned. No gaming servers seen, looks to be a internet-only LAN Party which means patching, new games and other data must come from WAN instead of local cached data which comes back to better interconnect speeds. No firewalling, just a router.
Why would you need a guide for the users? Just don't have ipv6 enabled for them and ensure they can't use other DNS servers via outgoing nat rules. (yes I'm aware that you can't prevent DNS over tls etc but that shouldn't be a problem for all of the various launcher)
If you were to set this up yourself, would you have even considered using Unifi switches and gateways? Or do you think it would have just been too much for those devices?
1 thing that so many get wrong about the torrenting thing is that your ISP does not detect you. the rights holders do and then complain to the ISP. in reality your ISP really couldn't care less about what you are doing on the internet as long as it doesn't a cause a problem for them and b doesn't cause another company to reach out to them.
What about torrenting for game patching ? World of Warships and a few other games use torrent protocol to do their updates. Do you use any form for game patch caching like lancache?
Heya, I wanted to thank you for the video of our event! It was great seeing you there! For those wanting to learn more about what makes PDXLAN special (Besides networking) check this out: th-cam.com/video/pNgF_vL5f60/w-d-xo.html
Why did they need larger than a /24? There’s no game that’s out there that doesn’t expect to route a end point out to a different subnet. Maybe back in the old StarCraft 1 days, where looking for a local server required it to be on the same vlan, but that shouldn’t be a thing now.
Question from someone that knows very little. Can the table play with other tables? That would be my only worry about only running 1Gb links. If someone is downloading a game maxing out the link would it not affects others lan to lan link?
i am surprised they did not have a cache server. Usually big LAN party's they install a steam cache server to off load game downloads and updates to a local server instead of having all the the client to go on the internet to get the files.
The admin told me he considered a cache server, but wanted to see what the 10Gbps pipe could do without caching. Turns out they never really came close to maxing it out.
Something I wonder is whether they used something like LanCache for caching Steam downloads. I mean, imagine a few hundred gamers trying to download the same game at the same time... Kind of a waste of bandwidth right?
I don''t agree with the bandwidth usage explanation of torrrenting. Especially since newer files of filesharing and downloading use cdn's which can push out more bandwidth than the 10gb can handle. On the surface torrenting looks like it uses a lot of bandwidth while a properly configured fileshare like Steam can use nearly as much. The "torrenting uses a lot of bandwidth" really comes from the old days where servers were the main bandwidth limit of connections and torrents were able to bypass that since it uses more peers than over http.
I can tell you from experience... you KNOW when someone at your table starts a client that uses peer-to-peer sharing. The entire table let's out a collective WTF as latency skyrockets. It's NOT fun.
Our main concern here is getting our 1 internet IP banned during the event. That'd be a show stopper. Add to that the legal implications for the venue.
@@Vectality I understand that concern and it's a valid one especially in the US. My comment was about the explanation of the bandwidth implications of bittorrent
better off to host dlc/mods locally than let lan saturate the link to the internet you can 100 Gb wan service back bone on the event and the local servers host that to the lan..
I have a question for you: What is the reason for two U6 mesh points being side by side? I thought they're omni directional. Also, how was the signal strength and range of the units?
I was using the mesh points on my phone throughout the event. The mesh was in the back, while I was sitting in the first row, up by the stage. I have four bars the entire time, even in the bathrooms.
I'd love to see a more in depth video on the torrent blocking setup
+1
+1
+1, anybody know how it is spelled? i'd love to look it up myself
@@dominikkusber5764 Suricata, The OG version is called Snort. I'm interested in the isolation aspect of the actions from suricata
Hi! Along with Clint, who posted down below, we're the network admins for PDXLAN, so can provide some basic info on this.
Suricata doesn't do the port isolation, it only monitors and detects the P2P traffic, the 'heavy lifting' is done with software called Packetfence, which also handles the captive portal. Packetfence uses the info from Suricata, along with the attendee's login information to map them to their specific switch port, then shuts the port, moves it to the 'naughty users' vlan, then unshuts the port, so they get the p2p warning and remediation page. Packetfence is very capable, and very frustrating software, but it's the best cheap solution we've come up with so far.
The discussion has been had amongst staff to develop our own software that does the same thing as what we use Packetfence for, but it hasn't really gotten past the discussion stage, since Packetfence works, even if it's not perfectly ideal for our use.
I am networking staff for an event very similar to PDXLAN, although of about half the size. The architecture of PDXLAN is pretty similar to the network I'm currently operating. You do hit a few points on the head of the nail when it comes to bandwidth distribution and network quality, and there's a few others I wanted to chime in about as well.
The network I administer operates at Layer 3 in the core, Layer 2 at the edge, with each "section" of the LAN being on its own Layer 3 Segment. Generally speaking, this has pros and cons. The biggest Pro to this is, in the event something goes wrong with a subnet, such as a jabbering NIC in a PC, someone creating a Broadcast or Multicast storm which isn't caught correctly by the network switches, a network switch fault, or in the event of say, a malware attack, problems with the network stay isolated to just that Layer 3 segment. Instead of having 600 people experience problems, maybe 40 people experience a problem. It's much less stress on networking staff when something goes wrong because of a client, and you get far less tickets about "lag spikes" in games. There have been numerous instances where, once you start pushing into 200+ person territory, the attachment of say, NVIDIA Shields or game consoles to the network, or the amount of Steam Server Discovery / SMB / Bonjour discovery traffic, starts to overwhelm the NICs in computers with multicast and broadcast traffic which must be discarded. You don't always get the choice of using hardware which can perform Multicast snooping or Storm limiting, either... at least not without cost (Juniper... grrrr).
The drawback to using small subnets is extra administrative overhead, so organization is key. But the other drawback is, you lose the ability to do peer-to-peer discovery when a friend in one section wants to host a game server on their PC. Some games only treat the local subnet as "LAN" and anything else is considered "Internet" so, that can make joining your friend's game a little less easy. Source games are usually well behaved. Other games, not so much. But this is quickly becoming a moot point as many popular games today ONLY support the use of Online servers provided by the publisher.
The other thing the network I operate does, is on-network caching for Steam, Origin, Blizzard, Windows Update etc. Using DNS hints, intercepts and spoofing, clients when downloading game content are directed to the on-network cache. The on-network cache performs Man-in-the-middle proxying of content for common gaming downloads and traffic, and this saves TERABYTES of traffic each event from hitting the Internet connection. So if say, someone downloads Call of Duty which is 140GB, and another friend decides to download it too, the Internet connection only eats that 140GB download once. The next download is handled on-network. It's also been extremely helpful when VALVe releases CS:Go updates mid-event, or there's a big Blizzard update. It takes only one person downloading the update to cache it for everyone else, and as I said, literal Terabytes of bandwidth are saved. You can use actual server hardware or, even Intel NUCs loaded up with large SSDs and Linux.
10Gbps Internet from a venue gets to be pretty pricy sometimes, so whatever you can do to save on bandwidth costs is key. I generally work with a 1Gbps connection, and bandwidth shaping/queue adjustments in Mikrotik or whatever you end up using for an Internet router is key. I also find that having less hardware in service helps with reliability. Virtualizing the UniFi Controller for example, is something else I do if I am using UniFi APs. If it breaks, you restore from a known snapshot in seconds.
For Torrenting, definitely all of the above. Copyright concerns, bandwidth concerns, but the other thing you forgot to mention is NAT Exhaustion. Most of these LANs aren't using IPv6, and they may not have multiple WAN IPs to pick from. Torrents consume a lot of connections and when you have hundreds or thousands of people trying to pipe through a single connection, at some point torrenting will overwhelm the number of connections which can be tracked by upstream equipment (some ISPs like AT&T have BUGGY CPE), or placed into the NAT. Mikrotik does P2P filtering and L7 however, compared to Suricata, it's nowhere near as good and it has many drawbacks. For example, P2P clients using UDP or encryption will thwart it, and blocking UDP ports has many, many consequences for applications like Discord, games, etc.
> the other drawback is, you lose the ability to do peer-to-peer discovery when a friend in one section wants to host a game server on their PC. Some games only treat the local subnet as "LAN" and anything else is considered "Internet" so, that can make joining your friend's game a little less easy
This is the main reason we have kept things to a single broadcast domain, there are still a number of classic titles commonly played within our community and at our events. In fact we intend to run some of these as "throwback" tournaments for our 20th anniversary event next year.
Instead of breaking this experience we instead do some targeted broadcast blocking to make sure we are limiting some of the more prolific broadcast spam from influencing the entire network, without killing all broadcast traffic. We have found that there is a pretty short list of major offenders that we aggressively don't care about and that, when blocked at the switch level (max of 24 users), will bring network-wide broadcast traffic down to a manageable level. Blocking MDNS hasn't been on the table yet, though the exceptionally noisy Oculus client will be what eventually drives us to this. SMB broadcast traffic also isn't huge by itself, even with 1000 clients.
Our main consideration, that I think most people miss, with game caching is the resources necessary to compete with a 10g internet pipe. You need some serious iops and storage to be as fast or faster with more than a handful of users. It is definitely something we could do, we have in the past when our connection was more limited, but the barrier to actually improving our experience with a cache is more expensive than tossing a couple consumer-grade SSDs in a box.
I like wires ,
Interesting insights into the technical details of a network admin for Game Center’s very cool! I would love to know how the man in the middle caching proxy works to save on downloading updates!
@@Knightfall23 look for this video in the Linus Tech Tips youtube channel "We’re running out of internet - Steam Game Caching Server"
Epic win ;)
8:30 I'm a network engineer at an ISP. I recently worked on the circuit for a well known streamer. While on stream I was curious so I looked up their current utilization. It was less than 5mbps. Gaming and even streaming doesn't use a ton of bandwidth, you just want lower latency.
people at this kind of events may need to download entire games or updates of them
@@marcovillalobos9762 This setup was in no way ideal, but "works" I'd have expected beefy caching servers for games
@@forid200 Yeah I was expecting a handful of game cache servers, a la PAX West a few years back where Quaid was the network admin for the whole event.
@@marcovillalobos9762 LTT did have a similar event, but they made a workaround.
@@forid200 Same as LTT did in a similar event.
The last lan I was at in Philly with 500 people had someone torrent and comcast threatened to turn off the 10g internet within a few hours of it occurring. The exact scenario you outlined.
Yup. With games these days needing to be on an internet connection, this is very scary - the idea you could have your connection shut off at the event. Very scary for the event owner :)
I heard about the Philly one. I’m from jersey but I was to late.
Super easy to block P2P with that Mikrotik router they're using.
@@TannerBugatti it's got limits though. Even with L7 inspection rules, any torrent client encrypting traffic or using UDP will skirt right around it. Going any harder starts to break games and Discord, and torrent clients can simply masquerade the traffic once again with what works.
Usually you would use an IDS solution like Suricata which can adaptively look at traffic and whack it. More premium solutions like what you find in a Juniper or Cisco will scan the entire payloads of traffic rather than the first 20KB like Mikrotik would.
You usually combine that approach along with DNS filtering to kill DNS over HTTPS, DNS over TLS, and NAT Rewrite rules to redirect standard DNS traffic to something you have control over, where known trackers/etc are blocked. Suricata can also whack known DHT IPs and the IPs of trackers. You also disable the ability to use UPnP so those torrent clients aren't likely to punch holes into the network and start seeding.
But yeah, it'd be fairly easy to block P2P traffic with that Dream Machine they have too, although I wouldn't trust that at all to route the LAN.
I'm a college student currently studying for my CCNA and recently discovered and am planning on going to PDXLan this year. This is pretty fascinating stuff! I felt like I could follow along this video well enough but it seems like there's so much more under the hood. I'll be checking out the rest of your channel now!
Hi! I'm with PDXLAN's staff. Be sure to keep an eye out for volunteer signups. This will give you a chance to look behind the scenes and ask the other on staff on how all of this is set up.
10Gb link, 72-core router, Suricata, 48-port switches, all that stuff...
...
...
...
...
...
and no IPv6 🙄
Explain why you feel this network needed IPv6 please.
No NAT. Going IPv6 removes all of the issues with NAT traversal for peer to peer connections and that whole set of headaches for everyone. Also no issues trying to go from the public wifi to LAN net which are each behind their own NAT, since every v6 address is globally addressable.
There are also advantages on the operator side that aren't as visible to participants like:
- You can lease less public IPv4 space since most CDN traffic will be v6 and won't consume public IP/port pairs
- Less load on the router since most CDN traffic will be v6 and won't go through NAT, which means it's more likely to hardware offload as well depending on the router (this particular router does *not* have L3 offload though)
- Uses multicast instead of broadcast for network management which can improve airtime use if you have a large broadcast domain combined with a wireless network, although mdns is usually a way bigger problem than DHCPv4 and friends
- Providing modern service instead of legacy-only service to users
@@CrosstalkSolutions Because you build a Network not a Museum. Its not a question of if this network needs it. Its a network, so it by default needs it. And as apalrd mentioned p2p connectivity which is especially important for many games.
@@CrosstalkSolutions IPv6 is global. Future is now oldman.
You are using an outdated technology (IPv4), stop making videos if you are not good at networking. @@CrosstalkSolutions
I would run 10gb branches to the desktop switchs and put 1gig links for end users with egress and rate limits in place to prevent a single user from bringing down the bandwidth for a table.
That what my thoughts were.
I wanna host a 12 - 16 player LAN party in the future and will likely do something similar.
I was there.. nice walkthrough.
I was hoping to say hi when I heard you were there, as I used your videos to help set up my UDM Pro..
Keep up the good work!!
WOW. I've been to previous PDXLAN parties, and I had NO IDEA it was this complex, but makes total sense!
Regarding the question you received on why only 1Gbps to the distribution switches, I think the driver for that question comes down to a generational gap. The last LAN party I attended was in the early 2000's, where it was truly a party to play multiplayer games and share files over the LAN, not a means for everyone to play disparate games across the WAN while in the same room.
Matt here, event founder.
It's actually simpler than that -we aren't playing old games. It's more than most games today need a internet connection to work. We simply don't "need" 10gb to the table for gaming. Sure downloading a game *could* be faster if we allowed 10gb to the table switch, but it comes with the risks then that one single person could saturate the internet.
There's also a massive cost-saving here by not going to 10gb distribution - we're a not-for-profit event, we don't have the budget for fancy toys used just 8 days a year :)
@@Vectality Do you pay for the 10Gbps bandwidth or the traffic as well? IMHO you could allow downloads (updates, patches) IF the overall bandwidth does not go too high as to impact latency
What kind of question is that, do you even understand how bandwidth works?
@@jfkastner One of the sponsors is Comcast. So Comcast is providing all the expensive "bits" :)
@@Vectality There are some pretty cheap ubiquity optical distribution switches that'd allow you a cost effective 10Gbps fibre connection to each switch. I'm in the process of upgrading the network at work and i'll have 2 x 25Gbps aggregated to the 2 hubs from main comms room and 2 x 10Gbps aggregated to each client switch.
I would love to see a suricata walkthru..
please make a video on the Suricata integration
Please do a video with the layer 7 redirect.
That would be awesome!!
Thanks for the deep dive on the setup. I would have thought they were using a cache server for the games.
The admin told me that he considered doing a cache server (and has done them in the past), but wanted to test maxing out the available bandwidth without one.
@@CrosstalkSolutions I'd second this - We've never run into the problem yet, and when / if we do, we'll adapt.
Perhaps the most intense bandwidth we've ever had was when we gave all the gamers in the room Battlefield 4. All of them needed to download the game. We 100% cached that :)
Hi, @@Vectality Would you be able to share about the packetfence+sucircata integration and how do you think and configure the packetfence to be used on those events?
Fook yeah! Thank you for all the content! You are THE plethora of knowledge.
I would like to hear more about the traffic monitoring and redirects. Sounds really interesting to me!
would love to see a full, detailed walkthrough of how to build the graphana services and how you setup the discord alerts and such. Very cool stuff
thanks for helping keep the lan tradition alive
SUPER interested in Suricata video. But could you do it in pfsense?
excellent setup for keeping it simple!!! kiss - letting it manage itself by limiting switch uplink speeds and keeping dhcp scope very large!!!! and port mirroring for content "inspection" prevents the content "filtering" from bottlenecking the main setup
Having all the edge switches and clients in the same L2 domain might be easy. But a lot can go wrong. Such as broadcast storms and network loops. You want some e segmentation, I would make every edge switch a /24. Only a few extra commands.
Would love a in depth video about the p2p detection and automation of end users to different VLANs
Up Vote - Video on Suricata
I was on setup team for this event. I no longer enjoy crimping ends :P
Don't lie, we know you'll be back volunteering in Spring for the 1,000 person one :)
*cue Rocky training montage*
A bit surprised something like LAN cache wasn't in place as there would be allot of the same game being downloaded.
Excellent insights! Always wanted a more in-depth tour of a big LAN event. I spoke to some net admins at DH Dallas and the setup was comporable. However, I've seen photos from older dreamhacks with networks that look like a Cisco Live demo rack!
the 1 gig limitation is their choice..however running a gamecache server pre seeded would have IMO been a better option. force all traffic through that gaming cache and it's controlled well that way..then you could do 10 gig to each switch. Linux tech tips actually demonstrated this and he was able to share his 2 gig link amongst a 50 person lan party he hoswted using a game caching server..:)
@crosstalk - At the very end you mentioned the 1gb links between switches and explained why you thought this made sense. The problem is that 1-2 computers in one row could cause the interswitch link to be saturated. Packets would be dropped between switches and could easily impact gamers in the same row.
The least complex ways prevent this is to either (1) set all end user ports to 100mb OR (2) use 10gb connections between switches. Another option is to use LACP to mitigate this if there are sufficient distribution switches. Unless there is a compelling reason to provide more than 100mb bandwidth or ability to get new access switches with 10gb uplinks, it seems that 100mb access ports are the way to go.
NOW thats some hardware PRON !! NIce !!
3:15 never do big L2 domains, I'd give every row of 48P Switches a /24 and route it appropriately at the Router. If there's a need you can enable HW flow offloading on the mikrotik and do it on any switch itself. Those I'd give another subnet.
As a LAN admin: I'd love to know how the Suricata implementation worked :)
Please do a in-depth video of the Suricata setup and rules setup with full details preventing the Torrenting. I would like to implement a similar setup in my own house.
You'd think a huge event like that would be running a lancache server. By having that it would also justify having greater than 1g links back to the core switch.
We haven't had the need actually. The bandwidth we get from Comcast has been enough for our setup. Most of our attendees come with their games pre-installed, and the ones that don't get *up to* 10% of the bandwidth to install it. We've used our own custom LAN-Cache for games that we've given to all attendees at the event, such as Battlefield 4 - we absolutely cached that.
Really, we haven't had the need for a LAN Cache, the network has been rock solid without. Actual gaming just doesn't take that much bandwidth.
I'd like to see the more in depth setup on that torrent blocking setup.
Would have also been cool if they were using pfsense, or the like, for the main router.
Me too
We used to use pfsense, years ago. When we bumped up to a 10gbit internet connection we had to move away as it couldn't sustain 10gbit at line rate. I personally use opnsense at home.
@@ClintWhiteside odd, maybe that was with an older version. I am running 10g right now with no problems. maybe it's because have plus? Glad you got something to work though! :D
@@neosmith80 The key phrase is "line-rate" - a lot of the traffic we really care about is the very small packets games tend to exchange. In our testing (years ago) pfsense could easily handle close to 10gbit w/ large packets but fell apart rather quickly as you started decreasing the packet size. This is easy to test w/ iperf2 (which is multi-threaded).
IIRC this was a FreeBSD kernel limitation, there are solutions (do processing in userspace) but I think Netgate turned that into their TNSR product instead of improving pfsense, though I no longer pay any attention to Netgate/pfsense so my information is likely out of date.
More advanced bandwidth management is needed - IF someone needs a windows or game update done he/she will be limited for no reason even if there are e.g. 9 Gbps available. Bandwidth should be used up till the latency becomes too bad for good gameplay, and with fiber that usually is in the 90%+ range
Amazing resoning and explinations. Thanks for the awesome video!
really looking forward to the video on the Suricata server.
I’m interested in the suracata setup
I won't lie, I had to chuckle that every answer he gave boiled down to, "you aren't a network administrator so you don't understand this is why we do it."
Nice video, well done! It would be great to hear more on the port mirroring and P2P blocking setup and on the discord alerts setup
Hm I disagree with their choice of 1g uplinks from access to core switches. I would've done 10g links then on the firewall I would guarantee each device a minimum amount of bandwidth.
This way people can share files between each other AND users have the ability to download much faster if needed without jeopardizing everyone else's guaranteed minimum speeds.
Agreed. I would either do 10gb between switches, or for even better QoS on a tight budget just set all edge port speeds to 100mb. At least one other commenter here (kinsel) said that's what they do at their event.
You might want to take a look at the Chaos Communication Congress setups
The core switches and the table switches are only on 1Gbit because their internet line isn't fast enough anyway, but if PDX ever wants to use LanCache, they would need to upgrade their entire network for this to work properly, well, maybe not the core switch, but still a large portion of switches and cables.
Yeah. Good video! When I game, my internet usage can be measured well below 1 Mb. You don’t need 500Gb service at that event. You just need to set it up in a smart way so everyone gets a piece of the internet pie!!
One interesting thing to me those looked like Cisco 5K and fexes (but didn’t see the switch at the table). That technology the sat switches are dumb, and don’t even do local switching. Every packet must go up to the distribution (Cisco 5k I assume). So that may really push a need for 10G at the tables, but obviously the team seems to monitor it, and if not a problem I agree, keep it simple!
Good eye. We only use a handful of FEXes and they typically have a pretty wide link back to the Nexus. This allows us to, for larger events, place the FEXes more centrally, run fiber back to the big noisy Nexus and servers, and reduce the length of our worst-case ethernet runs.
i'd not want nexus / fex at the tables for noise reasons. rather 4 tp-link 24 ports with 2/4x10g up instead. for bargain core I'd take two used mellanox msx1024 in mlag...
I'm running a /22 in my home for my main network. My other vlans for Iot/guest/private is a /24. I have a protect vlan (/28) for my protect cams and NVR that only talk to each other, NO internet. My NVR, using 10gb sfp+ for protect vlan and using the rj45 port for IoT vlan internet connection. Isolating my cameras
You just got yourself a subscriber my friend
Great video, thank you!
Great video and would love to see a more in-depth vid about torrent blocking and seracotta
When performance is needed all of them defaults to MikroTik.
By limiting switch uplink speeds from table to core and keeping DHCP pool very large; guess you are keeping with kiss method. Port mirroring for content mitigation to prevent the content filter from limiting the CORE is moving in the right direction. At this point if an event is not running a simple lancache on the LAN it just brings up questions. 10 gig to the table switch is rather low cost these days but I understand events have overhead.
But saying because you can doesn't mean you should is a no go that is not the answer to the PDXLAN gig limit to table they just don't have the switches that can do it. If it is the same table switches they have had for the last 6 years then of course there is a limit. With that said if they upgraded this year then everything I just said means nothing. That is inside baseball for another day. The PDXLAN folks are super awesome. Have had fun working with them in the past.
_________________________________
TH-cam comments no one cares.
Exactly my thoughts... so many useless bottlenecks. One gigabit link to 48 users connected via gigabit?? must be so damn overloaded when someone decide to download some game (specialy when group sitting next to each other decide to play something)... No wonder they were unable to utilize 10gbit internet. I do smaller events for 60 people, with 10gig uplinks between switches and local lancache, we download about 3TB of data from internet (half of it from lancache!). You can also easily solve traffic issues directly in mikrotik, using simple queues and traffic prioritization...
No one is gonna talk about those RJ45 Couplers what a mess 🤪
right, why not just run the cable to the switch... they had enough cable to do that. seems kinda silly to me! Plus, and just my 2c, i have never had a good experience with rj45 couplers.
Ha! This is funny.
Comes down to someone wanting to make it "look pretty" :)
Well looks like basic and much simpler than even my home network. Dreamhack like it was 15 years ago :)
Hi Chris, great video.
I see your Definitive Guide to Hosted UniFi 2021and is great. Do you plan a newer guide for 2022?
Thank's.
More interested in the Cisco switch, was it a Nexus 9k?
I would love to see the suricata video!
More in depth info on the Torrenting blocking, plx 🙏
Watched whole video and like it. You explained it well but that brings so much questions... I dont hear anything about caching server of any like Steam or something. Without it 10 Gbps can be saturated easily :) I have multiple 10 Gbps connections at my home and do share them with couple friends in my neighborhood... I happens couple of times to download a single game at same time with my friend to play and we can achieve near 200-300 MB/s for each of us trough single connection... Yes i have multiple but it is fine... Imagine what will happen if there are 20, 30 or 50 peopla who want to download particular game at the same time :) So that about 1 Gbit for 24 people is unnapropriate as well ;) There are so much things they can do better and they are even easier for doing than what they did like suricata, VLAN, etc. :)
Really interested on the suricata videos
I'd love to see a tutorial about Cerakata from Crosstalk solutions.
WAIT? You were there? Would have liked to say hello!
Very cool! Thank you!
am interested in knowing how suricata, vlan moving and automating the whole thing was done... can you shed some light on it?
Great video.
I need torrents for all my Linux ISOs.
Wide open subnet, grab yourself you’re own IP in a VM and see how to avoid the captive portal
Wouldn't you rather just have fun a the event though?
If you're available in January check out LANWar in Louisville, KY. They've been hosting large LAN parties for over 20 years and have had over 1,000 gamers at many events.
That WiFi setup sucks badly. Port mirrored traffic is monitoring only, not filtering. 1GbE backhaul from table switches is bad planning, especially when the core switch is 10GbE, I know what you said Chris but no redundancy and sounds like a cover for the before mentioned. No gaming servers seen, looks to be a internet-only LAN Party which means patching, new games and other data must come from WAN instead of local cached data which comes back to better interconnect speeds. No firewalling, just a router.
I liked this video a lot
Very interesting! Thanks
Yes please explain the user redirects for torrenting.
I am interesting to see a video about suricata.
Welcome to Scandinavia. This is a medium sized lan party. Nice video though 👍
I'd love a more indepth video on Torrent and other app blocking
10Gb links to satellite switches and use a lancache? Ofc needs a guide for users but could limit the Internet bandwidth requirements.
See LAN Cache comments above - we've just not had the need for it in most cases.
Why would you need a guide for the users? Just don't have ipv6 enabled for them and ensure they can't use other DNS servers via outgoing nat rules. (yes I'm aware that you can't prevent DNS over tls etc but that shouldn't be a problem for all of the various launcher)
If you were to set this up yourself, would you have even considered using Unifi switches and gateways? Or do you think it would have just been too much for those devices?
yes we are interesting Suricata.
1 thing that so many get wrong about the torrenting thing is that your ISP does not detect you. the rights holders do and then complain to the ISP. in reality your ISP really couldn't care less about what you are doing on the internet as long as it doesn't a cause a problem for them and b doesn't cause another company to reach out to them.
What about torrenting for game patching ? World of Warships and a few other games use torrent protocol to do their updates.
Do you use any form for game patch caching like lancache?
I'm a network admin major and it's very interesting to hear the terms I'm learning in an actual scenario. This is awesome man.
Heya, I wanted to thank you for the video of our event! It was great seeing you there!
For those wanting to learn more about what makes PDXLAN special (Besides networking) check this out: th-cam.com/video/pNgF_vL5f60/w-d-xo.html
Would you be able to share about the packetfence+sucircata integration and how do you think and configure the packetfence to be used on those events?
im interested in how the thing worked
Would love to see that surracata video
Why did they need larger than a /24? There’s no game that’s out there that doesn’t expect to route a end point out to a different subnet.
Maybe back in the old StarCraft 1 days, where looking for a local server required it to be on the same vlan, but that shouldn’t be a thing now.
Ohhhh pleeeeease do another video telling us how they do it 🙏🙏
And thank you so much for these knowledge🥹🥹😍😍
The organizers blocked P2P file sharing/torrenting for 3 reasons:
1. Bandwidth
2. Copyright/Intellectual property
3. Network security
Question from someone that knows very little. Can the table play with other tables? That would be my only worry about only running 1Gb links. If someone is downloading a game maxing out the link would it not affects others lan to lan link?
i am surprised they did not have a cache server. Usually big LAN party's they install a steam cache server to off load game downloads and updates to a local server instead of having all the the client to go on the internet to get the files.
The admin told me he considered a cache server, but wanted to see what the 10Gbps pipe could do without caching. Turns out they never really came close to maxing it out.
@@CrosstalkSolutionsHow much would caching benefit regardless? 🤔
Something I wonder is whether they used something like LanCache for caching Steam downloads.
I mean, imagine a few hundred gamers trying to download the same game at the same time... Kind of a waste of bandwidth right?
1+ we like to see the video how Suricata does that, thank you.
Im intrested to see a Vid on the P2P Or torrenting
mikrotik router can block torrenting site but too much cpu resources because using layer7 protocol
How much is worth all that equipment?
Portland and gamers. I can smell it from here.
I don''t agree with the bandwidth usage explanation of torrrenting. Especially since newer files of filesharing and downloading use cdn's which can push out more bandwidth than the 10gb can handle. On the surface torrenting looks like it uses a lot of bandwidth while a properly configured fileshare like Steam can use nearly as much. The "torrenting uses a lot of bandwidth" really comes from the old days where servers were the main bandwidth limit of connections and torrents were able to bypass that since it uses more peers than over http.
I can tell you from experience... you KNOW when someone at your table starts a client that uses peer-to-peer sharing. The entire table let's out a collective WTF as latency skyrockets. It's NOT fun.
Our main concern here is getting our 1 internet IP banned during the event. That'd be a show stopper. Add to that the legal implications for the venue.
@@Vectality I understand that concern and it's a valid one especially in the US. My comment was about the explanation of the bandwidth implications of bittorrent
Yes require the suricata
Using WAN load balancing on the mikrotik router ??
better off to host dlc/mods locally than let lan saturate the link to the internet you can 100 Gb wan service back bone on the event and the local servers host that to the lan..
which open source atp tool is used there?
I have a question for you: What is the reason for two U6 mesh points being side by side? I thought they're omni directional.
Also, how was the signal strength and range of the units?
I was using the mesh points on my phone throughout the event. The mesh was in the back, while I was sitting in the first row, up by the stage. I have four bars the entire time, even in the bathrooms.
@@CyberCrist Ok, TMI Cyber. :)
They were just there for guest wireless - two for redundancy, and on different wireless channels. Plenty of coverage for a big wide open room.