Interesting to see that after 10+ years Extreme is still having the same annoying bugs with stacks, snmp, loops which made us remove all E switches out of the network, we never looked back. But it in the end you have to love your job and use the equipment you have to deal with, thanks for the nice vids again! Take care!
In your setup, I would presume you are running MSTP, BPDU Guard, and similar features should help prevent loops. If a loop does occur, these settings typically trigger protections like blocking ports or disabling connections. To gather more details, you can run the -show spanning-tree summary- command.. at least on a Cisco switch. This will give you a snapshot of the spanning tree status and help identify any potential loop-related problems.
Yep, it was foggy in Modesto, CA area yesterday morning as well. I too hate upgrading firmware on production network switches but these days given the security vulnerabilities and bug issues we have to do them whenever possible. My biggest fear is total meltdown on failed firmware upgrades so I try to do them off hours and weekends. In your case you don't have that luxury for 24/7 hour operation. Also, the vendor should have still have helped you unless the firmware is several versions old which sounds like in this case. Hope you feel better soon!
Before I started watching your videos, I didn’t realize there were still any saints left in CA. 😂 Regarding upgrading-is there just one core switch or is it actually a redundant pair? If a pair, seems like you could take one offline to upgrade, then when it comes back up, do the other one and voila! But I know in real life it isn’t always that simple haha. 😅 As far as network loops, usually your spanning tree statistics will give you a good indication of if/where.
Are those the fibre pairs where you use SM SFP adapters with MM cables? You should check if cables and sfp adapters match up. No hate, just an advice. Regarding loop. Dont you use STP? Also what you can test. 1st of all allow SNMP gets from a different host. And when the switch goes red in your management software, do a snmpwalk to the switch from an other host (that was whitelisted before). If you get the data out without timeout, it is your management software or the windows base. If not, it may be a loop or something else. I suggest using not windows for the snmpwalk, because maybe MS broke something too.
Honestly I don't blame people wanting to run the latest version of the software. It eliminates so many potential causes (not to mention high uptime) and gives them more ammunition with development if it is in fact a bug. After all, I have no issues having end users reboot machines, check for updates, etc etc before spending a whole lot of brain energy on a weird problem. It has saved me tons of time before and will save me tons of time again.
You need to look at your spanning-tree instance(s). Network loops are impossible with properly implemented spanning-tree. You may be dealing with a spanning-tree instability. What variant of spanning-tree does your network use? Which switch is the spanning-tree root bridge? Which one is the backup for that? If you don't know the answers to these questions then you have been ignoring spanning-tree at your own peril. Hope you are feeling better soon.
Re: 'are you running the latest version'. I know this is frustrating for the customer, and I know how it can cause alot of pain. But the support people aren't wrong. I work in a similar field, and the number of fixes that go into a new version is HUGE. There's no way for a single support person to be able to catch each and every fix and fully understand the implications. So ya, VERY often, not just 50/50, but more like 90% of the time upgrading the firmware/software will fix the issue. And consider the other side: if updating fixes the issue, every single customer you try to help with that issue is basically wasted time for the support agent. Nothing more frustrating that discovering the problem you've been working on with a customer for a week is something already fixed.
If the STP protocols are doing their job correctly you really shouldn't have a loop and any switch in an enterprise like yours almost certainly has it enabled by default.
I have no clue why this video was recommended to me. But can say the entire 15:15 clearly showed absolutely no basic OSI trouble shooting?!? 🤦♂️ All I kept hearing is firmware this, SNMPv3 down! 🤢 What does the verbose logs show on the core (problem) switch??? Later I hear I’m only one person? Than I’m on call but never received any calls to come in?!? Clearly you’re NOT the only person in the entire IT Department . . . 🤦♂️ Regardless, later I hear (I have a fibre problem but can’t or won’t fix it until I / we figure out where??? 🙄 It’s clear to me this so called hospital doesn’t follow, conform, or must meet any UL / ISO standards! 🤢 All the logs especially the SysLog server should be able to provide the time, interval, and frequency of the problem. Later on all I hear is some grasping at straws about a loop?!? WTF??? Are you saying the entire IT Department just wild Willy have no clue how to setup a switch?!? People are just randomly making changes to a switch??? 😑 Come on . . . 👎 Than later I hear you say something about Microsoft?!? What the hell does that have to do with any of the switches and SNMPv3??? 🤦♂️ I could literally walk into that place and I’m sure it would be a complete cluster and mess. After I ignored all that crap I would be able to identify the problem in 20 minutes as to what it isn’t vs is! The lack of the most basic trouble shooting and grasp to use any tools is truly shocking and disgusting! Enough of this loop crap and focus on the infrastructure! ☝️
Interesting to see that after 10+ years Extreme is still having the same annoying bugs with stacks, snmp, loops which made us remove all E switches out of the network, we never looked back. But it in the end you have to love your job and use the equipment you have to deal with, thanks for the nice vids again! Take care!
Thankful for your channel. I can relate with many of the things you discuss. I’ll be praying that you will be able to quickly isolate the loop issue.
I'm thankful that people find this interesting. God bless!
God bless you! Thanks for you videos! I am up north of you in Portland OR.
You are so welcome! My two favorite towns in Oregon are Cottage Grove and Bend! God bless!
In your setup, I would presume you are running MSTP, BPDU Guard, and similar features should help prevent loops. If a loop does occur, these settings typically trigger protections like blocking ports or disabling connections. To gather more details, you can run the -show spanning-tree summary- command.. at least on a Cisco switch. This will give you a snapshot of the spanning tree status and help identify any potential loop-related problems.
I'm good at finding STP issues on the edge but not the core. Researching that. God bless!
I failed my Network+ exam last week. I'm going to try again. Thanks for your videos.
there is no try, there is only do or not do.
You can do it! God bless!
Nice discussion - thanks 🙂 God Bless.
Thanks for watching! God bless!
@@NetworkAdminLife You are quite welcome. And God Bless you.
Yep, it was foggy in Modesto, CA area yesterday morning as well. I too hate upgrading firmware on production network switches but these days given the security vulnerabilities and bug issues we have to do them whenever possible. My biggest fear is total meltdown on failed firmware upgrades so I try to do them off hours and weekends. In your case you don't have that luxury for 24/7 hour operation.
Also, the vendor should have still have helped you unless the firmware is several versions old which sounds like in this case.
Hope you feel better soon!
We'll get it done eventually. God bless!
Good luck finding the issue!!
Thank you! God bless!
Spanning tree should take care of your loop if it's there shouldn't it?
It should, but I just have trust issues when it comes to this stuff. God bless!
Before I started watching your videos, I didn’t realize there were still any saints left in CA. 😂 Regarding upgrading-is there just one core switch or is it actually a redundant pair? If a pair, seems like you could take one offline to upgrade, then when it comes back up, do the other one and voila! But I know in real life it isn’t always that simple haha. 😅 As far as network loops, usually your spanning tree statistics will give you a good indication of if/where.
There are many, many Christians in California. I believe we are actually the majority here. God bless!
If you have Extreme's ELRP set up, you can find the loops by looking at the ports it is disabling.
We do not have that set up yet. We should. God bless!
What antenna is that you have hanging from your garage door track? VHF slimjim?
Flexible J-pole. 73!
Doesn't Spanning Tree Protocol address broadcast loops?
Spanning Tree addresses physical loops too. God bless!
Are those the fibre pairs where you use SM SFP adapters with MM cables? You should check if cables and sfp adapters match up. No hate, just an advice.
Regarding loop. Dont you use STP?
Also what you can test.
1st of all allow SNMP gets from a different host. And when the switch goes red in your management software, do a snmpwalk to the switch from an other host (that was whitelisted before). If you get the data out without timeout, it is your management software or the windows base. If not, it may be a loop or something else.
I suggest using not windows for the snmpwalk, because maybe MS broke something too.
Advice is always welcome! We do use STP but I have had issues with it in the past. God bless!
May be it rely to a network congestion. I have seen something similar because of congestion on wan routers.
The uplinks have very low utilization. Some of the affected switches have 40GB uplinks. I'm onto something, more in an upcoming video. God bless!
Honestly I don't blame people wanting to run the latest version of the software. It eliminates so many potential causes (not to mention high uptime) and gives them more ammunition with development if it is in fact a bug. After all, I have no issues having end users reboot machines, check for updates, etc etc before spending a whole lot of brain energy on a weird problem. It has saved me tons of time before and will save me tons of time again.
Fair point. God bless!
You need to look at your spanning-tree instance(s). Network loops are impossible with properly implemented spanning-tree. You may be dealing with a spanning-tree instability. What variant of spanning-tree does your network use? Which switch is the spanning-tree root bridge? Which one is the backup for that? If you don't know the answers to these questions then you have been ignoring spanning-tree at your own peril. Hope you are feeling better soon.
And my peril has arrived. God bless!
Re: 'are you running the latest version'.
I know this is frustrating for the customer, and I know how it can cause alot of pain. But the support people aren't wrong. I work in a similar field, and the number of fixes that go into a new version is HUGE. There's no way for a single support person to be able to catch each and every fix and fully understand the implications. So ya, VERY often, not just 50/50, but more like 90% of the time upgrading the firmware/software will fix the issue.
And consider the other side: if updating fixes the issue, every single customer you try to help with that issue is basically wasted time for the support agent. Nothing more frustrating that discovering the problem you've been working on with a customer for a week is something already fixed.
Cool to hear it from that perspective. God bless!
Firmware upgrades are always troublesom.
I don't worry about it on my edge switches because I know that always goes smoothly. I have no reason to think that the cores won't but... God bless!
If the STP protocols are doing their job correctly you really shouldn't have a loop and any switch in an enterprise like yours almost certainly has it enabled by default.
We do have it enabled but things can get misconfigured at times. God bless!
I was thinking spanning tree
It's behaving today so not sure what's going on. God bless!
I have no clue why this video was recommended to me. But can say the entire 15:15 clearly showed absolutely no basic OSI trouble shooting?!? 🤦♂️
All I kept hearing is firmware this, SNMPv3 down! 🤢
What does the verbose logs show on the core (problem) switch??? Later I hear I’m only one person? Than I’m on call but never received any calls to come in?!?
Clearly you’re NOT the only person in the entire IT Department . . . 🤦♂️
Regardless, later I hear (I have a fibre problem but can’t or won’t fix it until I / we figure out where??? 🙄
It’s clear to me this so called hospital doesn’t follow, conform, or must meet any UL / ISO standards! 🤢
All the logs especially the SysLog server should be able to provide the time, interval, and frequency of the problem.
Later on all I hear is some grasping at straws about a loop?!?
WTF???
Are you saying the entire IT Department just wild Willy have no clue how to setup a switch?!? People are just randomly making changes to a switch??? 😑
Come on . . . 👎
Than later I hear you say something about Microsoft?!?
What the hell does that have to do with any of the switches and SNMPv3??? 🤦♂️
I could literally walk into that place and I’m sure it would be a complete cluster and mess. After I ignored all that crap I would be able to identify the problem in 20 minutes as to what it isn’t vs is!
The lack of the most basic trouble shooting and grasp to use any tools is truly shocking and disgusting!
Enough of this loop crap and focus on the infrastructure! ☝️
You should subscribe. I'm going to retire someday and if you are as good as you think you are then you would be perfect for the job. God bless!