Thanks Sarah for showing a real time trouble shooting sequence really brought back gobs of old memories from the 1970's. I can't count the number of faults exactly like that, shorted and or bent terminals, or a solder ball sitting on the banjo that I have cleared in my 31 years with just a quick visual inspection. Many a time I came in at 4pm and was handed a trouble that they had been working on the day shift with no resolution and I cleared it in a few minutes as I always did that visual inspection prior to getting out the books and tools. Bent contacts in connectors was another quick fix that everyone had overlooked. On trunk troubles I would sort them out as to trouble types and work on only one type at a time and there were only a few causes to that type and I would clear bunches at a time because I was lazy and made it easy on myself. I was actually mediocre at best but got a good reputation by being sneaky and fixing the easy stuff first then had plenty of time to spend on the head scratchers. My eyesight was always lousy so I was crap on the backplane, especially the newer wire-wrapped backplanes. Looking at all those close wire-wrapped terminals gave me virago. We had guns that tightened up and clipped off the cable ties at a specific pressure so as to not damage the wires. I still use one they are not that expensive.
A thermal camera might have helped locate the fault, if the short was causing the wiring and components to heat up -- but you'd probably consider that to be cheating!
I manage a team of maintenance technicians in manufacturing. You would be amazed at how frequently we run into similar situations with lack of documentation and needing to troubleshoot hardware the way you had to in the video. Good job!
You know. In a hundred years time, in the year of 2123, there will be more working 200 year old windup grammophones around than working CRT-televisions, AM/FM radios, videorecorders and analog telephones from the second half of the 1900's. Thinking of you guys keeping generations of telephone centrals alive and working is therefore of utmost importance for our common technological history due to their incredible rarity.
I have scars from reaching into server racks with zip-ties... One cut almost needed stitches. I like my Velcro straps, but eventually I would love to learn wax lacing. Great job as usual! It makes me happy seeing people take pride in a hobby, especially the electro-mechanical types of hobbies.
I learned how to lace in 1998 when I was installing muxes... and then when I started to work in resi POTS a few years later I completely forgot how to do it!
I've not done lacing, but one thing that I think is important is the wire ties or lacing can put too much pressure on a bundle and cause the insulation on the wires to deform. It's not a problem with these wires because they're cotton covered. But the plain pvc insulation could be deformed. This isn't a problem with velcro strips plus it's really great to be able to remove a strip and find the wire and then put the same strip back on without having to replace it with a new one.
I've learned wax lacing in the 90s. Never used it in ma professional career. And I'm glad about that. I don't really need the blisters on my fingers again.
Hey Sarah, thanks for that! I enjoyed every minute, the complexity of central offices from back then never ceases to amaze me. Sheer engineering marvels and I wish I'd learned about this stuff 40 years ago. Cheers from the UK.
Thanks Sarah. What a great video. You really laid out how to think and work like a veteran trouble shooter. I got a call one day from a guy who had been using an IBM PC XT for close to 20 years (yeah, I know). The PC had recently started acting like someone had tuned the big red power switch (remember those on IBM PCs?) off and quickly back on. The power supply appeared to have been undisturbed and the cover was installed with security Torx screws. With a little gentle rocking I heard something. I got my security Torx driver, took the cover off and found an extra cover screw lying between a heat sink and an inductor lead which had just a little solder tinning exposed above the circuit board. That screw had apparently been there since the power supply was manufactured and only recently made firm enough contact against the inductor lead to cause trouble. It is interesting how a latent short can sit around for years, in a panel frame, or PC power supply, and just the right temperature shift, vibration, or whatever, will finally push the short together… ensuring job security for a repair person.
Hey, in my most recent work as a radio Chief Engineer before fully retiring, we had a transmitter with redundant control power supplies. Dead, dead, dead. Both Polyfuses (one immediately physically above the other) burnt up. Replaced them and one of the two redundant supplies worked. Troubleshot the other and eventually found a factory wiring error, two wires swapped in a connector, shorting the supply to ground. How it got through factory final inspection and out the door like that rather a mystery. But after 20 years of opening to protect against the permanent short, the poor polyfuse burnt up and took out the one above it. Not the first time I found things like that over the years!
@@jumbie6 This one isn’t a short circuit, but a strange repair request. I got a call from a neuroscience lab where a hydraulic micro manipulator was being used to position electrodes. The control box had suddenly quit responding. I opened it up and found a green Radio Shack alligator clip had fallen off of two pieces of wire that it had been bridging. It was apparently a low volume piece of equipment and it had been sent from the factory without remembering to make a temporary engineering change permanent.
If you're worried about blowing a fuse, grab one of the 24v halogen clippin lights and use it as a fuse. Instead of consuming fuses you'll just turn the light on 😊
@@steveschulte8696 That's what the headset is for. It has a resistor built in so that when I touch things with the probe, it will click in my ear, but not blow the fuse. But if the probe touches ground *and* the thing I'm testing at the same time (because I was careless with my probe) then the resistor won't be in the circuit, and I will short to ground through the metal part of the probe. I could have put a resistance lamp in place of the fuse, but I use this probe so much, and I can't be bothered to do that every single time I use it.
Great job of troubleshooting! @ 31:36 just before you said "send it around the back" , you needed to take that short tag end and tie a half hitch with the long lacing end to lock it in. I've done a good amount of lacing as an AT&T tech, and that half hitch will lock that loop in very well.
That was an AWESOME troubleshooting and fixing video ! This is not only very interesting to watch but can help others in the future as a troubleshooting aid for the museum and for troubleshooting in general. The museum is SO much better with you there, Sarah ! Thank you !
Panel machines amaze me. I can still remember the very first time I walked into the PArkway central office in Seattle for a T-Carrier class and was met by the odd smell of hot grease and burnt cork. Smelled nothing like the Step-by-Step machine I worked. 😉Spent some time just watching the the machine work. Fascinating.
Thanks, Sarah, for a very interesting video. You brought up an excellent point that troubleshooting becomes a lot more intense when customers are out of service. I enjoyed the wax string lacing demonstration. I have always been impressed how great a job the Western Electric installers did when lacing up the back plane wiring no matter the type of switch.
Wow. That system is quite a maze to work your way through. I think you’re one of the few people left in the world that understands it so well. I’m always impressed by how you figure things out. A great analyst!
Awesome troubleshooting Sarah, and it your lacing was extremely impressive. I had learned how to lace back in the stone age, but have forgotten how to do it.
Wow, brings back loads of memories. The sound of the panel office…I’ve heard that spinning selector sound whenever something major happened in NYC (where I worked) An other sense that brings it all back is the smell….any time I’m near a box of crayons it smells like panel, because the wires and numerous other items were coated in paraffin. Thanks for the memories. 😊
Great troubleshooting video! It also illustrates the relationship between schematics and wiring diagrams, and the immense amount of labor required to construct these electromechanical switches.
It's amazing how similar the debugging process is to what I do today in web development. Tracing through what works, what doesn't, how far the signal gets, where it doesn't, or where it gets corrupted, until you find a single parenthesis out of place. xD I love watching you at work. Great fun!
Love the analysis. There are so many tips in your videos. I’m building a step by step switch and this video just taught me how to lace properly. Thank you .. J
I just wanted to let you know that I've been trying to find the time to offer my self to volunteer but due to mt own obligations, job and personal issues I've not ben able to do so Sarah. But seeing you and your passion in maintaining these systems bring me great joy and awaken my own passion and I hope to see more in the future! I hope to stop by again - Ryan
I used to work A-7s, it had a lot of relays and relay cards. The TA-7C almost exclusivity had relays and cards made in Iran. You couldn't get new after the Fall of the Shah. Please note, electromechanical relays do have a usable life, and those are teaching the end of theirs. The tantalum pads erode, corrode and Arc Over fusing the relay shut, the springs and leaves fracture and break leaving the circuit open. Everything ends.... everything.
@ 24:15 That is a standard "frame iron" , its plugged into an outlet that connects to AC power on a sliding track connection along with that ladder. It's powered on with a switch at the end of the aisle, along with any other iron plugged into an outlet on that aisle. They are more commonly used to solder the connections from the Vertical / Cable pair, to the horizontal / equipment connections, and are left on all day in a working central office.
It's so cool to see a mechanical switching office still working. I worked in a DMS100 office for about 10 years but we recently got moved to our call centre office which is a standard cubicle farm so don't get to see the equipment anymore. I just monitor stuff so don't really work on it but it was fun to be in that building and see things get added, changed etc over time.
Yeah, funnily enough the same zip tie issue from the old times translates into current times. I just so happened to slice my whole hand open on goddamn cableties in a data center
I was working in the Panel office that served the downtown and outlying areas of Newark N.J. ie. It was a large office that covered 2 floors, I will never forget the deafening sound of that office on the afternoon of November 22,1963 when word got out that Kennedy was assassinated.
We heard from switchmen in Seattle that our Panel offices pretty much fell over from the load. The sender selectors were going nuts, and there was enormous dial tone delay. Do you have any other memories of that day, or working in that office?
@@ConnectionsMuseum I remember everyone looking at each other wondering what was going on, when another tech walked in from lunch in tears and told us what had happened.
I have many memories that span from 1962 ( 17 years old 2 weeks out of High School) to 1995 when I retired, I started as a Framemen, Toll Transmission tester, #1 Xbar, #1 Xbar TSP tandem office, # 5 Xbar, # 5 Xbar ETS, (Cutover 2 of them) 2 B ess, IA Ess, #5 Ess, and lastly DMS100
There's a CO three blocks from my apartment that was built in the 1920's and still operates today. I can only dream of what kind of treasures are in that building (probably more internet stuff than POTS nowadays, so the cool stuff has long been silent), especially since South Central Bell bricked up all the windows sometime before I was born. I don't really know the story of how you guys got a CO and equipment to start a museum, but I would love to do something like that way down here in Alabama. Less than an hour away from my home in Birmingham is where the first ever 911 call was placed.
If someone were to place a call when this fault was happening, what would happen to their call? Hangs up? Some sort of busy signal? Stuck? Also why not trace that abandoned wire bundle and eliminate stuff like that? Is there a concern that moving it around too much may cause other faults?
I choose whether or not to eliminate extra wires depending on what they are, and how they look. If they were original to the switch, I leave them. If they were added after, in a haphazard way, I remove them. These were original. A call during this failure would just get stuck.
It’s nice to see you guys take care of this old equipment. Here in Norway the entire landline network is retired. The last subscribers were shut down last year. Tho the cables remain to supply other forms of communication for the time being (a few years). An era is over and it’s sad to see it go.
It will be some time until the US pulls the plug on copper telecom circuits. Even systems at major airport, when our company was having to move circuits to a new equipment office, still required copper / Mux fed T1's over the new equipment path.
@@poormanselectronicsbench2021 yes, for many local applications I’m sure it will be kept up here too. But the law in Norway used to say that everyone has the right to have a phone where they lived, back when we had one government owned phone company. This made them build a lot of infrastructure that would never economically exist if it was a private company. We are only about 5 million people in a huge country with lot of mountains and fjords. It resulted in an extensive network based on copper and microwave radio links. On the whole tho, the network is being scaled/taken down now that there are optional ways of providing a reliable communication option to all of us. If this is as secure and reliable is a huge concern and question. But that’s the road we are heading.
I assume that landlines still exist, but they don't use the landline network. They'll use VOIP over the Internet now. And I hope the government is now declaring that everyone has the right to have an Internet line where they live.
@@thewhitefalcon8539 Up until mid 2021, I handled both the equipment, as well as part of the copper loops for one large Telecom for my area in the Chicago suburbs and outlying areas. They (In Norway) most likely still employ a "copper loop" for some services, but I am betting that analog "POTS" (Plain Old Telephone Service) has been changed to VOIP over a ADSL/VDSL service. Replacing all the copper with a fiber overlay would be great, but most countries still haven't gotten that far, and end users still are using old SONET circuits like T1's and have not as of yet made plans to update, so copper loops will still be around for awhile.
@@poormanselectronicsbench2021 Yeah that's pretty much what I'm accustomed to in NZ, about 10-15 years ago when a similar transition was made in cities. DSL is a fine way to reuse old lines. Not every customer needs fiber speeds yet so if it ain't broke don't fix it. I'll mention that DSL is not reliable on every loop (experienced this a lot on MY loop!) so it's also good for the telephone company to keep the option to leave the landline alone - they can plug it into a VOIP adapter in the telephone office instead of the user premises.
That false ground would have caused an issue to use a signal tracer, it might have bled through a bit, but it could have just been a time waster if tried.
Reminds me of the wall of relays at my last job.. A 1950s nurse call system in a hospital. The coils had a light wired across them so you would know which relay had power. Chasing down glitches could be a nightmare.. That looks to be orders of magnitude more complicated than the nurse call system.
Well done Sarah! As an electronics engineer, I can truly applaud your debugging. Had the short been a metallic insect, you would have clearly found and eradicated the BUG ;o)
A housekeeping hint for all the cut cabling in the back of the panel. Lace them back into their own form then you can move them as a bundle. All the wires look to be color coded, but it appears that all the wires in the "Buss" are coded the same, or nearly. That is troubleshooting heck. The wires are bundled at the old Western Electric plant to aid in placing the laced loom in the frame. There was a wire board operator who spent his entire day running wire around a bunch of nails according his instructions. Then he would lace them all up and cut the wires. That whole assembly went to the operator that soldered all the terminals, and he would check his work with a "Beeper". Labor intensive but "Made in America".
Really interesting to watch. and think about when they were used in larger settings to be a tech and get that work order. i don't think they had a few hours to solve the problem. zip ties are great but when they are cut to short you get that minefield of sharp edges. so nice to see that cable stitching being done and very important to do.
We would drive past an office like this when I was a kid. I peered in as best I could as we drove past-And I would see what looked like a warehouse with rack after rack of stuff like this. Who knew what marvels went on inside!
Total respect for your attention to nicely lacing the wiring back together at the end. I'm curious what kind of string are you using? Thank you for the great video!
Ma Bell had a lacing waxed nylon cord, it was single ply and it was used for for that purpose. It is very strong, it could not be broken with you hands.
What would you say to a video answering a fuse alarm, from a major and/or minor initiation to possibly showing the alarm bay to zeroing in by way of aisle lights and panel lights Possibly other types of alarms. This small but very effective tool would demonstrate the conviction to service from the Bell System.
Wow what an intense debugging job! Great technique to chase down the problem though. I had not realized how intense and concentrated the wiring behind the frame is, ouch!
Fascinating. The largest relay I've had trouble with was a six-pole double-throw in a National NCX-5 transceiver. The engineers really pushed way past the specs, and they were switching 280 volts DC with contacts rated at 24 VDC. To make it worse they were switching 280 volts off with the break contacts and turning ground on with the make contacts, so a few milliseconds of time they were shorting. They didn't last more than a year or two at best. Ma Bell probably didn't push things that far. I wonder if you could have used a current-clamp type of meter to find the short? Or do you only use period-correct test equipment?
17:25 feels like a far too crude way to search for connections between 48 volts and ground. Discharge machining the contacts is often not a good way to make things last. I would have a suitable resistor, a small bridge rectifier, and an LED. Would light up visibly when in circuit, but not cause needless sparks. (A suitable resistor would be around 10K ohm or so.) edit: 24:30 is something I thought were a joke. That is a ridiculously large iron for the job. Heating up surrounding wires is a wonderful way to degrade the insulation and cause future shorts as the degraded isolation fails. I understand trying to be authentic to the period and such. But when it comes to maintaining the machines, it is better to use working practices that helps preserve the machine. 26:08 yes, those are improperly installed cable ties. They should be flush cut such that they won't cut someone's arm open in the future. (I know of companies that has fired people for not doing this properly. It is a liability for the company to ship products that can inflict such preventable injuries.) 26:22 "Does anybody ever do that?" Well, I do. To the point that I fix those that aren't done properly when I see them. 28:50 is a classic. Stiff wires getting pulled about over the years usually dislodges terminal blocks and other stuff over time. Now these days the issue is often cracked solder joints leading to intermittent connections, rather than bent metal terminals shorting out. But the root cause is the same, stiff wires getting forced about through years of service and eventually pulling something along with them..
May fav part of this video was the cable lacing. Those zip ties were really bugging me. Cable lacing really is a lost art and I would love to see it used more often.
You don't need a special tool to cut zip ties properly, small cutters as you were using to remove them will work just fine. My personal pet peeve is when they cut them on an angle leaving a sharp point just waiting to rip the back of my hand open.
I wonder if a thermal camera could have been useful, following the one wire carrying current for way longer than normal? The newer higher resolution thermal imagers are getting cheaper and cheaper these days, and have enough detail to be pretty useful.
Is Sarah only trying to use techniques and materials that were available when the panel switch was built 100 years ago? No thermal cameras, zip-ties, signal tracers, Velcro, etc.
Well that was today's lesson in humility. I started as a hobbyist in electronics at age 3.5, I am now in my late 30s. I spent +13 years working as a research engineer. I had no idea a special cutter existed for cable ties until now. 1. Thank you. 2. How is this tool for such a common use so obscure? I don't even know the name of it.
I tie cables like that in the heavy machinery I service, it makes it so much nicer to work on in the future. Funny enough, I picked it up from old telco stuff.
Brilliant as ever! I have just said to my other half: In 2024 I am travelling to visit the Connections Museum. He did not say: No. Incidentally I love they way you (and others from USA) say “sodder” rather than “solder”; in England it sounds really funny and somehow a bit naughty😉. I do hope you will be there still next year!
@@lurch1539 How very interesting. It is strange that I have never heard any English person say sodder. Many of my friends are from the telephony fraternity, particularly the Strowger engineers of the “last century” 😉. I am sure that your erudite dissertation is spot on the money though. Never the less, the pronunciation still sounds amusing to me 😂.
Sarah, since you know one end of the offending wire is one side of the SG magnet, why not just pick a working one, connect one side of the SG magnet to the meter, and just scan for continuity on the SF contacts? That would give you one side and observation of the SF contacts should show you the other side.
I don't use meters that much, and I didn't visualize the problem the same way you did. Your solution would have worked, but introducing a meter into the troubleshooting would have made things much worse for me.
@@ConnectionsMuseum No criticism intended, I'm just boggled that you keep so much high complexity in your head, much less keep it straight. I can get tangled following the sequence of a step switch :). I'm glad you folks are keeping this stuff alive :)
@@ConnectionsMuseum Coming at this from another perspective, instead of wiring in a meter, would it have been possible to use a split-core current transformer to tag the grounded wire on either a working or the troubled SG relay? You'd check for blips in the transformer output (induced voltage) when the relay changed state.
It would have been good to actually busy out the E decoder connector and then show that the call sim could work normally with just one bad component busied out. A lot of work lacing wires that don't actually go anywhere just to make it look good. Kudos!
Nice to see someone who knows how to lace cable. But, I am one of the people who uses cable ties.. but I do have the proper tool and use it every day :-)
As a Telcom engineer calling e1 and b channel/sigtran signaling ancient the marvel of mechanical switches is quite awesome! Then again I also very much like the old skool pinball machines :)
I'm wondering why they didn't use a toner and tracer to find out more about the wire routing. It's a great tool for just that purpose. Also, why tie the unconnected wires back? It seems to me that it would be better to tie them to the vertical run where they are out of the way and not blocking the view of all those soldered joints. Or else cut them shorter, since they will never be used again. They emphasized looking neat, well removing unneeded wiring makes it even neater. Thanks for the informative video. 👍
@@ConnectionsMuseum "Toning" works best when one lead of the tone signal is connected to a ground, so, in reality, the ground at the other end would just kill it. 30+ years as a cable splicer, toning cable pairs everywhere, definitely taught me that issue. Grounds were always the hardest to troubleshoot, the best device to find / measure a distance to them was to use a TDR. If you don't have an old Metro Tel or WE 76C test set for your museum, you should look into one.
Thanks Sarah, for another excellent video. It was not clear why you would leave all those unused spare wires in that cable bundle. Was this wiring left from some prior configuration? I would have pruned much of that out, since the bundle was open. However, that may be easier said than done.
About 80% of the wiring in that frame is unused, and if i took it all out, the frame would look naked on that side. Our museum configuration is so tiny, and that frame is meant to serve a much larger number of senders. I left it in for aesthetic reasons, so visitors could see the original wire forms.
Great vid for FIRST TIME viewer! Now I gotta learn MORE antique electronics, thanks MA BELL! Just learned how to brew a tetrode from scratch and how to make ic's from sand... Okay, last was not real. Just saying...
I used to be able to make that happen at my local CO back in the days of switched systems! I would take the CO down for a day or two before they could fix the problem.
Zip-ties are the devil, but when I have to use them, i've found that toenail clippers are a good inexpensive substitute for the special zip tie flush cutter!
The switch sounded like it was full of angry bees at the beginning when it couldn't find a sender. Nice video! (I am an old Nortel guy who cut my teeth on a SP-1)
I have a wonder for you @Connections Museum. When i search for foults in our relayboxes for railway, we have two ways to do. With a volt meter och a lamp with a magnet for the ground and the test lead. Is it not possible to do someting sumular in your applications?
BTW, we would never put all that back together till we verified we had completed the repair by testing. Yes, I know you are confident you found a problem, still there could, for example, be more to the problem.
I missed the step where you prove that the system no longer runs out of resources when you run the call generator. (In the spirit of bug tracking & fixing as described in "The pragmatic programmer".)
Hey so I'm have not watched all ur videos yet but I watched enough to understand that these are real calls. I would love to see a specific video on where are these phone calls coming from and how they work with modern centers as a assume they are connecting digitally but I don't know for sure
So you answered a bit, they are not real calls I guess. It would be cool if u some how connected it to real calls with a drop out to a modern center if any failure
I always am very pleased to see younger folks that both know HOW older technologies are supised to work, and care enough to do things correctly.
Thanks Sarah for showing a real time trouble shooting sequence really brought back gobs of old memories from the 1970's. I can't count the number of faults exactly like that, shorted and or bent terminals, or a solder ball sitting on the banjo that I have cleared in my 31 years with just a quick visual inspection. Many a time I came in at 4pm and was handed a trouble that they had been working on the day shift with no resolution and I cleared it in a few minutes as I always did that visual inspection prior to getting out the books and tools. Bent contacts in connectors was another quick fix that everyone had overlooked. On trunk troubles I would sort them out as to trouble types and work on only one type at a time and there were only a few causes to that type and I would clear bunches at a time because I was lazy and made it easy on myself. I was actually mediocre at best but got a good reputation by being sneaky and fixing the easy stuff first then had plenty of time to spend on the head scratchers. My eyesight was always lousy so I was crap on the backplane, especially the newer wire-wrapped backplanes. Looking at all those close wire-wrapped terminals gave me virago.
We had guns that tightened up and clipped off the cable ties at a specific pressure so as to not damage the wires. I still use one they are not that expensive.
A thermal camera might have helped locate the fault, if the short was causing the wiring and components to heat up -- but you'd probably consider that to be cheating!
I manage a team of maintenance technicians in manufacturing. You would be amazed at how frequently we run into similar situations with lack of documentation and needing to troubleshoot hardware the way you had to in the video. Good job!
You know. In a hundred years time, in the year of 2123, there will be more working 200 year old windup grammophones around than working CRT-televisions, AM/FM radios, videorecorders and analog telephones from the second half of the 1900's.
Thinking of you guys keeping generations of telephone centrals alive and working is therefore of utmost importance for our common technological history due to their incredible rarity.
I have scars from reaching into server racks with zip-ties... One cut almost needed stitches. I like my Velcro straps, but eventually I would love to learn wax lacing.
Great job as usual! It makes me happy seeing people take pride in a hobby, especially the electro-mechanical types of hobbies.
I learned how to lace in 1998 when I was installing muxes... and then when I started to work in resi POTS a few years later I completely forgot how to do it!
I've not done lacing, but one thing that I think is important is the wire ties or lacing can put too much pressure on a bundle and cause the insulation on the wires to deform. It's not a problem with these wires because they're cotton covered. But the plain pvc insulation could be deformed. This isn't a problem with velcro strips plus it's really great to be able to remove a strip and find the wire and then put the same strip back on without having to replace it with a new one.
I've learned wax lacing in the 90s. Never used it in ma professional career. And I'm glad about that. I don't really need the blisters on my fingers again.
@@MarcoTedaldi Amen to that
That's why you use flush cut snips. I've had cuts as well wiring in locomotive cabinets. I've only used Velcro for important Radio antenna cabling..
Hey Sarah, thanks for that! I enjoyed every minute, the complexity of central offices from back then never ceases to amaze me. Sheer engineering marvels and I wish I'd learned about this stuff 40 years ago. Cheers from the UK.
Thanks Sarah. What a great video. You really laid out how to think and work like a veteran trouble shooter. I got a call one day from a guy who had been using an IBM PC XT for close to 20 years (yeah, I know). The PC had recently started acting like someone had tuned the big red power switch (remember those on IBM PCs?) off and quickly back on. The power supply appeared to have been undisturbed and the cover was installed with security Torx screws. With a little gentle rocking I heard something. I got my security Torx driver, took the cover off and found an extra cover screw lying between a heat sink and an inductor lead which had just a little solder tinning exposed above the circuit board. That screw had apparently been there since the power supply was manufactured and only recently made firm enough contact against the inductor lead to cause trouble.
It is interesting how a latent short can sit around for years, in a panel frame, or PC power supply, and just the right temperature shift, vibration, or whatever, will finally push the short together… ensuring job security for a repair person.
Hey, in my most recent work as a radio Chief Engineer before fully retiring, we had a transmitter with redundant control power supplies. Dead, dead, dead. Both Polyfuses (one immediately physically above the other) burnt up. Replaced them and one of the two redundant supplies worked. Troubleshot the other and eventually found a factory wiring error, two wires swapped in a connector, shorting the supply to ground. How it got through factory final inspection and out the door like that rather a mystery. But after 20 years of opening to protect against the permanent short, the poor polyfuse burnt up and took out the one above it. Not the first time I found things like that over the years!
@@jumbie6 This one isn’t a short circuit, but a strange repair request. I got a call from a neuroscience lab where a hydraulic micro manipulator was being used to position electrodes. The control box had suddenly quit responding. I opened it up and found a green Radio Shack alligator clip had fallen off of two pieces of wire that it had been bridging. It was apparently a low volume piece of equipment and it had been sent from the factory without remembering to make a temporary engineering change permanent.
If you're worried about blowing a fuse, grab one of the 24v halogen clippin lights and use it as a fuse. Instead of consuming fuses you'll just turn the light on 😊
Yeah I know that trick. I also have a fuse with a lamp wired in place of the fuse element. I should have grabbed that, instead of being lazy.
Another is to get a automotive trouble shooting probe light, and put a -48 volt in it.
@@steveschulte8696 That's what the headset is for. It has a resistor built in so that when I touch things with the probe, it will click in my ear, but not blow the fuse. But if the probe touches ground *and* the thing I'm testing at the same time (because I was careless with my probe) then the resistor won't be in the circuit, and I will short to ground through the metal part of the probe.
I could have put a resistance lamp in place of the fuse, but I use this probe so much, and I can't be bothered to do that every single time I use it.
lamp turn on :)
Great job of troubleshooting! @ 31:36 just before you said "send it around the back" , you needed to take that short tag end and tie a half hitch with the long lacing end to lock it in. I've done a good amount of lacing as an AT&T tech, and that half hitch will lock that loop in very well.
Thanks for the tip!
That was an AWESOME troubleshooting and fixing video ! This is not only very interesting to watch but can help others in the future as a troubleshooting aid for the museum and for troubleshooting in general. The museum is SO much better with you there, Sarah ! Thank you !
Panel machines amaze me. I can still remember the very first time I walked into the PArkway central office in Seattle for a T-Carrier class and was met by the odd smell of hot grease and burnt cork. Smelled nothing like the Step-by-Step machine I worked. 😉Spent some time just watching the the machine work. Fascinating.
Thanks, Sarah, for a very interesting video. You brought up an excellent point that troubleshooting becomes a lot more intense when customers are out of service. I enjoyed the wax string lacing demonstration. I have always been impressed how great a job the Western Electric installers did when lacing up the back plane wiring no matter the type of switch.
I remember the old timers they took good care of those switches and you didn't dare touch THEIR switches !
Wow. That system is quite a maze to work your way through. I think you’re one of the few people left in the world that understands it so well. I’m always impressed by how you figure things out. A great analyst!
Awesome troubleshooting Sarah, and it your lacing was extremely impressive. I had learned how to lace back in the stone age, but have forgotten how to do it.
Wow, brings back loads of memories. The sound of the panel office…I’ve heard that spinning selector sound whenever something major happened in NYC (where I worked) An other sense that brings it all back is the smell….any time I’m near a box of crayons it smells like panel, because the wires and numerous other items were coated in paraffin. Thanks for the memories. 😊
Great troubleshooting video! It also illustrates the relationship between schematics and wiring diagrams, and the immense amount of labor required to construct these electromechanical switches.
It's amazing how similar the debugging process is to what I do today in web development. Tracing through what works, what doesn't, how far the signal gets, where it doesn't, or where it gets corrupted, until you find a single parenthesis out of place. xD
I love watching you at work. Great fun!
Love the analysis. There are so many tips in your videos. I’m building a step by step switch and this video just taught me how to lace properly. Thank you .. J
I just wanted to let you know that I've been trying to find the time to offer my self to volunteer but due to mt own obligations, job and personal issues I've not ben able to do so Sarah. But seeing you and your passion in maintaining these systems bring me great joy and awaken my own passion and I hope to see more in the future! I hope to stop by again - Ryan
I used to work A-7s, it had a lot of relays and relay cards. The TA-7C almost exclusivity had relays and cards made in Iran. You couldn't get new after the Fall of the Shah. Please note, electromechanical relays do have a usable life, and those are teaching the end of theirs. The tantalum pads erode, corrode and Arc Over fusing the relay shut, the springs and leaves fracture and break leaving the circuit open. Everything ends.... everything.
Very interesting Trouble and great troubleshooting, Sarah!
I love the in-depth videos like this, really shows a lot about how things were done and troubleshooted when these switches were still in use.
All this fun stuff is making me want to buy another electro-mech pinball machine to restore. Another awesome video!
You couldn't find a bigger soldering iron?! 🤣 Thankyou Sarah for another amazing video ♥
One of those new TS80S USB-C irons would probably be awesome in this setup
You should check out the one they have for doing soldering work on telephone lines. But yes, she could've found a bigger one x)
@ 24:15 That is a standard "frame iron" , its plugged into an outlet that connects to AC power on a sliding track connection along with that ladder. It's powered on with a switch at the end of the aisle, along with any other iron plugged into an outlet on that aisle. They are more commonly used to solder the connections from the Vertical / Cable pair, to the horizontal / equipment connections, and are left on all day in a working central office.
It's so cool to see a mechanical switching office still working. I worked in a DMS100 office for about 10 years but we recently got moved to our call centre office which is a standard cubicle farm so don't get to see the equipment anymore. I just monitor stuff so don't really work on it but it was fun to be in that building and see things get added, changed etc over time.
Yeah, funnily enough the same zip tie issue from the old times translates into current times. I just so happened to slice my whole hand open on goddamn cableties in a data center
I was working in the Panel office that served the downtown and outlying areas of Newark N.J. ie. It was a large office that covered 2 floors, I will never forget the deafening sound of that office on the afternoon of November 22,1963 when word got out that Kennedy was assassinated.
We heard from switchmen in Seattle that our Panel offices pretty much fell over from the load. The sender selectors were going nuts, and there was enormous dial tone delay.
Do you have any other memories of that day, or working in that office?
@@ConnectionsMuseum I remember everyone looking at each other wondering what was going on, when another tech walked in from lunch in tears and told us what had happened.
I have many memories that span from 1962 ( 17 years old 2 weeks out of High School) to 1995 when I retired, I started as a Framemen, Toll Transmission tester, #1 Xbar, #1 Xbar TSP tandem office, # 5 Xbar, # 5 Xbar ETS, (Cutover 2 of them) 2 B ess, IA Ess, #5 Ess, and lastly DMS100
If you'd like to share some of them, we'd love to listen. Drop us a line at info@connectionsmuseum.org
MEGA huge soldering iron, I love it
There's a CO three blocks from my apartment that was built in the 1920's and still operates today. I can only dream of what kind of treasures are in that building (probably more internet stuff than POTS nowadays, so the cool stuff has long been silent), especially since South Central Bell bricked up all the windows sometime before I was born. I don't really know the story of how you guys got a CO and equipment to start a museum, but I would love to do something like that way down here in Alabama. Less than an hour away from my home in Birmingham is where the first ever 911 call was placed.
They got the equipment because the museum is a CO that ran it until the 70s and upgraded to a single floor ESS
If someone were to place a call when this fault was happening, what would happen to their call? Hangs up? Some sort of busy signal? Stuck?
Also why not trace that abandoned wire bundle and eliminate stuff like that? Is there a concern that moving it around too much may cause other faults?
I choose whether or not to eliminate extra wires depending on what they are, and how they look. If they were original to the switch, I leave them. If they were added after, in a haphazard way, I remove them. These were original.
A call during this failure would just get stuck.
It’s nice to see you guys take care of this old equipment. Here in Norway the entire landline network is retired. The last subscribers were shut down last year. Tho the cables remain to supply other forms of communication for the time being (a few years). An era is over and it’s sad to see it go.
It will be some time until the US pulls the plug on copper telecom circuits. Even systems at major airport, when our company was having to move circuits to a new equipment office, still required copper / Mux fed T1's over the new equipment path.
@@poormanselectronicsbench2021 yes, for many local applications I’m sure it will be kept up here too. But the law in Norway used to say that everyone has the right to have a phone where they lived, back when we had one government owned phone company. This made them build a lot of infrastructure that would never economically exist if it was a private company. We are only about 5 million people in a huge country with lot of mountains and fjords. It resulted in an extensive network based on copper and microwave radio links. On the whole tho, the network is being scaled/taken down now that there are optional ways of providing a reliable communication option to all of us. If this is as secure and reliable is a huge concern and question. But that’s the road we are heading.
I assume that landlines still exist, but they don't use the landline network. They'll use VOIP over the Internet now. And I hope the government is now declaring that everyone has the right to have an Internet line where they live.
@@thewhitefalcon8539 Up until mid 2021, I handled both the equipment, as well as part of the copper loops for one large Telecom for my area in the Chicago suburbs and outlying areas. They (In Norway) most likely still employ a "copper loop" for some services, but I am betting that analog "POTS" (Plain Old Telephone Service) has been changed to VOIP over a ADSL/VDSL service. Replacing all the copper with a fiber overlay would be great, but most countries still haven't gotten that far, and end users still are using old SONET circuits like T1's and have not as of yet made plans to update, so copper loops will still be around for awhile.
@@poormanselectronicsbench2021 Yeah that's pretty much what I'm accustomed to in NZ, about 10-15 years ago when a similar transition was made in cities. DSL is a fine way to reuse old lines. Not every customer needs fiber speeds yet so if it ain't broke don't fix it. I'll mention that DSL is not reliable on every loop (experienced this a lot on MY loop!) so it's also good for the telephone company to keep the option to leave the landline alone - they can plug it into a VOIP adapter in the telephone office instead of the user premises.
always a pleasure watching! congrats!
woah, i LOVE to see proper troubleshooting like this ... Sarah you are amazing
🇷🇺
Great video! Do you never use a tone sender/tracer when trying to find a wire run? Or was that not possible because of the wire connected to ground?
That false ground would have caused an issue to use a signal tracer, it might have bled through a bit, but it could have just been a time waster if tried.
Great demonstration of lacing. Thanks for showing that. We sold spools of the waxed lacing tape at Vetco but I never got to see how it was done.
Reminds me of the wall of relays at my last job.. A 1950s nurse call system in a hospital. The coils had a light wired across them so you would know which relay had power.
Chasing down glitches could be a nightmare..
That looks to be orders of magnitude more complicated than the nurse call system.
Well done Sarah! As an electronics engineer, I can truly applaud your debugging. Had the short been a metallic insect, you would have clearly found and eradicated the BUG ;o)
its neat how you can see problems happening in real time on these old mechanical switches. it almost throws a tantrum haha.
Lovely work lacing up the looms (probably not how you say it in US) :)
Rigidity and neatness, nobody wants to work with a so-called Hay Baler when it comes to wire racks.
Great stuff. I remember looking at those banks of cables as a trainee thinking wtf. And the noise!
Nice troubleshooting, and good eye. (Also nice to see someone who still knows how to lace properly!)
A housekeeping hint for all the cut cabling in the back of the panel. Lace them back into their own form then you can move them as a bundle.
All the wires look to be color coded, but it appears that all the wires in the "Buss" are coded the same, or nearly. That is troubleshooting heck. The wires are bundled at the old Western Electric plant to aid in placing the laced loom in the frame. There was a wire board operator who spent his entire day running wire around a bunch of nails according his instructions. Then he would lace them all up and cut the wires. That whole assembly went to the operator that soldered all the terminals, and he would check his work with a "Beeper". Labor intensive but "Made in America".
Really interesting to watch.
and think about when they were used in larger settings to be a tech and get that work order.
i don't think they had a few hours to solve the problem.
zip ties are great but when they are cut to short you get that minefield of sharp edges.
so nice to see that cable stitching being done and very important to do.
We would drive past an office like this when I was a kid. I peered in as best I could as we drove past-And I would see what looked like a warehouse with rack after rack of stuff like this. Who knew what marvels went on inside!
Total respect for your attention to nicely lacing the wiring back together at the end. I'm curious what kind of string are you using? Thank you for the great video!
As a WECO installer, we used rolls of "12 cord" which was 12 ply waxed cotton cord. Nowadays, they use 8 & 9 ply waxed cord instead.
Ma Bell had a lacing waxed nylon cord, it was single ply and it was used for for that purpose. It is very strong, it could not be broken with you hands.
That is why it is necessary to wrap the place of soldering in an insulating tube so that it does not close with adjacent contacts.
What would you say to a video answering a fuse alarm, from a major and/or minor initiation to possibly showing the alarm bay to zeroing in by way of aisle lights and panel lights Possibly other types of alarms. This small but very effective tool would demonstrate the conviction to service from the Bell System.
Wow what an intense debugging job! Great technique to chase down the problem though. I had not realized how intense and concentrated the wiring behind the frame is, ouch!
Fascinating. The largest relay I've had trouble with was a six-pole double-throw in a National NCX-5 transceiver. The engineers really pushed way past the specs, and they were switching 280 volts DC with contacts rated at 24 VDC. To make it worse they were switching 280 volts off with the break contacts and turning ground on with the make contacts, so a few milliseconds of time they were shorting. They didn't last more than a year or two at best. Ma Bell probably didn't push things that far.
I wonder if you could have used a current-clamp type of meter to find the short? Or do you only use period-correct test equipment?
17:25 feels like a far too crude way to search for connections between 48 volts and ground.
Discharge machining the contacts is often not a good way to make things last.
I would have a suitable resistor, a small bridge rectifier, and an LED. Would light up visibly when in circuit, but not cause needless sparks.
(A suitable resistor would be around 10K ohm or so.)
edit:
24:30 is something I thought were a joke. That is a ridiculously large iron for the job. Heating up surrounding wires is a wonderful way to degrade the insulation and cause future shorts as the degraded isolation fails.
I understand trying to be authentic to the period and such. But when it comes to maintaining the machines, it is better to use working practices that helps preserve the machine.
26:08 yes, those are improperly installed cable ties. They should be flush cut such that they won't cut someone's arm open in the future. (I know of companies that has fired people for not doing this properly. It is a liability for the company to ship products that can inflict such preventable injuries.)
26:22 "Does anybody ever do that?" Well, I do. To the point that I fix those that aren't done properly when I see them.
28:50 is a classic. Stiff wires getting pulled about over the years usually dislodges terminal blocks and other stuff over time. Now these days the issue is often cracked solder joints leading to intermittent connections, rather than bent metal terminals shorting out. But the root cause is the same, stiff wires getting forced about through years of service and eventually pulling something along with them..
I can _smell_ that blueprint.
A strange mix of ammonia and ozone for the "diazo" process, lol
May fav part of this video was the cable lacing. Those zip ties were really bugging me. Cable lacing really is a lost art and I would love to see it used more often.
Look up "Chicago stitch lacing" and "Kansas City Stitch lacing" on YT, there's a few videos if you want to see more
You don't need a special tool to cut zip ties properly, small cutters as you were using to remove them will work just fine. My personal pet peeve is when they cut them on an angle leaving a sharp point just waiting to rip the back of my hand open.
I wonder if a thermal camera could have been useful, following the one wire carrying current for way longer than normal? The newer higher resolution thermal imagers are getting cheaper and cheaper these days, and have enough detail to be pretty useful.
Is Sarah only trying to use techniques and materials that were available when the panel switch was built 100 years ago? No thermal cameras, zip-ties, signal tracers, Velcro, etc.
@@GusFernCa I'm assuming part of the fun is trying to do it as accurately (for the time period) as possible.
Well that was today's lesson in humility. I started as a hobbyist in electronics at age 3.5, I am now in my late 30s. I spent +13 years working as a research engineer. I had no idea a special cutter existed for cable ties until now.
1. Thank you.
2. How is this tool for such a common use so obscure? I don't even know the name of it.
Awesome you answered some of my questions.
I tie cables like that in the heavy machinery I service, it makes it so much nicer to work on in the future. Funny enough, I picked it up from old telco stuff.
12:50 Wow, a schematic printed in cyanotype? A literal blueprint for the wiring
There are two main faults in electrical engineering: either there is a contact where it should not be, or there is no contact where it should be)
Now that you have discovered the terminals that provide that ground, do you go back and note that on the schematic?
Ah, thanks for the reminder!
Brilliant as ever! I have just said to my other half: In 2024 I am travelling to visit the Connections Museum. He did not say: No.
Incidentally I love they way you (and others from USA) say “sodder” rather than “solder”; in England it sounds really funny and somehow a bit naughty😉.
I do hope you will be there still next year!
@@lurch1539 How very interesting. It is strange that I have never heard any English person say sodder. Many of my friends are from the telephony fraternity, particularly the Strowger engineers of the “last century” 😉. I am sure that your erudite dissertation is spot on the money though. Never the less, the pronunciation still sounds amusing to me 😂.
Good to see you again Sarah
I do wonder, why keep the extra wires? Do they serve a use just being there? Purely for the look?
Sarah, since you know one end of the offending wire is one side of the SG magnet, why not just pick a working one, connect one side of the SG magnet to the meter, and just scan for continuity on the SF contacts? That would give you one side and observation of the SF contacts should show you the other side.
I don't use meters that much, and I didn't visualize the problem the same way you did. Your solution would have worked, but introducing a meter into the troubleshooting would have made things much worse for me.
@@ConnectionsMuseum No criticism intended, I'm just boggled that you keep so much high complexity in your head, much less keep it straight. I can get tangled following the sequence of a step switch :). I'm glad you folks are keeping this stuff alive :)
@@ConnectionsMuseum Coming at this from another perspective, instead of wiring in a meter, would it have been possible to use a split-core current transformer to tag the grounded wire on either a working or the troubled SG relay? You'd check for blips in the transformer output (induced voltage) when the relay changed state.
It would have been good to actually busy out the E decoder connector and then show that the call sim could work normally with just one bad component busied out.
A lot of work lacing wires that don't actually go anywhere just to make it look good. Kudos!
Nice to see someone who knows how to lace cable. But, I am one of the people who uses cable ties.. but I do have the proper tool and use it every day :-)
Woah... I've always got the probes out, but I work on 5V digital stuff... I don't get sparks... I'm not sure if I'm jealous or glad. ;)
As a Telcom engineer calling e1 and b channel/sigtran signaling ancient the marvel of mechanical switches is quite awesome! Then again I also very much like the old skool pinball machines :)
+pokes ground to random contacts+ 2 sparks - hey you found the battery :)
Busy plug has what component(s) in it? How does it make the piece of equipment unavailable?
The busy plug, will open a contact in the panel or place a ground through a contact via a lead which makes the unit show busy to any additional calls.
Is there a advantage (other than cost) of stitching over velcro-type ties?
Easy removal and adding more wires
I'm wondering why they didn't use a toner and tracer to find out more about the wire routing. It's a great tool for just that purpose.
Also, why tie the unconnected wires back? It seems to me that it would be better to tie them to the vertical run where they are out of the way and not blocking the view of all those soldered joints. Or else cut them shorter, since they will never be used again. They emphasized looking neat, well removing unneeded wiring makes it even neater.
Thanks for the informative video. 👍
The shorted wire was grounded, so the tone would just have spilled out everywhere.
@@ConnectionsMuseum "Toning" works best when one lead of the tone signal is connected to a ground, so, in reality, the ground at the other end would just kill it. 30+ years as a cable splicer, toning cable pairs everywhere, definitely taught me that issue. Grounds were always the hardest to troubleshoot, the best device to find / measure a distance to them was to use a TDR. If you don't have an old Metro Tel or WE 76C test set for your museum, you should look into one.
Could you continuity check to ground to figure out which wires might need to be removed?
Thanks Sarah, for another excellent video. It was not clear why you would leave all those unused spare wires in that cable bundle. Was this wiring left from some prior configuration? I would have pruned much of that out, since the bundle was open. However, that may be easier said than done.
About 80% of the wiring in that frame is unused, and if i took it all out, the frame would look naked on that side. Our museum configuration is so tiny, and that frame is meant to serve a much larger number of senders. I left it in for aesthetic reasons, so visitors could see the original wire forms.
In wiring pipe organ relays I have a saying, "Neatness Counts". It counts in telephone relays as well. Lacing in fun.
Great vid for FIRST TIME viewer! Now I gotta learn MORE antique electronics, thanks MA BELL! Just learned how to brew a tetrode from scratch and how to make ic's from sand... Okay, last was not real. Just saying...
I love this stuff. We lost the character of the phone system when we went digital. I wish this weren't on the other side of the country.
Great video Sarah
Do you have one of those RF wire trace and probe devices? They could make it possible to trace wires without cutting the bundles.
A "ground" would kill the tracing signal, so it wouldn't have gone well. That tech knew that also.
I used to be able to make that happen at my local CO back in the days of switched systems! I would take the CO down for a day or two before they could fix the problem.
So much of this reminds me of working on pipe organs. So much of the technology is the same, down to the giant soldering iron you used.
Zip-ties are the devil, but when I have to use them, i've found that toenail clippers are a good inexpensive substitute for the special zip tie flush cutter!
The switch sounded like it was full of angry bees at the beginning when it couldn't find a sender. Nice video! (I am an old Nortel guy who cut my teeth on a SP-1)
great job in fault reconciling in the code decoder
I have a wonder for you @Connections Museum.
When i search for foults in our relayboxes for railway, we have two ways to do.
With a volt meter och a lamp with a magnet for the ground and the test lead.
Is it not possible to do someting sumular in your applications?
this level of true-tech-mystery is addictive
Happy late birthday panel.
Also. A fellow trans girl being ideological about safety precautions and protocols will never not be cute and awesome.
Haven’t they used a line ringer to find the wires?
Have you ever put googly eyes on the thingies that go up and down?
BTW, we would never put all that back together till we verified we had completed the repair by testing. Yes, I know you are confident you found a problem, still there could, for example, be more to the problem.
Is this what they had in the USA? Were there none of those Strowger switches?
If this system is running what specifically is the purpose? Is it actually being used? A bit confused here.
I missed the step where you prove that the system no longer runs out of resources when you run the call generator. (In the spirit of bug tracking & fixing as described in "The pragmatic programmer".)
Hey so I'm have not watched all ur videos yet but I watched enough to understand that these are real calls. I would love to see a specific video on where are these phone calls coming from and how they work with modern centers as a assume they are connecting digitally but I don't know for sure
So you answered a bit, they are not real calls I guess. It would be cool if u some how connected it to real calls with a drop out to a modern center if any failure
You need a thermal camera. That'll show you where current is going when it shouldn't be
Man I was your place like this existed around where I live
Enjoyed very much.
Somebody in your tour bumped their butt up against the rack. That is the initial source of the problem.