I had almost completely forgotten that during storms when power went out (a frequent occurrence in the rural spot where we lived when I was little) the phones would all still work and we used to routinely call others in neighborhood towns to ask about conditions there... and even sometimes call the power company to get updates about repairs. ⛈️☎️
Yeah, I really miss when phones kept working with the power out, ever since telcos in my area switched to fiber-to-the-node they never change the batteries in the node boxes so after the first LONG outage in the winter, all the batteries froze when they discharged too far. Since then, phone and internet go out with the lights. We've tried telling em about the bad batteries, but they don't wanna spend the money, and it's a great reminder that just cause a system is newer, it isn't necessarily better. It's especially bad cause most of the cell towers in the area are ALSO on battery backups, and while they last for about 6 hours when the lights go out, they either don't HAVE auxiliary gensets or they never bothered to fuel them up after the ice storm so once they're dead you're cut off from the world. It sucks that we've somehow REGRESSED in terms of disaster preparedness, it used to be that the landline was THE most reliable utility and would keep on working after everything - record setting ice storms, tornadoes, the literal nuclear apocalypse, no matter what you could still call the emergency numbers on the fridge. Now, you gotta have a Baofeng with an illegal linear amp and a big aerial, and KNOW what FRS channel and squelch code the volunteer search-and-rescue dispatches on cause police/fire/EMS are all on encrypted digital radios, even CB channel 9 isn't monitored at dispatch like it used to be for emergencies.
Many years ago here in Worcester, England, the main exchange office had power problems and service was failing. A fuse of the grasshopper style failed however for some reason did not set the alarm off! That stopped emergency calls in the area. Not good.
That actually played in to a natural disaster I experienced about 10 years ago (flood). The power for the whole town was out for days. The cell towers died after a few hours. One person I knew had a land line that worked the whole time, and they were a major communication link for a group of us that worked together.
I spent 16 years as a C.O. tech (a so called 4 wall tech). When it was time for the yearly BOD test the part I always was concerned about was the coup de fouet readings. Watching the voltage drop and waiting for the bottom and the slow rise back to the operating voltage was always stressful regardless of the fact that it always did come back up. A 100 Gig circuit down was less stressful than almost any power issue. Thanks for the video. Most folks don’t realize how sophisticated a communication network was and is today. A cell phone call in a car on the interstate that doesn’t drop is a modern miracle that people take totally for granted.
I remember i once tried to call home with uncle's cell phone in 1998 but network coverage wasn't good back then in our city. I only heard noise and static (like what you hear in a radio). It feels strange and uncanny when i remember it and compare with what we have today.
Totally about 5-10 years ago there was a major scandal in my area because there was a road rage incident where the individual called 911 and the call dropped well they were on interstate 81. And everybody was like how come it dropped. Well first of all NIMBY how dare you put a cell phone tower in my backyard and we have Verizon, AT&T and T-Mobile all with their own isolated networks not sharing. So of course you're going in and out of service even though you're looking at 50 towers! The one good thing that came out of the 5G switch was that most of the networks have standardized to have the same back end for the most part, except for that weird US cellular and Verizon. I still think it's weird that as a T-Mobile customer I will Rome on AT&T and it will tell me that nowadays, although you can barely do anything because AT&T throttles it beyond belief supposedly it's just enough to make a phone call assuming your phones not trying to use the 64k data line.
As an electrical engineering student in the early 1990's I went on a field trip to visit a telephone exchange. We were shown into a huge room with many channels in the floor designed to carry many cables. Most of the room was empty except for a couple of racks of equipment in one corner and only a handful of cable channels had a couple of wires in them. We students assumed that this room had been designed for future expansion. In reality, it was just the opposite: the exchange had been upgraded to all-solid state equipment and subscriber lines were multiplexed in decentralized equipment cabintes, so all that was coming and going into the exchange were a handful of fiber optic cables!
@@splunge2222 -48v was distributed through Electronic Switching System (ESS) machines but all circuits I can think of used negative to ground circuits except for line feed and ringing. The 4 ESS ran on +130v with a lot of +3V logic.
oh yes, I remember the huge building that was the exchange when I was a kid, now its mostly empty, because its just a couple of optic fiber routers, all of the POTs signaling is generated near the consumer unit on small metal boxes you can find on the walkway every block.
Massachusetts electrician here, I remember in the '90s working in an AT&T building that was one of the last buildings built with the old technology in mind. 4th floor to it, they only used a quarter of one floor in the end. Would not lease out the other floors to businesses for security reasons. It was located on the main Street in a city. The entire face of the front of the building was designed for air intake!!!
In Germany utility companies have their own phone network and switches. The switch in the powerplant where I used to work had been replaced with a modern digital one just before I joined. Some folks were unhappy with the new switch because it needed some pretty long time to restart after a reset or power failure - the latter sounds kinda funny when talking about a power plant ;-) At the same time they still had the old copper wiring. One time a digger destroyed a cable of 750 (afair) pairs nearby. About 1.5m had to replaced so that's 3000 connections plus some more to fix damage at a third point.
Working for one of the independent telcos in FL many years ago, I was working in a CO when the power went out, the diesel generator attempted to crank over, and the starter battery promptly exploded. I thought the building had been struck by lightning. When the system gave up trying to crank that one, the turbine secondary generator whined to life a minute later. Trying to figure out the location where we’d been hit, we walked through the whole building, and when we came to the generator room, the carnage of the blown starter battery with shards of plastic all over the room and freshly sprayed acid made me very glad we didn’t typically have any reason to be in there.
You're so good at explaining and so knowledgeable! It warms my heart whenever young people put great interest in old tech, making sure that the knowledge will be kept for generations to come.
The story about Kennedy's assassination causing the switch overload reminds me of an incident when I was a teenager living in Memphis, TN. In mid-August of 1977 I was home working on my mini telephone office I had setup in a metal building in my parent's back yard. At some point I went into the house and mom was watching TV. At that moment, a very somber news bulletin interrupted her show to announce that Elvis Presley had died after being found slumped over in the master suite bathroom at Graceland. After the initial shock, mom and I both headed to our separate phones to call friends. Both phone lines were served by a crossbar 5 system and getting dial tone was difficult. It was mostly the hum of the office power over the phone until the dial tone finally kicked in. Even then, every single number dialed was met with a fast busy signal indicating busy trunks. I eventually got my call through via persistence, but mom had to wait a while to call anyone. So many people in Memphis were trying to call friends, all at once, who were Elvis fans to give the news and it seemed to overwhelm switches all over Memphis.
In the early '80s, I worked for a company that built video switching systems. We built one system for Pacific Bell; it was to be installed in a CO in Los Angeles (I don't know which one), where it would handle all of Pac Bell's West Coast TV network switching. Because it was to be installed in a CO, it had to run on 48V. We didn't have a 48V system at the assembly plant, so we improvised one with four car batteries in series. An old welding transformer that produced a 50V output charged them. To limit the current, someone went to Sears and got a clothes dryer heating element. The whole thing was built onto a pallet, with the "ballast resistor" standing on ceramic insulators. Although it did not get red hot, it did get too hot to touch, and I always worried that it would ignite the hydrogen offgassing from the batteries. We kept a box fan blowing on the rig when the batteries were charging.
There's an exchange near me in Edinburgh that allegedly has the word CHLORIDE debossed into the battery room ceiling after a misoperation of the plant caused the batteries to gas severely and explode, firing the cell lids upwards. I've never remembered to check for myself.
I never heard of anyone dropping a wrench in a distribution unit but I saw a pair of HVAC techs dropping a duct over the bus bars. Molten galvanized steel smells awful. Keep up your excellent work!
@@simonforget280 back around 1990, the Forman at a major AT&T building in Minneapolis told me someone dropped a 1” diameter steel prybar across the battery output. They had to wait more than 8 hours for it to burn through. All 3/8” threaded rod within 10’ of the fault vaporized. The fire department could only vent the hydrogen from the batteries.
@@rgsparber1I'm surprised it would take 8 hours to burn through unless there was something severely limiting the current. Styropyro did a video some months ago where he wired 100 car batteries up for 60 volts, and he found that larger steel items burned through faster than smaller ones, because the lower resistance allowed them to pass more current (up to 55,000 amps through a 1" bolt iirc). On top of that, one of the biggest problems he faced was trying to keep everything connected long enough for it to melt as the induced magnetic fields kept blowing everything apart. Everything he tested only lasted at most a couple seconds before melting or burning up.
Something similar happened at a power plant, they use DC to provide field to the generators among other things. A worker dropped a tool on the DC bus, the current wasn't high enough to vaporize anything but the wrench just welded itself to the bus and glowed like a big heater. They had to wait quite awhile for the batteries to discharge and then cut the wrench off the bus with a torch. In the meantime the plant is off line. Nobody got hurt but procedures had to be revised for future work in that area.
In the mid 1970s I worked maintenance at Holiday Inn and I had keys to every room in the motel, One of those keys was to the onsite telephone battery room. It was a room about 20 feet wide and 50 feet long that had glass cased batteries in steel racks like the ones in your video that went from floor to ceiling filling that entire room. I was just kind of awestruck by the number of batteries it took for for 200 motel phones, The motel switchboard was the telex system but I was not responsible for repairing that although later in life I repaired four line small business systems. I only went in that battery room once, It is possible it supplied the whole town. Everything in that room was Bell system property so they may have just been sharing the building with us. I wish I had gone in there more then once and studied how it all worked, Your video has enlightened me a little.
I'm always surprised at how much useful (or at least fascinating) information you pack into these videos. For example, the prevention of electrolysis with a positive ground.... Thank you to you and all your team!
Listening to you talk about the level of traffic the exchange could carry, reminded me that in the 80s I was part of the team that were in the Guinness book of world records for carrying IIRC 1.6 million busy-hour call attempts (BHCA) on our model System-X local exchange. Reed relays and silicon do not have the same beauty and sound as a Strowger exchange with its motor selectors and two motion switches!
What an excellent will made video. My knowledge is with X-ray equipment. When I started working installing and repairing X-ray machines in the 1970s many of the systems out in rural hospitals and clinics still operated on relays and vacuum tubes. I recall working on an one million volt industrial X-ray unit that used a dynamotor to convert power. That machine was used to inspect the fuse inside 155mm shells. Back when I was a child in the 1950s I recall going into the local independent telephone office and watch two women work the plug and cord switchboard. The blue glow from the big rectifier tubes in the back office was very eerie.
It's really amazing how much inertia is in telecom. While individual components have changed out over the years, the topology of a modern CO's power plant really hasn't changed all that much. It's still negative 48V. There's still a string of batteries in the basement, although the rectifiers are indeed all solid state. Each rack still has its own set of fuses, complete with monitoring, although that's all electronic these days. Also worth mentioning is why 48V was chosen. It was a balance of safety and conductor size, while still being high enough to reliably break through oxidation on relay contacts.
You forgot to mention a little ditty when you spoke about the grasshopper fuses and the alarms. The big, cartridge fuses had no way of tripping the alarms so they wired a grasshopper fuse in the circuit. Now when a 400 amp cartridge fuse blows, the 1/4 amp grasshopper blows along with it thus tripping the alarm. But you don't know how often I have seen a "tech" standing at a BDFD with a pile of blown 1/4 amp fuses at his feet. "The damn thing keeps blowing every time I replace it!". Then I would show him that the fuse he was trying to replace was actually a "tell tail fuse" for a cartridge fuse!
Yeah, I'm calling bs on that last part. Sure, one fuse blows then replace it. The replacement immediately blew? You have a real problem and you should not try again until the problem is found and fixed. If the answer to a blowing fuse is to blindly keep jamming replacements in then why is the fuse there at all?
@@uzlonewolf unfortunately, these people should have never been CO techs. Yes it's logical for you and I to say that is a stupid thing to do but believe me, these were people with long records of being brainless. These are people who didn't care about their jobs and the union protected them.
Hello Switch Witch, I watched a lot of your videos and learned a lot about telephone switch system types that were completely unknown to me. Thanks for that! Your enthusiasm for this systems is contagious and can be felt all the way here, 7000km away. I am impressed! Can I have some more, please...:-) Greetings from Kiel (Germany) Holger
My father worked in a 5X5 crossbar office from cutover in Nov of 1949 till retirement in Dec 1973 and their power plant changed over the years. When cut into service they replaced operators, and the locals couldn't figure out the "new" system. They had three GE motor generator sets the were 200 amp which evolved into one being replaced with a Westinghouse 400 amp machine, it was a screamer sound wise compared to the GE generators. Finally in the last few years of service they installed a solid-state rectifier that became the number one in the string and ran at full output of around 500 amp. Originally there was no standby power for AC, later they installed a 100kw diesel unit that had to be manually started, just after Dad's retirement a Solar gas turbine unit was installed a 250kw unit. At retirement the exchange had two full units 232 and 439, now there are five exchange units and are all ESS and are in the basement, whole first floor has nothing left but the main frame. Haven't been in the place since ESS as there is no regular staff. Remotely monitored and "nothing can possibly go wrong!"
I spent about 10 years maintaining DC power plants and backup generators. At one site we had 3 2.2 megawatts Solar Generators operating in parallel 400 gallons of fuel per hour I also saw what happens when plumbers working on a water sprinkler system above the bus bar shorted out a ladder from the bus bar to ground. It put a hole in the 1/4 “bus bar. And destroying the ladder When working on these power plants a good safety reminder is to keep one hand in your pocket
As an electrical engineer and master electrician in New York City, it's pretty funny seeing some of these switch gear in a museum, yet. I still see them in major Telecom COs to this day.
A story I heard (take with a grain of salt). At a central office the power (utility) went out. OK, fine we start up the standby generator. At the time this was a gas powered turbine. To start said turbine, a blast of compressed air was sent through to start the blades running, and it was followed by the burning gas. That's fine. Generator up to speed. Now flip the switch to supply the load. Well, over the years, (since the last power failure) loads were added and the generator couldn't handle it very well, so the turbine stalled and died. Not thinking very well, the procedure was tried again (the compressed air supply was good for two starts). Rinse, repeat. So now there is no more air to start the turbine, and all they could do is watch the CO slowly die. The procedure was changed to make the charging of the compressed air supply a top priority, and the loads on the emergency supply side of things was more closely monitored. No, I was not a witness to this, it is a second (or third) hand story. I was told that this happened in San Francisco. Live and learn!
A friend of mine worked for AT&T / Ameritech. He told me there was big fiasco at I believe it was the Bowery Street central office in Akron, Ohio. The building is a fairly tall multistory office. HVAC chillers occupy the top floor and the next floor down has a bank of turbine generators. A chiller failure dowsed the turbine generators and switch gear, bringing down 9-1-1 service and large part of the CO switching. Truck based generators were brought in from an AT&T equipment yard, but the portable generators did not want to start in cold winter weather which was making the fuel gel. It apparently took several days to get everything sorted out and running again. The moral is: don’t mount your generators where they could get wet because they eventually will get wet. My old employer learned that my putting the 50 KVA MG set in the basement under a fire main which broke. TEPCO learned that at the nuke plant in Fukushima where they put the Diesels in the basement. Another fiend had an Internet business site which was hosted at an independent telco in Doylestown, Ohio (not far from Akron, Ohio) in the 1990s. A utility power failure caused a an automatic transfer switch try to start and cut in a Diesel generator, but the transfer switch exploded, knocking his Internet servers out for almost a week. I was an engineer at small data center and we test ran our Diesel every two weeks. The first test of the month was no load. The second test transferred the entire data center to the Diesel for 30 minutes. I always got the classic adrenaline rush every time I pressed the transfer button, remembering the story I was told about Doylestown. When I first started the data center job, there was not a program in place to exercise and test the Diesel generator, and very little preventative maintenance had been done. It was about 2:00 AM on a February night with an ice storm, and I was awakened by my pager. I looked at the pager and saw some power transfer notices from the Symmetra UPS. A little while later, there was another pager message from the Symmetra. Sleepy, I assumed the last notice was about the bump which occurs when the automatic transfer switch goes back to mains from Diesel. When I looked at my pager as I got up to go to work, I was horrified to see the message was the two minute warning from the Symmetra that it was about to shut down. Luckily in that last two minutes, the utility power had come back and saved my bacon. What had happened is that the Diesel had shut off almost immediately because the seal on its water pump blew out. The data center had been running on its UPS batteries for about 45 minutes. Bringing the data center up from being completely off is probably a several hour job and we would have had customer complaints. Karma was kind to us that night.
Decades ago, heard one about the graveyard technician who went to the head: first alarm: the mains went down, the second alarm: the first automatic generator failed to start, the 3rd alarm: the second automatic generator failed to start. He was moving fast by then, as the third and final generator failed to start. (you can also take this one with a grain of salt) LOL Same source; ‘The Painter and his Bucket of Paint’. Bus bar and bucket. Didn’t event cause a blip! Won’t say much about the painter (shaken, but alive). That was the kind of crowd that I was running around with back then. 80’s
One of the problems with the turbines was that once the turbine shut down and the power went out again an exhaust gas temperature sensor would not let the turbine restart if the power went off a second time a short time later. If I remember correctly, the two turbines at Anaheim Lemon (ANHMCA01) were removed and diesel generators put back in because of this.
@@edwinsinclair9853 Was that one of the Solar Gas Turbines? In the late 1960's , we had a Solar 200kw gas turbine set at ARCDCA11 that blew up ... I was on duty when the turbine, running a monthly full load test ate a gland seal (we think), failed to shut down on over speed because fuel was being supplied by way of the gear oil in the planetary gear box, air damper was slammed shut but that only crushed the flexible joint in the plenum, splitting it open and allowing room air to supply the turbine. Over speed failed so the beast just continued until it shutdown, I guess the bearings failed when speed reached some ungodly speed. I was not in the room when all of this happened, but knew something was amiss when all the building lights went off, every alarm known to man came on. Auto transfer switch failed, stuck between commercial and emergency positions so there was no emergency power, or commercial power available. I had to manually restore commercial power breaker. Total down time was about 30 minutes, just enough to get the fire department next door to extract the smoke from the power room, before it worked it's way to the #5xbar switch room on the second floor. Of course, the end cell switch operated, probably because 48 volt plant voltage had dropped down low enough. Long day, Lots of alarms that day, but no real damage to customer service or the switch machines......
I've heard a story (very likely apocryphal, but still interesting) which supposedly took place in a telephone exchange with a manual change-over for the generator, and for some reason, it didn't have a mutually exclusive change-over switch but two on-off switches. One connected the external power lines to the exchange, while the other connected the generator to the exchange. The correct procedure was, of course, to start the generator then, when it got up to speed, disconnect the external power lines then connect the generator. According to the story, during a blackout someone neglected to disconnect the external power lines before connecting the generator, which of course caused the generator to be connected to the entire town. Supposedly (and this is the truly unbelievable part) the generator stalled under the massive load but the engine driving the generator kept running (at least for a few seconds), ripping itself off of its mountings and rolling itself right through a brick wall. Why do I think this is just a fantasy tale? Well, even back then, I would expect there to have been fuses between the generator and the switch, and those fuses would have blown out long before the generator could stall. Of course, safety wasn't always the "big thing" that it is these days, so it's barely possible that it actually did happen.
All of our MG sets were 400 amp, 52 volts They were all Western Electric /GE open frame sets, similar to what you guys have, only painted gray instead of black.. We also had 400 amp solid state rectifiers (NOISEY!) . We had 4 strings of batteries made up of 23- 2.17 Volt cells each, and at least one end cell per string. Emergency power was supplied by a 200 KW Solar Gas Turbine. Our switching system was 2 units of SXS, and 1 unit of #5 XBar. We also had a 130 volt string of cells for ring generator, and Before Ma Bell got rid of them, teletypes requiring 130 V battery. Those cells were I think 2180 AH rating each, and were manufactured by either Excide or CD . If the battery voltage dropped below 48 volts, the emergency end cell switch would cut in extra cells as needed, to keep the voltage at 49.91 volts. Battery strings were connected to overhead copper bus bars hanging from the ceiling on large red insulators. Our main switch panel was all knife switches, some rated at 600 amps. Might be interesting to tell the people how those high amperage fuses were changed, with the knife switches! Dozens and dozens of fuses were used to supply the various switch frames on the 1st and second floors. We were a small local end office CO, but up the street a couple of miles, we had a 4 story CO that was also a major toll hub with lots of microwave. Their emergency power was 2-750 KW Solar gas turbines. I worked at both of these CO's, nearly 10 years before moving east, all the COEM's worked power at some time, we all trained,..I can still remember using that Weston single cell voltmeter, and taking Sp. gravity readings as a weekly routine. I ran the Emergency gas turbine once per week with no load, and once per month with full load. . That little office today is all 5ESS.
I worked in Exchange Construction, and had some rivalry with the Exchange Maintenance staff. We would modify those 'Grasshopper Fuses' so the back part contacted the alarm circuit, but the front flag did not jump out. After the maintenance staff had given up looking for the blown fuse, we would remove it from the unused circuit, and save it for another day.
Not a telco power story but amusing nonetheless. Back in the 80s I was working in DMERT development at Bell Labs. There were several labs, each one consisting of a room with a 3B20D processor and an adjoining room containing a DEC PDP-11/70 computer, which was used to control and monitor the 3B20. Feeding these two rooms were aluminum busbars carrying 208 VAC 3-phase at some horrendous number of amps. One day I was in the lab and a couple of the busbars decided to end it all. There was a deafening ka-BOOM! and I jumped about three feet into the air. I made a beeline for the door, hitting the EPO button on the way out. (The power was already off but, hey, I hit the switch anyway.) The electricians began work on replacing the busbars and temporarily left the old ones lying in the hallway. I was standing around with a few co-workers when two guys walked by. One glanced at these melted, gnarled-up busbars, turned to his friend, and said, "Obviously a software problem."
@@meepferret The labs were supplied with 208 VAC 3-phase. The PDP-11 ran on that directly. To power the 3B20, the AC was stepped down and rectified. Don't forget, this wasn't an entire switch. It was, essentially, just the processor out of a 5ESS. For this application there was no need for a battery room.
I had a summer job for P.T.& T. at 555 Pine St. in San Francisco in 1962. One of the switchmen there told me about the earthquake of 1957. He was working in the crossbar no.5 office. When the earthquake hit the office went silent. Everyone who was on a call stayed on the phone. Those who were about to make a call froze. Then, all of a sudden, the office went crazy with everyone trying to make calls. In short order it locked up as the registered filled up.
The interesting thing about dropping a spanner on busbars is it makes more of a BAMPF noise than a bang per se. The most memorable story I heard of telco power though, was a team of painters who decided that the busbars were a handy place to put a full and sealed can of paint. Did you know UK style System X style switches will work perfectly happily even when covered with white emulsion?
I remember in the 80s (and maybe even early 90s) sometimes when making a landline call, you could hear faint tones leaking into your line whilst waiting for things to connect or whatever was happening. And definitely remember phones working in power outages
Another very interesting video. While I was not able to locate this scene quickly, there are two spots in the video where you show the batteries themselves. The second scene, the one with a woman standing in front of the racks, shows batteries manufactured by the company I worked at for 19 years - C&D Technologies. In earlier times they were known as C&D Battery. Batteries in that configuration normally had multiple cells in them and there was all kinds of strapping (series and/or series/parallel) to deliver 48 volts. Remember that the chemistry of lead-acid batteries tells us that an individual cell only produces 2 volts. This means that you need to wire cells in series to increase the output voltage. I was surprised that there was no mention of AT&T's contribution to the battery industry. I would expect that many of the old timers commenting here will remember the round batteries that they must have encountered during their careers. AT&T as "Ma Bell" was always concerned with longevity and reliability in anything they produced. AT&T engineers set out to create a lead-acid battery that would last longer than traditional batteries. They did this by literally changing the entire configuration. Typical lead-acid batteries are in rectangular "jars". Think of your car battery or the wastebasket sized units on those racks. The plates are all vertical. What happens over time is that the oxide paste that is held by the plates (they are actually lead grids that are filled with paste) ages and some falls to the bottom. Once there is enough to form a connection between plates the battery shorts out and fails. The Round Cell tried to eliminate that by using horizontal plates and then using a cylindrical jar. C&D was the only manufacturer licensed by AT&T to produce the Round Cell. They were in production in various configurations until about 2015. When I hired on at C&D in 2002 I saw a huge room full of these at C&D's Leola, PA manufacturing facility charging away. The other variant AT&T had was a much longer "formation" step. The first application of charging energy to a freshly created battery "forms" the positive and negative paste. The Round Cell took about 30 days to complete this step; normal lead-acid batteries take much, much less time.
The batteries used in the museum are in the basement. The museum is on the second floor of an active CenturyLink central office, and borrows power from the active CO’s power plant.
9th grade field trip was to our local CO - my biggest memory was the power - the rows of lead acid batteries, and then the giant IH Solar gas turbine generator that had notices that it could start automatically without notice - Probably this field trip is what sparked my interest in telco - Also apparently it was rare for a smaller CO to have one of the Solar generators Looking forward to my visit to your museum next month!
This is a great presentation. I’m sure it will generate lots of comments and stories. I have one for the group. In 1967 I was in the Army assigned as an SP4 Dial Central Office Repairman at Ft. Benning, Georgia. Ft. Benning is now Ft. Moore. General Benning fought on the losing side of the Civil War. My job was night and weekend inside plant man for the post. I watched over the four telephone exchanges at the fort during non-working hours. Sounds awful, but it was really pretty neat. I got to work on equipment. The Army Dial Central Office Repairmen on regular duty hours cut the grass while civilian employees worked on equipment. But on to the story - I had just come into the Main Post central office to start my 8:00 AM Saturday to 8:00AM Sunday shift. I took off my jacket, stretched my arms, then the lights went out, followed by the sound of the motor-generator winding down. We are on the batteries alone and we had no backup power source. I ran to cut the switches to the motor-generator so it wouldn’t try to suddenly start up before I had time to figure out what as going on. I called the Wire Chief to report what was going on then called the Post Engineers to find out what happened to my power. The power was off to perform scheduled power line upgrade. Scheduled with who? Would have been good to know in advance. Our battery was 20 years old and the end cells cut in very soon after the power went down. I won’t make you sit through the next three hours while I watched the volt meter slowly wind down. It was tense. But the power did come back on. I cranked the control for the voltage of the motor-generator down before restarting it. I was afraid the exchange load plus the load of recharging the battery might blow the big fuse like the ones you saw in this presentation. We didn’t have a spare for that either. Hey, don’t look at me, I was just a lowly E-4. Planning and decisions were above my pay grade. Anyway, the motor-generator did come back on. I put it on-line and spent the rest of the day slowly cranking the voltage back up to the normal level. Whew.
As a boy scout about 1947 i toured t Colorado Springs central office, a manual exchange. I remember much of the power equipment you show here. They had a V8 truck engine for backup power running off the gas mains. Thanks for the memories.
The 4A machine where I spent years, had a almost city block of 5 foot tall flooded lead-acid batteries. Not as old as yours in the museum so we had big rectifiers to float the batteries.
The negative 48v makes a lot of sense. One other major user of negative DC voltage is London Underground's fourth rail system, with the third rail at +420v and the fourth at -210v.
I worked in AT&T's Integrated Test Network (ITN) as a UNIX SA in the 90's. At the time, it was the 3rd largest network behind AT&T's "real" network and MCI. We had a full battery -48V power plant. I remember walking by the control unit (not sure what it was called) one day and it was supplying -51V at 5600 Amps!
24:00 battery plus on chassis/ground was used in cars until they managed to properly protect wires&contacts from water and found the electrolytic corrosion on the chassis more problematic on the long term.
I remember getting a tour of our local CO as a kid and was always impressed by the big batteries and big DC cabling. Eventually got a job and worked in that building for a good 10 years as a NOC tech. We don't really touch anything we just monitor it though. When there are storms we have to keep track of all the COs that have no power so we can deploy techs with generators if the batteries get too low and it can get quite hectic. Cell towers work the same way, -48v as well. Our office got moved to the call centre building about a year ago though, I miss being in the CO, it was neat in there even though I never really got to touch anything. Interestingly enough I've been setting up a CO power plant at home to keep all my server stuff running, as well as my workstation. It's a work in progress but I have the basics, just saving up for the batteries now, and proper cabling as what I have now is temporary.
I saw it often when I was a kid tagging along with my dad on a weekend call. Remember the old mechanical switches and it was loud click clack all the time.
I definitely remember POTS working during power outages. When we moved to Santa Maria, CA in 1979 the phone numbers for the city were 5 digits. Probably the 5 digits were the local numbers just for Santa Maria. Then one day the phone company told us that we must use 92 in front of our 5 digit numbers. Some years later I had a job at a college - THEY had their own 48 volt lead-acid batteries in their basement for their elaborate 70's style phone system. So cool.
When I was a telephone guy at Kapaun AB (1990s), we had these scheduled PMIs we had to do. One of them was to stress test our batteries, so every Friday we would disconnect the backup generators and commercial power, make sure we stayed up for 4 hours, and then reconnect everything. Well, we are people right, and eventually a mistake was made, and someone forgot to reconnect the power and went home for the weekend. Well...I know everybody here is smart enough to figure out what happened, next lol. So, after all of us had to stand tall on the Commander's carpet getting ripped new ones on Saturday morning, we scratched that stress test from out PMI schedule and never did it again.
@@thewhitefalcon8539This is why in the IT space doing Friday afternoon releases should be avoided if at all possible. And if not avoidable you put extra people on it to mitigate things being missed. :P
Another super vid thanks Sarah! I worked in a large exchange in the UK in 1975 that had a battery stack with 60 gallons (72 US gal) of acid per cell. The Big Switch at the end of the string was scary large, SO tempting to pull the lever...
I really like your presentation style! I found this entire video really interesting. It's sometimes hard to believe that all of this stuff was designed and manufactured, often by hand, before anything resembling a computer existed. Some day I would like to visit your museum. The fact that you have the knowledge and expertise to keep a sample of such old equipment in operation is actually really remarkable. 💖
Another great video Sarah. I spent a lot of time around many of these old power boards in the 80’s. My job was design of the AC side especially for the very large COs. Redundancy was key and no single point failure modes were acceptable. Some central offices were so large that they had as many as 7 AC substations and 6 emergency generators! And there were still a few small substations with M-G sets still running. The museum is an invaluable record of what was achieved in the 20th century. Sadly, it is disappearing day by day. Wire line services are going away at an incredible rate. Even cellular will likely become obsolete within 20 years in favor of an all satellite network. In 50 years, people will have to visit the museum to see what the telephone system looked like.
Thank you for these lovely explainers, which I know take considerable time. I watch a lot, but the complexity of the phone system even pre-digital era is just staggering to me. Down the road, if you can do a large general explainer with some graphics flowcharts on how all these components link together, that would be golden. The engineering here just blows my mind.
I've seen the battery room with its row upon row of clear-case batteries at the old telephone exchange in my town. I've also seen the generator they had for standby power. They used to run it for some time (I don't know exactly how long, but I'd guess at least an hour) each Monday, just to keep the oil flowing and the battery charged. I seem to recall one night when the power went out for a while and I stepped outside the house to look at the darkened town, and it was so quiet that I could hear the generator at the exchange rumbling away, even from around 3-4 blocks away in a more or less diagonal direction. I recall being told that the generator wasn't actually designed to be used for emergency backup, because it tended to throw oil around until it warmed up properly and the seals heated up and closed the gaps properly. It was intended for continuous usage in a situation where the grid wasn't available. I presume the exchange building still has a backup generator, but it would be supplying the Internet equipment and the mobile/cell phone tower. I'm pretty sure the old generator was retired years ago and they have a replacement. And now I'm remembering the time that around half of the distribution equipment in the exchange went out. You could pick up a phone and stand there hearing nothing while the uniselector desperately attempted to find an available path. I think it was out for around 30 minutes until whatever the problem was (I never found out any more) was located and fixed. You could occasionally get a dial tone, but it wasn't easy and you weren't guaranteed to be able to connect. By the way, the first fully automatic telephone exchange was designed by Almon Strowger, which is why the step-by-step switch was also called a Strowger Switch. He designed it using cut-out paper pieces. According to the story, Strowger, an undertaker, was convinced that the manual telephone operator, who was related to his only rival in town, was deliberately directing all calls for "The Undertaker" to the rival company instead of evenly dividing them between the two companies as was the policy. Strowger was motivated by this to work out a way for subscribers to dial the number themselves. As I understand it, Strowger's exchange required 4 wires, with the 2 extra wires being used for number signalling, but later developments allowed the numbers to be conveyed through just two wires along with ring current and sound/voice.
Funny to see you using Audacity to measure noise. As a retired phone company flunky, I’ve been shocked by ring voltage in hot attics more times than I want to admit, and lugged around 20 lb HP test sets that we used to do what Audacity is doing for you here. I can’t remember what test set we used for noise. We had frequency analyzers, TDRs for locating damage, and of course all the digital test equipment for digital circuits like DS1s, ISDNs, etc. But I’m drawing a blank on what we used for noise. It might have just been an ohm meter that had a special “kick” feature, which sorta turned a circuit into a temporary capacitor. Hard to explain, but it was easy to find damaged or wet copper by filling the pair up with a bunch of electrons, then monitoring how long it took to discharge. Of course we also measured echo return loss on designed voice circuits that class 4 and 5 PBXs spit out. Fun times.
Greetings Sarah! Thanks for the very detailed explanation of how power is provided to the equipment. Seeing the battery plant and the heavy gauge wire brought back memories of working in a telecom development lab back in the 80's. -48 power came from a stack of gel batteries that looked like those small blue plastic freezer packs, only much much bigger. Thanks for taking the time to produce this video and all the others. You provide so much information and you have a knack for explaining technical things in a way that makes sense. Your passion for what you do is quite evident.
A benefit of -48 V that could never have been foreseen by the inventor is that you can switch it with N-MOSFETs which are much more efficient than P-MOSFETs. When I did a design for a manufacturer of modern battery backups for cell phone base stations, I was quite happy that I didn't need complicated high-side switches. (Yes, -48 V is still the telecom standard.) But I have a hard time rewiring my brain for positive GND.
In the 1960s, where I live was served by an SxS office. It was neat to hear the electromechanical sounds after going off hook, dialing and ringing. It got quieter when the talk path was cut in at the last selector, but faint electromechanical sounds could still be heard. When making long distance calls, periodic bursts of multi frequency tones could be heard in the background.
Very interesting. During WWII Bell Labs and Western Electric worked together on a top secret weapon called a Mark 24 "Mine". It was an acoustic homing torpedo developed to sink submerged submarines. Bell Labs developed the electronics that used 4 underwater microphones (Hydrophones) to steer the torpedo towards the swishing sound of the U boats propeller. Western Electric used a standard 48 volt telephone battery for power. General Electric supplied a standard 48 volt commercial wash machine motor to power the propeller. The batteries and laundry motor were chosen because they would fit inside a standard torpedo case. Code name for this weapon was "FIDO". It was first used in March, 1943 and was hugely successful. This weapon allowed the supplies for the Normandy invasion to begin being sent to England with less fear they would be sunk. This weapon was classified Top Secret for 40 or 50 years after the war ended.in case it needed to be used during the Cold War. There are TH-cam videos on the Mark 24 "FIDO".
The collective knowledge you all have on these beautiful systems is just astonishing and makes me feel all warm and fuzzy! Then again, it is Summer in Texas so I’m I’ll upgrade that to prickly and sun-stroked. Either way, I really hope to come visit the museum some day and get a chance to meet all of you.
I remember when the phone guy came to change the batteries for my families phone. They were in a wooden box mounted underneath the basement stairs. I remember helicoptering the worker. I think that had to be in the late 60's. I ended up being in the CATV business. We dealt with large battery banks too. Marcos went to put a labeler on top of the DC breakers box. The box didn't have a top. He quickly opened the door and grabbed it. Not noticing that he hit a couple of breakers and knocked out some critical gear that caused an outage (ONS).
It was wonderful to see End Cell switching. That has to be the craziest part of telecom power. I’m not sure you mentioned that power is not disrupted while adding a cell. This means the cell is first shorted out, then put in series with the other cells, and then the short removed. I’m assuming all end cells work the way I was taught.
I was going to ask about End Cell switching. I've heard that it is done by a motorized control that is designed such that nothing can stop it once the process is started. The non-stop feature protects the shunt that is being fried during the process.
Very interesting throughout, and interesting to hear about the grasshopper fuses. At the BBC in the UK the 50V fuses were very similar but were called butterfly fuses . . .
5:30 interesting; I haven't seen autotransformers used for starting motors, but here in Europe, with 400/230V 3-phase AC used for most motors up to tens of kilowatts (and higher voltages like 6kV if you need even more power), we use wye/delta switches; wye puts 230V on the windings and is used for startup at less power, and delta puts the full 400V working voltage on the windings. Slip rings and shunt resistors are sometimes used, they also used to be used as speed control devices, but this function has largely been taken over by variable frequency drives. Give my best to Jay - she's a friggin' genius (how else can I call someone who makes their own vacuum tubes?), next generation Glasslinger coming up :) That's one absolutely gorgeous breaker! Multi-stage switching, remote control... A thing of beauty and a joy for ever. "Scientist: It's much too unpredictable - don't let it overcharge! Barney: What do you mean, overcharge?" Mercury vapor rectifier... I miss PhotonicInduction a LOT. DDoS attack on an old phone switch, eh? Might be one of the first recorded examples. It's very interesting to learn that the negative voltage has a real deal reason to it. All the fuses look interesting too. Thanks a lot for the insightful tour!
After a long stint at Chicago 1 telegraph I spent a couple of years in the 12th floor power room. Once a week Illinois Bell would test their generators and almost never matched phase. Kaboom! BDFB fuses popping everywhere! Once one of the big boys popped and pretty much destroyed the wall facing it. The scariest job I had was routining the 330 volt rectifiers which was done live
Even if it was in Europe they blew the fuse, that would be in excess of 12 kW. 4800 A @ 48 V is 230.4 kW (!) That’s 20 to 40 times the Stones extravagances…
your story telling style reminds me of Tim Hunkin and The Secret Life of Machines... if you haven't seen that old series something makes he think you would like it.
Sarah - you are the Museum’s Tim Hunkin. Thank you for bringing all these systems to life for us. It really is gratifying to see that the art of technical storytelling is alive and well.
Curious electronics student here with lots of questions: 4:32 So the power is always supplied from the generators, and the batteries are only there to smooth out power fluctuations? And if so, how large do the batteries even need to be relative to the load, i.e. how long would they hold up without (/with inadequate) generator input? 7:40 If I had to guess, that's because it also contains enough copper to be very appealing to scrap. Very cool mechanical system, though, I'd definitely be interested in a more detailed discussion of it (Where are the remote actuators built in? How does it distinguish manual and overvoltage/back-EMF shutdown? What's the bottom contact pair with the jacobs-ladder-y arc extinguishers for? Why the three-stage contact mechanism and not just two stages?) 11:23 So are the end cells intended to increase battery voltage temporarily? But if you still regulate to 48V with additional cells in series, the average cell voltage will drop, which is apparently pretty bad (14:16, although I'm surprised that 1.9V instead of 2V already damages lead-acid cells, I thought those were pretty resilient to overdischarge) 3:31 This may be too fundamental a question, but I'm confused how exactly this switch-over works - particularly for the caller in a long-distance call, where the final connection is made in some far-away office: how does the originating office know the call is now connected, so it can switch to the talking battery? And how does the switch-over and resulting voltage drop not result in the connection being dropped?
Good questions! I'll try to provide helpful answers. 1. In the earliest days of common battery telephones (1900s-1920s) the equipment and network ran off of the batteries. When voltage became too low, Battery A (that had been supplying power) was cut over to the charging equipment, and Battery B (which had previously been charging) was cut over to run the equipment. Eventually, they moved to a "floating" style of charging, where the equipment is running on the batteries, but those same batteries are kept continuously charged. The batteries always served two main purposes. To smooth out ripple in the charging circuit, and to provide reliability during power failures. Batteries could run the exchange for a few hours to a few days after a power failure. Exchanges also had diesel generators to charge the batteries, and those could run as long as fuel lasted. AT&T had pretty fantastic supply chain management and they could get fuel anywhere very fast. 2. The circuit breaker is surprisingly complex, and different breakers could be outfitted with different options depending on what the customer needed. The jacobs-laddery-contact-thing near the bottom opens the circuit to the main operating solenoid just after it starts to pull the breaker to the closed position. The solenoid uses A LOT of current, so we have to break it quickly and we don't want to do that with the button that actually starts its operation. Think of it like the "snapping" part of a household light switch. 3. The end cells are only cut in when the voltage of the main battery string is low enough to require them. We don't necessarily regulate down to 48V. That's just a "nominal" voltage. For example, the voltage level at the museum is 52.6 volts. (We don't actually get to decide on that number. That's just what Lumen, the telephone company, provides for us) 4. Talking battery is actually supplied in two places. For the originating side of the call, it is provided by the local office. For the terminating end of the call, it is provided by the far-end office. There is no need to apply talking battery for all of the links in the middle, since the talking battery is really only necessary for the telephones and local hook-state signaling. The human voice AC "wiggles" can travel along the wires just fine without that DC offset. However, you are correct that the switches at either end need to know when to cut in the talking battery to their respective telephones at each end of the connection. This has to do with a concept that we in the telephone business call "supervision". As a part of its design, the switch "knows" when a call is completed, when a subscriber as answered the phone, and generally "what the state of the call is". So all we have to do is provide a relay somewhere in the call circuit that cuts off the "signal" battery and cuts-in the "talking" battery. The connection is not dropped because the switch over is very fast, and the relays responsible for holding the call up basically don't notice the change. Exactly how the "supervision" of the call is transmitted between telephone switches can be quite complex, because it varies over time and between different systems. For now, we'll just have to say that "it works" and leave it at that. For a little more info on supervision, you can check out my video on the 2600hz signaling system here: th-cam.com/video/8PmkUPBhL4U/w-d-xo.html&pp=ygUGMjYwMGh6
Excellent! Funny that you have a Power Designs Inc. model 5015T power supply just like I have around the 7 minute mark! I got it from a ham radio friend of mine who has passed away. He worked for Bell Labs back in the 1970's. It is a great 50 volt 1.5 amp current and voltage limited power supply. Great job on all of these videos, I really enjoy them.
I inherited a 1920s Siemens 380V three phase AC to 60V 300 Amps DC motor-generator from my grandfather. He bought it cheap at a surplus store and converted it to a crazy powerful welding/gouging machine in the 1950s. I still use that monster occasionally for carbon gouging stuff like excavator buckets. What a machine!
in my younger days working for western electric co as an installer, i was working in a new P.R.T.D. bay cutting in new 750 ccm cables. before starting work i went down to the power room and checked the fuse board for that new P.R.T.D. and all the fuses were out and all the circuits were tagged do not make live. i went back up stars and started working. i did not know it but some one removed the tags on the fuse board and put in all the 800 amp fuses, i was using insulated power wrenches but left my ring on. i made contact between the cable end and my ring and frame ground . my ring vaporized with a flash and my hand and face was sprayed with molten metal. my finger was burned to the bone. . i was flash blind but my safety glasses saved my eyes. i was out for almost a year with skin grafts . the supervisor that removed the tags and put in the fuses was fired from his supervisor position .
Wow, you are lucky, we had a commercial electrician working with WECO installing new flourescent lights between the bays in our SXS floor, when the kid he was working with shoved a metal fish tape through the conduit into the open switch box he was working in, he wasn't so lucky. I remember seeing the embedded metal in his safety glasses, severe burns to his arms and chest area.
Very interesting. In the 1970s I worked on mainframe computer systems. We used motor-alternators to supply power to the system. These consisted of a 440v 3 phase motor driving a 440v 3 phase alternator. Their purpose was to give us a clean mains supply.
I think you hinted at it - there is no master power switch. There are also faults in the power bus that occur before fuses. You just wait for these faults to burn through and use fans to blow away the hydrogen cloud.
I always thought that there was a pair of CRAY-1 super computers in the basement and the waste heat was used to boil water into steam. That steam was then used to power a pair of steam turbines that in turn spun the big generators. Or maybe I was thinking of somewhere else. Love all of your videos.
I love coming across your page! I. found one of these rooms in an old hospital that was abandoned in 2019. They had a newer telephony room that ran off of the newer grid, but It was wild seeing this and learning how it worked!
Sarah; will you ever put on a video about CO alarms? the gongs, the alarm bells, etc? I miss those gongs, the clatter of the #5 switch, and the sound of a call going through the SXS office at 4 AM, and at 81 yrs of age, only hear them in my dreams -
My own story related to a battery plant.. Working at AT&T in the 90s.. had just started, toured the building.. was taken into the battery room.. first (and only) time I'd ever seen OPEN CELL BATTERIES filling the room.. oh the smell. lol.
As a teen in the 60's I would connect a high gain mic input to the telephone line and listen to all the things there was to hear. Needless to say it was pretty interesting
This is fantastic. I'm due to visit Seattle in a couple of weeks, and I seriously hope to see the museum. I'm in IT now, but electromechanical, analog stuff still blows my mind, and I would love to see it in action. Thanks so much!
You spoke of the 1963 assassination of Kennedy and the overload of the phone systems. I was 4. I remember my mom struggling to call people and didn't understand why. I understood intimately in 1989 Loma Prieta earthquake (impacting greater San Francisco Bay area). No cell phones in those days. I was in a San Francisco train station with a couple thousand other people waiting to use 12 phones. Sometimes it took 5 minutes for a phone to connect because the phone company couldn't handle the calls. It was hardest to connect to another circuit within the bay area, so people began connecting with people outside the area and have THOSE people call our loved ones at our homes to assure them we were safe.
Could you talk about what happens to the equipment when it loses power? Are all calls dropped when that huge fuse blows - and again when the switch is flipped after replacement? Do they disconnect when it does a "reset" on powerup? Does it just keep working in the last-known state? Are there redundant power paths to every piece of equipment to reduce the impact of blown upstream fuses like on modern servers?
Another great Vid as always. Love your hair, been trying to work out my new colour and green/yellow is one of my favourites....All the best to yourself Sarah and the Team. Joni.
I work in a data center. Our power lineups are configured to swap back to utility one at a time to prevent putting a huge sudden load onto the grid. Also, that beautiful breaker reminds me of a much more elegant version of our big Eaton and Schneider air breakers (4kA instantaneous).
I had almost completely forgotten that during storms when power went out (a frequent occurrence in the rural spot where we lived when I was little) the phones would all still work and we used to routinely call others in neighborhood towns to ask about conditions there... and even sometimes call the power company to get updates about repairs. ⛈️☎️
Yeah, I really miss when phones kept working with the power out, ever since telcos in my area switched to fiber-to-the-node they never change the batteries in the node boxes so after the first LONG outage in the winter, all the batteries froze when they discharged too far. Since then, phone and internet go out with the lights. We've tried telling em about the bad batteries, but they don't wanna spend the money, and it's a great reminder that just cause a system is newer, it isn't necessarily better. It's especially bad cause most of the cell towers in the area are ALSO on battery backups, and while they last for about 6 hours when the lights go out, they either don't HAVE auxiliary gensets or they never bothered to fuel them up after the ice storm so once they're dead you're cut off from the world.
It sucks that we've somehow REGRESSED in terms of disaster preparedness, it used to be that the landline was THE most reliable utility and would keep on working after everything - record setting ice storms, tornadoes, the literal nuclear apocalypse, no matter what you could still call the emergency numbers on the fridge. Now, you gotta have a Baofeng with an illegal linear amp and a big aerial, and KNOW what FRS channel and squelch code the volunteer search-and-rescue dispatches on cause police/fire/EMS are all on encrypted digital radios, even CB channel 9 isn't monitored at dispatch like it used to be for emergencies.
Many years ago here in Worcester, England, the main exchange office had power problems and service was failing. A fuse of the grasshopper style failed however for some reason did not set the alarm off! That stopped emergency calls in the area. Not good.
That actually played in to a natural disaster I experienced about 10 years ago (flood). The power for the whole town was out for days. The cell towers died after a few hours. One person I knew had a land line that worked the whole time, and they were a major communication link for a group of us that worked together.
In the wild! Fancy seeing you here!
@@Alexis-lt3zy hiya! 👋☺️👍
I spent 16 years as a C.O. tech (a so called 4 wall tech). When it was time for the yearly BOD test the part I always was concerned about was the coup de fouet readings. Watching the voltage drop and waiting for the bottom and the slow rise back to the operating voltage was always stressful regardless of the fact that it always did come back up. A 100 Gig circuit down was less stressful than almost any power issue. Thanks for the video. Most folks don’t realize how sophisticated a communication network was and is today. A cell phone call in a car on the interstate that doesn’t drop is a modern miracle that people take totally for granted.
I remember i once tried to call home with uncle's cell phone in 1998 but network coverage wasn't good back then in our city. I only heard noise and static (like what you hear in a radio). It feels strange and uncanny when i remember it and compare with what we have today.
Back when people cared about the customer, the physical plant, and reputation.
Totally about 5-10 years ago there was a major scandal in my area because there was a road rage incident where the individual called 911 and the call dropped well they were on interstate 81. And everybody was like how come it dropped. Well first of all NIMBY how dare you put a cell phone tower in my backyard and we have Verizon, AT&T and T-Mobile all with their own isolated networks not sharing. So of course you're going in and out of service even though you're looking at 50 towers! The one good thing that came out of the 5G switch was that most of the networks have standardized to have the same back end for the most part, except for that weird US cellular and Verizon. I still think it's weird that as a T-Mobile customer I will Rome on AT&T and it will tell me that nowadays, although you can barely do anything because AT&T throttles it beyond belief supposedly it's just enough to make a phone call assuming your phones not trying to use the 64k data line.
@@imark7777777 They pay you two or three grand a month to have one of those cell towers in your yard.
As an electrical engineering student in the early 1990's I went on a field trip to visit a telephone exchange. We were shown into a huge room with many channels in the floor designed to carry many cables. Most of the room was empty except for a couple of racks of equipment in one corner and only a handful of cable channels had a couple of wires in them. We students assumed that this room had been designed for future expansion. In reality, it was just the opposite: the exchange had been upgraded to all-solid state equipment and subscriber lines were multiplexed in decentralized equipment cabintes, so all that was coming and going into the exchange were a handful of fiber optic cables!
Those negative voltage brainworms also infect you if you work on old Volkswagens.
@@splunge2222 -48v was distributed through Electronic Switching System (ESS) machines but all circuits I can think of used negative to ground circuits except for line feed and ringing. The 4 ESS ran on +130v with a lot of +3V logic.
oh yes, I remember the huge building that was the exchange when I was a kid, now its mostly empty, because its just a couple of optic fiber routers, all of the POTs signaling is generated near the consumer unit on small metal boxes you can find on the walkway every block.
Massachusetts electrician here, I remember in the '90s working in an AT&T building that was one of the last buildings built with the old technology in mind. 4th floor to it, they only used a quarter of one floor in the end. Would not lease out the other floors to businesses for security reasons. It was located on the main Street in a city. The entire face of the front of the building was designed for air intake!!!
In Germany utility companies have their own phone network and switches. The switch in the powerplant where I used to work had been replaced with a modern digital one just before I joined. Some folks were unhappy with the new switch because it needed some pretty long time to restart after a reset or power failure - the latter sounds kinda funny when talking about a power plant ;-)
At the same time they still had the old copper wiring. One time a digger destroyed a cable of 750 (afair) pairs nearby. About 1.5m had to replaced so that's 3000 connections plus some more to fix damage at a third point.
Working for one of the independent telcos in FL many years ago, I was working in a CO when the power went out, the diesel generator attempted to crank over, and the starter battery promptly exploded. I thought the building had been struck by lightning. When the system gave up trying to crank that one, the turbine secondary generator whined to life a minute later. Trying to figure out the location where we’d been hit, we walked through the whole building, and when we came to the generator room, the carnage of the blown starter battery with shards of plastic all over the room and freshly sprayed acid made me very glad we didn’t typically have any reason to be in there.
You're so good at explaining and so knowledgeable! It warms my heart whenever young people put great interest in old tech, making sure that the knowledge will be kept for generations to come.
The story about Kennedy's assassination causing the switch overload reminds me of an incident when I was a teenager living in Memphis, TN.
In mid-August of 1977 I was home working on my mini telephone office I had setup in a metal building in my parent's back yard. At some point I went into the house and mom was watching TV. At that moment, a very somber news bulletin interrupted her show to announce that Elvis Presley had died after being found slumped over in the master suite bathroom at Graceland. After the initial shock, mom and I both headed to our separate phones to call friends. Both phone lines were served by a crossbar 5 system and getting dial tone was difficult. It was mostly the hum of the office power over the phone until the dial tone finally kicked in. Even then, every single number dialed was met with a fast busy signal indicating busy trunks. I eventually got my call through via persistence, but mom had to wait a while to call anyone. So many people in Memphis were trying to call friends, all at once, who were Elvis fans to give the news and it seemed to overwhelm switches all over Memphis.
In the early '80s, I worked for a company that built video switching systems. We built one system for Pacific Bell; it was to be installed in a CO in Los Angeles (I don't know which one), where it would handle all of Pac Bell's West Coast TV network switching. Because it was to be installed in a CO, it had to run on 48V. We didn't have a 48V system at the assembly plant, so we improvised one with four car batteries in series. An old welding transformer that produced a 50V output charged them. To limit the current, someone went to Sears and got a clothes dryer heating element. The whole thing was built onto a pallet, with the "ballast resistor" standing on ceramic insulators. Although it did not get red hot, it did get too hot to touch, and I always worried that it would ignite the hydrogen offgassing from the batteries. We kept a box fan blowing on the rig when the batteries were charging.
Now that's some hillbilly shit right there
I love it!!! (and i'd probably do the same...)
There's an exchange near me in Edinburgh that allegedly has the word CHLORIDE debossed into the battery room ceiling after a misoperation of the plant caused the batteries to gas severely and explode, firing the cell lids upwards. I've never remembered to check for myself.
Scotland? I'm guessing US telco technology was always similar to that of the UK?
@@gavinstirling7088Yes there's a lot of commonality between the two.
I never heard of anyone dropping a wrench in a distribution unit but I saw a pair of HVAC techs dropping a duct over the bus bars. Molten galvanized steel smells awful.
Keep up your excellent work!
Metal + bus bars = **bad**
@@carpetbomberz But I bet its quite exciting!
@@simonforget280 back around 1990, the Forman at a major AT&T building in Minneapolis told me someone dropped a 1” diameter steel prybar across the battery output. They had to wait more than 8 hours for it to burn through. All 3/8” threaded rod within 10’ of the fault vaporized. The fire department could only vent the hydrogen from the batteries.
@@rgsparber1I'm surprised it would take 8 hours to burn through unless there was something severely limiting the current. Styropyro did a video some months ago where he wired 100 car batteries up for 60 volts, and he found that larger steel items burned through faster than smaller ones, because the lower resistance allowed them to pass more current (up to 55,000 amps through a 1" bolt iirc). On top of that, one of the biggest problems he faced was trying to keep everything connected long enough for it to melt as the induced magnetic fields kept blowing everything apart. Everything he tested only lasted at most a couple seconds before melting or burning up.
Something similar happened at a power plant, they use DC to provide field to the generators among other things. A worker dropped a tool on the DC bus, the current wasn't high enough to vaporize anything but the wrench just welded itself to the bus and glowed like a big heater. They had to wait quite awhile for the batteries to discharge and then cut the wrench off the bus with a torch. In the meantime the plant is off line. Nobody got hurt but procedures had to be revised for future work in that area.
In the mid 1970s I worked maintenance at Holiday Inn and I had keys to every room in the motel, One of those keys was to the onsite telephone battery room. It was a room about 20 feet wide and 50 feet long that had glass cased batteries in steel racks like the ones in your video that went from floor to ceiling filling that entire room. I was just kind of awestruck by the number of batteries it took for for 200 motel phones, The motel switchboard was the telex system but I was not responsible for repairing that although later in life I repaired four line small business systems. I only went in that battery room once, It is possible it supplied the whole town. Everything in that room was Bell system property so they may have just been sharing the building with us. I wish I had gone in there more then once and studied how it all worked, Your video has enlightened me a little.
I'm always surprised at how much useful (or at least fascinating) information you pack into these videos. For example, the prevention of electrolysis with a positive ground.... Thank you to you and all your team!
Listening to you talk about the level of traffic the exchange could carry, reminded me that in the 80s I was part of the team that were in the Guinness book of world records for carrying IIRC 1.6 million busy-hour call attempts (BHCA) on our model System-X local exchange.
Reed relays and silicon do not have the same beauty and sound as a Strowger exchange with its motor selectors and two motion switches!
What an excellent will made video. My knowledge is with X-ray equipment. When I started working installing and repairing X-ray machines in the 1970s many of the systems out in rural hospitals and clinics still operated on relays and vacuum tubes. I recall working on an one million volt industrial X-ray unit that used a dynamotor to convert power. That machine was used to inspect the fuse inside 155mm shells. Back when I was a child in the 1950s I recall going into the local independent telephone office and watch two women work the plug and cord switchboard. The blue glow from the big rectifier tubes in the back office was very eerie.
It's really amazing how much inertia is in telecom. While individual components have changed out over the years, the topology of a modern CO's power plant really hasn't changed all that much. It's still negative 48V. There's still a string of batteries in the basement, although the rectifiers are indeed all solid state. Each rack still has its own set of fuses, complete with monitoring, although that's all electronic these days.
Also worth mentioning is why 48V was chosen. It was a balance of safety and conductor size, while still being high enough to reliably break through oxidation on relay contacts.
You forgot to mention a little ditty when you spoke about the grasshopper fuses and the alarms. The big, cartridge fuses had no way of tripping the alarms so they wired a grasshopper fuse in the circuit. Now when a 400 amp cartridge fuse blows, the 1/4 amp grasshopper blows along with it thus tripping the alarm.
But you don't know how often I have seen a "tech" standing at a BDFD with a pile of blown 1/4 amp fuses at his feet. "The damn thing keeps blowing every time I replace it!". Then I would show him that the fuse he was trying to replace was actually a "tell tail fuse" for a cartridge fuse!
Yeah, I'm calling bs on that last part. Sure, one fuse blows then replace it. The replacement immediately blew? You have a real problem and you should not try again until the problem is found and fixed. If the answer to a blowing fuse is to blindly keep jamming replacements in then why is the fuse there at all?
@@uzlonewolf unfortunately, these people should have never been CO techs. Yes it's logical for you and I to say that is a stupid thing to do but believe me, these were people with long records of being brainless. These are people who didn't care about their jobs and the union protected them.
You had to be careful that the spring part had tension on it.
Another brilliant video from Sarah and the team. One of the best technical channels on youTube.
Love these explainer videos. Amazing to see all the details of a centrals office's power systems.
Hello Switch Witch,
I watched a lot of your videos and learned a lot about telephone switch system types that were completely unknown to me. Thanks for that!
Your enthusiasm for this systems is contagious and can be felt all the way here, 7000km away.
I am impressed!
Can I have some more, please...:-)
Greetings from Kiel (Germany)
Holger
My father worked in a 5X5 crossbar office from cutover in Nov of 1949 till retirement in Dec 1973 and their power plant changed over the years. When cut into service they replaced operators, and the locals couldn't figure out the "new" system. They had three GE motor generator sets the were 200 amp which evolved into one being replaced with a Westinghouse 400 amp machine, it was a screamer sound wise compared to the GE generators. Finally in the last few years of service they installed a solid-state rectifier that became the number one in the string and ran at full output of around 500 amp. Originally there was no standby power for AC, later they installed a 100kw diesel unit that had to be manually started, just after Dad's retirement a Solar gas turbine unit was installed a 250kw unit. At retirement the exchange had two full units 232 and 439, now there are five exchange units and are all ESS and are in the basement, whole first floor has nothing left but the main frame. Haven't been in the place since ESS as there is no regular staff. Remotely monitored and "nothing can possibly go wrong!"
I spent about 10 years maintaining DC power plants and backup generators. At one site we had 3 2.2 megawatts Solar Generators operating in parallel 400 gallons of fuel per hour
I also saw what happens when plumbers working on a water sprinkler system above the bus bar shorted out a ladder from the bus bar to ground. It put a hole in the 1/4 “bus bar. And destroying the ladder
When working on these power plants a good safety reminder is to keep one hand in your pocket
As an electrical engineer and master electrician in New York City, it's pretty funny seeing some of these switch gear in a museum, yet. I still see them in major Telecom COs to this day.
As an electrical engineer, that was an awesome video.
A story I heard (take with a grain of salt). At a central office the power (utility) went out. OK, fine we start up the standby generator. At the time this was a gas powered turbine. To start said turbine, a blast of compressed air was sent through to start the blades running, and it was followed by the burning gas. That's fine. Generator up to speed. Now flip the switch to supply the load. Well, over the years, (since the last power failure) loads were added and the generator couldn't handle it very well, so the turbine stalled and died. Not thinking very well, the procedure was tried again (the compressed air supply was good for two starts). Rinse, repeat. So now there is no more air to start the turbine, and all they could do is watch the CO slowly die. The procedure was changed to make the charging of the compressed air supply a top priority, and the loads on the emergency supply side of things was more closely monitored.
No, I was not a witness to this, it is a second (or third) hand story. I was told that this happened in San Francisco. Live and learn!
A friend of mine worked for AT&T / Ameritech. He told me there was big fiasco at I believe it was the Bowery Street central office in Akron, Ohio. The building is a fairly tall multistory office. HVAC chillers occupy the top floor and the next floor down has a bank of turbine generators. A chiller failure dowsed the turbine generators and switch gear, bringing down 9-1-1 service and large part of the CO switching. Truck based generators were brought in from an AT&T equipment yard, but the portable generators did not want to start in cold winter weather which was making the fuel gel. It apparently took several days to get everything sorted out and running again.
The moral is: don’t mount your generators where they could get wet because they eventually will get wet. My old employer learned that my putting the 50 KVA MG set in the basement under a fire main which broke. TEPCO learned that at the nuke plant in Fukushima where they put the Diesels in the basement.
Another fiend had an Internet business site which was hosted at an independent telco in Doylestown, Ohio (not far from Akron, Ohio) in the 1990s. A utility power failure caused a an automatic transfer switch try to start and cut in a Diesel generator, but the transfer switch exploded, knocking his Internet servers out for almost a week. I was an engineer at small data center and we test ran our Diesel every two weeks. The first test of the month was no load. The second test transferred the entire data center to the Diesel for 30 minutes. I always got the classic adrenaline rush every time I pressed the transfer button, remembering the story I was told about Doylestown.
When I first started the data center job, there was not a program in place to exercise and test the Diesel generator, and very little preventative maintenance had been done. It was about 2:00 AM on a February night with an ice storm, and I was awakened by my pager. I looked at the pager and saw some power transfer notices from the Symmetra UPS. A little while later, there was another pager message from the Symmetra. Sleepy, I assumed the last notice was about the bump which occurs when the automatic transfer switch goes back to mains from Diesel. When I looked at my pager as I got up to go to work, I was horrified to see the message was the two minute warning from the Symmetra that it was about to shut down. Luckily in that last two minutes, the utility power had come back and saved my bacon. What had happened is that the Diesel had shut off almost immediately because the seal on its water pump blew out. The data center had been running on its UPS batteries for about 45 minutes. Bringing the data center up from being completely off is probably a several hour job and we would have had customer complaints. Karma was kind to us that night.
Decades ago, heard one about the graveyard technician who went to the head: first alarm: the mains went down, the second alarm: the first automatic generator failed to start, the 3rd alarm: the second automatic generator failed to start. He was moving fast by then, as the third and final generator failed to start. (you can also take this one with a grain of salt) LOL
Same source; ‘The Painter and his Bucket of Paint’. Bus bar and bucket. Didn’t event cause a blip! Won’t say much about the painter (shaken, but alive).
That was the kind of crowd that I was running around with back then. 80’s
One of the problems with the turbines was that once the turbine shut down and the power went out again an exhaust gas temperature sensor would not let the turbine restart if the power went off a second time a short time later. If I remember correctly, the two turbines at Anaheim Lemon (ANHMCA01) were removed and diesel generators put back in because of this.
@@edwinsinclair9853 Was that one of the Solar Gas Turbines?
In the late 1960's , we had a Solar 200kw gas turbine set at ARCDCA11 that blew up ... I was on duty when the turbine, running a monthly full load test ate a gland seal (we think), failed to shut down on over speed because fuel was being supplied by way of the gear oil in the planetary gear box, air damper was slammed shut but that only crushed the flexible joint in the plenum, splitting it open and allowing room air to supply the turbine. Over speed failed so the beast just continued until it shutdown, I guess the bearings failed when speed reached some ungodly speed. I was not in the room when all of this happened, but knew something was amiss when all the building lights went off, every alarm known to man came on. Auto transfer switch failed, stuck between commercial and emergency positions so there was no emergency power, or commercial power available. I had to manually restore commercial power breaker. Total down time was about 30 minutes, just enough to get the fire department next door to extract the smoke from the power room, before it worked it's way to the #5xbar switch room on the second floor. Of course, the end cell switch operated, probably because 48 volt plant voltage had dropped down low enough.
Long day, Lots of alarms that day, but no real damage to customer service or the switch machines......
I've heard a story (very likely apocryphal, but still interesting) which supposedly took place in a telephone exchange with a manual change-over for the generator, and for some reason, it didn't have a mutually exclusive change-over switch but two on-off switches. One connected the external power lines to the exchange, while the other connected the generator to the exchange. The correct procedure was, of course, to start the generator then, when it got up to speed, disconnect the external power lines then connect the generator.
According to the story, during a blackout someone neglected to disconnect the external power lines before connecting the generator, which of course caused the generator to be connected to the entire town.
Supposedly (and this is the truly unbelievable part) the generator stalled under the massive load but the engine driving the generator kept running (at least for a few seconds), ripping itself off of its mountings and rolling itself right through a brick wall.
Why do I think this is just a fantasy tale? Well, even back then, I would expect there to have been fuses between the generator and the switch, and those fuses would have blown out long before the generator could stall.
Of course, safety wasn't always the "big thing" that it is these days, so it's barely possible that it actually did happen.
All of our MG sets were 400 amp, 52 volts They were all Western Electric /GE open frame sets, similar to what you guys have, only painted gray instead of black.. We also had 400 amp solid state rectifiers (NOISEY!) . We had 4 strings of batteries made up of 23- 2.17 Volt cells each, and at least one end cell per string. Emergency power was supplied by a 200 KW Solar Gas Turbine. Our switching system was 2 units of SXS, and 1 unit of #5 XBar. We also had a 130 volt string of cells for ring generator, and Before Ma Bell got rid of them, teletypes requiring 130 V battery. Those cells were I think 2180 AH rating each, and were manufactured by either Excide or CD . If the battery voltage dropped below 48 volts, the emergency end cell switch would cut in extra cells as needed, to keep the voltage at 49.91 volts. Battery strings were connected to overhead copper bus bars hanging from the ceiling on large red insulators. Our main switch panel was all knife switches, some rated at 600 amps. Might be interesting to tell the people how those high amperage fuses were changed, with the knife switches! Dozens and dozens of fuses were used to supply the various switch frames on the 1st and second floors. We were a small local end office CO, but up the street a couple of miles, we had a 4 story CO that was also a major toll hub with lots of microwave. Their emergency power was 2-750 KW Solar gas turbines. I worked at both of these CO's, nearly 10 years before moving east, all the COEM's worked power at some time, we all trained,..I can still remember using that Weston single cell voltmeter, and taking Sp. gravity readings as a weekly routine. I ran the Emergency gas turbine once per week with no load, and once per month with full load. . That little office today is all 5ESS.
Neat to learn why -48 volts. I really enjoyed this video & look forward to more deep dives.
I worked in Exchange Construction, and had some rivalry with the Exchange Maintenance staff. We would modify those 'Grasshopper Fuses' so the back part contacted the alarm circuit, but the front flag did not jump out. After the maintenance staff had given up looking for the blown fuse, we would remove it from the unused circuit, and save it for another day.
Not a telco power story but amusing nonetheless.
Back in the 80s I was working in DMERT development at Bell Labs. There were several labs, each one consisting of a room with a 3B20D processor and an adjoining room containing a DEC PDP-11/70 computer, which was used to control and monitor the 3B20.
Feeding these two rooms were aluminum busbars carrying 208 VAC 3-phase at some horrendous number of amps.
One day I was in the lab and a couple of the busbars decided to end it all. There was a deafening ka-BOOM! and I jumped about three feet into the air. I made a beeline for the door, hitting the EPO button on the way out. (The power was already off but, hey, I hit the switch anyway.)
The electricians began work on replacing the busbars and temporarily left the old ones lying in the hallway. I was standing around with a few co-workers when two guys walked by. One glanced at these melted, gnarled-up busbars, turned to his friend, and said, "Obviously a software problem."
I thought telco was 48V DC, albeit at a LOTTA AMPS
@@meepferret The labs were supplied with 208 VAC 3-phase. The PDP-11 ran on that directly. To power the 3B20, the AC was stepped down and rectified. Don't forget, this wasn't an entire switch. It was, essentially, just the processor out of a 5ESS. For this application there was no need for a battery room.
I had a summer job for P.T.& T. at 555 Pine St. in San Francisco in 1962. One of the switchmen there told me about the earthquake of 1957. He was working in the crossbar no.5 office. When the earthquake hit the office went silent. Everyone who was on a call stayed on the phone. Those who were about to make a call froze. Then, all of a sudden, the office went crazy with everyone trying to make calls. In short order it locked up as the registered filled up.
Really happy I had the chance to visit in person last month!
The interesting thing about dropping a spanner on busbars is it makes more of a BAMPF noise than a bang per se.
The most memorable story I heard of telco power though, was a team of painters who decided that the busbars were a handy place to put a full and sealed can of paint. Did you know UK style System X style switches will work perfectly happily even when covered with white emulsion?
I remember in the 80s (and maybe even early 90s) sometimes when making a landline call, you could hear faint tones leaking into your line whilst waiting for things to connect or whatever was happening. And definitely remember phones working in power outages
Another very interesting video. While I was not able to locate this scene quickly, there are two spots in the video where you show the batteries themselves. The second scene, the one with a woman standing in front of the racks, shows batteries manufactured by the company I worked at for 19 years - C&D Technologies. In earlier times they were known as C&D Battery. Batteries in that configuration normally had multiple cells in them and there was all kinds of strapping (series and/or series/parallel) to deliver 48 volts. Remember that the chemistry of lead-acid batteries tells us that an individual cell only produces 2 volts. This means that you need to wire cells in series to increase the output voltage.
I was surprised that there was no mention of AT&T's contribution to the battery industry. I would expect that many of the old timers commenting here will remember the round batteries that they must have encountered during their careers. AT&T as "Ma Bell" was always concerned with longevity and reliability in anything they produced. AT&T engineers set out to create a lead-acid battery that would last longer than traditional batteries. They did this by literally changing the entire configuration. Typical lead-acid batteries are in rectangular "jars". Think of your car battery or the wastebasket sized units on those racks. The plates are all vertical. What happens over time is that the oxide paste that is held by the plates (they are actually lead grids that are filled with paste) ages and some falls to the bottom. Once there is enough to form a connection between plates the battery shorts out and fails. The Round Cell tried to eliminate that by using horizontal plates and then using a cylindrical jar. C&D was the only manufacturer licensed by AT&T to produce the Round Cell. They were in production in various configurations until about 2015. When I hired on at C&D in 2002 I saw a huge room full of these at C&D's Leola, PA manufacturing facility charging away. The other variant AT&T had was a much longer "formation" step. The first application of charging energy to a freshly created battery "forms" the positive and negative paste. The Round Cell took about 30 days to complete this step; normal lead-acid batteries take much, much less time.
The batteries used in the museum are in the basement. The museum is on the second floor of an active CenturyLink central office, and borrows power from the active CO’s power plant.
9th grade field trip was to our local CO - my biggest memory was the power - the rows of lead acid batteries, and then the giant IH Solar gas turbine generator that had notices that it could start automatically without notice - Probably this field trip is what sparked my interest in telco - Also apparently it was rare for a smaller CO to have one of the Solar generators Looking forward to my visit to your museum next month!
This is a great presentation. I’m sure it will generate lots of comments and stories. I have one for the group.
In 1967 I was in the Army assigned as an SP4 Dial Central Office Repairman at Ft. Benning, Georgia. Ft. Benning is now Ft. Moore. General Benning fought on the losing side of the Civil War. My job was night and weekend inside plant man for the post. I watched over the four telephone exchanges at the fort during non-working hours. Sounds awful, but it was really pretty neat. I got to work on equipment. The Army Dial Central Office Repairmen on regular duty hours cut the grass while civilian employees worked on equipment. But on to the story - I had just come into the Main Post central office to start my 8:00 AM Saturday to 8:00AM Sunday shift. I took off my jacket, stretched my arms, then the lights went out, followed by the sound of the motor-generator winding down. We are on the batteries alone and we had no backup power source. I ran to cut the switches to the motor-generator so it wouldn’t try to suddenly start up before I had time to figure out what as going on. I called the Wire Chief to report what was going on then called the Post Engineers to find out what happened to my power. The power was off to perform scheduled power line upgrade. Scheduled with who? Would have been good to know in advance.
Our battery was 20 years old and the end cells cut in very soon after the power went down. I won’t make you sit through the next three hours while I watched the volt meter slowly wind down. It was tense. But the power did come back on. I cranked the control for the voltage of the motor-generator down before restarting it. I was afraid the exchange load plus the load of recharging the battery might blow the big fuse like the ones you saw in this presentation. We didn’t have a spare for that either. Hey, don’t look at me, I was just a lowly E-4. Planning and decisions were above my pay grade. Anyway, the motor-generator did come back on. I put it on-line and spent the rest of the day slowly cranking the voltage back up to the normal level. Whew.
As a boy scout about 1947 i toured t Colorado Springs central office, a manual exchange. I remember much of the power equipment you show here. They had a V8 truck engine for backup power running off the gas mains. Thanks for the memories.
The 4A machine where I spent years, had a almost city block of 5 foot tall flooded lead-acid batteries. Not as old as yours in the museum so we had big rectifiers to float the batteries.
The negative 48v makes a lot of sense. One other major user of negative DC voltage is London Underground's fourth rail system, with the third rail at +420v and the fourth at -210v.
I worked in AT&T's Integrated Test Network (ITN) as a UNIX SA in the 90's. At the time, it was the 3rd largest network behind AT&T's "real" network and MCI. We had a full battery -48V power plant. I remember walking by the control unit (not sure what it was called) one day and it was supplying -51V at 5600 Amps!
24:00 battery plus on chassis/ground was used in cars until they managed to properly protect wires&contacts from water and found the electrolytic corrosion on the chassis more problematic on the long term.
THIS WAS ABOUT 1955 OR 1956, WHEN CAR MFGS CHANGED OVER TO 12 VOLT SYSTEMS, also about the time that generators were changed over to alternators.
@@henrythompson7595"+ on GND" was kept in cars from Japan well into the 1980ies, resulting many oopies for people doing car audio retrofits.
I remember getting a tour of our local CO as a kid and was always impressed by the big batteries and big DC cabling. Eventually got a job and worked in that building for a good 10 years as a NOC tech. We don't really touch anything we just monitor it though. When there are storms we have to keep track of all the COs that have no power so we can deploy techs with generators if the batteries get too low and it can get quite hectic. Cell towers work the same way, -48v as well. Our office got moved to the call centre building about a year ago though, I miss being in the CO, it was neat in there even though I never really got to touch anything. Interestingly enough I've been setting up a CO power plant at home to keep all my server stuff running, as well as my workstation. It's a work in progress but I have the basics, just saving up for the batteries now, and proper cabling as what I have now is temporary.
I saw it often when I was a kid tagging along with my dad on a weekend call. Remember the old mechanical switches and it was loud click clack all the time.
That is some cool old equipment being preserved! Never would have thought such massive power equipment was in a telephone office.
I definitely remember POTS working during power outages. When we moved to Santa Maria, CA in 1979 the phone numbers for the city were 5 digits. Probably the 5 digits were the local numbers just for Santa Maria. Then one day the phone company told us that we must use 92 in front of our 5 digit numbers. Some years later I had a job at a college - THEY had their own 48 volt lead-acid batteries in their basement for their elaborate 70's style phone system. So cool.
When I was a telephone guy at Kapaun AB (1990s), we had these scheduled PMIs we had to do. One of them was to stress test our batteries, so every Friday we would disconnect the backup generators and commercial power, make sure we stayed up for 4 hours, and then reconnect everything. Well, we are people right, and eventually a mistake was made, and someone forgot to reconnect the power and went home for the weekend. Well...I know everybody here is smart enough to figure out what happened, next lol. So, after all of us had to stand tall on the Commander's carpet getting ripped new ones on Saturday morning, we scratched that stress test from out PMI schedule and never did it again.
Move the test to Wednesday morning.
@thewhitefalcon8539 that was discussed, but we were ordered to stop doing it.
@@thewhitefalcon8539This is why in the IT space doing Friday afternoon releases should be avoided if at all possible. And if not avoidable you put extra people on it to mitigate things being missed. :P
Awesome video and well-explained.
Another super vid thanks Sarah! I worked in a large exchange in the UK in 1975 that had a battery stack with 60 gallons (72 US gal) of acid per cell. The Big Switch at the end of the string was scary large, SO tempting to pull the lever...
I really like your presentation style! I found this entire video really interesting. It's sometimes hard to believe that all of this stuff was designed and manufactured, often by hand, before anything resembling a computer existed. Some day I would like to visit your museum. The fact that you have the knowledge and expertise to keep a sample of such old equipment in operation is actually really remarkable. 💖
Another great video Sarah. I spent a lot of time around many of these old power boards in the 80’s. My job was design of the AC side especially for the very large COs. Redundancy was key and no single point failure modes were acceptable. Some central offices were so large that they had as many as 7 AC substations and 6 emergency generators! And there were still a few small substations with M-G sets still running.
The museum is an invaluable record of what was achieved in the 20th century. Sadly, it is disappearing day by day. Wire line services are going away at an incredible rate. Even cellular will likely become obsolete within 20 years in favor of an all satellite network. In 50 years, people will have to visit the museum to see what the telephone system looked like.
Satellite will only replace cellular in rural areas, urban areas are just too dense even for the new LEO constellations they're launching.
I really need to make a run to Seattle to check out this museum - so cool!
Thank you for these lovely explainers, which I know take considerable time. I watch a lot, but the complexity of the phone system even pre-digital era is just staggering to me. Down the road, if you can do a large general explainer with some graphics flowcharts on how all these components link together, that would be golden. The engineering here just blows my mind.
I've seen the battery room with its row upon row of clear-case batteries at the old telephone exchange in my town. I've also seen the generator they had for standby power. They used to run it for some time (I don't know exactly how long, but I'd guess at least an hour) each Monday, just to keep the oil flowing and the battery charged. I seem to recall one night when the power went out for a while and I stepped outside the house to look at the darkened town, and it was so quiet that I could hear the generator at the exchange rumbling away, even from around 3-4 blocks away in a more or less diagonal direction.
I recall being told that the generator wasn't actually designed to be used for emergency backup, because it tended to throw oil around until it warmed up properly and the seals heated up and closed the gaps properly. It was intended for continuous usage in a situation where the grid wasn't available.
I presume the exchange building still has a backup generator, but it would be supplying the Internet equipment and the mobile/cell phone tower. I'm pretty sure the old generator was retired years ago and they have a replacement.
And now I'm remembering the time that around half of the distribution equipment in the exchange went out. You could pick up a phone and stand there hearing nothing while the uniselector desperately attempted to find an available path. I think it was out for around 30 minutes until whatever the problem was (I never found out any more) was located and fixed. You could occasionally get a dial tone, but it wasn't easy and you weren't guaranteed to be able to connect.
By the way, the first fully automatic telephone exchange was designed by Almon Strowger, which is why the step-by-step switch was also called a Strowger Switch. He designed it using cut-out paper pieces.
According to the story, Strowger, an undertaker, was convinced that the manual telephone operator, who was related to his only rival in town, was deliberately directing all calls for "The Undertaker" to the rival company instead of evenly dividing them between the two companies as was the policy. Strowger was motivated by this to work out a way for subscribers to dial the number themselves.
As I understand it, Strowger's exchange required 4 wires, with the 2 extra wires being used for number signalling, but later developments allowed the numbers to be conveyed through just two wires along with ring current and sound/voice.
Funny to see you using Audacity to measure noise. As a retired phone company flunky, I’ve been shocked by ring voltage in hot attics more times than I want to admit, and lugged around 20 lb HP test sets that we used to do what Audacity is doing for you here. I can’t remember what test set we used for noise. We had frequency analyzers, TDRs for locating damage, and of course all the digital test equipment for digital circuits like DS1s, ISDNs, etc. But I’m drawing a blank on what we used for noise. It might have just been an ohm meter that had a special “kick” feature, which sorta turned a circuit into a temporary capacitor. Hard to explain, but it was easy to find damaged or wet copper by filling the pair up with a bunch of electrons, then monitoring how long it took to discharge.
Of course we also measured echo return loss on designed voice circuits that class 4 and 5 PBXs spit out.
Fun times.
Greetings Sarah! Thanks for the very detailed explanation of how power is provided to the equipment. Seeing the battery plant and the heavy gauge wire brought back memories of working in a telecom development lab back in the 80's. -48 power came from a stack of gel batteries that looked like those small blue plastic freezer packs, only much much bigger.
Thanks for taking the time to produce this video and all the others. You provide so much information and you have a knack for explaining technical things in a way that makes sense. Your passion for what you do is quite evident.
A benefit of -48 V that could never have been foreseen by the inventor is that you can switch it with N-MOSFETs which are much more efficient than P-MOSFETs. When I did a design for a manufacturer of modern battery backups for cell phone base stations, I was quite happy that I didn't need complicated high-side switches. (Yes, -48 V is still the telecom standard.) But I have a hard time rewiring my brain for positive GND.
In the 1960s, where I live was served by an SxS office. It was neat to hear the electromechanical sounds after going off hook, dialing and ringing. It got quieter when the talk path was cut in at the last selector, but faint electromechanical sounds could still be heard. When making long distance calls, periodic bursts of multi frequency tones could be heard in the background.
Very interesting. During WWII Bell Labs and Western Electric worked together on a top secret weapon called a Mark 24 "Mine". It was an acoustic homing torpedo developed to sink submerged submarines. Bell Labs developed the electronics that used 4 underwater microphones (Hydrophones) to steer the torpedo towards the swishing sound of the U boats propeller. Western Electric used a standard 48 volt telephone battery for power. General Electric supplied a standard 48 volt commercial wash machine motor to power the propeller. The batteries and laundry motor were chosen because they would fit inside a standard torpedo case. Code name for this weapon was "FIDO". It was first used in March, 1943 and was hugely successful. This weapon allowed the supplies for the Normandy invasion to begin being sent to England with less fear they would be sunk. This weapon was classified Top Secret for 40 or 50 years after the war ended.in case it needed to be used during the Cold War. There are TH-cam videos on the Mark 24 "FIDO".
The collective knowledge you all have on these beautiful systems is just astonishing and makes me feel all warm and fuzzy! Then again, it is Summer in Texas so I’m I’ll upgrade that to prickly and sun-stroked. Either way, I really hope to come visit the museum some day and get a chance to meet all of you.
I remember when the phone guy came to change the batteries for my families phone. They were in a wooden box mounted underneath the basement stairs. I remember helicoptering the worker. I think that had to be in the late 60's. I ended up being in the CATV business. We dealt with large battery banks too. Marcos went to put a labeler on top of the DC breakers box. The box didn't have a top. He quickly opened the door and grabbed it. Not noticing that he hit a couple of breakers and knocked out some critical gear that caused an outage (ONS).
It was wonderful to see End Cell switching. That has to be the craziest part of telecom power. I’m not sure you mentioned that power is not disrupted while adding a cell. This means the cell is first shorted out, then put in series with the other cells, and then the short removed. I’m assuming all end cells work the way I was taught.
I was going to ask about End Cell switching. I've heard that it is done by a motorized control that is designed such that nothing can stop it once the process is started. The non-stop feature protects the shunt that is being fried during the process.
Very interesting throughout, and interesting to hear about the grasshopper fuses. At the BBC in the UK the 50V fuses were very similar but were called butterfly fuses . . .
5:30 interesting; I haven't seen autotransformers used for starting motors, but here in Europe, with 400/230V 3-phase AC used for most motors up to tens of kilowatts (and higher voltages like 6kV if you need even more power), we use wye/delta switches; wye puts 230V on the windings and is used for startup at less power, and delta puts the full 400V working voltage on the windings. Slip rings and shunt resistors are sometimes used, they also used to be used as speed control devices, but this function has largely been taken over by variable frequency drives.
Give my best to Jay - she's a friggin' genius (how else can I call someone who makes their own vacuum tubes?), next generation Glasslinger coming up :)
That's one absolutely gorgeous breaker! Multi-stage switching, remote control... A thing of beauty and a joy for ever.
"Scientist: It's much too unpredictable - don't let it overcharge!
Barney: What do you mean, overcharge?"
Mercury vapor rectifier... I miss PhotonicInduction a LOT.
DDoS attack on an old phone switch, eh? Might be one of the first recorded examples.
It's very interesting to learn that the negative voltage has a real deal reason to it. All the fuses look interesting too. Thanks a lot for the insightful tour!
This is so facinating. Your content has gotten better and better and this, and the power plant, probably my favorite.
After a long stint at Chicago 1 telegraph I spent a couple of years in the 12th floor power room. Once a week Illinois Bell would test their generators and almost never matched phase. Kaboom! BDFB fuses popping everywhere! Once one of the big boys popped and pretty much destroyed the wall facing it. The scariest job I had was routining the 330 volt rectifiers which was done live
The Rolling Stones blowing a 50 amp fuse got nothing on that 4,800 amp fuse!
Even if it was in Europe they blew the fuse, that would be in excess of 12 kW.
4800 A @ 48 V is 230.4 kW (!) That’s 20 to 40 times the Stones extravagances…
your story telling style reminds me of Tim Hunkin and The Secret Life of Machines... if you haven't seen that old series something makes he think you would like it.
I *love* The Secret Life of Machines! Being mentioned in the same sentence as Tim is quite an honor :)
Well, you’re definitely up there my technical friend. You have the original technical skills to be the advisor.
We need to get Sarah a TV budget and a Rex Garrod to drive the stunt cars etc
Sarah - you are the Museum’s Tim Hunkin. Thank you for bringing all these systems to life for us. It really is gratifying to see that the art of technical storytelling is alive and well.
ALL of Tims Shows are on TH-cam.. Most with bonus footage at the end....
I love this channel, this is incredible valuable content 😍
Curious electronics student here with lots of questions:
4:32 So the power is always supplied from the generators, and the batteries are only there to smooth out power fluctuations? And if so, how large do the batteries even need to be relative to the load, i.e. how long would they hold up without (/with inadequate) generator input?
7:40 If I had to guess, that's because it also contains enough copper to be very appealing to scrap. Very cool mechanical system, though, I'd definitely be interested in a more detailed discussion of it (Where are the remote actuators built in? How does it distinguish manual and overvoltage/back-EMF shutdown? What's the bottom contact pair with the jacobs-ladder-y arc extinguishers for? Why the three-stage contact mechanism and not just two stages?)
11:23 So are the end cells intended to increase battery voltage temporarily? But if you still regulate to 48V with additional cells in series, the average cell voltage will drop, which is apparently pretty bad (14:16, although I'm surprised that 1.9V instead of 2V already damages lead-acid cells, I thought those were pretty resilient to overdischarge)
3:31 This may be too fundamental a question, but I'm confused how exactly this switch-over works - particularly for the caller in a long-distance call, where the final connection is made in some far-away office: how does the originating office know the call is now connected, so it can switch to the talking battery? And how does the switch-over and resulting voltage drop not result in the connection being dropped?
Good questions! I'll try to provide helpful answers.
1. In the earliest days of common battery telephones (1900s-1920s) the equipment and network ran off of the batteries. When voltage became too low, Battery A (that had been supplying power) was cut over to the charging equipment, and Battery B (which had previously been charging) was cut over to run the equipment. Eventually, they moved to a "floating" style of charging, where the equipment is running on the batteries, but those same batteries are kept continuously charged.
The batteries always served two main purposes. To smooth out ripple in the charging circuit, and to provide reliability during power failures. Batteries could run the exchange for a few hours to a few days after a power failure. Exchanges also had diesel generators to charge the batteries, and those could run as long as fuel lasted. AT&T had pretty fantastic supply chain management and they could get fuel anywhere very fast.
2. The circuit breaker is surprisingly complex, and different breakers could be outfitted with different options depending on what the customer needed. The jacobs-laddery-contact-thing near the bottom opens the circuit to the main operating solenoid just after it starts to pull the breaker to the closed position. The solenoid uses A LOT of current, so we have to break it quickly and we don't want to do that with the button that actually starts its operation. Think of it like the "snapping" part of a household light switch.
3. The end cells are only cut in when the voltage of the main battery string is low enough to require them. We don't necessarily regulate down to 48V. That's just a "nominal" voltage. For example, the voltage level at the museum is 52.6 volts. (We don't actually get to decide on that number. That's just what Lumen, the telephone company, provides for us)
4. Talking battery is actually supplied in two places. For the originating side of the call, it is provided by the local office. For the terminating end of the call, it is provided by the far-end office. There is no need to apply talking battery for all of the links in the middle, since the talking battery is really only necessary for the telephones and local hook-state signaling. The human voice AC "wiggles" can travel along the wires just fine without that DC offset.
However, you are correct that the switches at either end need to know when to cut in the talking battery to their respective telephones at each end of the connection. This has to do with a concept that we in the telephone business call "supervision". As a part of its design, the switch "knows" when a call is completed, when a subscriber as answered the phone, and generally "what the state of the call is". So all we have to do is provide a relay somewhere in the call circuit that cuts off the "signal" battery and cuts-in the "talking" battery. The connection is not dropped because the switch over is very fast, and the relays responsible for holding the call up basically don't notice the change.
Exactly how the "supervision" of the call is transmitted between telephone switches can be quite complex, because it varies over time and between different systems. For now, we'll just have to say that "it works" and leave it at that. For a little more info on supervision, you can check out my video on the 2600hz signaling system here: th-cam.com/video/8PmkUPBhL4U/w-d-xo.html&pp=ygUGMjYwMGh6
Excellent! Funny that you have a Power Designs Inc. model 5015T power supply just like I have around the 7 minute mark! I got it from a ham radio friend of mine who has passed away. He worked for Bell Labs back in the 1970's. It is a great 50 volt 1.5 amp current and voltage limited power supply. Great job on all of these videos, I really enjoy them.
I inherited a 1920s Siemens 380V three phase AC to 60V 300 Amps DC motor-generator from my grandfather. He bought it cheap at a surplus store and converted it to a crazy powerful welding/gouging machine in the 1950s. I still use that monster occasionally for carbon gouging stuff like excavator buckets. What a machine!
r/endcells is a WAY better community than r/incels 😂
Omg Dev lmao 😂
I could bet my ass on that!
I seriously thought "nice that they put the subtitles there."
What about r/in-cell for the lowdown on prison life?
Sarah, awesome video. Thank you very much, and keep it up.
Aloha.............................
Yall are so freaking cool. This is fascinating
A place for the girls like us! I'd absolutely love to go there someday, and hang out with Sarah, Astrid, Jay and others.
What is a *Marker* as described at 16:00? Also, mentioned is a *sender* but without any explanation.
There are other videos on this channel that describe markers and senders. I will probably do more in the future.
great video! thanks! I remember when I canceled my POTS line to get VOIP, it was a sentimental day for me. My friends were like, what the heck???
in my younger days working for western electric co as an installer, i was working in a new P.R.T.D. bay cutting in new 750 ccm cables. before starting work i went down to the power room and checked the fuse board for that new P.R.T.D. and all the fuses were out and all the circuits were tagged do not make live. i went back up stars and started working. i did not know it but some one removed the tags on the fuse board and put in all the 800 amp fuses, i was using insulated power wrenches but left my ring on. i made contact between the cable end and my ring and frame ground . my ring vaporized with a flash and my hand and face was sprayed with molten metal. my finger was burned to the bone. . i was flash blind but my safety glasses saved my eyes. i was out for almost a year with skin grafts . the supervisor that removed the tags and put in the fuses was fired from his supervisor position .
Wow, you are lucky, we had a commercial electrician working with WECO installing new flourescent lights between the bays in our SXS floor, when the kid he was working with shoved a metal fish tape through the conduit into the open switch box he was working in, he wasn't so lucky. I remember seeing the embedded metal in his safety glasses, severe burns to his arms and chest area.
Very interesting. In the 1970s I worked on mainframe computer systems. We used motor-alternators to supply power to the system. These consisted of a 440v 3 phase motor driving a 440v 3 phase alternator. Their purpose was to give us a clean mains supply.
Absolutely love these videos! So fascinating and well presented! (Also loving the hair!)
Thanks so much. It's all really interesting. If I could clone a younger version of myself, she'd definitely join your team!
Love your videos! Thanks! I always learn something new.
I think you hinted at it - there is no master power switch. There are also faults in the power bus that occur before fuses. You just wait for these faults to burn through and use fans to blow away the hydrogen cloud.
I always thought that there was a pair of CRAY-1 super computers in the basement and the waste heat was used to boil water into steam. That steam was then used to power a pair of steam turbines that in turn spun the big generators. Or maybe I was thinking of somewhere else. Love all of your videos.
Good video ! Lots of great info, very well presented!
Another wonderful video.... thank you.
I love coming across your page! I. found one of these rooms in an old hospital that was abandoned in 2019. They had a newer telephony room that ran off of the newer grid, but It was wild seeing this and learning how it worked!
Sarah; will you ever put on a video about CO alarms? the gongs, the alarm bells, etc? I miss those gongs, the clatter of the #5 switch, and the sound of a call going through the SXS office at 4 AM, and at 81 yrs of age, only hear them in my dreams -
This was an awesome freaking video! More like this. Thanks Sarah
very cool to look at old systems!!! I do remimber the old dial princes phones of the 70's
I will buy your merch - the tees are awesome and the channel is even better! Visiting the museum is high up in my list, btw.
I Love the AvE warning sticker on the fuseboard.
My own story related to a battery plant.. Working at AT&T in the 90s.. had just started, toured the building.. was taken into the battery room.. first (and only) time I'd ever seen OPEN CELL BATTERIES filling the room.. oh the smell. lol.
another fantastic video, you guy's are top notch.
I was missing you videos from inside the museum. Can’t wait to visit. I’m coming soon
As a teen in the 60's I would connect a high gain mic input to the telephone line and listen to all the things there was to hear. Needless to say it was pretty interesting
This is fantastic. I'm due to visit Seattle in a couple of weeks, and I seriously hope to see the museum. I'm in IT now, but electromechanical, analog stuff still blows my mind, and I would love to see it in action. Thanks so much!
Sarah, that was fascinating! Thanks for the deep dive :)
You spoke of the 1963 assassination of Kennedy and the overload of the phone systems. I was 4. I remember my mom struggling to call people and didn't understand why. I understood intimately in 1989 Loma Prieta earthquake (impacting greater San Francisco Bay area). No cell phones in those days. I was in a San Francisco train station with a couple thousand other people waiting to use 12 phones. Sometimes it took 5 minutes for a phone to connect because the phone company couldn't handle the calls. It was hardest to connect to another circuit within the bay area, so people began connecting with people outside the area and have THOSE people call our loved ones at our homes to assure them we were safe.
Could you talk about what happens to the equipment when it loses power? Are all calls dropped when that huge fuse blows - and again when the switch is flipped after replacement? Do they disconnect when it does a "reset" on powerup? Does it just keep working in the last-known state? Are there redundant power paths to every piece of equipment to reduce the impact of blown upstream fuses like on modern servers?
Excellent presentation.
Great to see a new video, love your hair.
Another great Vid as always. Love your hair, been trying to work out my new colour and green/yellow is one of my favourites....All the best to yourself Sarah and the Team. Joni.
I work in a data center. Our power lineups are configured to swap back to utility one at a time to prevent putting a huge sudden load onto the grid.
Also, that beautiful breaker reminds me of a much more elegant version of our big Eaton and Schneider air breakers (4kA instantaneous).
Darn it Sarah, I don't know how you're able to make things so interesting, but you do it! Thanks for the great content everyone!
Sarah?