This is good. Our company designed and built the IMUs, which interacted with the Guidance Computers and the DSKYs to actually direct the whole Guidance System. I myself spent countless hours, often overtime at nights, working on various problems. I still have some pinout tools for the various mil-connectors. One common problem was that the wiring was always made as short as possible, to reduce weight. As such, there were "stress points" at many connectors, which caused wire breakage right at the joint with the pin. Mostly due to vibration or thermal shrinkage (often not considered by the mechanical engineers). We had special "repair kits" for these. Also, one time at Boeing, they had wrapped a wire bundle with some kind of very tight tie-down, and that tie-down actually smashed the wire insulation, and caused wire shorting. It took us two whole days to figure this out, since they were in a vacuum chamber, and nobody wanted to dump the vacuum. The tiedowns were never needed but they just followed normal procedure, and created an additional problem. This was actually common during those programs, from people wanting to do too much, too fast. Murphy's Law alway intervenes.
I wouldn't complain about people wanting to do too much, too fast. It got us to the moon, and it got us the longest-operating space probes in history. These days people want to do too little too safely. Result: We can't even get people to our own space station and back any more. If we want to get anywhere in space exploration, we need that pioneer spirit back, that just embraces the risks.
Incredible work, guys! The effort and expertise you put into reviving the Apollo comms system is beyond impressive. Watching you command the spacecraft's original tech is such a treat! Can't wait for the next episode!
You know... we used to joke you guys will bring back the program one part at a time... but man. The more you guys dig, the more i think you should receive a blank cheque and all the manpower you can muster...
Well, Carl is currently in Florida restoring period correct launch site hardware at the Cape Canaveral space force base. So presumably, if the team manages to restore a capsule and a rocket from spare parts, they'd have a place to launch from...
During this entire sequence of videos its very clear that Systems Engineering as a discipline really came of age with Apollo. Each one of those subsystems is fiendishly complex ( the cumulative aggregation of the various radio signals is a thing of beauty) yet they all meshed in harmony. The immense cost of Apollo really comes into focus when you consider how much engineering effort just went into the parts on the desk.
OMG, such intricate prose...!!! " ...fiendishly complex ( the cumulative aggregation of the various radio signals is a thing of beauty) yet they all meshed in harmony."
It may be just a light or a simulated console, but it's beyond fascinating seeing all this equipment up and communicating with one another. Having someone who has figured all this stuff out explain how they were able to do what they did in the era that they did it, gives you a much greater appreciation for what they accomplished. It's one of those things that just never gets old. From the most complicated systems to the most seemingly mundane, I could watch this stuff all day and never get tired of it.
Noooo, we need to see a collab between Dr. Maggie Lieu, Curious Marc, Chornobyl Family and Everyday Astronaut for a real re-flight of Apollo to the moon.
I’m still in awe at how all this was developed and executed. The forethought and implementation is astounding! Congratulations to your team for figuring it all out and getting it working.
As a person who grew up with the space program from childhood, I have found the entire series on rebuilding and saving Apollo programming and communications to be totally fascinating stuff. It's technology I did not understand as a kid, and was simply amazed at when a teen knowing very little about computer construction. But this channel has taught me more about electronics than I ever knew before, and even more it's allowed me to understand the deep layers of technical achievement it took to go from suborbital flight to the moon. You and the team have done some amazing things, taken me places I have only seen on live TV in black and white as a child when launches were a normal part of school assemblies, and brought back forward for all the importance of these technological achievements, with great explanation and teaching.
I thought you guys would never top the AGC restoration, but man was I wrong! This is absolutely astonishing! Both the work, foresight etc by the original engineers, but also you guys reverse engineering skills. I’m flabbergasted. What you are doing, is documenting the stone age of modern technology in a fascinating and modern way. Hats off to you - this effort is just amazing! 🎉
They topped the restoration by shrinking the AGC to fit on a solderless breadboard! Fascinating episode, and very welcome. You guys are certainly having fun and it's wonderful to see how this piece of Apollo worked. The different endian-ness of the command fields seems to show how the various design groups thought things should be and then any disconnects got fixed in the software. Wonder if that's actually true? The Apollo project was on such an aggressive timeline, it's even money that it was true.
I laughed out loud at the absurdity of all that equipment just to turn a light on and back off... but love every minute of the entire Apollo comms series.
How is it possible that every Apollo clip made here on this channel AMAZES me? You sirs are doing an EXCELLENT job in quality, content, everything! Thank you so much for this!!
Well done guys.. Watching the tape commands remotely control the AGC is amazing. Also it's incredible what the engineers managed to design and build back in the 1960's. Please keep going with this series as far as you possibly can, it's fascinating, but also wonderful to see a piece of living history preserved.
Amazing work from Marc, Eric, Mike, and all the others. I struggled for 20 minutes trying to connect a bluetooth device yesterday, and here you are doing punch tape to PC talking.
It's not surprising struggling with modern technology. There are no obvious switches and lights. For example I learned the other day that to put Samsung ear buds into pairing mode you take them out of the case, then put them back in, then close and open the case. That's a ritual, not a technical procedure!
Two things make me dream, the 300 metric-tons booster which is delicately embraced by choptsitcks and the curious marc dream team. You're truly amazing guys! Not forgetting all due respect for those who designed this marvellous control chain over half a century ago. All the best from Oleron Island - France.
I am really gasping for air. This is of another level. Analog computing and communication that worked. I never knew that they had invented all this amazing technology. Those days must have been awesome to work at NASA. You guys revived Frankenstein. 😁
Wow, that‘s like watching some guys figuring out how the pyramids were built for real! Not some thought experiment rock stacking, but the real technique! Amazing stuff you‘re doing there. Keep up the fascinating work!
It's fascinating to contemplate that the punched paper tape invented by Joseph Marie Jacquard in 1801 was used to send data to a computer at the surface of the moon a hundred sixty heights years later.
Just imagine. All these things , man made. I kudos everyone , and thank you Mark for showing us the "remote desktop" version of Apolo. Thank you again !
5:40 There's also another auto abort trigger which I don't think is mentioned in that schematic-exceeding the LV angle-of-attack (alpha) limits, measured using a sensor on the launch escape system. In any case, the launch vehicle rates referenced for an abort comes from the IU's inertial system, rather than the AGC-one of the things that helped Apollo 12's launch.
Ah yes, the Q ball! I believe you are entirely correct. I wonder where the wiring for that is hidden, I did not see it on this page. It might be going directly into one of the event sequencers, which controlled the complex escape tower firing sequence. And you are also correct on the source for the rates, it’s from the IU in the Saturn V, which I did not show at the left of the page. That certainly saved the day on Apollo 12, as the command module IMU lost power, but the Saturn V systems kept on going.
Wow, another awesome episode to behold. Together you guys are a real dream team. I find it so fascinating you are able to reverse engineer all of this tech and understand the thinking of that time on the how and why. Thank you so very much Marc and team, it was really entertaining.
Wonderful as usual. What would be great is if you could do a HOW TO episode on what software and workflow you use to do your lovely coloured schematic explanations.
It now makes me appreciate the sheer size of the Saturn 5 rocket, as I begin to truly understand just how heavy and bulky everything was.... It's just amazing they could DO this at all in the 60's.
"Hello and Welcome. As you know, we have landed at the Apollo 15 site and are currently restoring the LRV communications system. In the last episode Master Ken has reverse-engineered the remote camera control circuits and Mike has written a transistor-level emulator of the uplink encoder..."
Defense Simulations occasionally has Master Specialties stuff. One thing I was very surprised to see in their catalog was an original PerkinElmer KB-35 35mm mission strike camera, a motion picture camera which was used to record air-to-ground missile strikes.
Thanks for answering my hacker questions, this was satisfying. I love how they implemented that firewall using a big master switch and a relay reset switch. Fascinating the astronauts could overrule everything but a single light bulb for Abort. I love the "remote desktop" functionality of the AGC. genius. Only leaves me to wonder how much the ground could NOT control. Seems like it's basically everything else that isn't AGC and the few shadowed switches. Which in the end, seems like a whole lot. So there's no way Apollo could have been entirely flown without a crew, had it been a desired feature. (I'm thinking like Buran could be entirely flown remotely). So looking forward about the video on the logistics with the DSN and all the ground support for this. This sounds fascinating as well.
Next up, CuriousMarc swaps out the fancy pants for a pirate hat and commandeers the SS Cape Island to convert back to _its_ original Apollo configuration.
Fascinating. This channel is such a gem! Seeing these fantastic technologies and systems come to live in real time is absolutely brilliant. You people are a national treasure.
Agree with your comment. As a babyboomer who grew up in vacuum tube (valves on the other side of the pond) the level of integration today is mind boggling.
25:18 is there like... a Cryptography diagram that can encapsulate all this? xDD god this feels like code breaking at this point! amazing work. Looking forward to next vid and crazy tricks needed...
Fascinating video, as always! I find myself with the same question as from the old Soviet clock videos though... why would they want/need to reset the mission clock? That seems very counter-intuitive!
I think it’s a by-product of the adjustment routine. The first thing it needs to do before adjusting the time is to reset it to zeros before it starts counting up. And of course it gets reset at lift-off.
Frankly, I wish I could visit your laboratory and help out some day. I have experience with vintage electronics and soldering but I would love the opportunity to learn more. Like for example, how to build my own Apollo Guidance Computer System Mk. II. I have data on how to build the Mk. I unit from early missions but I prefer the Mk. II from 11 to 16. I have surplus IC chips that I would like to do something with. Also, Where did your friend get an FPGA equivalent fabricated? I want one.
A few years ago in this series Ben Krasnow from Applied science came out of nowhere with an fully working luminescent DSKY display. Then he appeared in Steve Jurvetsons private space collection and talked about how he would like to restore one of the fuel cells. I hope it doesn't take too long until he appears again with an working fuel cell.
We are still talking about the fuel cells, and Mike recently discovered much needed documentation on it. But it is so scary. Sending high pressure H2 and O2 in a 50 year old piece of high power electrical equipment. I can’t think of a single thing that could go wrong…
10 points to Gryffindor for the beautiful vignette of an HP (11C??) in binary mode in its classic spot on top of an engineer's notebook, analogue pencil AND eraser at the ready. :)
A few times every year I ask what is the name of that outro music,, and I've never gotten an answer,, so here I am trying again... Just to be clear I've gotten a couple wrong answers but I've never gotten the correct answer...
I'm sorry if I am repeating a question, but were these diagrams and schematics hand drawn or plotted by an X-Y plotter? And those switches are really beautiful. I cannot express my respect for how much I admire the engineers, manufacturers and shop floor people who made the Apollo program possible.
On the schematics you've shown, there's a lot of symbols that look like a large Z with numbers in various places around it. They seem to be references to another drawing, somehow, but I don't quite see how. What do they mean? Thanks for bringing us all into your lab to see all these things you're so lucky to have to play with. Keep being awesome!
You are correct. It references to a sheet number and letter/digit sector on that sheet, which has that circuit, hope it makes sense :) Eg. Sheet number 6.5, sector A/3
@@meerkatmcr No, it's opposite - sheet number is stated under "Z" and grid reference are on each side of "Z". If there is no number under Z, then that symbol refers to the same sheet. E.g. at 29:44 on the right there is “Central timing equipment” box with Z symbol inside. It refers to sheet no. 7.1 grid reference D/2. At the same time stamp you can see x2 Z’s in the middle of the screen - they both refer to the same sheet which is shown (hence no sheet number underneath “Z”), grid reference R/7 (At that R/7 sector you will find those “9” and “10” reference points). It is hard to describe it in the comments without screenshots so I hope it makes sense
I was just typing this question as you referred ti the issue at the end. Did they actually use paper tape to upload commands? Hard to think they’d do that in real time to, eg, turning on that abort light.
It’s complicated, I need to do another video on how that was done. But yes, you guessed right, Abort was from a button press that triggered a pre-recorded message right at the tracking site. A special priority one at that, that would interrupt any ongoing uplink.
What's the logic that drives the UPLINK ACTY light? (Is it something like "we are waiting for input and the first key in this interaction came over uplink"?)
@@Spookieham Yes, the Apollo system did compensate for the time delay caused by the signal traveling between Earth and the spacecraft. This was an important consideration due to the significant distance between Earth and the Moon, which could result in a round-trip signal delay of up to 2.5 seconds (1.25 seconds each way when the spacecraft was near the Moon, since light takes about 1.28 seconds to travel the roughly 384,000 km). ### Here's how it worked: 1. **Time Delay Estimation**: The Apollo system was aware of the current position and velocity of the spacecraft relative to Earth. NASA's tracking stations, such as those in the Deep Space Network (DSN), continuously tracked the spacecraft's position. This allowed precise calculation of the signal travel time at any moment. 2. **Compensating for Signal Travel Time**: When mission control needed to send commands or data (like clock updates) to the spacecraft, they calculated the one-way signal delay. The signal was then adjusted so that, by the time it reached the spacecraft, it would reflect the correct Earth time. 3. **Doppler Shift Considerations**: In addition to the basic delay due to the distance, engineers also had to account for the Doppler shift caused by the spacecraft's motion relative to Earth. As the spacecraft moved toward or away from Earth, the frequency of signals shifted slightly, which could also affect precise timing. These shifts were accounted for in the communications and timing systems. ### Real-time Control: Mission control used this knowledge of signal delays to ensure that all time-sensitive operations (such as guidance corrections, telemetry, and even syncing clocks) could be managed accurately. The onboard clock of the spacecraft, while critical for navigation and operation, could be updated from Earth in a way that accounted for these delays, ensuring that the systems on both Earth and the spacecraft remained synchronized. This level of precision was crucial for a mission like Apollo, where even small timing errors could have led to navigation problems or data misinterpretation. The time precision in the Apollo program, particularly for onboard systems and ground communications, was remarkably high given the technological constraints of the 1960s. Here's a breakdown of how precise the timing was in different aspects of the mission: ### 1. **Onboard Timing Precision (Guidance Systems)** The Apollo spacecraft had two primary guidance systems with timekeeping mechanisms: - **Command Module (CM) Guidance System**: This used the **Primary Guidance, Navigation, and Control System (PGNCS)**, which relied on an onboard clock with a precision of **milliseconds**. The onboard computer, the Apollo Guidance Computer (AGC), needed to run precise navigation and control calculations continuously, so clock synchronization and accuracy were vital. - **Lunar Module (LM) Guidance System**: Similarly, the LM’s PGNCS also relied on highly accurate timing for landing on the Moon, making decisions in real-time about descent and course corrections. In both cases, the precision of the onboard clocks was critical to ensure that navigation and control solutions were calculated accurately, often within **tens of milliseconds**. ### 2. **Earth-Based Tracking Systems (Deep Space Network)** The **Deep Space Network (DSN)**, which tracked the spacecraft, had a timekeeping precision on the order of **microseconds (µs)**. These ground-based stations measured the spacecraft's position and velocity extremely accurately by tracking the Doppler shift of the signals transmitted from the spacecraft. The higher precision on the ground was necessary to track the spacecraft's location within a few hundred meters across the hundreds of thousands of kilometers between Earth and the Moon. ### 3. **Communication and Telemetry Timing** For the Apollo spacecraft's communication system, the timing precision needed to account for: - **Time Delay Compensation**: Signal travel time between Earth and the spacecraft, which could vary between **1 to 1.25 seconds one way** depending on distance. - **Clock Synchronization**: The spacecraft’s clocks could be updated from Earth, and the precision of these updates was essential for mission-critical operations. The updates were made by accounting for signal travel time, which was computed to within **milliseconds** to ensure synchronization. ### 4. **Timing in Critical Events (Lunar Landing)** During critical operations such as the lunar landing, timing needed to be incredibly precise. The entire landing sequence relied on accurate timing for thruster burns, altitude measurements, and velocity checks, often within **milliseconds**. For example, during the descent to the lunar surface, the Lunar Module’s computer (which also used the AGC) processed real-time sensor data and issued control signals based on precisely timed inputs. The entire sequence of events had to be executed with **millisecond precision** to ensure a smooth landing. ### 5. **Synchronization Between Ground and Spacecraft** Although the onboard clocks were not perfectly synchronized with Earth clocks over long periods (due to drift), regular updates from Earth ensured that the clocks remained within **milliseconds** of Earth time for most of the mission. These updates were especially important for precise navigation and communication. ### 6. **Doppler Shift and Frequency Timing** The Doppler effect, caused by the spacecraft's movement relative to Earth, affected the timing and frequency of communications signals. Tracking stations compensated for this with extremely precise measurements of frequency shifts, down to **fractions of a hertz**, which required timing precision on the order of **microseconds**. ### Summary of Time Precision Levels: - **Onboard Computers (AGC)**: Milliseconds precision for navigation and guidance. - **Earth-based Tracking (DSN)**: Microseconds precision for spacecraft tracking. - **Time Delay Compensation**: Calculated within milliseconds to synchronize clocks and signals. Overall, while the spacecraft's onboard systems operated with precision in the millisecond range, Earth-based systems and signal tracking were precise to the microsecond level to ensure the highest possible accuracy in navigation and mission operations.
Yes, they sure had to compensate for propagation delay. For that reason, the CTE update is the one command that was not computed nor sent by Houston. It was computed at the tracking station and sent from there. Adding the delay was quite simple, since they had a direct measurement of it right at the station to a fraction of a millisecond, obtained from the pseudorandom ranging signal turnaround time. Then they added the deterministic command processing time, which is indicated on the drawing if you look closely.
But don't they have to confirm everything before sending it, ie, wouldn't a wrong command crash the spacecraft. Did they run it through a duplicate system on the ground like Marc's setup.
You are absolutely right, and they did confirm many times, particularly for the AGC uploads. Before it was sent, while sending, after sending. And uploads were held in an upload buffer in the AGC before they’d be applied. The buffer would be re-checked by telemetry, then they’d finally send a short validation command to apply the upload. I’ll go over it in the next video.
Both. We’ll cover that in another episode, but when doing manual keypresses, flight controllers were supposed to wait for reading it back on their control screen DSKY telemetry return before pressing the next key. With the propagation delay both ways, that meant up to 3 seconds between keystrokes. When doing a ground computer assisted load, keystrokes were sent about 300 ms apart (take this with a grain of salt, I need to double check that figure), without waiting for confirmation, in order to go faster. However, the keystrokes were accumulated in a keystroke buffer in the AGC, and held there. At the end of the computer load uplink, which took 20s, the controller (and the ground computer) would both check the entire buffer content via telemetry. If it checked good, the FC (and not the computer) would enter “Verb 33 Enter” manually from the console, causing the AGC to accept the buffered load and process the keystrokes. So they were very cautious and deliberate with their AGC uploads.
„So you have a highly complicated update system, what you use for most channels, must be really tricky functions“ „oh yes! Two different methods to turn on a warning lights!“ reality of spaceflight in Apollo age is equally mindblowing and ridiculous 😮 It beeps! No no we can make a muuuuch more impressive demo, look at that light!
Fascinating functionality. I wonder why they didn't use this for the Apollo 14 hot fix for the stuck abort switch rather than having the astronauts punch in the commands and possibly hit a wrong key.
One I had never considered, what if an adversary did in fact develop the ability to communicate with the vehicle and send it malicious messages. Was there any level of authentication between ground control systems and the craft’s remote control interface? Doesn’t seem like it. I smell a Hollywood movie idea.
I wonder if those two lights could have been used by Houston to send visual morse coded messages, in a weird situation where the only communication link still working was the UDL. Or is this a completely nuts idea, because if the UDL is working, then by definition, something better to communicate is necessarily working as well. Can we imagine a failure mode that would require Houston to use a light for morse code?
This is good. Our company designed and built the IMUs, which interacted with the Guidance Computers and the DSKYs to actually direct the whole Guidance System. I myself spent countless hours, often overtime at nights, working on various problems. I still have some pinout tools for the various mil-connectors. One common problem was that the wiring was always made as short as possible, to reduce weight. As such, there were "stress points" at many connectors, which caused wire breakage right at the joint with the pin. Mostly due to vibration or thermal shrinkage (often not considered by the mechanical engineers). We had special "repair kits" for these. Also, one time at Boeing, they had wrapped a wire bundle with some kind of very tight tie-down, and that tie-down actually smashed the wire insulation, and caused wire shorting. It took us two whole days to figure this out, since they were in a vacuum chamber, and nobody wanted to dump the vacuum. The tiedowns were never needed but they just followed normal procedure, and created an additional problem. This was actually common during those programs, from people wanting to do too much, too fast. Murphy's Law alway intervenes.
I wouldn't complain about people wanting to do too much, too fast. It got us to the moon, and it got us the longest-operating space probes in history. These days people want to do too little too safely. Result: We can't even get people to our own space station and back any more. If we want to get anywhere in space exploration, we need that pioneer spirit back, that just embraces the risks.
There was a lot of people and organizations involved, the idea of common practice becoming a liability is as old as sin. Thanks for your work!
You guys certainly did an awesome job!
Incredible work, guys! The effort and expertise you put into reviving the Apollo comms system is beyond impressive. Watching you command the spacecraft's original tech is such a treat! Can't wait for the next episode!
You know... we used to joke you guys will bring back the program one part at a time... but man. The more you guys dig, the more i think you should receive a blank cheque and all the manpower you can muster...
It is amazing and incredibly important work. Preserving a key part of Apollo history that is far too often overlooked.
Marc does these things not because they are easy... 😁
Well, Carl is currently in Florida restoring period correct launch site hardware at the Cape Canaveral space force base. So presumably, if the team manages to restore a capsule and a rocket from spare parts, they'd have a place to launch from...
@@ReneSchickbauerthey will soon realize the impossibility of NASA’s Obviously absurd Claims…
During this entire sequence of videos its very clear that Systems Engineering as a discipline really came of age with Apollo. Each one of those subsystems is fiendishly complex ( the cumulative aggregation of the various radio signals is a thing of beauty) yet they all meshed in harmony. The immense cost of Apollo really comes into focus when you consider how much engineering effort just went into the parts on the desk.
OMG, such intricate prose...!!! " ...fiendishly complex ( the cumulative aggregation of the various radio signals is a thing of beauty) yet they all meshed in harmony."
It may be just a light or a simulated console, but it's beyond fascinating seeing all this equipment up and communicating with one another. Having someone who has figured all this stuff out explain how they were able to do what they did in the era that they did it, gives you a much greater appreciation for what they accomplished. It's one of those things that just never gets old. From the most complicated systems to the most seemingly mundane, I could watch this stuff all day and never get tired of it.
Noooo, we need to see a collab between Dr. Maggie Lieu, Curious Marc, Chornobyl Family and Everyday Astronaut for a real re-flight of Apollo to the moon.
I’m still in awe at how all this was developed and executed. The forethought and implementation is astounding! Congratulations to your team for figuring it all out and getting it working.
As a person who grew up with the space program from childhood, I have found the entire series on rebuilding and saving Apollo programming and communications to be totally fascinating stuff. It's technology I did not understand as a kid, and was simply amazed at when a teen knowing very little about computer construction. But this channel has taught me more about electronics than I ever knew before, and even more it's allowed me to understand the deep layers of technical achievement it took to go from suborbital flight to the moon. You and the team have done some amazing things, taken me places I have only seen on live TV in black and white as a child when launches were a normal part of school assemblies, and brought back forward for all the importance of these technological achievements, with great explanation and teaching.
28:29 _“Sort of Remote Desktop for Apollo…”_ Now, *that* quote made my day… 😂
I thought you guys would never top the AGC restoration, but man was I wrong! This is absolutely astonishing! Both the work, foresight etc by the original engineers, but also you guys reverse engineering skills. I’m flabbergasted. What you are doing, is documenting the stone age of modern technology in a fascinating and modern way. Hats off to you - this effort is just amazing! 🎉
They topped the restoration by shrinking the AGC to fit on a solderless breadboard! Fascinating episode, and very welcome. You guys are certainly having fun and it's wonderful to see how this piece of Apollo worked. The different endian-ness of the command fields seems to show how the various design groups thought things should be and then any disconnects got fixed in the software. Wonder if that's actually true? The Apollo project was on such an aggressive timeline, it's even money that it was true.
Apollo Project Part 234: Testing, Ignition, and Force Measurement of our Salvaged Rocketdyne F-1
Thanks for another 37 minutes of genius
I laughed out loud at the absurdity of all that equipment just to turn a light on and back off... but love every minute of the entire Apollo comms series.
How is it possible that every Apollo clip made here on this channel AMAZES me? You sirs are doing an EXCELLENT job in quality, content, everything! Thank you so much for this!!
Well done guys.. Watching the tape commands remotely control the AGC is amazing. Also it's incredible what the engineers managed to design and build back in the 1960's. Please keep going with this series as far as you possibly can, it's fascinating, but also wonderful to see a piece of living history preserved.
With some luck, CuriousMarc might actually put man back on the moon before NASA does.
At the rate he's going, @primitivetechnology9550 may beat NASA to the moon.
MagicMarc. 🙂
Amazing work from Marc, Eric, Mike, and all the others. I struggled for 20 minutes trying to connect a bluetooth device yesterday, and here you are doing punch tape to PC talking.
Yes, but we struggled for two years to get to that point!
It's not surprising struggling with modern technology. There are no obvious switches and lights. For example I learned the other day that to put Samsung ear buds into pairing mode you take them out of the case, then put them back in, then close and open the case. That's a ritual, not a technical procedure!
@@gshinglesonly certain prayers trigger Bluetooth pairing, exact distance necessary, must be kneeled down for it to work.
@@snowzZzZz 😂👍
Two things make me dream, the 300 metric-tons booster which is delicately embraced by choptsitcks and the curious marc dream team. You're truly amazing guys! Not forgetting all due respect for those who designed this marvellous control chain over half a century ago. All the best from Oleron Island - France.
I am really gasping for air. This is of another level. Analog computing and communication that worked. I never knew that they had invented all this amazing technology. Those days must have been awesome to work at NASA. You guys revived Frankenstein. 😁
Wow, that‘s like watching some guys figuring out how the pyramids were built for real! Not some thought experiment rock stacking, but the real technique! Amazing stuff you‘re doing there. Keep up the fascinating work!
Your videos are always so immensely enjoyable! Thanks so much for making them.
It's fascinating to contemplate that the punched paper tape invented by Joseph Marie Jacquard in 1801 was used to send data to a computer at the surface of the moon a hundred sixty heights years later.
And two more decades for data storage of lost of things
Just imagine. All these things , man made. I kudos everyone , and thank you Mark for showing us the "remote desktop" version of Apolo. Thank you again !
I know from experience you must have plenty of spare CM387 light bulbs on hand for those number display readouts, and relays, too.
And… your experience is correct, we do!
37 minutes of bliss. Thanks, guys, you made my day.
5:40 There's also another auto abort trigger which I don't think is mentioned in that schematic-exceeding the LV angle-of-attack (alpha) limits, measured using a sensor on the launch escape system. In any case, the launch vehicle rates referenced for an abort comes from the IU's inertial system, rather than the AGC-one of the things that helped Apollo 12's launch.
Ah yes, the Q ball! I believe you are entirely correct. I wonder where the wiring for that is hidden, I did not see it on this page. It might be going directly into one of the event sequencers, which controlled the complex escape tower firing sequence. And you are also correct on the source for the rates, it’s from the IU in the Saturn V, which I did not show at the left of the page. That certainly saved the day on Apollo 12, as the command module IMU lost power, but the Saturn V systems kept on going.
I love this series. And I'm a bit envious. Way, way cool beyond words. Kudos to all of you!
You and the crew are amazing. Great work!
Awsome reversing and demonstration. The tape reader makes the commands feel so distinct, also like all the redundancy in the design.
Well worth the wait. What a great job you are doing.
Wow, another awesome episode to behold. Together you guys are a real dream team. I find it so fascinating you are able to reverse engineer all of this tech and understand the thinking of that time on the how and why. Thank you so very much Marc and team, it was really entertaining.
Incredible engineering for back in the day. Wow.
Wonderful as usual.
What would be great is if you could do a HOW TO episode on what software and workflow you use to do your lovely coloured schematic explanations.
phew for a brief little moment I tought that this magnificent series had ended.
Bonus points for having an HP-16C I still have mine, along with an original 15C
Hehe. I have my 15C from the Bell Labs days and the 16C thar you saw at the ready in the lab. And the re-release of the 12C in my office.
It now makes me appreciate the sheer size of the Saturn 5 rocket, as I begin to truly understand just how heavy and bulky everything was.... It's just amazing they could DO this at all in the 60's.
In episode 100 I wouldn't be surprised if they just go ahead and rebuild an apollo capsule
"Hello and Welcome. As you know, we have landed at the Apollo 15 site and are currently restoring the LRV communications system. In the last episode Master Ken has reverse-engineered the remote camera control circuits and Mike has written a transistor-level emulator of the uplink encoder..."
@@ReneSchickbauer hahaha 😂
At this rate they'll launch a video on the moon in episode 300 - As a flag they will put "This episode is brought to you by PCB Way!!!"
Buttons and Blinkenlights! Great stuff.
Defense Simulations occasionally has Master Specialties stuff. One thing I was very surprised to see in their catalog was an original PerkinElmer KB-35 35mm mission strike camera, a motion picture camera which was used to record air-to-ground missile strikes.
Really enjoying this series, just amazing. Given such a clear insight into the technology they used and how it was designed.
This is an amazing achievement! Bleat! Nice job!
Got a flu shot today and don't feel like doing anything active this evening. What a perfect time to settle in with a nice juicy Apollo comms episode!
Thanks for answering my hacker questions, this was satisfying. I love how they implemented that firewall using a big master switch and a relay reset switch. Fascinating the astronauts could overrule everything but a single light bulb for Abort.
I love the "remote desktop" functionality of the AGC. genius.
Only leaves me to wonder how much the ground could NOT control. Seems like it's basically everything else that isn't AGC and the few shadowed switches. Which in the end, seems like a whole lot. So there's no way Apollo could have been entirely flown without a crew, had it been a desired feature. (I'm thinking like Buran could be entirely flown remotely).
So looking forward about the video on the logistics with the DSN and all the ground support for this. This sounds fascinating as well.
Mind-bending as to how they managed to design it all!
love this channel. love your videos and the work you do!
Fascinating! Thanks Marc and Team. Looking forward to the next vid.
Really looking forward to the next episode on how ground control did it end to end, even all the way to Parkes in Australia.
Probably the most complicated blink test that has ever been devised :D
Next up, CuriousMarc swaps out the fancy pants for a pirate hat and commandeers the SS Cape Island to convert back to _its_ original Apollo configuration.
Just brilliant, many thanks to the team.
Superb effort in every way: loved it!
Please keep at it!❤
Fascinating. This channel is such a gem! Seeing these fantastic technologies and systems come to live in real time is absolutely brilliant. You people are a national treasure.
Always fascinating!
Drillenium Falcon ❤ darn you are a nerd hero! - already knew you where the master, but i just love it.
Brilliantly done. 👏
next from curiousmarc, they reconstruct an ATT long lines link to demonstrate how to get coms to and from teresterally....
Boeing should have hired you guys to rebuild an Apollo capsule instead of Starliner. Would have saved a lot of money on debugging.
What a beautiful sound 1:05
It is crazy to think that you can fit all the gates in the AGC to a tiny FPGA. It is like the same computer but very smollll
Agree with your comment. As a babyboomer who grew up in vacuum tube (valves on the other side of the pond) the level of integration today is mind boggling.
25:18 is there like... a Cryptography diagram that can encapsulate all this? xDD god this feels like code breaking at this point! amazing work. Looking forward to next vid and crazy tricks needed...
Fascinating video, as always! I find myself with the same question as from the old Soviet clock videos though... why would they want/need to reset the mission clock? That seems very counter-intuitive!
I think it’s a by-product of the adjustment routine. The first thing it needs to do before adjusting the time is to reset it to zeros before it starts counting up. And of course it gets reset at lift-off.
Frankly, I wish I could visit your laboratory and help out some day. I have experience with vintage electronics and soldering but I would love the opportunity to learn more. Like for example, how to build my own Apollo Guidance Computer System Mk. II. I have data on how to build the Mk. I unit from early missions but I prefer the Mk. II from 11 to 16. I have surplus IC chips that I would like to do something with.
Also, Where did your friend get an FPGA equivalent fabricated? I want one.
It seems that you are capable of doing something that will be happening 55 years forward!
I can here the original meetings in todays conversations 🤔👍🏻🤩🤓
What is the FPGA name/part number @30:42 ?
Thank you for the video
Magnifique!! 👍👍
A few years ago in this series Ben Krasnow from Applied science came out of nowhere with an fully working luminescent DSKY display. Then he appeared in Steve Jurvetsons private space collection and talked about how he would like to restore one of the fuel cells. I hope it doesn't take too long until he appears again with an working fuel cell.
We are still talking about the fuel cells, and Mike recently discovered much needed documentation on it. But it is so scary. Sending high pressure H2 and O2 in a 50 year old piece of high power electrical equipment. I can’t think of a single thing that could go wrong…
@@CuriousMarc "that's the crater wher California used to be!"
"TH-camr sends himself to the moon."
At the rate you're going, you will have an Apollo capsule back to the moon before Boeing stops their farce from leaking. 😂
In a few years, we're going to see Marc knock out a wall to his basement so that he can remove the Apollo Command Module that he built.
10 points to Gryffindor for the beautiful vignette of an HP (11C??) in binary mode in its classic spot on top of an engineer's notebook, analogue pencil AND eraser at the ready. :)
HP 16C it is! I appreciate the points.
This is so cool. Well done! :)
Thanks so much for sharing. 😉👌🏻
love it! keep up the Work
A few times every year I ask what is the name of that outro music,, and I've never gotten an answer,, so here I am trying again...
Just to be clear I've gotten a couple wrong answers but I've never gotten the correct answer...
I'm sorry if I am repeating a question, but were these diagrams and schematics hand drawn or plotted by an X-Y plotter? And those switches are really beautiful. I cannot express my respect for how much I admire the engineers, manufacturers and shop floor people who made the Apollo program possible.
They look hand drawn. They are drop-dead gorgeous drawings. I’ll post the links when I have a minute.
On the schematics you've shown, there's a lot of symbols that look like a large Z with numbers in various places around it. They seem to be references to another drawing, somehow, but I don't quite see how. What do they mean?
Thanks for bringing us all into your lab to see all these things you're so lucky to have to play with. Keep being awesome!
You are correct. It references to a sheet number and letter/digit sector on that sheet, which has that circuit, hope it makes sense :) Eg. Sheet number 6.5, sector A/3
@@deathblowhere so the sheet numbers are split each side of the Z, and the sector (is that like a grid?) is underneath?
@@meerkatmcr No, it's opposite - sheet number is stated under "Z" and grid reference are on each side of "Z". If there is no number under Z, then that symbol refers to the same sheet.
E.g. at 29:44 on the right there is “Central timing equipment” box with Z symbol inside. It refers to sheet no. 7.1 grid reference D/2. At the same time stamp you can see x2 Z’s in the middle of the screen - they both refer to the same sheet which is shown (hence no sheet number underneath “Z”), grid reference R/7 (At that R/7 sector you will find those “9” and “10” reference points). It is hard to describe it in the comments without screenshots so I hope it makes sense
@@deathblowhere now it makes perfect sense (when I look at that timestamp in the video and your explanation)! Thank you!
@@meerkatmcr You're welcome :) Glad it helped 👍
I was just typing this question as you referred ti the issue at the end. Did they actually use paper tape to upload commands? Hard to think they’d do that in real time to, eg, turning on that abort light.
No way, too much room for human errors. But I'm also curious.
It’s complicated, I need to do another video on how that was done. But yes, you guessed right, Abort was from a button press that triggered a pre-recorded message right at the tracking site. A special priority one at that, that would interrupt any ongoing uplink.
ooh spoilers
What's the logic that drives the UPLINK ACTY light? (Is it something like "we are waiting for input and the first key in this interaction came over uplink"?)
Thanks!
What distro is Mike running on his laptop that does the AGC emulation?
Any chance you'll tie the comm package from the lunar rover into the system?
you're really going to have a complete apollo control system before long, next up, "restoring the apollo capsule"
wow and peace be upon you sir from me
If that feels cool today, imagine how it must have felt back then!
What about time delays for signals between earth and Apollo while setting the clock?
I imagine they would know the delay from command generation to timer update occurring and adjust accordingly
@@Spookieham
Yes, the Apollo system did compensate for the time delay caused by the signal traveling between Earth and the spacecraft. This was an important consideration due to the significant distance between Earth and the Moon, which could result in a round-trip signal delay of up to 2.5 seconds (1.25 seconds each way when the spacecraft was near the Moon, since light takes about 1.28 seconds to travel the roughly 384,000 km).
### Here's how it worked:
1. **Time Delay Estimation**: The Apollo system was aware of the current position and velocity of the spacecraft relative to Earth. NASA's tracking stations, such as those in the Deep Space Network (DSN), continuously tracked the spacecraft's position. This allowed precise calculation of the signal travel time at any moment.
2. **Compensating for Signal Travel Time**: When mission control needed to send commands or data (like clock updates) to the spacecraft, they calculated the one-way signal delay. The signal was then adjusted so that, by the time it reached the spacecraft, it would reflect the correct Earth time.
3. **Doppler Shift Considerations**: In addition to the basic delay due to the distance, engineers also had to account for the Doppler shift caused by the spacecraft's motion relative to Earth. As the spacecraft moved toward or away from Earth, the frequency of signals shifted slightly, which could also affect precise timing. These shifts were accounted for in the communications and timing systems.
### Real-time Control:
Mission control used this knowledge of signal delays to ensure that all time-sensitive operations (such as guidance corrections, telemetry, and even syncing clocks) could be managed accurately. The onboard clock of the spacecraft, while critical for navigation and operation, could be updated from Earth in a way that accounted for these delays, ensuring that the systems on both Earth and the spacecraft remained synchronized.
This level of precision was crucial for a mission like Apollo, where even small timing errors could have led to navigation problems or data misinterpretation.
The time precision in the Apollo program, particularly for onboard systems and ground communications, was remarkably high given the technological constraints of the 1960s. Here's a breakdown of how precise the timing was in different aspects of the mission:
### 1. **Onboard Timing Precision (Guidance Systems)**
The Apollo spacecraft had two primary guidance systems with timekeeping mechanisms:
- **Command Module (CM) Guidance System**: This used the **Primary Guidance, Navigation, and Control System (PGNCS)**, which relied on an onboard clock with a precision of **milliseconds**. The onboard computer, the Apollo Guidance Computer (AGC), needed to run precise navigation and control calculations continuously, so clock synchronization and accuracy were vital.
- **Lunar Module (LM) Guidance System**: Similarly, the LM’s PGNCS also relied on highly accurate timing for landing on the Moon, making decisions in real-time about descent and course corrections.
In both cases, the precision of the onboard clocks was critical to ensure that navigation and control solutions were calculated accurately, often within **tens of milliseconds**.
### 2. **Earth-Based Tracking Systems (Deep Space Network)**
The **Deep Space Network (DSN)**, which tracked the spacecraft, had a timekeeping precision on the order of **microseconds (µs)**. These ground-based stations measured the spacecraft's position and velocity extremely accurately by tracking the Doppler shift of the signals transmitted from the spacecraft. The higher precision on the ground was necessary to track the spacecraft's location within a few hundred meters across the hundreds of thousands of kilometers between Earth and the Moon.
### 3. **Communication and Telemetry Timing**
For the Apollo spacecraft's communication system, the timing precision needed to account for:
- **Time Delay Compensation**: Signal travel time between Earth and the spacecraft, which could vary between **1 to 1.25 seconds one way** depending on distance.
- **Clock Synchronization**: The spacecraft’s clocks could be updated from Earth, and the precision of these updates was essential for mission-critical operations. The updates were made by accounting for signal travel time, which was computed to within **milliseconds** to ensure synchronization.
### 4. **Timing in Critical Events (Lunar Landing)**
During critical operations such as the lunar landing, timing needed to be incredibly precise. The entire landing sequence relied on accurate timing for thruster burns, altitude measurements, and velocity checks, often within **milliseconds**.
For example, during the descent to the lunar surface, the Lunar Module’s computer (which also used the AGC) processed real-time sensor data and issued control signals based on precisely timed inputs. The entire sequence of events had to be executed with **millisecond precision** to ensure a smooth landing.
### 5. **Synchronization Between Ground and Spacecraft**
Although the onboard clocks were not perfectly synchronized with Earth clocks over long periods (due to drift), regular updates from Earth ensured that the clocks remained within **milliseconds** of Earth time for most of the mission. These updates were especially important for precise navigation and communication.
### 6. **Doppler Shift and Frequency Timing**
The Doppler effect, caused by the spacecraft's movement relative to Earth, affected the timing and frequency of communications signals. Tracking stations compensated for this with extremely precise measurements of frequency shifts, down to **fractions of a hertz**, which required timing precision on the order of **microseconds**.
### Summary of Time Precision Levels:
- **Onboard Computers (AGC)**: Milliseconds precision for navigation and guidance.
- **Earth-based Tracking (DSN)**: Microseconds precision for spacecraft tracking.
- **Time Delay Compensation**: Calculated within milliseconds to synchronize clocks and signals.
Overall, while the spacecraft's onboard systems operated with precision in the millisecond range, Earth-based systems and signal tracking were precise to the microsecond level to ensure the highest possible accuracy in navigation and mission operations.
Yes, they sure had to compensate for propagation delay. For that reason, the CTE update is the one command that was not computed nor sent by Houston. It was computed at the tracking station and sent from there. Adding the delay was quite simple, since they had a direct measurement of it right at the station to a fraction of a millisecond, obtained from the pseudorandom ranging signal turnaround time. Then they added the deterministic command processing time, which is indicated on the drawing if you look closely.
But don't they have to confirm everything before sending it, ie, wouldn't a wrong command crash the spacecraft. Did they run it through a duplicate system on the ground like Marc's setup.
You are absolutely right, and they did confirm many times, particularly for the AGC uploads. Before it was sent, while sending, after sending. And uploads were held in an upload buffer in the AGC before they’d be applied. The buffer would be re-checked by telemetry, then they’d finally send a short validation command to apply the upload. I’ll go over it in the next video.
Out of curiosity, is the bitrate on the dsky presses from Houston limited to a human level or can it read as fast as it can be sent?
Both. We’ll cover that in another episode, but when doing manual keypresses, flight controllers were supposed to wait for reading it back on their control screen DSKY telemetry return before pressing the next key. With the propagation delay both ways, that meant up to 3 seconds between keystrokes. When doing a ground computer assisted load, keystrokes were sent about 300 ms apart (take this with a grain of salt, I need to double check that figure), without waiting for confirmation, in order to go faster. However, the keystrokes were accumulated in a keystroke buffer in the AGC, and held there. At the end of the computer load uplink, which took 20s, the controller (and the ground computer) would both check the entire buffer content via telemetry. If it checked good, the FC (and not the computer) would enter “Verb 33 Enter” manually from the console, causing the AGC to accept the buffered load and process the keystrokes. So they were very cautious and deliberate with their AGC uploads.
Yey new marc :3
A Steren breadboard? Wow!
incredible!
Let's hope that Apollo 11 has actually landed already, and that you don't put the guys up there in trouble with your experiments
„So you have a highly complicated update system, what you use for most channels, must be really tricky functions“ „oh yes! Two different methods to turn on a warning lights!“ reality of spaceflight in Apollo age is equally mindblowing and ridiculous 😮
It beeps! No no we can make a muuuuch more impressive demo, look at that light!
Out of curiousity. Is the agc on the pallet still kicking arround and is there a chance you can get it up and running?
Ah! When every engineer had a HP calculator. You need a hp35 to match the rest of the tech
Wonderful!
Fascinating functionality. I wonder why they didn't use this for the Apollo 14 hot fix for the stuck abort switch rather than having the astronauts punch in the commands and possibly hit a wrong key.
I doubt the LM had the same capability.
I love you guys.
One I had never considered, what if an adversary did in fact develop the ability to communicate with the vehicle and send it malicious messages. Was there any level of authentication between ground control systems and the craft’s remote control interface? Doesn’t seem like it.
I smell a Hollywood movie idea.
Nope, no encryption. We were listening to the Soviets, the Soviets listened to us, but everybody behaved.
5:41 what about the signals for the LV not being intact?
I wonder if those two lights could have been used by Houston to send visual morse coded messages, in a weird situation where the only communication link still working was the UDL.
Or is this a completely nuts idea, because if the UDL is working, then by definition, something better to communicate is necessarily working as well. Can we imagine a failure mode that would require Houston to use a light for morse code?
I would not be surprised if it was in a procedure document somewhere
Sorry Buzz and Neil, we sent an abort command but Marc got the wiring wrong any way you might as well go land on the moon whilst we check the wiring.