OT: I really enjoyed The Increment when it was first being released. It felt like the first software engineering practitioner's publication and introduced me to a lot of new people to follow.
"From the dawn of the Space Age through the present, NASA has relied on resilient software running on redundant hardware to make up for physical defects, wear and tear, sudden failures, or even the effects of cosmic rays on equipment."
An interesting case study in this domain is to compare the Saturn V Launch Vehicle Digital Computer with the Apollo Guidance Computer
Now the LVDC, that was a real flight computer, triply redundant, every stage in the processing pipeline had to be vote confirmed, the works.
Compare the AGC, with no redundancy. a toy by comparison. But the AGC was much faster and lighter so they just shipped two of them(three if you count the one in the lunar module) and made sure it was really good at restarting fast.
There is a lesson to be learned here but I am not sure what it is. Worse is better? Can not fail vs fail gracefully?
The contrast with modern software development is striking. Today we often rely on fast iteration and patching problems in production. Spacecraft software is the opposite
On the other hand a lot of SpaceX's success can be attributed to applying modern software development methodology on spacecraft. They are very much doing agile development, betting on velocity enabling fast iteration.
That has lead to some of the best rockets ever developed, and the largest satellite constellation by far. But part of the secret sauce is creating situations where you can take risks. Traditionally anything space-related deals in one-offs or tiny production volumes, so any risk is expensive. A lot of SpaceX's strategy is about changing this, whether that's by testing in flight phases the customer doesn't care about, being their own best customer to have lower-risk flights, or building constellations so big that certain failure scenarios aren't a big issue (while other scenarios still have to be treated as high-risk high-impact)
> running identical software on multiple computer systems is the name of the software-architecture game
In the railway signalling industry (which for historically obvious reasons is obsessed with reliability) there even is a pattern of running different software implementing the same specification, written by different team, running on a different RTOS and different CPU architecture.
interesting point about patching in production – it's a totally different mindset. we had a similar issue with a legacy system at my old job, felt like a constant firefighting situation.
15 comments
[ 4.2 ms ] story [ 40.4 ms ] threadWe know Glenn is loquacious.
An interesting case study in this domain is to compare the Saturn V Launch Vehicle Digital Computer with the Apollo Guidance Computer
Now the LVDC, that was a real flight computer, triply redundant, every stage in the processing pipeline had to be vote confirmed, the works.
https://en.wikipedia.org/wiki/Launch_Vehicle_Digital_Compute...
Compare the AGC, with no redundancy. a toy by comparison. But the AGC was much faster and lighter so they just shipped two of them(three if you count the one in the lunar module) and made sure it was really good at restarting fast.
There is a lesson to be learned here but I am not sure what it is. Worse is better? Can not fail vs fail gracefully?
A transient bit failure in digital circuits? Then reboot and away you go.
A coding / algorithmic defect. Reboot and you are back in the same place.
Also, the AGC was directly interfaced to an astronaut. They could decide to ignore erroneous outputs from the AGC.
I think this is because an AGC failure is recoverable in most phases of flight, while an LVDC failure is not.
That has lead to some of the best rockets ever developed, and the largest satellite constellation by far. But part of the secret sauce is creating situations where you can take risks. Traditionally anything space-related deals in one-offs or tiny production volumes, so any risk is expensive. A lot of SpaceX's strategy is about changing this, whether that's by testing in flight phases the customer doesn't care about, being their own best customer to have lower-risk flights, or building constellations so big that certain failure scenarios aren't a big issue (while other scenarios still have to be treated as high-risk high-impact)
In the railway signalling industry (which for historically obvious reasons is obsessed with reliability) there even is a pattern of running different software implementing the same specification, written by different team, running on a different RTOS and different CPU architecture.