> Orion utilizes two Vehicle Management Computers, each containing two Flight Control Modules, for a total of four FCMs. But the redundancy goes even deeper: each FCM consists of a self-checking pair of processors.
Who sits down and determines that 8 is the correct number? Why not 4? Or 2? Or 16 or 32?
I'm a big fan of Dissimilar Redundancies (but didn't know that was the term until today) for building system software.
Build for various Linux distros, and some of the BSDs. You'll encounter weird compile errors or edge cases that will pop up. Often times I've found that these will expose undefined behaviour or incorrect assumptions that you wouldn't notice if you were building for a single platform.
I recall OpenBSD operated in a similar way, building the system on various architectures, big and little endian, VAX, SPARC, Luna88K, etc. Quickly highlights any hardware assumptions and helped make base more robust.
The engineering behind Artemis and SLS is a masterclass in safety-critical design. The quad redundant Primary runs on
on a quadruple config PPC-750 CPU with the Green Hills Integrity OS and ARINC653 framework
While the Back up is on a LEON 3 (SPARKV8) CPU using the VxWorks and NASA's CFS framework. (https://github.com/nasa/cFS)
NASA actually makes all this publicly available information available on their NTRS server.
I remember a secret storage server that used Shamir secret sharing to shard secrets across 5 different servers with 3 shards needed to read. But the real killer was that it ran on Windows Linux and a BSD and he even wrote it in different languages!. Just an amazing work of obsessive genius but I cannot remember the name or find it.
> The self-checking pairs ensure that if a CPU performs an erroneous calculation due to a radiation event, the error is detected immediately and the system responds.
How does a pair determine which of the pair did the calculation correctly?
Candidly, while I understand the need for some amount of redundancy, I'm curious what this level of redundancy adds in terms of complexity to the system of a whole and whether or not that complexity-add almost outweighs the higher redundancy. I'm sure NASA has calculated the trade off, but I'd be curious to see the thoughts behind that.
I feel in a similar vein when learning of certain aircraft accidents over the years, where it feels like the redundancy of certain systems and the complexity it adds has been the indirect cause of accidents instead of preventing them. I suppose there's not really a way to quantify the accidents that it's prevent to be able to compare them directly.
There’s an obvious example of this with twin-engine airplanes. Having two engines obviously makes you a lot safer since you still have power if one fails. But dealing with an engine failure takes some skill, and your probability of experiencing a failure doubles. Airlines train their pilots to handle it, but if you’re a more casual pilot and you’re flying a twin, you have to be careful to ensure it’s actually making you safer.
The acceptable loss of crew risk for Artemis is 1 in 30 (3.3%), and I gather figures like that feed into the engineering constraints dictating the design (level of redundancy, materials selection, etc).
I interned at a company called Stratus which did hardware fault tolerant computers in the 80s/90s. I think they called it a “Pair and spare” approach, where every component had 3 copies running and comparing state every cycle. If one component’s state stopped matching the other 2, the failing component would be taken offline and the system would call home for a replacement to be fedexed overnight. I think just about every component was hotswappable too. Pretty cool, but expensive, and other architectures for improving availability, or mitigating impact from loss of availability, won out (except for a handful of exotic use cases).
20 comments
[ 4.3 ms ] story [ 49.8 ms ] threadWho sits down and determines that 8 is the correct number? Why not 4? Or 2? Or 16 or 32?
https://en.wikipedia.org/wiki/Lockstep_(computing)
Example: https://www.st.com/resource/en/datasheet/spc574k72e5.pdf
I other words, how over engineered is it.
Build for various Linux distros, and some of the BSDs. You'll encounter weird compile errors or edge cases that will pop up. Often times I've found that these will expose undefined behaviour or incorrect assumptions that you wouldn't notice if you were building for a single platform.
NASA actually makes all this publicly available information available on their NTRS server.
Primary and BFS Info: https://ntrs.nasa.gov/api/citations/20190000011/downloads/20... Orion BFS: https://ntrs.nasa.gov/api/citations/20230002185/downloads/FS...
Sometimes it makes the system more complicated, but it definitely also reveals where the rough corners are.
How does a pair determine which of the pair did the calculation correctly?
I feel in a similar vein when learning of certain aircraft accidents over the years, where it feels like the redundancy of certain systems and the complexity it adds has been the indirect cause of accidents instead of preventing them. I suppose there's not really a way to quantify the accidents that it's prevent to be able to compare them directly.