Ahah yes when I worked in the space industry and something failed we had a good time.... wait we didn't. Bosses were angry and the responsibles were sacked.
Sacking someone for making a mistake[1] is often more about petty revenge than any desire to improve things. It's a stupid policy. Someone who's made an error and learned from it is less likely to make it again than someone who hasn't made the error in the first place. This is the exact reason why experienced people are better qualified to do jobs than new people. Experience is valuable.
[1] By that I mean an actual, hard-to-foresee mistake that no one else caught rather than just incompetence or lack of ability. It's safe to assume that people without ability don't get in to decision making positions at NASA though.
an over-the-air software update was broadcast to the Viking 1 lander to correct a battery charging issue. The update had an error in it and accidentally overwrote the wrong bits resulting in a bricked lander.
That problem has been (partially) solved even in devices here on Earth: there's a small amount of memory which is either much more difficult or impossible to modify, and that holds "recovery mode" code which is responsible for the basic operations including updating the main application code. This is known as the bootloader, boot block, or boot ROM. It's designed to be simple and thus nearly bug-free as possible.
It seems that the architecture of Viking was such the bootloader would have needed to perform gimbal control of the antenna. That isn't super simple. My guess is it would have needed to know what time it was, along with a control loop and astronomical algorithms for a vector to Earth.
I wonder if they had a way to test this software update on the ground before they sent it.
Martin Marietta built what they called the "Proof Test Capsule", which was a similar to the pair that flew and was used for ground-based system integration testing. According to these [2] notes there was a separate analog/digital computer that simulated the descent from orbit to surface.
Edit: The report referenced in [2] is available for download at [3]. Starting at page 184 there is a description of the Viking Test Language, which appears to have been used for scripting tests to be run against the flight hardware/software. And this was back in 1977.
This is very interesting. And how little things have changed between 1977 and 2015 (from [3]):
"Originally it had been planned to slowly take STE programmers off of the Viking Project. By the time the STE was moved to KSC for operational test support, there were to be no programmers left. As matters turned out, the number of programmers assigned to STE peaked during this period. This
was only partially due to the problems with maintaining the H-632. New requirements for software functions constantly arose, both in Denver and at KSC. Two new programs were written at the Cape to assure that Flight loads would be correct."
Added to that - a hardware based watchdog, which if not properly exercised by running code in the main software firmware, resets the device back into recovery mode/LKG/Safe Mode - whatever you want to call it.
8 comments
[ 3.3 ms ] story [ 30.2 ms ] thread[1] By that I mean an actual, hard-to-foresee mistake that no one else caught rather than just incompetence or lack of ability. It's safe to assume that people without ability don't get in to decision making positions at NASA though.
That problem has been (partially) solved even in devices here on Earth: there's a small amount of memory which is either much more difficult or impossible to modify, and that holds "recovery mode" code which is responsible for the basic operations including updating the main application code. This is known as the bootloader, boot block, or boot ROM. It's designed to be simple and thus nearly bug-free as possible.
I wonder if they had a way to test this software update on the ground before they sent it.
Edit: The report referenced in [2] is available for download at [3]. Starting at page 184 there is a description of the Viking Test Language, which appears to have been used for scripting tests to be run against the flight hardware/software. And this was back in 1977.
[1] http://www.unmannedspaceflight.com/index.php?s=bf5c742cc7bc7...
[2] http://www.retrotechnology.com/memship/viking_software.txt
[3] http://www.dtic.mil/get-tr-doc/pdf?AD=ADA040770
"Originally it had been planned to slowly take STE programmers off of the Viking Project. By the time the STE was moved to KSC for operational test support, there were to be no programmers left. As matters turned out, the number of programmers assigned to STE peaked during this period. This was only partially due to the problems with maintaining the H-632. New requirements for software functions constantly arose, both in Denver and at KSC. Two new programs were written at the Cape to assure that Flight loads would be correct."
[1] http://appel.nasa.gov/2011/05/13/galileos-rocky-road-to-jupi...