Brett Buck · Jan 24, 2004 05:46 PM
#0 sourcehttp://www.spaceflightnow.com/mars/mera/status.html
Pretty much tells the story. Reading between the lines, they seem to have solved the most immediate problem, that is, getting it to sleep so that they end up with a positive power balance. It's supposed to sleep at night when it's not getting sun on the solar array, and the batteries run heaters to keep it going. What had been happening was that when it tried to shut down, it ran a shutdown routine, and that aborted. This is not like a desktop computer - it has to handle faults on it's own. Typically, purpose-built hardware or a simple external processor detects faults, and resets the whole comptuer to get it going again. The ability to diagnose faults and determine the appropriate action autonomously is a very, very difficult process that in many ways is far beyond the state of the art, so the logic is quite prone to not doing the ideal thing in every situation. This is true on all spacecraft that have this sort of protection, not just the MERs. In this case, on a shutdown fault, it appears to restart the machine. This is probably just what you don't want if there is a persistent hardware fault of some kind, or even worse, a software bug that show variable results depending on the conditions. The net result appears to be that the computer would try to sleep, get reset, run for a while, then try to shutdown, have another fault. on and on.
The most important thing to concern yourself with in a spacecraft anomaly is to put it into a power and thermal-safe condition. This reboot loop was preventing this, and since it didn't have enough power to run the computer all night, and still get the batteries charged the next day, the net effect being that the battery was cycling up and down in depth of discharge, with the low point getting lower and lower. At some point, the battery discharges completely at night, meaning that the heaters don't work, and everything gets cold. It may tunr on again the next day when the sun comes up, but now it's cycling up and donw in temperature - I don't know how cold it gets, but th environment is -100->-150C for 13 hours. Some parts probably stay warmer than that, but there's no doubt that the temerpature swings a lot. This fatigues the components, and eventually something will fail.
They managed to guess/determine (usually there isn't enough information to unequiviocally determine the cause of any failure, so there's an art to making good theories and guesses) that a likely cause was in the interaction with the non-volatile memory. It has regular RAM, and flash RAM to save information while the computer is off at night. That's how is stores the data overnight until it can send it back to Earth either directly or through one of the two relay satellites. It also probably is used under normal circumstances to store restart information so that it doesn't wake up like every time is the first time
What they have done is disable the use of the flash memory, and just boot up off it's EEPROM for the code/database and use the regular RAM for live storage. Now, it appears to working as expected in this "cripple" mode, and they have managed to get it to go to sleep. This is allowing the batteries to charge up and the rover now has a positive power balance. This will keep it alive. Now, the trick will be to determine how where the problem in the flash RAM process might be. It could be almost anything (to the extent I can understand it from afar just from reading the press release, and knowing a bit about the computer), from a single bad memory location, to broken interface hardware, to a bug in the software that drives the I/O.
This seems very positive. The guys working on the problem are not the cretins normally encountered on consumer tech support lines - these are experts that know the guts of the thing on a bit-by-bit and resistor-by-resistor basis. If it can be fixed, it will be fixed, and if there is any possible workaround it will be found. Keeping it alive long enough to do so was the big problem, and this seems to be well in hand.
Brett
