15 comments

[ 447 ms ] story [ 2132 ms ] thread
So fun to read. In my early career I used to think that replacing memory in servers used for production was risky. What if I could not get it done in time? Chaos!

Later I worked in a refinery when replacing mission critical items could seriously mean injury or death if mishandled. What if I made a mistake? Chaos!

Now, I realize what I was doing was child's play!

Do it wrong and the life support systems for you and your friends up there go out and you all die. Also, this entire $150,000,000,000 international investment may lose guidance and altitude control and burn up in the atmosphere, or maybe even rain down debris on some unlucky city.

No pressure!

Iirc, mission critical aerospace computers have triple redundancy since like ... forever.

So I can't believe the ISS computer doesn't have some sort of redundancy in place to prevent a catastrophe when there's so much money and lives at stake.

But I liked your joke.

Did you read the article? If you did you'd know that they run two computers. If one fail, a third automatically activates and the faulty is disabled. They also keep a fourth to swap out the failed one.

But it still doesn't change the fact that swapping out with a completely new specification can have unintended consequences. They waited until a computer failed to swap, so if it did not work, they'd run at reduced redundancy.

Did you read the article?

From the HN guidelines:

Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that."

Nobody reads anything on HN, guidelines included.
Yeah the point of this article is to say that they had no spare parts anymore since the machine is 20 years old.

Someone got an idea that they could manufacture the old missing parts (using new manufacturing process) and now they did and it works.

I've worked on financial platforms for most of my career. It's a strange feeling to know that a mistake will delete someone else's money. I always thought I had it hard. But that the end of the day, it's not "real". Certainly not life-critical. Working on something like this has got to be on a whole other level.
It can absolutely be life-critical, for people living paycheck to paycheck.
I wonder what kind of hardware and software (CPU and OS) we’re talking about here.
It's a Russian built system from the 90s, so think embedded system type interface. The hardware is probably rad-hardened 80s era chips. It has an important job, but that job doesn't necessarily require a ton of CPU cycles or oceans of memory.
(comment deleted)
It was remarkably difficult to dig up any specifics on this. But I finally found the linked PDF below. It says this about the DMS-R:

"Another example is EADS DMS-R, a computer board based on ERC32 (Atmel TSC695) processor, and a fault-tolerant TMR version using 3 boards for the ATV of the ISS (Autonomous Transport Vehicle of the International Space Station)."

I looked up the Atmel chip and it is a radiation hardened implementation of the SPARC v7 architecture. I was a little surprised it wasn't a RAD6000, since I saw elsewhere that BAE was involved. Their RAD line of hardened Power PC processors is quite popular in spacecraft systems.

https://webee.technion.ac.il/~ran/papers/SurveySpaceProcesso...

From my days as a repair techie, sounds like just another board-pulling call-out job, I could do it for $343 (per hour) including VAT. But NASA would have to provide transport.
Does your business insurance cover equipment valued at $100B, though?