You'd hope an air traffic control centre would have a big box of popsicle sticks and black pen markers to cover the fallover of power | backup power | digital systems.
Because large systems are complex and have very complex and hard to detect failure modes.
What is more impressive is how rare these events are, given the complexity of the underlying systems. Redundancy is not without its own problems (source of truth, for instance).
The IT is just poor. Tech firms routinely change much more massive and complex systems at a far faster pace compared to the stagnation found in airline IT, and yet total failures are not more frequent.
Im not sure that's true. If something makes the news it's likely to be of interest to a large amount of people, that doesn't immediately qualify it as something rare.
A lot of times large scale outages like this are because of the redundancy. The whole system is interlinked with automatic failover. then it hits a corner case that was not engineered into the fail model and you get cascading system failure where each node starts bringing down other nodes automatically. basically the lesson is: In highly interlinked systems you get highly interlinked failures.
And then after a lot of angry words and finger pointing this new failure gets added to the failure model.
My personal takeaway after chasing the long tail of automatic failover on a few projects, is quite often it is better to drop a few 9's from your service goal, decouple some of the systems, and accept that while some parts of the system may go down, it should not bring everything down with it.
Complexity tends to hide problems. An over-engineered system is going to be less stable than a simple one where you know how things can break and how to bring them online pretty quickly.
The US air traffic control crash at the start of the year turned out to be a decades old system which had not been upgraded, I think the UK one is a little bit more recent, but like alot of software, people dont want to pay for the real costs of software developments.
They agiled software to the point where it’s written by product managers and wanna be engineers that cargo cult “good practice” without understanding what it’s meant to do. This types of issues are common in british made software.
Lots of US bound traffic took very southerly routings today to avoid UK airspace. My guess is that this will continue until the outage is completely resolved.
I am genuinely convinced that the UK is in collapse. The nhs is flattened, policing is meh, the infrastructure is pretty banged up, bills are through the roof, inflation and interest rates are mad high, and now this. Not to mention a noticeable drop in service quality throughout the private sector, with everyone blaming “staff shortage” and a government desperate to suppress the very same wages that are meant to attract more qualified staff. Seriously this is madness.
Most of these are global issues though in the developed world, even healthcare to an extent due to ageing population. I'm not sure what this has to do with an air traffic control outage either?
It does seem that a lot of infrastructure that was put in place during a golden age cannot be adequately maintained, but AFAIK this seems to be the case in USA and Europe too.
NATs has reported that the system is now back up and running. It is possible that the secondary system took over with data syncing having to take place before the switchover?
My original post was about my interest in the technical aspects of redundancy and failover mechanisms in something as mission-critical as air traffic control.
I come from a background in broadcasting, where failover is critical, and redundancy is built into any broadcast chain. e have multiple backups and jumping-off points to deal with any issues that arise. It's pretty rare we could ever go to “black”.
24 comments
[ 3.0 ms ] story [ 55.9 ms ] threadWhoever has the stick is responsible for the aircraft - it was a literal "unique token".
What is more impressive is how rare these events are, given the complexity of the underlying systems. Redundancy is not without its own problems (source of truth, for instance).
https://www.ft.com/content/65544730-8216-11e4-b9d0-00144feab...
And that's just the UK. Airport systems crashing is a regular occurrence. Paris crashed due to still relying on a Windows 3.1 system:
https://www.zdnet.com/article/a-23-year-old-windows-3-1-syst...
The IT is just poor. Tech firms routinely change much more massive and complex systems at a far faster pace compared to the stagnation found in airline IT, and yet total failures are not more frequent.
https://www.dw.com/en/lufthansa-system-failure-causes-massiv...
Though this is not due to outdated IT, but lack of redundancy of a critical component of the system instead.
And then after a lot of angry words and finger pointing this new failure gets added to the failure model.
My personal takeaway after chasing the long tail of automatic failover on a few projects, is quite often it is better to drop a few 9's from your service goal, decouple some of the systems, and accept that while some parts of the system may go down, it should not bring everything down with it.
https://www.wprost.pl/amp/11364934/atak-na-polska-infrastruk...
It does seem that a lot of infrastructure that was put in place during a golden age cannot be adequately maintained, but AFAIK this seems to be the case in USA and Europe too.
NATs has reported that the system is now back up and running. It is possible that the secondary system took over with data syncing having to take place before the switchover?
My original post was about my interest in the technical aspects of redundancy and failover mechanisms in something as mission-critical as air traffic control.
I come from a background in broadcasting, where failover is critical, and redundancy is built into any broadcast chain. e have multiple backups and jumping-off points to deal with any issues that arise. It's pretty rare we could ever go to “black”.