>Agency officials are attempting to determine whether the two people made the changes accidentally or intentionally, and if there was any malicious intent, the person said.
I've worked with my share of utter fools in software development. Never attribute to malice that which can be explained by stupidity.
People are remarkably failure-prone in the absence of controls and guardrails. Sadly, I suspect the individuals will get the rap instead of whatever crap process made it possible for the changes to pass muster and get into production without being caught and corrected.
I’m in complete agreement with you. Could you imagine 2 contractors from somewhere like Accenture, who are already underpaid and overworked, not making a mistake?
No, I cannot picture someone from ACN failing to make a mistake.
Malicious changes to the IT systems were done outside the FAA by messing directly with the elections of officials who run federal agencies like the FAA.
Americans always look for a movie plot when the very real threats to “the homeland” looking to abuse their access are operating in the open and have been for many years… and they’re not low level software engineers.
Couldn’t agree more. Its totally insane that backwards middle eastern billionaires are allowed to just buy up American tech and media companies. At least this one has a sense of humor.
i don't know that dude in particular, but 'backwards middle eastern' sounds a bit racist.
for instance, maybe that person has said something kind of evil, imo -- like maybe that an entire country should be ceded to some other country -- but even then i wouldn't want to overgeneralize -- i def don't like being associated with the most famous/rich americans when i go abroad, or interact online.
but i would agree generally that non-us investors buying up us politicians, pr firms, and tech and media companies is something we/us should be careful about.
>i def don't like being associated with the most famous/rich americans
On some level Americans are responsible for what our government and economic leaders do. We cant identify as and benefit from being Americans and then chide other groups for being racist when they complain about what our leaders do in our name. The reverse is equally true
you have it completely backwards and I can show it
the people that pushed, bragged and elbowed their way into executive management at a massive, eternally funded bureacracy are experts at disregarding the viewpoint, input and most of all directions from others
The first thing to do in such a position is to reward an inner loyal circle by increasing their benefits, authority and lessen the actual work, via contractors. This thread shows anecdotal evidence that setups involve far more than half the working headcount as contractors. The contractors do real work, take real blame, and are routinely pushed around since that is what people really do to each other.
At what stage of any of this do "we" take responsibility and make change?!? it is fortified income protection, not responsive systems, described here..
I suggest reading the Simple Sabotage Field Manual; the individual and organizational behaviors you are describing are indistinguishable from those of a skilled saboteur:
"Any sufficiently advanced incompetence is indistinguishable from malice."
Fact remains that a system wherein this is possible is a system that has been exploited. Marshaling the forces to fix the system is the preferred outcome, unless you want the system to remain exploited (ie, you benefit from the situation).
I think the comment you're replying to is trying to say something about the management being compromised, deliberately hiring incompetent people and being lax about controls, and wanting the failures to happen for some unspecified conspiratorial reason. That merely bumps the question up another level, it's a question of malice or incompetence. Conspiracy theories tend to end up invoking some shadowy malicious cabal at the highest levels, above even leaders of nation-states, to explain just plain old human nature.
When a society is run by people who care only for their own small wealthy in-group and nothing for the common good all kinds of incompetence is bound to ensue. It doesn't take a tin foil hat to see how that has happened to America.
I tend to find that idea is adjacent to conspiracy thinking.
If society is being “run” by anybody then either (a) they could do a much better job of it for themselves, or (b) they are very good at hiding.
“Powerful” people seem to often have the same problems as the rest of us: they seem to be on the receiving end of systemic issues they don’t understand as much as any of us, they often work long hours, they still get tragic family issues and chronic health issues.
Yes the wealthy mega-donors in American politics could absolutely do better by American workers than they are. How do these people being old and having health problems excuse their treatment of American workers?
What about the software developers? A data entry problem or file corruption should not lead to a production outage! Robust software should gracefully reject bad input, show or log an error, and continue on. Those contractors should be able to feed emoji or line noise into the system and not cause an outage. This is why we fuzz test.
If the take-away from this is “contractors should follow procedure and not make mistakes when entering data” then the FAA has not fixed anything.
The FAA had earlier referred to it as a "damaged database file". Which I suspect translates to something like a index file + data file pair, or a raw file for some kind of database. It sounds like the contractors accidentally overwrote something like that.
So, yes, the system should have thrown some kind of "invalid magic number" or "unable to read header" error. Though that would have probably only helped a little. They were still slow to recover even after they understood the issue. Meaning they are missing still other things, like procedures to start over using fresh data. They had, for example, lots of partners that already had all the "known good data before the issue", but no way to take that and run.
The failure was in a technical organization architecting a solution that allowed a simple human error in data entry to take down the entire system. This is classic system design and process failure. Bad input should have been caught before it got pushed to prod, before it got pushed to backup databases, etc. There was clearly no testing, no change process, no backup verification, etc.
People in senior management positions are at fault for creating a broken org and product. At a private company a failure like this would see the CTO immediately fired and even the CEO having to beg the board for forgiveness and keeping their job. The fault is not with the two contractors, it's with the bad engineering that enables a mistake to destroy the product.
I agree that the ultimate root fault isn’t the recent data entry.
> At a private company a failure like this would see the CTO immediately fired and even the CEO having to beg the board for forgiveness and keeping their job.
Right. Engineer closest to the fault gets scapegoated and fired, new hire gets hired in without any idea what they’re coming into, and the show goes on.
Good interview question to ask: “am I replacing a recent departure? If so, why?” You either get the truth or deception signal once hired. Both are helpful to know, and asking the question is free.
I think it also matters on what the response to the incident was, in particular the answer to the questions of: who was responsible and how can we prevent this from happening again?
If the answer to the first question is a bunch of finger pointing or the answer to the second is along the lines of "nothing", heads will likely roll.
My first boss as a FTE at a trading shop made a change to a production database outside of process before a weekend. Come Monday morning, production trading systems couldn't come up for market open. We were also an options market maker. He refused to accept responsibility for his actions and denied there was anything done to prevent it. He was quickly behind closed doors in his boss's office. I had to undo the changes he made the week before (hooray for audit tables) as soon as I could. We were potentially liable for a substantial penalty from the exchange for being out of the market, and every minute mattered. Don't know if we ever did get penalized, but he was fired and out the door by 10AM that day.
I've certainly seen and made lots of mistakes over the decades since, but that's the only one for which I've seen someone canned. Response matters.
>Right. Engineer closest to the fault gets scapegoated and fired, new hire gets hired in without any idea what they’re coming into, and the show goes on.
You can tell what part of the world a poster comes from when they're smugly pretending that companies still don't have plenty of levers to pull to push you out the door, doing so by making you miserable.
Oh, and the rampant racism and classism, thanks to the standard practice of including a photo on one's resume...
In the places I've experienced with poor management and engineering cultures, what I've seen is management dictates all sorts of horrible, asinine things (like story points being treated as deadlines, etc.) that leads directly to problems like this because engineering is not given the space it needs to create robust solutions in the first place.
The prevailing winds are, "Just get it done." The outcome is constant fire fighting and triage because of simple problems like bad data finding it's way into the system.
Funny. I'm reading The Phoenix Project and just finished the about the CEO chewing out various employees. I keep thinking, responsiblity flows upstream, or should. Ultimately it is the CEO's job to put his people and the project in a position to suceed.
Maybe the CEO in the book gets his due?
Blaming a contractor? To save face? Isn't saving face. It's an embarrassment.
Maybe even more pertinent is that they had a similar one in 2016 (smaller scale, but similar cause/effect) and then didn't fix anything, so it could happen again recently.
There's running lean and then there is running recklessly.
Yeah you are right. Also, there is another side of this coin. I used to work for a FAANG and here is an outage story for you. We had every angle covered of preventing user errors and stopping a single data entry to take down systems. We had reviews of changes, multiple approval required etc. One of my co-workers was working on a change that got approved he had to change the IP of a dns entry pointing to a load-balancer. The change involved inputting the IP address of the DNS server and the load-balancer's IP as well. Needless to say he mixed up the two IPs causing a several hour outage. The moral of the story is that you can architect the shit our of everything try to avoid these problems but there is always going to be a case that you did not cover and it can cause serious outages just like this one.
If manual entries are critical to a system, you can have one person enter the info and someone else to confirm the entry is correct. Not sure how common this is in other industries, but this was the method used at an extrusion manufacturing plant I once worked at.
It is very common in aviation. That is why at any point in time one pilot is flying the plane and another one is monitoring what the former is doing. The pilot monitoring is responsible for verifying the actions made by pilot flying not to punish or blame anyone, but just to make sure that intended action (pronounced by pilot flying) is executed correctly. It is possible that, for instance, you wanted to set value to be X but accidentally rotated a switch a little more. There are also other responsibilities of course. But, yes, making action in pairs is something critical in aviation. That does not happen often when it comes to ground software though. You need to remember that NOTAMs are user-generated content, many systems which allows pushing data to these systems are older than me. Nevertheless database maintenance is possible to do in a safe way. It’s not a DNS nor BGP - it’s high level software where tampering files is absolutely avoidable.
The FAA earlier called it a "damaged database file". I'm not convinced it was data entry exactly.
For example, running "/some/script > data-file" instead of "/some/script < data file" still fits the very generic wording the FAA is sharing publicly.
Could have been accidentally overwriting some idx/dat pair, or some raw database file, etc. Though, yeah, there should still be some controls, troubleshooting tools, logging, descriptive errors, etc, that would have made things more clear and recoverable.
An earlier report said that the operator had unintentionally clobbered a database file (the newspaper did not use the word "clobbered" but it is the correct term of art).
It explains that procedures were circumvented and something in production was edited - something that was not allowed.
You can put as many procedures in place as you want, but unless the humans follow these procedures (I.e. not circumvent them deliberately) then someone always has some level of access where they can cause mayhem.
Sure you can programmatically enforce this to a large extent, but ultimately there are always some humans with prod access, or BGP editing access, or firewall write access, or access to that validation logic itself! And it requires that the humans follow the organisational procedures in place.
From what the article says, it sounds like the humans deliberately circumvented procedures, in which case, it is the engineers fault, and they should be disciplined.
Another one: sometimes, especially in large organizations, there are so many rules that you can't find anyone who actually knows them. Making it unlikely they're being followed.
In large org thinking, if something bad happens, and you add a rule, then it's fixed.
The problem is that this is not compatible with human psychology.
This leads to many rules, the rules are unstructured and impossible to learn, the rules probably also contradict each other.
In my experience, a rule without automatic enforcement should be the absolute last thing to depend on. If you do this, your org is broken.
The Toyota Production System with it's blame on process design instead of humans is still the best way to avoid these issues IMO.
Do 5-whys where human error is not allowed as a root cause. Design your process with poka yoke in mind. If somebody can forget something they will. Don't depend on it.
There's no mention of deliberate circumvention. It doesn't even say there was circumvention, only that the files were damaged with the rules in place. If the rule is "don't damage these files when writing them" and it's your job to write them, it's adds no safety.
> Like other computer systems that are critical to operating flights, the FAA has imposed procedures to ensure data aren’t damaged by technicians working on them, said the person. The file or files were altered in spite of rules that prohibit those kind of changes on a live system.
> Agency officials are attempting to determine whether the two people made the changes accidentally or intentionally, and if there was any malicious intent, the person said.
The lede reads, verbatim: "The computer failure that prompted a halt of all US flight departures was caused when a data file was damaged as a result of a failure to follow government procedures, the Federal Aviation Administration said Thursday."
It is clearly implied that had the rules been followed, the corruption wouldn't have occurred.
Your blame-the-FAA scenario where the rules were incomplete or flawed isn't ruled out, but I don't see value in asserting it without evidence. I think it's perfectly plausible that someone just cheated a bit. That's hardly a rare situation in any IT environment, and so it seems like Occam favors the article's interpretation.
>At a private company a failure like this would see the CTO immediately fired and even the CEO having to beg the board for forgiveness and keeping their job.
If that sort of thing actually happened a lot more C level people would have been fired for over-hiring, but that rarely happens.
> At a private company a failure like this would see the CTO immediately fired and even the CEO having to beg the board for forgiveness and keeping their job.
I don't think this is a thing that actually happens.
I don't know if this is true of the CTO and senior tech staff but SWA's CEO had just been in there for 8 months, so I wouldn't expect anything to happen with him.
But the previous CEO looks like he was the spreadsheet type and rejected these kinds of ideas. So who is to blame?
> […] a data file was damaged as a result of a failure to follow government procedures, […]
The more critical processes humans are expected to perform, the worse the expected results.
That goes 10x for outside or temporary “contractors” who are not going to be experts in a systems unique, esoteric, or unexpectedly irresponsible points of fragility
Bruh you’re complaining about a system that had about perfect uptime for 30 years. It’s only newsworthy because it was so perfect
Second, nothing would happen at a private company because they would involve marketing and marketing wouldn’t blame anyone in a press release. The only difference with FAA is that they didn’t let a marketing department write their release.
But really the real issue here is that everyone is posting to make wide generalizations from a one time event.
> At a private company a failure like this would see the CTO immediately fired and even the CEO having to beg the board for forgiveness and keeping their job.
Can you share an example of where that has actually happened?
> CTO immediately fired and even the CEO having to beg the board for forgiveness and keeping their job.
I doubt that. More likely: the defect would be identified and fixed, someone would start working on a training-wheels editor for the file to ensure the problem doesn't happen again. Any change in revenues or profits would be used by the CEO to drive change in whatever thing needs changing, and the whole incident would be forgotten. Source: have owned four private companies. If everyone is fired immediately, no one learns a lesson and everyone is terrified to give leaders bad news.
The article did not say how the bad data was introduced. There is plenty of room for contractor malfeasance or incompetence to be a factor. Working around the safety measures in place, for example. Using an import tool improperly. Directly modifying a database through console, or, custom application software written without customer knowledge. Abusing privileged accounts in non-authorized ways. I'm sure there are many more.
The engineering precautions in system design always involve trade-offs. There is no perfect system that could be created for acceptable costs. Ultimately it is always going to come down to humans following the rules, hopefully as few of those very-trustworthy humans as possible, though that's where the FAA may have been lax.
Hey now, I received my $5.21 check from equifax for their huge data breach! I’m sure that taught them a lesson, but I don’t think it’s the lesson I’m hoping for….
> The failure was in a technical organization architecting a solution that allowed a simple human error in data entry to take down the entire system. This is classic system design and process failure. Bad input should have been caught before it got pushed to prod, before it got pushed to backup databases, etc. There was clearly no testing, no change process, no backup verification, etc.
> Transportation Secretary Pete Buttigieg told NBC News that he has asked the FAA, "to make sure that there are enough safeguards built into the system that this level of disruption can't happen because of an individual person’s decision or action or mistake."
People are blaming the system designers a lot in this thread, but:
> The file or files were altered in spite of rules that prohibit those kind of changes on a live system.
Can anyone here argue that, given arbitrary write access, intimate knowledge of the system, and the ability to disable any deployment and runtime checks I want, I wouldn’t be able to take out their production system?
> given arbitrary write access, intimate knowledge of the system, and the ability to disable any deployment and runtime checks I want, I wouldn’t be able to take out their production system?
This is a fair point. This press release was not aimed at software engineers, and "rules that prohibit those kind of changes" is vague. It could mean anything from a sign on the wall that says "please don't!" to a robust access control and deployment pipeline that they intentionally circumvented.
If the policy was violated then the policy was violated. If it was properly communicated and taken seriously by management (and it may not have been) then the person who violated it is at fault along with management for choosing the wrong person for the job. Given that this is a PR release, I suspect management’s diligence in the selection and training of the contractors is not being critically examined.
Good policies account for people not following the policy. Since this is aviation related, let's take "no smoking on a plane" as an example. There are "No smoking" signs everywhere and it's announced over the speakers several times prior to takeoff. Yet there's still an ashtray in the lavatory in case somebody disregards every warning so that they're not throwing a lit cigarette into a bin filled with paper products.
Think about the request from air crew to switch phones to the flight mode. Many people assume it’s for safety reasons. In fact would it be dangerous for the airplane systems there will be a real measures in place which are enforcing the rule - like a kill switch for all phones, scanners for not turned off devices, collecting devices before boarding a plane; but nothing like that in place. It’s because this particular request is to reinforce that passengers must comply with air crew orders, regardless of what they request (to some reasonable boundaries of course)
Imagine how many laptops, bluetooth speakers, tablets, and other devices are dinging "the waves" for wireless of all kinds from beneath seats and in overhead bins. Peer-to-peer, speakers, wifi, cell, ....
I just realized I have never disabled cell access on my watch.
It isn't clear to me why airplane mode is requested on airplanes. It doesn't reinforce crew control, it reinforces complete lack of crew control, i.e. crew control theatre.
> If the policy was violated then the policy was violated. If it was properly communicated and taken seriously by management (and it may not have been) then the person who violated it is at fault along with management for choosing the wrong person for the job.
There has to be a limit to this reasoning, though. There is a threshold where an organization is so negligent that they can’t concentrate the blame on an individual, regardless of whether there was a policy violation.
I want more details about that, too. That could be a reckless cowboy but I’ve also seen cases where people routinely bypass the official process because it doesn’t handle all of the work they need to do, or is slow/unreliable in ways incompatible with the volume of work, etc.
It does suggest a big lapse in oversight: if this wasn’t the first time they did this, their security monitoring should have picked it up.
My guess is that this is going to be one of those situations where someone broke the rules in a way their immediate management knew about but sanctioned because the official process was dysfunctional. In those cases it’s really common to crucify the operator and ignore the larger problem.
This is a great example of the general principle that what often gets dismissed as operator error is better recognized as design flaw.
When SpaceShipTwo broke up mid-flight, the FAA did not place all the blame on the pilot who accidentally engaged the feathering mechanism during ascent. They thoroughly dragged the aircraft's designers for producing a design that made it so easy to do such a thing in the first place.
They weren't willing to accept, "We said not to do it in the manual," as an appropriate safeguard in that case. Why are they doing it here?
If we want to call ourselves software engineers, we need to start holding ourselves and each other to higher standards. We can do better.
Software is still developing at a pace that burdensome regulatory red tape would represent a completely unacceptable destruction of value.
Defaulting to red tape as the mechanism to improve the quality of software is also very short-sighted. Software engineering has the absolutely unique property among all engineering disciplines that we are working with abstract systems which are subject to formal verification. We can get arbitrarily high levels of assurance, with zero need whatsoever for some stodgy accreditation body.
This is not happening much at the moment because money is (artificially) cheap, so it makes sense to hack together some crap and get it out the door as quickly as possible. There are solutions to this; slapping regulations on software is not the best one.
> burdensome regulatory red tape would represent a completely unacceptable destruction of value.
Typical holier-than-thou software industry arrogance and hubris. There is nothing forcing red tape regulation on all engineers nor would it for software engineers.
Establishing a consistent level of competence and training for entrants is important for consistent quality in the field. It will also help companies to stop inventing their own unique interview process and just look for verified accreditation when determining whether to hire someone.
It's absolutely bonkers that some companies don't trust the general population of devs with 5+ years of verified experience and still gives them technical interviews. But you can't really point them to a good standard to hire with either.
Only metrics resistant to Goodhart will work here, and most of them are illegal to use in the US under the current employment law paradigm. One of the reasons software as an industry makes so much money is there are still ways to implement Goodhart-resistant hiring in the US. Going into too much detail here is somewhat hazardous, so I will leave it there.
>Going into too much detail here is somewhat hazardous
I don't exactly believe that.
And we have other professional exams/processes that generate good professionals. Folding over and saying "Oh we can't do this, Goodhart's law exists" just seems like a weak reason to me. https://en.wikipedia.org/wiki/Perfect_is_the_enemy_of_good
But it wasn't the FAA that placed the blame on the SpaceShipTwo designers, it was the NTSB. Hopefully there's a similarly independent agency that can lead an investigation of the NOTAM debacle.
Yep, FAA was owned by Boeing before the 737MAX debacle. It amazes me the public flies on that airframe which has the engines so far forward it can pitch up uncontrollably.
Grew up around the FAA. I will say this of the organization: they are extremely thorough about root cause analysis and red tape, so I doubt this ever happens again.
There’s several comments in this thread pointing out that given a certain level of access, intentionally bypassing policy is possible in any organization. What I don’t think is being given due consideration is how often the people with this access are contractors, and worse often foreign nationals, which even in civilian orgs like the FAA has serious national security implications.
The West has an addiction to foreign IT contractors. It’s a many hundreds of billions yearly industry globally, and is only possible because the largest and most critical institutions in our society refuse to invest in people to keep this knowledge in house and grow their own trained workforce.
It's always easier to blame contractors than to determine why and how. And Snowden talked about in his book that roughly 95% of the fed gov is contractors. Reason is that getting money is easy, but getting OMB approval for more hires is ridiculous.
These people had a public trust or a clearance. They've worked their way in the fed gov. I wouldn't immediately question malice or otherwise. You don't just throw away a career with sabotage.
Also having worked in a similar environment, I can guarantee you that although there may be a test env, it is nowhere near similar to prod. I've seen it myself with different firewall rules, different GPOs, different software hardening packages. And I also complained that there should be no distinctions between stage and prod - insomuch that DNS and SSL should be the only real differences.
I'm also acutely aware if my project has a similar incident, that I too would be scapegoated and fired. It's part of the course of this job.
I worked there for 10 years. What is this test env you speak of? /s
We had dev and prod. No version control and no testing requirements. Was a while ago for sure (i used svn locally for sanity). But from what i hear not a lot has changed. This was not flight related software. They did have better tools and procedures but not by leaps and bounds
Oh, we always have a test environment :) Sometimes its also called prod.
Ive had the displeasure of having another team write the Cloudformation scripts, and they wrote them in such a way that none could be redeployed, or changed. And since they're domain joined windows machines, effectively un-redeployable.
My first start, which died in situ, was to do the work to get stage up and running, and then clone stage into prod and changing the DNS and naming bits. Since this was a no-go, ended up piecemealing everything. It still stings too. We're nearly done and ready to go live as well. Ive been pushing for live since November. But when a single person speaks out, "delays get delayed by meetings to discuss in meetings to have meetings about delays" (note: I'm not making this up)...
I shouldn't complain too much. A good 4h a workday is filled with gaming and stuff, since the enforced delays and "hurry up and waits" require me to literally do nothing. But when I do work, I'm going at the 5-10x speed I'm accustomed to going. But when it takes weeks to open up SMB from internal machines to other internal machines... there's not a lot I can do to fix broken process.
And, I'm just a lousy contractor out of hundreds of thousands.
There's not enough information to be sure, but it sounds kind of like an SQL injection type of issue. Which would be very surprising as that's been known about for many years and all systems should have some kind of input validation setup in place to prevent it. I suppose such any such setup could be overridden by someone with the appropriate level of access however.
OH I see they edited the file that reads "DONT EDIT THIS FILE". A cronjob that relies upon the letter X appearing at character position 450 then failed.
Even if you ignore the design issues that let contractors corrupt the database, it's still a massive failure of system design & policy because they clearly didn't actually have a backup system.
> When the system began having problems Tuesday night, technicians switched to a backup. But because the backup was attempting to access the same damaged data, it also didn’t work, the person said.
Their entire backup plan was to fail over to an identical secondary system that was supposedly redundant. But, it was clearly sharing the same database system as the primary, thus they didn't actually have a fully redundant system. They had 1 system with some parts of it having a secondary, i.e. not an actual DR / failover plan.
Crypto-skeptics downvoted my previous suggestion (https://news.ycombinator.com/item?id=34349576) that NOTAM's sort of simple textual bulletin log – perhaps not even 150MB of data per year! – could be implemented as a distributed-consensus database.
But blockchains are specifically designed to keep on appending, no matter the mistakes of individuals, or even motivated sabotage attempts of sophisticated attackers.
Essentially, blockchains have a giant, full-coverage set of unit-tests in their shared & public consensus code – reviewed and running across many systems. And many of their designs have proven uptimes in adversarial environments for years.
And the task gets even easier when there's a small fixed set of established authorities with privileged keys.
Protecting those keys is also, of course, a hard problem. But compromised credentials is probably already a fatal threat for the legacy system(s) that were brought down by "contractors who introduced data errors". So a hypothetical modern crypto-consensus solution's failure modes remain a subset of those of the legacy system.
I don't think it helps to characterise this as a simple textual bulletin log.
I am by no means an expert, however to me this system appears to include: a globally agreed upon data format, a transmission protocol hardware distributed across the planet in ground stations and air vehicles, and 70 years of legacy.
And all of that is enshrined in legislation across many countries.
And while there is a modernisation effort in progress, the nature of this system means that things will move slowly.
In 2023 anyone can suggest a better data format and distribution system if they start from scratch. Good luck getting the world to use it and good luck getting anyone to care.
You're right that the main challeges are political & bureaucratic inertia. That shouldn't stop us from honestly & plainly describing its core, & the industry's state-of-the-art for similar systems that don't have the legacy system's encumbrances.
The "globally agreed-upon data format" is essentially text blobs with a standard set of abbreviations, that assist search/filtering by flight-paths & planned waypoints. It seems to be a mere 150MB a year.
The inherently "move slowly" modernization effort can only be helped by a clearer outline of what's now possible. This is too important to prematurely dumb-down brainstorming ideas, based on low-expectations for the adoption process!
And a modern architecture for distributed updates & reliability could trivially mimic the old system's pilot-visible outputs, to remain familiar to tenured personnel.
Um, an immutable journal means you can't delete bogus data. If, for example, a system chokes when it sees a 3-byte UTF-8 code point, and you accidentally insert such a thing in the journal, then you're hosed. The system can't come back up until the software is mad-panic patched. Or (again hypothetically) somebody fat-fingers an update, deletes ATL from the airport list, thus causing a crash when software sees now-orphaned NOTAMs.
Nor does distributed-consensus seem to apply in this case. For many datasets you want one source of truth (e.g. airport codes, approved abbreviations).
Crypto blockchains deal with very, very simplistic global states. Bitcoin, for example, has (a) 256 bits from the prior block, (b) a megabyte or so of transaction data. Every state machine update (mined block) totally replaces the prior state. And it crucially depends upon backtracking never being needed; for example, if a prior block's hash made it impossible for a new block to be created with enough zeroes in the hash (I don't think that's possible), then a global, manual fork would be needed.
If the instructive content is invalid, you send a invalidation message. But if there's a violation of constraints, modern practice (both in and outside crypto projects) can do a pretty good job of precluding it, with test cases, wide review, multiple implementations, formal verification, and so forth.
And when something slips by, many eyes & many implementations make a consensus fix very fast. No one has 'fat-fingered' the major blockchains in a long time! Some proprietary/permissioned chains have had problems. And stuff as complicated as smart contracts can go awry in subtle ways. But NOTAM seems to be just a simple text log!
Distributed consensus inherently achieves one-source-of-truth without single-point-of-failure. As the notices relevant to individual flight-paths are inherently geo-local, with many independent submitters (like airports), any routing "through the FAA in DC" introduces risks – even if the FAA has final authority over who (which keys) can add entries.
Yes, Bitcoin's consensus state is relatively simple – but also far more compicated than NOTAM's even-simpler message. And continually redundantly verified & replicated by many independent actors & codebases, very successfully, for 10+ years.
> could be implemented as a distributed-consensus database.
Yes it can. That is not the question. The question is why do you think a bog standard, but correctly implemented and administered database could not handle the same?
Imagine the following: the task is to move two wheels of cheese 5 km from A to B. Your proposal is akin to someone comming and suggesting that a suborbital rocket can do the job. And you are right. It can. But so can a Toyota Corolla without breaking a sweat.
So tell me, why should we use a “hypothetical modern crypto-consensus solution” here, with all its complexity, instead of a well understood and bog standard and well implemented database?
Any bog-standard database might work. But if so, curious the FAA hasn't just done that: say, SQLite replicated across many places.
But the centralized team running it could still "fat-finger" it. Any "master DB" system has problems that blockchain-like approaches have definitively solved, allowing progress on valid messages when large portions of the usual maintenance systems have gone down or been compromised.
Of course any major RDBMS would have no problem with the data. If the reports here are accurate, you could fit the last 5-20 years of all NOTAM messages into the RAM of a Raspberry Pi running SQLite.
But if that's enough, why hasn't the FAA solved the problem already?
And even a "major RDBMS" is vulnerable to stodgy management leaving design in a state where it's vulnerable to a single central point of failure, where "2 contractors who introduced data errors" (!) can bring it down.
Modern highly-distributed consensus systems are not vulnerable to that. Neither accident nor malice from 2 government contractors could prevent all authorized updates worldwide in a distributed consensus system like Ethereum.
DC & northern Virginia (or wherever the FAA's systems hosted) could even spontaneously disappear into a lava pit and the official consensus set of NOTAMs would continue to update.
Because this was all standardized back when interoperability meant standard fixed-length data formats that everyone had to understand rather than versioned API's and the inertia of the established file formats is too great.
Hell, if you _really_ wanted to insist on an exchange data format, XML would work just fine.
`When the system began having problems Tuesday night, technicians switched to a backup. But because the backup was attempting to access the same damaged data, it also didn’t work, the person said. `
That's not a backup. Sounds like they have standby hardware for one tier of the architecture which they switched to, but since it wasn't a hardware failure it didn't change anything.
One hopes they review not only their processes which allowed (one assumes) a configuration change to be applied without being verified, but also their backup strategy...
> When the system began having problems Tuesday night, technicians switched to a backup. But because the backup was attempting to access the same damaged data, it also didn’t work, the person said.
That's not a backup. Sounds like they have standby hardware for one tier of the architecture which they switched to, but since it wasn't a hardware failure it didn't change anything.
One hopes they review not only their processes which allowed (one assumes) a configuration change to be applied without being verified, but also their backup strategy...
To create a complete backup, one must first create a snapshot of the state of the universe. Then to be sure it worked, one must restore that backup and run regression tests.
OK, so I know nothing about NOTAMs and even less about the FAA infrastructure.
But: data errors introduced by a bad disk? In 2022? Storing your data on a hardware-backed (or, ZFS-backed, if you must) RAID6 (or equivalent) partition has been bog-standard for the past decade or so, and a partition is either online (and OK) or degraded (but still OK), or offline, at which point 'restoring the latest backup' (or just failing over to a replica) should bring you back to an OK state.
Then: apparently, restoring the offline partition caused bad data to be ingested, over and over, crashing the system again and again. Even if we pretend that solutions like SQLite have not been available for the past decade or so, 'skip past the head of the corrupted data file' has been best-practice since I last saw abominations requiring such hacks (say: Novell MHS), and sorry to repeat myself, over a decade ago?
TL;DR: please pay me US$ 200M, and I'll make sure all your problems are at least not of the kind solved a long time ago?
I'm going to disagree that "skip past the head of the corrupted data file" is something that should be done in a safety-critical environment. What happens if the head of the data file contains the location of a hazard that 100 planes need to be cautious of?
Sure, "skipped past ## corrupted octets at offset ####, offending contents saved as `/mnt/log/data/yyyyMMdd/seq.json` and discarded" should absolutely be logged and trigger the highest alert level. It should not, however, impact the availability of the system.
FAA specifically should know better than blaming individuals. Whatever happened was a systemic issue because in a well designed system individuals simply should not be able to bring this much chaos. Neither accidentally nor maliciously.
Is there anyone there who heard about just culture, and root cause analysis? I can’t even believe they would publish this irresponsible stuff.
I wonder if it's just a coincidence that the FAA started accepting public applications today[1] for 12 general engineer vacancies (of the program management office type) in the DC locality.
130 comments
[ 3.2 ms ] story [ 214 ms ] threadPeople are remarkably failure-prone in the absence of controls and guardrails. Sadly, I suspect the individuals will get the rap instead of whatever crap process made it possible for the changes to pass muster and get into production without being caught and corrected.
Malicious changes to the IT systems were done outside the FAA by messing directly with the elections of officials who run federal agencies like the FAA.
Americans always look for a movie plot when the very real threats to “the homeland” looking to abuse their access are operating in the open and have been for many years… and they’re not low level software engineers.
https://twitter.com/Alwaleed_Talal/status/158597522656711065...
@Alwaleed_Talal Dear friend "Chief Twit" @elonmusk
Together all the way @Twitter
for instance, maybe that person has said something kind of evil, imo -- like maybe that an entire country should be ceded to some other country -- but even then i wouldn't want to overgeneralize -- i def don't like being associated with the most famous/rich americans when i go abroad, or interact online.
but i would agree generally that non-us investors buying up us politicians, pr firms, and tech and media companies is something we/us should be careful about.
On some level Americans are responsible for what our government and economic leaders do. We cant identify as and benefit from being Americans and then chide other groups for being racist when they complain about what our leaders do in our name. The reverse is equally true
the people that pushed, bragged and elbowed their way into executive management at a massive, eternally funded bureacracy are experts at disregarding the viewpoint, input and most of all directions from others
The first thing to do in such a position is to reward an inner loyal circle by increasing their benefits, authority and lessen the actual work, via contractors. This thread shows anecdotal evidence that setups involve far more than half the working headcount as contractors. The contractors do real work, take real blame, and are routinely pushed around since that is what people really do to each other.
At what stage of any of this do "we" take responsibility and make change?!? it is fortified income protection, not responsive systems, described here..
https://archive.org/download/SimpleSabotageFieldManualStrate...
"Any sufficiently advanced incompetence is indistinguishable from malice."
Fact remains that a system wherein this is possible is a system that has been exploited. Marshaling the forces to fix the system is the preferred outcome, unless you want the system to remain exploited (ie, you benefit from the situation).
[1] http://wikidumper.blogspot.com/2007/07/greys-law.html
I tend to find that idea is adjacent to conspiracy thinking.
If society is being “run” by anybody then either (a) they could do a much better job of it for themselves, or (b) they are very good at hiding.
“Powerful” people seem to often have the same problems as the rest of us: they seem to be on the receiving end of systemic issues they don’t understand as much as any of us, they often work long hours, they still get tragic family issues and chronic health issues.
https://thefederalist.com/2019/12/05/tucker-carlsons-critiqu...
Yes the wealthy mega-donors in American politics could absolutely do better by American workers than they are. How do these people being old and having health problems excuse their treatment of American workers?
A note is taped to the side of some filing cabinet that says
If the take-away from this is “contractors should follow procedure and not make mistakes when entering data” then the FAA has not fixed anything.
So, yes, the system should have thrown some kind of "invalid magic number" or "unable to read header" error. Though that would have probably only helped a little. They were still slow to recover even after they understood the issue. Meaning they are missing still other things, like procedures to start over using fresh data. They had, for example, lots of partners that already had all the "known good data before the issue", but no way to take that and run.
People in senior management positions are at fault for creating a broken org and product. At a private company a failure like this would see the CTO immediately fired and even the CEO having to beg the board for forgiveness and keeping their job. The fault is not with the two contractors, it's with the bad engineering that enables a mistake to destroy the product.
> At a private company a failure like this would see the CTO immediately fired and even the CEO having to beg the board for forgiveness and keeping their job.
That part doesn’t match my experience at all.
Good interview question to ask: “am I replacing a recent departure? If so, why?” You either get the truth or deception signal once hired. Both are helpful to know, and asking the question is free.
That happens. But not every time, and even "usually" requires some statistics collecting to be sustained.
Very often people just say "yes, we learned an expensive lesson", and change something that may or may not help the next time it happens.
If the answer to the first question is a bunch of finger pointing or the answer to the second is along the lines of "nothing", heads will likely roll.
My first boss as a FTE at a trading shop made a change to a production database outside of process before a weekend. Come Monday morning, production trading systems couldn't come up for market open. We were also an options market maker. He refused to accept responsibility for his actions and denied there was anything done to prevent it. He was quickly behind closed doors in his boss's office. I had to undo the changes he made the week before (hooray for audit tables) as soon as I could. We were potentially liable for a substantial penalty from the exchange for being out of the market, and every minute mattered. Don't know if we ever did get penalized, but he was fired and out the door by 10AM that day.
I've certainly seen and made lots of mistakes over the decades since, but that's the only one for which I've seen someone canned. Response matters.
That part doesn’t match my experience at all.
Good question though..
Oh, and the rampant racism and classism, thanks to the standard practice of including a photo on one's resume...
In the places I've experienced with poor management and engineering cultures, what I've seen is management dictates all sorts of horrible, asinine things (like story points being treated as deadlines, etc.) that leads directly to problems like this because engineering is not given the space it needs to create robust solutions in the first place.
The prevailing winds are, "Just get it done." The outcome is constant fire fighting and triage because of simple problems like bad data finding it's way into the system.
Probably not in this case, but you couldn't assume for every...
Maybe the CEO in the book gets his due?
Blaming a contractor? To save face? Isn't saving face. It's an embarrassment.
TBH IDK the book was mentioned on HN and I figured I'd check it out.
Exactly! There wasn't any real fallout for Southwest, who had a shutdown less than a month ago, for similar outdated technology reasons.
There's running lean and then there is running recklessly.
I joke but the right amount of bureaucracy is good, but people are very careful about adding more of it because you can’t undo it
For example, running "/some/script > data-file" instead of "/some/script < data file" still fits the very generic wording the FAA is sharing publicly.
Could have been accidentally overwriting some idx/dat pair, or some raw database file, etc. Though, yeah, there should still be some controls, troubleshooting tools, logging, descriptive errors, etc, that would have made things more clear and recoverable.
It explains that procedures were circumvented and something in production was edited - something that was not allowed.
You can put as many procedures in place as you want, but unless the humans follow these procedures (I.e. not circumvent them deliberately) then someone always has some level of access where they can cause mayhem.
Sure you can programmatically enforce this to a large extent, but ultimately there are always some humans with prod access, or BGP editing access, or firewall write access, or access to that validation logic itself! And it requires that the humans follow the organisational procedures in place.
From what the article says, it sounds like the humans deliberately circumvented procedures, in which case, it is the engineers fault, and they should be disciplined.
It can be justified as "the instructions are out of date", "this is an urgent change", "just get it done!", etc.
Heck, sometimes following the written rules is derided as "working to rule" or "takes longer than others to complete the task".
I'm just saying that blame for these issues only sometimes belongs on the line workers.
In large org thinking, if something bad happens, and you add a rule, then it's fixed.
The problem is that this is not compatible with human psychology.
This leads to many rules, the rules are unstructured and impossible to learn, the rules probably also contradict each other.
In my experience, a rule without automatic enforcement should be the absolute last thing to depend on. If you do this, your org is broken.
The Toyota Production System with it's blame on process design instead of humans is still the best way to avoid these issues IMO.
Do 5-whys where human error is not allowed as a root cause. Design your process with poka yoke in mind. If somebody can forget something they will. Don't depend on it.
> Like other computer systems that are critical to operating flights, the FAA has imposed procedures to ensure data aren’t damaged by technicians working on them, said the person. The file or files were altered in spite of rules that prohibit those kind of changes on a live system.
> Agency officials are attempting to determine whether the two people made the changes accidentally or intentionally, and if there was any malicious intent, the person said.
It is clearly implied that had the rules been followed, the corruption wouldn't have occurred.
Your blame-the-FAA scenario where the rules were incomplete or flawed isn't ruled out, but I don't see value in asserting it without evidence. I think it's perfectly plausible that someone just cheated a bit. That's hardly a rare situation in any IT environment, and so it seems like Occam favors the article's interpretation.
If that sort of thing actually happened a lot more C level people would have been fired for over-hiring, but that rarely happens.
I don't think this is a thing that actually happens.
But the previous CEO looks like he was the spreadsheet type and rejected these kinds of ideas. So who is to blame?
(Various elements and automations of the system have been created and updated, so these could have included improvements in design.)
> […] a data file was damaged as a result of a failure to follow government procedures, […]
The more critical processes humans are expected to perform, the worse the expected results.
That goes 10x for outside or temporary “contractors” who are not going to be experts in a systems unique, esoteric, or unexpectedly irresponsible points of fragility
Second, nothing would happen at a private company because they would involve marketing and marketing wouldn’t blame anyone in a press release. The only difference with FAA is that they didn’t let a marketing department write their release.
But really the real issue here is that everyone is posting to make wide generalizations from a one time event.
Can you share an example of where that has actually happened?
I doubt that. More likely: the defect would be identified and fixed, someone would start working on a training-wheels editor for the file to ensure the problem doesn't happen again. Any change in revenues or profits would be used by the CEO to drive change in whatever thing needs changing, and the whole incident would be forgotten. Source: have owned four private companies. If everyone is fired immediately, no one learns a lesson and everyone is terrified to give leaders bad news.
The engineering precautions in system design always involve trade-offs. There is no perfect system that could be created for acceptable costs. Ultimately it is always going to come down to humans following the rules, hopefully as few of those very-trustworthy humans as possible, though that's where the FAA may have been lax.
Sec. Buttigieg apparently agrees: https://www.nbcnews.com/news/us-news/software-blamed-faa-out...
> Transportation Secretary Pete Buttigieg told NBC News that he has asked the FAA, "to make sure that there are enough safeguards built into the system that this level of disruption can't happen because of an individual person’s decision or action or mistake."
> The file or files were altered in spite of rules that prohibit those kind of changes on a live system.
Can anyone here argue that, given arbitrary write access, intimate knowledge of the system, and the ability to disable any deployment and runtime checks I want, I wouldn’t be able to take out their production system?
I'm reminded of the punchline to an old joke:
"Doctor, it hurts when I do this!"
"Don't do that."
I just realized I have never disabled cell access on my watch.
It isn't clear to me why airplane mode is requested on airplanes. It doesn't reinforce crew control, it reinforces complete lack of crew control, i.e. crew control theatre.
What should they have done in this case that didn't involve flat out preventing it?
There has to be a limit to this reasoning, though. There is a threshold where an organization is so negligent that they can’t concentrate the blame on an individual, regardless of whether there was a policy violation.
It does suggest a big lapse in oversight: if this wasn’t the first time they did this, their security monitoring should have picked it up.
My guess is that this is going to be one of those situations where someone broke the rules in a way their immediate management knew about but sanctioned because the official process was dysfunctional. In those cases it’s really common to crucify the operator and ignore the larger problem.
When SpaceShipTwo broke up mid-flight, the FAA did not place all the blame on the pilot who accidentally engaged the feathering mechanism during ascent. They thoroughly dragged the aircraft's designers for producing a design that made it so easy to do such a thing in the first place.
They weren't willing to accept, "We said not to do it in the manual," as an appropriate safeguard in that case. Why are they doing it here?
If we want to call ourselves software engineers, we need to start holding ourselves and each other to higher standards. We can do better.
But I think the term "engineer" means more to you than it does most practitioners, unfortunately.
There's a limit to individual accountability. You need professional accreditation which brings training, standards, and legal resources.
Defaulting to red tape as the mechanism to improve the quality of software is also very short-sighted. Software engineering has the absolutely unique property among all engineering disciplines that we are working with abstract systems which are subject to formal verification. We can get arbitrarily high levels of assurance, with zero need whatsoever for some stodgy accreditation body.
This is not happening much at the moment because money is (artificially) cheap, so it makes sense to hack together some crap and get it out the door as quickly as possible. There are solutions to this; slapping regulations on software is not the best one.
Typical holier-than-thou software industry arrogance and hubris. There is nothing forcing red tape regulation on all engineers nor would it for software engineers.
It's absolutely bonkers that some companies don't trust the general population of devs with 5+ years of verified experience and still gives them technical interviews. But you can't really point them to a good standard to hire with either.
Only metrics resistant to Goodhart will work here, and most of them are illegal to use in the US under the current employment law paradigm. One of the reasons software as an industry makes so much money is there are still ways to implement Goodhart-resistant hiring in the US. Going into too much detail here is somewhat hazardous, so I will leave it there.
I don't exactly believe that.
And we have other professional exams/processes that generate good professionals. Folding over and saying "Oh we can't do this, Goodhart's law exists" just seems like a weak reason to me. https://en.wikipedia.org/wiki/Perfect_is_the_enemy_of_good
https://www.cnsnews.com/commentary/hans-bader/faa-endangers-...
A corrupt file led to the FAA ground stoppage – also found in backup system - https://news.ycombinator.com/item?id=34348388 - Jan 2023 (388 comments)
US Halts Flights Nationwide After Key FAA System Goes Down - https://news.ycombinator.com/item?id=34337807 - Jan 2023 (36 comments)
FAA NOTAM System Outage - https://news.ycombinator.com/item?id=34337158 - Jan 2023 (214 comments)
The West has an addiction to foreign IT contractors. It’s a many hundreds of billions yearly industry globally, and is only possible because the largest and most critical institutions in our society refuse to invest in people to keep this knowledge in house and grow their own trained workforce.
LOL - let me show you Dubai
These people had a public trust or a clearance. They've worked their way in the fed gov. I wouldn't immediately question malice or otherwise. You don't just throw away a career with sabotage.
Also having worked in a similar environment, I can guarantee you that although there may be a test env, it is nowhere near similar to prod. I've seen it myself with different firewall rules, different GPOs, different software hardening packages. And I also complained that there should be no distinctions between stage and prod - insomuch that DNS and SSL should be the only real differences.
I'm also acutely aware if my project has a similar incident, that I too would be scapegoated and fired. It's part of the course of this job.
We had dev and prod. No version control and no testing requirements. Was a while ago for sure (i used svn locally for sanity). But from what i hear not a lot has changed. This was not flight related software. They did have better tools and procedures but not by leaps and bounds
Ive had the displeasure of having another team write the Cloudformation scripts, and they wrote them in such a way that none could be redeployed, or changed. And since they're domain joined windows machines, effectively un-redeployable.
My first start, which died in situ, was to do the work to get stage up and running, and then clone stage into prod and changing the DNS and naming bits. Since this was a no-go, ended up piecemealing everything. It still stings too. We're nearly done and ready to go live as well. Ive been pushing for live since November. But when a single person speaks out, "delays get delayed by meetings to discuss in meetings to have meetings about delays" (note: I'm not making this up)...
I shouldn't complain too much. A good 4h a workday is filled with gaming and stuff, since the enforced delays and "hurry up and waits" require me to literally do nothing. But when I do work, I'm going at the 5-10x speed I'm accustomed to going. But when it takes weeks to open up SMB from internal machines to other internal machines... there's not a lot I can do to fix broken process.
And, I'm just a lousy contractor out of hundreds of thousands.
It could literally have been a lower case letter that didn't encode into whatever archaic language they use.
> When the system began having problems Tuesday night, technicians switched to a backup. But because the backup was attempting to access the same damaged data, it also didn’t work, the person said.
Their entire backup plan was to fail over to an identical secondary system that was supposedly redundant. But, it was clearly sharing the same database system as the primary, thus they didn't actually have a fully redundant system. They had 1 system with some parts of it having a secondary, i.e. not an actual DR / failover plan.
But blockchains are specifically designed to keep on appending, no matter the mistakes of individuals, or even motivated sabotage attempts of sophisticated attackers.
Essentially, blockchains have a giant, full-coverage set of unit-tests in their shared & public consensus code – reviewed and running across many systems. And many of their designs have proven uptimes in adversarial environments for years.
And the task gets even easier when there's a small fixed set of established authorities with privileged keys.
Protecting those keys is also, of course, a hard problem. But compromised credentials is probably already a fatal threat for the legacy system(s) that were brought down by "contractors who introduced data errors". So a hypothetical modern crypto-consensus solution's failure modes remain a subset of those of the legacy system.
I am by no means an expert, however to me this system appears to include: a globally agreed upon data format, a transmission protocol hardware distributed across the planet in ground stations and air vehicles, and 70 years of legacy.
And all of that is enshrined in legislation across many countries.
And while there is a modernisation effort in progress, the nature of this system means that things will move slowly.
In 2023 anyone can suggest a better data format and distribution system if they start from scratch. Good luck getting the world to use it and good luck getting anyone to care.
The "globally agreed-upon data format" is essentially text blobs with a standard set of abbreviations, that assist search/filtering by flight-paths & planned waypoints. It seems to be a mere 150MB a year.
The inherently "move slowly" modernization effort can only be helped by a clearer outline of what's now possible. This is too important to prematurely dumb-down brainstorming ideas, based on low-expectations for the adoption process!
And a modern architecture for distributed updates & reliability could trivially mimic the old system's pilot-visible outputs, to remain familiar to tenured personnel.
Nor does distributed-consensus seem to apply in this case. For many datasets you want one source of truth (e.g. airport codes, approved abbreviations).
Crypto blockchains deal with very, very simplistic global states. Bitcoin, for example, has (a) 256 bits from the prior block, (b) a megabyte or so of transaction data. Every state machine update (mined block) totally replaces the prior state. And it crucially depends upon backtracking never being needed; for example, if a prior block's hash made it impossible for a new block to be created with enough zeroes in the hash (I don't think that's possible), then a global, manual fork would be needed.
And when something slips by, many eyes & many implementations make a consensus fix very fast. No one has 'fat-fingered' the major blockchains in a long time! Some proprietary/permissioned chains have had problems. And stuff as complicated as smart contracts can go awry in subtle ways. But NOTAM seems to be just a simple text log!
Distributed consensus inherently achieves one-source-of-truth without single-point-of-failure. As the notices relevant to individual flight-paths are inherently geo-local, with many independent submitters (like airports), any routing "through the FAA in DC" introduces risks – even if the FAA has final authority over who (which keys) can add entries.
Yes, Bitcoin's consensus state is relatively simple – but also far more compicated than NOTAM's even-simpler message. And continually redundantly verified & replicated by many independent actors & codebases, very successfully, for 10+ years.
Yes it can. That is not the question. The question is why do you think a bog standard, but correctly implemented and administered database could not handle the same?
Imagine the following: the task is to move two wheels of cheese 5 km from A to B. Your proposal is akin to someone comming and suggesting that a suborbital rocket can do the job. And you are right. It can. But so can a Toyota Corolla without breaking a sweat.
So tell me, why should we use a “hypothetical modern crypto-consensus solution” here, with all its complexity, instead of a well understood and bog standard and well implemented database?
But the centralized team running it could still "fat-finger" it. Any "master DB" system has problems that blockchain-like approaches have definitively solved, allowing progress on valid messages when large portions of the usual maintenance systems have gone down or been compromised.
Is there a major RDBMS out there that struggles with 150MB of data?
But if that's enough, why hasn't the FAA solved the problem already?
And even a "major RDBMS" is vulnerable to stodgy management leaving design in a state where it's vulnerable to a single central point of failure, where "2 contractors who introduced data errors" (!) can bring it down.
Modern highly-distributed consensus systems are not vulnerable to that. Neither accident nor malice from 2 government contractors could prevent all authorized updates worldwide in a distributed consensus system like Ethereum.
DC & northern Virginia (or wherever the FAA's systems hosted) could even spontaneously disappear into a lava pit and the official consensus set of NOTAMs would continue to update.
Hell, if you _really_ wanted to insist on an exchange data format, XML would work just fine.
That's not a backup. Sounds like they have standby hardware for one tier of the architecture which they switched to, but since it wasn't a hardware failure it didn't change anything.
One hopes they review not only their processes which allowed (one assumes) a configuration change to be applied without being verified, but also their backup strategy...
That's not a backup. Sounds like they have standby hardware for one tier of the architecture which they switched to, but since it wasn't a hardware failure it didn't change anything.
One hopes they review not only their processes which allowed (one assumes) a configuration change to be applied without being verified, but also their backup strategy...
But: data errors introduced by a bad disk? In 2022? Storing your data on a hardware-backed (or, ZFS-backed, if you must) RAID6 (or equivalent) partition has been bog-standard for the past decade or so, and a partition is either online (and OK) or degraded (but still OK), or offline, at which point 'restoring the latest backup' (or just failing over to a replica) should bring you back to an OK state.
Then: apparently, restoring the offline partition caused bad data to be ingested, over and over, crashing the system again and again. Even if we pretend that solutions like SQLite have not been available for the past decade or so, 'skip past the head of the corrupted data file' has been best-practice since I last saw abominations requiring such hacks (say: Novell MHS), and sorry to repeat myself, over a decade ago?
TL;DR: please pay me US$ 200M, and I'll make sure all your problems are at least not of the kind solved a long time ago?
Is there anyone there who heard about just culture, and root cause analysis? I can’t even believe they would publish this irresponsible stuff.
[1] https://www.usajobs.gov/job/700094300