Calling the anonymous S3 Team Member the "root cause" is both insulting and untrue. System designs must assume humans are fallible.
The Amazon post itself[0] identifies several other contributing factors beyond "Sysop Error". Root cause analysis would ask the following questions:
Why is a potentially risky operation being done manually in the first place?
Why was the human error of restarting a larger-than-usual percentage of servers not flagged for user confirmation before continuing?
Why was a restart of the indexing subsystem and the placement subsystem never tested in the past several years?
etcetera, etcetera.
In short, there were significant process failures that led to "S3 Team Member" even being in a position to accidentally cause this outage.
Perhaps whoever is behind this "light-hearted" pledge drive can't see beyond the human factor because they don't have experience at anything close to this scale, and systems many orders of magnitude smaller simply can't justify the sort of investment in extra belts and suspenders that is done on larger systems.
Any of the big hosts, including Amazon, invests quite a lot in infrastructure and tooling precisely to prevent or at least mitigate this type of error, but this particular corner of their system didn't have enough protections in place nor was the recovery process tested.
Those process failures are not "S3 Team Member"'s fault. At. All.
As far as I am concerned, this was a bug, and "S3 Team Member" should get a prize for finding it, however inadvertently. May I suggest some extra vacation time?
7 comments
[ 3.1 ms ] story [ 34.3 ms ] threadI remember removing a 10gig port channel (by a typo) on a regional switch. That was in 2012. The horror.
I think they might just prefer to stay anonymous.
Sounds a lot like a scam.
The Amazon post itself[0] identifies several other contributing factors beyond "Sysop Error". Root cause analysis would ask the following questions:
Why is a potentially risky operation being done manually in the first place?
Why was the human error of restarting a larger-than-usual percentage of servers not flagged for user confirmation before continuing?
Why was a restart of the indexing subsystem and the placement subsystem never tested in the past several years?
etcetera, etcetera.
In short, there were significant process failures that led to "S3 Team Member" even being in a position to accidentally cause this outage.
Perhaps whoever is behind this "light-hearted" pledge drive can't see beyond the human factor because they don't have experience at anything close to this scale, and systems many orders of magnitude smaller simply can't justify the sort of investment in extra belts and suspenders that is done on larger systems.
Any of the big hosts, including Amazon, invests quite a lot in infrastructure and tooling precisely to prevent or at least mitigate this type of error, but this particular corner of their system didn't have enough protections in place nor was the recovery process tested.
Those process failures are not "S3 Team Member"'s fault. At. All.
As far as I am concerned, this was a bug, and "S3 Team Member" should get a prize for finding it, however inadvertently. May I suggest some extra vacation time?
[0] https://aws.amazon.com/message/41926/