The power drill mention in the headline is a bit click-baity because in the end while a power drill was used it was unnecessary and was not the solution to the problem. Had they known how to properly use the hardware security devices they had the power drill wouldn't have been deployed at all.
> "At this point, the engineers in Australia decided that a brute-force approach to their safe problem was warranted and applied a power drill to the task. An hour later, the safe was open—but even the newly retrieved cards triggered the same error message."
What happened here (from what I recall) was far funnier than this does it credit.
The SREs first attempted to use a mallet (hammer) on the safe (which they had to first buy from the local hardware store - don't worry it got expensed later), then after multiple rounds of "persuasion" they eventually called in a professional (aka. a locksmith) who used a drill+crowbar to finally liberate the keycard.
The postmortem had fun step by step photos of the safe in various stages of disassembly.
What is this, sitcom slapstick? The slapstick of storing the security combination to the safe on the system that is locked by the card which inside the safe; and the slapstick of "You're inserting it wrong"...
Real life sometimes competes with the stupidity of TV sitcom humor. A friend just complained that his wife lost access to her computer, even though she kept the password on a sticky note. Excuse me, a Sticky Note(tm). The app. On the computer itself.
> It took an additional hour for the team to realize that the green light on the smart card reader did not, in fact, indicate that the card had been inserted correctly.
I'm not sure which is worse: bad UI/UX use of lights, or inadequately trained engineers who misunderstood the lights.
I would say of all companies that have great SRE, I would not have expected Google to be one of them were this process was so brutaly flawed:
- Storing the safes password - which is required for the password manager to start - ... in this very same password manager?
- Failing at trying to insert the card in multiple ways into the card reader (it's like USB, you're using it the wrong way around). I would have tried that before (while?) drilling the safe.
- Having no clue (no documentation) how to restart the service, despite it having passwords in it? If passwords are lost, all encrypted stuff is lost, forever.
If there's one thing I think is central to document personal or corporate), it is how to get accesss to passwords _fast and reliable_ whenever there's a disaster recovery.
Sorry for the offtopic comment, but it's bizarre to me that Google is hosting their book on Github with a github.io domain. Their previous two SRE books are hosted at https://sre.google on Google-owned IPs.[0]
What was that decision process? "We're Google, and we're literally writing a book about how good we are at hosting services. But hosting some static HTML files that are almost entirely text? That's a tough one. We'd better outsource that to one of our competitors."
sre.google links to this book, but it links to github.io. The other two books linked on sre.google point to within the sre.google domain, so this is the odd one out.
> restart required a hardware security module (HSM) smart card.
Out of curiosity, does anyone know why? My guess would be the PW DB would be encrypted with some token generated from this card.
I've had lots of "I have a secret and the server needs it" type problems but I've never been very happy with my solutions- smart cards seem like potentially an elegant solution.
It's mostly a rather dense engineering textbook, but it contains lots of things I found insightful. I most particularly remember a segment like "We make things more reliable by adding more layers of Swiss cheese, on the assumption that failure modes are uncorrelated and it's only when all the failures take place that the system breaks. But this doesn't work when the system is being attacked by an intelligence, because an intelligence will explicitly correlate failures."
The book is very much designed for Google-scale systems, though: everything is assumed to be microservices, for example.
I don't know anything about Google but I glean this Password Manager service was of low importance and was shared by employees. I'm thinking this would've been a non issue with a low tech solution like a shared document of passwords and services or a wiki page, and by virtue of being hosted on a more common platform would benefit from a better SLA.
17 comments
[ 3.1 ms ] story [ 38.2 ms ] threadWhat happened here (from what I recall) was far funnier than this does it credit.
The SREs first attempted to use a mallet (hammer) on the safe (which they had to first buy from the local hardware store - don't worry it got expensed later), then after multiple rounds of "persuasion" they eventually called in a professional (aka. a locksmith) who used a drill+crowbar to finally liberate the keycard.
The postmortem had fun step by step photos of the safe in various stages of disassembly.
I'm not sure which is worse: bad UI/UX use of lights, or inadequately trained engineers who misunderstood the lights.
- Storing the safes password - which is required for the password manager to start - ... in this very same password manager? - Failing at trying to insert the card in multiple ways into the card reader (it's like USB, you're using it the wrong way around). I would have tried that before (while?) drilling the safe. - Having no clue (no documentation) how to restart the service, despite it having passwords in it? If passwords are lost, all encrypted stuff is lost, forever.
If there's one thing I think is central to document personal or corporate), it is how to get accesss to passwords _fast and reliable_ whenever there's a disaster recovery.
What was that decision process? "We're Google, and we're literally writing a book about how good we are at hosting services. But hosting some static HTML files that are almost entirely text? That's a tough one. We'd better outsource that to one of our competitors."
[0] https://sre.google/books/
It seems this github.io URL is more like a CI run of the book, and the one on sre.google is the "published" one.
Out of curiosity, does anyone know why? My guess would be the PW DB would be encrypted with some token generated from this card.
I've had lots of "I have a secret and the server needs it" type problems but I've never been very happy with my solutions- smart cards seem like potentially an elegant solution.
The book is very much designed for Google-scale systems, though: everything is assumed to be microservices, for example.
I don't know anything about Google but I glean this Password Manager service was of low importance and was shared by employees. I'm thinking this would've been a non issue with a low tech solution like a shared document of passwords and services or a wiki page, and by virtue of being hosted on a more common platform would benefit from a better SLA.
Not too long ago here on HN, a user was saying they were locked out of their accounts because A required B, but B required C, and C required A.