Tangent: I hate this trend of following titles like that.
To my mind SRE != Sysadmin; SRE is a principle of tackling "Sysadmin" as if there were no sysadmins- engaging software solutions and engineering to track recurrent problems with a top-down approach, often with little understanding of high-availability in hardware or OS design.
Sysadmin is historically a role of automation and reliability, but working from the bottom up. I (and others) make sure operating systems are not exhausted and that the hardware can support various reliability metrics.
Personally, I think these roles are complementary because an auto-healing system that has a stable platform is going to be more reliable than something that is very over-engineered to deal with hardware faults as a common occurrence.
I don't think title inflation is necessary.
Don't get me started on "DevOps" engineers. It's either rebranded sysadmins doing the same thing but maybe with some CI/CD. Or Developers who have been thrown to the wolves. Hardly anyone is actually using the "there are no fullstack people, only full-stack teams" mantra.
I don't disagree with your analysis, but my (semi-serious) point was rather that most good SAs could do both, and that there might be a pay differential between SA and SRE as SRE is more popular currently.
> Don't get me started on "DevOps" engineers. It's either rebranded sysadmins doing the same thing but maybe with some CI/CD. Or Developers who have been thrown to the wolves.
True, although startup navel gazing is not as common as one would expect. Current top 10:
Swift Playgrounds for macOS (apps.apple.com)
Judge Orders Navy to Release USS Thresher Disaster Documents (usni.org)
Where are all the animated SVGs? (getmotion.io)
Stage is a minimalistic 2D, cross-platform HTML5 game engine (piqnt.com)
How the CIA used Crypto AG encryption devices to spy on countries for decades (washingtonpost.com)
N26 will be leaving the UK (n26.com)
The coming IP war over facts derived from books (abe-winter.github.io)
Growing Neural Cellular Automata: A Differentiable Model of Morphogenesis (distill.pub)
A popular self-driving car dataset is missing labels for hundreds of pedestrians (roboflow.ai)
Investigating the Performance Overhead of C++ Exceptions (pspdfkit.com)
Something about her writing style to me is offputting. I prefer more formal blogs in general. There is a sense of adventure though which obviously people find appealing.
Screwing up things is normal. One thing I started doing is that when a junior member of the team "screws up", I'd laugh it off and tell them about a major screw up of mine.
The thing is that often my screw-up as a junior was worse (short version: broke a key part of the 'boot' system, was detected friday evening, and we had a major scheduled release on monday morning) and it just puts people at ease. I'll tell it in a humorous way as well. It's important (I think) that they don't feel bad about it.
I've often had colleagues join in on the conversation as well. We're human, we'll make mistakes, no need to stress out over it.
EDIT: added a bit more explanation of _why_ I do so.
Some places use eg P1 P2 P3 instead of SEV1 SEV2 SEV3
the classification can be in terms of impact to the business. E.g. "P1" could be reserved for issues severe enough to prevent the business from functioning as a business. (E.g. the bank that cannot process customer transactions, the cdn that cannot distribute content)
P3 might mean some features of a service are broken and 1% of your customers are pissed but there is a workaround available or the features aren't really critical
Depending on who you ask, severity, site event, serious event, or something else. Rachel probably picked up the nomenclature at FB, and even there the origin of the term is kind of lost in the mists of time.
Here is a screw-up/near miss share. I was migrating a pensions service to a cloud vendor. Part of this involved a very large ETL. Practice makes perfect and we ran the custom process regularly to ensure I'd go smoothly on the big day. The last time we ran the practice I somehow managed to get my prod creds mixed up and I started to restore a week old back up over the top of production. Thankfully the first part of the process is a disk check and I realised my mistake and cancelled the job before any destructive actions happened. I was minutes away from destroying the pensions records for two FTSE 100 businesses.
I was once (partially) responsible for the deaths of dozens of virtual machines at a distance of about three and a half years.
Fun fact: none of these VMs had rebooted in that time, or they wouldn't have crashed.
Anyway, back in 2014 or so I dropped a bunch of transmit packet completions. In most cases I also double completed packets which was immediately fatal. Kernels get mad about that sort of thing.
Turns out, not all of the affected VMs died. Some of them lived on with head indices forever unequal to tail indices (until they rebooted).
In 2018 a developer realized there was a potential bug in waiting for VMs entering a quiescent state -- a truly idle networking stack had retired all Tx packets that it had admitted. Having unequal indices was impossible under correct operating conditions. They fixed the glitch.
This change rolled out gradually.
Gradually, the kernel panics appeared.
The change rolled back, halting the impact, but then the analysis began. What had we broken?
Another fun fact: Linux often includes an uptime in dmesg logs.
Slowly a pattern appeared. The dmesg logs included unusually large numbers for uptimes. Plotting these, there was a clear cliff in terms of a minimum uptime. Historical deployment logs showed a noteworthy release at that date, years past. Noteworthy in that it was rolled back for my bug, years prior.
On the plus side, I realized this was almost certainly my years prior fuckup slightly sooner than anyone else, so at least I got to call myself out :)
Haha, @rachelbythebay (whatever your HN username is), I think your writings are great and your in-the-trenches wisdom is extremely useful and interesting, but...
"Yes, I do screw up sometimes, here are a handful of anecdotes from 30 years ago to prove it" doesn't exactly ring the humility bell. :D
One of my screw up's was when I was SYSAD (head admin) for the Prime 550 at the UK office of a large consulting engineers.
We had our field engineer in doing a PM and he needed a scratch disk and I said oh you can use xxxx and pointed at the sticky label which had all the disk id's on.
Turns out that some one had been using this for a big GIS project in Amman and ended up wiping 6 month's work
I recall at my first job one of our computer rooms had the emergency stop button on the wall - just at head Hight.
One time I or My Boss (I cant recall who) stepped backwards and hit the off button with his head - we had our electrician fit a molly guard after that.
Several jobs ago, we had a datacenter where the PDUs for the racks were mounted at the very top. Normally, this wasn't an issue as this was well above head height...
One day we hired an engineer who was a Sikh. Turns out the PDUs were almost exactly at turban (dastar) height. Cue the outage alerts (and the installation of mollyguards).
I admire that the author actually responded to some of criticism, accepted it and took it in stride. It's something I find myself having difficulties with more often than I'd like.
However, "even in turds you can sometimes find a peanut". I mean, come on...
It's definitely off-putting. I'm not sure what your mean by not reading it "too literally". Obviously no one thinks the author is speaking literally... It's still reasonable to think the phrasing is gross.
What is your criticism exactly? You quoted the article and said simply, "come on".
IMO this is only validating the criticism the article levels at comment sections like HN's. You have picked out some random sentence and expressed no more than idle disagreement. Maybe I personally wouldn't compare your comment to a turd, but there's not a whole lot of nutritional value in it either.
Perhaps the reason the article expresses this concern in this specific way is because it is warranted. Because people insist on disassembling articles coming from this domain sentence by sentence and posting comments that really don't say anything helpful or sometimes anything at all.
Ok... Do you have any evidence of me (or the OP commenter) doing that? Rachel is one of my favorite tech bloggers, and has been in my RSS feed for years.
At least it's clear that this is just you White-Knighting....
One of my favorite job interview questions for sysadmins is asking about a time that they screwed up and broke production. If they don't have one, then it makes me nervous. Either they are lying, or they don't have enough experience, or they will be too conservative and will block all progress.
"ifconfig eth0 down" on the production bastion host, instead of on my localhost terminal -- and no hands on in the datacenter which was 160km away. Of course the bastion host was the only one not hooked up to remote power reset services.. and only 2 hours left in the service window.. sinking feeling
Oh, yes, that feeling when you mix up “init 5” and “init 6” on a machine 300 miles away. “Huh, it’s taking longer than usual to reboot...”. 20 years later and I still remember the name of the guy I had to call at two in the morning to go power it back on.
Back in the 1990s, something like that lead to a major local data center having to admit that the “24 hour support” meant 12 hours a day with someone who could answer the phone but didn’t have access to the server room.
I've done that a couple of times, although in better circumstances; also fun variants like "the new kernel didn't have the right ethernet driver in it".
I think the first one in my career was discovering that "killall" does something very different on Solaris from Linux.
For tmux users: put something like this in your .tmux.conf on production servers:
set -g window-style 'fg=red,bg=black'
It will color the text red, hopefully reminding you to be extra careful. Adjust according to your preferences.
One other "defensive scripting" trick I frequently use is starting any `rm` command with `ls`, double checking its output (or triple checking if it's a recursive one), and then replacing `ls` with `rm`. It barely takes any extra time if you're proficient with emacs-style readline hotkeys:
Something like this to obviously distinguish environments is good practice - at one company we implemented scripts for terminal color-coding like this after some downtime caused by a destructive backup restore accidentally being run in the production environment instead of an acceptance testing system, which was in all aspects identical to the production system.
And I've seen a solution for more secure environments where physical separation was used with the operator having separate monitors/keyboards, and the "important" system having a different color keyboard and monitor frame.
I once ran a script on production to re-push some old data for a customer based on log entries. This script used the log timestamps to decide which data to re-push. Didn't realize that the timestamps in the log files were UTC, and I just ran it with the default timestamp provided by the library (which is the one the host system uses). Lucky for me, the system's default timezone was also UTC, but nonetheless, the moment I realized it and the 10 minutes it took me to read the documentation and to check the host system's timezone felt like hours.
Remember when Gitlab had their famous DB incident? From that we had some sort of an inside joke in my then-workplace. If you're gonna do something big and potentially prod breaking just "don't be _that_ guy" (said in the same spirit as "break a leg").
I became _that_ guy.
My then-workplace didn't always have enough funds, though as an employer they were generally generous especially considering their actual finances. This is relevant to the story because this employer:
1. was very lenient when it came to office attendance. So we frequently worked remotely in odd hours; that was normal. But as a matter of professionalism, I always tried to be conscientous when it came to the hours I put in. Most weeks I probably did more than usual, merits of which is another discussion entirely.
2. periodically organized events to promote the business. But being short on funds, they didn't have money to hire an actual photographer. So they'd ask me to shoot because I was interested enough in photography to, at the very least, have the gear for it.
The day I became _that_ guy they had this event I'm supposed to shoot but they really communicated the time badly to me. I expected to be able to do at least three, maybe four, hours of work before I'm needed with my camera. This is what I communicated to my TL.
Turns out they needed me _earlier_, such that I only had an hour of work done so far. Again, office culture was lenient about such things so my TL didn't really mind if I left then. The event was some kind of a big deal besides.
I'd generally start my "hours" in the afternoon, way after lunch. So by the time this event was done, it was already pretty late in the evening. I had my dinner and received a message from my TL. Nonverbatim:
"Hey can you update PostgreSQL (9->10) tonight? It shouldn't take too long and here's the steps..."
It was still in to my "usual" working hours but a couple of things that night made this request result to disaster:
1. I was tired from the event. Honest to goodness tired. I should've called it off when I couldn't even entertain myself enough to stay awake waiting for one of the given steps to finish. But I didn't because...
2. I didn't have the heart to beg off on this task when I've only done one hour of technical/engineering work for the day. To be fair, my TL always abided by the rule "Don't touch prod when tired; you will make things worse". Pretty sure he would've understood if I explained the state I was in. We could've done it the next night. But when you're tired and embarassed at having only done one hour of work for the day so far your decision making is exceptionally unsound, for lack of a stronger adjective.
Unfortunately the technical bits of this story gets fuzzy; it's been two years ago. But two years ago we have just migrated to Kubernetes and a couple of months in the team was still adjusting their mental models from servers to containers/deployments/statefulsets/pods. From just thinking between HDD vs SSD tradeoffs to Persistent Volume architecture issues. This is also why upgrading Postgres was such an ad hoc process for us then. We simply didn't know better (if something not "ad hoc" even exists).
Part of the instructions was to "delete the old data directory of Postgres" (cue: I have read this in a postmortem before...). Because I was tired and lazy I wrote a script so the update could go without my (much needed!) supervision. The instructions were sound and deletion would've been safe--assuming all the steps prior to the deletion finished successfully. It did not and I did not use `set -e`. Which meant I just deleted all the prod data in master. I was efficient. The realization woke me up harder than sugar ever did.
To cut this already long story short, I at least had the sense to concede at that point and wake up my TL with the bad news. Mu...
I've made mistakes, and even big ones, but I've never worked for an (technical) employee abusive enough that I thought they were going to fire me for a single mistake.
It's not that they don't fire for mistakes, it's that they work with you to correct your behavior first and they give plenty of warning to someone who is in danger of that.
And I've never been given that warning.
edit: Added the word "technical". I've worked for companies that would fire at the drop of a hat, but they were all retail minimum-wage jobs.
Hmm. I'm not sure I've ever had a screw up I thought would get me fired and I've been in tech for 13 years. I've definitely had some pretty big screw ups (turned the lights off for over 200 people, see my reply to parent). I guess I've always had managers who had my back.
The younger you are when you commit the mistake, generally the more fearful you are of the result of that mistake. That same mistake might get a "well that sucks" from someone who's been around the block.
Tech for 12 years and I've never made a mistake disastrous enough to be fearful for my job. The worst costing ~$20k in hardware (couple server CPUs). Told my manager right after without hesitation (this was also at a startup).
I would not stress that much anyway now if it were to occur. Having been through mass layoffs from startups twice before, you change and become hardier. I will be careful but will never be fearful of employment. Short of doing a Desk Pop[0], I'm falling asleep every night with both eyes closed. Life is too short as is. Let me go and I will spend my next morning on a nearby beach with a good book.
Just curious, how did you destroy the CPUs? Was it physical maintenance or did you set the wrong voltage/etc (usually this happens during overclocking in consumer-grade machines which is why I’m curious how this happens on server machines).
I've made mistakes, though thankfully no huge outage-causing ones. The only "I'm going to get fired" mistake was a political one, where I accidentally emailed a report to someone, and was informed after the fact that it was a very bad thing.
It's also a fallacy to believe that making mistakes in the past and learning from them precludes making many novel mistakes in the future. Kind of like the disclaimer that's always given about investing. Both the individual and the organization must simply bake in as many failsafes in their processes as possible. And even then, sometimes Murphy's Law manages to slip through all the holes at once.
Adam Savage and Matt Parker recently had a conversation that spent a lot of time covering the topic of "screwing up" and how we should respond when we do (Matt's new book is about math screw ups that have had real world consequences). It's a great interview in general, in my opinion.
I ran a sql script that migrated a database, it all looked as if it had worked perfectly, but the category Id's had changed. The main website handled this fine but I found out a separate system that sent out daily offers by email proudly advertised Domestos Bleach as the drink of the day.
I find the premise[1] of this post amusing. The sort of comment "You always criticize others, but what about you?" seems to nearly always signal emotional damage on the part of the commentator who felt blamed for something. And that is nearly always a sign of bad management practice.
People don't always get things right, they screw up, they do stupid things for good reasons, and sometimes good things for stupid reasons. As a manager I always want folks to be observant and thoughtful, and try to keep such things not about "who" screwed up but how that screw up came to be (the good or stupid reasons) and how one might think about the action ahead of time that would alert you to the potential problem that would result from a given action.
And the key of all that is making the discussion about how to think so you don't have the problem in the first place, rather than making it a blame-fest on some hapless engineer who chose poorly.
I was fortunate to have a manager early in my career who was very proactive at solving problems and moving forward, not affixing blame. He would say "Ignorance is the natural state before learning, only if it persists in the presence of learning opportunities does it become a problem."
I've always tried to learn by what I observe and what I do, which is why I enjoy Rachel's stories of finding root causes. They teach the principles that needed to be understood prior to the action. All without experiencing the feeling of dread that you've just taken production off line :-).
[1] That being that commenters feeling badly that the author doesn't seem to show their own flaws in the stories.
That's something I gotta ask on my way to my next job. My company is extremely conservative in every meaning of the word, and is not unlike patio11's description of Japanese megacorps in mannerisms.
"What happened the last time production went down?" should produce a quite illuminating answer. Do they go through a detailed root-cause analysis? Do they answer with marketing-speak meant for a legally minimal disclosure? Do they blame "that moron" whom I'm meant to replace?
As it stands here, the official corporate policy is everything happens perfectly until someone who shouldn't be there messes things up, and the problem is best solved with a public and angry firing letter. Quoth our business partner: "We're allowed to change our minds, but you're not allowed to be in error. Even if we give you bad data, you're expected to infer proper data and give us proper output. BTW we're not paying for testing"
157 comments
[ 2.1 ms ] story [ 228 ms ] threadTo my mind SRE != Sysadmin; SRE is a principle of tackling "Sysadmin" as if there were no sysadmins- engaging software solutions and engineering to track recurrent problems with a top-down approach, often with little understanding of high-availability in hardware or OS design.
Sysadmin is historically a role of automation and reliability, but working from the bottom up. I (and others) make sure operating systems are not exhausted and that the hardware can support various reliability metrics.
Personally, I think these roles are complementary because an auto-healing system that has a stable platform is going to be more reliable than something that is very over-engineered to deal with hardware faults as a common occurrence.
I don't think title inflation is necessary.
Don't get me started on "DevOps" engineers. It's either rebranded sysadmins doing the same thing but maybe with some CI/CD. Or Developers who have been thrown to the wolves. Hardly anyone is actually using the "there are no fullstack people, only full-stack teams" mantra.
I just lament the truth of your statement. :(
This made me laugh! This is so true.
There is some truth in advertising!
I feel like so many of the posts I read and enjoy could lead with that statement.
The thing is that often my screw-up as a junior was worse (short version: broke a key part of the 'boot' system, was detected friday evening, and we had a major scheduled release on monday morning) and it just puts people at ease. I'll tell it in a humorous way as well. It's important (I think) that they don't feel bad about it.
I've often had colleagues join in on the conversation as well. We're human, we'll make mistakes, no need to stress out over it.
EDIT: added a bit more explanation of _why_ I do so.
Other places may use similar terminology but OP is at FB.
She's not worked there for nearly 2 years now: https://rachelbythebay.com/w/2018/03/10/free/
the classification can be in terms of impact to the business. E.g. "P1" could be reserved for issues severe enough to prevent the business from functioning as a business. (E.g. the bank that cannot process customer transactions, the cdn that cannot distribute content)
P3 might mean some features of a service are broken and 1% of your customers are pissed but there is a workaround available or the features aren't really critical
Screw ups lead to uncertainty and research suggests we learn best in uncertainty - https://www.aau.edu/research-scholarship/featured-research-t...
Everyone makes mistakes! It's how we learn :-)
Fun fact: none of these VMs had rebooted in that time, or they wouldn't have crashed.
Anyway, back in 2014 or so I dropped a bunch of transmit packet completions. In most cases I also double completed packets which was immediately fatal. Kernels get mad about that sort of thing.
Turns out, not all of the affected VMs died. Some of them lived on with head indices forever unequal to tail indices (until they rebooted).
In 2018 a developer realized there was a potential bug in waiting for VMs entering a quiescent state -- a truly idle networking stack had retired all Tx packets that it had admitted. Having unequal indices was impossible under correct operating conditions. They fixed the glitch.
This change rolled out gradually.
Gradually, the kernel panics appeared.
The change rolled back, halting the impact, but then the analysis began. What had we broken?
Another fun fact: Linux often includes an uptime in dmesg logs.
Slowly a pattern appeared. The dmesg logs included unusually large numbers for uptimes. Plotting these, there was a clear cliff in terms of a minimum uptime. Historical deployment logs showed a noteworthy release at that date, years past. Noteworthy in that it was rolled back for my bug, years prior.
On the plus side, I realized this was almost certainly my years prior fuckup slightly sooner than anyone else, so at least I got to call myself out :)
"Yes, I do screw up sometimes, here are a handful of anecdotes from 30 years ago to prove it" doesn't exactly ring the humility bell. :D
We had our field engineer in doing a PM and he needed a scratch disk and I said oh you can use xxxx and pointed at the sticky label which had all the disk id's on.
Turns out that some one had been using this for a big GIS project in Amman and ended up wiping 6 month's work
http://www.jaruzel.com/blog/dont-screw-up-a-vaxcluster-tale
that's what differentiate a good engineer from not-so-good - they learn on own mistakes!
One time I or My Boss (I cant recall who) stepped backwards and hit the off button with his head - we had our electrician fit a molly guard after that.
One day we hired an engineer who was a Sikh. Turns out the PDUs were almost exactly at turban (dastar) height. Cue the outage alerts (and the installation of mollyguards).
However, "even in turds you can sometimes find a peanut". I mean, come on...
I'm confused about why folks seem so upset by people expressing this opinion .
IMO this is only validating the criticism the article levels at comment sections like HN's. You have picked out some random sentence and expressed no more than idle disagreement. Maybe I personally wouldn't compare your comment to a turd, but there's not a whole lot of nutritional value in it either.
Perhaps the reason the article expresses this concern in this specific way is because it is warranted. Because people insist on disassembling articles coming from this domain sentence by sentence and posting comments that really don't say anything helpful or sometimes anything at all.
1. unpallatable 2. indiscriminantly rude towards an entire community?
> Because people insist on disassembling articles coming from this domain sentence by sentence
I feel like I may have walked into something where I don't have much context. I'm not sure what you mean by that.
Also, I find it strange people are so fixated on my criticism, and nobody has commented anything about the praise I made in the very same post.
At least it's clear that this is just you White-Knighting....
Hardly an unreasonable description of hackernews comments.
I think the first one in my career was discovering that "killall" does something very different on Solaris from Linux.
One other "defensive scripting" trick I frequently use is starting any `rm` command with `ls`, double checking its output (or triple checking if it's a recursive one), and then replacing `ls` with `rm`. It barely takes any extra time if you're proficient with emacs-style readline hotkeys:
I do the tmux color trick too-- color coded by environment for each bastion.
And I've seen a solution for more secure environments where physical separation was used with the operator having separate monitors/keyboards, and the "important" system having a different color keyboard and monitor frame.
You live and you learn, I'd say :)
I became _that_ guy.
My then-workplace didn't always have enough funds, though as an employer they were generally generous especially considering their actual finances. This is relevant to the story because this employer:
1. was very lenient when it came to office attendance. So we frequently worked remotely in odd hours; that was normal. But as a matter of professionalism, I always tried to be conscientous when it came to the hours I put in. Most weeks I probably did more than usual, merits of which is another discussion entirely.
2. periodically organized events to promote the business. But being short on funds, they didn't have money to hire an actual photographer. So they'd ask me to shoot because I was interested enough in photography to, at the very least, have the gear for it.
The day I became _that_ guy they had this event I'm supposed to shoot but they really communicated the time badly to me. I expected to be able to do at least three, maybe four, hours of work before I'm needed with my camera. This is what I communicated to my TL.
Turns out they needed me _earlier_, such that I only had an hour of work done so far. Again, office culture was lenient about such things so my TL didn't really mind if I left then. The event was some kind of a big deal besides.
I'd generally start my "hours" in the afternoon, way after lunch. So by the time this event was done, it was already pretty late in the evening. I had my dinner and received a message from my TL. Nonverbatim:
"Hey can you update PostgreSQL (9->10) tonight? It shouldn't take too long and here's the steps..."
It was still in to my "usual" working hours but a couple of things that night made this request result to disaster:
1. I was tired from the event. Honest to goodness tired. I should've called it off when I couldn't even entertain myself enough to stay awake waiting for one of the given steps to finish. But I didn't because...
2. I didn't have the heart to beg off on this task when I've only done one hour of technical/engineering work for the day. To be fair, my TL always abided by the rule "Don't touch prod when tired; you will make things worse". Pretty sure he would've understood if I explained the state I was in. We could've done it the next night. But when you're tired and embarassed at having only done one hour of work for the day so far your decision making is exceptionally unsound, for lack of a stronger adjective.
Unfortunately the technical bits of this story gets fuzzy; it's been two years ago. But two years ago we have just migrated to Kubernetes and a couple of months in the team was still adjusting their mental models from servers to containers/deployments/statefulsets/pods. From just thinking between HDD vs SSD tradeoffs to Persistent Volume architecture issues. This is also why upgrading Postgres was such an ad hoc process for us then. We simply didn't know better (if something not "ad hoc" even exists).
Part of the instructions was to "delete the old data directory of Postgres" (cue: I have read this in a postmortem before...). Because I was tired and lazy I wrote a script so the update could go without my (much needed!) supervision. The instructions were sound and deletion would've been safe--assuming all the steps prior to the deletion finished successfully. It did not and I did not use `set -e`. Which meant I just deleted all the prod data in master. I was efficient. The realization woke me up harder than sugar ever did.
To cut this already long story short, I at least had the sense to concede at that point and wake up my TL with the bad news. Mu...
Tell me about a time you made a mistake that you thought was going to get you fired.
1. Everyone has one. If you don't, you haven't been doing this long enough and I want you to make a couple of those mistakes elsewhere first.
2. If you didn't learn anything from it, you're going to make that and bigger mistakes in your hubris. I'd rather you do that elsewhere.
It's not that they don't fire for mistakes, it's that they work with you to correct your behavior first and they give plenty of warning to someone who is in danger of that.
And I've never been given that warning.
edit: Added the word "technical". I've worked for companies that would fire at the drop of a hat, but they were all retail minimum-wage jobs.
Tech for 12 years and I've never made a mistake disastrous enough to be fearful for my job. The worst costing ~$20k in hardware (couple server CPUs). Told my manager right after without hesitation (this was also at a startup).
I would not stress that much anyway now if it were to occur. Having been through mass layoffs from startups twice before, you change and become hardier. I will be careful but will never be fearful of employment. Short of doing a Desk Pop[0], I'm falling asleep every night with both eyes closed. Life is too short as is. Let me go and I will spend my next morning on a nearby beach with a good book.
[0] https://www.urbandictionary.com/define.php?term=Desk%20Pop
https://youtu.be/ig-2xlXfex4
People don't always get things right, they screw up, they do stupid things for good reasons, and sometimes good things for stupid reasons. As a manager I always want folks to be observant and thoughtful, and try to keep such things not about "who" screwed up but how that screw up came to be (the good or stupid reasons) and how one might think about the action ahead of time that would alert you to the potential problem that would result from a given action.
And the key of all that is making the discussion about how to think so you don't have the problem in the first place, rather than making it a blame-fest on some hapless engineer who chose poorly.
I was fortunate to have a manager early in my career who was very proactive at solving problems and moving forward, not affixing blame. He would say "Ignorance is the natural state before learning, only if it persists in the presence of learning opportunities does it become a problem."
I've always tried to learn by what I observe and what I do, which is why I enjoy Rachel's stories of finding root causes. They teach the principles that needed to be understood prior to the action. All without experiencing the feeling of dread that you've just taken production off line :-).
[1] That being that commenters feeling badly that the author doesn't seem to show their own flaws in the stories.
"What happened the last time production went down?" should produce a quite illuminating answer. Do they go through a detailed root-cause analysis? Do they answer with marketing-speak meant for a legally minimal disclosure? Do they blame "that moron" whom I'm meant to replace?
As it stands here, the official corporate policy is everything happens perfectly until someone who shouldn't be there messes things up, and the problem is best solved with a public and angry firing letter. Quoth our business partner: "We're allowed to change our minds, but you're not allowed to be in error. Even if we give you bad data, you're expected to infer proper data and give us proper output. BTW we're not paying for testing"