Apologies if this is not quite on topic, but this definition of shame really spoke to me:
“When a person feels that he or she is becoming his or her undesired or feared self.”
I feel like a lot of my motivation to better myself and help improve my environment comes from that feeling. It's as if i have a very clear picture of who/what i dont want to become, but not so clear on what the "ideal me" would be.
Is this something that motivates HN as well or do you try to realize an ideal(?) version of you through your thoughts/actions
I could probably write one, but writing books is hard. I am still smarting from the time I wrote a 400-page book, that was out of date before it could be published.
"Vague" systemic exploration is interesting, having live prototypes to modify and refine gradually, to gain knowledge and maybe readjust the substructures.
It’s a social media-like app, based on aggregate servers. I wrote the servers, myself, years ago. One is currently a world standard, and heavily in use. The other, was a server I wrote a few years ago as a “practice” project.
The app, itself, has been under development for over a year.
A big reason for that, is the team had only a vague idea of what it would do. It wasn’t really feasible to get them to hash out a more detailed plan, so I set up a “breadboard” system, based on Apple’s TestFlight beta-test system [0]. TestFlight is usually only engaged towards the very end of a project, but we’ve been using it since one month after my initial checkin.
TestFlight requires a fairly “complete” app, that runs without tripping any alarms (like crashing). Apple vets each release.
I can produce a fairly full-featured app, quite rapidly, so I was able to meet Apple’s quality bar, pretty quickly.
Because we are using TestFlight, we have some significant advantages:
1. The app is constantly at “beta” quality. Even with incomplete functionality, what’s there, works well. I cannot overstate the importance of this.
2. We can have non-technical stakeholders constantly “test driving” the UI, and providing relevant, useful feedback, on a continuous basis. They are also highly motivated and engaged.
3. Because Apple is constantly vetting the app, we know that we probably won’t have any nasty surprises, when we finally release.
During this year, we have explored a number of avenues that have resulted in “dead ends.” I’ve tossed out months’ worth of work.
That’s great. The best code I write, is the code I don’t write. I’m happy to tear out huge chunks of code.
The result is that we now have a clear, robust, ultra-high-quality app that is in the final stages of development. I still have a couple of areas of functionality to add, but we’re done prevaricating. We know where we’re going, and everyone is on board and psyched.
The quality of the app is also, quite frankly, jaw-dropping. We’re all thrilled to death.
Using TestFlight has allowed us to work through a lot of stuff that is often encountered during early MVP stages; without the brand damage and public humiliation.
TDD has some excellent philosophy behind it, but I struggle to implement it because I get discouraged by failure, and move on to other topics.
If the TDD approach said "thank you" for the good parts (code structure, not breaking 500 other test cases), that would make the 13 errors less scary. One of the things I like about software is that it's "soft" - it should be possible to make changes without the fear of getting it wrong, and be able to fix them now while I see the problem, rather than procrastinating until later when I've forgotten about it.
The same "thank you" system applies when working with external contractors (e.g. language translations). We can assume they tried their best, and therefore might not know what else to say. Reporting 19 problems will get us a reputation for being nitpicky, hostile, even toxic. Rather, we should thank them for the 730 that they got right, and suggest improvements for the 19 that we think might be better.
With thankfulness and encouragement, TDD could really change the industry for the better.
My deep seated fear is that some fundamental infrastructure will fail, Database, dns, DDoS, there are so many things that can go wrong. Esp with big old code bases.
Many developers really dont care what happens in production, leaving a subset to stress out. Perhaps the solution is just be like the people who just expect problems and accept it.
I hear you, it isn't great when people are apathetic to what happens in prod, especially when there are things that can be done.
That being said, here's another way to view it: of all of those deep seated fears you have, what do you know you can do to avoid certain issues and what can you actually not do?
Let's take the database for example. What's the deep seated fear here? Maybe:
> The disks crash, the data center bursts into flames and all the data is gone!
A solution to those is perform backups and get the backups off-prem. You have to explain the risk and justifications to the company as best as you can. Once you've done that, what is left?
Technically speaking, bugs in the backup script could exist, or maybe the backups get corrupted. Beyond that, there are the human ones. Maybe somebody else manages the script and breaks it, or maybe backups get unintentionally poisoned by malware getting into the backups. Or maybe even before that happens the company goes "eh, risk worth taking to save costs!" and you don't even get to implement the backups.
Here's the thing: these things are _not_ in your control at this point! They are out of your hands, and with humans like us who make mistakes like us. So problems are going to be expected, especially ones you don't realize right now. But they're not in your control! And yet, you did your part, and things will continue on.
As someone with a background in infrastructure operations, it always felt like we were the ones who cared and stressed out, and development didn't (have to) care.
These days, particularly in microservices, more and more developers are on call for their work, so they care more.
Still, I find a radical misalignment between my mentality, where you CANNOT make mistakes because you're working without a net, and development, where typically there are more layers of safety. I'm far, far more cautious and anxious about making changes and breaking things; they've historically been paid to produce. If I try to have a conversation about it, they're unable to grok that their threshold for the acceptability of making mistakes is much higher than mine is.
You NEED to be able to make mistakes in order to be able to produce, but it's a hard mentality shift.
Yeah my sympathies, its tougher than it used to be. Wish we had you on our team, our first level support doesn't do anything except bounce processes then escalate. :)
Someone who never makes mistakes or someone who deals with their mistakes openly and directly?
We know the first one doesn't exist. Some people like to make us believe that, but it's bullshit.
However, one big caveat: Owning mistakes, being open about them doesn't absolve us from making them, especially not in the future. We still _made_ the mistake. Should we dwell on that or get punished? Likely not. Severity, intention and so on matter too. But should we simply not feel bad about it? "Embrace mistakes"? Not so sure about that either.
When we say "embrace mistakes", what we really mean is "embrace learning". Mistakes are opportunities to learn for ourselves and others indirectly.
We shouldn't be victims of our bad feelings and beat ourselves up. I did that for years. It crushed my confidence and well-being into submission and years after I'm still recovering from that. But the solution is not to try and make us feel good about ourselves regardless. It's not _just_ a psychological problem.
Mistakes happen for a reason, even if they are as simple as "I didn't write that down". They happen all the time, we can't prevent that, but what we can do is decide what kind of mistakes we want to face and ultimately what kind of problems we want to solve, be able to solve - for us and others.
Don't get bogged down, don't beat yourself up and don't let others beat you up for your mistakes. But make yourself aware that you can actually learn from them and decide for yourself when you want or need to put the energy into doing that.
I get why people are saying things like "normalize mistakes" etc, living in some kind of police state environment that punishes every misstep is just miserable. Especially if you're working in software, where "no mistakes" is not a reasonable standard.
OTOH it shouldn't become an excuse for complacency, and especially not an excuse to tolerate incompetence and/or totally unforced errors.
And sometimes even if you are an expert, you may still make a mistake if you aren't in an optimal frame of mind - for example, if you didn't sleep or eat properly or had an altercation with someone, or your normal routine got messed up ... sometimes it is just that - a shitty day in your life with nothing to learn but to just go through it.
I have a friend that used to drive trucks for mobsters, in New York.
He tells me a story about how he drove his truck under a low bridge, and did about $20,000 worth of damage to it.
When he got back, he was expecting to get fired (or worse).
Instead, his boss asked him what he'd learned from the mistake, and gave him his next assignment. When my friend asked (incredulously) why he didn't get fired, his boss answered "I just spent twenty thousand bucks, training you to be careful of clearance. You think I'm gonna toss that out?"
My friend says he learned a lot about management from these goombahs. They were managing very dangerous people, who were big investments, and couldn't be treated badly or disrespectfully.
Maybe he read the book, or maybe this was a fairly common practice, amongst a certain demographic (it was in the early 1990s), and Horowitz heard about it.
No, this guy really experienced it. It's likely his boss had a cultural introduction to the technique. I actually know a number of folks [formerly] from that subculture (I hang out with some ... interesting ... people).
I'm sure it didn't cost him too much. He used 20K as a nice, fat number, but they probably only needed to shell out a couple hundred bucks out of pocket. It was an easy act to do. Impressed my friend, but he didn't really think about the background.
They are like that. Not exactly The Sopranos. Quite different from TV.
Exactly. We tend to underestimate the power of compounding wrt learning from your mistakes. Two things that are important here: honesty (you can't fix what you don't know) and consistency (tight feedback loop = faster compounding).
This extends far beyond software programming. It applies to any form of self-development. I actually developed a habit formation program framework [0] built off of these principles (and a couple others, e.g aligned incentives, bias towards action, writing is thinking).
Whenever I start something new, I accept failure immediately. It frees my brain to enjoy my mistakes; the more mistakes, the more I learn. The worst thing that can happen with something new is I do nothing wrong.
I think acceptance and comfortability with error messages (i.e. small failures nearly every step of the way) is the basis of one's career as a developer. We start knowing almost nothing, producing tons of errors as we go (which reduce relative to experience). And learning what those errors mean-- they guide us into proper development.
Errors are good given that they lead to learning. Frustration is fun.
You're one of those people that never takes risks and never really does anything and just has a plain jane life, aren't you. But, obviously, you're quick to laugh at others' misfortunes for trying to do things.
And seeking out or even opening yourself up to a chance of failure gives certain people a risk of falling into a spiral of depression. I am not even sure that going for desirable yet challenging goals is even an option for these people if too much failure will destroy their confidence about life in general, whether or not that is a consequence that logically follows.
If you're weighing an option that opens you up to a higher risk of failure than other options, then depending on what type of person you are, you may have to evaluate whether you have enough of a support network or other "cushion" to absorb the mental blow of failure first. Failure is something that can be gotten used to, but before it is, you have no choice but to get past the damage to your mental health the failure causes, and some people are not capable of managing that on their own.
This is why I have to sometimes frame my decisions as: "Because I will fail considerably more often if I choose this path, I'm giving myself a considerable risk of becoming a miserable wreck incapable of doing the things I'm capable of now, or worse. So for the sake of preserving my mental stability, I will have to do something else."
The fact that this is on a pager app company's site immediately sets off some alarm bells.
"Systematically overcome your fear of failure" is a great principle, but why is PagerDuty of all companies promoting this? I can't help but think this is an attempt to market some kind of product, but I can't figure out what it is.
I think whole swathes of the SWE community are behind the times when it comes to responding to crisis.
There are models for how to handle this stuff in other industries. What comes to mind is "Tiger Teams" and "EMS services". Both of these involve the deliberate creation of special organizational structures that respond to crisis. For both of them that means contingency planning and, importantly, practice. It's largely about training for things that might happen and thus being ready when they do happen-- even if they're not exactly what was expected.
Sadly, the model for crisis management in SWE (in most places) is just throwing around the word "accountability", putting people in the hot-seat, and hoping for the best. You end up with "hero-based" crisis management. In fiction, that works out great (James Bond types always win), but in reality you just end up with stressed-out people and flaky systems.
I think some orgs are getting there, with SRE teams and red team exercises and the like. But I agree the overall industry is still deep in the 19th century, as far as managing this specific set of risks goes.
> What comes to mind is "Tiger Teams" and "EMS services"
I know EMS services of course, but what are Tiger Teams? A cursory Google search didn't turn up anything useful. Any pointers where I could learn more about them?
"Tiger Teams" are relatively common in large technical organizations (usually outside of computing).
I have seen them at a linear accelerator facility and also in large factories. I think the first time the name was used was in the NASA space program (and looking at the wikipedia link someone provided that appears to be correct).
Basically, they are teams that are put together to deal with ongoing critical cross-cutting problems in the organization. They are, by nature, cross-functional. Large orgs tend to put talent into silos. Silo'd talent can make the org vulnerable to weird problems where no one feels enough agency to act using the excuse-- "it's not my job". A Tiger Team is intended to work across silos.
Ours is a small company with about 50 employees. Whenever we encounter an especially mission-critical blocker we assemble a group of people that put their heads together to work on it. The goal is investigate all possible angles and relentlessly work on it till it is resolved.
Some companies, ones large enough to have serious SRE staffs, have 24 hour staffing of an operations center, and a War Room where people can huddle until a problem is fixed.
If they are smart, they have ways to roll back offending code.
But any company smaller than say a regional bank is going to have trouble affording this.
Now, if you using a cloud provider, you can design in rollback.
But of course there are surprises. A scenario I just thought up, when you discover on Black Friday that your order system can't handle more than 1048575 (that's 2 to the 20th minus 1, just to pick a slightly plausible number) orders in one day, and this limit is deep in your code and architecture.
The trend I've seen in big tech is that microservices architectures kind of negate a lot of this benefit though. Yes, there might be SRE's who are on-call for existential company-wide outages, and ideally they have some general knowledge about some of the biggest systems and common causes of issues. But for any issues that are due to bugs in the code or configuration of one particular service (which is usually most issues), there is a separate on-call rotation for each service. These folks are mostly just regular engineers that are primarily responsible for building features and delivering under time pressure and don't have the luxury of a lot of training & practice dealing with incidents until they happen for real.
So after a year of two of oncall at Amazon (it was long enough ago that we were oncall for the whole thing, long before SRE roles were ever invented) the biggest thing I learned was that nobody was dying and it wasn't actually fire-fighting and we shouldn't be using analogies like War Rooms and all these other War/EMS/fire analogies.
I've never worked in one of those jobs, but I've actually dragged dead bodies out of the water due to scuba diving accidents and analogizing IT outages to real life fatalities is almost sort of like how we think about holocaust/nazi comparisons and shouldn't use those casually. The industry really needs to chill out a bit about the impact, nobody is generally dying for the kinds of things we're talking about where the website is just down (obviously some IT really is life-affecting, and in finance people can be hauled off to jail by the SEC, but we're generally not talking about those industries here).
There are some processes such as accident analysis from aviation and aerospace which are useful, but could be used a bit more dispassionately. Blameless postmortems are also better than blamestorming, although I think there is a bit of a limit (at some point you may really have a bad team producing poor architecture which is constantly knocking the website over and you need to kick someone out, that isn't the purpose of an engineering postmortem though, that should happen at a management level).
I'm also weird and I don't really feel a lot of guilt (never raised religious might have something to do with it) and I just naturally understand how "shoulds" are bad and find it mildly appalling how Millennials love to throw the word "shame" around for situations where it really doesn't apply (your sportsball team shouldn't really be "shamed" if they lose). The broader population could do with a reassessment of shame and guilt and perfectionism and "shoulds".
Back in the day (and still today, sadly) SWEs had nothing to do with how their code was run. You paid Operations Engineers to staff a NOC and build an entire practice around preventing and responding to crises. The SWEs would commit their code, push a deploy button, and go home for the day (at 5PM on a Friday), oblivious to any chaos that might ensue.
I still meet SWEs who seem to believe they have no responsibility for any failures resulting from their code if it "worked for them in dev". No responsibility to design their code to resist failures or security issues in production, nor re-design it if the current design is failure-prone. No responsibility to performance test, functional test, or check with other teams on what a change might impact at scale. No need to come up with a rollback plan or test it.
If their product makes a ton of money, upper management enables this attitude, because nobody wants to kill the golden goose or risk losing its developers. It's a question of priorities. If the business does not require SWEs to care about risk, failure, reliability, etc, then they won't.
We have to somehow hack ourselves and make our brains love winning/succeeding.
I think we mostly don't love winning or succeeding per se, even when that happens it's not the joy of winning and succeeding as much as it is the relief of not having crashed and burned.
The whole mental process is dominated by loss aversion, not by victory enthusiasm.
When interviewing a while back, I was asked to reflect on my failures. Only then did I realize that failure was not in my mental model of my experiences. I had a hard time thinking of a "failure". It's not that I had never had any, it's just that I had been so accustomed to the idea of thinking of them as learning experiences rather than failures. I wouldn't say I'm the most accomplished person by any means, but I certainly have a tried-and-true way of getting through so-called "failures", and part of that has to do with not even including failure in my vocabulary of my experience.
Learn to not care. Learn to not care about being fired, losing everything, and dying in general. For me, the moment was like a longer more drawn out experience like that of Office Space. But it became that. I didn't care what happened, who screwed what up, or what expectations were of me. Fire me if you don't like it. Now you're really gonna be screwed having to pick up the slack without me there.
Once I unlocked this, the world became flat. I didn't care about the journey, just the goal. Whip something up. Learn how to socially manipulate expectations. Make it seem like you work. Half ass something and convince people there were significant difficulties.
None of your code matters. You are paid to support. Not to reinvent the wheel. So support it with the easiest code possible. Quit trying to do something fancy. It doesn't help when layoffs come about. What does is cozying up to VP's and management who know you work and get things done. Your portfolio means fuck all if not a single person knows what you do, even if you are a better programmer.
If you were working on a production system that had a failure, you'd probably want to dig into the why behind the failure and take some steps to keep it from happening in the future. Why not apply the same to your own failures and treat them as an opportunity to improve yourself? I like the idea of tracking my own personal performance to help mitigate future problems.
There is probably only going to be a retro for the incident if it was something really bad. Even then it's not going to focus on you, it'll be more about cross team efforts and higher level issues.
Having a personal tracker may help you understand your own weak spots, common sources of problems, and trends in your work. For each incident look at things like why it happened, what could have been done to avoid it, and the context that the mistake was made in (e.g. we had three issues going at the time and were under a product deadline for X which was due in the next two weeks).
To measure performance you could classify the mistake type and plot it over time. It could be something general like the preventable/complex/innovative buckets stated in the article with technical details like: missed the bug during PR review, misunderstood the system architecture, missed handling an edge case, etc. Having the issues clearly stated for yourself should help you not make a similar mistake going forward. If you have a trend in the data you can focus your personal development on areas where you're weak. Keeping a reference of issues that have happened should help with understanding the problem history a year later too. That could be good for planning future work since you'll know where some of the hard spots might be.
If you're having issues with the emotions and psychology of failure or anything else, it may be worth it to use your extended benefits and see a mental health professional.
DIY is great, and an outside professional perspective can also super charge your efforts and put you years ahead.
48 comments
[ 2.9 ms ] story [ 108 ms ] thread“When a person feels that he or she is becoming his or her undesired or feared self.”
I feel like a lot of my motivation to better myself and help improve my environment comes from that feeling. It's as if i have a very clear picture of who/what i dont want to become, but not so clear on what the "ideal me" would be.
Is this something that motivates HN as well or do you try to realize an ideal(?) version of you through your thoughts/actions
I don’t really subscribe to TDD, but they have a valid point: Start with something that doesn’t work, and work on it, until it works.
I tend to develop my software in a similar way, but I also start with a “fuzzy” design, which would give TDD people fits.
I fail every day. Repeatedly. It’s how I end up with some damn good software, at the end of the day.
I do have a blurb that discusses my design philosophy here: https://littlegreenviper.com/miscellany/evolutionary-design-...
I’m developing a fairly ambitious iOS app.
It’s a social media-like app, based on aggregate servers. I wrote the servers, myself, years ago. One is currently a world standard, and heavily in use. The other, was a server I wrote a few years ago as a “practice” project.
The app, itself, has been under development for over a year.
A big reason for that, is the team had only a vague idea of what it would do. It wasn’t really feasible to get them to hash out a more detailed plan, so I set up a “breadboard” system, based on Apple’s TestFlight beta-test system [0]. TestFlight is usually only engaged towards the very end of a project, but we’ve been using it since one month after my initial checkin.
TestFlight requires a fairly “complete” app, that runs without tripping any alarms (like crashing). Apple vets each release.
I can produce a fairly full-featured app, quite rapidly, so I was able to meet Apple’s quality bar, pretty quickly.
Because we are using TestFlight, we have some significant advantages:
1. The app is constantly at “beta” quality. Even with incomplete functionality, what’s there, works well. I cannot overstate the importance of this.
2. We can have non-technical stakeholders constantly “test driving” the UI, and providing relevant, useful feedback, on a continuous basis. They are also highly motivated and engaged.
3. Because Apple is constantly vetting the app, we know that we probably won’t have any nasty surprises, when we finally release.
During this year, we have explored a number of avenues that have resulted in “dead ends.” I’ve tossed out months’ worth of work.
That’s great. The best code I write, is the code I don’t write. I’m happy to tear out huge chunks of code.
The result is that we now have a clear, robust, ultra-high-quality app that is in the final stages of development. I still have a couple of areas of functionality to add, but we’re done prevaricating. We know where we’re going, and everyone is on board and psyched.
The quality of the app is also, quite frankly, jaw-dropping. We’re all thrilled to death.
Using TestFlight has allowed us to work through a lot of stuff that is often encountered during early MVP stages; without the brand damage and public humiliation.
[0] https://developer.apple.com/testflight/
If the TDD approach said "thank you" for the good parts (code structure, not breaking 500 other test cases), that would make the 13 errors less scary. One of the things I like about software is that it's "soft" - it should be possible to make changes without the fear of getting it wrong, and be able to fix them now while I see the problem, rather than procrastinating until later when I've forgotten about it.
The same "thank you" system applies when working with external contractors (e.g. language translations). We can assume they tried their best, and therefore might not know what else to say. Reporting 19 problems will get us a reputation for being nitpicky, hostile, even toxic. Rather, we should thank them for the 730 that they got right, and suggest improvements for the 19 that we think might be better.
With thankfulness and encouragement, TDD could really change the industry for the better.
I've always preferred carrots over sticks.
Many developers really dont care what happens in production, leaving a subset to stress out. Perhaps the solution is just be like the people who just expect problems and accept it.
That being said, here's another way to view it: of all of those deep seated fears you have, what do you know you can do to avoid certain issues and what can you actually not do?
Let's take the database for example. What's the deep seated fear here? Maybe:
> The disks crash, the data center bursts into flames and all the data is gone!
A solution to those is perform backups and get the backups off-prem. You have to explain the risk and justifications to the company as best as you can. Once you've done that, what is left?
Technically speaking, bugs in the backup script could exist, or maybe the backups get corrupted. Beyond that, there are the human ones. Maybe somebody else manages the script and breaks it, or maybe backups get unintentionally poisoned by malware getting into the backups. Or maybe even before that happens the company goes "eh, risk worth taking to save costs!" and you don't even get to implement the backups.
Here's the thing: these things are _not_ in your control at this point! They are out of your hands, and with humans like us who make mistakes like us. So problems are going to be expected, especially ones you don't realize right now. But they're not in your control! And yet, you did your part, and things will continue on.
These days, particularly in microservices, more and more developers are on call for their work, so they care more.
Still, I find a radical misalignment between my mentality, where you CANNOT make mistakes because you're working without a net, and development, where typically there are more layers of safety. I'm far, far more cautious and anxious about making changes and breaking things; they've historically been paid to produce. If I try to have a conversation about it, they're unable to grok that their threshold for the acceptability of making mistakes is much higher than mine is.
You NEED to be able to make mistakes in order to be able to produce, but it's a hard mentality shift.
Someone who never makes mistakes or someone who deals with their mistakes openly and directly?
We know the first one doesn't exist. Some people like to make us believe that, but it's bullshit.
However, one big caveat: Owning mistakes, being open about them doesn't absolve us from making them, especially not in the future. We still _made_ the mistake. Should we dwell on that or get punished? Likely not. Severity, intention and so on matter too. But should we simply not feel bad about it? "Embrace mistakes"? Not so sure about that either.
When we say "embrace mistakes", what we really mean is "embrace learning". Mistakes are opportunities to learn for ourselves and others indirectly.
We shouldn't be victims of our bad feelings and beat ourselves up. I did that for years. It crushed my confidence and well-being into submission and years after I'm still recovering from that. But the solution is not to try and make us feel good about ourselves regardless. It's not _just_ a psychological problem.
Mistakes happen for a reason, even if they are as simple as "I didn't write that down". They happen all the time, we can't prevent that, but what we can do is decide what kind of mistakes we want to face and ultimately what kind of problems we want to solve, be able to solve - for us and others.
Don't get bogged down, don't beat yourself up and don't let others beat you up for your mistakes. But make yourself aware that you can actually learn from them and decide for yourself when you want or need to put the energy into doing that.
I get why people are saying things like "normalize mistakes" etc, living in some kind of police state environment that punishes every misstep is just miserable. Especially if you're working in software, where "no mistakes" is not a reasonable standard.
OTOH it shouldn't become an excuse for complacency, and especially not an excuse to tolerate incompetence and/or totally unforced errors.
He tells me a story about how he drove his truck under a low bridge, and did about $20,000 worth of damage to it.
When he got back, he was expecting to get fired (or worse).
Instead, his boss asked him what he'd learned from the mistake, and gave him his next assignment. When my friend asked (incredulously) why he didn't get fired, his boss answered "I just spent twenty thousand bucks, training you to be careful of clearance. You think I'm gonna toss that out?"
My friend says he learned a lot about management from these goombahs. They were managing very dangerous people, who were big investments, and couldn't be treated badly or disrespectfully.
I'm sure it didn't cost him too much. He used 20K as a nice, fat number, but they probably only needed to shell out a couple hundred bucks out of pocket. It was an easy act to do. Impressed my friend, but he didn't really think about the background.
They are like that. Not exactly The Sopranos. Quite different from TV.
This extends far beyond software programming. It applies to any form of self-development. I actually developed a habit formation program framework [0] built off of these principles (and a couple others, e.g aligned incentives, bias towards action, writing is thinking).
Here are examples of programs using it:
* https://themoai.org/work-life-balance
* https://themoai.org/intentional-technology
[0] https://themoai.org
Errors are good given that they lead to learning. Frustration is fun.
Tony Hsieh from Zappos famously does one thing a day that scares him. At a certain point, potential failure is as scary as just trying something new.
https://www.nytimes.com/2021/01/26/technology/tony-hsieh-dea...
If you're weighing an option that opens you up to a higher risk of failure than other options, then depending on what type of person you are, you may have to evaluate whether you have enough of a support network or other "cushion" to absorb the mental blow of failure first. Failure is something that can be gotten used to, but before it is, you have no choice but to get past the damage to your mental health the failure causes, and some people are not capable of managing that on their own.
This is why I have to sometimes frame my decisions as: "Because I will fail considerably more often if I choose this path, I'm giving myself a considerable risk of becoming a miserable wreck incapable of doing the things I'm capable of now, or worse. So for the sake of preserving my mental stability, I will have to do something else."
"Systematically overcome your fear of failure" is a great principle, but why is PagerDuty of all companies promoting this? I can't help but think this is an attempt to market some kind of product, but I can't figure out what it is.
To drive traffic and increase sales of PagerDuty. It reads like pretty standard content marketing to me.
There are models for how to handle this stuff in other industries. What comes to mind is "Tiger Teams" and "EMS services". Both of these involve the deliberate creation of special organizational structures that respond to crisis. For both of them that means contingency planning and, importantly, practice. It's largely about training for things that might happen and thus being ready when they do happen-- even if they're not exactly what was expected.
Sadly, the model for crisis management in SWE (in most places) is just throwing around the word "accountability", putting people in the hot-seat, and hoping for the best. You end up with "hero-based" crisis management. In fiction, that works out great (James Bond types always win), but in reality you just end up with stressed-out people and flaky systems.
> What comes to mind is "Tiger Teams" and "EMS services"
I know EMS services of course, but what are Tiger Teams? A cursory Google search didn't turn up anything useful. Any pointers where I could learn more about them?
I have seen them at a linear accelerator facility and also in large factories. I think the first time the name was used was in the NASA space program (and looking at the wikipedia link someone provided that appears to be correct).
Basically, they are teams that are put together to deal with ongoing critical cross-cutting problems in the organization. They are, by nature, cross-functional. Large orgs tend to put talent into silos. Silo'd talent can make the org vulnerable to weird problems where no one feels enough agency to act using the excuse-- "it's not my job". A Tiger Team is intended to work across silos.
If they are smart, they have ways to roll back offending code.
But any company smaller than say a regional bank is going to have trouble affording this.
Now, if you using a cloud provider, you can design in rollback.
But of course there are surprises. A scenario I just thought up, when you discover on Black Friday that your order system can't handle more than 1048575 (that's 2 to the 20th minus 1, just to pick a slightly plausible number) orders in one day, and this limit is deep in your code and architecture.
I've never worked in one of those jobs, but I've actually dragged dead bodies out of the water due to scuba diving accidents and analogizing IT outages to real life fatalities is almost sort of like how we think about holocaust/nazi comparisons and shouldn't use those casually. The industry really needs to chill out a bit about the impact, nobody is generally dying for the kinds of things we're talking about where the website is just down (obviously some IT really is life-affecting, and in finance people can be hauled off to jail by the SEC, but we're generally not talking about those industries here).
There are some processes such as accident analysis from aviation and aerospace which are useful, but could be used a bit more dispassionately. Blameless postmortems are also better than blamestorming, although I think there is a bit of a limit (at some point you may really have a bad team producing poor architecture which is constantly knocking the website over and you need to kick someone out, that isn't the purpose of an engineering postmortem though, that should happen at a management level).
I'm also weird and I don't really feel a lot of guilt (never raised religious might have something to do with it) and I just naturally understand how "shoulds" are bad and find it mildly appalling how Millennials love to throw the word "shame" around for situations where it really doesn't apply (your sportsball team shouldn't really be "shamed" if they lose). The broader population could do with a reassessment of shame and guilt and perfectionism and "shoulds".
I still meet SWEs who seem to believe they have no responsibility for any failures resulting from their code if it "worked for them in dev". No responsibility to design their code to resist failures or security issues in production, nor re-design it if the current design is failure-prone. No responsibility to performance test, functional test, or check with other teams on what a change might impact at scale. No need to come up with a rollback plan or test it.
If their product makes a ton of money, upper management enables this attitude, because nobody wants to kill the golden goose or risk losing its developers. It's a question of priorities. If the business does not require SWEs to care about risk, failure, reliability, etc, then they won't.
I think we mostly don't love winning or succeeding per se, even when that happens it's not the joy of winning and succeeding as much as it is the relief of not having crashed and burned.
The whole mental process is dominated by loss aversion, not by victory enthusiasm.
Once I unlocked this, the world became flat. I didn't care about the journey, just the goal. Whip something up. Learn how to socially manipulate expectations. Make it seem like you work. Half ass something and convince people there were significant difficulties.
None of your code matters. You are paid to support. Not to reinvent the wheel. So support it with the easiest code possible. Quit trying to do something fancy. It doesn't help when layoffs come about. What does is cozying up to VP's and management who know you work and get things done. Your portfolio means fuck all if not a single person knows what you do, even if you are a better programmer.
There is probably only going to be a retro for the incident if it was something really bad. Even then it's not going to focus on you, it'll be more about cross team efforts and higher level issues.
Having a personal tracker may help you understand your own weak spots, common sources of problems, and trends in your work. For each incident look at things like why it happened, what could have been done to avoid it, and the context that the mistake was made in (e.g. we had three issues going at the time and were under a product deadline for X which was due in the next two weeks).
To measure performance you could classify the mistake type and plot it over time. It could be something general like the preventable/complex/innovative buckets stated in the article with technical details like: missed the bug during PR review, misunderstood the system architecture, missed handling an edge case, etc. Having the issues clearly stated for yourself should help you not make a similar mistake going forward. If you have a trend in the data you can focus your personal development on areas where you're weak. Keeping a reference of issues that have happened should help with understanding the problem history a year later too. That could be good for planning future work since you'll know where some of the hard spots might be.
DIY is great, and an outside professional perspective can also super charge your efforts and put you years ahead.