But I disagree about it being something special to leaders.
And the way he says it, makes it sound like maybe he thinks leaders shouldn't apologize in general. Which I think is a fairly common belief that leaders use to justify not admitting they made a mistake. And is 100% different from not apologizing for bugs. Since as he points out, bugs are not mistakes, they are a normal part of development.
When the manager makes a significant mistake it's not like a bug, it's a poor judgement that probably affected everyone. And sometimes if it's a very negative affect then an apology is warranted. But at least, it's usually necessary to acknowledge when you had people going in the wrong direction.
So sure, bugs happen, regardless of how good you are.
But there really are some people who constantly make mistakes, and those that rarely do.
There are people on my team where, when there is a bug, I just start looking at diffs of what they have committed because more likely than not, they caused the problem.
I think it's important to reinforce an attitude of trying to become better and people who give a shit when they make mistakes tend to also be the ones who make less of them.
> There are people on my team where, when there is a bug, I just start looking at diffs of what they have committed because more likely than not, they caused the problem.
If you can tell the bug from looking at diffs, why was the bug not caught in code review or by tests? Most obvious bugs should be caught this way.
In my experience, it's because Thing 1 and Thing 2 can review each others' code, and have no compunctions about potential bugginess. Somebody else gets to pick up after both of them, and is always outvoted during team code review.
Code review also isn't great at catching the kinds of edge cases that lead to many production bugs. Even tests only help you so much, because if you forget about input X or state Y or api result Z while writing the code, you're likely to forget about it when writing tests too. This is why bugs are more or less inevitable in systems beyond a certain level of complexity that are being actively worked on, and why engineers who think very carefully about anything that could go wrong will produce far fewer bugs than sloppy engineers who try to use process as a safety net.
As a manager, I've always striven to apologize for my errors.
I do agree that errors are unavoidable and are bound to happen. That's why I actually never cared at all whether programmers apologize for bugs. Fixing bugs is also part of the software development process after all.
However, I've always believed an apology is important to show humility to my team mates. As a manager, I do not want to appear entitled. Of course, as soon as the error is uncovered, we should prioritize it and fix it. But I do not believe that proceeding to fixing an error is orthogonal to apologizing for it.
We are trying our best to build a company without a guilt culture. In such a position, is apologizing a bad move as a leader? What would you do?
I'm a manager and I appoligize if it was a stupid mistake made from carelessnes. Apologize if you did something you wouldn't want other people doing.
I think context matters too. The industry you're in (what are the consequences of bugs) as well as the part of the codebase (presentation or data integrity) change the acceptability of bugs.
Maybe don’t apologise for it, but I definitely think it’s useful to acknowledge if it was your code as it removes an awkwardness around it. Obviously this doesn’t have to be done for every little bug but if there’s a discussion around one then owning it can help.
Probably because there is always a bigger picture: culture, process (e.g. testing, code review, time pressure, feature centric performance metrics).
Surely, the person making the mistake is the proximal cause for a given bug, and surely all other things being equal personal attitude matters (i.e. a more careless person is going to make more mistakes), but if you zoom out to the big picture you see other responsibilities and more importantly you see opportunities for improvement that vastly exceed the playoffs of "just putting more care" into each commit.
Understood. Thanks for the explanation. I have to say I don’t intuitively like the position, because taken broadly you could conclude that no-one in an organisation is ever personally responsible for their actions, because some system should have prevented the mistake. This feels wrong
Indeed, it does feel wrong because it is, technically, wrong.
The intent of a "no blame culture" however is to find a new sweet spot on the myriad of tradeoffs an engineering organization does, as an many paradigm shifts we do, it is accompanying with a certain amount of bullshit-sounding maxims meant to be thought provoking and dislodge the readers from their suboptimal local maximums.
But that's just reassigning or distributing 'blame' to the approver.
So what is the responsibility of the key-puncher if it isn't turning requirements into robust, correct code? Many comments here seem to suggest that programmers are just there to produce some sort of fuzzy first-stage approximation of what the code could be.
In that case, just employ an automated code generator and push the burden onto reviewers.
I think it would be great if we a group of people whose entire job was to watch over the programmer's shoulder, and every time they make any kind of mistake, SHAME THEM. Because shaming the developers is more important than anything else, apparently. All we need is other people who can 100% reliably detect errors in code... oh. They could simply be the developers instead!
> So what is the responsibility of the key-puncher if it isn't turning requirements into robust, correct code?
How can it be a responsibility to never make a mistake? Nobody can never make a mistake.
> Many comments here seem to suggest that programmers are just there to produce some sort of fuzzy first-stage approximation of what the code could be.
Many comments here seem to suggest that producing high quality code is a team effort supported by a company or department which values high quality code. If you had mythical no-mistake employees you could do away with most everything else. Why even have tools, tests, revision control systems if you're not making mistakes?
And if you are using tools, tests, etc. as part of not making mistakes, why not reviewers, bug testers, quality analysts, as well?
One person wrote the requirements and acceptance criteria, another person signed them off, a developer wrote the code (probably using parts of code that many others have written), another developer peer reviewed the code, someone else QA'd the feature, someone else checked the feature was deployable, and so on. Code is never written by a single person on anything but the very simplest projects.
I'm not sure if you can extend bugs to that many people, since usually bugs and awful code are not tied into the requirements or the acceptance criteria. That being said, I do hold code reviewers partially responsible if shit gets past to production.
A code reviewer is able to see that someone is building SQL from raw input, the QA isn't. And if I see that a developer writes that kind of a thing and another person approves it, the first question I have is if they actually read through the code or just glanced at it.
Would you be ok with your dentist being OK with her slip-ups?
“I really should have checked the x-ray better before I extracted your front tooth. But you know, I’m not sorry. It was an external constraint — there was a queue of patients waiting”.
I can’t believe how low we let this profession sink. I hope in 15 years we will look at thies era like we look at the Wild West today. Exciting from a distance but nobody would really like to live like that.
What I think should be common in these two cases is the attitude of a professional towards their work.
Building software should be an engineering discipline with all its perks and responsibilities just like building bridges. We rarely see it and it propagates to the craftsmen who seriously believe that good quality software is impossible to write under the slightest of constraints. Like business pressure.
We had this ability in the past when programming was way more difficult, tedious and error prone. Look what IBM did with their “Clean Room” approach.
I’m sure we still have it at many places but we choose to spread the myth that it cannot be done. This call to stop apologizing is the next step towards not even trying.
> What I think should be common in these two cases is the attitude of a professional towards their work.
I think everyone would not deny this fact. Engineering is more of a discipline.
But if you see things practically, this issue of "bad" code may not happen for various number of cases. For example, unclear project requirements, use cases etc., That is where 'context' of what happened helps to better understand the situation.
Building bridges are not done agile and it has a learnings from decades or even centuries. Computer Engineering is mostly agile where you push changes based on changing requirements.
In some countries at least engineering is a professional title and properly gated, but until we stop allowing people to call themselves engineers after a six week bootcamp in websites design this downpath will keep going on.
Like on that dentist's case, my hope is that as liability persecutions and lawsuits due to faulty software increase, that will eventually be sorted out.
This sentiment reminds me of Gerald Weinberg's observation of four plus decades ago - "If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization." I was trained as a computer engineer, and the difference between those who were taught to approach software development as a craft and those who think bugs are ok and it's really about feeling good about myself at the end of the day is clear and obvious in most organizations and in the product that comes out of them.
At the same time, if programmers had to deal with the same kinds of rigid constraints as builders, writing a simple MVP would cost millions, and we'd have a lot less new technology in the world.
All I can add is that, as a freelancer, I always explain (like they are 5) what went wrong. My customers don't care about the details but they like it when they know I made an effort of creating good software (bugs included).
I think it's worth acknowledging over your customers that the bug has caused them lost time or other kinds of issues. Whether you do that with an apology, it doesn't really matter, but I know I'd personally feel better when a vendor emphasizes with me.
Something you definitely don't want is to sound like it's an inconvenience for you that they're reporting bugs. Bug reports are extremely helpful and you should be grateful that they're taking the time to report them to you.
Bugs categories are also important. My project and my leadership was destroyed when a non-technical wanna-be "operation" manager without experience in software development, started to see only bugs and reduce all development to bug fixing. Some "bugs" are in reality wrong or misunderstood requirements. Some are missing features. Sometimes the joke "it's not a bug it's a feature" is real. Sometimes "bugs" are edge case we didn't want to solve immediately, until the edge case it's not edge anymore.
I think what is dangerous it's seeing ONLY bugs, focus dev ONLY on bugs. Shift a little the point of view from bug fixing to software evolution.
Why do software engineers constantly try to project that they are a special snowflake of a profession?
Would you accept such nonsense from your plumber, or your dentist, or your electrician?
Software engineering is one of the most fundamental building blocks of modern civilization, and yet it continues to be so fragile because of this kind of cavalier attitude.
Also, apologizing doesn't mean you have a moral failing. It means accepting at least partial responsibility, and being humble about it. Any good org knows that you don't just design your system with the hope that people won't screw up, you need to design it to work despite some amount of screwing up. But in either case, it is still both an individual and collective responsibility.
And yet my house has never caught fire due to a miswiring or got flooded due to a misplumbing but let me not even count the ways in which software has failed me.
Maybe there's a reason you rarely hear one category apologizing and hear the other category coming up with excuses.
Plumbing and electrical doesn’t change all too often. Once you learn the ins and outs it doesn’t change much over your career. Not the case with software development.
I know people who’ve programmed microcontrollers in C all their lives and still written bugs. It’s not like switching tooling once in a while is the source of bugs.
> but let me not even count the ways in which software has failed me.
Failed you? This sounds either very entitled or sensational. Has software caused you major grief in any way financial or the loss of limbs or loved ones? While this happens, very few people are affected by bugs in such a way, at least confirmed.
>Failed you? This sounds either very entitled or sensational. Has software caused you major grief in any way financial or the loss of limbs or loved ones? While this happens, very few people are affected by bugs in such a way, at least confirmed.
Many people have been significantly affected by software bugs. Therac-25[1] is one I constantly remind myself of; it killed 5 people. Here's a list for further reading: https://en.wikipedia.org/wiki/List_of_software_bugs
I'm not sure what your definition of "very few people" is but we'd all do well to aim for 0 adversely-affected people as a result from using software we write.
Read my reply again, note especially the part "While this happens". Obviously I am acutely aware of what you are saying, no matter how many Wikipedia articles you link, so your comment is not only superfluous but also makes no sense, as I was asking OP if he was personally affected in such a way because he seemed bitter. Your reply does not answer this, and you should let him answer for himself.
> I'm not sure what your definition of "very few people"
I think my definition is about the same as every accepted definition of "very few", the number of people affected by such bugs falls within this definition, considering how many users software has.
> Many people have been significantly affected by software bugs. Therac-25[1] is one I constantly remind myself of; it killed 5 people.
5 does not fit the definition many in anyone's book.
When someone pays for software, they are perfectly entitled to expect it to be fit for purpose. Failures don't have to cause major grief for me to be justifiably irritated and to expect redress. A conscientious developer should apologize for wasting people's time or selling them products that don't do what they are supposed to.
Plenty of people are hurt or killed through clerical errors by medical professionals, but you're not going to get anyone admitting that. Even for something simple, if you have any accusation to make in a medical context, it's probably time to evaluate your options and consider a new doctor, and only then make your accusation.
The worst thing code I have written ever did was cause a small power outage. But that was a one-time thing, I haven't had the opportunity to write code that even could do that in years.
This is about the dynamic between people within the organization (coworkers, boss, etc.), not about external communication to customers. That is, we aren't talking about whether an individual software engineer should apologize to the customer or the end user.
So the analogous question would be whether someone who has worked as a plumber has ever heard a coworker plumber apologize for messing up a job. I'm hoping the answer to that is yes. A plumber apologizing directly to a customer would be more analogous to a company making a public announcement about a bug, which is a different issue than how responsibility is determined within a team.
Sure you have a point, but then I do not force my plumber to do his job in far less than the time required, keep changing the plumbing requirements and make him start before the final requirements are seven known.
You are right. This is the flip side of not treating software as an engineering discipline. But both parties are at fault. The business treats us like non-engineers and we don’t behave like ones.
This is not a cavalier attitude. The article is not minimizing the harm caused by bugs. It is attempting to direct the blame away from individuals and towards systemic causes of defects. This is necessary to be able to effectively address them.
"Just be more careful" has pretty much proven not to be a successful strategy to reduce defects.
No, it continues to be so fragile because software engineers build virtual machines with enormous numbers of moving parts that need to change all the time but nobody wants to pay the price for exhaustive testing (not to speak of the price of formal verification).
We are pretty good at writing software with a very small number of bugs, it just costs a few orders of magnitude more than normal software.
No it's because a piece of software with a few hundred thousand lines of code has a state space that boggles the mind. And then that software interacts with a dozen other services of similar size. And then new features need to be added every few weeks. It's completely incomparable to wiring up a normal house.
In buildings with nontrivial electrical systems, shit hits the fan really quickly too. Just look at the absolute mess that is the fire prevention system in Berlin's new airport for example.
I see what you're saying, but I think electronics is equally or more complex than software, but over a longer time period
Copper, plastic, other things which I have no authority to talk about, were all complex at one point in time
You do make a good point about scope creep, I wonder if the early days of electronic had a similar problem because nobody knew what they wanted or what was possible?
I'd think so. If you were to install one of Tesla's early first AC motors, it'd sometimes do unexpected things or stop working intermittently, I suppose
This is the same thing I was thinking. We write imperfect software because the market (and the law) does not incentivize the significant extra work required for perfect software. Blaming individual engineers or organizations for that doesn't make any sense.
Yeah I'd accept it from my dentist. A couple of years ago, they gave me a dental crown for one of my upper right molars. Two months later, it chipped. It felt weird in my mouth.
Why wouldn't this be acceptable? They source that crown from a supplier, and apparently there was a fault with it.
Make sure you're comparing them equally here. So don't compare the plumber coming to your house to fix a problem with the sink, with a software engineer working at a airspace center for example. The home-plumber vs the home-hacker and the enterprise-plumber vs the enterprise-developer is a more fair comparison, and while I'm not a plumber, I think you underestimate the complexity they deal with when working on a larger scale.
Really incredible, when you have to go through university for at least 5 years, with heavy technical and at least for me mathematical lectures. Also incredible, when the complexity of IT systems is ever rising. Incredible, when we basically have to build mathematically correct systems, to make sure things work exactly like they should, including weird user wanted behavior, just for historical reasons. Also weird, that working in IT often requires a degree and that degree has requirements in educational background, which do not exist for plumbing.
Besides all this incredibility, yes, I do work professionally in an engineering role.
Limiting a statement is not the same as a completely making it meaningless. Of course there are plumbing projects, which are more complex than a static blog. Besides that, I would also like to remind you of the little part of the phrase "I'd wager that". I did not know, that it is an "ignorant" statement, to express ones believes.
I think it's normal to assume averages when comparing things.
I would be interested in some experiments of average developers learning average plumbing vs average plumbers learning average development. I'd wager theres an order of magnitude difference in the results(time spent, quality etc).
I've worked as a structural engineer (EIT) before. There are some situations where structures are complex (earthquake modelling and really large bridges) but honestly software is way, way more complex than structures. I'm not saying I agree with the contributed article (I think people should be sorry for bugs they missed, at least to a point) but I have some empathy for the fact that even seasoned developers let bugs in despite all the tests, and lints, and monitoring, and best practices. Sometimes you just don't anticipate something and nobody else around you did either. It's a more complex domain than the forces in the physical world, and unlike the physical world in most cases, the same mistake goes to everyone. One electrician screws up one time and one house burns down to the ground. One software engineer screws up one time and well everyone's shit breaks at the same time.
>It reinforces the idea that any one person or piece of code can be blamed for a given failure.
That can absolutely be the case. How could it not sometimes be the case?
>Short of malice, this is never the case.
Never? Outside of somebody deliberately harming a system, there has never once been such an instance?
>It gives the impression that, when you wrote the code, you should have written it better. This is a counterfactual that rarely holds up to examination.
Lots of assertions in this piece with no supporting information. There are plenty of reasons substandard code is written by some individuals. Laziness is a big one. Apathy is another.
>It positions shame as the correct emotion to feel about bugs in your code: if you were a better engineer – a better teammate – the bug wouldn’t exist.
Call it whatever you want, but concern is certainly warranted.
>If you’re a more senior engineer on your team, the effects of these anti-patterns are magnified: people see you apologizing for bugs, so they think that they should be striving to write bug-free code. They may feel ashamed if their code has bugs.
People look up to experienced individuals within their profession. In fact, I would argue that it is even more important these senior individuals are able to accept responsibility and act accordingly; that's setting an example, something from which everyone will benefit.
I think the important point is to let go of ego issues. A bug is a technical problem, how do we address a technical problem? By being level-headed and not playing the blame game. Being sorry doesn't add any relevant information and generate a strange atmosphere, like admitting a moral failing really. Any matter of substandard performance is better dealt by the management, well outside the scope of bug fixing.
No, elaborate insistence on repentance rituals, shaming, and ideologies of "personal responsibility" over systems thinking and data driven improvement is all about ego.
> Would you accept such nonsense from your plumber, or your dentist, or your electrician?
In the UK...
Dentists: you have to have a BDS or BChD degree from a dental school at a university
Plumbers: have to get NVQ diploma as well as pass Gas Safe accreditation to fit boilers and heating systems, plus get a CSCS card to work on construction sites
Electricians: have to get an NVQ diploma, and to work in construction, they must also have an ECS card to work on a building site
Software developers: [this is left blank as an exercise for the reader]
(IMO / 2 cents) because as a software developer, you can create a huge amount of functionality in a short amount of time if you cut some corners. This just isn't the case, doesn't have nearly the fast return-on-investment that you get in other fields of engineering.
I mean in electrical engineering, idea-to-working-prototype involves a lot of design and assembly first. In structural engineering, you just can't build a bridge on your own. It goes on.
While in software development, I can cough up ten distinct features in a day if not more without forward design or testing.
> Would you accept such nonsense from your plumber, or your dentist, or your electrician?
This would be a reasonable comparison if 90% of software developers could reliably produce bug-free code. This would be a (maybe less) reasonable comparison if 75% of software developers could reliably produce bug-free code. This would be a middling comparison if 50% of software developers could reliably produce bug-free code. This is a completely unreasonable comparison given that there have been 0 software developers ever, in the history of software, who have been able to reliably produce bug-free code. So yes, software development is orders of magnitude more difficult than plumbing, dentistry or electrician(-ing?). If you think it isn't, please feel free to jump in and prove all of us wrong.
Look, if I genuinely screwed something up, sure, I'll apologize and either way I'll fix it. But if I didn't test some corner case because the software takes 30 minutes to build, another 30 minutes to start up, and can't be run without a live database and crashes with an out of memory error after it runs for more than 30 more minutes and no time has ever been allocated to improving the code quality because that's "low priority to the business" and the fix was one of the 20 I had assigned to me in this two-week "sprint", then I don't owe anybody an apology. If you think you can find somebody better, be my guest.
Orders of magnitude harder? Get over yourself. You sit in a comfy climate controlled office with perks like free food and drinks, gyms, entertainment, etc. You make a mistake, and a program doesn't run correctly. The horror!!!! An electrician, plumber, or dentist makes a mistake and that could result in someone getting hurt or even someone to die.
A dentist doesn't get to run a dev/staging/prod environment before drilling into a tooth. A dentist can't roll back to a more stable previous version of the tooth they just cracked from a mistake that was made. There is no control-z in the real world outside of software devs.
Try working in military applications, aerospace, tax software, medical devices, banking. Making one mistakes can affect the fates of hundreds if not millions. It's way more responsibility and risk than the job you quote.
By the way. A dentist gets to try on fake tooth before drilling a real tooth. They actually don't have a choice, that's part of the mandatory training. Doctors gets to train on real people too, medicine schools go to great length to procure bodies for training purpose.
In programming, one would be lucky to get any sort of test environment.
> A dentist gets to try on fake tooth before drilling a real tooth.
Are you saying you went straight to writing production code before you ever studied any software engineering? Sounds unusual for practitioners of an industry endowed with such a critical responsibility.
A dentist can easily kill someone through malpractice... There's no code review, there's no test environment, and there are always difficult (often time-sensitive) decisions to make and difficult people to deal with.
If all you know is estimating story points and merging pull requests to make your burn-down chart look pretty you probably shouldn't be so confident that your profession is more difficult or more superior than one that you have no understanding of.
A software engineer can easily kill tens of people with a typo. Medical devices have a ton of software and it's only increasing.
Medical staff have the equivalent of code review and test environments. Non-routine operations have always multiple people in the room. They have bodies to practice on.
Do yourself a favour and drop the attitude. Medical and software industries both have some very difficult jobs.
You have got to be kidding me. Do you even understand the consequences of making mistakes in other fields? In some cases an apology isn't even enough. In some cases imprisonment and an utterly ruined career are the result of mistakes. So the idea that apologizing shouldn't happen as a programmer is complete rubbish.
> In some cases imprisonment and an utterly ruined career are the result of mistakes
This is quite an exaggerated point; the number of people imprisoned for making mechanical, civil, or electrical engineering errors is very close to zero in the US.
I think more programmers have actually been charged than any other field, although I can't quote a statistic to back that up.
With plumbing the tools of the trade have been the same for 100+ years. Pipes, wrenches, valves. It seems like every two years in the software engineering realm you have to learn to use a completely new tool in order to be valuable to a company.
When you design a hammer and sell it to the end user, there are limitations on how they can use it. You simply cannot encompass and address every situation that the end user would subject the hammer to. What if they left the hammer in a hot 1500 C oven? The designer cannot design it such that it can universally work in every situation. There are usage guidelines and limitations.
A hammer that melts in a furnace is considered a bug?
The possibilities and test paths a piece of code can possibly take exceeds a very large number in a complex software application.
Actually situations like this are still considered a bug in software. It is always the responsibility of the software to be resilient against externalities.
Zoom should work and gracefully fail if there is not a single microphone device found on the client’s PC.
Everything exists in an economic context. Errors in plumbing, dentistry, and electrical work can result in thousands or hundreds of thousands of dollars in damage, and they often can't roll out a fix in a matter of hours. Some software bugs can also result in lots of damage (financial/health data security, etc), and we do take those bugs seriously via compliance (although compliance isn't a guarantee for good software just like it's not a guarantee for good dentistry). But no one is going to pay for the effort required to perfect every software release, nor should they.
>Why do software engineers constantly try to project that they are a special snowflake of a profession?
I agree but this point is orthogonal to the main point. Plumbers and electricians build to code in an environment where physical reality remains constant and they operate on expected inputs. A civil engineer is not held to account for a failure if gravity suddenly reverses itself. If I shock a circuit board with static electricity and it fails is that an electrical engineer's fault? If I put 20,000 psi through a household plumbing system and it fails is it the plumber's fault? These designs are only expected to operate in a narrow range on a small number of expected inputs / variables and that is easier.
Software can basically expect infinite permutations of inputs at any time and is expected to give good outputs and no bugs 100% of the time?
I hear from plumbers and electicians all the time that the last guy to work on something was an idiot. Medical errors are routine and managed through a combination of procedure and insurance, not by admonishing doctors.
“Thanks for reporting this, I’ll take care of it” is accepting responsibility while avoiding a morality judgement.
I totally agree with you. In fact, some branches of software engineering do have equally high standards as other engineering fields. If you write the software for a medical device or an airplane, you better make sure it doesn't have any critical bugs, otherwise people will die. If you write audio software for professional musicians, you better make sure that it doesn't have any critical bugs, so you don't ruin a live show. I think people claiming that nobody can write bug free code and you shouldn't even try simply work in environments where failures don't cause irreversible harm.
I earn a living from my SaaS. I am a solo founder, responsible for everything. I wrote all the code. My customers pay subscription fees because the software makes their work easier (it's a B2B SaaS, https://partsbox.com/).
Bugs are solely my responsibility: there is no one else to blame. And bugs mean that my customers paid money for a product which did not work as intended (certainly not as I intended it to work). I feel it is definitely right to apologize for bugs!
I disagree with the author: if I worked at a workshop and dropped a hammer onto my customer's foot, I should apologize. No matter what the reason: I could be clumsy, or you could say that no one could be expected to never drop a tool — still, an apology is in order, because I did drop that hammer, and it made my customer's day worse.
Therefore, I always apologize for bugs.
What I did, however, learn to do, and it is quite important, is not to apologize when you are not sorry. An example is when a customer is unhappy with the pricing, or complains about a missing feature. Many people will begin a reply with "I'm sorry, but…", and it's the wrong thing to do. I only say I'm sorry when I really am.
If you fuck up I don't really care about an apology. I want to know:
1. How will you fix it?
2. How will you prevent it from happening again?
An apology, in my head atleast, is for when you have caused emotional distress to someone and there isn't much you can physically do to fix it other than to convey how you now feel about what you have done. When you make a practical mistake an apology is implied by you fixing it.
There are perhaps two parts to an apology. One is an acknowledgement that you did something wrong, and that you should not have done it. The other is a sort of ritualistic, performative (in the original sense!) use of language that may make someone feel better. This is the difference between:
"I made a mistake when i dereferenced that null pointer"
and:
"I'm sorry i dereferenced that null pointer"
I absolutely agree with you that the ritualistic part is pointless, and perhaps harmful if it is performed instead of actually fixing the problem (see also "thoughts and prayers").
But i wouldn't want to stop doing the acknowledgement part.
I wouldn’t refer to the first stage as an apology, it is simply an acknowledgement.
I very rarely apologize for a bug at work. I don’t expect others to apologize to me either. Apologies should be reserved for personal/relationship things, like forgetting to invite somebody to a meeting.
Shipping with bugs is not an inevitability. Bugs come from code being used in cases other than the ones that the programmer anticipated. But it is possible to understand how code will be used. You can test against it, either manually or with some automated testing tool. And you can produce code that behaves the way you expect it to every time you run it. Every once and a while you might mistype >= as <=, but you can catch it and fix it with only a modicum of effort.
But you first need to believe it is in your own power to write less buggy code. If you can identify patterns that frequently produce bugs, you can try to avoid them (e.g. copying & pasting: very buggy!). You can identify, isolate, and document footguns and complex API. And you can be proactive in simplifying systems when feasible to do so.
Don't beat yourself up if you accidentally have a bad commit, but don't excuse sloppy behavior. Demand better from yourself. Don't produce garbage software.
A lot of the replies and analogies here are interpreting this as a business-consumer apology. Of course if a business has disrupted a customer's experience through buggy software they should issue post-mortem/apologies/etc.
But I read this as though the author is saying apologies to fellow teammates shouldn't be necessary, which I do at least in part agree with.
This reads a bit like one of those feel-good "managers should be good to employees! agree?" posts on linkedin. The bugs I see in PRs from even some mature people are things like input_array[0] without checking the array for null or size. Anyone beyond cs101 can tell you this is unacceptable and careless, and the author likely had a "fuck it" attitude, and was in a hurry to get back to Netflix or whatever was really on their mind.
Should we expect for them to go into a full-on dogeza Japanese kneeling bow apology? Of course not. But there are many cases where a person glaringly half-assed something, cut corners, quietly committed something directly to master. And yes, this person should feel some kind of regret, or should be brought to stop such behavior because it is extremely destructive and is totally avoidable and caused not by overworking or accident but by persistent carelessness and neglect, so it cannot just be cheerfully swept under the rug.
Why doesn't your team have a process (integration tests, etc) that makes it impossible to check in something like `input_array[0]` without breaking a build? Why is it allowed to directly push to master?
How does the old saying go? The road to where is paved with good intentions? You could engage in moralizing against "carelessness" and "neglect" and presuming that a teammate somehow cares more about watching Netflix than doing their job (which they presumably outperformed other candidates to even get), and maybe that will make you feel better. You are the good, careful, model employee, and that other one -- they're careless, slovenly, apathetic and lazy. But if that's the case, how did they get past the door in the interview process? If the whole company is like that, why do you work there as opposed to a company where the bar is higher? Something does not add up. In all likelihood, your explanation is a rationalization and not the most obvious answer, which is that your process could be improved but it hasn't been because such investments are not viewed by your companies executives as improving the long term bottom line to be worth the short term investment cost.
Take a look at history and figure out how companies in the industry have solved this problem before by building more bulletproof runbooks, processes and tools. These companies, as they approach enormous scale, necessarily have to determine how to deal with employee reversion to mean. It turns out, surprisingly, that process eats good intentions and care for breakfast. You'd be surprised at how quickly those good intentions become useless if your company is successful and you hit hypergrowth and scale. It's ironic that for a profession where it is so tractable to automate away mundane or repetitive tasks, where we study spacetime complexity in data structures and algorithms, that we have so many practitioners that default to witch hunting and moralizing and seem unable to apply spacetime complexity or procedural analysis to their own software development lifecycle. I expect to see this trend change as our still young industry continues to mature.
Because we were a 4-person company at the time starting to hire first employees and I realized I can't trust new people as much, so most certainly you cannot anymore do things like input_array[0] and committing to master because i took measures immediately after it happened, so that is not a problem.
The problem is that the person(s) for whom I had to add these baby bumpers not to do input_array[0] will - and has - screwed up much bigger on everything that is slightly more complicated than that, too. Because if you are an engineer and you don't care, then no amount of padding around you will make you curious, careful, detail-oriented, non-lazy and capable of tackling hard problems that life throws at you every day at a tiny startup.
To commenters who compare work of programmer with the work of civil engineer:
There is an explanation to this discrepancy. Risk = probability of fault * cost of fault. Programmers usually work with the data. Data has a pretty very unique property that it is easily and cheaply copied and restored. It also means data can be replicated more easily, which allows to build systems with large amount of redundancy. Therefore, price of a software bug is usually orders of magnitude less than the price of breaking something in the physical world. This allows to drive up probability of fault parameter and allows for much faster evolution and less regulation.
I think a part that needs to be considered in the comparison is also the concept of safety factors.
Many structural engineering projects are built with a safety factor greater than 1, which allows for - as you'd imagine - safety. In software, it could absolutely be done that every function has explicit validation checks for each input, and that more safety, error handling, and checking is written. The economics just don't make that viable though except in the most extreme situations where this is required. In aerospace, with the exception of the recent Boeing debacle, there's been very few software related issues (that I know of) relative to the number of CPU cycles run and flights taken - I'd be willing to be there's far more mechanical issues that cause grounding and maintenance cycles.
Now look at civil engineering and infrastructure. How many roads do we drive over daily filled with potholes? How many bridges get patches and repairs every few years because of concrete cracking and rebar rusting? (I live in Quebec, so there may be bias in this statement). Maintenance like this is just paying down tech-debt - the same as in any field.
Software absolutely can be built to a high level of robustness (and it is) - it's just not economically sensible in most cases.
Replication costs have nothing to do with anything here.
The damage caused by a software bug can range from nil (even negative, since a bug might have desirable effects) to incalculable. Consider a bug in software used for landing SpaceX boosters... Or a bug in a trading platform. Or a bug in an automated medicine dispenser. Or...
You can compare the work of a programmer to the work of a civil engineer, but it's true that it's difficult because most bugs cost little, much software is not positioned to cause much damage if buggy, software bugs are so prevalent, and it's so difficult to write bug-free code. But the cost of replication of software has nothing to do with any of this.
> Replication costs have nothing to do with anything here.
Replication is the cornerstone of fault-tolerance. When logic is written in high-level languages with some minimal good practices, most SW bugs result in crashes rather than data corruption, hence data replication plays important role in protection against bugs as well as HW problems.
> Consider a bug in software used for landing SpaceX boosters... Or a bug in a trading platform. Or a bug in an automated medicine dispense
I specifically mentioned: "Programmers _usually_ work with the data. <...>" In the examples you listed above software manipulates physical world rather than pure data. I thought it was too obvious to explain.
I wish I could find that one guy I worked with about 19 years ago that tried to load an ActiveX component in his VbScript running in Netscape and send him this article, true he never apologized - but I just want to make sure he doesn't feel bad about never apologizing.
The biggest career lesson that I've had is that 80% of development is testing.
Should you apologize for bugs? No. But if you're doing professional quality work, (as opposed to prototyping or researching,) a large portion of your time (as a developer) should be writing automated tests (like unit tests) to test the code you write.
Bugs are the responsibility of the process that brought them there. In most cases it is shared responsibility.
Why doesn't your dentist mess up 1 out of 5 interventions? They have gone through years of strict training and there are giant liabilities in place.
In development? A lot of self-taught developers, who can always jump to a next gig because of never-ending demand. This sets up the scene for a totally different scenario - never-ending stream of bugs.
If you want your team to have less bugs, then as a manager make sure you have a process in place that optimizes for that. Start with your hiring process and add requirements for code quality practices. Then build a process that includes strict commit guidelines, documentation, code reviews and QA process. Most importantly, add time to estimates that ensures all of the above is checked properly. After all that is in place, define in a clear way some sort of liability for bugs.
Your dentist goes through all of that, why not your developers doing arguably equally important work?
Are there 100% medical procedures with 100% success rate?
To me it seems that failures happens pretty often, generally due to incompetence, limitations in the science regarding the problem or insufficient resources. Seems very similar to bugs to me.
The question is how bad is failure, obviously if someone's health is at risk, you have a much bigger incentive to have processes on the safer side.
Maybe you feel "bugs" are more frequent in software because the software you're referring to has an incentive towards innovating rather than stability.
I'm sure if you studied software used in critical systems, the failure rate would be more in line with a procedure at a dentist. You can't justify the kind of cost that kind of development has for most apps.
I believe the answer to your question of why developers don't generally treat quality the way dentists do is one of economics.
The world needs enough software, and the world's requirements for quality are low enough, and the average developer writes reliable-enough code, that in most cases money can be made without special attention to quality.
In some domains, quality is paramount. I imagine that quality and bug mitigation are the primary concerns of someone who manages, say, an X-ray machine firmware project, or avionics software. Software for these purposes that is not extremely reliable cannot compete in that market. For other kinds of software people are paid to write -- like social networks for pets -- reliability is a much lower priority.
Certain kinds of software needs to be rewritten constantly to meet the demands of highly-evolving markets. Games are an example of this. Reliability is important, but not paramount, because the underlying business can only continue to compete by at some point stopping work on the reliability of existing games, and moving on to developing new games.
I'm sympathetic to the respect you have for the value of high quality software. But the reality is, the value of quality varies dramatically within the software industry -- much more than it does within, say, dentistry, or medicine, or aviation.
Fewer bugs isn't necessarily the primary concern of those using and buying any particular kind of software.
May be - may be not. Would you trust your dental work to a self-taught dentist? What makes developers a special case? I am agnostic to it and genuinely curious.
> Through this lens, apologizing for bugs may seem innocuous. But it contributes to a bunch of cultural anti-patterns:
> * If you’re a more senior engineer on your team, the effects of these anti-patterns are magnified: people see you apologizing for bugs, so they think that they should be striving to write bug-free code.
Imagine writing this. Imagine actually sitting down at your keyboard and writing down that people should not be striving to write bug-free code.
The article sounds much worse if replace "bug in code" with "made a mistake at work". Imagine if we're talking about a civil engineer, an electrician, or a doctor.
I agree with you, it's outrageous. What an awful article. I get the point (you should have robust and multi-layered quality systems) but just the complete rejection of accountability is embarassing.
Doh, of course no programmer intentionally writes buggy code, yet there isn't a single programmer in the world who doesn't write buggy code (and everybody who claims to write 'bug-free code' is either very inexperienced or a goddam liar).
The point is to not have a "shame culture" for writing bugs, because they are normal part of the workflow. Instead focus on efficiently identifying and fixing bugs (efficient debugging is an important skill that's unfortunately often neglected).
Otherwise you end up with people writing overly defensive code, and get into a situation like the Stalin-era airplane designers who each made their own parts just a bit stronger then necessary because they feared ending up in Gulag or against the wall if it was "their part" that broke. End result was planes that were so much more heavier than planned that they were essentially useless for the tasks they were designed for.
Of course we have to accept that bugs will be written. But that doesn't mean we shouldn't feel bad about writing bugs! Feeling bad is an excellent motivator to write fewer bugs!
John Carmack put this better than i can [1]:
> Left to themselves, most people have a tremendous ability to ignore their flaws, and it hampers their growth. A bit of shame is often a positive motivation. I am ashamed of a lot of code I wrote last year. I have reasons why it is the way it is, some of which are defensible, but some are just “WTF was I thinking?” If you don’t have nagging bits of guilt about your recent body of work, it might well be a benefit for someone to point out problems in terms that break through your defenses.
I wouldn't call it "feeling bad", but "feeling responsible" for your own bugs. No finger pointing and blame games, just sit down and fix your own bugs, try to not run into the same problem next time, and inform your team mates about what the reason for the bug was and how the fix looks like.
But there's no need for a "formal apology" to your team either. The user might deserve an apology by the PR department, but this shouldn't single out the actual developer who wrote the bug.
193 comments
[ 3.0 ms ] story [ 214 ms ] threadBut I disagree about it being something special to leaders.
And the way he says it, makes it sound like maybe he thinks leaders shouldn't apologize in general. Which I think is a fairly common belief that leaders use to justify not admitting they made a mistake. And is 100% different from not apologizing for bugs. Since as he points out, bugs are not mistakes, they are a normal part of development.
When the manager makes a significant mistake it's not like a bug, it's a poor judgement that probably affected everyone. And sometimes if it's a very negative affect then an apology is warranted. But at least, it's usually necessary to acknowledge when you had people going in the wrong direction.
But there really are some people who constantly make mistakes, and those that rarely do.
There are people on my team where, when there is a bug, I just start looking at diffs of what they have committed because more likely than not, they caused the problem.
I think it's important to reinforce an attitude of trying to become better and people who give a shit when they make mistakes tend to also be the ones who make less of them.
If you can tell the bug from looking at diffs, why was the bug not caught in code review or by tests? Most obvious bugs should be caught this way.
It's like giving a minority report, honestly.
Even if these people suck at coding and are still committing code without someone else double-checking their work, they're not the real problem.
However, I've always believed an apology is important to show humility to my team mates. As a manager, I do not want to appear entitled. Of course, as soon as the error is uncovered, we should prioritize it and fix it. But I do not believe that proceeding to fixing an error is orthogonal to apologizing for it.
We are trying our best to build a company without a guilt culture. In such a position, is apologizing a bad move as a leader? What would you do?
I think context matters too. The industry you're in (what are the consequences of bugs) as well as the part of the codebase (presentation or data integrity) change the acceptability of bugs.
> It reinforces the idea that any one person or piece of code can be blamed for a given failure.
Why is that a fallacy?
Surely, the person making the mistake is the proximal cause for a given bug, and surely all other things being equal personal attitude matters (i.e. a more careless person is going to make more mistakes), but if you zoom out to the big picture you see other responsibilities and more importantly you see opportunities for improvement that vastly exceed the playoffs of "just putting more care" into each commit.
The intent of a "no blame culture" however is to find a new sweet spot on the myriad of tradeoffs an engineering organization does, as an many paradigm shifts we do, it is accompanying with a certain amount of bullshit-sounding maxims meant to be thought provoking and dislodge the readers from their suboptimal local maximums.
So what is the responsibility of the key-puncher if it isn't turning requirements into robust, correct code? Many comments here seem to suggest that programmers are just there to produce some sort of fuzzy first-stage approximation of what the code could be.
In that case, just employ an automated code generator and push the burden onto reviewers.
> So what is the responsibility of the key-puncher if it isn't turning requirements into robust, correct code?
How can it be a responsibility to never make a mistake? Nobody can never make a mistake.
> Many comments here seem to suggest that programmers are just there to produce some sort of fuzzy first-stage approximation of what the code could be.
Many comments here seem to suggest that producing high quality code is a team effort supported by a company or department which values high quality code. If you had mythical no-mistake employees you could do away with most everything else. Why even have tools, tests, revision control systems if you're not making mistakes?
And if you are using tools, tests, etc. as part of not making mistakes, why not reviewers, bug testers, quality analysts, as well?
A code reviewer is able to see that someone is building SQL from raw input, the QA isn't. And if I see that a developer writes that kind of a thing and another person approves it, the first question I have is if they actually read through the code or just glanced at it.
“I really should have checked the x-ray better before I extracted your front tooth. But you know, I’m not sorry. It was an external constraint — there was a queue of patients waiting”.
I can’t believe how low we let this profession sink. I hope in 15 years we will look at thies era like we look at the Wild West today. Exciting from a distance but nobody would really like to live like that.
I believe we need to get the "context" before judging any situation.
Building software should be an engineering discipline with all its perks and responsibilities just like building bridges. We rarely see it and it propagates to the craftsmen who seriously believe that good quality software is impossible to write under the slightest of constraints. Like business pressure.
We had this ability in the past when programming was way more difficult, tedious and error prone. Look what IBM did with their “Clean Room” approach. I’m sure we still have it at many places but we choose to spread the myth that it cannot be done. This call to stop apologizing is the next step towards not even trying.
I think everyone would not deny this fact. Engineering is more of a discipline.
But if you see things practically, this issue of "bad" code may not happen for various number of cases. For example, unclear project requirements, use cases etc., That is where 'context' of what happened helps to better understand the situation.
Building bridges are not done agile and it has a learnings from decades or even centuries. Computer Engineering is mostly agile where you push changes based on changing requirements.
Like on that dentist's case, my hope is that as liability persecutions and lawsuits due to faulty software increase, that will eventually be sorted out.
You did not test it? Apologize.
You were sloppy? Apologize.
In most other cases I can agree with the author.
All I can add is that, as a freelancer, I always explain (like they are 5) what went wrong. My customers don't care about the details but they like it when they know I made an effort of creating good software (bugs included).
Not on every case. I once had a customer who literally removed the "Testing" part of my offer because it seemed to expensive to him.
As testing often takes a considerable amount of time I do list it explicitly in an offer and don't "hide" it in the numbers for the other tasks.
Something you definitely don't want is to sound like it's an inconvenience for you that they're reporting bugs. Bug reports are extremely helpful and you should be grateful that they're taking the time to report them to you.
As a company, by some PR representative, yeah, sure, maybe, but individual developers should not apologize.
Working at Instagram, probably not, since it wouldn't be my job to handle support.
Would you accept such nonsense from your plumber, or your dentist, or your electrician?
Software engineering is one of the most fundamental building blocks of modern civilization, and yet it continues to be so fragile because of this kind of cavalier attitude.
Also, apologizing doesn't mean you have a moral failing. It means accepting at least partial responsibility, and being humble about it. Any good org knows that you don't just design your system with the hope that people won't screw up, you need to design it to work despite some amount of screwing up. But in either case, it is still both an individual and collective responsibility.
Maybe there's a reason you rarely hear one category apologizing and hear the other category coming up with excuses.
Failed you? This sounds either very entitled or sensational. Has software caused you major grief in any way financial or the loss of limbs or loved ones? While this happens, very few people are affected by bugs in such a way, at least confirmed.
Many people have been significantly affected by software bugs. Therac-25[1] is one I constantly remind myself of; it killed 5 people. Here's a list for further reading: https://en.wikipedia.org/wiki/List_of_software_bugs
I'm not sure what your definition of "very few people" is but we'd all do well to aim for 0 adversely-affected people as a result from using software we write.
[1]https://en.wikipedia.org/wiki/Therac-25
> I'm not sure what your definition of "very few people"
I think my definition is about the same as every accepted definition of "very few", the number of people affected by such bugs falls within this definition, considering how many users software has.
> Many people have been significantly affected by software bugs. Therac-25[1] is one I constantly remind myself of; it killed 5 people.
5 does not fit the definition many in anyone's book.
Well, IIRC, there's Watership Down, a literal book, but I assume you meant that figuratively.
...
>I was asking OP if he was personally affected
So 5 people dead is insignificant, but a single person, adversely affected, is cause for concern?
So the analogous question would be whether someone who has worked as a plumber has ever heard a coworker plumber apologize for messing up a job. I'm hoping the answer to that is yes. A plumber apologizing directly to a customer would be more analogous to a company making a public announcement about a bug, which is a different issue than how responsibility is determined within a team.
"Just be more careful" has pretty much proven not to be a successful strategy to reduce defects.
We are pretty good at writing software with a very small number of bugs, it just costs a few orders of magnitude more than normal software.
Imagine if an electricians invented new types of wires for every job they did
I'm not sure why software is the way it is, maybe it will be more like a trade in a few decades...
In buildings with nontrivial electrical systems, shit hits the fan really quickly too. Just look at the absolute mess that is the fire prevention system in Berlin's new airport for example.
Copper, plastic, other things which I have no authority to talk about, were all complex at one point in time
You do make a good point about scope creep, I wonder if the early days of electronic had a similar problem because nobody knew what they wanted or what was possible?
Very interesting to think about IMO
Why wouldn't this be acceptable? They source that crown from a supplier, and apparently there was a fault with it.
This is an incredibly ignorant statement. Have you worked in that profession? Any professional engineering roles?
But that all doesn't really matter because the statement becomes meaningless when qualified with "Depends on the project[...]"
Besides all this incredibility, yes, I do work professionally in an engineering role.
Limiting a statement is not the same as a completely making it meaningless. Of course there are plumbing projects, which are more complex than a static blog. Besides that, I would also like to remind you of the little part of the phrase "I'd wager that". I did not know, that it is an "ignorant" statement, to express ones believes.
I would be interested in some experiments of average developers learning average plumbing vs average plumbers learning average development. I'd wager theres an order of magnitude difference in the results(time spent, quality etc).
>It reinforces the idea that any one person or piece of code can be blamed for a given failure.
That can absolutely be the case. How could it not sometimes be the case?
>Short of malice, this is never the case.
Never? Outside of somebody deliberately harming a system, there has never once been such an instance?
>It gives the impression that, when you wrote the code, you should have written it better. This is a counterfactual that rarely holds up to examination.
Lots of assertions in this piece with no supporting information. There are plenty of reasons substandard code is written by some individuals. Laziness is a big one. Apathy is another.
>It positions shame as the correct emotion to feel about bugs in your code: if you were a better engineer – a better teammate – the bug wouldn’t exist.
Call it whatever you want, but concern is certainly warranted.
>If you’re a more senior engineer on your team, the effects of these anti-patterns are magnified: people see you apologizing for bugs, so they think that they should be striving to write bug-free code. They may feel ashamed if their code has bugs.
People look up to experienced individuals within their profession. In fact, I would argue that it is even more important these senior individuals are able to accept responsibility and act accordingly; that's setting an example, something from which everyone will benefit.
Why not a little bit of both? Can we not think about system and data driven improvement while also at the same time apologize for our mistake?
In the UK...
Dentists: you have to have a BDS or BChD degree from a dental school at a university
Plumbers: have to get NVQ diploma as well as pass Gas Safe accreditation to fit boilers and heating systems, plus get a CSCS card to work on construction sites
Electricians: have to get an NVQ diploma, and to work in construction, they must also have an ECS card to work on a building site
Software developers: [this is left blank as an exercise for the reader]
In fact, there's a (very popular) school of though that the more credentials you have, the less qualified you are to develop software: https://blog.alinelerner.com/how-different-is-a-b-s-in-compu...
I mean in electrical engineering, idea-to-working-prototype involves a lot of design and assembly first. In structural engineering, you just can't build a bridge on your own. It goes on.
While in software development, I can cough up ten distinct features in a day if not more without forward design or testing.
This would be a reasonable comparison if 90% of software developers could reliably produce bug-free code. This would be a (maybe less) reasonable comparison if 75% of software developers could reliably produce bug-free code. This would be a middling comparison if 50% of software developers could reliably produce bug-free code. This is a completely unreasonable comparison given that there have been 0 software developers ever, in the history of software, who have been able to reliably produce bug-free code. So yes, software development is orders of magnitude more difficult than plumbing, dentistry or electrician(-ing?). If you think it isn't, please feel free to jump in and prove all of us wrong.
A dentist doesn't get to run a dev/staging/prod environment before drilling into a tooth. A dentist can't roll back to a more stable previous version of the tooth they just cracked from a mistake that was made. There is no control-z in the real world outside of software devs.
Also the environment in which this happens is completely irrelevant.
By the way. A dentist gets to try on fake tooth before drilling a real tooth. They actually don't have a choice, that's part of the mandatory training. Doctors gets to train on real people too, medicine schools go to great length to procure bodies for training purpose.
In programming, one would be lucky to get any sort of test environment.
Are you saying you went straight to writing production code before you ever studied any software engineering? Sounds unusual for practitioners of an industry endowed with such a critical responsibility.
A dentist can easily kill someone through malpractice... There's no code review, there's no test environment, and there are always difficult (often time-sensitive) decisions to make and difficult people to deal with.
If all you know is estimating story points and merging pull requests to make your burn-down chart look pretty you probably shouldn't be so confident that your profession is more difficult or more superior than one that you have no understanding of.
Medical staff have the equivalent of code review and test environments. Non-routine operations have always multiple people in the room. They have bodies to practice on.
Do yourself a favour and drop the attitude. Medical and software industries both have some very difficult jobs.
This is quite an exaggerated point; the number of people imprisoned for making mechanical, civil, or electrical engineering errors is very close to zero in the US.
I think more programmers have actually been charged than any other field, although I can't quote a statistic to back that up.
A hammer that melts in a furnace is considered a bug?
The possibilities and test paths a piece of code can possibly take exceeds a very large number in a complex software application.
Zoom should work and gracefully fail if there is not a single microphone device found on the client’s PC.
I agree but this point is orthogonal to the main point. Plumbers and electricians build to code in an environment where physical reality remains constant and they operate on expected inputs. A civil engineer is not held to account for a failure if gravity suddenly reverses itself. If I shock a circuit board with static electricity and it fails is that an electrical engineer's fault? If I put 20,000 psi through a household plumbing system and it fails is it the plumber's fault? These designs are only expected to operate in a narrow range on a small number of expected inputs / variables and that is easier.
Software can basically expect infinite permutations of inputs at any time and is expected to give good outputs and no bugs 100% of the time?
“Thanks for reporting this, I’ll take care of it” is accepting responsibility while avoiding a morality judgement.
Bugs are solely my responsibility: there is no one else to blame. And bugs mean that my customers paid money for a product which did not work as intended (certainly not as I intended it to work). I feel it is definitely right to apologize for bugs!
I disagree with the author: if I worked at a workshop and dropped a hammer onto my customer's foot, I should apologize. No matter what the reason: I could be clumsy, or you could say that no one could be expected to never drop a tool — still, an apology is in order, because I did drop that hammer, and it made my customer's day worse.
Therefore, I always apologize for bugs.
What I did, however, learn to do, and it is quite important, is not to apologize when you are not sorry. An example is when a customer is unhappy with the pricing, or complains about a missing feature. Many people will begin a reply with "I'm sorry, but…", and it's the wrong thing to do. I only say I'm sorry when I really am.
"I made a mistake when i dereferenced that null pointer"
and:
"I'm sorry i dereferenced that null pointer"
I absolutely agree with you that the ritualistic part is pointless, and perhaps harmful if it is performed instead of actually fixing the problem (see also "thoughts and prayers").
But i wouldn't want to stop doing the acknowledgement part.
I very rarely apologize for a bug at work. I don’t expect others to apologize to me either. Apologies should be reserved for personal/relationship things, like forgetting to invite somebody to a meeting.
But you first need to believe it is in your own power to write less buggy code. If you can identify patterns that frequently produce bugs, you can try to avoid them (e.g. copying & pasting: very buggy!). You can identify, isolate, and document footguns and complex API. And you can be proactive in simplifying systems when feasible to do so.
Don't beat yourself up if you accidentally have a bad commit, but don't excuse sloppy behavior. Demand better from yourself. Don't produce garbage software.
The company could apologize to customers if needed. But it would be insane to make the individual developer apologize to customers.
But I read this as though the author is saying apologies to fellow teammates shouldn't be necessary, which I do at least in part agree with.
Should we expect for them to go into a full-on dogeza Japanese kneeling bow apology? Of course not. But there are many cases where a person glaringly half-assed something, cut corners, quietly committed something directly to master. And yes, this person should feel some kind of regret, or should be brought to stop such behavior because it is extremely destructive and is totally avoidable and caused not by overworking or accident but by persistent carelessness and neglect, so it cannot just be cheerfully swept under the rug.
... where a developer was given 30 JIRA tickets each "sprint" and whose "performance" was evaluated entirely on how many they completed.
How does the old saying go? The road to where is paved with good intentions? You could engage in moralizing against "carelessness" and "neglect" and presuming that a teammate somehow cares more about watching Netflix than doing their job (which they presumably outperformed other candidates to even get), and maybe that will make you feel better. You are the good, careful, model employee, and that other one -- they're careless, slovenly, apathetic and lazy. But if that's the case, how did they get past the door in the interview process? If the whole company is like that, why do you work there as opposed to a company where the bar is higher? Something does not add up. In all likelihood, your explanation is a rationalization and not the most obvious answer, which is that your process could be improved but it hasn't been because such investments are not viewed by your companies executives as improving the long term bottom line to be worth the short term investment cost.
Take a look at history and figure out how companies in the industry have solved this problem before by building more bulletproof runbooks, processes and tools. These companies, as they approach enormous scale, necessarily have to determine how to deal with employee reversion to mean. It turns out, surprisingly, that process eats good intentions and care for breakfast. You'd be surprised at how quickly those good intentions become useless if your company is successful and you hit hypergrowth and scale. It's ironic that for a profession where it is so tractable to automate away mundane or repetitive tasks, where we study spacetime complexity in data structures and algorithms, that we have so many practitioners that default to witch hunting and moralizing and seem unable to apply spacetime complexity or procedural analysis to their own software development lifecycle. I expect to see this trend change as our still young industry continues to mature.
The problem is that the person(s) for whom I had to add these baby bumpers not to do input_array[0] will - and has - screwed up much bigger on everything that is slightly more complicated than that, too. Because if you are an engineer and you don't care, then no amount of padding around you will make you curious, careful, detail-oriented, non-lazy and capable of tackling hard problems that life throws at you every day at a tiny startup.
There is an explanation to this discrepancy. Risk = probability of fault * cost of fault. Programmers usually work with the data. Data has a pretty very unique property that it is easily and cheaply copied and restored. It also means data can be replicated more easily, which allows to build systems with large amount of redundancy. Therefore, price of a software bug is usually orders of magnitude less than the price of breaking something in the physical world. This allows to drive up probability of fault parameter and allows for much faster evolution and less regulation.
Many structural engineering projects are built with a safety factor greater than 1, which allows for - as you'd imagine - safety. In software, it could absolutely be done that every function has explicit validation checks for each input, and that more safety, error handling, and checking is written. The economics just don't make that viable though except in the most extreme situations where this is required. In aerospace, with the exception of the recent Boeing debacle, there's been very few software related issues (that I know of) relative to the number of CPU cycles run and flights taken - I'd be willing to be there's far more mechanical issues that cause grounding and maintenance cycles.
Now look at civil engineering and infrastructure. How many roads do we drive over daily filled with potholes? How many bridges get patches and repairs every few years because of concrete cracking and rebar rusting? (I live in Quebec, so there may be bias in this statement). Maintenance like this is just paying down tech-debt - the same as in any field.
Software absolutely can be built to a high level of robustness (and it is) - it's just not economically sensible in most cases.
The damage caused by a software bug can range from nil (even negative, since a bug might have desirable effects) to incalculable. Consider a bug in software used for landing SpaceX boosters... Or a bug in a trading platform. Or a bug in an automated medicine dispenser. Or...
You can compare the work of a programmer to the work of a civil engineer, but it's true that it's difficult because most bugs cost little, much software is not positioned to cause much damage if buggy, software bugs are so prevalent, and it's so difficult to write bug-free code. But the cost of replication of software has nothing to do with any of this.
Replication is the cornerstone of fault-tolerance. When logic is written in high-level languages with some minimal good practices, most SW bugs result in crashes rather than data corruption, hence data replication plays important role in protection against bugs as well as HW problems.
> Consider a bug in software used for landing SpaceX boosters... Or a bug in a trading platform. Or a bug in an automated medicine dispense
I specifically mentioned: "Programmers _usually_ work with the data. <...>" In the examples you listed above software manipulates physical world rather than pure data. I thought it was too obvious to explain.
Should you apologize for bugs? No. But if you're doing professional quality work, (as opposed to prototyping or researching,) a large portion of your time (as a developer) should be writing automated tests (like unit tests) to test the code you write.
Why doesn't your dentist mess up 1 out of 5 interventions? They have gone through years of strict training and there are giant liabilities in place.
In development? A lot of self-taught developers, who can always jump to a next gig because of never-ending demand. This sets up the scene for a totally different scenario - never-ending stream of bugs.
If you want your team to have less bugs, then as a manager make sure you have a process in place that optimizes for that. Start with your hiring process and add requirements for code quality practices. Then build a process that includes strict commit guidelines, documentation, code reviews and QA process. Most importantly, add time to estimates that ensures all of the above is checked properly. After all that is in place, define in a clear way some sort of liability for bugs.
Your dentist goes through all of that, why not your developers doing arguably equally important work?
To me it seems that failures happens pretty often, generally due to incompetence, limitations in the science regarding the problem or insufficient resources. Seems very similar to bugs to me.
The question is how bad is failure, obviously if someone's health is at risk, you have a much bigger incentive to have processes on the safer side.
Maybe you feel "bugs" are more frequent in software because the software you're referring to has an incentive towards innovating rather than stability.
I'm sure if you studied software used in critical systems, the failure rate would be more in line with a procedure at a dentist. You can't justify the kind of cost that kind of development has for most apps.
The world needs enough software, and the world's requirements for quality are low enough, and the average developer writes reliable-enough code, that in most cases money can be made without special attention to quality.
In some domains, quality is paramount. I imagine that quality and bug mitigation are the primary concerns of someone who manages, say, an X-ray machine firmware project, or avionics software. Software for these purposes that is not extremely reliable cannot compete in that market. For other kinds of software people are paid to write -- like social networks for pets -- reliability is a much lower priority.
Certain kinds of software needs to be rewritten constantly to meet the demands of highly-evolving markets. Games are an example of this. Reliability is important, but not paramount, because the underlying business can only continue to compete by at some point stopping work on the reliability of existing games, and moving on to developing new games.
I'm sympathetic to the respect you have for the value of high quality software. But the reality is, the value of quality varies dramatically within the software industry -- much more than it does within, say, dentistry, or medicine, or aviation.
Fewer bugs isn't necessarily the primary concern of those using and buying any particular kind of software.
I mostly agree with the rest of your post.
Code is _always_ a flawed approximation of an ideal system. You can move bugs around, but you cannot get rid of them: software _is_ bugs.
> * If you’re a more senior engineer on your team, the effects of these anti-patterns are magnified: people see you apologizing for bugs, so they think that they should be striving to write bug-free code.
Imagine writing this. Imagine actually sitting down at your keyboard and writing down that people should not be striving to write bug-free code.
I agree with you, it's outrageous. What an awful article. I get the point (you should have robust and multi-layered quality systems) but just the complete rejection of accountability is embarassing.
The point is to not have a "shame culture" for writing bugs, because they are normal part of the workflow. Instead focus on efficiently identifying and fixing bugs (efficient debugging is an important skill that's unfortunately often neglected).
Otherwise you end up with people writing overly defensive code, and get into a situation like the Stalin-era airplane designers who each made their own parts just a bit stronger then necessary because they feared ending up in Gulag or against the wall if it was "their part" that broke. End result was planes that were so much more heavier than planned that they were essentially useless for the tasks they were designed for.
John Carmack put this better than i can [1]:
> Left to themselves, most people have a tremendous ability to ignore their flaws, and it hampers their growth. A bit of shame is often a positive motivation. I am ashamed of a lot of code I wrote last year. I have reasons why it is the way it is, some of which are defensible, but some are just “WTF was I thinking?” If you don’t have nagging bits of guilt about your recent body of work, it might well be a benefit for someone to point out problems in terms that break through your defenses.
[1] https://www.wanzafran.com/posts/2020/shame-and-code/
But there's no need for a "formal apology" to your team either. The user might deserve an apology by the PR department, but this shouldn't single out the actual developer who wrote the bug.
> It reinforces the idea that any one person or piece of code can be blamed for a given failure. Short of malice, this is never the case.
Never the case? Never?