Ask HN: How do I feel less guilty about bugs?
I'm a DevOps engineer and I have been working at my current company (a startup) for less than a year. For context without giving too much information, we make enterprise software that makes certain jobs a lot easier through automation. Our customers use our app basically as their full time job, many tabs open for hours at a time. Our customers are people who used our trial, then begged their bosses to sign up for a yearly plan with us, which costs 5 figures.
I feel guilty when our customers have repeated bugginess in our app. For example they might spend an hour working on a task and then suddenly our code runs into an exception and their data is not saved properly. These issues are rare from a numerical standpoint, we track exceptions and we have a low percentage of errors, but it can still be a serious negative user experience. We have even had a customer recently complain about a bug, which we fixed in a few hours, but the next week they had to complain about the bug again because it was back. We have also had complaints that they were pulled into a meeting with their CEO or VP or whatever and they needed the app and it wasn't working properly and it was really embarassing. One customer has requested access to our API because they want to write a script to make frequent backups of their data because they are nervous about our data integrity. I feel like we owe it to our customers who had advocated for our app to their bosses and made a significant financial investment to have an experience that is as polished of an experience as possible, since it would make them look really bad if our app is a buggy mess.
This is my second job out of school. At my first job, I'd search Twitter and see people complaining about our app being buggy and I would feel bad, but less strongly than I do here. That is because we had a QA team triple the size of the DevOps team that triaged bugs. Whenever we had an incident, I figured if it got through a whole QA team then it must have been a tough one to prevent. Nevertheless, I did have at least some bad feelings about the complaints.
I have casually asked my boss how they feel about all the random bugs in our product. They feel that as a startup of our size, our number 1 priority is making sure we have enough features. Our prospective customers tell us things like "we can't sign up until you support X number of features," and my boss and the founder believe that our number 1 indication of success is the downward trend of X. I understand what they mean (it doesn't matter how bug-free your code is if you run out of funding and go out of business) but I still feel guilty. My boss also is correct when they say that these issues happen less than 1% of the time. I don't feel like I have buy-in to make any changes to this process. Alerts are ignored, and some people don't even have PagerDuty set up, their alerts go nowhere.
I wouldn't call myself a perfectionist, I write bugs all the time. But when I see the bug I feel a responsibility to triage it and fix it.
I think about people who work at big companies and all the complaints people have about them, like becoming demonitized on Youtube for no reason or getting their app banned from the App Store for no reason. Those things would make me feel really bad working at those companies, especially if there was no way I could address them. But after doing some thinking, I realize the majority of the employees there probably don't feel this way. So it's probably an issue with myself that needs fixing, right?
92 comments
[ 2.8 ms ] story [ 164 ms ] threadIt's a good thing to want to develop a good product, and it's good to take pride in your work. But if your goal is 0 bugs, it's just not possible.
Don't feel bad about the bug. Bugs happen. But the company should feel bad about the process that let it come back.
Can the process be fixed? That should be top priority.
So given that, I don't feel very comfortable pushing hard for any process changes. I already feel like I'm bordering on annoying.
(Alternatively. you can try to be the force that turns this ship around -- but I don't recommend it.)
* Trying to figure out a way to improve processes at your current job. (There might be a way that isn't or doesn't feel like "pushing hard". OR, your current job might just be completely uninterested and this is never going to happen).
* Finding a new job. (The job market is as good for software engineers right now as it ever has been for anyone ever, pretty much)
* Deciding to live with it
In the sense that you care about the product.
Some people don't, they're there for the paycheck and don't uphold their end of the employment bargain. Or justify it by saying they're "too busy".
Some people might not litter on purpose, but occasionally something falls out of their car or bag, and they say, "Oops. Can't be bothered. It goes against the principle of leaving things better than how you found it.
Apart from ensuring that all parts of the code are covered via appropriate tests, and being responsive to the customers' complaints, there isn't much you can do.
Think of this in contrast to working at a place where, if there is a production bug the lead’s first instinct once finding said bug is to git blame who did it.
Leadership should be encouraging breakages as long as there is a review of the bug, remediation, and a fix put in place to prevent it from happening again. That should be rewarded in any performance reviews.
There's a classic Joel on Software post where he talks about making workflow changes as a grunt. An important point is that you want to get your daily work done first:
https://www.joelonsoftware.com/2001/12/25/getting-things-don...
"… I also knew that making a good first impression was crucial. So I allocated the first seven hours of every day to just writing code, as was expected of me. There’s nothing like a flurry of checkins to make you look good to the rest of the development team. But I reserved another hour every afternoon before going home to improving the process. I used that time to fix things that made it hard to debug our product. I set up a daily build and a bug database. I fixed all the longstanding annoyances that made development difficult… Slowly, the process got better and better."
But also pay attention to your company culture. A "blame culture" is one of the worst enemies of good software/product. If your management work by making people feel guilty, you can't have candid conversations about your problems. You'll never find what you need to fix, because you won't be able to discover the real problems.
I've worked in a company where the "blame game" mode were always on. It was terrible. People didn't want to fix a problem, just pass it to the next guy. You'd never find the original cause of problems or have your management invest to solve it.
If you have a culture problem, don't let your employer make you feel bad. Take care of your mental health. A job change may be the best option.
But wow, some of this post is incredibly toxic and passive aggressive. Create your own personal bug tracker, then refuse to fix any bugs unless everyone else uses it? "Neutralize the bozos" by sending them incessant bug reports, until they're no longer productive? Can you imagine working on a team like that?
I totally agree that you should incrementally work towards hardening your systems. Inasmuch as it's something you can fit into your workday, I'd say that's just part of your job.
- The Tao of Programming: https://www.mit.edu/~xela/tao.html
I had a Google spreadsheet that I put together two weeks ago. Used some custom code in "App Script", a few very simple functions that made some of the spreadsheet formulas much easier.
A week later they stopped working. Google app script was just not working for a lot of people. It could have been embarrasing because I had shared this spreadsheet with several hundred people to communicate some real-time information, and now it was not working. But, that's software. It always lets you down eventually, and never at a convenient time.
Here's the specific issue, still unresolved after many days. This is Google. Google. If their software is buggy, what hope does your small startup have?
https://issuetracker.google.com/issues/222342097
It's part of the reason that I hope to as much as possible evict computers, phones, and the internet from my life when I retire. I feel like Charlie Brown kicking the football over and over, thinking that this time she won't pull the ball away.
short term, I used to feel this way as well and would try to figure out how to fix all the things, just to realize it cannot be done. Not to pass the buck but it's the person above you whom probably has more information about whether or not these bugs are going to sink the ship, if the company needs to create less bugs someone will need to prioritize ways to prevent them
2) increase the effort you spend on tests; write less code; implement simpler features; code review every change. Don't listen to anyone who tells you it is ok to test less; do what will make the company the most money long term; it is better to work somewhere else than to work for someone who doesn't care about product quality. Alpha/Beta test more.
3) implement a checkpoint/rollback system (using git can save you effort) Save config changes to a temp location so it is never lost. Allow test configs to be validated and then allow the user to roll out the new config gradually. Save the old config so users can rollback easily. Implement a monitoring system that can auto detect failed rollouts and trigger rollbacks.
Unfortunately the extent to which we can take professional pride in our work is limited by priorities above our pay grade.
1) they don't have to use your software. they could write their own, or choose another vendor, or do whatever your software automates manually. if they are using your software, they have chosen that option because, on balance, it meets their needs. that's not to say it's perfect, but it must do more right than it does wrong.
> For example they might spend an hour working on a task and then suddenly our code runs into an exception and their data is not saved properly.
Nobody should ever do this! Spending an hour working on unsaved work is not sensible. Either your software should save it along the way, or they should do their work in some other place (notepad?) that saves it incrementally, then copy and paste it in as a last step.
This is an example of where you need to step back and see the whole picture - it shouldn't be a case of doing unsaved work for 1 hour then saving with 100% reliability, it should save incrementally. Even if the save code were 100% reliable, they would still lose their work if they lose power, have a network disconnect, etc.
Unless you step back and understand why they are choosing your application, despite these problems, you won't be able to see the most important aspects of the business.
Guilt is probably not going to contribute to that.
> if they are using your software, they have chosen that option because, on balance, it meets their needs
This doesn't hold in any enterprise setting.
Coming to this case that OP describes, I think OP, you have high empathy and are passionate about your product. Since you're likely an entry-mid level engineer, sometimes others have more context of the business setting.
It would be good to implement ideas like: 1. Tracking crash reports 2. Coming up with a test framework that helps avoid regressions 3. Coming up with a script that can analyze errors, which reduces manual ops. 4. Setting up a job that compiles these errors and send it out over email every week, so managers and business automatically learn about what users are seeing.
You are feeling guilty because you care about your (your app) customers, this is not (IMHO) an issue, it is only you being a "good" person (like hopefully most people).
But 1% (if it is 1%) can (still IMHO) be a lot.
Let's say (for the sake of reasoning) that the app is about invoicing.
One of your customers used to be able to prepare 8-10 invoices per hour and now, thanks to your app, is able to make 16 instead, BUT every 6 hours or around 100 invoices one fails, the work done is lost and it takes "a few hours" to solve the problem.
Hardly an increase in productivity.
0.01% or 1 in 10,000 would be probably acceptable in this completely hypothetical scenario.
Yep. Perhaps an unpopular opinion but developers should care. They should feel bad about their bugs. At the same time, recognize that shit happens and you're not going to be perfect. But take that negative energy and use it at motivation to reflect on how you made that mistake and what steps you could take to avoid it in the future.
Also your programming technique and style of programming influences the amount of bugs.
Generally to reduce bugs, go with a functional programming style and make sure the language has robust type checking. Functional has a bad connotation in some circles so another way to think about it is to use immutable variables as much as possible. Avoid mutation wherever you can.
This is the first step. The next steps are testing and QA, but most teams have that side covered.
It's better to focus on clear and simple code styles and architectures. The goal is that when you're debugging the code or explaining it to someone else, you don't want any situations where things don't make sense, or something happens in a weird order for no apparent reason. Immutability can help here, for sure, but FP is a double edged sword. Sometimes it's fantastic but at others it makes your code into an unintelligible mess.
So yeah: Don't go rushing to rewrite your perfectly functional Java/Ruby/Python app across to Haskell or Clojure. It won't help you much at all. Rather, think about how you might use some of the ideas like immutability and function composition, to make your Java/Ruby/Python app easier to work with.
Basically immutability serves to create more invariants and reduce complexity of the program such that it is much more predictable.
I'm advocating exactly your conclusion hence the reason why I specified that immutability is the keyword, not monads and other advanced concepts associated with functional programming.
Do everything you're supposed to do as a software engineer; then you won't feel guilty about the remaining bugs that slip through.
Also, don't feel too guilty if you break something which someone else developed that wasn't (1) documented and (2) covered by regression tests to catch the breakage.
When users report a bug, don't feel guilty if they are relying on some undocumented behavior, and it's not working they way they imagine. (Of course, security, reliability and robustness to bad inputs shouldn't have to be documented; that's not what my remark here is about.)
What keeps bugs from happening is processes, not programmer conscientiousness. Processes begin with a test suite, and enough time spent on maintaining the test suite. And generally, enough time spent on managing bugs.
The way for the software to have fewer bugs is for the organization to spend more resources on processes to reduce bugs. If they are choosing not to do that, perhaps thinking they won't gain/lose enough business over it to justify the investment... well, your boss has already told you that they think the number of bugs you have is appropriate for business requirements and there is no reason to invest in having fewer. So, you can accept that, and give up your feeling of responsibility to make things different (your boss has already said they don't want things to be different), or you can look for a different job, I guess!
I admire you for feeling guilty for putting out crap software, and don't want to tell you you are wrong. Too many of us get paid to put out crap software. I think we are responsible for making people's lives just a bit crappier having to deal with our crappy software. I don't want to tell you to become a soulless automaton who doesn't mind producing crap that makes people's lives crappier as long as you are getting paid.
(Alternatively, maybe it's not crap software? Even high quality software sometimes has bugs. Maybe you are being overly critical? But there is so much crap software out there, that's not what I'd assume).
But it sounds like there isn't a lot you can do about it at this job.
Or... is there? We can brainstorm. It's not spending your time doing things your boss has told you you should not spend time on, though, to try to save the company from itself. That won't be good for your career, or your personal stress level, and also won't help. Whatever it is will about trying to pitch your boss/team on new processes.
While maybe it's not sustainable, or maybe it doesn't work for large companies, if you're at a startup or small company, conscientiousness absolutely plays a large role. That's not to say that process cannot help reduce the mental burden of developing though by reducing the need to be conscientious.
But I (who has worked my entire decade+ career on very small teams/companies/projects) think that, past a certain point of not being just apathetic/irresponsible, "trying harder not to have bugs when I write the code the first time" has much much less effect on outcome than code review (including even code review by yourself, but giving yourself the time to do that, including some resting time in between when you write and review your own code), test suite, QA, error monitoring (with allowance to spend time diagnosing and fixing any errors reported; actually just having that allowance in general, otherwise bugs keep building on bugs), etc.
Plus if you think bugs are a result of "not trying hard enough" or "not focusing hard enough", you just stress yourself out. Even more so on a small (or especially solo) team. There are ways to create high-quality low-bug software without stressing yourself out. I don't think they include a model where bugs are your fault for not trying hard enough.
But sure, everyone should try to write good code, and if you don't care about what you are doing and don't even try, you will have more bugs, it's true.
That said, it is a good idea to look into your bugs, and figure out why they happened in the first place. With this you might find a way that allows you to avoid them. Good luck!
Thank you, I'm going to ask about this in interviews from now on. If they say # of features is more important than quality, I'm not taking the gig.
On a tactical level,
First work with your development team on setting up temporary stores(autosaves) workflows either at the browser/ app level and/or in the backend with redis/Kafka etc.
If your users can think they can safely refresh and un submitted forms would be there , then they won't worry that much.
Secondly from a devops point of view, there is a lot of things you can do improve tooling for your teams. Canarying deployments, CI/CD automation, performance monitoring (APM) robust traceability and observablity, better error tracking and deeper data captures(session /request replays), to help solve bugs faster, production like environments with good mock data, failure point simulations with network etc, isolated dev environments (virtual clusters) . Ability to run code in a PR etc. [1]
The idea is you catch bugs before it goes to users or before the user notices.
----
From a biz perspective your boss will not change his view, even with some bugs you are better than the market and decision makers in companies buy a product for features, not because it is bug free , they don't care because the buyer is not the user. Your boss will only start caring if you start losing customers over bugs.
There is no right answer, many times your boss is right, sometimes not fixing bugs can explode technical debt and slow your growth or kill the product too, very hard to know that balance.
---
[1] these are things devops could do without product engg involvement for the most part, writing units tests or fixing architecture is not a practical initiative for you to take up although it may pay off more .