Ask HN: Why is software quality an afterthought for many people/companies?
I've seen many discussions around this topic lately, but I'm particularly curious to understand why most people think that software/code quality is something secondary and could be addressed late in the process, for instance with peer review.
Why isn't the idea that software quality starts way before you write any line of code the predominant mindset amongst engineers / the industry?
I have the feeling that most companies don't hold discussions about what software quality means and how it should be measured.
To which extent do you agree or disagree with this feeling?
99 comments
[ 3.2 ms ] story [ 145 ms ] threadIn fact these days, people kinda know what they want, but really only understand what they want when they have something in front of them. I'd go so far as to suggest you can over-engineer a solution too easily and then spend a lot of time refactoring it. YMMV
This is also a large factor in why software projects tend to overrun both in terms of time and budget (the other large factor is bad project management).
And companies are built to optimise other stuff.
So in regards to your question, it would seem that the market reality is that a lot of the time, it is better for a company to have a quickly-cobbled-together piece of software that mostly does what the customers want (and maybe get to the market first) even if it is low-quality, than it is to have a piece of high-quality software that does less, or is finished later, but is maintainable, and potentially scalable in the future (which you'll never get to enjoy because the worse-is-better people already conquered the market).
The state of the software is currently much worse in my opinion.
Quality is not easily quantified while the price is. Metrics at the customer end are hard to collect (it requires software development too, raising the costs), and in the current state of the art, it also requires having customer support staff too which is still costlier. As a result, quality does not even gets quantified properly. A natural result of which is quality reducing below what would customers desire.
This isn't much different than where quality of MP3 players, laptops and smartphones was headed. Perhaps quality then was being measured just by percentage of customer returns, not by customer satisfaction. Steve Jobs then changed the game. Apple's products would just "feel right" to the customers. Apple iPod took over the market even after being much costlier. It then took a couple years for the rest of the laptop/smartphone manufacturers to catch up.
"Feel right" is definitely a kind of quality that software can compete on. I'd probably put Chrome in this category (relative to other browsers that were around at the time it launched). Sublime Text, maybe. Blizzard games (especially those of a certain era).
Note, though, that this quality is principally about doing what the users want and being pleasant to use while doing so.
It's something that you can definitely focus on deliberately in your work and projects, but I'd argue that a lot of the current mantras that get recited when software quality comes up (test coverage, continuous delivery, maybe even code reviews) are not especially helpful for achieving this kind of user-perceived quality. Maybe even a distraction, in some cases. Getting your code in front of users and listening to feedback can help, certainly. But having a strong, clear, vision of what you're trying to build in the first place might be even more important. And I don't think that's something that's achieved with tools and processes.
Sadly, things sometimes get rushed out the door and it's not until some time has passed that they realise the enormous tech debt they've incurred.
That means that the number one consideration for the software is profitability. For internal-only software, this means that cost is the prime consideration.
In support of that, often software startups are trying to capture a winner-takes-all market, so time-to-market is critical.
Thirdly, consumer protection law is weak in the US, and product liability is almost nonexistant for software everywhere. The cost of failure is very low even if you leak all your customers' data or your product ceases to work after 18 months because you've "pivoted".
Fourthly, a lot of software is ""free"" or ad-funded. This further weakens the cost of failure.
There are techniques for delivering extremely high quality software. Few sectors of the industry care about them because it's not required and is unprofitable, but the aerospace people can usually get it right and the security people can usually get it right (when dealing with security products, not general purpose junk like Flash).
The automotive industry is kind of on a boundary. The Toyota "unintended acceleration" bug revealed some tremendously poor quality software. This is one of the main worries about self-driving cars: how minimal is the quality assurance going to be?
Dan Ariely (author of the book Predictably Irrational) called allowing free apps on AppStores to be a mistake made by the industry. The customers have now gotten used to free apps, making it harder for the industry/developers to offer better quality.
I don't agree with the idea that free apps are inherently bad - that would rule out Open Source / Free Software, and it would also put the boundary vs "free" web pages in a strange place.
"Free"+adsupported and "Free"+IAPs have certainly produced some strange and terrible incentives though. As has the incredibly bad discovery process on app stores.
I do find open-source software to be generally lower in quality than paid products. Though there are many exceptions at this point in time where open-source is rather highly superior.
However, even in cases where someone could produce a better paid software for an open-source alternative, they practically cannot as it is hard to compete with free.
For instance, in the latest iOS there is a stupid bug where the calculator blocks the buttons if you press them too fast. So if you enter 1+2+3 the display will show 23.
In open source this would be trivial to fix. In closed source you have to wait for Apple to do its implementation, testing, distribution.
In open source you would have a customizable calculator with a million generally useless buttons though because it’s so easy to add them.
I won't spend $10 on an app sight unseen because if it's crap or even just plain doesn't do what I need, there's no way to know that aside from trying to parse it out of the reviews that the developer probably bought from a spammer.
I will spend $10 in a heartbeat on an app that I've tried for two weeks and don't want to go back to living or working without.
But AFAICT, Apple explicitly forbids that business model in its store. Dunno whether or why Google Play apps don't use it more, tho.
Something like this really should be a store-level feature
> In support of that, often software startups are trying to capture a winner-takes-all market, so time-to-market is critical.
I think we software engineers should embrace this reality, and learn to live with it. Your employer is willing to spend years to develop top-notch quality software? Great, you can employ all the software engineering best practices. That's not the case? Well, we should have a standard approach for gracefully handling strong time constraints without completely giving up on quality.
One way this unhappiness with our lot is expressed is Technical Debt. For me this just cognitive dissonance on the developers part trying to reconcile / justify why the codebase is a mess and why all those shortcuts were taken to get the thing shipped. If you want to pursue your craft and deliver a result you would be proud of then probably commercial software companies are not for you.
Well all might be great writers at heart but if all the employers want is a pool of people to write pulp fiction and romance novels the sooner we get over it the better.
Of course one solution to this identity crisis is sort of mapped out with Erik Dietrich's Developer Hegemony, https://leanpub.com/developerhegemony but it might take a while to get there.
Did Toyota ever recall the affected ETCs?
Are you refererring to adherence to standards like Misra and DO-178B or something else?
The Chinook fiasco is, like most quality issues, really a project management fiasco. The decision to do special software rather than get Boeing to do it, then a series of oversight failures on known problems.
If the point is to reduce the number of errors then it helps to at least have a checklist of the errors, and someone reminding the team of the checklist. Checklist process is one of the easiest quality and safety tools to implement.
Having a premade checklist that makes sense in the form of a process plan makes things easier.
I'm not sure why you are mentioning "stupid tools", whatever they are - they would not even be means to an end.
I'm not sure being ad-funded weakens the cost of failure. Losing users or having down time impacts revenue immediately if you're ad-funded.
Anyway, this is tangential to this thread, so I won't go on about it.
Reading along with HN for a few years now is making me more terminology-oriented in areas of coding and capitalism, two of the major themes.
So to me it's only software if it's intended for sale (or profitable distribution), otherwise it's just computer programs.
Same with hardware, if it's not built for profit then it's not wares, just equipment.
Nothing wrong with building in quality for profit, but you may not be able to compete with low-quality-focused operators, especially ones which are strongly established.
First, to extend what you said about startups, you generally aren't totally sure that the market is really there. If it turns out that nobody cares about what you're building, it doesn't matter what quality you built it with. Therefore, as long as it's cheaper to build it with lower quality, startups are rational to build it with little concern for quality.
Second, Toyota: I'm not going to be any kind of apologist for Toyota's horrible software. From what I read about the situation, the way it was written was appalling. And they rightly got a lot of heat over it. (Arguably, they should have gotten more.)
But I wonder if the quality bar isn't being set too high in this situation. If Toyota didn't implement things in software, they would have had to implement it in hardware (either mechanical or electrical). That hardware would have some failure modes and failure rate. If the software has a lower failure rate than the hardware, that's progress, even if the software has a higher failure rate than it should have.
Our discussion of the Toyota flaws is colored by the fatalities. Still, hardware flaws can kill people, too...
Having worked for an AV vendor, I assure you that is not the case. Just check Project Zero[1], most of them do parsing of complex binary formats in kernel mode, 'nuff said.
This is just one example, but all major vendors have had issues:
[1]https://googleprojectzero.blogspot.ro/2016/06/how-to-comprom...
-----------------
OTOH, there are a few individuals who show a great deal of care about software correctness. Daniel Bernstein comes to mind, but many other people are offering big bounties for their personal projects, and have a track record of delivering correct software. But even in cases such as these, there are probably some hidden bugs in there, because of the inherent complexity. Nobody has the time to verify the fine interactions between the compiler, OS, libraries etc.
At the end of the day, if you want higher quality software, you have to incentivize it, as others have mentioned.
Quality and security become increasingly important as we depend even more on software systems for essential functions such as cars, power grid management, agriculture, etc. Unfortunately, this situation is all too similar to how many opt for the emergency room over preventative care.
We should also consider that many businesses wouldn't exist if not for lax quality requirements for software products. How many product V1s are chock full of bugs and exploits, and to what extent is that okay? What about open source? As usual, it's pretty complicated.
When software is created without a profit motive, then it is to coolness of the idea that motivates the creators. Focusing on quality would only slow down the "creative" process.
Software is not made by grownups. And for the most part the development is not managed by grownups. The problem is that so many can get away with childish behaviour.
I'm unconvinced by this argument. Maybe compliance with "best practices" would improve a bit, but I see very little evidence that the results would be good for the average user.
I'm pretty certain, for example, Rich Hickey was having fun at least some of the time when building Clojure -- his belief in it is palpable whenever I've seen one of his talks -- yet it's an incredibly well-thought-out, solid piece of software.
Lots of classic games were passion projects of an individual or a very small and close-knit team. Whether that's "quality" I guess depends on perspective. For me, a lot of classic games very definitely were, though (and a bug or two doesn't necessarily detract from the overall experience).
But this is certainly not the majority.
Basically it comes down to management that is willing and able to tell a client no, or convince the client to budget to do things right, and not management like my current company, which has in the past threatened to disallow even unit test writing and code review as slowing the process down too much.
"Code quality is time and money you're saving your future self" is an argument that only makes sense to people who write code, apparently, until you actually lose a client to avoidable problems.
Up to a point, as soon as you start losing market share to competitors because your customer complains your application crashes every other day. Suddenly, the focus switch back to quality. (Until the next cycle).
What is frustrating as engineer is to release something you know you'll have to fix in 6 months after a customer's complaint. But maybe from a sales point of view that was the right decision.
When developers talk about "quality," they mean tech debt. Addressing tech debt is problematic for businesses because it's something that never ends. You allocate one month for tech debt and the devs will ask for two. Allocate two and they will ask for three. There is no agreed-upon standard at which devs will stop and say, "Now our code is clean."
Add to this the fact that there are many developers who just always seem to have an agenda about the code they're working on. They never work a project except they're dying to add some pattern or change some aspect of the code, even if it's something that they used to favor a year ago.
A common thing that new hire developers do is call for "a complete rewrite", they do this because when they first approach a large old code base, its daunting and seems impenetrable. Of course they are right, but naive in thinking a "rewrite" will help. Any new rewrite will eventually just grow to be just as impenetrable once all features and edge-cases are accounted for.
Fundamentally, any software product is trying to model some aspect of the real world...and the real world is messy, very messy. Governments pass laws that contradict each other, some laws change drastically state by state, employees try new and novel ways to embezzle, different languages and units of measure exist, changing prices for commodities can suddenly cause complete upheavals in manufacturing process, etc. All this must try to be accounted for and its nearly an impossible task.
The bugs that persist are almost never "I click Button A and it does the wrong thing", but almost always "In case that Situation A + B + C all simultaneously exist, the result as interpreted by Agency X is not optimal". Obvious and real bugs get squashed pretty quickly, but those complex situational bugs can linger for a long time. As a manager, you sometimes just need to shrug, because the effort required to fix each and everyone of these would produce little to no tangible business value. Moreover, an environmental change could come along to render your "fix" invalid anyways.
Sometimes even during design discussions we are completely aware we are creating "a bug", but the decision is made that the amount of people that want both Feature A and where Situation B exist will produce relatively little overlap. Most often we just design a manual workaround, instead of trying to completely eliminate the bug.
I'm always refreshed and excited by dealing with young devs, particularly for their zeal to fix problems, simplify things, and generally improve the product. Yet, I do feel a bit of sadness in that I know reality is going to temper their enthusiasm after a decade or so. Reality is a very hard thing to model with any semblance of being "bug-free".
But one of the benefits of a rewrite is that you can dump all the features and edge-cases that are no longer required. Or fold old edge-cases into new generalities because the business has changed since then.
> the real world is messy, very messy.
Cannot disagree -but- it's nowhere near as messy as the people (often those who are to blame) defending the byzantine software stacks using that argument.
> reality is going to temper their enthusiasm after a decade or so.
I've been doing this professionally for two decades and my enthusiasm for "chuck it away and do it right" hasn't waned one bit.
>I've been doing this professionally for two decades and my enthusiasm for "chuck it away and do it right" hasn't waned one bit.
I think it's probably somewhere in between the two extremes. I think you should have good unit tests and then refactor parts of your code where you see better generalities, or where basic code cleanliness was disregarded before.
But throwing all of it away is rarely possible without endangering the profitability of the company for a while.
Well, obviously I don't mean "turn it off and wait for the new system to be finished". You build the new one whilst the old one is in maintenance mode and swap in new bits as and when you can.
For example, at current $WORK, the backoffice system is a horror show of overcomplex PHP that is riddled with bugs and no-one really understands how it all works. Replacing that would be a huge boon both humanly and monetarily to the company because CS use it heavily every day.
Continuous incremental improvement of a production system may, over time, have the same net effect as a an idealized big-bang replacement, but it's a very different process (it's usually what people who are saying you should never do a ground-up replacement prefer instead, because actual big-bang replacements, unlike idealized ones, are usually a shitstorm: and the reason is that they are usually done to the kind of systems you describe, overcomplicated key systems with inadequate documentation or institutional memory, and they are done instead of trying to get a firm grasp on each component of the existing system before replacing that component. And so they end up, at best, being exceedingly well designed, but overlooking key elements of business function discovered and implemented, but not durably documented, in the original system.)
> Cannot disagree -but- it's nowhere near as messy as the people (often those who are to blame) defending the byzantine software stacks using that argument.
This. As a still relatively young developer, I can almost guarantee you that the initial reaction of "nuke it from orbit!!!" doesn't come from a couple of minor abstraction problems. You get this reaction when every second bug you try to fix ends in a trip to Klendathu.
You will probably appreciate this little piece of anecdata, last July there was an update change in some fiscal Laws in Italy, so that a number of firms had to make a certain payment of taxes within the 31st of July, BUT the change was communicated/published almost "last minute" (as often happens) and a software house had to update their accounting program in a very strict time. The payment code (on the government side) was the same of another payment (already known to be due on the 31/07/2017) so the programmers, in order to distinguish the two payments "anticipated" (virtually) the date on the database, so that two payments were resulting, one on the 30th and one on 31st.
This (intentionally) "queer" behaviour was not explained (or not explained well enough) to the users.
Most users "trusted" the program and everything went well for them, those that noticed the anomaly managed to "force" both payments on the same date, and this resulted in a "single" payment (instead of the two separate ones required), thus messing up the whole thing.
Seems like one needs to be controlling the spec and writing the code not to get stuck in this trap.
Well, not always.
That applies ONLY to those that find such problems or inconsistencies and workaround them in an incorrect manner.
And this brings us back right to Chesterton's fence:
https://en.wikipedia.org/wiki/Wikipedia:Chesterton's_fence
that can be invoked both when users do silly things, but also when programmers do them.
Partly yes, but only partly, as they did publish the "peculiar" workaround that they (the programmers) used in the update (though not giving to it the relevance that should have been given to it), but NO user actually reads the boring text that comes with the updates of course.
In this peculiar case the non-reading users were divided in three:
1) non-reading users that didn't even notice the anomaly
2) non-reaading users that noticed the anomaly and that either read the accompanying text or called to ask why the anomaly presented itself and were given a reasonable explanation.
3) non-reading users that noticed the anomaly, but, assuming that the programmers were a bunch of good-for-nothing morons [1], forced or "overruled" the settings without asking anything (and of course without even asking themselves if they were possibly causing an issue later on)
Of course both the #1 and #2 were fine, with the difference that the #1 were simply lucky, whilst the #2 "deserved" their success, as they had the curiosity to delve deeper in the issue.
The #3 are the main reason why I posted the Chesterton Fence reference, but it is applicable more generally.
Now that they were (all, users and programmers, in different ways) bitten by the issue, most probably the programmers in next release will add a field to the database so that you can have more than one payment with the same government code on the same day.
Still I can bet that in a few years the new kid on the block (among the programmers) will notice that there is a field in the database that is always set to 1, the memory of the reason why that field was added will be lost and he will probably remove that field by saying "Ha! I optimized the database by removing an unneeded field." and falling in the same fallacy.
[1] BTW, not that the opinion specifically was completely wrong, though I am not a programmer (nor an accountant) I had to deal with some of these guys to import some inventory data coming from another accounting program and it was a nightmare.
* Defect rate (does it do what it's meant to do?)
* Does it do what the user wants? (not always the same as the above...)
* Is it pleasant and efficient to use (definitely not the same as either of the above).
* Is it developed in a way the management are comfortable with? (which often seems to lean towards sufficiently "under control", replaceable developers).
How do you balance these? The answer will be quite different depending on whether you're landing on Mars or writing a free-to-play game.
They pay lip service to it, sure, but when it comes down to it most don't care enough until it actually starts to affect the bottom line. And longer, more expensive development processes are already affecting the bottom line, so come on, get it out the door!
Plus a lot of engineers see quality considerations as a drag. If they can find a home in a company that doesn't want all this "extra" stuff done then, well, this is what you get.
There are notable counter-examples in companies - Big Blue has a huge focus on quality, and their teams put a lot of effort into it (note I am saying nothing about usability here...) which is possible because a lot of stuff there moves slowly anyway. It's also because IBM are very, very good at measuring their cashflows and costs and have figured out just how much lack of quality can impact their bottom line.
There are also many individual engineers in smaller companies who put quality up front, and try their damnedest to push it through even where the business may not really care.
( * mostly SMEs are terrible for this, IMHO, though one or two large corporates I've worked with haven't been that great either)
Theoretically, you could write software by just having an incredibly long list of test cases and a random string generator.
The quality of that code would probably be terrible, but it would still work as long as your test cases are restrictive enough.
If you have mistaken issues of style for issues of quality, than that might seem to be the case. True quality in software is measurable primarily in the defect rate, and secondarily in the amount of effort needed to enhance it.
> Theoretically, you could write software by just having an incredibly long list of test cases and a random string generator.
Putting aside the time and concurrency issues, the quality would be determined by the correctness and thoroughness of your test cases.
Unfortunately, you learn that some time after you write the poor-quality code...
There are two big trade-offs in time alone: missing the chance to be first to market (mongo vs rethink comes to mind, albeit not quite accurate), and the need to get feedback early and often enough to pivot if the idea isn't quite right.
Then the layers of lava come- not enough time to rewrite everything now that the domain is better understood, the prototype becomes the foundation, and cruft builds up.
There is a misconception that code quality comes in a price of developing time but in my experience I have released much faster the final product when I use unit testing, incremental releases, use code reviews and other techniques that help maintain the code quality high.
By not using these techniques you get faster initial release but a much slower final, bug free, release.
Your feeling about the magnitude of the issue (most companies) is wrong. Programmers discuss CQ principles among themselves. However, the discussion becomes more challenging with management.
Management is responsible for accomplishing business objectives. Development and testing timeframes are at odds with business objectives. Adopting CQ delays product. If you're going to delay product, but the product will be beautifully efficient, idiomatic, elegant and possibly a little faster than the first pre-CQ version, you're not going to win an argument in an organizational context where delivery timelines matter.
Time and effort are not a programmer's friend in a task-driven organizational setting. Fortunately, real-time linters tell programmers not only about material errors but present stylistic warnings (such as Python pep8 linters). Further, static analysis tools such as Quantified Code [1] conduct an in-depth analysis of code and suggest stylistic improvements. I suspect that this is an area where machine learning will advance Code Quality further. Maybe, just as there are language servers, there will be code quality servers.
It is worth noting that the QuantifiedCode entity shuttered in the Summer of 2017. It's not clear why the company closed-- did they fail to monetize automated code review? Were they acquired?
In conclusion, the more you can automate code quality-related improvements, the more likely you can promote your Code Quality ideals.
[1] https://github.com/quantifiedcode/quantifiedcode
:points madly around the room: