Shops who set the bar this high and mean it are exceptionally rare (I look for them when I'm on the market). Where I am now, an hour of downtime costs us my salary for weeks, and still we cut corners that don't always sit right with me.
But having generalist engineers create functional and integration test plans? Not only do we tend to cost more, but we have less experience at it than QA specialists. And a dev shouldn't test their own work, because they won't notice any unwarranted assumptions they made in design and coding.
My wife has a freelance QA consultancy, and while I understand that there is often the sense that QA is a 'nag', I can honestly say that whenever she is involved in my work, the quality of my output is much better, and takes less time to get it there.
A good QA analyst / team does much more than point at a problem and say 'there it is'. A good QA analyst can determine why you made an error, and can anticipate other errors you might have made in the same way. A good QA analyst will learn your formulae, learn your equations, understand your workflow and try to find root cause, even if they're not developers.
Having a good QA person with a trained eye for knowing what I'm likely to have overlooked is also FAR more cost effective than your high-paid developers doing the same task. If they know that 'heads will roll' when they get errors, I'm also betting that makes your releases even slower.
Off-topic completely, but I generally prefer non-developer QA analysts, as they generally prove themselves to 'think too much like me' to be effective.
People who know how to test are extremely valuable. Specially when doing peer review/pair programming with (solving the blind's eye thing)
Testing as a whole is not an easy task as it involves testing some really hard things: memory leaks, performance, UX, correctness of parallel and distributed algorithms/programs/systems.
However, Wealthfront engineers probably put this position more into perspective [1]:
We do not have QA team and do not want to have one, the reasoning is that if a human is involved in testing then there is a higher chance of missing things and you simply can't test all the site dozens of times a day.
Granted, Wealthfront uses continuous deployment[2], while still being regulated by the SEC. I really enjoy their lean startup approach to minimize inventory (ie: code not in production), while still keeping an immune system. And investors seem to agree it is really paying off [3].
At least in my wife's case, she generally does pre-release QA, and doesn't necessarily do continuous QA. Still though, even if you need support for continuous deployment, you probably want a QA person to set up your automated test cases to test the output of a site in a way that, in my opinion, unit tests don't seem to catch.
Having worked in Silicon Valley for 12 years, I have yet to find QA people who could successfully automate testing in a non-fragile way or without using an expensive, proprietary and crappy testing harness. Those who know how to program are invariably better at writing automated tests. One recent CS graduate should be able to put a team of QA out of a job. (Disclosure: I work for Wealthfront)
Considering I work with one of the most incompetent QA engineers I've ever had the displeasure of encountering, I couldn't agree more. This person wastes my time, and the organization's time, and has not actually increased the quality of software (as far as I can tell). It is a waste of a developer's time to decipher a bug that states "it's broken," without clearly defining "broken," or "it," or how to reproduce the behavior. Often, reproduction involves user error (like forgetting to sign into the vpn). If it weren't for this bozo, our team could get more work done in less time.
Do you think that your unpleasant situation is attributable to QA being an unhelpful discipline, or to a particular person performing poorly in their QA role?
I don't think this would be a good idea. When developing I don't always have the time to be as thorough as a QA team is in testing my own code. It's very helpful to have somebody else with a non-technical background to help spot problems that I am blind to.
Well, that's certainly helpful. But what I'm saying is more that, without a QA team, while I would I would almost certainly attempt to quality control my own code more, it would be ignorant to deny that I or anybody else has blind spots.
I believe it's a lot more difficult to see your own errors than somebody else's. I believe QA is a fulltime job. I believe the relationship between developers and QA people is mutually beneficial to both people.
No, that you're blind to because it's in the nature of software development. Spotting flaws in something you've spent the last $X hours convincing yourself is correct is hard to do. It's easy to make invalid assumptions and internalize them in your thinking to the point you don't notice them. This problem exists, and largely is solved by dragging in someone else to help with it. That someone else, unless you'd like to squabble over semantics, is doing QA.
The author plays to the stereotype of developers who recklessly "commit and forget it". In a properly run organization, QA takes note of developers who consistently commit show-stopper bugs. If the QA team is positioned as a clean up crew for shabby code, this is a failing of the organization rather than the developers.
In an organization in which tests are developed for any code committed, and developers are careful not to break a build, soft issues are still caught by the QA team. Things that work but not as expected, minor cross-platform glitches, and permutations you'd never think to check, are all commonplace catches for a professional QA tech working with a well functioning development team.
The author of the article really doesn't sound like he's worked with a decent dev or QA team.
It's also helpful to have someone that can run through endless permutations, paying careful attention to each, finding small issues that popup with countless applied revisions. It is not an easy job.
My head is more often buried in code than in the product's UI. Our QA team has a better nose for when interaction or UI details are off than I do. In general, they spend more time using the application, and have a better instinct for when a given behaviour is wrong.
Some of the bugs they report would fall under the automatically testable category, but many don't.
I suspect that dogfooding is more effective at companies where the product itself is useful to developers or is a consumer app. e.g. Basecamp, FogBugz, Facebook, Netflix, Gmail, etc.
QA and dev are two different skills. It's like saying that we save on design by having the devs do it, or having your quarterback also play noseguard. Sure it works in high school, but not when you're a pro.
I'd much rather hire someone who knows testing and QA well and let my devs focus on writing bug-free product code.
I disagree. Developers who don't know how to test are bad developers, they should go. And conversely, if you have QA, they should know how to develop, otherwise you're paying them to hunt, peck, and click. Why not have QA write the code to have computer do that and save like 1000x the time? That's efficiency AND quality.
Of course QA knows how to program. But their skillset is about breaking code. It's about writing this crazy C++ function that shows how your API has a security bug in this case. As developers their like dev tools developers as their customers are generally devs.
Skillset maybe isn't the right word... mindset maybe better word. Both should be good programmers -- in fact you'll often have more test code than product code. But you likely won't be able to switch your dev team to QA and vice-versa and get the same results.
Ah it's a single anecdotal story and comes with no specifics about who it is or in what way things supposedly improved, just some broad calls for firing QA.
Forrester will ring a few more people they picked out of the phone at random, write it up as a report for their library complete with beautiful graphs, and then begin to charge thousands of dollars per copy to tell other consultants what the "Industry Best Practice" is at the moment.
a large fraction of errors that turn up in QA are conceptual errors made by the development team; like the blind spot in the eye, you don't know what you don't know, and there are certain problems that you'll never find yourself.
Phil Crosby would say that management is responsible for quality; management writes the specifications and has the responsibility to hire and fire developers and QA people. The buck stops with management.
Efficient quality requires practices to be work correctly across the organization, but it starts in the specification process. If you don't know where you're going, you're sure as hell never going to get there.
"If you don't know where you're going, you're sure as hell never going to get there." -> Also not always true. Think Twitter. Think Christopher Columbus.
"Amerigo Vespucci's travel journals, published 1502-4, convinced Martin Waldseemüller that the discovered place was not India, as Columbus always believed, but a new continent"
Not sure this takes into account everyone else who died at sea. Occasional serendipitous results are to be expected, but don't represent what is probably a large body of associated failures to this approach.
If a developer's reaction to a client-facing defect is "QA didn't catch it", you have cultural problems that firing a QA team probably won't fix. If you don't have a culture of personal responsibility, a finger can always be pointed elsewhere.
Developers should absolutely write and use unit tests. Where QA really shines in the larger scale: validating actual business requirements.
Exactly. For me, every client-facing defect is an embarrassing personal failure. Having them caught in QA simply lessens the shame, and allows me to dedicate all the time I would spend freaking out about the newfound emergency to figuring out improvements that eliminate that entire class of errors.
What is the metric for quality? Is the article saying that having fewer people looking for bugs resulted in more bugs being found? Or fewer bugs? Or just fewer bugs found?
I 100% agree with this article. Quality is everyone's job, not just the QA teams' and I believe that, in most cases, developers can and should be responsible for developing, testing, and releasing their code. When developers can't "throw code over the wall to QA" because there is no QA, they will do a better job testing it themselves.
I don't buy the popular opinion that developers can't test their own code like there is some sort of magic that goes on in the mind of QA engineers that causes them to be able to find bugs that developers couldn't find themselves. In fact, I think the opposite is true - developers are better suited to test the features they are writing because they know the code and know the risk areas, where things need to be regression tested, where more attention should be placed, etc.
I do think that it can be helpful for another set of eyes to look at a developed feature and this is where pair programming can be useful. It does not need to be QA doing this.
One caveat is that this approach does not work for all teams. This works for small, agile teams where roles and responsibilities are less defined. If you fire your QA team but your developers can't test their own code because they are set in their ways of writing crap because they expect QA to find the bugs, you may end up firing your dev team too....
Having been in a high pressure position in which my team and I were writing mission critical code which could "have no bugs whatsoever" first without QA, and then later with QA, I can say confidently the quality of the code and occurrence of bugs dropped dramatically after we introduced our QA engineers.
Our prior condition was very similar to telling a newspaper writer "DONT WRITE TYPOS". You can scan a page you've written several times carefully and still miss your mistakes, even if you are a conscientious writer. And the newspaper industry isn't the only industry that prevents mistakes through multi-party review processes, either.
Don't be so quick to brush this off. The argument isn't so different from saying that we don't want to distinguish "front-end" and "back-end" programmers. It's probably a special circumstance that you need a team that is entirely dedicated to verification.
Feynman talks about this arrangement in his essay on the shuttle program (http://www.fotuva.org/feynman/challenger-appendix.html). In a nutshell, the core software team is responsible for checking the code. Then there is a totally removed group that is adversarial, i.e., it is measured on its ability to trip up the software team. This is the same a "setting consequences" as this author describes.
Now, the shuttle program is truly life critical. I think a good software team can produce quality results internally.
The argument isn't so different from saying that we don't want to distinguish "front-end" and "back-end" programmers.
I don't want to distinguish front-end and back-end programmers. I would like to distinguish front and back-end development.
I do everything currently but I still need put on a conceptual "testing hat", "GUI hat" and "back-end hat". That requires mental effort and costs time. A real Q&A would save that even if I was exactly as skilled as the Q&A person.
Yeah, that's true. I think the point of the source is that the first line of defense is with the originator of the code. Sure, the context switch is expensive, but that's the price of higher quality.
It's not really about firing the QA team. Instead, it's about having a QA function that does not entirely relieve the original author of his duty to ensure quality.
By the time anything gets to the QA person at my dayjob, the devs have usually tried every damn thing that they could think of to break their own code.
The usual reaction to the bugs she finds is "what made you think of trying that?"
Most of the other comments already voice my reaction to this post. I will only add that a "heads will roll" attitude towards motivating devs on the part of management would prompt me to switch to some other place very fast. Simply saying "you have to do X as well as Y or heads will roll" just sounds like trying to save money on QA and squeezing more work out of devs - who could be putting that time and effort to much better use improving the product.
> They better get it right, or heads will roll. As British author Samuel Johnson famously put it, "The prospect of being hanged focuses the mind wonderfully."
Gee, sounds like a wonderful place to work. When can I sign up?
I work in QA in a high-reliability industry. Correct operation of some our products saves millions and saves lives. We need super high quality. How we do QA is pretty traditional. In particular, we have functional test teams. The perspective we take is that the testers take the specs and develop tests to exercise the functionality, and theoretically, we (QA) cover any spots the developers missed, as well as providing assurance that the developers aren't talking out of the wrong hole when they say they're done. Seems to work most of the time except when mutual brain-farts develop.
I don't know that I would go so far as to recommend firing myself to increase quality. I would have to visit with many teams and really gain experience in the way they work. I do think that we could take a step forward by having a standard across the company to have fully automated test suites running continuous builds and tests. That, however, runs facefirst into the problem of Legacy Stuff and ROI. Automation just plain takes time when you are interfacing against external programs. It's not always worth it in the short/medium term.
There's no silver bullet to the software quality problem. But me, I think that continuous integration provides a copper bullet.
Peopleware has a chapter about "the black team" that was at IBM. The chapter is about team work but they describe how in the process of gelling the QA team delighted in finding programmer bugs. They describe how creation of a reputation and ethos in the black team not only made them a little better than other QA teams but significantly better.
While it's easy to say that QA isn't important many of the QA teams I've worked with weren't respected or valued, and often saw QA as a stepping stone. If you want to increase the quality of the code, don't fire the QA team, motivate them.
47 comments
[ 2.0 ms ] story [ 99.8 ms ] threadShops who set the bar this high and mean it are exceptionally rare (I look for them when I'm on the market). Where I am now, an hour of downtime costs us my salary for weeks, and still we cut corners that don't always sit right with me.
But having generalist engineers create functional and integration test plans? Not only do we tend to cost more, but we have less experience at it than QA specialists. And a dev shouldn't test their own work, because they won't notice any unwarranted assumptions they made in design and coding.
A good QA analyst / team does much more than point at a problem and say 'there it is'. A good QA analyst can determine why you made an error, and can anticipate other errors you might have made in the same way. A good QA analyst will learn your formulae, learn your equations, understand your workflow and try to find root cause, even if they're not developers.
Having a good QA person with a trained eye for knowing what I'm likely to have overlooked is also FAR more cost effective than your high-paid developers doing the same task. If they know that 'heads will roll' when they get errors, I'm also betting that makes your releases even slower.
Off-topic completely, but I generally prefer non-developer QA analysts, as they generally prove themselves to 'think too much like me' to be effective.
Testing as a whole is not an easy task as it involves testing some really hard things: memory leaks, performance, UX, correctness of parallel and distributed algorithms/programs/systems.
However, Wealthfront engineers probably put this position more into perspective [1]:
We do not have QA team and do not want to have one, the reasoning is that if a human is involved in testing then there is a higher chance of missing things and you simply can't test all the site dozens of times a day.
Granted, Wealthfront uses continuous deployment[2], while still being regulated by the SEC. I really enjoy their lean startup approach to minimize inventory (ie: code not in production), while still keeping an immune system. And investors seem to agree it is really paying off [3].
[1] http://eng.wealthfront.com/2010/04/findbugs-husdon-and-pizza...
[2] http://www.eishay.com/2010/07/continuous-deployment-at-kachi...
[3] http://techcrunch.com/2010/10/19/kaching-gets-100m-under-man...
I assume that others have had better experiences.
I believe it's a lot more difficult to see your own errors than somebody else's. I believe QA is a fulltime job. I believe the relationship between developers and QA people is mutually beneficial to both people.
In an organization in which tests are developed for any code committed, and developers are careful not to break a build, soft issues are still caught by the QA team. Things that work but not as expected, minor cross-platform glitches, and permutations you'd never think to check, are all commonplace catches for a professional QA tech working with a well functioning development team.
The author of the article really doesn't sound like he's worked with a decent dev or QA team.
Some of the bugs they report would fall under the automatically testable category, but many don't.
I suspect that dogfooding is more effective at companies where the product itself is useful to developers or is a consumer app. e.g. Basecamp, FogBugz, Facebook, Netflix, Gmail, etc.
I'd much rather hire someone who knows testing and QA well and let my devs focus on writing bug-free product code.
Skillset maybe isn't the right word... mindset maybe better word. Both should be good programmers -- in fact you'll often have more test code than product code. But you likely won't be able to switch your dev team to QA and vice-versa and get the same results.
a large fraction of errors that turn up in QA are conceptual errors made by the development team; like the blind spot in the eye, you don't know what you don't know, and there are certain problems that you'll never find yourself.
Phil Crosby would say that management is responsible for quality; management writes the specifications and has the responsibility to hire and fire developers and QA people. The buck stops with management.
Efficient quality requires practices to be work correctly across the organization, but it starts in the specification process. If you don't know where you're going, you're sure as hell never going to get there.
Quoting from wiki (http://en.wikipedia.org/wiki/Christopher_Columbus):
"Amerigo Vespucci's travel journals, published 1502-4, convinced Martin Waldseemüller that the discovered place was not India, as Columbus always believed, but a new continent"
Taken out of context. Twitter wasn't a result of good or bad QA. In fact, it's got nothing to do with that.
I hope all of you reading the blog post realise that this brainfart will cost your organisation $5,500 per copy downloaded.
Developers should absolutely write and use unit tests. Where QA really shines in the larger scale: validating actual business requirements.
I don't buy the popular opinion that developers can't test their own code like there is some sort of magic that goes on in the mind of QA engineers that causes them to be able to find bugs that developers couldn't find themselves. In fact, I think the opposite is true - developers are better suited to test the features they are writing because they know the code and know the risk areas, where things need to be regression tested, where more attention should be placed, etc.
I do think that it can be helpful for another set of eyes to look at a developed feature and this is where pair programming can be useful. It does not need to be QA doing this.
One caveat is that this approach does not work for all teams. This works for small, agile teams where roles and responsibilities are less defined. If you fire your QA team but your developers can't test their own code because they are set in their ways of writing crap because they expect QA to find the bugs, you may end up firing your dev team too....
Our prior condition was very similar to telling a newspaper writer "DONT WRITE TYPOS". You can scan a page you've written several times carefully and still miss your mistakes, even if you are a conscientious writer. And the newspaper industry isn't the only industry that prevents mistakes through multi-party review processes, either.
Feynman talks about this arrangement in his essay on the shuttle program (http://www.fotuva.org/feynman/challenger-appendix.html). In a nutshell, the core software team is responsible for checking the code. Then there is a totally removed group that is adversarial, i.e., it is measured on its ability to trip up the software team. This is the same a "setting consequences" as this author describes.
Now, the shuttle program is truly life critical. I think a good software team can produce quality results internally.
I don't want to distinguish front-end and back-end programmers. I would like to distinguish front and back-end development.
I do everything currently but I still need put on a conceptual "testing hat", "GUI hat" and "back-end hat". That requires mental effort and costs time. A real Q&A would save that even if I was exactly as skilled as the Q&A person.
It's not really about firing the QA team. Instead, it's about having a QA function that does not entirely relieve the original author of his duty to ensure quality.
The usual reaction to the bugs she finds is "what made you think of trying that?"
Gee, sounds like a wonderful place to work. When can I sign up?
I don't know that I would go so far as to recommend firing myself to increase quality. I would have to visit with many teams and really gain experience in the way they work. I do think that we could take a step forward by having a standard across the company to have fully automated test suites running continuous builds and tests. That, however, runs facefirst into the problem of Legacy Stuff and ROI. Automation just plain takes time when you are interfacing against external programs. It's not always worth it in the short/medium term.
There's no silver bullet to the software quality problem. But me, I think that continuous integration provides a copper bullet.
While it's easy to say that QA isn't important many of the QA teams I've worked with weren't respected or valued, and often saw QA as a stepping stone. If you want to increase the quality of the code, don't fire the QA team, motivate them.