In making the case for building up a QA org at my current startup, I repeat the mantra that QA is both a skillset and a mindset. Automated tests can tell us a lot, but skilled QA testers are amazing at edge cases to break things and providing human feedback about what looks and feels good gor users.
Microsoft is probably the one case where this sticks out the most, at least for me anyways. Noticeably more bugs in updates since they dropped their QA team, in Windows as well as cloud products.
>The revenue keeps rolling in so, clearly, they made the right business decision... >sigh<
This is exactly the point I was going to make. Their stock price is doing great! So obviously they've done the right thing for their position in the market: the market has rewarded them for not wasting money on QA and just letting users suffer with the bugs.
>I wish there was some kind of consequence to them for their declining quality.
If people keep insisting on throwing money at them no matter how bad their software is, then there's no reason for them to improve their quality.
Another part is that there is barely any training for QA people. Even your average CS course will only graze the top of the topic, most usually some prof droning on about Java unit tests on some really old version of Java and a testing framework just as old.
There are no "Software Quality Assurance" academic degrees, there's barely any research into testing methodologies, there's barely any commercial engagement in the space aside from test run environments (aka, selling shovels to gold diggers), and let's face truth, also in tooling. And everything but software QA is an even worse state, with "training" usually consisting of a few weeks of "learning on the job".
Basic stuff like "how does one even write a test plan", "how does one keep track of bugs", "how to determine what needs to be tested in what way (unit, integration, e2e)" is at best cargo-culted in the organization, at worst everyone is left to reinvent the wheel themselves, and you end up with 13 different "testing" jobs in a manually clicked-together Jenkins server, for one project.
> Defect Investigation: Reproduction, or “repro”, is a critical part of managing bugs. In order to expedite fixes, somebody has to do the legwork to translate “I tried to buy a movie ticket and it didn’t work” into “character encoding issues broke the purchase flow for a customer with a non-English character in their name”.
And this would normally not be the job of a QA person, that's 1st level support's job, but outsourcing to Indian body shops or outright AI chatbots is cheaper than hiring competent support staff.
That also ties in to another aspect I found lacking in the article: users are expected to be your testers for free aka you sell bananaware. No matter if it's AAA games, computer OSes, phones, even cars...
To play devil's advocate here, I did not _really_ get training for my software engineering role. I got a little bit from 1-2 college courses, but the vast majority of my role I had to pick up on the job or on my own.
I can tell you, I definitely didn't get training for QA tasks, but here I am doing them anyways. It's just work that needs to be done.
> To play devil's advocate here, I did not _really_ get training for my software engineering role
Yeah and that is my point. It would be way better for the entire field of QA if there were at least a commonly agreed base framework of concepts and ways to do and especially to name things, if alone because the lack of standardization wrecks one's ability to even get testers and makes onboarding them to your "in house standards" a very costly endeavour.
> if there were at least a commonly agreed base framework of concepts and ways to do and especially to name things
All of the above exist, the International Software Testing Qualification Board is one of the entities offering certifications in this: https://www.istqb.org/
ISTQB/ASTQB's definitions and teachings aren't commonly agreed on by the testing profession, at least in North America, and their certification isn't seen as being of any particular value. I hear they're held in higher regard in Europe and elsewhere.
A lot of people took it so they could put it on their CV/resume and get past the job filters, consultancies got their testers to do it so they could get past client filters ( from my experience when I was working in the UK)
> There are no "Software Quality Assurance" academic degrees, there's barely any research into testing methodologies,
There's a lot of this actually. Entire communities of people working on software quality assurance. Practitioners in this space call their field "resilience engineering".
The field likes to talk a lot about system design. Especially in the intersection of humans and machines. Stuff like "How do you set up a system (the org) such that production bugs are less likely to make it all the way to users"
This might be my personal experience, but I've never encountered a QA team that actually writes the tests for engineering.
I have only had QA teams that wrote "test plans" and executed them manually, and in rarer cases, via automated browser / device tests. I consider these types of tests to be valuable, but less so than "unit tests" or "integration tests".
With this model, I have found that the engineering team ends up being the QA team in practice, and then the actual QA team often only finds bugs that aren't really bugs, just creating noise and taking away more value than they provide.
I would love to learn about QA team models that work. Manual tests are great, but they only go so far in my experience.
I'm not trying to knock on QA folks, I'm just sharing my experience.
in the classic model, most QA orgs were a useless appendage. partially by construction, but largely because QA gets squeezed out when dev is late (when does that happen?). they aren't folded in early, so they twiddle their thumbs doing 'test infrastructure' and 'test plans', until they finally get a code drop and a 48 hr schedule to sign off, which they are under extreme pressure to do.
but every once and a while you ran across a QA organization that actually had a deep understanding of the problem domain, and actually helped drive development. right there alongside dev the entire way. not only did they improve quality, but they actually saved everyone time.
Saying "useless appendage" sounds to me like it's the QA team that's the problem, when what you're really saying is that it's the organization and process that pushed QA teams into irrelevance. I agree with your assessment overall, and those issues were one of the driving forces behind companies dispensing with QA and putting it all on the developers.
I worked on a team where we structured things so we had working groups of 2-3 devs + 1 QA engineer for individual features, which were typically less than a week. Idea being the QA person was involved in the writing of the acceptance criterion and the testing at the end, etc, but also along-the-way testing.
It worked exceptionally well, in my opinion but the QA revolted; they didn't like it, said that it felt like we were trying to make them into software developers. And so it ended, and it was back to "throw it over the wall".
This is because there is no formal way to run a QA org, people get hired and are told to “figure it out”. Then as other posters said the other orgs ignore the QA org because they have no understanding of the need. What you’re describing is a leadership problem, not a QA usefulness problem.
the engineering team are usually great at writing tests that test their code, a good QA can test alongside them to find cases they've missed and issues that automated code tests can't find. The QA person doesn't have to spend time checking that the app basically works, they can be confident in that and spend their time testing for other 'qualities'
But yes, I've known QA teams that will only find bugs that no one cares about or are never likely to happen - often because they are not trained on the product to be able to dig deep
It seems so obvious to me that your typical engineer, who spent hours / days / whatever working on a feature, is never going to test the edge cases that they didn’t conceive of during implementation. And if they didn’t think of it, I bet they’re not handling it correctly.
Sometimes that’ll get caught in code review, if your reviewer is thinking about the implementation.
I’ve worked in payroll and finance software. I don’t like it when users are the ones finding the bugs for us.
I started off as a dev, wanted to change to being a tester/QA but was told by the CEO that "the customers are better at finding bugs than we are so just give the app a quick look over and ship it out" - I left soon after that.
From what I've seen, the value in QA is product familiarity. Good QA'ers know more about how the product actually works than anybody else. More than PM's, more than sales, and more than most dev teams. They have a holistic knowledge of the entire user-facing system and can tell you exactly what to expect when any button gets pushed. Bad QA'ers are indeed a source of noise. But so are bad devs, sysadmins, T1/2 support, etc.
Agreed! I did have some good experiences at my last job with the QA team, but it was definitely a unique model. They were really a "Customer Success" team, it was a mix of QA, sales, and customer support.
These "Customer Support" reps, when functioning as QA, knew the product better than product or eng, exactly how you're describing. I did enjoy that model, but they also did not write tests for us. They primarily executed manual test plans, after deploys, in production. They did provide more value than creating noise, but the engineering team still was QA, at least from an automated test standpoint.
We had no dedicated QA, but would consistently poach "Customer Success" team members for critical QA work for the exact reasons your listed. Worked quite well for us.
Especially for complex products that are based on users chaining many building blocks together to create something useful, devs generally have no visibility into how users work and how to test.
I completely agree with that. It really comes down to having the right skills as a QA person. If you don't know how the product is used and only click on some buttons, you will never reach the states in the software that real users reach and therefor you will also not be able to reproduce them.
> Good QA'ers know more about how the product actually works than anybody else. More than PM's, more than sales, and more than most dev teams.
Not disagreeing with this, but there's one thing they won't always be aware of. They won't always know what code a dev touched underneath the hood and what they might need to recheck (short of a full regression test every single time) to verify everything is still working.
I know that the component I adjusted for this feature might have also affected the component over in spots X, Y, and Z, because I looked at that code, and probably did a code search or a 'find references' check at some point to see where else it's getting called, and also I usually retest those other places as well (not every dev does, though. I've met some devs that think it's a waste of time and money for them to test anything and that's entirely QA's job).
A good QA person might also intuit other places that might be affected if it's a visible component that looks the same (but either I haven't worked with too many good QA people or that intuition is pretty rare, I'm guessing it's the latter because I believe I have worked with people who were good at QA). Because of that, I do my best to be proactive and go "oh by the way this code might have affected these other places, please include those in your tests".
> They won't always know what code a dev touched underneath the hood and what they might need to recheck (short of a full regression test every single time) to verify everything is still working.
This is a good point, but there are some QA that do review code (source: me - started career in QA and transitioned to dev). When making a test plan, an important factor is risk assessment. If QA has a hunch, or better when the dev lead flags complex changes, the test plan should be created and then the code diffs should be reviewed to assess whether or not the plan needed revising. For example, maybe the QA env doesn’t have a full replica of prod but a query is introduced that could be impacted if one of the joining tables is huge (like in prod). So maybe we’d adjust the plan to run some benchmarks on a similar scale environment.
I’m definitely biased since I started in QA and loved it. To me, good QA is a cross section of many of the things people have mentioned - technical, product, ops, security - with a healthy dash of liking to break things. However, reality is that the trend has been to split that responsibility among people in each of those roles and get rid of QA. Works great if people in each of those job functions has the bandwidth to take on that QA work (they’ll all have a much deeper knowledge of their respective domains). But you’ll lose coverage if any one of those people don’t have time to dedicate to proper QA.
(I’ll also completely acknowledge that it’s rare to have a few, let alone a full team, of QA people who can do that.)
> Not disagreeing with this, but there's one thing they won't always be aware of. They won't always know what code a dev touched underneath the hood and what they might need to recheck (short of a full regression test every single time) to verify everything is still working.
It doesn't necessarily matter what code was changed, a change in code in Module A can cause a bug in Module B that hasn't been changed in a year. A QA test plan should cover the surface area of the product as used by consumers whoever they might be. While knowing some module had fixes can inform the test plan or focus areas when the test schedule is constrained, only testing changes is the road to tears.
Test plans never account for everything, at least in my experience, especially edge cases. And it's rare that I've seen any QA team do a full regression test of the entire site. There's only been a few times where I've seen it authorized, and that's usually after a major refactoring or rewrite.
I'm not in QA, I write code, so I defer to whatever they decide for these things usually, these are just observations from what I've seen.
I just try to make sure I test my code enough that there isn't anything terribly broken when I check it in and fixes I need to make tend to be relatively minor (with a few exceptions in my past).
Also I'm not necessarily talking basic functionality here. I'm currently working for a client that's very picky about the look and feel, so if a few pixels in padding get adjusted where it's noticeable, or a font color or size gets adjusted a bit, in one place and it affects something else, there could be complaints. And a test plan is not likely to catch that, at least not any on any projects I've worked on.
>They won't always know what code a dev touched underneath the hood and what they might need to recheck (short of a full regression test every single time) to verify everything is still working.
Not really. As QA I always reviewed the checkins since yesterday before opening up the daily build. Between the bug comments and the patch comments, even if the patch itself is a bit Greek to me, I can tell what was going on enough to be a better tester of that area.
Let's say that there's some sort of list component that's used in like six other places (that's maybe not obvious because there's different configurations or something).
Six months later I work on a new page that incorporates that component that requires some sort of change in that component that could theoretically break it elsewhere, at least how it looks or something, like alters the padding or something in a way that shoves things onto the next line in one of the instances.
If I don't mention what other places that component is used in my comments, are you going to know where all to look for the other instances of that component? Even if you look at the source code, it will only show changes, it won't say 'oh this component is used elsewhere' unless you actively bring up the source code yourself and do a code search yourself.
Because it's a visible component, maybe you'll just have the memory that something that looks like this is elsewhere, but maybe you won't remember. Or maybe you're fairly new to the project and aren't aware of these other parts of the site using it.
What about something that's even less visible, like something that takes in data and then sorts, transforms, and adds metadata to it? Then I make a change to it to accomodate for some new requirement on a new page. Maybe it's used for four other locations, but the changes only shows the function being altered and the new page using it. That change I made broke it elsewhere, but I don't mention where else to check in my notes. Are you going to think to check each place where data is loaded to make sure it's working correctly, without me explicitly telling you to check those spots?
If you can, great, you've got a real intuition for this that I doubt many QA people have. I don't even know to check these other places sometimes, and I worked on the code myself and can do a quick text search to see where else it's being used.
This was why I said what I did. It's not just about 'that area', the change could potentially affect another application in different repository entirely sometimes, especially if a shared library is involved. I've literally had like four or five of these situations happen in the past six months of my current project, where my change broke something elsewhere.
Usually I caught it, but I also told QA to check it explicitly anyway since I don't usually test things 100% (since I'm not QA). When I was a more junior developer, I missed these things more often and QA would sometimes miss it also since they didn't know to retest something.
This is a great model, until those people so familiar with the business needs end up.. doing business things instead. It's really hard to keep people like that in a QA role once the business recognizes their value. Kind of the same problem with QA automation people - once they become really good at test automation, they are effectively software developers, and want to go there.
I think that's a compensation problem more than anything else. I've known some QA folks who enjoyed QA and would have stayed in that role if they could have justified the massive differential in comp between QA and SWE or product development. If we valued QA and compensated it at the same level we do those other roles then there would be a lot less difficulty retaining good QA folks.
I have never once heard of a problem that QA folks end up in project or product management too often, and almost always have the problem of not being able to escape the QA org despite many years. Most companies are extremely resistant to people moving tracks, especially from a “lower status” org like QA or CS. It’s the exception not the rule.
To your point, the QA team is the customer's advocate. As you say, they know the product, from the customer's perspective, better than anyone else in the development organization.
Where I've seen QA teams most effective is providing more function than "just" QA. I've seen them used for 2nd tier support. I've seen them used to support sales engineers. I've also seen QA teams that take their manual test plans and automate their execution (think Selenium or UiPath) and have seen those automations included in dev pipelines.
Finally, the QA team are the masters and caretakers of your test environment(s), all the different types of accounts you need for testing, they should have the knowledge of all the different browsers and OSes your customers are using, and so forth.
That also means they test from a different perspective than the dev does. If I get a requirement my build is based on my understanding of the requirement, and so is my testing.
A separate QA person coming at it from the customer's perspective will do a test that's much more likely to reflect reality.
You're not wrong, but a good, well resourced QA org can both help write or develop more flexible tests, and also help fix brittle tests when they do break. The idea of brittle tests that break often being a blocker is predicated on practices like running every type of test on every commit that exist to deal with a lack of QA effort in the first place.
Maybe recorded integration tests are run on every release instead of every commit? Maybe the QA team uses them less to pass/fail work and more to easily note which parts of the product have changed and need attention for the next release? There's lots of possibilities.
Absolutely hellish to maintain once you get into territories of different localization, A/B testing, engineering teams that constantly change small things in UI (for valid reasons). I think it might work on a product that changes once a year, but not sure about maintainability past that.
Every time I worked with dedicated QA teams, we (core engineering) ended up leading the effort of writing maintainable automated tests. At some point, it was much easier to just do it ourselves.
Well, that's kinda the point, isn't it? Have the tests run after each PR, and it flags all the changes for you. For the intentional ones, that means the tests have to be updated (often just changing a line or two). But it also catches all your unintentional changes that you didn't mean to break.
This is especially important in complex UIs where everything from localization to state management to A/B testing can accidentally break things you didn't even mean to touch.
Isn't it better for the devs to spend a few minutes looking at a broken test (which they can easily fix themselves) than for a user to discover it and have to go through triage months later, when it's out of everyone's memories already?
Realistically the build will fail, but since QAs used some action capturing thing, devs will have to either recapture the same flow or delegate it back to QA. Now you have the pipeline blocked, QAs most likely working on some other stuff so they need to reprioritize, devs are also annoyed as their build is not going through.
I completely understand there are pros and cons to every set up, but if higher velocity is more important than complete correctness, such set up might be detrimental.
In our setup, we had the ability to either fix the test ourselves, comment out those tests as useless, or just force a build and bypass any hooks if we needed to force a new deploy and we were absolutely certain the tests aren't important. But at least they would be caught, even if they served as a soft warning rather than a forcible barrier.
> but if higher velocity is more important than complete correctness, such set up might be detrimental.
I guess it's a cultural value thing: move fast and break things vs polish before shipping. As a user I always prefer the latter, but hell, I don't run these multibillion dollar companies that are always shipping buggy things, lol
Totally agreed, it really depends on the case. I’ve been user of both types of products where I value extreme correctness (think of anything financial, or core software that I want to never crash like a browser), but also really don’t care if something like Spotify or Twitter doesn’t work 5% of the time I use it. Obviously that barrier is different for everyone, but that’s where the product development planning and prioritization comes into play.
> but I've never encountered a QA team that actually writes the tests for engineering.
I have a few times. But the only common thing in the QA industry, is that every company does it differently and think they're doing it the "normal way".
In my company the engineering team mostly writes unit tests. Then there was a weekly manual QA exercise where the oncall engineer followed a checklist with our actual mobile app on an actual phone before it went to the store. When this started to take almost the entire day, we hired a contract workforce for it. The contract workforce is in the process of automating those tests, but the most important ones still get human eyes on.
I was surprised by the opposite of this (after entering my first real job at Google, after startup founder => seller.)
People wrote off QA completely unless it meant they didn't have to write tests, but, it didn't track from my (rather naive) perspective that tests are _always_ part of coding.
From that perspective, it seemed QA should A) manage go/nogo and manual testing of releases B) keep the CI green and tasks assigned for red (bonus points if they had capacity to try fixing red) C) longer term infra investments, ex. what can we do to migrate manual testing to integration testing, what can we do to make integration testing not-finicky in the age of mobile
I really enjoyed this article because it also indicates the slippery slide I saw there: we had a product that had a _60% success rate_ on setup. And the product was $200+ dollars to buy. In retrospect, the TL was into status games, not technical stuff, and when I made several breakthroughs that allowed us to automate testing of setup, they pulled me aside to warn me that I should avoid getting into it because people don't care.
It didn't compute to me back then, because leadership _incessantly_ talked about this being a #1 or #2 problem in quarterly team meetings.
But they were right. All that happened was my TL got mad because I kept going with it, my skip manager smiled and got a bottle of wine to celebrate with, I got shuffled off to work with QA for next 18 months, and no one ever really mentioned it again.
At least from my knowledge of the gaming world there are QA devs who do find the issues and fix them if they've the ability to do so, point out code that should be taken a look at, and all of that.
I find it extremely valuable to have another set of eyes in the code with a very more focused perspective, sometimes different from the dev.
yeah my experience is basically the same, usually if a place has qa at all, it's one person in a team who doesnt have an adequate environment or data set to test with and they effectively end up just watching the developer qa their own work and i end up screaming into a pillow every time i see "Tester: hi" pop up on my screen.
the one exception to this was when i was qa (never again) and i made sure we only ever did automated tests. unfortunately management was nonexistent, devs made zero effort to work with us, and naturally we were soon replaced by a cheap offshore indian team who couldn't tell you the difference between a computer and a fridge anyway.
i think a lot of it just stems from companies not caring about qa, not knowing who to hire, and not knowing what they want the people they hire to achieve. "qa" is just like "agile", where nobody can be bothered to actually learn anything about it, so they make something up and then pat themselves on the back for having it.
Weird, I have had the opposite experience that most shit slips through the cracks of automated testing & manual testing by an experienced QA is 10x more effective.
I think the thing missing from a lot of these conversations is what problem domain you're working in. The more "technical" your problem domain is the more valuable automated testing will be over manual. For almost anything based on user experience and especially mass-market customer-facing products, human QA is far more necessary.
In either case, the optimal operating model is that QA is embedded in your product team. They participate in writing tickets, in setting test criteria and understanding the value of the work being done. "Finding bugs" is a low value task that anyone can do. Checking for product correctness requires a lot more insight and intuition. Automated test writing can really go either direction, but typically I'd expect engineers to write unit tests and QA to write e2e tests and only as much or as little as it actually saves time and can satisfactorily indicated success or failure of a user journey.
I share the same experience that the QA team writes test plans not and “code level” tests
That said, those test plans are gold. They form the definition of the product’s behavior better than any Google Doc, Integration Test, or rotating PM ever could.
A good QA person is to a software developer as a good editor is to a writer. Both take a look at your hard work and critique it ruthlessly. Annoying as hell when it's happening, but in my experience well worth it because the end result is much higher quality.
I might just be too old, but I remember when QA people didn't typically write tests, they manually tested your code and did all those weird things you were really hoping users wouldn't do. They found issues and bugs that would be hard to universally catch with tests.
Now we hoist QA on the user.
Working with younger devs I find that the very concept of QA is something that is increasingly foreign to them. It's astounding how often I've seen bugs get to prod and ask "how did it work when you play around with it locally?" only to get strange looks: it passed the type checker, why not ship it?
Programmer efficiency these days is measured in PRs/minute, so introducing bugs is not only not a problem, but great because it means you have another PR you can push in a few days once someone else notices it in prod! QA would have ruined this.
This drives me crazy. It's a cheap way of saying we're ok shipping crap. In the past, I've been part of some QA audits where the developers claimed their customer support log sufficed as their test plan. This wasn't safety-critical software, but it did involve what I would consider medium risk (e.g., regulatory compliance). The fact that they openly admit they are okay shipping bad products in that environment just doesn't make sense to me.
I remember Steve Maguire saying this in Writing Solid Code (that they're both necessary, and both types of testing complement the other). He criticized Microsoft employees who relied on QA to find their bugs. He compared QA testing to a psychologist sitting down with a person and judging whether the person is insane after a conversation. The programmer can test from the inside out, whereas QA has to treat the program like a black box, with outputs and effects resulting from certain inputs.
+1 to this. My whole career has been spent with QA teams that behave this way.
I worked at a place with a 10 year old legacy product and a 10,000 test case spreadsheet of each manual action a QA tester must perform on that product to greenlight any individual change. Obviously this lead to huge wait times to get anything deployed. Also was pretty amusing to catch all the bugs in production that their exhaustive spreadsheet totally overlooked. Almost as though it did not have value in the first place.
Microsoft used to test this way, at least in the team I worked with. SDEs still wrote unit tests. But SDETs wrote a lot of automated tests and whatever random test tools that ended up being needed. The idea was to free up more SDE time to focus on the actual product.
I think that era is over after the great SDET layoffs of 2014/2015? Now I guess some SDE teams are tasked with this kind of dev work.
Anecdotally, from my time on Windows Vista I remember an old-school tester who didn't write any code, just clicked on stuff. From what I could tell, in terms of finding serious bugs he was probably more valuable than any of the SDETs who did write code. His ability to find UI bugs was just amazing (partly due to familiarizing himself with the feature specs, I think, and partly due to some mysterious natural ability).
I've seen good QA teams who own and develop common infrastructure, and can pursue testing initiatives that just don't fit with engineering teams. When developing a new feature, the team developing it will write new tests to cover the functionality of the feature, and will own any failures in those tests moving forward. But while they're doing that, the QA team is operating a database of test failures, performance metrics, etc. that can provide insight into trends, hot spots needing more attention, etc. They're improving the test harnesses and test frameworks so it's easier for the engineering teams to develop new tests quickly and robustly. While the engineering team probably owns all of the unit tests and some integration tests - a dedicated QA team focuses on end-to-end tests, and tests that more accurately recreate real world scenarios. Sometimes there are features that are hard to test well because of non-deterministic behavior, lots of externalities, etc., and I think QA should be seen as an engineering specialty - sometimes they should collaborate with the feature teams to help them do that part of their job better and teach them new testing techniques that may be appropriate that perhaps aren't obvious or common.
I would also second another comment that pointed out that good QA folks often know the real surface area of the product better than anyone. And good QA folks also need to be seen as good QA folks. If you have a corporate culture that treats QA folks like secondary or lesser engineers, that will quickly be a self-fulfilling prophecy. The good ones will leave all the ones who fit your stereotype behind by transitioning into dev roles or finding a new team.
The last two organizations I worked for had full QA teams with people who wrote the tests, not just test plans. The devs sometimes provided features to facilitate it, but the QA teams were the ones that constructed the tests, ran them, and decided if the software was ready to be released. Some things had manual tests, but a large percentage was fully automated.
I've only ever had an official QA team in one job, at a Fortune 1000. When I started we didn't have anyone yet, but eventually they hired an mid-manager from India and brought him over (as in relocated his whole family). He then brought on a QA person he had worked with previously.
I did not work well with the mid-manager, who was both my new boss and the QA person's (not too relevant here). However, I do give him credit for the person he hired.
That QA person, a young Indian woman with some experience, was actually phenomenal at her job, catching many mistakes of ours both in the frontend and in the APIs.
She not only did a bunch of manual testing (and thus discovered many user-facing edge cases the devs missed), she wrote all the test cases (exhaustively documented them in Excel, etc. for the higher-ups), AND the unit tests in Jest, AND all the end-to-end tests with Playwright. It drastically improved our coverage and added way more polish to our frontend than we otherwise would've had.
Did she know everything? No, there was some stuff she wasn't yet familiar with (namely DOM/CSS selectors and Xpath), and it took some back-and-forth to figure out a system of test IDs that worked well enough for everyone. She also wasn't super fluent with the many nuances of Javascript (but really, who is). There was also a bit of a language barrier (not bad, but noticeable). Overall, though, I thought she was incredible at her job, very bright, and ridiculously hard-working. I would often stay a little late, but she would usually be there for hours after the end of the day. She had to juggle both the technical dev/test tasks, the cultural barriers, and managing both up and across (as in producing useless test case reports in Excel for the higher ups, even though she was also writing the actual tests in code), dealing with complex inter-team dynamics, etc.
I would work with her again any day, and if I were in management, I'd have promoted the heck out of her, trained her in whatever systems/languages she was interested in learning, or at least given her a raise if she wanted to stay in QA. To my knowledge the company didn't have a defined promotion system though, so for as long as I was there, she remained QA :( I think it was still better than the opportunities she would've had in India, but man, she deserved so much more... if she had the opportunities I did as an American man, she'd probably be a CTO by now.
Hello. I am QA that writes tests for engineering. Technically, my title is a Software Development Engineer in Test (SDET). Not only do I write "test plans", I work on the test framework, infrastructure and the automation of those test plans.
Every company is different on how they implement the QA function. Whether it be left to customer, developers, customer support, manual only QA, or SDET. It really comes down to how much leadership values quality or how leadership perceives QA.
If a company has a QA team, I think the most success comes when QA get involved early in the process. If it is a good QA team, they should be finding bugs before any code is written. The later they are involved, the later you find bugs (whether the bugs are just "noise" or not) and then the tighter they get squeezed between "code complete" and release. I think that the QA team should have automation skills so more time is spent on new test cases instead of re-executing manual test cases.
Anyways, from my vantage point, the article really hits hard. QA are sometimes treated as second class citizens and left out of many discussions that can give them the context to actually do their job well. And it gets worse as the good ones leave for development or product management. So the downward spiral is real.
Unit tests are great when you provide data that the methods expect and are sane. It's not until users get in front of the UI and submit data that you never even thought about testing with your unit tests.
To me, unit tests are great to ensure the code doesn't have silly syntax errors and returns results as expected on the happy path of coding. I would never consider that QA no matter how much you randomize the unit test's input.
Humans pushing buttons, selecting items, hover their mouse over an element, doing all sorts of things that have no real reason but yet they are being done anyways will almost always wreck your perfect little unit tests. Why do you think we have session playback now, because no matter what a dev does to recreate an issue, it's never the exact same thing the user did. And there's always that one little WTF does that matter type of thing the user did without even knowing they were doing anything.
A good QA team are worth their weight in $someHighValueMineral. I worked with one person that was just special in his ability to find bugs. He was savant like. He could catch things that ultimately made me look better as the final released thing was rock solid. Even after other QA team members gave a thumbs up, he could still find something. There were days were I hated it, but it was always a better product because of his efforts.
Unit tests are used to test functions that have only defined inputs, and whose outputs depend only on those inputs.
You can extract a lot of business logic into those kinds of functions. There's a whole art in writing "unit testable code". Those unit tests have value.
What's left is the pile code and scenarios that need to be tested in other ways. But part of the art is in shrinking down that pile as much as possible.
The Q&A teams I've seen worked the way you describe initially except they were valuable.
They weren't there for engineering, they were there for product quality. Their expertise was that they knew what the product was supposed to do and made it did it. Things like "unit tests" help development but they don't make sure the product satisfies client requirements.
If engineering is really on top of it, they learn from QA and QA seems to have nothing to do. But don't let that situation fool you into thinking they are "just creating noise and taking away more value than they provide"
FWIW, I have seen that same model have some success, provided management is willing to stand-up for QA. When QA isn't actively writing tests, they can still provide some balance against human biases that tend toward following the easiest path. In these cases, QA provides an objective viewpoint and backstop to cost and schedule pressures that might lead to bad decisions. This might be most valuable on safety-critical code, but I suppose it can still apply at various levels of risk.
I've seen where this has went poorly as QA was slowly eroded. It became easier and easier to justify shoddy testing practices. Low-probability events don't come around often by their very nature and it can create complacency. I've seen some aerospace applications have some close calls related to shortcomings in QA integration; in those cases, luck saved the day, not good development practices.
I work with the regulated drug development industry, and believe there is a useful and important distinction between Quality Control (QC) and Quality Assurance (QA). I wonder if perhaps this distinction would be useful to software quality too.
QC are the processes that ensure a quality product: things like tests, monitoring, metrology, audit trails, etc. No one person or team is responsible for these, rather they are processes that exist throughout.
QA is a role that ensures these and other quality-related processes are in place and operating correctly. An independent, top level view if possible. They may do this through testing, record reviews, regular inspections and audits, document and procedure reviews, analyzing metrics.
Yes, they will probably test here and there to make sure everything is in order, but this should be higher level - testing against specifications, acceptability and regulatory, perhaps some exploratory testing, etc.
Critically they should not be the QC process itself: rather they should be making sure the QC process is doing its job. QA's value is not in catching that one rare bug (though they might), but in long term quality, stability, and consistency.
I think that complicates conversations like this. I’ve seen a range of QA people ranging from the utterly incompetent to people who knew the product and users better than anyone else to people writing code and tackling gnarly performance or correctness issues.
If your company hires the low end of that scale, any approach is going to have problems because your company has management problems. It’s very easy to take a lesson like “QA is an outdated concept” because that’s often easier than acknowledging the broken social system.
We have SDETs for that. And they do a great job. But QA is where polish happens. When you get good QA people, who know the app better than the developers, better than the users, who anticipate how the users will use the product? These people should be paid their weight in gold.
> Focus: There is real value in having people at your company whose focus is on the quality of your end product. Quality might be “everybody’s job”…but it should also be “somebody’s job”.
Yes indeed, naturally every person have just one focus, having dedicated person focus on QA is important.
Another practice, or buzz word (or used to be buzz word:) ), Exploratory Testing, which can pretty much be conducted only by dedicated QA.
That's pretty much my role - I don't write test cases, I'll explore the system and try to find issues that the devs have missed. Then they learn from what they missed so I have to explore more to find other types of issues.
Getting rid of QA teams are a slow ticking bomb frequently imho. Cause some issues are might not be even breaking functionality. You can mess up some tracking/analytics and managers will make wrong decisions based on incorrect data.
But personally I feel like within few years everything might change a lot. Machines 100% will be be better at coding, maintenance and testing things.
This is a symptom of a larger issue in tech where C/E suites are trying really hard to turn engineers into some sort of fungible cogs in the system that can be swapped in and out and in different parts of the system and still have everything work perfectly in order.
Yep I see this a lot where people want swappable engineers and no one is able to understand if you have an engineer working in the Frontend most of the time they will not be acquainted with backend work. Nor is there a need or logical way to keep everyone working across the stack to keep them in this swappable state. Each time you change a persons job they need retraining and orientation. Code is code but a drop down menu is not a database insert.
QA Engineers are some of the best debuggers too. They have their hands in the pipeline, src and test directories, and often work with all aspects of developing and deploying the application.
When I was a QA lead I often ran into software engineers that couldn’t be bothered to read a pipeline error message (and would complain daily in Slack) and when it came to optimizing the pipeline they would ignore base problems and pretend the issues that stemmed from the base problems were magical and not understood. Wasting days guessing at a solution.
The disrespect a QA engineer sees is not exaggerated in this article. Since most companies with QA orgs do not have a rigorous interviewing process like the Engineering orgs, the QA engineers are seen as lesser. The only SWE that have respect for them that I’ve met are the people who worked in QA themselves. The disrespect is so rampant that I myself have switched back to the Engineering org (I tried using seniority as a principal engineer and even shifted as a manager to make changes, but this failed because Engineering could not see past their own hubris and leadership peoples will not help you). My previous company before I was laid off hired a new CTO who claimed we could just automate away QA needs but had no examples of what she was talking about. This is the level of respect poured down from the top about building good software.
I stepping-stoned through QA on my way into development, now a decade something ago, and this part stands out as especially true in my experience:
> This created a self-reinforcing spiral, in which anyone “good enough at coding” or fed up with being treated poorly would leave QA. Similarly, others would assume anyone in QA wasn’t “good enough” to exit the discipline. No one recommends the field to new grads. Eventually, the whole thing seemed like it wasn’t worth it any more. Our divorce with QA was a cold one — companies just said “we’re no longer going to have that function, figure it out.”
I've worked with a handful of talented career software QA people in the past. The sanity they can bring to a complex system is amazing, but it seems like a shrinking niche. The writing was on the wall for me that I needed and wanted to get into fulltime dev as QA got increasingly squeezed, mistreated, and misunderstood. At so many companies QA roles went into a death spiral and haven't recovered.
Now, as the author points out, a lot of younger engineers have never worked with a QA team in their career. Or maybe have worked with a crappy QA team because its been devalued so much. So many people have never seen good QA that no one advocates for it.
This is why. Engineers have some of the most inflated egos that they set an extremely high bar for being “part of the club”. Sometimes that’s corporate policy (hire better than you) and sometimes it’s just toxicity (I am better than you). Without realizing that the most valuable skills they could learn are soft skills. I’m open to finding anyone willing to code. Whether it’s from QA, sales, Gerry’s nephew, recent CS grad, Designer turned coder, or that business analyst that taught themselves Python/Pandas.
A good QA team is sorely missed. A bad QA team turns the whole notion of QA teams sour. Just the same for development teams :D
I think devs are first line of defense. Unit tests etc. QA is second line (should we release?), feature testing, regression, UX continuity, etc. There’s value in it if you can afford it.
I had a boss at Yahoo who gave our QA to another team because "Facebook doesn't use QA, we shouldn't either". I can't remember if it was Facebook or MS, but he was willing to buy all of us a book talking about how amazing it was.
Long story short, it wasn't. It was like taking away a crutch. Of course we could have been more diligent about testing before having QA validate it, but it slowed development down so much trying to learn all the things we never thought to test that QA did automatically.
A bit recent to have affected Yahoo - but it sells a good story.
We would celebrate the first time someone broke something.
Let anyone touch any part of the codebase and get in there to fix a bug or build a feature. Yes, this can cause bugs. Yes, that is an acceptable tradeoff. We had a shared mythology about the times that the site went down and we all rallied together to quickly bring it back up.
Sounds like hell: running as close to the edge of the cliff as you can. Presumably totally ignoring thousands of papercuts of slightly broken functionality. Optimising to produce an infinite number of shallow bugs.
The difference is that bugs in the social network parts of Facebook (the ones where you see your friends and family's pictures and posts) are not directly making money for Facebook. The only real stuff that matters for the money is all the tracking.
I bet the people responsible for Facebook Ads Manager are a lot less enthusiastic about "move fast and break things", although I'd be interested to hear an opposing viewpoint from anyone here who's worked for that group.
This point is brought up in the article but I think it is at the real heart of the issue.
QA is almost always seen as a 'cost center' by the business and upper management. I have a hypothesis that you never ought to work in a department that is seen as a 'cost center'. The bonuses, the recognition, and the respect always goes to the money makers. The cost center is the first place to get more work with less hands, get blamed for failures, and ultimately fired when the business needs to slim up. I think the same thing applies to IT.
This spiral is why QA will always be a harder career than just taking similar skills and being a developer. It self reinforces that the best people get fed up and switch out as soon as they can.
No. That’s not actually how most orgs break it down. R&D, marketing, sales is “bringing new business” so are profit centers. This means their budget grows with revenue. Manufacturing, QA, IT and service are cost centers so get squeezed year-over-year even if revenue is flat.
It depends on the org. In my company, which is a SaaS-like company, all of product and engineering is a cost center, despite creating the product the company sells. It’s just the way they do their accounting.
Even as a developer (mobile app developer) I feel like one has to be careful not to work on "cost center" things.
Accessibility, observability, good logging, testing infrastructure improvements, CI/CD tweaks, stability, better linting and analyzer issues are all important, but you will be rewarded if you ship features fast.
This year I spent too much time on the former because I felt like that's what the team and app needed, because nobody on the team priorized these issues, and I'll be sweating at the end of the year performance reviews.
Now knowing this, I understand why the others didn't want to work on these items, so next year, I'll be wiser, and I'll focus on shipping features that get me the most visibility.
Sorry for the bugs in the app, but I need a job to pay my mortgage.
The purpose behind all those things you were pursuing (apart from accessibility) should have been to increase the rate at which the team is able to ship features. If your work on these items over the course of a year haven't demonstrably improved delivery speed, then what value did they actually bring? If they have improved delivery speed and you can show evidence for that, why would you be nervous going into a review?
Thinking this can be reduced to a single metric is the blight of modern software (and business in general, I think). Mapping an individual change to improved delivery speed is in the vast majority of cases an impossible task and any decent developer knows this. It's management+ that wants simple easy metrics since they lack the deep understanding required to do their job well. Software development is - despite management's hopes - not line work. It's much more akin to R&D. The line work gets eaten up by AWS.
If you can’t prove your contributions are worthwhile, you’re not going to get recognition. People putting out fires get recognition because they are solving visible and urgent problems. If your work reduces the chances of those fires, it should be measurable otherwise what is the point? Do you honestly believe someone who has spent a year “improving processes” but cannot measure the impact of that year of work deserves glowing praise?
And it’s not that managers don’t understand developers and the work they do. It’s that a lot of developers don’t engage at all with what a business actually does. They are working at IKEA while trying to convince management to use the nice dovetail joints instead of that garbage dowel based assembly. Not only do dovetails look better, they are substantially stronger! All very true statements from a craftsman woodworker. But a complete failure to understand the business and how and why it operates as it does and the value they are expected to provide within it.
I agree with the 'cost center' sentiment, but I'll try to add some nuance from my experience.
1) Some organizations have come to really value what QA/QC brings to the table. From my experience, this seems to be more visible in manufacturing than software. I speculate this is because software is more abstract by its very nature and waste is harder to track.
2) The really good QAs are those who really believe in its mission, rather than those who are looking for the path of least resistance.
Both of those underscore the value lies in organizations and individuals who really buy-in to the QA ethos. There are lots of examples of both who are simply going through the motions.
I've worked in cost center groups almost all my life, and I wouldn't trade it for anything.
I'm fine with bonuses, etc going to other groups. I'm paid well as a tech worker, and many of those jobs would make absolutely miserable -- assuming I turned out to actually be good at them.
I work on our frontend platform team and my perf work, CI design and process engineering definitely have a harder time getting recognition and promotions than folks who ship things that translate to dollars in the bank.
I don’t care though. I enjoy making things better and more robust. It makes my soul feel better. I’ll leave fucking things up to the cynics.
I completely agree with the sentiment of this article: It is a big problem that being a "software tester" is not at all as prestigious as being a software engineer. Having someone who really understands how the users interact with the software and systematically covers all behavior in test cases is very valuable.
I experienced both worlds: I worked in an organization where 4 QA engineers tested each release that was built by 6 software engineers. Now I'm in a situation where 0 QA engineers test the work of 8 software engineers. In the second case the software engineers actually do all the testing, but not that systematically because it's not their job to optimize the testing process.
Having someone with the capabilities of a software engineer who's daily work is uncovering defects and verifying functionality is important. Paying someone who owns the testing process is more than justified commercially. The problem is: You don't find those people. For various reasons. Therefor you are stuck with making the software engineers do the testing.
But there is hope. There is a new standard released for my industry that requires organizations to have a QA department that is independent of the software engineering department. If they don't have that, they are not allowed to role out there software product complaint to the standard. Maybe this will help to reintroduce the QA Engineer as an important and prestigious role.
Mechanical engineering and shopfloor software for factory automation in Europe. There is a new IT security standard released that also includes requirements for the software.
The main problem with QA teams is the same problem with IT teams or even management. If they are doing their jobs well they appear to be doing nothing.
This often creates a situation where people need to "justify" their jobs. Usually this happens due to an over reliance upon metrics (see Goodhart's Law) rather than understanding what the metrics are proxying and what the actual purpose of the job is. A bad QA team is one who is overly nitpicky, looking to ensure they have something to say. A good QA team simultaneous checks for quality as well as trains employees to produce higher quality.
I do feel like there is a lack of training going on in the workforce. We had the "95%-ile isn't that good"[0] post on the front page not long ago and literally it is saying "It's easy to get to the top 5% of performers in any field because most performers don't actively train or have active feedback." It's like the difference between being on an amateur sports team vs a professional. Shouldn't businesses be operating like the latter? Constantly training? Should make hiring be viewed differently too, as in "can we turn this person into a top performer" rather than "are they already" because the latter isn't as meaningful as it appears when your environment is vastly different than the one where success was demonstrated.
When I work with new devs I can often trip them up with basic tests of double-clicking, leading spaces, using the back button on Android. They then learn these and from then on these issues dont appear ( well, OK, it might take a couple of times of a ticket being rejected because of these but they do quickly learn all my tricks ) I don't get measured on bugs found so there's no pressure on me to find stupid bugs just to boost my figures.
I had a buddy who liked to do that kind of thing. I think his favorite trick was just to enter nothing in a form and hit enter to see what happens. It's probably his favorite because it deleted the database on some little thing I wrote at one point and we got a good laugh over it and I got a good lesson out of it.
> I don't get measured on bugs found so there's no pressure on me to find stupid bugs just to boost my figures.
Sounds like the right incentive structure. If you don't mind, how are you judged? Do you feel like the system you're in is creating the appropriate incentives and actually being effective? Certainly this example is but I'd like to know more details from an expert so I can update my understanding.
The system is really effective, I wanted to work at a place where the cliche "everyone cares about quality" is actually true and I found it - devs test, designers test, I test, customer has the chance to test the latest build every 2 weeks so that we can check that our quality checks are aligning with theirs. It gets to be a game of 'can the devs get it past my checks' and 'can I find new ways to trip them up' which builds up confidence in each others skill levels.
> Shouldn't businesses be operating like the latter? Constantly training?
There's a rampant cultural mind-virus that argues that 95%th percentile is somehow tons of work (rather than a lack of unforced mistakes), so everyone just writes it off. It's on full display at this very site. Just look on any post involving software quality, and read a bunch of comments suggesting widespread apathy from engineers.
Obviously every situation is different, but people seem to be pretty okay with relinquishing agency on these things and just going along with whatever local maxima their org operates in. It's not totally their fault, but they're not blameless either.
Yeah it is weird that it is believed that there is a linear scale to work in and quality considering how well known pareto/power distributions are. These distributions are extremely prolific too. I mean we even codify that sentiment in the 80/20 rule or say that 20% of time is writing code and 80% is debugging it. What's interesting is this effect is scalable. Like you see this when comparing countries by population but the same distribution (general shape) exists when looking at populations of states/regions/cities (zooming in for higher resolution) for any given country or even down to the street level.
> Obviously every situation is different, but people seem to be pretty okay with relinquishing agency on these things and just going along with whatever local maxima their org operates in. It's not totally their fault, but they're not blameless either.
I agree here, to the letter. I don't blame low level employees for maximizing their local optima. But there's two main areas (among many) that just baffles me. The first is when this is top down. When a CEO and board are hyper focused on benchmarks rather than the evaluation. Being unable to distinguish the two (benchmarks and metrics are guides, not answers). The other is when you have highly trained and educated people actively ignoring this situation. To have an over-reliance on metrics and refusing to acknowledge that metrics are proxies and considering the nuances that they were explicitly trained to look for and is what meaningfully distinguishes them from less experienced people. I've been trying to coin the term Goodhart's Hell to describe this more general phenomena because I think it is a fairly apt and concise description. The general phenomena seems prolific, but I agree that the blame has higher weight to those issuing orders. Just like a soldier is not blameless for their participation in a war crime but the ones issuing the orders are going to receive higher critique due to the imbalance of power/knowledge.
Ironically I think we need to embrace the chaos a bit more. But that is rather in recognizing that ambiguity and uncertainty is inescapable rather than abandonment of any form of metric all together. I think modern society has gotten so good at measuring that we often forget that our tools are imprecise whereas previously the imprecision was so apparent that it was difficult to ignore. One could call this laziness but considering its systematic I'm not sure that's the right word.
> To have an over-reliance on metrics and refusing to acknowledge that metrics are proxies and considering the nuances that they were explicitly trained to look for and is what meaningfully distinguishes them from less experienced people. I've been trying to coin the term Goodhart's Hell to describe this more general phenomena because I think it is a fairly apt and concise description.
> This often creates a situation where people need to "justify" their jobs.
On DevOps teams I see this constantly. Usually the best-compensated or most senior "Ops" guy or whatever they're called at the company spends a lot of his time extinguishing fires that were either entirely of his own creation/incompetence, which makes it look like he's "doing something." You automate away the majority of the toil there and this person doesn't have a job, yet this pattern is so insanely common. There's little incentive to do it right when doing it right means management thinks you sit there all day and do nothing.
That's a fantastic example of what I'm trying to describe. It's kinda like thinking hours worked is directly proportional to widgets produced. There certainly are jobs and situations where this relationship holds (can't sell widgets if you aren't manning the store or can't produce turn crank widgets if the crank isn't being turned). But modern world widgets don't work that way and are more abstract. Sometimes fewer hours creates more widgets, sometimes the reverse. But widget production is now stochastic and especially in fields where creativity and brain power are required. (Using widgets for generalization -- in the economic sense--, insert any appropriate product or ask and I'll clarify)
Which is one reason DevOps teams don't make sense. DevOps is a skill developers need to have. It needs to be embedded within the development team, not some other team's responsibility who only focuses on "DevOps" work. You create the build and deployment pipelines and move on to other project work. If you give someone a role and say their job is to do "DevOps" they will HAVE to invent things to do because that's such a small part of a project and once implemented doesn't need a ton of maintenance.
It is not about DevOps. In every organization if you do your work good and have no fuckups - you will not be recognized and promoted. But if you are hero-firefighter - managers will love you and help with promotion.
Because visibility is a key! Key to everything in the corporate life. If your work is not visible for managers - you are doing nothing.
Placing value based on the work assigned? Obviously insanely easier said than done. But I think we should embrace the fuzziness a bit. Your manager should have a very good idea of this. If they don't, then they're the ones that should be let go because this is a significant aspect of their job. (may be learned through indirect methods)
> That doesn’t seem a little insane to you?
Honestly, a lot of business practices and economics sounds insane to me. Similarly a lot of alternative suggestions (especially the latter) since they tend to not address the underlying issues but be bandaids. The ones that don't sound insane are often boring and very reasonable, but I think we've established that that's considered undesirable when you're evaluated by visibility. I think this is probably a significant part of the negative feedback loop.
The fortunate thing is there are measurable metrics you can hit and improve upon as an X-Ops team, but implementing those takes buyin from the org or enough freedom to go rogue and do it on your own.
Definitely agree. In a sister comment[0] I mention a concept I've been calling "Goodhart's Hell" for a more general term. But I think what you are specifically mentioning is sometimes laughably called "Loud Laboring"[1]. I think this all falls under the broader umbrella of metric hacking and thus Goodhart's Law.
Idk why, but it really does seem like metric hacking is extremely pervasive in our modern society, and can be found nearly everywhere. What upsets me the most is that it too is found in the sciences. There also appears to be a strong correlation between the popularity (or hype) of a field and metric hacking.
Sometimes people want to be firefighters to protect the people they serve but others love how being in a crisis makes them feel alive. The later are why arson investigators first look at firefighters when doing an arson investigation. Some of them need the fires and will start them just to fight them.
There are a lot of these types that gravitate to crisis management roles like DevOps.
> Shouldn't businesses be operating like the latter? Constantly training?
"But then they'll leave for somewhere else for more money." </sarcasm>
Literally every company I have worked for. Meaningful training was always an uphill battle.
However, good training also requires someone good and broad in your technical ladder. They may not be the most up-to-date, but they need to be able to sniff out bullshit and call it out.
I noticed this trend a couple of years ago, they called it Shift-Left.
Basically, it was a way to get a developer to do more testing during development before handing over a feature to QA. This sets up the imagined possibility of firing all QA staff and having developers write perfect code that never needs to be tested thoroughly. Looks great on paper...
At a previous company, they started firing all of the manual QA devs and replacing them with offshore QA people who could do automated testing development (Cypress tests). The only problem was that those fired QA team members had significant business domain knowledge that never was transferred or written down. The result was a dumpster fire every week, with the usual offshore nightmares and panicked responses.
Make no mistake about this, it's just a cost-cutting measure with an impact that is rarely felt immediately by management. I've worked with stellar manual QA people and our effort resulted in bulletproof software that handled edge cases, legacy customers, API versions, etc. Without these people, your software eventually becomes IT-IS-WHAT-IT-IS and your position is always threatened due to the lack of quality controls.
QA has always been about risk management. There are multiple ways to manage risk, and some of those ways can be more cost effective to a business. As software shifted towards SaaS offerings, deployments (and rollbacks) became quicker, customer feedback loops also got lightning fast. Team's can manage the risk of a bug more efficiently by optimizing for mean-time-to-recovery. This muscle is not one that QA teams are particularly optimized for, thus their effectiveness in this new model was reduced. I've found that holding on to QA function in this environment can severely dilute the ownership of quality as a requirement from engineers.
QA is still extremely valuable in any software that has long deployment lead times. Mobile apps, On-Prem solutions, anything that cannot be deployed or rolled back within minutes can benefit from a dedicated QA team that can manage the risk appropriately.
There are so so many instances where "rolling back" is just not a feasible solution. Working for a SaaS company with mobile/web/api and huge db's, migrations, payroll uses in the product, rolling back is and should always be a LAST RESORT. In 99% of the cases, something significant enough to want to roll back usually results in a "hot patch" workflow instead because rolling back or etc has its own risk.
> QA has always been about risk management.
100%.
QA should be related to identifying risk, likelihood of failure, impact of failure to user, client and company. The earlier this is done in the varying processes, the better. ("shift left" but I've seen a ton of differences with how people describe this, but generally QA should start getting involved in the "design phase")
Another example from my own first-hand experience:
A company I worked for made a product that plugged into machines that were manufacturing parts, and based on your parameters it would tell you whether or not the part was "good" or "bad".
When interviewing the leadership of the company, as well as the majority of the engineering group, "what is the biggest risk with this product" they all said "if the product locks up!". Upon further discussion, I pulled out a much larger, insidious risk; "what if our product tells the client that the part is 'good' when it is not?"
In this example, the part could be involved in a medical device that keeps someone alive.
Agree with the sentiment of this article but the disturbing ai generated images every paragraph were definitely not necessary - do people actually need to see these?
Systems used by humans have to be tested by humans. Those testers can either be your customers, or your QA team - as your dev and sales teams will be busy doing other things.
> The most conscientious employees in your organization are the most bitter. They see the quality issues, they often address them, and they get no recognition for doing so. When they speak up about quality concerns, they get treated like mouthbreathers who want to slow down. They watch the “move fast and break things” crowd get rewarded time after time, while they run around angrily cleaning up their messes. To these folks, it feels like giving a damn is a huge career liability in your organization. Because it is.
This is the bitter truth, no one wants to acknowledge.
DBAs and Infra, are in the same boat as QAs. Pendulam will swing back in not so long time frame i hope.
The hard part for most people is learning to tell the devil in a necktie to fuck off every time they try to sweet talk you into volunteering for that kind of pain. Also with no recognition or compensation.
All it takes is one person on my team to defect and support an untenable amount of tech debt, and everyone on my team has to pay for it.
While QA and testing is important, I’m not sure you can convince the people only concerned about profits… I think the “release it fast and patch it later” concept here to stay due to the internet being so accessible. Why bother spending tons of money and time when the users will just report the bugs and you can release updates over the internet they can download. Ever since physical copies of video games and software were replaced mainly by downloads, it seems like patching is cheaper. Of course this leads to horrendous security issues, bad user experience, etc. but who cares as long as the guy on top is maximizing profits.
“release it fast and patch it later” You are testing just different things eg is this product viable in the market? QA has its place but you should not cram it into every corner of software development and also the same time, not everything should be "release fast and break things".
I’m not so sure. I think software trends in the last 10-15 years or so have been driven heavily by a small number of influential players (FAANG, mega-VCs).
An influx of dumb money like that can shape and distort the market and overwhelm the feedback loops that would otherwise give consumers influence.
Perhaps now rates are up, the equilibrium changes, but I think it’s still easy to overestimate the number of “first movers” in an industry and the power of tacit or unconscious collusion.
Yeah this hit home so hard.
Had to leave the startup i loved working at, because started doing all this release fast crap. And somehow people think releasing fintech stuff fast and untested is fine.
I've worked with conscientious engineers. Sometimes they are right but their delivery mechanism is broken. Sometimes they are just in the wrong place. If we're building a POC SASS product, it really doesn't need the quality of a avianoics microcontroller. All these trade-offs come with risk and cost, good engineers need to know the difference.
I think it is a matter of perspective as who has "broken delivery mechanism".
Would you consider expectations of fixing unique key constraints in 8 year old database tables, which are regularly cause of bugs, "avionics grade quality expectations"?
And this is one of many such things I have seen first hand.
Having a QA team is like having an Ops team with stuff being 'thrown over the wall' to the downstream.
There's two kinds of tests. Regression testing, that should be automated and written and maintained by devs. New feature or change testing should be done by those that defined them, namely Product people. In the best case it's an iterative and collaborative process, where things can be evaluated in dev/test environments, staging environments, or production for beta flag enabled test users.
I worked at two companies 15-20 years ago that invested in top-tier QA teams. They were worth their weight in gold. The products were world class because the QA team were fantastic at finding bugs we developers didn't think of looking for because we were too close to the problem. We are too used to looking at the happy path.
One key attribute to both companies is that it was dictated from on high that the QA team had final say whether the release went to production or not.
These days companies think having the developers write automated tests and spend an inordinate amount of time worrying over code coverage is better. I can't count how many products I've seen with 100% code coverage that objectively, quantifiably doesn't work.
I'm not saying automated testing is bad. I'm saying, just as the author does, that doing away with human QA testers is.
I've seen QA/QE greatness and it was similar to how you describe. A different chain of command for deciding if releases are certified for production. Different incentive structures as well.
Not to mention, at one recent employer, the QE team wrote an enormous amount of code to perform their tests - It was more LOC than the modules being tested/certified.
I had a team like that once. It was glorious. And ultimately, I'm convinced it led to overall faster development cycles because the baseline code quality & documentation was so much better than it would have been without such a great QA manager. The QA team, of course, was also technical -- mostly with SWE backgrounds -- and they were primarily colo'd in the same office as the dev team. I still remember the epiphany everyone had one planning cycle when it was mutually understood that by generally agreeing to use TDD, the QA team could participate actively in the real engineering planning and product development process.
... Then I left and my CIO let go the onshore QA team in favor of near term cost savings. Code quality went way down and within a year or two several apps needed to be entirely rewritten. Everything slowed down and people started pointing fingers, and before you knew it, it was time for "cloud native rearchitecting/reengineering" which required an SI to come in with "specialists".
At the first company we released a few times a year. You have to remember that CI/CD wasn't a thing back then. While I worked on a web-based product at this company, it was a time when it was still very common to release software on physical media like CD's.
At the second company we integrated Agile so we released every two or weeks (I can't recall which). The first time I heard about CI/CD was at this company from co-workers who came back from a conference all excited about it.
I love my really thorough QA’s. Yes it’s an antipattern to let me as a dev lean too much on them catching what I won’t. But where I dread even running the code for a minute, they enjoy it. They take pride in figuring out edge cases far beyond any spec. They are definitely worth their weight in gold. It lets developers have confidence when changing things in the same sense a good type system does. For some classes of very interactive apps (e.g games) having unit and integration tests just doesn’t cover the parameter space.
People here are talking about skillful QA worth their weight in gold.
Unfortunately people in the industry who have actual power in planning budgets don't think so. An article is right. QA engineers now are viewed as janitors: no one respects then, better to outsource to cheap location.
The QA culture has to be there, not just dictates that the QA has final say.
I've seen companies where that's true and it was still trash because the QA were mostly low-paid contract workers who only did exactly what they were told and no more.
That’s not because “QA had the final say”, it’s because your QA team were ass clowns. Any QA team that discourages dev from writing or running tests needs to be burnt to ground and rebuilt.
Test code is just code. If you can write test code you can write production code. If you can write production code you can write tests.
If your concern is that devs don't have the right mindset for testing, you can have them collaborate with a QA specialist to define the test cases and review the test implementation.
Developers need to write tests so they understand how to structure the application in a manner that can be tested. The test engineer brings a fresh set of eyes and different expectations about how the application should work. That's valuable too.
Devs that don't test their own code are usually wasting the QA team's time with garbage. It also tends to cultivate (or is a symptom of) an environment where groups are throwing projects over the wall, so to speak, without tightly integrating QA into the process. This wastes significant amounts of time.
Testing is continuous multilayered multistaged process.
Testing should start before first developer wrote first line of code for the project. Architecture blueprints, set of requirements should be tested as early as possible. But that is not happening in real life, only in books. In real life pm will bring cheap contractor from India one month before the target release date.
> I've seen developers writing tests with no assertions
This can be OK if the code executing without throwing exceptions is itself testing something. If you have a lot of assertions written directly into the code, as pre- or post-conditions for instance. But I'm guessing that wasn't the case here.
Why would it be ok to not test the assertions would be triggered if the conditions are bad? How would you verify that the assertions are correct with just a happy path? If you run a code with pre-post assertions, then remove all the assertions the same test will continue to pass.
Agreed: such tests are not sufficient to test the pre- and post-conditions themselves, and are definitely happy-path focused. But such tests may be sufficient to test the happy-path code itself (which the in-code checks are also testing). It would be ridiculous to have an entire test suite with no assertions for these reasons, but it's not ridiculous (in isolation) to see some unit tests with no assertions. In theory -- and this is not advice: theory and practice are different -- but in theory, if you've absolutely air-tightly tested the "happy path", then you've basically proven no other paths occur (short of cosmic rays).
But you could actually make this same argument about unit tests with assertions: do you have any test that those assertions are written correctly? Do you unit test your unit tests? Pre- and post-conditions living "in situ" are the equivalent thing to "control" and test assertions (respectively) living in your unit tests. Would you be more comfortable if the in-situ checks were simply cut/pasted into the unit test? I wouldn't be!
I can't count how many products I've seen with 100% code coverage that objectively, quantifiably doesn't work.
That’s because code coverage doesn’t find the bugs that result from code you didn’t write, but should have. Code coverage is but one measure, and to treat it as the measure is folly.
(But, yes, I have heard a test manager at a large software company we’ve all heard of declare that test team was done because 100% coverage.)
It's more because code coverage isn't measured correctly. What you want is branch coverage.
If you have something like this:
if condition1:
do_something1()
if condition2:
do_something2()
if condition3:
do_something3()
There are 8 possible paths for the code to follow here, but you can cover 100% of lines of code in one test. If your code coverage measure tells you that testing the case where conditions 1, 2, and 3 are all true achieves 100% coverage, your measure is worthless. Covering this code cannot be done in less than eight tests.
They do, but that's not the point GP made. An example how this could fail is if the first branch sets up something that branch two uses. Two tests are written that call branch 1&2 and not call them. 100% code coverage, even the non-happy path was tested.
But in reality, if those branches have any interaction, you would need to write eight test cases for every combination of branches being run and not being run.
Well, again, eight is a lower bound. If the software uses any values beyond the bare minimum (here) of three separate booleans, you're likely to need more.
Take this simplified code:
if condition1:
do_something()
You need one test for code coverage and two for branch coverage, but if there's a bug in the code as written you'll find that you actually need more. Assume that we want to run do_something() when, and only when, condition2 is true.
Now we have four cases:
- condition1 && condition2: do_something() runs, which is correct.
- condition1 && !condition2: do_something() runs, which is incorrect.
- !condition1 && !condition2: do_something() does not run, which is correct.
- !condition1 && condition2: do_something() does not run, which is incorrect.
If you happened to write tests for the first and third of those options, your tests will make it look like your code works, and your (official) coverage metrics will look comprehensive. But because your coverage metrics (in reality) are actually terrible, your code is secretly not working.
The big problem we've just run into is that the number of test cases we need depends on the number of bugs in our code. If the goal of writing tests is to prove that we don't have bugs, this is a terrible result; it means our goal is fundamentally impossible to achieve. We don't know whether we have bugs, so we don't know how many tests we need.
No, it's more because when the bug was filed, you said to yourself, "oh, was I supposed to update the $HOOZIWIZIT in that transaction, too? Oops." Code coverage won't save you from that. Matching test cases to requirements, or summat, can be helpful, but good golly it's a bunch more paperwork.
Your point is a good one, but not where I was going.
The best approach I’ve seen is to do all of the above.
- Have QA run pessimal versions of real use cases. Trying to sell a word processor to lawyers? Format the entire US legal code and a bajillion contracts in it, then duplicate it 10x and start filing usability/performance bugs.
- Have the engineers test everything with randomly generated workloads before committing. Run those tests nightly, and fix all the crashes / failures.
- Have Product Management (remember them?) work with marketing and sales to figure out what absolutely has to ship, and when.
Make sure it only takes one of the above three groups to stop ship, and also to stop non-essential development tasks.
We try to do this, QA owns deployment to production and has the final say if a feature needs a re-write. The almost adversarial incentives of dev and QA are why it works. Dev wants to close ticket, QA wants the feature to work, Product wants to tick a box, all collaborate so that the closed ticket delivers a working feature and not just one of those three things.
276 comments
[ 4.3 ms ] story [ 270 ms ] threadI use a lot of MSFT software and services in the "day job". I wish there was some kind of consequence to them for their declining quality.
This is exactly the point I was going to make. Their stock price is doing great! So obviously they've done the right thing for their position in the market: the market has rewarded them for not wasting money on QA and just letting users suffer with the bugs.
>I wish there was some kind of consequence to them for their declining quality.
If people keep insisting on throwing money at them no matter how bad their software is, then there's no reason for them to improve their quality.
There are no "Software Quality Assurance" academic degrees, there's barely any research into testing methodologies, there's barely any commercial engagement in the space aside from test run environments (aka, selling shovels to gold diggers), and let's face truth, also in tooling. And everything but software QA is an even worse state, with "training" usually consisting of a few weeks of "learning on the job".
Basic stuff like "how does one even write a test plan", "how does one keep track of bugs", "how to determine what needs to be tested in what way (unit, integration, e2e)" is at best cargo-culted in the organization, at worst everyone is left to reinvent the wheel themselves, and you end up with 13 different "testing" jobs in a manually clicked-together Jenkins server, for one project.
> Defect Investigation: Reproduction, or “repro”, is a critical part of managing bugs. In order to expedite fixes, somebody has to do the legwork to translate “I tried to buy a movie ticket and it didn’t work” into “character encoding issues broke the purchase flow for a customer with a non-English character in their name”.
And this would normally not be the job of a QA person, that's 1st level support's job, but outsourcing to Indian body shops or outright AI chatbots is cheaper than hiring competent support staff.
That also ties in to another aspect I found lacking in the article: users are expected to be your testers for free aka you sell bananaware. No matter if it's AAA games, computer OSes, phones, even cars...
I can tell you, I definitely didn't get training for QA tasks, but here I am doing them anyways. It's just work that needs to be done.
Yeah and that is my point. It would be way better for the entire field of QA if there were at least a commonly agreed base framework of concepts and ways to do and especially to name things, if alone because the lack of standardization wrecks one's ability to even get testers and makes onboarding them to your "in house standards" a very costly endeavour.
All of the above exist, the International Software Testing Qualification Board is one of the entities offering certifications in this: https://www.istqb.org/
There's a lot of this actually. Entire communities of people working on software quality assurance. Practitioners in this space call their field "resilience engineering".
The field likes to talk a lot about system design. Especially in the intersection of humans and machines. Stuff like "How do you set up a system (the org) such that production bugs are less likely to make it all the way to users"
What training is in your opinion needed?
I have only had QA teams that wrote "test plans" and executed them manually, and in rarer cases, via automated browser / device tests. I consider these types of tests to be valuable, but less so than "unit tests" or "integration tests".
With this model, I have found that the engineering team ends up being the QA team in practice, and then the actual QA team often only finds bugs that aren't really bugs, just creating noise and taking away more value than they provide.
I would love to learn about QA team models that work. Manual tests are great, but they only go so far in my experience.
I'm not trying to knock on QA folks, I'm just sharing my experience.
but every once and a while you ran across a QA organization that actually had a deep understanding of the problem domain, and actually helped drive development. right there alongside dev the entire way. not only did they improve quality, but they actually saved everyone time.
It worked exceptionally well, in my opinion but the QA revolted; they didn't like it, said that it felt like we were trying to make them into software developers. And so it ended, and it was back to "throw it over the wall".
Sometimes that’ll get caught in code review, if your reviewer is thinking about the implementation.
I’ve worked in payroll and finance software. I don’t like it when users are the ones finding the bugs for us.
These "Customer Support" reps, when functioning as QA, knew the product better than product or eng, exactly how you're describing. I did enjoy that model, but they also did not write tests for us. They primarily executed manual test plans, after deploys, in production. They did provide more value than creating noise, but the engineering team still was QA, at least from an automated test standpoint.
Especially for complex products that are based on users chaining many building blocks together to create something useful, devs generally have no visibility into how users work and how to test.
Not disagreeing with this, but there's one thing they won't always be aware of. They won't always know what code a dev touched underneath the hood and what they might need to recheck (short of a full regression test every single time) to verify everything is still working.
I know that the component I adjusted for this feature might have also affected the component over in spots X, Y, and Z, because I looked at that code, and probably did a code search or a 'find references' check at some point to see where else it's getting called, and also I usually retest those other places as well (not every dev does, though. I've met some devs that think it's a waste of time and money for them to test anything and that's entirely QA's job).
A good QA person might also intuit other places that might be affected if it's a visible component that looks the same (but either I haven't worked with too many good QA people or that intuition is pretty rare, I'm guessing it's the latter because I believe I have worked with people who were good at QA). Because of that, I do my best to be proactive and go "oh by the way this code might have affected these other places, please include those in your tests".
This is a good point, but there are some QA that do review code (source: me - started career in QA and transitioned to dev). When making a test plan, an important factor is risk assessment. If QA has a hunch, or better when the dev lead flags complex changes, the test plan should be created and then the code diffs should be reviewed to assess whether or not the plan needed revising. For example, maybe the QA env doesn’t have a full replica of prod but a query is introduced that could be impacted if one of the joining tables is huge (like in prod). So maybe we’d adjust the plan to run some benchmarks on a similar scale environment.
I’m definitely biased since I started in QA and loved it. To me, good QA is a cross section of many of the things people have mentioned - technical, product, ops, security - with a healthy dash of liking to break things. However, reality is that the trend has been to split that responsibility among people in each of those roles and get rid of QA. Works great if people in each of those job functions has the bandwidth to take on that QA work (they’ll all have a much deeper knowledge of their respective domains). But you’ll lose coverage if any one of those people don’t have time to dedicate to proper QA.
(I’ll also completely acknowledge that it’s rare to have a few, let alone a full team, of QA people who can do that.)
It doesn't necessarily matter what code was changed, a change in code in Module A can cause a bug in Module B that hasn't been changed in a year. A QA test plan should cover the surface area of the product as used by consumers whoever they might be. While knowing some module had fixes can inform the test plan or focus areas when the test schedule is constrained, only testing changes is the road to tears.
I'm not in QA, I write code, so I defer to whatever they decide for these things usually, these are just observations from what I've seen.
I just try to make sure I test my code enough that there isn't anything terribly broken when I check it in and fixes I need to make tend to be relatively minor (with a few exceptions in my past).
Also I'm not necessarily talking basic functionality here. I'm currently working for a client that's very picky about the look and feel, so if a few pixels in padding get adjusted where it's noticeable, or a font color or size gets adjusted a bit, in one place and it affects something else, there could be complaints. And a test plan is not likely to catch that, at least not any on any projects I've worked on.
Not really. As QA I always reviewed the checkins since yesterday before opening up the daily build. Between the bug comments and the patch comments, even if the patch itself is a bit Greek to me, I can tell what was going on enough to be a better tester of that area.
Six months later I work on a new page that incorporates that component that requires some sort of change in that component that could theoretically break it elsewhere, at least how it looks or something, like alters the padding or something in a way that shoves things onto the next line in one of the instances.
If I don't mention what other places that component is used in my comments, are you going to know where all to look for the other instances of that component? Even if you look at the source code, it will only show changes, it won't say 'oh this component is used elsewhere' unless you actively bring up the source code yourself and do a code search yourself.
Because it's a visible component, maybe you'll just have the memory that something that looks like this is elsewhere, but maybe you won't remember. Or maybe you're fairly new to the project and aren't aware of these other parts of the site using it.
What about something that's even less visible, like something that takes in data and then sorts, transforms, and adds metadata to it? Then I make a change to it to accomodate for some new requirement on a new page. Maybe it's used for four other locations, but the changes only shows the function being altered and the new page using it. That change I made broke it elsewhere, but I don't mention where else to check in my notes. Are you going to think to check each place where data is loaded to make sure it's working correctly, without me explicitly telling you to check those spots?
If you can, great, you've got a real intuition for this that I doubt many QA people have. I don't even know to check these other places sometimes, and I worked on the code myself and can do a quick text search to see where else it's being used.
This was why I said what I did. It's not just about 'that area', the change could potentially affect another application in different repository entirely sometimes, especially if a shared library is involved. I've literally had like four or five of these situations happen in the past six months of my current project, where my change broke something elsewhere.
Usually I caught it, but I also told QA to check it explicitly anyway since I don't usually test things 100% (since I'm not QA). When I was a more junior developer, I missed these things more often and QA would sometimes miss it also since they didn't know to retest something.
Where I've seen QA teams most effective is providing more function than "just" QA. I've seen them used for 2nd tier support. I've seen them used to support sales engineers. I've also seen QA teams that take their manual test plans and automate their execution (think Selenium or UiPath) and have seen those automations included in dev pipelines.
Finally, the QA team are the masters and caretakers of your test environment(s), all the different types of accounts you need for testing, they should have the knowledge of all the different browsers and OSes your customers are using, and so forth.
That's a lot for the dev team to take on.
A separate QA person coming at it from the customer's perspective will do a test that's much more likely to reflect reality.
So you can make your QA teams create plenty of tests if you give them the right tools.
Maybe recorded integration tests are run on every release instead of every commit? Maybe the QA team uses them less to pass/fail work and more to easily note which parts of the product have changed and need attention for the next release? There's lots of possibilities.
That would limit the frequency of releases.
Every time I worked with dedicated QA teams, we (core engineering) ended up leading the effort of writing maintainable automated tests. At some point, it was much easier to just do it ourselves.
This is especially important in complex UIs where everything from localization to state management to A/B testing can accidentally break things you didn't even mean to touch.
Isn't it better for the devs to spend a few minutes looking at a broken test (which they can easily fix themselves) than for a user to discover it and have to go through triage months later, when it's out of everyone's memories already?
I completely understand there are pros and cons to every set up, but if higher velocity is more important than complete correctness, such set up might be detrimental.
> but if higher velocity is more important than complete correctness, such set up might be detrimental.
I guess it's a cultural value thing: move fast and break things vs polish before shipping. As a user I always prefer the latter, but hell, I don't run these multibillion dollar companies that are always shipping buggy things, lol
I have a few times. But the only common thing in the QA industry, is that every company does it differently and think they're doing it the "normal way".
People wrote off QA completely unless it meant they didn't have to write tests, but, it didn't track from my (rather naive) perspective that tests are _always_ part of coding.
From that perspective, it seemed QA should A) manage go/nogo and manual testing of releases B) keep the CI green and tasks assigned for red (bonus points if they had capacity to try fixing red) C) longer term infra investments, ex. what can we do to migrate manual testing to integration testing, what can we do to make integration testing not-finicky in the age of mobile
I really enjoyed this article because it also indicates the slippery slide I saw there: we had a product that had a _60% success rate_ on setup. And the product was $200+ dollars to buy. In retrospect, the TL was into status games, not technical stuff, and when I made several breakthroughs that allowed us to automate testing of setup, they pulled me aside to warn me that I should avoid getting into it because people don't care.
It didn't compute to me back then, because leadership _incessantly_ talked about this being a #1 or #2 problem in quarterly team meetings.
But they were right. All that happened was my TL got mad because I kept going with it, my skip manager smiled and got a bottle of wine to celebrate with, I got shuffled off to work with QA for next 18 months, and no one ever really mentioned it again.
the one exception to this was when i was qa (never again) and i made sure we only ever did automated tests. unfortunately management was nonexistent, devs made zero effort to work with us, and naturally we were soon replaced by a cheap offshore indian team who couldn't tell you the difference between a computer and a fridge anyway.
i think a lot of it just stems from companies not caring about qa, not knowing who to hire, and not knowing what they want the people they hire to achieve. "qa" is just like "agile", where nobody can be bothered to actually learn anything about it, so they make something up and then pat themselves on the back for having it.
In either case, the optimal operating model is that QA is embedded in your product team. They participate in writing tickets, in setting test criteria and understanding the value of the work being done. "Finding bugs" is a low value task that anyone can do. Checking for product correctness requires a lot more insight and intuition. Automated test writing can really go either direction, but typically I'd expect engineers to write unit tests and QA to write e2e tests and only as much or as little as it actually saves time and can satisfactorily indicated success or failure of a user journey.
That said, those test plans are gold. They form the definition of the product’s behavior better than any Google Doc, Integration Test, or rotating PM ever could.
I might just be too old, but I remember when QA people didn't typically write tests, they manually tested your code and did all those weird things you were really hoping users wouldn't do. They found issues and bugs that would be hard to universally catch with tests.
Now we hoist QA on the user.
Working with younger devs I find that the very concept of QA is something that is increasingly foreign to them. It's astounding how often I've seen bugs get to prod and ask "how did it work when you play around with it locally?" only to get strange looks: it passed the type checker, why not ship it?
Programmer efficiency these days is measured in PRs/minute, so introducing bugs is not only not a problem, but great because it means you have another PR you can push in a few days once someone else notices it in prod! QA would have ruined this.
This drives me crazy. It's a cheap way of saying we're ok shipping crap. In the past, I've been part of some QA audits where the developers claimed their customer support log sufficed as their test plan. This wasn't safety-critical software, but it did involve what I would consider medium risk (e.g., regulatory compliance). The fact that they openly admit they are okay shipping bad products in that environment just doesn't make sense to me.
I worked at a place with a 10 year old legacy product and a 10,000 test case spreadsheet of each manual action a QA tester must perform on that product to greenlight any individual change. Obviously this lead to huge wait times to get anything deployed. Also was pretty amusing to catch all the bugs in production that their exhaustive spreadsheet totally overlooked. Almost as though it did not have value in the first place.
I think that era is over after the great SDET layoffs of 2014/2015? Now I guess some SDE teams are tasked with this kind of dev work.
I would also second another comment that pointed out that good QA folks often know the real surface area of the product better than anyone. And good QA folks also need to be seen as good QA folks. If you have a corporate culture that treats QA folks like secondary or lesser engineers, that will quickly be a self-fulfilling prophecy. The good ones will leave all the ones who fit your stereotype behind by transitioning into dev roles or finding a new team.
I did not work well with the mid-manager, who was both my new boss and the QA person's (not too relevant here). However, I do give him credit for the person he hired.
That QA person, a young Indian woman with some experience, was actually phenomenal at her job, catching many mistakes of ours both in the frontend and in the APIs.
She not only did a bunch of manual testing (and thus discovered many user-facing edge cases the devs missed), she wrote all the test cases (exhaustively documented them in Excel, etc. for the higher-ups), AND the unit tests in Jest, AND all the end-to-end tests with Playwright. It drastically improved our coverage and added way more polish to our frontend than we otherwise would've had.
Did she know everything? No, there was some stuff she wasn't yet familiar with (namely DOM/CSS selectors and Xpath), and it took some back-and-forth to figure out a system of test IDs that worked well enough for everyone. She also wasn't super fluent with the many nuances of Javascript (but really, who is). There was also a bit of a language barrier (not bad, but noticeable). Overall, though, I thought she was incredible at her job, very bright, and ridiculously hard-working. I would often stay a little late, but she would usually be there for hours after the end of the day. She had to juggle both the technical dev/test tasks, the cultural barriers, and managing both up and across (as in producing useless test case reports in Excel for the higher ups, even though she was also writing the actual tests in code), dealing with complex inter-team dynamics, etc.
I would work with her again any day, and if I were in management, I'd have promoted the heck out of her, trained her in whatever systems/languages she was interested in learning, or at least given her a raise if she wanted to stay in QA. To my knowledge the company didn't have a defined promotion system though, so for as long as I was there, she remained QA :( I think it was still better than the opportunities she would've had in India, but man, she deserved so much more... if she had the opportunities I did as an American man, she'd probably be a CTO by now.
Every company is different on how they implement the QA function. Whether it be left to customer, developers, customer support, manual only QA, or SDET. It really comes down to how much leadership values quality or how leadership perceives QA.
If a company has a QA team, I think the most success comes when QA get involved early in the process. If it is a good QA team, they should be finding bugs before any code is written. The later they are involved, the later you find bugs (whether the bugs are just "noise" or not) and then the tighter they get squeezed between "code complete" and release. I think that the QA team should have automation skills so more time is spent on new test cases instead of re-executing manual test cases.
Anyways, from my vantage point, the article really hits hard. QA are sometimes treated as second class citizens and left out of many discussions that can give them the context to actually do their job well. And it gets worse as the good ones leave for development or product management. So the downward spiral is real.
To me, unit tests are great to ensure the code doesn't have silly syntax errors and returns results as expected on the happy path of coding. I would never consider that QA no matter how much you randomize the unit test's input.
Humans pushing buttons, selecting items, hover their mouse over an element, doing all sorts of things that have no real reason but yet they are being done anyways will almost always wreck your perfect little unit tests. Why do you think we have session playback now, because no matter what a dev does to recreate an issue, it's never the exact same thing the user did. And there's always that one little WTF does that matter type of thing the user did without even knowing they were doing anything.
A good QA team are worth their weight in $someHighValueMineral. I worked with one person that was just special in his ability to find bugs. He was savant like. He could catch things that ultimately made me look better as the final released thing was rock solid. Even after other QA team members gave a thumbs up, he could still find something. There were days were I hated it, but it was always a better product because of his efforts.
You can extract a lot of business logic into those kinds of functions. There's a whole art in writing "unit testable code". Those unit tests have value.
What's left is the pile code and scenarios that need to be tested in other ways. But part of the art is in shrinking down that pile as much as possible.
They weren't there for engineering, they were there for product quality. Their expertise was that they knew what the product was supposed to do and made it did it. Things like "unit tests" help development but they don't make sure the product satisfies client requirements.
If engineering is really on top of it, they learn from QA and QA seems to have nothing to do. But don't let that situation fool you into thinking they are "just creating noise and taking away more value than they provide"
I've seen where this has went poorly as QA was slowly eroded. It became easier and easier to justify shoddy testing practices. Low-probability events don't come around often by their very nature and it can create complacency. I've seen some aerospace applications have some close calls related to shortcomings in QA integration; in those cases, luck saved the day, not good development practices.
QC are the processes that ensure a quality product: things like tests, monitoring, metrology, audit trails, etc. No one person or team is responsible for these, rather they are processes that exist throughout.
QA is a role that ensures these and other quality-related processes are in place and operating correctly. An independent, top level view if possible. They may do this through testing, record reviews, regular inspections and audits, document and procedure reviews, analyzing metrics.
Yes, they will probably test here and there to make sure everything is in order, but this should be higher level - testing against specifications, acceptability and regulatory, perhaps some exploratory testing, etc.
Critically they should not be the QC process itself: rather they should be making sure the QC process is doing its job. QA's value is not in catching that one rare bug (though they might), but in long term quality, stability, and consistency.
If your company hires the low end of that scale, any approach is going to have problems because your company has management problems. It’s very easy to take a lesson like “QA is an outdated concept” because that’s often easier than acknowledging the broken social system.
> Focus: There is real value in having people at your company whose focus is on the quality of your end product. Quality might be “everybody’s job”…but it should also be “somebody’s job”. Yes indeed, naturally every person have just one focus, having dedicated person focus on QA is important.
Another practice, or buzz word (or used to be buzz word:) ), Exploratory Testing, which can pretty much be conducted only by dedicated QA.
When I was a QA lead I often ran into software engineers that couldn’t be bothered to read a pipeline error message (and would complain daily in Slack) and when it came to optimizing the pipeline they would ignore base problems and pretend the issues that stemmed from the base problems were magical and not understood. Wasting days guessing at a solution.
The disrespect a QA engineer sees is not exaggerated in this article. Since most companies with QA orgs do not have a rigorous interviewing process like the Engineering orgs, the QA engineers are seen as lesser. The only SWE that have respect for them that I’ve met are the people who worked in QA themselves. The disrespect is so rampant that I myself have switched back to the Engineering org (I tried using seniority as a principal engineer and even shifted as a manager to make changes, but this failed because Engineering could not see past their own hubris and leadership peoples will not help you). My previous company before I was laid off hired a new CTO who claimed we could just automate away QA needs but had no examples of what she was talking about. This is the level of respect poured down from the top about building good software.
> This created a self-reinforcing spiral, in which anyone “good enough at coding” or fed up with being treated poorly would leave QA. Similarly, others would assume anyone in QA wasn’t “good enough” to exit the discipline. No one recommends the field to new grads. Eventually, the whole thing seemed like it wasn’t worth it any more. Our divorce with QA was a cold one — companies just said “we’re no longer going to have that function, figure it out.”
I've worked with a handful of talented career software QA people in the past. The sanity they can bring to a complex system is amazing, but it seems like a shrinking niche. The writing was on the wall for me that I needed and wanted to get into fulltime dev as QA got increasingly squeezed, mistreated, and misunderstood. At so many companies QA roles went into a death spiral and haven't recovered.
Now, as the author points out, a lot of younger engineers have never worked with a QA team in their career. Or maybe have worked with a crappy QA team because its been devalued so much. So many people have never seen good QA that no one advocates for it.
This is why. Engineers have some of the most inflated egos that they set an extremely high bar for being “part of the club”. Sometimes that’s corporate policy (hire better than you) and sometimes it’s just toxicity (I am better than you). Without realizing that the most valuable skills they could learn are soft skills. I’m open to finding anyone willing to code. Whether it’s from QA, sales, Gerry’s nephew, recent CS grad, Designer turned coder, or that business analyst that taught themselves Python/Pandas.
A good QA team is sorely missed. A bad QA team turns the whole notion of QA teams sour. Just the same for development teams :D
I think devs are first line of defense. Unit tests etc. QA is second line (should we release?), feature testing, regression, UX continuity, etc. There’s value in it if you can afford it.
Long story short, it wasn't. It was like taking away a crutch. Of course we could have been more diligent about testing before having QA validate it, but it slowed development down so much trying to learn all the things we never thought to test that QA did automatically.
https://blog.southparkcommons.com/move-fast-or-die/
A bit recent to have affected Yahoo - but it sells a good story.
Sounds like hell: running as close to the edge of the cliff as you can. Presumably totally ignoring thousands of papercuts of slightly broken functionality. Optimising to produce an infinite number of shallow bugs.I bet the people responsible for Facebook Ads Manager are a lot less enthusiastic about "move fast and break things", although I'd be interested to hear an opposing viewpoint from anyone here who's worked for that group.
QA is almost always seen as a 'cost center' by the business and upper management. I have a hypothesis that you never ought to work in a department that is seen as a 'cost center'. The bonuses, the recognition, and the respect always goes to the money makers. The cost center is the first place to get more work with less hands, get blamed for failures, and ultimately fired when the business needs to slim up. I think the same thing applies to IT.
This spiral is why QA will always be a harder career than just taking similar skills and being a developer. It self reinforces that the best people get fed up and switch out as soon as they can.
Well everything involved in making a product is seen as a cost, that includes the entire development team - QA, Developers, Devops, PM ....
Accessibility, observability, good logging, testing infrastructure improvements, CI/CD tweaks, stability, better linting and analyzer issues are all important, but you will be rewarded if you ship features fast.
This year I spent too much time on the former because I felt like that's what the team and app needed, because nobody on the team priorized these issues, and I'll be sweating at the end of the year performance reviews.
Now knowing this, I understand why the others didn't want to work on these items, so next year, I'll be wiser, and I'll focus on shipping features that get me the most visibility.
Sorry for the bugs in the app, but I need a job to pay my mortgage.
Thinking this can be reduced to a single metric is the blight of modern software (and business in general, I think). Mapping an individual change to improved delivery speed is in the vast majority of cases an impossible task and any decent developer knows this. It's management+ that wants simple easy metrics since they lack the deep understanding required to do their job well. Software development is - despite management's hopes - not line work. It's much more akin to R&D. The line work gets eaten up by AWS.
And it’s not that managers don’t understand developers and the work they do. It’s that a lot of developers don’t engage at all with what a business actually does. They are working at IKEA while trying to convince management to use the nice dovetail joints instead of that garbage dowel based assembly. Not only do dovetails look better, they are substantially stronger! All very true statements from a craftsman woodworker. But a complete failure to understand the business and how and why it operates as it does and the value they are expected to provide within it.
https://en.wikipedia.org/wiki/McNamara_fallacy
1) Some organizations have come to really value what QA/QC brings to the table. From my experience, this seems to be more visible in manufacturing than software. I speculate this is because software is more abstract by its very nature and waste is harder to track.
2) The really good QAs are those who really believe in its mission, rather than those who are looking for the path of least resistance.
Both of those underscore the value lies in organizations and individuals who really buy-in to the QA ethos. There are lots of examples of both who are simply going through the motions.
That's why I don't work in an IT department of a traditional business.
I'm fine with bonuses, etc going to other groups. I'm paid well as a tech worker, and many of those jobs would make absolutely miserable -- assuming I turned out to actually be good at them.
I don’t care though. I enjoy making things better and more robust. It makes my soul feel better. I’ll leave fucking things up to the cynics.
I experienced both worlds: I worked in an organization where 4 QA engineers tested each release that was built by 6 software engineers. Now I'm in a situation where 0 QA engineers test the work of 8 software engineers. In the second case the software engineers actually do all the testing, but not that systematically because it's not their job to optimize the testing process.
Having someone with the capabilities of a software engineer who's daily work is uncovering defects and verifying functionality is important. Paying someone who owns the testing process is more than justified commercially. The problem is: You don't find those people. For various reasons. Therefor you are stuck with making the software engineers do the testing.
But there is hope. There is a new standard released for my industry that requires organizations to have a QA department that is independent of the software engineering department. If they don't have that, they are not allowed to role out there software product complaint to the standard. Maybe this will help to reintroduce the QA Engineer as an important and prestigious role.
This often creates a situation where people need to "justify" their jobs. Usually this happens due to an over reliance upon metrics (see Goodhart's Law) rather than understanding what the metrics are proxying and what the actual purpose of the job is. A bad QA team is one who is overly nitpicky, looking to ensure they have something to say. A good QA team simultaneous checks for quality as well as trains employees to produce higher quality.
I do feel like there is a lack of training going on in the workforce. We had the "95%-ile isn't that good"[0] post on the front page not long ago and literally it is saying "It's easy to get to the top 5% of performers in any field because most performers don't actively train or have active feedback." It's like the difference between being on an amateur sports team vs a professional. Shouldn't businesses be operating like the latter? Constantly training? Should make hiring be viewed differently too, as in "can we turn this person into a top performer" rather than "are they already" because the latter isn't as meaningful as it appears when your environment is vastly different than the one where success was demonstrated.
[0] https://news.ycombinator.com/item?id=38560345
Sounds like the right incentive structure. If you don't mind, how are you judged? Do you feel like the system you're in is creating the appropriate incentives and actually being effective? Certainly this example is but I'd like to know more details from an expert so I can update my understanding.
There's a rampant cultural mind-virus that argues that 95%th percentile is somehow tons of work (rather than a lack of unforced mistakes), so everyone just writes it off. It's on full display at this very site. Just look on any post involving software quality, and read a bunch of comments suggesting widespread apathy from engineers.
Obviously every situation is different, but people seem to be pretty okay with relinquishing agency on these things and just going along with whatever local maxima their org operates in. It's not totally their fault, but they're not blameless either.
> Obviously every situation is different, but people seem to be pretty okay with relinquishing agency on these things and just going along with whatever local maxima their org operates in. It's not totally their fault, but they're not blameless either.
I agree here, to the letter. I don't blame low level employees for maximizing their local optima. But there's two main areas (among many) that just baffles me. The first is when this is top down. When a CEO and board are hyper focused on benchmarks rather than the evaluation. Being unable to distinguish the two (benchmarks and metrics are guides, not answers). The other is when you have highly trained and educated people actively ignoring this situation. To have an over-reliance on metrics and refusing to acknowledge that metrics are proxies and considering the nuances that they were explicitly trained to look for and is what meaningfully distinguishes them from less experienced people. I've been trying to coin the term Goodhart's Hell to describe this more general phenomena because I think it is a fairly apt and concise description. The general phenomena seems prolific, but I agree that the blame has higher weight to those issuing orders. Just like a soldier is not blameless for their participation in a war crime but the ones issuing the orders are going to receive higher critique due to the imbalance of power/knowledge.
Ironically I think we need to embrace the chaos a bit more. But that is rather in recognizing that ambiguity and uncertainty is inescapable rather than abandonment of any form of metric all together. I think modern society has gotten so good at measuring that we often forget that our tools are imprecise whereas previously the imprecision was so apparent that it was difficult to ignore. One could call this laziness but considering its systematic I'm not sure that's the right word.
McNamara fallacy or quantitative fallacy.[1]
[1] https://en.wikipedia.org/wiki/McNamara_fallacy
On DevOps teams I see this constantly. Usually the best-compensated or most senior "Ops" guy or whatever they're called at the company spends a lot of his time extinguishing fires that were either entirely of his own creation/incompetence, which makes it look like he's "doing something." You automate away the majority of the toil there and this person doesn't have a job, yet this pattern is so insanely common. There's little incentive to do it right when doing it right means management thinks you sit there all day and do nothing.
Because visibility is a key! Key to everything in the corporate life. If your work is not visible for managers - you are doing nothing.
> “We have no idea what you did this year and no one has any idea of the value you’re providing, but what the hell. Here’s a huge bonus!”
That doesn’t seem a little insane to you?
Placing value based on the work assigned? Obviously insanely easier said than done. But I think we should embrace the fuzziness a bit. Your manager should have a very good idea of this. If they don't, then they're the ones that should be let go because this is a significant aspect of their job. (may be learned through indirect methods)
> That doesn’t seem a little insane to you?
Honestly, a lot of business practices and economics sounds insane to me. Similarly a lot of alternative suggestions (especially the latter) since they tend to not address the underlying issues but be bandaids. The ones that don't sound insane are often boring and very reasonable, but I think we've established that that's considered undesirable when you're evaluated by visibility. I think this is probably a significant part of the negative feedback loop.
Question(s):
- What are the metrics?
- How aligned are the metrics with the actual goal?
- What isn't covered by the metrics?
- How is what's not covered by metrics evaluated?
- Can what's not currently covered by metrics theoretically be covered by some potentially unknown metric? (best guess)
Definitely agree. In a sister comment[0] I mention a concept I've been calling "Goodhart's Hell" for a more general term. But I think what you are specifically mentioning is sometimes laughably called "Loud Laboring"[1]. I think this all falls under the broader umbrella of metric hacking and thus Goodhart's Law.
Idk why, but it really does seem like metric hacking is extremely pervasive in our modern society, and can be found nearly everywhere. What upsets me the most is that it too is found in the sciences. There also appears to be a strong correlation between the popularity (or hype) of a field and metric hacking.
[0] https://news.ycombinator.com/item?id=38647582
[1] https://news.ycombinator.com/item?id=37147707 || https://www.cnbc.com/2023/08/09/forget-quiet-quitting-loud-l...
There are a lot of these types that gravitate to crisis management roles like DevOps.
"But then they'll leave for somewhere else for more money." </sarcasm>
Literally every company I have worked for. Meaningful training was always an uphill battle.
However, good training also requires someone good and broad in your technical ladder. They may not be the most up-to-date, but they need to be able to sniff out bullshit and call it out.
FAANG is no exception, either.
Basically, it was a way to get a developer to do more testing during development before handing over a feature to QA. This sets up the imagined possibility of firing all QA staff and having developers write perfect code that never needs to be tested thoroughly. Looks great on paper...
At a previous company, they started firing all of the manual QA devs and replacing them with offshore QA people who could do automated testing development (Cypress tests). The only problem was that those fired QA team members had significant business domain knowledge that never was transferred or written down. The result was a dumpster fire every week, with the usual offshore nightmares and panicked responses.
Make no mistake about this, it's just a cost-cutting measure with an impact that is rarely felt immediately by management. I've worked with stellar manual QA people and our effort resulted in bulletproof software that handled edge cases, legacy customers, API versions, etc. Without these people, your software eventually becomes IT-IS-WHAT-IT-IS and your position is always threatened due to the lack of quality controls.
QA is still extremely valuable in any software that has long deployment lead times. Mobile apps, On-Prem solutions, anything that cannot be deployed or rolled back within minutes can benefit from a dedicated QA team that can manage the risk appropriately.
> QA has always been about risk management.
100%.
QA should be related to identifying risk, likelihood of failure, impact of failure to user, client and company. The earlier this is done in the varying processes, the better. ("shift left" but I've seen a ton of differences with how people describe this, but generally QA should start getting involved in the "design phase")
Another example from my own first-hand experience:
A company I worked for made a product that plugged into machines that were manufacturing parts, and based on your parameters it would tell you whether or not the part was "good" or "bad".
When interviewing the leadership of the company, as well as the majority of the engineering group, "what is the biggest risk with this product" they all said "if the product locks up!". Upon further discussion, I pulled out a much larger, insidious risk; "what if our product tells the client that the part is 'good' when it is not?"
In this example, the part could be involved in a medical device that keeps someone alive.
You're not going to be able to roll that back.
it's not one or the other. in my experience it's a decision of "no images" vs "ai images".
in this case, probably "no images" would've been better for the reading experience. but there was never any illustrator getting paid
Every time I looked at one of those AI images my brain just kept seeing all the little weird parts that didn't make sense. Like a brain itch.
This is the bitter truth, no one wants to acknowledge.
DBAs and Infra, are in the same boat as QAs. Pendulam will swing back in not so long time frame i hope.
All it takes is one person on my team to defect and support an untenable amount of tech debt, and everyone on my team has to pay for it.
Ultimately it’s up to the customers. Will they walk because of bugs and outages or stay because of shiny new features?
An influx of dumb money like that can shape and distort the market and overwhelm the feedback loops that would otherwise give consumers influence.
Perhaps now rates are up, the equilibrium changes, but I think it’s still easy to overestimate the number of “first movers” in an industry and the power of tacit or unconscious collusion.
I've worked with conscientious engineers. Sometimes they are right but their delivery mechanism is broken. Sometimes they are just in the wrong place. If we're building a POC SASS product, it really doesn't need the quality of a avianoics microcontroller. All these trade-offs come with risk and cost, good engineers need to know the difference.
There's two kinds of tests. Regression testing, that should be automated and written and maintained by devs. New feature or change testing should be done by those that defined them, namely Product people. In the best case it's an iterative and collaborative process, where things can be evaluated in dev/test environments, staging environments, or production for beta flag enabled test users.
One key attribute to both companies is that it was dictated from on high that the QA team had final say whether the release went to production or not.
These days companies think having the developers write automated tests and spend an inordinate amount of time worrying over code coverage is better. I can't count how many products I've seen with 100% code coverage that objectively, quantifiably doesn't work.
I'm not saying automated testing is bad. I'm saying, just as the author does, that doing away with human QA testers is.
Not to mention, at one recent employer, the QE team wrote an enormous amount of code to perform their tests - It was more LOC than the modules being tested/certified.
... Then I left and my CIO let go the onshore QA team in favor of near term cost savings. Code quality went way down and within a year or two several apps needed to be entirely rewritten. Everything slowed down and people started pointing fingers, and before you knew it, it was time for "cloud native rearchitecting/reengineering" which required an SI to come in with "specialists".
And not even superior wheels, lower quality, more fragile wheels by clumsier wheel designers.
So much code. I hope they had a QAQA team to test all that.
At the second company we integrated Agile so we released every two or weeks (I can't recall which). The first time I heard about CI/CD was at this company from co-workers who came back from a conference all excited about it.
Unfortunately people in the industry who have actual power in planning budgets don't think so. An article is right. QA engineers now are viewed as janitors: no one respects then, better to outsource to cheap location.
I've seen companies where that's true and it was still trash because the QA were mostly low-paid contract workers who only did exactly what they were told and no more.
QA also REFUSED to let developers write automation tests, also REFUSED to let us run them ourselves.
What a nightmare.
YMMV, but just having the final say is not a silver bullet for sure.
That QA was not a clown, they've seen some shit...
Would you let QA to write features in your production code?
If your concern is that devs don't have the right mindset for testing, you can have them collaborate with a QA specialist to define the test cases and review the test implementation.
In practice, devs will not write good tests for their features, QA will be kept away from committing to production code.
Btw, if it is just code - why developers cannot implement features without bugs? No need in QA in such ideal world.
QA is a safety net, but they are not the first line of defense.
Devs that don't test their own code are usually wasting the QA team's time with garbage. It also tends to cultivate (or is a symptom of) an environment where groups are throwing projects over the wall, so to speak, without tightly integrating QA into the process. This wastes significant amounts of time.
It's why Google has SRE instead of devops people largely
Also, the SREs should be knowledgable enough to work on the fix, not just block.
have at your claims good sir.
The same people have been asking: why should I write tests if I can write new features?
And then one senior QA comes and destroys everything.
Once I found that if I press f5 50 times during a minute then backend will go in outOfMemory while spinning requests to the database.
Testing should start before first developer wrote first line of code for the project. Architecture blueprints, set of requirements should be tested as early as possible. But that is not happening in real life, only in books. In real life pm will bring cheap contractor from India one month before the target release date.
This can be OK if the code executing without throwing exceptions is itself testing something. If you have a lot of assertions written directly into the code, as pre- or post-conditions for instance. But I'm guessing that wasn't the case here.
But you could actually make this same argument about unit tests with assertions: do you have any test that those assertions are written correctly? Do you unit test your unit tests? Pre- and post-conditions living "in situ" are the equivalent thing to "control" and test assertions (respectively) living in your unit tests. Would you be more comfortable if the in-situ checks were simply cut/pasted into the unit test? I wouldn't be!
That’s because code coverage doesn’t find the bugs that result from code you didn’t write, but should have. Code coverage is but one measure, and to treat it as the measure is folly.
(But, yes, I have heard a test manager at a large software company we’ve all heard of declare that test team was done because 100% coverage.)
If you have something like this:
There are 8 possible paths for the code to follow here, but you can cover 100% of lines of code in one test. If your code coverage measure tells you that testing the case where conditions 1, 2, and 3 are all true achieves 100% coverage, your measure is worthless. Covering this code cannot be done in less than eight tests.But in reality, if those branches have any interaction, you would need to write eight test cases for every combination of branches being run and not being run.
Take this simplified code:
You need one test for code coverage and two for branch coverage, but if there's a bug in the code as written you'll find that you actually need more. Assume that we want to run do_something() when, and only when, condition2 is true.Now we have four cases:
- condition1 && condition2: do_something() runs, which is correct.
- condition1 && !condition2: do_something() runs, which is incorrect.
- !condition1 && !condition2: do_something() does not run, which is correct.
- !condition1 && condition2: do_something() does not run, which is incorrect.
If you happened to write tests for the first and third of those options, your tests will make it look like your code works, and your (official) coverage metrics will look comprehensive. But because your coverage metrics (in reality) are actually terrible, your code is secretly not working.
The big problem we've just run into is that the number of test cases we need depends on the number of bugs in our code. If the goal of writing tests is to prove that we don't have bugs, this is a terrible result; it means our goal is fundamentally impossible to achieve. We don't know whether we have bugs, so we don't know how many tests we need.
Your point is a good one, but not where I was going.
- Have QA run pessimal versions of real use cases. Trying to sell a word processor to lawyers? Format the entire US legal code and a bajillion contracts in it, then duplicate it 10x and start filing usability/performance bugs.
- Have the engineers test everything with randomly generated workloads before committing. Run those tests nightly, and fix all the crashes / failures.
- Have Product Management (remember them?) work with marketing and sales to figure out what absolutely has to ship, and when.
Make sure it only takes one of the above three groups to stop ship, and also to stop non-essential development tasks.
An uncountable number of products have died or devolved because "we don't have time to do it that way, put in the quick fix"