I think product owners should do (manual) QA. On their own, and when necessary for scalability reasons also via a small team under their direct control.
+1 - I think they should be able to for sure - with current tooling, this is basically impossible without manually doing it; which I've seen a lot of product folks do!
Yeah, this is the only thing that practically worked, in a complex product I used to work on.
- If you ask the devs to do QA, you'll get no bugs other than the ones they already caught during testing and deployment.
- If you have a mostly independent QA team, they will find somewhat silly/trivial bugs like the login page not working in an extreme edge case scenario.
- However, when you ask your Product team to own QA, you get the real good stuff - "Why does this feature not actually work well with this other feature when you combine the configuration" etc. It's great!
When I was a PM, I was lucky to have a big, talented QA team, but I still knew I'd have to do a "smoke test" myself after every major feature release. I cared the most, and I knew the most about the intricacies of the product.
Also: Bliss is when you as a product owner get to such a place that you trust the QA team that you've worked with so closely so much that you only have have to some basic tests a few hours after the release.
On independent QA teams finding trivial bugs, I think this is a social problem. Specifically an alignment one. If a bug goes out, whose fault is it? Eng for making it or QA for not finding it? The answers are different at different orgs, and they have more to do with power dynamics than anything else.
QA is pretty easy to fake (for a while). The last thing you want to do is quality check your QA team. So the further the incentives of the QA team are from the success of the product, the worse things get. It's a spectrum from the other cofounder doing QA to someone in another country charging by the hour.
There are opposing forces to this. It's inherently difficult for people to check themselves. Also, QA is its own skill. It's one more thing to ask devs to be great at. Maybe it's worth having some specific experts around. It's kind of a specific mindset that not all devs have.
This mindset issue is where I'm not entirely sold on what looks like a new strategy from Rainforest QA (where I used to work) of strongly targeting product teams. I haven't seen any QA tool so good that it removes the complete need for skill like indoor heating means you don't need to know anything about how to tend a fire. The best ones are more like how a modern gas range helps a chef. So I question how great the results will be if you have a PM or CSM doing it. Still, I wish them the best of luck.
We had a very large QA silo in a previous company. ~150 people all working under a former QA (non-developer) person turned into a "QA VP". It was a disaster in so many ways. Empire-building tendencies, large numbers of bad hires, etc.
The solution was to break up this silo and getting these people into the product teams where more technical people could handle management and hiring.
Thanks for the luck Travis! Def something we've been edging around for a while. From what we've been seeing, a lot of PMs and no/low code folks already do this kind of thing manually, and don't have tooling for it. RF automation is now way better / easier to use than when you were with us (imho, of course).
> However, when you ask your Product team to own QA, you get the real good stuff - "Why does this feature not actually work well with this other feature when you combine the configuration" etc. It's great!
That would require a skilled product management that is on equal footing with sales and has a veto right before stuff gets sold to customers. All too often however, the only thing that matters is customer wishes for new features, with no one taking care that the tugs the customers are pulling on don't rip the product apart.
Having previously spent a decade in that role for a $1B company: I never got a veto right before stuff got sold to customers. I really, really wanted that in the beginning.
In the end what I ended up doing was spending a lot of time packaging the product as well as educating the sales force. If you don't have well-packaged products, the sales force will sell anything, even if it doesn't exist. And they'll get backed up by a typically slightly disconnected exec team, because they also want revenue really, really badly.
This strategy worked surprisingly well. I think both the sales team and the ever slightly disconnected exec team benefited from the artificial structure that I spent so much effort making up. I guess people instinctively just like following clear structures over chaos.
(This was 5-10 years ago, and not in Silicon Valley.)
>However, when you ask your Product team to own QA, you get the real good stuff - "Why does this feature not actually work well with this other feature when you combine the configuration" etc. It's great!
This works right up until you hit a complexity point product can't handle, or worse, you find out that product is building something completely different from the requirements SME's are giving.
Your QA group (or misguided product group) can do jack squat about badly translated reqs or the right questions not being asked, which in the presence of the best devs and QA's degenerates into building the wrong thing perfectly, which has to get redone over and over again. A good QA that's been around the block is usually pretty good at sussing out major communication defects, but it can be tough to pull out of.
It's a surreal class of miscommunication to behold, and as close a death spiral as you can end up in, because everybody day in day out is working hard and getting frustrated but no one's getting closer to the end goal.
> - If you have a mostly independent QA team, they will find somewhat silly/trivial bugs like the login page not working in an extreme edge case scenario.
Fair but you have to keep in mind that part of QA's job is to explore the silly and extreme edge cases too. One of the nastiest bugs I ever found started as a joke of a test. What happens if you enter a massive long text string (I think it was something like 600 kilobytes) into the password field and enter it?
I expected the UI to barf and a bit of mischief. What I got was the entire server backend crashing and loosing every user session with it when it restarted. The joke test revealed a uncovered a serious denial of service attack vector that was trivial to execute, trivial to automate, difficult to detect, and incredibly effective.
I agree it would be silly if QA insisted that trivial or edge cases be fixed; every bug has to be judged on the cost of fixing vs the risk of it happening in the field. But finding those silly edge case bugs is still very well worth the time to explore. If not because chasing the edge can reveal serious problems, but also because it also helps define where the limits of sanity really are instead of were it is imagined to be.
I've been on teams where the "siloed" QA model seemed to work pretty well -- we seemed to find a decent balance between test coverage and frequency of releases.
But this was at a cash-rich startup that had lots of money to spend on making sure we had plenty of QA headcount. That seems to be the exception, rather than the rule. Lots of startups I talk to are quite constrained in terms of dedicated QA, so the argument for empowering product managers to own quality does make some sense to me.
This; especially now more apps are being done with no-code tooling (bubble, anyone?) - not a lot of existing tooling work with it, and code-based testing isn't viable even if it did.
The last place I worked would pull devs randomly to do QA tasks when QA got behind. Of course, since we weren't trained in the QA processes, and didn't do it all the time, it often wound up taking longer to do the testing, plus the devs would get behind in their tasks.
It's also common practice; usually caused by testing getting under-resourced, bad tooling, or longer shipping cycles (more stuff to test, more pressure to get it out).
It's hard to get them to provide details to the engineers. I haven't seen many who write user stories. Asking them to build no-code automated test cases, that's ... ambitious.
The complexity of our application necessitates that our business owners do a lot of the final integration testing.
It also requires that our product owners handle a large part of the implementation & QA efforts.
For our application, a single piece of code might be reused 40 different ways across 100 different customers based on configuration-time items.
To ask a developer to figure out why something is broken would be a fool's errand for us. Only with the aid of trace files & configuration references are we able to perform RCA and clean up an issue from live usage. For us, 99% of the time it is a configuration thing (either on our end or some 3rd party system). If the code works for 99 clients and 39 other configurations, it might be your client+config that is wrong, not the code.
Essentially yes. We are trying to move to a hybrid model where we can send excel workbooks to the customer for completion, and then directly import these into the system. This would get us ~80% of the way there.
One huge upside with configuration is that it can be copied really easily. If you have a model customer that is very similar to others, you can use it as a starting point and save 99% of the work effort.
Makes sense. Did y'all try to automate any of it? I've seen situations where things are highly-configurable, and massive at the same time - mostly in the medical industry. Test plans are hard due to interactions between config being highly-complex. No sense testing all of the setups, as not all are used or would even make sense.
I think QA-specific people that are specifically not SDETs create bad habits on teams. I think asking developers and product owners/managers to "own" QA fixes the "just throw it over the wall and hope for the best" problem.
But then you're settling for low quality testing. There's an excellent methodology a good test team uses on devs that throw it over the wall. They throw it back with the first bug report.
That said, it's sometimes the right thing for the team when something gets thrown over raw. A good test team should be fine with that and play their part.
> But then you're settling for low quality testing.
Not necessarily though. I would also argue that manual testing is just an insurmountable timesink. You'll never have enough time because manual testing balloons to the time allotted.
> There's an excellent methodology a good test team uses on devs that throw it over the wall. They throw it back with the first bug report.
Sure, but continuing a cycle of "not my problem" helps no one and wastes time. Removing QA as a team/role/specialization, and instead making it a step in the process a dev goes through to ship software, you're fixing the broken feedback cycle QA teams create.
I've yet to see the automated test suite that replaces a skilled, sapient, human, functional tester. The automation takes away the drudgery of repeating tests, but it takes a skilled human to figure out what risks are in the code and figure out coverage to determine whether those risks are realized. If you have developers write good unit and integration tests, and build their work to their local environment to make sure it basically works, you avoid most of the "throw it over the wall" mentality. You also deliver way fewer needles in the haystack, if you will. Testers are free to look for more impactful bugs manually, and then to automate coverage as appropriate.
Totally agree that automation will never fully replace the value of human-powered testing. (Though it is great for the rote regression-testing stuff. The "drudgery", as you put it.)
Isn't the problem with relying too much on unit and integration tests that they don't consider the end-to-end app experience? (Which, in my mind, is what ultimately matters when customers think of "quality".)
IMHO, yep - it's a balance, but the great thing is they can be quick to run, and run locally-easily; which is great for developers to get fast-feedback. Unit testing is unlikely to catch all end-user level issues though; traditional automation too, which is why human-testing is still valuable today.
I think you're pretty on point here; today human nuance is needed to decide WHAT to test as well as HOW MUCH to test. It's also useful for executing some types of tests too (which is why we support both automation, and human testing). IMHO Unit and integration tests should be a given. Human testers should be used with the highest leverage where possible. Being a functional-tester though, today is partly limited by what tooling is accessible to you - which we want to fix.
I've seen them used to do a decent smoke test to allow the QA people to make forward progress on other testing. I worked at a place that had a large number of computers running some testing software (robo something?) that ran through all the mainline scenarios. They constantly added new scenarios so the QA folks were working on much deeper flows until those too became automated.
Company from the article is not claiming that but all the business people are talking about it.
They would like that "test-automation" would mean instead of 2 QA engineers now you need 1 doing 2x more.
Sad reality is that with "test-automation" you actually still need those 2 QA engineers and maybe part time dev to help them out. The upside would be that those QA people would spend less time repeating clicks and improve quality of checks.
Business thinks that you can do test automation in a vacuum without any domain knowledge and without knowing the system.
You always need people to understand the system, to understand the automation and to keep domain knowledge in house.
If your whole QA team leaves and you have automation (or even documentation) - you are still in deep shit because hiring new people and training them on the system will still be needed and only then they can learn to run the automation. They still need to spend time reading documentation and understanding it, because that something is written down or documented does not mean other person has the same understanding of it and the same reference framework to use that documentation.
Well you can hire some new people and let them run automation built by previous team but ... good luck with that.
Writing code is generally easy. Figuring out what code to write is hard. I can spend 80+% of my day thinking about an overall system, what things are important, and how to design things so that they both work and don't prevent change later. Then sitting down and writing the code to do that is (usually) the easy part.
Writing automated tests is generally easy. Figuring out what tests to write is hard. Does the code break on normal cases? Does it break on edge cases? Does it handle all the known use cases? Does the implementation actually achieve what the client is asking for? Does what the client asked for actually make sense?
You can't just automate away testing. Automated testing (unit, functional, system, etc) is a power multiplier for both dev and QA... not a total solution.
We've suffered from the same problems as highlighted here. What helped us was a low code solution, https://smashtest.io, which is basically an English wrapper over Selenium. The developers spend less time on tests, and the QAs aren't an afterthought.
In a model where developers adhere to a devops philosophy and are product owners, there's no need for a split. Developers should not be measured only quantity of releases, but also on metrics related to availability, customer impact, latency, operational costs, etc.
I'm not opposed to a model where non-technical roles are empowered to define test cases and contribute to approval workflows of a release pipeline, but that doesn't absolve developers of the primary responsibility of being end-to-end owners.
I know "devops" is an overloaded term that means different things to different people. To me, it's when developers are involved in product research, requirements gathering, user studies, design, architecture, implementation, testing, release management, and ongoing operations. They're supported by a dedicated product manager who leads the research/requirements gathering/user studies initiatives.
This is how I've worked for most of the last decade. Empowered devs that are accountable for their releases and supported by metrics related to customer experience want good quality releases. Ownership works.
Agree but a lot of companies, particularly larger ones, don't have this sort of culture. They are filled with people whos sense of worth(at the company at least) is defined by a narrow job title and set of skills. Their worldview requires them to categorize everyone else on a narrow set of attributes as well:
"Oh, you're a programmer so you must not be good at soft-skills and communicating with the business."
"Ah, you have soft-skills and you also program? You must not actually be very good at programming so you should stop that and just become a PM."
"Ah, they are PMs/Sales/Whatever so SQL is too hard for them and they can't be expected to learn git or markdown; that's for nerds."
This leads to cringy, insulting sentiments like:
> The most valuable thing you can be doing is writing code
This feels a lot like a reincarnation of cucumber/gherkin, except they've replaced business-facing text DSL with a no-code visual UI. The intention is the same - to have the customer own the tests.
This looks like it has a shallower learning curve to get started, but I would imagine that after a certain point it winds up being less productive to use the UI than to write code and this is ultimately why visual coding hasn't taken off. At some point someone will need to become a power user and be their organization's resident expert on Rainforest, but at that point they're spending all of their time in Rainforest and they're no longer a product owner embedded in the business.
At the end of the day the business owner should be writing specifications, the engineers should be using them to create tests and they each should be collaborating closely on both.
I've had a couple goes at teams trying to roll out cucumber tests, and I still don't understand quite what the point is.
Nobody but developers could actually manage to write any tests, and it was harder than just using the normal tools, plus maintaining all the glue besides.
The gherkin language that cucumber uses for its specifications, written well, are brilliant at distilling what the customer wants, what the developer is going to build and how the QA is going to assert that acceptance is measured. But it has to be written collaboratively by all of those together and by the end of it you get a shared understanding. This is the most important part.
The glue code behind it is a different fish, and needs someone with software engineering trainging to build it and love it.
If you want something it's closer to, it's sikuli script. Visually looking at the page (or whatever, tbh), then manipulating it using the keyboard and mouse. It's basically done using a KVM, so much closer to what a user would be able to see and do than something like cucumber or gherkin.
However, we also allow you to test using a crowd of humans, should folks need more nuanced feedback about things, or have much more complex things to ask.
Disagree on who should be writing tests; I think that's the case today as tooling doesn't support anyone but engineers (QA or not) automation things, or manual tests.
Engineers should absolutely be writing unit tests and integration tests for their APIs. Personally I find that it brings a lot more integrity and a sense of ownership into the process when engineers are required to deliver tests.
I disagree with the problem's that you mention in this article:
Developers aren’t incentivized to prioritize QA testing
Developers are typically evaluated based on the quantity of software they ship, and how fast they ship it.
That's an organizational problem that is not universal and certainly won't be solved by a QA automation tool.
Developers’ job satisfaction goes down when they’re in charge of QA
Expanding upon one of the previous points:, we’ve seen that most developers just don’t enjoy doing QA.
This might be the case for manual testing but for automated testing the opposite is true. Delivering tests along with code increases integrity, makes debugging significantly easier, and helps clearly communicate the intention of each feature. This only works if there is collaboration between the developers and the product owner.
Clearly you have a viable product that works for many organizations, but it's certainly not a one-size-fits-all solution nor a best practice.
Thanks. I agree re unit-tests and integration tests. Mostly we're focused on (and talking about) testing what humans end up using directly - e.g. interfaces to web apps or mobile apps.
I think you're wrong re:organizational problems, it's part of it - but most developers (in my expereince) do not want to do QA outside of unit-tests and maybe integration tests. They want to write code, and ship things. Automation, at least traditionally is brittle as well as slow to write, and few love it. Whilst tooling like Cypress does improve things over Selenium, still, I've not met a developer that actually enjoys that kind of testing.
The right person for doing QA for code written by developer A is developer B, who is motivated to show that there is breakage in the code written by A. This is just good old peer review, in the sphere of development.
The industry uses dedicated QA people based on the assumption that you can get two for the price of a developer.
In fact, if you have two developers, one of whom is more clever than the other, you want to give the code cranking to the less clever one, and use the clever one to verify the code cranking and make improvement suggestions.
If you have clever people designing the system, and clever people finding problems in the commits, you can just have average coders cranking out the bulk of it. Most code doesn't need cleverness, just persistence and consistency.
My org has mixed teams. We find devs are terrible at being QAs. This is not through lack of willingness but they're always either overtesting or undertesting and usually missing the test boundaries. So we use the SDET model, software developers trained in testing to build out the automation and high quality QAs with lots of domain knowledge to do the static and exploratory testing.
In my experience (hiring manager for QAs), there's no way I can get SDETs for anything like the price of a journeyman dev. I'm probably looking at a 20-40% premium right now for someone decent.
Devs doing QA worked fine for us. You need to have a team that actually cares about the product, and you need figurehead devs - not necessarily seniors or team leads, but charismatic people others will fall into line with - who model the behaviours you want.
The problem with almost anything else is that it increases the number of hand-offs between groups of differently-aligned people on the route to production. If you're aiming at multiple deployments per day, with sub-hour commit-to-prod times, the latency in those hand-offs adds up way too fast.
My main problem with devs doing QA is that QA is huge, knowing the entire product is hard.
In my case, devs were just testing what they changed and never noticed something horribly broken somewhere else.
I think all devs should test that what they built works in production (you don't need QA for that), you need QA to keep everything else tested and working.
> devs were just testing what they changed and never noticed something horribly broken somewhere else.
There's your problem. If they broke it but didn't test what they broke, then they changed it but didn't realise they had. "Horribly broken elsewhere" sounds like the sort of thing that should show up in an automated test, no?
If the problem is "we don't have a good enough automated test suite and our architecture doesn't limit the blast radius of changes" then I can see how throwing QA headcount at it might look tempting.
A person doing manual testing, which is a gate to deployment would certainly be hard to make work for deploys as fast as you're targeting. It may still be valuable to have someone separate to the devs testing your product (in production, not as a gate to release) looking for things your automated testing, customer feedback or metrics may have missed. Whether this would be valuable is really context-dependent.
Why can’t the QA people just live on the dev team? Why do they need to be siloed away or not exist?
I had this in my past job. Have 1 QA per two developers, the QA sits with those two developers, and you are constantly telling them when a build is done and on staging. They write tests, do some manual checks, and then tell you how it is going relatively immediately. They also handle the problem of reproducing bugs and narrowing down exactly where they occur, which is not trivial.
For all the faults in that organization (including letting our test runner be broken for months), we didn’t put out a lot of bugs and we found all manner of strange edge cases very quickly.
> Why can’t the QA people just live on the dev team?
Because there is little incentive for devs to police themselves and there could be multiple dev teams spanning client/server that needs to be integrated and tested.
A slightly better org to own QA would be product team.
Each team should have some QA resources. Server can be tested independent of client, etc. Then the QA staff work together to ensure appropriate systems integration tests.
Siloed QA team runs the risk of becoming a bottleneck as work from disparate teams comes through.
Sure there’s incentives for devs to police themselves. I’d rather catch problems early than have a to deal with a shitstorm from pissed off product owners and VPs when bugs are found in production. But our QA person is on another team. I’d much rather have them on my team.
The key part in that is communication. That’s makes all the difference whether it be with BA’s, QA’s, or the end user. Speeds up the development cycle and greatly reduces the number of bugs.
The best experience I had was on a team that had essentially 2 BA’s and 7 devs. There was constant communication to clarify actual requirements, devs would build automated tests off them, BA’s would test against the requirements and then a business user would do a final look over. All in all features were able to be released usually within a day and there would be days we’d get out 3 or 4 releases. Only in one case did a major bug get released to production and the root cause of that was poorly worded regulations which had a clarifying errata come out at the 11th hour.
For as many faults as that company had that caused me to move on I’ve yet to run across a team that functioned as well as that one did.
Communication is great until someone becomes unreasonable and doesn't want to do something. Trust but the chain-of-command must verify. Shouldn't need to, but it should be there as insurance.
People don’t just randomly become unreasonable halfway through. If they’d be unreasonable, they’d do so from the start. If it happens midway, there’s almost always some reason.
That said, I do I agree that the chain of command should always be aware of what’s going on, or have a reliable way to find out.
Yep. People aren't rational, compliant, cooperative actors 24/7. Agendas. Bad days. Illnesses. Family events. So then it's unreasonable and foolish to extend trust unconditionally, perpetually, and without auditing.
If all people were angels, no government (or whoever regulates) would be necessary.
If all people (and systems) were perfect, no backups would be needed.
QA here, disagree. In this situation QA is focused on the quality of the new features, so they're already excellent when they go out into the current release.
do that too. Tests for new features also include regression testing, as does integration tests and automation CI/CD tests.
In our working agile environments, you know the team's velocity. Managers know the team's velocity. Points per story include QA work. If they want something faster in this sprint, something else has to be taken out. Firm but fair, and it works very well.
I agree in theory and there are certainly teams where this could happen, but what happened on this team was that the bugs were found so quickly than the fixes were also quick (I would always get a bug report the same day and often within the hour), making shipping stuff with few bugs relatively painless.
Was there pressure to keep releasing? Yes. But with the rapid feedback, it never became too onerous to get them done anyway.
That's false. QA can also be concerned about growing new features in the next release. We have embedded QA's on our teams. They QA new features, as well as help to triage current bugs.
Anecdotal too, but I've been QA lead on a team for a 9 figure project in aviation, concerns and complexity very significant, and it was very successful.
I work at a startup so we have a small team. But for us it goes:
1. Develop some feature (me and the other engineer are full stack)
2. Send to other engineer for code review
3. Deploy to a staging site, with test data. Send to product girl to QA
If she finds a bug (or realizes her design doesn’t look as right in practice as she envisioned) she sends back with notes, we implement or fix them, and then start back at 1-2.
I think QA is different for every org but I believe there should be steps. At bigger orgs there should be QA for every feature developed (so each team should have a QA person) and then an integration QA of the current beta.
QA is different for every type of engineering organization.
It shouldn't be automatically thrown around or diffused as something tertiary that can be subordinate wherever it lands. That works in a 3 person startup, but not if you're building an aircraft carrier or an iPhone.
Makes a lot of sense to me. The best and highest quality software I've ever worked on (people could die if it failed) had the devs doing testing and writing automated tests, with an intermingled QA team doing further testing, and often jumping in on dev.
I think the ideal team setup is one you allude to: a cross-functional team with experts from each domain collaborating together. My intent was not to say that can't happen - but that it typically doesn't, because of specialization and organizational politics. The vast majority of software teams have a siloed QA team. Why? That's for another thread :)
We had that in one of my assignments. It kinda works, but at the same time, they were a gatekeeper, a barrier, and usually a release was delayed for a day while they tested out their new testing approach and went through their excel sheets of things to test for.
I’ve experienced both, were the dev team also had to do QA, and a dev team with QA people.
I personally hated writing automated tests, but it did give me a much deeper understanding of the product. Especially when writing tests for features I wasn’t working on.
That said, having a dedicated QA person within the team is far more effective. The dev team can build features, while our QA person works with the business to come up with test cases and test data.
* It eventually becomes too much work for the PM as the product grows or the PM just gets tired of the tediousness of creating tests for all the edge cases
* PM hires someone to help with creating the automation tests
We have a team that helps with testing infrastructure but the tests are created by developers on the functional team. Everything is automated. Seems to work well.
Dev sometimes aren't the best QA because they think like developers and not like end users. The mindset prevents you from doing things on the corners that end users do. Its like your instincts kick and a keep you safe without the railing where an end user might plow ahead thinking their path is ok and fall off the edge.
Devs should do automated unit and functional tests, but after that, get some good QA that do not have the same boss (at least at the first level) as the developers.
This can absolutely be learned, enabled, and encouraged by the right guidance and, if necessary, training at the dev-team level. It's just part of the job.
Can be, yes. Devs aren't always the best group to do that with though.
I've had quite a bit of user training in a product I work on, but unsurprisingly the QA group of people who in their primary roles use it daily and have degrees in the problem domain are way more effective at finding non-obvious problems.
I really don't think any Dev team will be as good at QA as a dedicated QA team. The mindset is very different and the context switch is hard. Checking your own work is also not the greatest of ideas.
I can't find the bit where he talks explicitly about this as I've lost access to the book I read it in (and don't even remember the name). I'll see if I can dig it out.
In the passage I am unable to find, he gets into the issue of having a worker making widgets, and another worker checking the widgets. In our world this would be a developer and a test author. He states that this reduces quality: the maker knows that he can make poor parts when he is in a hurry, because the tester will catch them; the tester knows the worker personally, and trusts that he will make good parts, so doesn't have to be very thorough. Instead, we need the worker making the widget to be responsible for its quality, and further to be empowered to do so.
I think this article misses two important points on siloed qa teams
1. Qa doesn’t know what can be covered by unit or integration tests
2. Since they treat our code like a black box, they may create permutations of tests which which cover the same functionality
Maybe this is part of the draw of having a qa team. Feature coverage rather than code coverage. The downside is this can create a huge number of expensive to run manual tests which may be hitting the same code paths in functionally identical ways.
The tooling for automating manual tests of web apps is almost there: puppeteer, recording user inputs and network calls, replaying everything and diffing screenshots.
Since qa tests are tied to features and not code, There’s also the problem of having to run all qa tests even if you’re releasing minor code changes. My build tools are smart enough to return cached results for unit tests whose dependencies didn’t change, but there’s no equivalent for qa tests.
Yeah, this article is shallow and avoiding the deficiencies inherent to Rainforest's offering. They are defining QA challenges as a nail so they can sell you their hammer.
Asking developers to own QA is broken because developers are naturally biased towards the happy path. If you want to build a bar, you need someone to order -1 beers[1].
Handing off QA to an external team is broken because those people don't have the necessary experience with the product, nor can they quickly and easily engage with development to get to the heart of a problem (and a fix.)
Having QA rely exclusively on automation brings quality down as the application's complexity goes up. Writing tests to cover every possible edge case before the product ships isn't feasible.
The best solution I've seen in the decades I've been in software development has been to have QA and Dev as part of a team. Automation covers as much of the product as possible, and continually grows as issues are identified that can be covered by CI from here on out. QA become essential at the boundary of new features, and require a deep understanding of the product in order to know what knobs to turn that might break the application in ways the developer never thought about. Once those sharp edges are also automated, QA pushes the boundary forward.
I've also arrived at this approach and don't think it's that uncommon - IMO the article is presenting a false dichotomy.
There are still gotchas to look out for in the team-embedded QA approach. In typical team sizes, you often end up with only one QA per team - you need to make sure they have cover (everyone needs a break), and they need support in their discipline (do something to share QA knowledge across teams).
QA also works best for features that the user can meaningfully interact with. The more esoteric, interconnected, or complex the failure is, the more it gets muddled with other reports and weird conditions (it breaks on 3pm when the moon is in equinox and I have my mouse pointed north). That’s an over exaggeration but the sentiment is accurate. QA will frequently lack a deeper understanding of how the software works and is connected. That ignorance can be valuable (finds blind spots) but has trade offs (assumptions are made based on imperfect observation of cause and effect where human biases lead you down dark hallways). Speaking from experience with lots of manual QA which is actually a rarity these days.
The other thing to consider is, if you’re careful, user feedback can be obtained more quickly. If you can keep things stable, then you won’t upset your users too badly and you’ll avoid the need for QA to act as a stand in.
I agree, this all makes sense. Although I think the team-embedded QA is generally the right thing, I wouldn't use it blindly in all cases. Some teams I manage only produce HTTP API's, these are ideal candidates for automated testing (incl. end-to-end integrated tests) and the developers are happy to own this without a QA on the team.
Most projects I've worked on have been games where the studio was split into teams of about 6-10 people with one dedicated in-house QA member for some but not all teams, a few in-house QA workers not assigned to a specific team, plus a much larger external QA team (from the parent company/ publisher). That has worked great.
I've also been on projects with only external QA. It works, but the lack of internal QA has been a frequent annoyance.
Yeah - in my experience 6-10 people per team is typical, often with one of them being dedicated QA. Having this, plus a separate central QA team is one way to address the pitfalls of embedded QA I was pointing out - they can cover for time away for team QA, or act as bench capacity.
Agreed it's presenting a false dichotomy. The article ends with a pitch for Rainforest's "no-code" QA platform so it makes sense they want to present their product as the clear solution. I could take the article more seriously if it at least mentioned a more integrated approach.
Absolutely. Disappointing to see this sort of shallow sales pitch blogspam making it to the frontpage. I'm surprised more HN readers don't see through this.
The entire purpose of this article is to self-servingly attempt to convince the reader that their product is the only solution to QA problems.
It correctly identifies some challenges with QA, but this solution is certainly not the only way to have effective QA. That Rainforest is resorting to such a disingenuous presentation of solutions to QA issues makes me think they probably don't actually solve QA problems very well.
"We have researched what makes QA successful and X, Y, and Z are what we found. Here's how we believe we're solving Z for our customers" would be a much more honest pitch.
I totally see your point here - it's frustrating that how I presented it undermined the impact of the core point, which is that siloed ownership of quality is an anti-pattern, and an anti-pattern which is often a response to the limitations and design principles of quality tooling.
We are to my knowledge the only product built for this kind of cross-functional ownership, that's why I don't recommend other products.
While I agree the post is too marketing-y - to be honest I didn't expect it to appeal so much to the HN audience - it was written in good faith. We built the product because after 7 years of building a product for one of those silos, we became convinced that the only long-term durable solution was to empower everyone to own quality. Clearly I need to figure out how to strike the balance that you outline in your last sentence, which was the goal!
Oh c'mon. Of course a blog on the website of a company is ultimately trying to pitch their product, one way or another. That does not invalidate the points they bring up per se.
It obviously needs a bit of critical thinking when reading it and taking everything with a grain of salt, but that's something I really hope we can expect people on HN to be capable of?
i don’t agree with this at all. i’ve seen qa automation team members work solely as a resourcing issue: that if you have X developers you always have X feature developers and never enough people to automate. it’s not that devs aren’t capabale. i fundamentally disagree with rainforests mission that you need “other people” to achieve success in testing. your top engineers have no problem succeeding here
[author here] You are absolutely right: the optimal setup is a cross-functional team of domain experts collaborating on solving the customer's problem. The article is meant to point out that siloing QA to either just dev or just QA is an anti-pattern that is unstable and will not work, long term, without significant pain.
I think the challenge is that, while we agree on the optimal setup, it's almost never done like this. There's lots of reasons why, but the main one (imo) is organizational scaling. The logic of scaling teams tends towards specialization, especially because developer time is extremely expensive.
There are many ways to address the problem of QA. From our perspective, if they don't broaden the ownership of QA from just developers or just QA engineers (which is where all the current products are targeted), they will exacerbate this specialization problem.
I think that to say developers should not be interest or responsible for quality is most certainly a cop out, there can and should be a level of pride of workmanship that is focused on the customer.
The main challenge with software is the possibility of unexpected regression in areas that are non-obvious when implementing a given feature. One of the more scalable ways of catching these regressions is via some sort of automated testing. I think that a QA support team can provide the tools and support that allow for teams to easily add end to end testing, which is vitally important when build large distributed software.
If the QA team is responsible for testing every feature from many teams there is definite risk of bus factor and finger pointing.
In the super ideal world the testing is specified via something like Cucumber and the orginal author could be the product owner with feedback from the engineer as he works to that specification. But I have never seen this though have heard it can happen, rarely.
Hi author, I found parts of this downright wrong, especially on the Cypress side, which is where I have the most familiarity:
>Compare this to code-based automation solutions like Selenium or Cypress, where a coder would need to manually update the code across all the relevant tests any time the product changes.
... Well, no. Those tools can have reusable chunks and references that work just like your steps and are organized in the same manner, so that things used multiple places are stored once and reused. That's a basic programming good practice. When a change is needed to, say, a login flow, you don't change 100 UI tests, you change the login function that those tests are all calling.
> Tests created in code-based automation solutions like Selenium and Cypress evaluate what’s going on in the code behind the scenes of your product’s UI. That is, these solutions test what the computer sees, not what your users see.
and
> For example, if the HTML ID attribute of the signup button in the example above changes, a Selenium or Cypress coded test that identifies that button by its ID would likely break.
This is all very disingenuous in my book. In these tools the tests interact with the rendered web page and _can_ be made to use IDs, classes, and other brittle stuff as selectors, but that's a naive approach that a professional wouldn't typically use, because the brittleness surfaces early and there are better patterns that avoid it. One of those is to use selectors based on something that itself should be present - eg, click a button based on its label being "Log In". If that test fails because the button label was changed or removed, that's a reasonable failure, because labels are important in the UI. In the case where we are selecting/making assertions about DOM elements that don't have meaningful user-facing labels, dedicated test-id attributes solve the brittleness problem. But by and large, if something is interactive to the user, it should have a label, so the fallback to test attrs for selectors is rare, and used for the case where we want to measure some specific effect took place that is not itself an interactive element.
It's unfair to suggest that these tools inherently create brittle tests.
> Rainforest QA tests don’t break when there are minor, behind-the-scenes code changes that don’t change the UI.
but
> It works by pixel matching and using AI to find and interact with (click, fill, select, etc.) the correct elements.
Look, maybe your tool is GREAT, I haven't tried it. But a functional UI test based on label contents may be more robust than something that breaks when the visual structure of a page changes, which happens itself pretty frequently. Maybe the AI is good enough to know that a bunch of stuff moved around but the actual flow is the same and can find its way through. With tests based on labels and dedicated selectors, devs can make massive structural changes to the DOM and not break any tests if all the workflows are the same.
I would sure love if this post contained more info on how the visual AI system of selecting elements is better than using labels, test attributes etc as is conventional in automated testing done by professionals. Likewise it would be good to compare with an actual competitor like Ghost Inspector, where non-programmers have been able to generate step-based tests like this for years. The main gripe I had with Ghost Inspector was that it creates brittle selectors and so tests need to be changed a lot in response to DOM changes, unless a developer gets in there and manually reviews/picks better selectors.
If what you have a is a tool that _makes more robust tests than Ghost Inspector_ but is _as easy for a PM to use_, then that is interesting.
I actually support this underlying idea completely - I love systems where non-developers can create and maintain tests that reflect what they care about in the UI. I even love it when tools like this create low-quality tests that ar...
We have a free trial - you should try it instead of criticizing theory, I would love to get your feedback on the actual product, here or directly (I'm fred@rainforestqa.com).
Regarding DOM interaction, you're missing the point. All automation that tests the front end code, regardless of how it attaches, is using a path that is different from your end user. The end user interacts with the application visually. That's why we test visually. A decrease in code-based brittleness is just a nice side effect. And as you note, this is a very high-level post outlining one key idea about quality ownership. You may be interested in this, which is one of our front end folks talking about why we believe testing visually is superior: https://www.rainforestqa.com/blog/the-downfall-of-dom-and-th...
We have been selling a QA solution for almost 10 years. In that time we've seen thousands of setups and directly worked with hundreds of teams. Your claim "weaknesses of automated test frameworks [...] are compensated for quite easily and routinely" is, quite simply, not true for the majority of engineering teams - few QA leaders, including proponents of Cypress, would agree with you.
> The end user interacts with the application visually. That's why we test visually.
Not all users interact visually. Selecting interactive elements through accessible labels, not based on visual appearance, is a better practice imo. I want important parts of the DOM that are critical to building a correct accessibility tree to be a part of the test, and to fail the test if we change something that makes the accessibility tree incorrect. Because that is the API for assistive technology and it communicates the user interface for lots of people. "Correct behavior" of an app or website includes "the accessibility tree accurately reflects the structure and nature of the content" and "form fields are labelled correctly". I might be in the minority in thinking that, but I believe it 100%.
Nothing I've seen so far (including the post you linked) suggests that the OCR-like approach can tell us anything about the accessibility tree.
The post does make a similar point to mine:
> If your app changes visually, your test will fail. This is of course a tradeoff and a conscious decision to couple your tests to your UI instead of your code, but the upsides bring much more value than that the downsides take away.
I disagree on the tradeoff. I can run screenshot diffs etc to catch unexpected UI changes, they are a little noisy, but I'm ok with that tradeoff because I care about more than just the visual appearance of the app.
A "visually correct" app with click handlers on divs and no semantic HTML is a liability (legally, maintenance wise, etc).. I'd like the E2E testing tool to assert that the app is working correctly, which does mean some assertions about the DOM are appropriate to me. I agree with the author of the linked post that "we want a solution that is not brittle in unwanted ways." We can be selective about what DOM things are important.
In the linked post the author says "In particular, DOM tests are tightly coupled to the structure of your code." and gives an example about a Selenium test that uses a brittle xpath that depends on a specific DOM structure.
Maybe I have not been exposed to enough of the industry to know that there are thousands of setups relying on flaky xpaths to target elements for testing. To me, it is not true that DOM tests are tightly couple to the structure of your code by default. It's a false statement made for marketing purposes and it is gross.
DOM tests "can be flaky", "are sometimes coupled to DOM structure" or whatever, is a fair assertion, but flakiness in DOM-driven testing is not a fact, it's a sign of badly written DOM-based tests. This is often the first thing I address in a code review of new tests written by somebody who does not write lots of FE test code, and they easily learn how to avoid it.
Maybe I'm wrong but it seems really really basic stuff to not create brittle selectors that fail tests for reasons we don't care about.
I like the OS-level interaction and agree that provides some advantages. I totally disagree that these advantages mean your solution wins at the "best way" to test, but it does clearly cover a different surface area than other automated solutions for E2E testing, and it seems like tests are pretty quick to knock out.
This solution could be a complement to other automated E2E tests, and I would see no reason that a PM or other party couldn't spin up and maintain some tests this way as quick-to-create set of tests to run against various builds, knowing that design changes will break them but that this is OK cause in theory it is quick to rewrite them.
But I couldn't see using this tool as the only E2E testing solution as though it is a superset of what Cypress/Selenium/Whatever tests are capable of. It is actually not a competitor with those tools. It's addresses different concerns with a little bit of overlap.
I'm happy to check out the free trial and see if I'm miss...
I would agree with this. In a past job doing UI work, I had a tester sitting across from me. As soon as I got things working reasonably, I would give her a build to test. It was great, because we found a lot of bugs super early, very shortly after the code was written...
Problem comes from BigCo approach to people. Such corporations would like to have a QA team that should be utilized to a 100%.
So that if they don't have much work on project X you just drop them into project Y.
In reality we all know that is just not possible, because people have to have domain knowledge and keep up with changes. It is not like if you come back to project after 2 months you will be proficient with it as the world changed.
This way if project X has slow times, dedicated QA will have less to do and BigCo has to eat that "loss".
In small startup it is easier to have QA incorporated into dev/product team because as software is constantly changed there is work to be done all the time.
>Asking developers to own QA is broken because developers are naturally biased towards the happy path.
I've been arguing the same thing for a while now. There's a clear mismatch in incentives. If you hand off the QA to developers you might initially get less bug reports but that doesn't mean your software isn't garbage.
Some of the best QA engineers I've worked with were developers in a previous life but they've since spent years cultivating a very different way of thinking about software. Believing you can get your engineers to adopt that mindset overnight is pure fantasy.
For the first ~15 years of my career, multiple teams, the model was always that engineering was responsible for all the test coverage of all areas. It was wonderful. A separate dedicated QA team will never, almost no matter how good they are, have the same deep insight into the code both big picture and small picture. So they won't be able to exercise it via tests as well as the engineers who wrote it. I always thought, this is the only way that makes sense.
To my surprise in subsequent roles, I've seen many development teams, even good ones, who don't have that deep instinct to break their code via tests. Which was very alien to me. I've seen enough cases now to realize it's somewhat common. For such teams, separate QA engineers are a must.
I still feel having quality responsibility in the engineering team ultimately can offer the best quality outcome, but only if you can build a team with the right attitude about testing. If not, having a separate QA team is vital.
> Asking developers to own QA is broken because developers are naturally biased towards the happy path
This overlooks that "no customer calls with product issues that dev needs to get fixed asap on a Saturday evening" is part of the happy path to optimize for. If those things happen, the dev has deviated from the happy path, including the embarrassment in front of fellow devs "but that's really a case we should have tested". This is another of those details the article is lacking.
Code review. Read the results of someone thinking through a process. Spot more than they will, simply by throwing more eyes at it. Actually fairly effective: getting a senior dev to cast even a lazy eye over everything gives more opportunities to discuss Why It's Done This Way and Why We Don't Do That and Why This Framework Sucks And How To Deal With It with specific concrete examples which the other dev is currently thinking about. But it's still easier to write the code yourself than review it, and things still get missed no matter how careful you try to be, so it's still just another layer.
Unit tests. They cover the stuff we think to check and actually encountered in the past (ie. regressions). Great for testing abstractions, not so great for testing features, since the latter typically rely on vast amounts of apparently-unrelated code.
Integration tests. Better for testing features than specific abstractions, and often the simplest ones will dredge up things when you update a library five years later and some subtle behaviour changed. Slow sanity checks fit here.
UI-first automation (inc. Selenium, etc). Code or no-code, it's glitchy as hell for any codebase not originally designed to support it; tends to get thrown out because tests which cry wolf every other day are worse than useless. Careful application to a few basics can smoke-test situations which otherwise pass internal application sanity checks, and systems built from the start to use it can benefit a lot.
Manual testing. Boring, mechanical, but the test plans require less active fiddling/maintenance because a link changed to a button or something. Best for exploratory find-new-edge-cases, but throwing a bunch of students at a huge test plan can sometimes deliver massive value for money/coffee/ramen. Humans can tell us when the instructions are 'slightly off' and carry on regardless, distinguishing the actual important breakage from a trivial 2px layout adjustment or a CSS classname change.
So that's the linear view. Let's go meta, and combine techniques for mutual reinforcement.
Code review benefits from local relevance and is hampered by action at a distance. Write static analysers which enforce relevant semantics sharing a lexical scope, ie. if two things are supposed to happen together ensure that they happen in the same function (at the same level of abstraction). Encourage relevant details to share not just a file diff, but a chunk. Kill dynamic scoping with fire.
Unit and Integration tests can be generated. Given a set of functions or types, ensure that they all fit some specific pattern. This is more powerful than leveraging the type system to enforce that pattern, because when one example needs to diverge you can just add a (commented) exception to the generative test instead of rearchitecting lots of code, ie. you can easily separate sharing behaviour from sharing code. Write tests which cover code not yet written, and force exceptions to a rule to be explicitly listed.
UI testing is rather hard to amplify because you need to reliably control that UI in abstractable ways, and make it easy to combine those operations. I honestly have no idea how to do this in any sane way for any codebase not constructed to enable it. If you're working on greenfield stuff, congratulations; some of us are working on stuff that's been ported forwards decade by decade... Actual practical solutions welcome!
That's my best shot at a 2D (triangular?) view: automated tests can enforce rules which simplify code review, etc. The goal is always to force errors up the page: find them as early as possible as cheaply as possible and as reliably as possible.
The machine can't check complex things without either missing stuff or crying wolf, but it can rigidly enforce simple rules which let humans spot the outliers more e...
Code reviews should be about project structure and abstractions and keeping approach in order or to use team common approach instead of each team member doing whatever, well syntax/code should be linted and formatted automatically nothing for reviewer. Second thing is checking by second pair of eyes if they understand code in question in the same way.
Unit and integration tests should not be generated. Those should be written by people if they find code that they are writing doing complex things like some specific calculation. It is more as a tool for understanding what you are doing and then maybe leave some tests behind for regression. But don't generate BS tests that will only slow down system and people. People have to understand what is going on and be on top of it and never "just run the tests" because tests that are passing green but are actually wrong are really bad.
UI testing should not be abstractable - it should be only augmenting manual UI testing - so tester should be automating his own work after he has done it manually. That tester should also find things that take him long time or have to be done multiple times and are not changing often so he wins time to do more important things. QA person should also be always engaged with the system and automation because that is the only way you can keep domain knowledge.
It depends how you count the unit/integration tests, really.
If you've got a general rule which must apply across an entire system, generate the necessary tests so that they fail granularly and don't require messing around to find the exact case which breaks. IMO that's one test, just applied to a range of cases.
An example might be mappings for Entity Framework (or similar ORMs, etc). Auto-generated migrations simply do not work if you need to limit migration downtime and maintain certain data invariants (which can't be specified in the schema, and yes, those always exist). So you need to write database migrations manually. This introduces risk of desync of entity mappings and schema.
So don't just spot-test roundtripping entities (a nontrivial system will have hundreds and something always gets missed). Instead, write a tool to introspect the DB schema and the Framework's mappings, and check that they match sufficiently closely. Every time someone adds an entity or property or something, it's already covered.
Similar cases exist when dealing with any interface between separate systems, especially when you don't control one of them. If you're regularly mapping between two models, use something like Automapper which can be asked to verify its mappings to check that every property is handled in some way.
(Granted, Automapper doesn't catch everything, but it builds a model that could probably be introspected over to spot encountered bugs and check that other possible examples of those bugs don't exist. Doing so generatively catches future additions of possible cases for free. If you're really paranoid, define some means of marking manually-written tests which cover each case, and test that a test exists for each case.)
Computers are really good at force-multiplication. They should trivially be capable of spotting other instances of known categories of bug. This is not hard to do and doesn't require wooly nebulous machine-learning shit: we've had introspectable ASTs since the dawn of compilers.
I agree with not asking the developers of the feature. I don't think it should be part of product. QA is a deeply technical job. At GitLab QA gets own department called Quality. They are on the same level as development, design, development, and infrastructure. The people in it are mostly Software engineers in test. For more information please see https://about.gitlab.com/handbook/engineering/quality/
Some of QA is technical needlessly; we think that's a common issue with tooling today. QA today is just not accessible to folks that think about and manage change to the product, namely product folks.
Why don't you think it should be part of product?
Love that your department is called "Quality", implies more metrics-less-feeling before reading the doc.
Thanks Russell. In our experience a lot of QA improvements require automation which is more of a software engineering job than a product management one.
I think this is mostly true today, but something which we think of as broken and we're fixing. We believe it should be accessible to a wider audience outside of just engineering; there is a lot to do, but we're starting with UI automation.
> folks that think about and manage change to the product, namely product folks
It's weird to me that the software engineers building the product are not viewed as "product folk".
The people designing the architecture for and building the actual product aren't managing the changes to it or concerned about it? The people who have to address any defects in it aren't concerned about their processes as they pertain to quality?
This sort of worldview seems anti-agile and anti-lean TBH.
Engineering needs to own and focus on quality. It's construction that determines quality. It's construction that addresses defects. It's engineering that owns construction.
If engineers aren't owning and focused on quality the business either doesn't really believe there are quality issues or there is a huge alignment issue.
192 comments
[ 4.2 ms ] story [ 232 ms ] thread- If you ask the devs to do QA, you'll get no bugs other than the ones they already caught during testing and deployment.
- If you have a mostly independent QA team, they will find somewhat silly/trivial bugs like the login page not working in an extreme edge case scenario.
- However, when you ask your Product team to own QA, you get the real good stuff - "Why does this feature not actually work well with this other feature when you combine the configuration" etc. It's great!
Also: Bliss is when you as a product owner get to such a place that you trust the QA team that you've worked with so closely so much that you only have have to some basic tests a few hours after the release.
QA is pretty easy to fake (for a while). The last thing you want to do is quality check your QA team. So the further the incentives of the QA team are from the success of the product, the worse things get. It's a spectrum from the other cofounder doing QA to someone in another country charging by the hour.
There are opposing forces to this. It's inherently difficult for people to check themselves. Also, QA is its own skill. It's one more thing to ask devs to be great at. Maybe it's worth having some specific experts around. It's kind of a specific mindset that not all devs have.
This mindset issue is where I'm not entirely sold on what looks like a new strategy from Rainforest QA (where I used to work) of strongly targeting product teams. I haven't seen any QA tool so good that it removes the complete need for skill like indoor heating means you don't need to know anything about how to tend a fire. The best ones are more like how a modern gas range helps a chef. So I question how great the results will be if you have a PM or CSM doing it. Still, I wish them the best of luck.
The solution was to break up this silo and getting these people into the product teams where more technical people could handle management and hiring.
That would require a skilled product management that is on equal footing with sales and has a veto right before stuff gets sold to customers. All too often however, the only thing that matters is customer wishes for new features, with no one taking care that the tugs the customers are pulling on don't rip the product apart.
In the end what I ended up doing was spending a lot of time packaging the product as well as educating the sales force. If you don't have well-packaged products, the sales force will sell anything, even if it doesn't exist. And they'll get backed up by a typically slightly disconnected exec team, because they also want revenue really, really badly.
This strategy worked surprisingly well. I think both the sales team and the ever slightly disconnected exec team benefited from the artificial structure that I spent so much effort making up. I guess people instinctively just like following clear structures over chaos.
(This was 5-10 years ago, and not in Silicon Valley.)
A good QA who is part of the team can do that as well.
This works right up until you hit a complexity point product can't handle, or worse, you find out that product is building something completely different from the requirements SME's are giving.
Your QA group (or misguided product group) can do jack squat about badly translated reqs or the right questions not being asked, which in the presence of the best devs and QA's degenerates into building the wrong thing perfectly, which has to get redone over and over again. A good QA that's been around the block is usually pretty good at sussing out major communication defects, but it can be tough to pull out of.
It's a surreal class of miscommunication to behold, and as close a death spiral as you can end up in, because everybody day in day out is working hard and getting frustrated but no one's getting closer to the end goal.
Fair but you have to keep in mind that part of QA's job is to explore the silly and extreme edge cases too. One of the nastiest bugs I ever found started as a joke of a test. What happens if you enter a massive long text string (I think it was something like 600 kilobytes) into the password field and enter it?
I expected the UI to barf and a bit of mischief. What I got was the entire server backend crashing and loosing every user session with it when it restarted. The joke test revealed a uncovered a serious denial of service attack vector that was trivial to execute, trivial to automate, difficult to detect, and incredibly effective.
I agree it would be silly if QA insisted that trivial or edge cases be fixed; every bug has to be judged on the cost of fixing vs the risk of it happening in the field. But finding those silly edge case bugs is still very well worth the time to explore. If not because chasing the edge can reveal serious problems, but also because it also helps define where the limits of sanity really are instead of were it is imagined to be.
Or maybe "my" process only scales so far...
It worked well for me at this scale: 30 developers, 10 QA people, 3 ux people, 2 product people "vs" 50 sales people.
There was also a separate delivery/consulting team of about 50 people which was important early on for revenue.
But this was at a cash-rich startup that had lots of money to spend on making sure we had plenty of QA headcount. That seems to be the exception, rather than the rule. Lots of startups I talk to are quite constrained in terms of dedicated QA, so the argument for empowering product managers to own quality does make some sense to me.
But what do you mean?
It also requires that our product owners handle a large part of the implementation & QA efforts.
For our application, a single piece of code might be reused 40 different ways across 100 different customers based on configuration-time items.
To ask a developer to figure out why something is broken would be a fool's errand for us. Only with the aid of trace files & configuration references are we able to perform RCA and clean up an issue from live usage. For us, 99% of the time it is a configuration thing (either on our end or some 3rd party system). If the code works for 99 clients and 39 other configurations, it might be your client+config that is wrong, not the code.
One huge upside with configuration is that it can be copied really easily. If you have a model customer that is very similar to others, you can use it as a starting point and save 99% of the work effort.
That said, it's sometimes the right thing for the team when something gets thrown over raw. A good test team should be fine with that and play their part.
Not necessarily though. I would also argue that manual testing is just an insurmountable timesink. You'll never have enough time because manual testing balloons to the time allotted.
> There's an excellent methodology a good test team uses on devs that throw it over the wall. They throw it back with the first bug report.
Sure, but continuing a cycle of "not my problem" helps no one and wastes time. Removing QA as a team/role/specialization, and instead making it a step in the process a dev goes through to ship software, you're fixing the broken feedback cycle QA teams create.
Isn't the problem with relying too much on unit and integration tests that they don't consider the end-to-end app experience? (Which, in my mind, is what ultimately matters when customers think of "quality".)
They say that instead of being developers writing codee, they should be non technical product managers using no-code.
They would like that "test-automation" would mean instead of 2 QA engineers now you need 1 doing 2x more.
Sad reality is that with "test-automation" you actually still need those 2 QA engineers and maybe part time dev to help them out. The upside would be that those QA people would spend less time repeating clicks and improve quality of checks.
Business thinks that you can do test automation in a vacuum without any domain knowledge and without knowing the system.
You always need people to understand the system, to understand the automation and to keep domain knowledge in house.
If your whole QA team leaves and you have automation (or even documentation) - you are still in deep shit because hiring new people and training them on the system will still be needed and only then they can learn to run the automation. They still need to spend time reading documentation and understanding it, because that something is written down or documented does not mean other person has the same understanding of it and the same reference framework to use that documentation.
Well you can hire some new people and let them run automation built by previous team but ... good luck with that.
Writing code is generally easy. Figuring out what code to write is hard. I can spend 80+% of my day thinking about an overall system, what things are important, and how to design things so that they both work and don't prevent change later. Then sitting down and writing the code to do that is (usually) the easy part.
Writing automated tests is generally easy. Figuring out what tests to write is hard. Does the code break on normal cases? Does it break on edge cases? Does it handle all the known use cases? Does the implementation actually achieve what the client is asking for? Does what the client asked for actually make sense?
You can't just automate away testing. Automated testing (unit, functional, system, etc) is a power multiplier for both dev and QA... not a total solution.
I'm not opposed to a model where non-technical roles are empowered to define test cases and contribute to approval workflows of a release pipeline, but that doesn't absolve developers of the primary responsibility of being end-to-end owners.
I know "devops" is an overloaded term that means different things to different people. To me, it's when developers are involved in product research, requirements gathering, user studies, design, architecture, implementation, testing, release management, and ongoing operations. They're supported by a dedicated product manager who leads the research/requirements gathering/user studies initiatives.
"Oh, you're a programmer so you must not be good at soft-skills and communicating with the business."
"Ah, you have soft-skills and you also program? You must not actually be very good at programming so you should stop that and just become a PM."
"Ah, they are PMs/Sales/Whatever so SQL is too hard for them and they can't be expected to learn git or markdown; that's for nerds."
This leads to cringy, insulting sentiments like:
> The most valuable thing you can be doing is writing code
Ughh.
This looks like it has a shallower learning curve to get started, but I would imagine that after a certain point it winds up being less productive to use the UI than to write code and this is ultimately why visual coding hasn't taken off. At some point someone will need to become a power user and be their organization's resident expert on Rainforest, but at that point they're spending all of their time in Rainforest and they're no longer a product owner embedded in the business.
At the end of the day the business owner should be writing specifications, the engineers should be using them to create tests and they each should be collaborating closely on both.
Nobody but developers could actually manage to write any tests, and it was harder than just using the normal tools, plus maintaining all the glue besides.
The glue code behind it is a different fish, and needs someone with software engineering trainging to build it and love it.
However, we also allow you to test using a crowd of humans, should folks need more nuanced feedback about things, or have much more complex things to ask.
Disagree on who should be writing tests; I think that's the case today as tooling doesn't support anyone but engineers (QA or not) automation things, or manual tests.
I disagree with the problem's that you mention in this article:
Developers aren’t incentivized to prioritize QA testing
Developers are typically evaluated based on the quantity of software they ship, and how fast they ship it.
That's an organizational problem that is not universal and certainly won't be solved by a QA automation tool.
Developers’ job satisfaction goes down when they’re in charge of QA
Expanding upon one of the previous points:, we’ve seen that most developers just don’t enjoy doing QA.
This might be the case for manual testing but for automated testing the opposite is true. Delivering tests along with code increases integrity, makes debugging significantly easier, and helps clearly communicate the intention of each feature. This only works if there is collaboration between the developers and the product owner.
Clearly you have a viable product that works for many organizations, but it's certainly not a one-size-fits-all solution nor a best practice.
I think you're wrong re:organizational problems, it's part of it - but most developers (in my expereince) do not want to do QA outside of unit-tests and maybe integration tests. They want to write code, and ship things. Automation, at least traditionally is brittle as well as slow to write, and few love it. Whilst tooling like Cypress does improve things over Selenium, still, I've not met a developer that actually enjoys that kind of testing.
The industry uses dedicated QA people based on the assumption that you can get two for the price of a developer.
In fact, if you have two developers, one of whom is more clever than the other, you want to give the code cranking to the less clever one, and use the clever one to verify the code cranking and make improvement suggestions.
If you have clever people designing the system, and clever people finding problems in the commits, you can just have average coders cranking out the bulk of it. Most code doesn't need cleverness, just persistence and consistency.
In my experience (hiring manager for QAs), there's no way I can get SDETs for anything like the price of a journeyman dev. I'm probably looking at a 20-40% premium right now for someone decent.
The problem with almost anything else is that it increases the number of hand-offs between groups of differently-aligned people on the route to production. If you're aiming at multiple deployments per day, with sub-hour commit-to-prod times, the latency in those hand-offs adds up way too fast.
In my case, devs were just testing what they changed and never noticed something horribly broken somewhere else.
I think all devs should test that what they built works in production (you don't need QA for that), you need QA to keep everything else tested and working.
There's your problem. If they broke it but didn't test what they broke, then they changed it but didn't realise they had. "Horribly broken elsewhere" sounds like the sort of thing that should show up in an automated test, no?
If the problem is "we don't have a good enough automated test suite and our architecture doesn't limit the blast radius of changes" then I can see how throwing QA headcount at it might look tempting.
I had this in my past job. Have 1 QA per two developers, the QA sits with those two developers, and you are constantly telling them when a build is done and on staging. They write tests, do some manual checks, and then tell you how it is going relatively immediately. They also handle the problem of reproducing bugs and narrowing down exactly where they occur, which is not trivial.
For all the faults in that organization (including letting our test runner be broken for months), we didn’t put out a lot of bugs and we found all manner of strange edge cases very quickly.
Because there is little incentive for devs to police themselves and there could be multiple dev teams spanning client/server that needs to be integrated and tested.
A slightly better org to own QA would be product team.
Siloed QA team runs the risk of becoming a bottleneck as work from disparate teams comes through.
The best experience I had was on a team that had essentially 2 BA’s and 7 devs. There was constant communication to clarify actual requirements, devs would build automated tests off them, BA’s would test against the requirements and then a business user would do a final look over. All in all features were able to be released usually within a day and there would be days we’d get out 3 or 4 releases. Only in one case did a major bug get released to production and the root cause of that was poorly worded regulations which had a clarifying errata come out at the 11th hour.
For as many faults as that company had that caused me to move on I’ve yet to run across a team that functioned as well as that one did.
It sounds like a sentence about programs, not people. Consider people tend to make mistakes and being flexible in their intentions.
If all people were angels, no government (or whoever regulates) would be necessary.
If all people (and systems) were perfect, no backups would be needed.
The dev team manager is focused on growing new features in the next release.
QA is focused on the excellent of the current release.
Subordinating quality to new releases is the result.
In a similar situation, devops supposedly addressed similar tensions of dev and ops.
What happens when the dev mgr asks QA to rush testing? Been There, Done That, Bought The T-Shirt, Said "No," and Tendered a Resignation
In our working agile environments, you know the team's velocity. Managers know the team's velocity. Points per story include QA work. If they want something faster in this sprint, something else has to be taken out. Firm but fair, and it works very well.
Was there pressure to keep releasing? Yes. But with the rapid feedback, it never became too onerous to get them done anyway.
Embedded QA doesn't scale for projects where the teams are larger, or the concerns and complexity are significant.
https://qualitysystemsanalyst.com/2017/10/06/independence-th...
1. Develop some feature (me and the other engineer are full stack)
2. Send to other engineer for code review
3. Deploy to a staging site, with test data. Send to product girl to QA
If she finds a bug (or realizes her design doesn’t look as right in practice as she envisioned) she sends back with notes, we implement or fix them, and then start back at 1-2.
I think QA is different for every org but I believe there should be steps. At bigger orgs there should be QA for every feature developed (so each team should have a QA person) and then an integration QA of the current beta.
It shouldn't be automatically thrown around or diffused as something tertiary that can be subordinate wherever it lands. That works in a 3 person startup, but not if you're building an aircraft carrier or an iPhone.
I personally hated writing automated tests, but it did give me a much deeper understanding of the product. Especially when writing tests for features I wasn’t working on.
That said, having a dedicated QA person within the team is far more effective. The dev team can build features, while our QA person works with the business to come up with test cases and test data.
* It eventually becomes too much work for the PM as the product grows or the PM just gets tired of the tediousness of creating tests for all the edge cases
* PM hires someone to help with creating the automation tests
* That guy now becomes "the QA guy"
* Repeat till you've re-invented "the QA team"
Devs should do automated unit and functional tests, but after that, get some good QA that do not have the same boss (at least at the first level) as the developers.
I've had quite a bit of user training in a product I work on, but unsurprisingly the QA group of people who in their primary roles use it daily and have degrees in the problem domain are way more effective at finding non-obvious problems.
if your product is e.g docker, git - then yea, easily
if it's soft for different industry, then complexity increases
In the passage I am unable to find, he gets into the issue of having a worker making widgets, and another worker checking the widgets. In our world this would be a developer and a test author. He states that this reduces quality: the maker knows that he can make poor parts when he is in a hurry, because the tester will catch them; the tester knows the worker personally, and trusts that he will make good parts, so doesn't have to be very thorough. Instead, we need the worker making the widget to be responsible for its quality, and further to be empowered to do so.
1. Qa doesn’t know what can be covered by unit or integration tests
2. Since they treat our code like a black box, they may create permutations of tests which which cover the same functionality
Maybe this is part of the draw of having a qa team. Feature coverage rather than code coverage. The downside is this can create a huge number of expensive to run manual tests which may be hitting the same code paths in functionally identical ways.
The tooling for automating manual tests of web apps is almost there: puppeteer, recording user inputs and network calls, replaying everything and diffing screenshots.
Since qa tests are tied to features and not code, There’s also the problem of having to run all qa tests even if you’re releasing minor code changes. My build tools are smart enough to return cached results for unit tests whose dependencies didn’t change, but there’s no equivalent for qa tests.
Handing off QA to an external team is broken because those people don't have the necessary experience with the product, nor can they quickly and easily engage with development to get to the heart of a problem (and a fix.)
Having QA rely exclusively on automation brings quality down as the application's complexity goes up. Writing tests to cover every possible edge case before the product ships isn't feasible.
The best solution I've seen in the decades I've been in software development has been to have QA and Dev as part of a team. Automation covers as much of the product as possible, and continually grows as issues are identified that can be covered by CI from here on out. QA become essential at the boundary of new features, and require a deep understanding of the product in order to know what knobs to turn that might break the application in ways the developer never thought about. Once those sharp edges are also automated, QA pushes the boundary forward.
[1]: https://twitter.com/brenankeller/status/1068615953989087232
There are still gotchas to look out for in the team-embedded QA approach. In typical team sizes, you often end up with only one QA per team - you need to make sure they have cover (everyone needs a break), and they need support in their discipline (do something to share QA knowledge across teams).
The other thing to consider is, if you’re careful, user feedback can be obtained more quickly. If you can keep things stable, then you won’t upset your users too badly and you’ll avoid the need for QA to act as a stand in.
it's not.
OpenOffice won't print on Tuesdays https://bugs.launchpad.net/ubuntu/+source/cupsys/+bug/255161...
etc
Most projects I've worked on have been games where the studio was split into teams of about 6-10 people with one dedicated in-house QA member for some but not all teams, a few in-house QA workers not assigned to a specific team, plus a much larger external QA team (from the parent company/ publisher). That has worked great.
I've also been on projects with only external QA. It works, but the lack of internal QA has been a frequent annoyance.
That's difficult to hear, repeatedly, from the same person. And I've seen a lot of developers not react well to the underlying discomfort.
When it's an external team, at arm's length, that relationship can get pretty bad.
At least when it's someone on your team, then you have non-QA moments and interactions to help balance out the unfortunate bug truth.
And the end result is faster, more thorough, more productive bidirectional communication between QA and coders.
The entire purpose of this article is to self-servingly attempt to convince the reader that their product is the only solution to QA problems.
It correctly identifies some challenges with QA, but this solution is certainly not the only way to have effective QA. That Rainforest is resorting to such a disingenuous presentation of solutions to QA issues makes me think they probably don't actually solve QA problems very well.
"We have researched what makes QA successful and X, Y, and Z are what we found. Here's how we believe we're solving Z for our customers" would be a much more honest pitch.
We are to my knowledge the only product built for this kind of cross-functional ownership, that's why I don't recommend other products.
While I agree the post is too marketing-y - to be honest I didn't expect it to appeal so much to the HN audience - it was written in good faith. We built the product because after 7 years of building a product for one of those silos, we became convinced that the only long-term durable solution was to empower everyone to own quality. Clearly I need to figure out how to strike the balance that you outline in your last sentence, which was the goal!
It obviously needs a bit of critical thinking when reading it and taking everything with a grain of salt, but that's something I really hope we can expect people on HN to be capable of?
I think the challenge is that, while we agree on the optimal setup, it's almost never done like this. There's lots of reasons why, but the main one (imo) is organizational scaling. The logic of scaling teams tends towards specialization, especially because developer time is extremely expensive.
There are many ways to address the problem of QA. From our perspective, if they don't broaden the ownership of QA from just developers or just QA engineers (which is where all the current products are targeted), they will exacerbate this specialization problem.
The main challenge with software is the possibility of unexpected regression in areas that are non-obvious when implementing a given feature. One of the more scalable ways of catching these regressions is via some sort of automated testing. I think that a QA support team can provide the tools and support that allow for teams to easily add end to end testing, which is vitally important when build large distributed software.
If the QA team is responsible for testing every feature from many teams there is definite risk of bus factor and finger pointing.
In the super ideal world the testing is specified via something like Cucumber and the orginal author could be the product owner with feedback from the engineer as he works to that specification. But I have never seen this though have heard it can happen, rarely.
Read the piece - it says no such thing. It's talking about 'ownership', which is distinct from participation and most certainly from interest.
>Compare this to code-based automation solutions like Selenium or Cypress, where a coder would need to manually update the code across all the relevant tests any time the product changes.
... Well, no. Those tools can have reusable chunks and references that work just like your steps and are organized in the same manner, so that things used multiple places are stored once and reused. That's a basic programming good practice. When a change is needed to, say, a login flow, you don't change 100 UI tests, you change the login function that those tests are all calling.
> Tests created in code-based automation solutions like Selenium and Cypress evaluate what’s going on in the code behind the scenes of your product’s UI. That is, these solutions test what the computer sees, not what your users see.
and
> For example, if the HTML ID attribute of the signup button in the example above changes, a Selenium or Cypress coded test that identifies that button by its ID would likely break.
This is all very disingenuous in my book. In these tools the tests interact with the rendered web page and _can_ be made to use IDs, classes, and other brittle stuff as selectors, but that's a naive approach that a professional wouldn't typically use, because the brittleness surfaces early and there are better patterns that avoid it. One of those is to use selectors based on something that itself should be present - eg, click a button based on its label being "Log In". If that test fails because the button label was changed or removed, that's a reasonable failure, because labels are important in the UI. In the case where we are selecting/making assertions about DOM elements that don't have meaningful user-facing labels, dedicated test-id attributes solve the brittleness problem. But by and large, if something is interactive to the user, it should have a label, so the fallback to test attrs for selectors is rare, and used for the case where we want to measure some specific effect took place that is not itself an interactive element.
It's unfair to suggest that these tools inherently create brittle tests.
> Rainforest QA tests don’t break when there are minor, behind-the-scenes code changes that don’t change the UI.
but
> It works by pixel matching and using AI to find and interact with (click, fill, select, etc.) the correct elements.
Look, maybe your tool is GREAT, I haven't tried it. But a functional UI test based on label contents may be more robust than something that breaks when the visual structure of a page changes, which happens itself pretty frequently. Maybe the AI is good enough to know that a bunch of stuff moved around but the actual flow is the same and can find its way through. With tests based on labels and dedicated selectors, devs can make massive structural changes to the DOM and not break any tests if all the workflows are the same.
I would sure love if this post contained more info on how the visual AI system of selecting elements is better than using labels, test attributes etc as is conventional in automated testing done by professionals. Likewise it would be good to compare with an actual competitor like Ghost Inspector, where non-programmers have been able to generate step-based tests like this for years. The main gripe I had with Ghost Inspector was that it creates brittle selectors and so tests need to be changed a lot in response to DOM changes, unless a developer gets in there and manually reviews/picks better selectors.
If what you have a is a tool that _makes more robust tests than Ghost Inspector_ but is _as easy for a PM to use_, then that is interesting.
I actually support this underlying idea completely - I love systems where non-developers can create and maintain tests that reflect what they care about in the UI. I even love it when tools like this create low-quality tests that ar...
Regarding DOM interaction, you're missing the point. All automation that tests the front end code, regardless of how it attaches, is using a path that is different from your end user. The end user interacts with the application visually. That's why we test visually. A decrease in code-based brittleness is just a nice side effect. And as you note, this is a very high-level post outlining one key idea about quality ownership. You may be interested in this, which is one of our front end folks talking about why we believe testing visually is superior: https://www.rainforestqa.com/blog/the-downfall-of-dom-and-th...
We have been selling a QA solution for almost 10 years. In that time we've seen thousands of setups and directly worked with hundreds of teams. Your claim "weaknesses of automated test frameworks [...] are compensated for quite easily and routinely" is, quite simply, not true for the majority of engineering teams - few QA leaders, including proponents of Cypress, would agree with you.
Not all users interact visually. Selecting interactive elements through accessible labels, not based on visual appearance, is a better practice imo. I want important parts of the DOM that are critical to building a correct accessibility tree to be a part of the test, and to fail the test if we change something that makes the accessibility tree incorrect. Because that is the API for assistive technology and it communicates the user interface for lots of people. "Correct behavior" of an app or website includes "the accessibility tree accurately reflects the structure and nature of the content" and "form fields are labelled correctly". I might be in the minority in thinking that, but I believe it 100%.
Nothing I've seen so far (including the post you linked) suggests that the OCR-like approach can tell us anything about the accessibility tree.
The post does make a similar point to mine:
> If your app changes visually, your test will fail. This is of course a tradeoff and a conscious decision to couple your tests to your UI instead of your code, but the upsides bring much more value than that the downsides take away.
I disagree on the tradeoff. I can run screenshot diffs etc to catch unexpected UI changes, they are a little noisy, but I'm ok with that tradeoff because I care about more than just the visual appearance of the app.
A "visually correct" app with click handlers on divs and no semantic HTML is a liability (legally, maintenance wise, etc).. I'd like the E2E testing tool to assert that the app is working correctly, which does mean some assertions about the DOM are appropriate to me. I agree with the author of the linked post that "we want a solution that is not brittle in unwanted ways." We can be selective about what DOM things are important.
In the linked post the author says "In particular, DOM tests are tightly coupled to the structure of your code." and gives an example about a Selenium test that uses a brittle xpath that depends on a specific DOM structure.
Maybe I have not been exposed to enough of the industry to know that there are thousands of setups relying on flaky xpaths to target elements for testing. To me, it is not true that DOM tests are tightly couple to the structure of your code by default. It's a false statement made for marketing purposes and it is gross.
DOM tests "can be flaky", "are sometimes coupled to DOM structure" or whatever, is a fair assertion, but flakiness in DOM-driven testing is not a fact, it's a sign of badly written DOM-based tests. This is often the first thing I address in a code review of new tests written by somebody who does not write lots of FE test code, and they easily learn how to avoid it.
Maybe I'm wrong but it seems really really basic stuff to not create brittle selectors that fail tests for reasons we don't care about.
I like the OS-level interaction and agree that provides some advantages. I totally disagree that these advantages mean your solution wins at the "best way" to test, but it does clearly cover a different surface area than other automated solutions for E2E testing, and it seems like tests are pretty quick to knock out.
This solution could be a complement to other automated E2E tests, and I would see no reason that a PM or other party couldn't spin up and maintain some tests this way as quick-to-create set of tests to run against various builds, knowing that design changes will break them but that this is OK cause in theory it is quick to rewrite them.
But I couldn't see using this tool as the only E2E testing solution as though it is a superset of what Cypress/Selenium/Whatever tests are capable of. It is actually not a competitor with those tools. It's addresses different concerns with a little bit of overlap.
I'm happy to check out the free trial and see if I'm miss...
So that if they don't have much work on project X you just drop them into project Y.
In reality we all know that is just not possible, because people have to have domain knowledge and keep up with changes. It is not like if you come back to project after 2 months you will be proficient with it as the world changed.
This way if project X has slow times, dedicated QA will have less to do and BigCo has to eat that "loss".
In small startup it is easier to have QA incorporated into dev/product team because as software is constantly changed there is work to be done all the time.
I've been arguing the same thing for a while now. There's a clear mismatch in incentives. If you hand off the QA to developers you might initially get less bug reports but that doesn't mean your software isn't garbage.
Some of the best QA engineers I've worked with were developers in a previous life but they've since spent years cultivating a very different way of thinking about software. Believing you can get your engineers to adopt that mindset overnight is pure fantasy.
To my surprise in subsequent roles, I've seen many development teams, even good ones, who don't have that deep instinct to break their code via tests. Which was very alien to me. I've seen enough cases now to realize it's somewhat common. For such teams, separate QA engineers are a must.
I still feel having quality responsibility in the engineering team ultimately can offer the best quality outcome, but only if you can build a team with the right attitude about testing. If not, having a separate QA team is vital.
This overlooks that "no customer calls with product issues that dev needs to get fixed asap on a Saturday evening" is part of the happy path to optimize for. If those things happen, the dev has deviated from the happy path, including the embarrassment in front of fellow devs "but that's really a case we should have tested". This is another of those details the article is lacking.
Code review. Read the results of someone thinking through a process. Spot more than they will, simply by throwing more eyes at it. Actually fairly effective: getting a senior dev to cast even a lazy eye over everything gives more opportunities to discuss Why It's Done This Way and Why We Don't Do That and Why This Framework Sucks And How To Deal With It with specific concrete examples which the other dev is currently thinking about. But it's still easier to write the code yourself than review it, and things still get missed no matter how careful you try to be, so it's still just another layer.
Unit tests. They cover the stuff we think to check and actually encountered in the past (ie. regressions). Great for testing abstractions, not so great for testing features, since the latter typically rely on vast amounts of apparently-unrelated code.
Integration tests. Better for testing features than specific abstractions, and often the simplest ones will dredge up things when you update a library five years later and some subtle behaviour changed. Slow sanity checks fit here.
UI-first automation (inc. Selenium, etc). Code or no-code, it's glitchy as hell for any codebase not originally designed to support it; tends to get thrown out because tests which cry wolf every other day are worse than useless. Careful application to a few basics can smoke-test situations which otherwise pass internal application sanity checks, and systems built from the start to use it can benefit a lot.
Manual testing. Boring, mechanical, but the test plans require less active fiddling/maintenance because a link changed to a button or something. Best for exploratory find-new-edge-cases, but throwing a bunch of students at a huge test plan can sometimes deliver massive value for money/coffee/ramen. Humans can tell us when the instructions are 'slightly off' and carry on regardless, distinguishing the actual important breakage from a trivial 2px layout adjustment or a CSS classname change.
So that's the linear view. Let's go meta, and combine techniques for mutual reinforcement.
Code review benefits from local relevance and is hampered by action at a distance. Write static analysers which enforce relevant semantics sharing a lexical scope, ie. if two things are supposed to happen together ensure that they happen in the same function (at the same level of abstraction). Encourage relevant details to share not just a file diff, but a chunk. Kill dynamic scoping with fire.
Unit and Integration tests can be generated. Given a set of functions or types, ensure that they all fit some specific pattern. This is more powerful than leveraging the type system to enforce that pattern, because when one example needs to diverge you can just add a (commented) exception to the generative test instead of rearchitecting lots of code, ie. you can easily separate sharing behaviour from sharing code. Write tests which cover code not yet written, and force exceptions to a rule to be explicitly listed.
UI testing is rather hard to amplify because you need to reliably control that UI in abstractable ways, and make it easy to combine those operations. I honestly have no idea how to do this in any sane way for any codebase not constructed to enable it. If you're working on greenfield stuff, congratulations; some of us are working on stuff that's been ported forwards decade by decade... Actual practical solutions welcome!
That's my best shot at a 2D (triangular?) view: automated tests can enforce rules which simplify code review, etc. The goal is always to force errors up the page: find them as early as possible as cheaply as possible and as reliably as possible.
The machine can't check complex things without either missing stuff or crying wolf, but it can rigidly enforce simple rules which let humans spot the outliers more e...
Unit and integration tests should not be generated. Those should be written by people if they find code that they are writing doing complex things like some specific calculation. It is more as a tool for understanding what you are doing and then maybe leave some tests behind for regression. But don't generate BS tests that will only slow down system and people. People have to understand what is going on and be on top of it and never "just run the tests" because tests that are passing green but are actually wrong are really bad.
UI testing should not be abstractable - it should be only augmenting manual UI testing - so tester should be automating his own work after he has done it manually. That tester should also find things that take him long time or have to be done multiple times and are not changing often so he wins time to do more important things. QA person should also be always engaged with the system and automation because that is the only way you can keep domain knowledge.
If you've got a general rule which must apply across an entire system, generate the necessary tests so that they fail granularly and don't require messing around to find the exact case which breaks. IMO that's one test, just applied to a range of cases.
An example might be mappings for Entity Framework (or similar ORMs, etc). Auto-generated migrations simply do not work if you need to limit migration downtime and maintain certain data invariants (which can't be specified in the schema, and yes, those always exist). So you need to write database migrations manually. This introduces risk of desync of entity mappings and schema.
So don't just spot-test roundtripping entities (a nontrivial system will have hundreds and something always gets missed). Instead, write a tool to introspect the DB schema and the Framework's mappings, and check that they match sufficiently closely. Every time someone adds an entity or property or something, it's already covered.
Similar cases exist when dealing with any interface between separate systems, especially when you don't control one of them. If you're regularly mapping between two models, use something like Automapper which can be asked to verify its mappings to check that every property is handled in some way.
(Granted, Automapper doesn't catch everything, but it builds a model that could probably be introspected over to spot encountered bugs and check that other possible examples of those bugs don't exist. Doing so generatively catches future additions of possible cases for free. If you're really paranoid, define some means of marking manually-written tests which cover each case, and test that a test exists for each case.)
Computers are really good at force-multiplication. They should trivially be capable of spotting other instances of known categories of bug. This is not hard to do and doesn't require wooly nebulous machine-learning shit: we've had introspectable ASTs since the dawn of compilers.
The broken practice in our industry is management...
They made a nice looking product, they can just sell their product by saying they're making existing QAs more efficient / write less code / cheaper.
Why don't you think it should be part of product?
Love that your department is called "Quality", implies more metrics-less-feeling before reading the doc.
It's weird to me that the software engineers building the product are not viewed as "product folk".
The people designing the architecture for and building the actual product aren't managing the changes to it or concerned about it? The people who have to address any defects in it aren't concerned about their processes as they pertain to quality?
This sort of worldview seems anti-agile and anti-lean TBH.
If engineers aren't owning and focused on quality the business either doesn't really believe there are quality issues or there is a huge alignment issue.