Thank you! I apologize for being 8 days behind on this. :)
It's still mystifying that there was no iOS or Mac version for such a long (albeit temporary) period of time.
Also mystifying that I apparently deserved a -4 for this comment. Seems valid to me to note Yahoo not having an app on iOS or Mac OS for well over a year.
I can see why this would work in a place that uses QA as a crutch.
All developers should aim for no bugs and test their stuff themselves but of course when deadlines are looming its easier to just code and let the QA team pick it up.
This was exactly the situation at a previous employer. There were twice as many testers as devs. I tried to advocate for comprehensive unit and integration tests and was met with a "if it doesn't work, the testers will let us know" attitude. Baffling. At my current job we have no QA testers and write much better software.
Comprehensive unit and integration tests strike me as a waste of time. From what I've seen, getting around 70%~80% code coverage is usually going to be around the tipping point where you start to get diminishing returns.
Don't get me wrong, I think testing is important. But there's tons of code where you get no value by writing tests for it. (At least in frontend development.)
We don't have a QA person, but I think it'd be great to have one. You can't write automated tests to check that things all look like they should. You can't write automated tests to check that all of the interactions are behaving as expected.
Point taken. I've never done frontend work, but I can understand the value of a human tester when a complex UI is involved. The kind of work I do is very data-oriented and quite easy to get high coverage when the code is well written.
Well, that takes away from the other work than can do - and may incentivize them to more thoroughly check their work as they go. It could be a fairly big improvement to efficiency.
I think that is a slightly harsh assessment. They forced the engineers to stop shipping shoddy code, so the QA team wasn't necessary in their opinion.
However I think there is probably a middle ground where your engineers deliver quality code and you also have a QA team to increase that quality even further.
A failing company cutting QA teams is nothing but a desperate attempt to save money. Yahoo's margins are falling and it's easier to fire people than it is to raise revenue.
I have worked in small companies before where there were no janitorial services, so we emptied our trash cans once a week, and took turns taking the larger bags to the dumpster outside. What's the big deal? And that's not a rhetorical question. I truthfully want to know why you think something like that is silly.
Because Yahoo isn't a small company? They have been losing a lot of engineers and it must be hell to recruit people to join Yahoo, negative press like this is only going to make it worse.
As for why I think having Yahoo engineers take over janitorial duties is silly... If you're paying Silicon Valley engineering wages to someone (which they are, no one is going to Yahoo for stock options!) you should make sure they are working on engineering level tasks. It would be a giant waste of resources to have people do tasks they are overqualified for.
It's impossible to ask engineers to do more work in a day - you can only change the work being done and/or the time you give them to do it. My guess is that they'll be asked to do more automated & sanity testing as part of their cycle, and as a result their timelines will be extended.
Imho, this is the way engineering teams should be structured anyways.
> "It's impossible to ask engineers to do more work in a day - you can only change the work being done and/or the time you give them to do it."
False. If engineers are typically working 9-5 or 9-6, and you give them new responsibilities but keep existing deadlines and scope, it in essence has the effect of forcing them to work longer hours to get the work done (or else...).
Not sure about that. Many companies have successfully creeped up the responsibilities of people while keeping their pay the same. Add a few tasks here and there, tell them that this will look great come their review, but then lead them along.
Not saying this is good or that the practice doesn't result in increased turnover, but that doesn't stop companies from doing it with some success.
They fired the QA team to force the devs to do a better job of designing for testability. When you have to write and plough through your own tests, you integrate tests earlier, and you modularize your code better to support that testing. There's a lot in the early phases of engineering that the initial devs can do that QA cannot. QA is handed a black box; dev gets to change the box.
It's a very similar big-picture realization as the move from "system administration" to SREs/devOps. Having a bunch of people throwing #@*( over the wall that other people then have to make work is a poor model for optimizing the big picture.
This is a good move. It's Mayer taking another play from the Google playbook and trying to improve the process at Yahoo.
I think we must be misunderstanding each other. Developers are human. (At least, I think we are.) We respond to the incentives and structure of the environment in which we're developing. A culture of "throw crap over the wall to QA" naturally creates an incentive for quickly writing up features, without balancing that with robustness, because finding the problems "is QA's fault". A culture of "the devs are responsible for the quality of their code" produces better code. The way to treat the devs well is to have the QA folks go off and create or bring in easy-to-use automated testing and CI frameworks, to make it easy for the devs to do the right thing.
So are Q&A people. They are the developer's teams members not their minions.
> We respond to the incentives and structure of the environment in which we're developing. A culture of "throw crap over the wall to QA" naturally creates an incentive for quickly writing up features, without balancing that with robustness, because finding the problems "is QA's fault". A culture of "the devs are responsible for the quality of their code" produces better code.
I'm wondering why the presence of a Q&A team would in any way change the professional responsibility of developers to create correct, testable code. If your developers have developed a culture of "throw crap over the wall to QA", then that's on the developers.
> The way to treat the devs well is to have the QA folks go off and create or bring in easy-to-use automated testing and CI frameworks, to make it easy for the devs to do the right thing.
That is not really the function of QA. Developers need to use proper testing tools on their own, but QA has its own function beyond just testing the code. They also test the assumptions and specification that the "business" ask the developer's to code. They look for bugs that go beyond what a certain set of test criteria instructs. It is also important to supply Q&A with tools to do fast, automated testing. Developer tools are not the function of Q&A.
Q&A has a job to do, and its irresponsible for Developers use them as a substitute for their responsibilities. As an example from other industries, safety is not just the Safety Officer's problem.
// I've been equal years developer and system admin working with good and bad Q&A people
Great. I worked for a company that didn't invest in QA, it was consistently a !@#$% mess. When you do this, the need simply shifts to the customer. I wouldn't install our software until the 4th or 5th hotfix patch was available.
Certainly, I'm an advocate of a more responsible dev team sharing the quality tasks and continuous integration too. But no QA at all? Hahah... maybe if you're a web portal that no one depends on for business-critical needs.
Where are these magical places that do invest in QA? In nearly 20 years of professional development, I've never seen an organization in which the criteria for shipping was anything other than "works for me". I have never seen an organization in which there was either budget or managerial patience for proper QA, let alone anything other than VERY basic acceptance testing.
When I worked at Microsoft many years ago they had an large QA team with equal footing to Development and Product in determining what shipped and when. This was the same for every development team I was a part of there for over 9 years. It wasn't perfect, and the ship cycles were slow, but the quality bar was set higher than what I've seen outside of Microsoft.
I once worked in an organization that has 1.5-2x QA then dev. I now work in a place which has 1 QA for every 8 devs -- and there are far less bugs here then the other.
I think the reason is proper tooling, a culture of thorough automated testing, and ownership of code.
Healthcare. Many companies developing medical software require two layers of QA (programmer review and QA; then a separate QA team) before release. "Works for me" is not appropriate when the "me" in question is not a subject-matter expert, and the subject matter is potentially deadly.
This is actually a story about the triumph of continuous integration and staged rollout.
By shipping code constantly, but slowly rolling it out to users - bugs can be detected very quickly by the users themselves, instead of employing large QA teams.
Keeping a central code repository, automating builds, frequent commits and automatic tests for code are taking away a lot of load for QA teams.
You are correct! I'm a programmer at Yahoo -- deploying multiple times a day to production, with the confidence your code will work, feels great.
Manual ("batch-release") deployments have been forbidden for over a year, which is a forcing function to change development process to allow deploying to production continuously multiple times a day. This requires robust test and deployment automation and for engineers to better understand what they build. It's pretty nice overall!
Forgive the throwaway -- but how does Yahoo define "will work?" Ignoring the calls against the UX change of several years ago, your own user feedback pages at https://yahoo.uservoice.com/forums/207809 make it pretty clear that longstanding issues such as spam (the same ring of spammers has operated for multiple years as Ultimate Stock Alerts, PennyStockAlerts and ExplosiveOTC and others -- simple Bayesian filtering could have solved this years ago) and things like the fact that the ignore function (a pretty core piece of functionality) has never actually ignored users - merely greyed them out, but they still take up space on the screen.
My point isn't to be negative about the state of Yahoo Finance; you probably don't work in that department, and after three years of neglect, most of the users are long gone.
My point is that if an organization is going to rely on end users to report bugs, the organization must actually respond to those bugs. Sometimes the answer might be "No, we're not going back to the Web 1.0 UX." But ignoring the top bugs for multiple years suggests a breakdown in the feedback mechanism. If Yahoo doesn't care, that's fine, it's just business. But it seems more likely that Yahoo doesn't even know there's a problem, because there's no way for user feedback to make it to the developers.
...triumph of continuous integration and staged rollout.
This makes sense to me. The fact that this wouldn't have worked at the old-style "enterprisey" places we used to work at doesn't say much in general. (It may not work at Y! either, but it seems possible...)
This is actually a story about the triumph of ecosystem lock-in and training consumers to exchange high quality for low price or new shiny. It's outsourcing your testing to your customers. When you are either shipping free/freemium apps, or are a company that rigidly controls all APIs to their system (Twitter/Facebook), you can force the users to swallow any level of quality you choose to give them.
The stereotype that QA-is-pointless in Silicon Valley is persistent and actually annoying. There will always be issues that even the most comprehensive test suite will miss.
Startups still glorify Facebook's "Move Fast and Break Things" without noting that Facebook has backpedaled from that. After all, people expect startup software to have issues, so what's the harm? Technical debt? Pfft.
Engineers are not the best QA for their own code since they may be adverse to admitting errors in their own code. QA engineers are not as empathetic.
Disclosure: I am a Software QA Engineer in Silicon Valley.
Where I work teams are made up of a product designer, UX designer or two, devs, and QAs. The QAs are responsible for high level automated tests as well as manual testing for the parts that are hard to automate. It makes communication between devs/qa/design much faster as we all sit together.
I was on a large, fast moving project, with dozens of components. Some genius decided that devs will do all QA, automated testing, blah blah. It was a disaster.
The insane deadlines required devs to write lots of poor-to-average quality code (tried code reviews and peer programming ... no time for that so it fell on the wayside). The automated testing done by devs was terrible but understandable. If you are up until 2am hacking out code (without any precise requirements), then why bother with testing? We ended up having one -somewhat- central component that had "gating tests". Everyone stuck their tests there. That made things worse since that one component was the one that kept seeing failed tests. The PMs did frantic "user-like" testing before demos. You can imagine how fun that was.
In my opinion, the decision to not have a dedicated team of testers was the big mistake in all of this. When you have many teams, many components and no precise requirements, you need an independent QA team to coordinate and prevent people from passing the buck. In a time critical project (what projects are not time critical today?), you don't have precise requirements and devs have to "sling" code. I accept this reality. But I don't except the "no QA will make you mature devs" stupidity. If I was being a mature dev, I would refuse to code until the requirements were clear. None of this 2 week agile-scrum nonsense.
Oh .. and one other big thing. The project was a cloud project that needed to be up 24/7 while we were developing it (for beta users). It was like going from one outage to the next. What a disaster!
So what I learned is this: not only have a QA team, but have a 24 by 7 QA team for the kind of project I was on. Note... not all projects are the same!
> tried code reviews and peer programming ... no time for that so it fell on the wayside
You don't think that could be related? If you had a sane development cycle, that may have helped. Having QA would likely have helped as well, but a healthy dev cycle is a good start.
Actually, it sounds like the main problem for the kind of project you were on was realistic timelines, not a QA team.
And I don't think an independent QA team helps to coordinate people and avoid buck-passing, IME, the more different teams are essential to delivering a piece of software to the customer, the more opportunities for buck passing, and the higher level of management the buck passing occurs at.
That sounds like a problem with lack of resources and bad planning, not just lacking qa. If you were to choose between a) hire 3 devs to offload your work so that you can focus on your tasks better, or b) hire 3 qa guys that will keep throwing back the shit you produce because you are overloaded. I don't think any qa would help in such a situation
The thing that I think is missed in the "getting rid of QA" debate is the size of releases. When you're deploying continuously, the size of what's changing drops substantially. Adding QA as a gating step in the process between developers and production is inefficient and, worse yet, QA will be overwhelmed by the number of rounds of QA they're forced to perform. It's not uncommon for QA regression testing to take days so when there are multiple releases per day, you can see the problem with putting QA between development and production.
That's not to say that dedicated QA doesn't have a purpose. They can and should be testing the live code on a regular basis (weekly or daily, depending on how long the testing cycle takes). But automated tests are what's responsible for ensuring that nothing arrives in production in a completely-broken state and it's assumed that the benefits of continually shipping software will outweigh the downsides of occasionally having subtle bugs in production. And even with QA involved, bugs will make it through the process. At that point, the expedited process of pushing code to production becomes a huge win. Testing strategies often aim for the best MTBF, but when that comes at the cost of MTTF, it's not always a good thing. We've had bugs that were fixed in production less than 10 minutes after the bug is filed.
The other point that gets missed is that people assume that there's no manual QA happening and a developer's careless change just goes to production and wreaks havoc. This ignores the code review process, which is crucial to delivering quality software in a continuous deployment scenario. On my team, changes require 2 +1s before being merged into master, subjected to continuous integration again and eventually deployed to production. Moreover, if any engineer reviewing the code isn't sure they fully understand the change or otherwise wants to see the code running, a single command that runs in under 10 min will spin up an environment in AWS using the code in the pull request so that they can do any manual testing they need to feel comfortable adding their +1. When their done, a single command cleans up that environment.
The thing to keep in mind when designing a testing strategy is the context your software runs in. I would not advocate this testing strategy for code that runs in a vehicle where a bug could cause physical destruction. Likewise, I wouldn't use it for an application with access to highly-sensitive medical or financial information where a leak or data corruption could mean millions of dollars in losses/fines. But for the majority of internet software, the stakes just aren't that high and the gains from a streamlined development process will outweigh the losses from bugs that find their way into production.
Disclosure: I manage a team that deploys continually, usually upwards of 20 times per day. We're responsible for our own QA and have a significantly lower defect rate than other teams in the company with a more traditional QA strategy. However we still draw on QA resources when we feel like we're pushing something risky.
You can be sure that any system at Yahoo! or anywhere else that is the bread and butter of the operation (ad systems, accounting, fraud tracking) still involves a software QA team.
Facebook could afford to "move fast and break things" when they weren't taking money from anyone. Now that they're a real company, that kind of attitude can get them adverse attention from the SEC if it causes an accounting discrepancy or from trial lawyers if it upsets their advertisers.
Surprised that people are finding this unusual, in web/mobile anyways. In my experience most engineers do some level of QA themselves, particularly in start-ups < 1000 people. In what ways does an engineer being their own QA negatively impact the company?
Yup me too. At AWS we didn't have test engineers either. Everyone was responsible for testing their own code. We didn't even have SREs, so everyone was also on call.
This just meant that everyone made sure they were writing well tested code before it got released because you didn't want to be the guy who made yourself, or worse, your coworker have to fix something at 3am.
Of course I can see how this could be bad too, like if developers really dislike writing tests. On the other hand, the people who write the code seem best equipped to understand how to set up automated testing most efficiently.
> This just meant that everyone made sure they were writing well tested code before it got released because you didn't want to be the guy who made yourself
ha, yeah right.
What it really means at Amazon is build your service then bail for a new team before you have to maintain it.
Exaggeration, but somewhat true. I have a friend there now who has an oncall rotation that's split into day/night. nighttime oncall basically means you are working graveyard this week. Of course that is really because that team should have a proper support staff, it's a vital service.
Amazon, to me, is the epitome of a company that combines dev/qa/support because it's cheap rather than because it's actually good.
Well, I should have said that the idea is people would write well tested code before it gets released. I can also attest it didn't necessarily work out that ideally in practice.
On the other hand, I'll disagree that they combine dev/qa/support because they're cheap. This actually makes little sense to me because typically QA roles are paid less than traditional software developers. That being the case, it doesn't make sense to get rid of them and let your developers do that stuff if all you want to do is save money.
What I heard was that they used to have support staff but changed it because it wasn't working out. I wasn't there so I don't know if that actually happened, but I can see how it's both harder and slower to have someone who doesn't know the code base fix bugs on the spot.
Honestly I can see it both ways. I think there are a lot of benefits to having the developers write tests themselves. On the other hand, when a project gets big enough, I can see how it makes sense to have people only working on tools like automated test frameworks or build stuff.
Mostly that you're asking the developers to become experts in software testing and verification, in addition to their existing knowledge. If you have good mentoring & examples & guidelines, then everybody can learn and move along roughly the same path. That takes time and effort to set up, so in those small startups you're likely to see wildly divergent approaches to testing + the friction when people think they should standardize, or when they actually do need to, or when people switch teams.
>I've worked with many a QA who would get bent up over a detail outside of the spec that didn't really matter, and where all QA testing was manual.
This is where you need management (or someone from the product side) who can set priorities, where needed, and put and end to pointless side-disputes that can and do crop up.
Surprised to see the negativity here. I have worked in environments with traditional manual QA, and environments where all development is test-driven and nobody is allowed to merge a feature that lacks automated test coverage.
Both the productivity and the quality were higher in the places with fully automated testing. Which is not shocking at all: does anybody really think a human can run through 800 test cases better than a computer can?
It's not a magic way to save money -- the developers obviously end up spending time writing tests. But the long-term value of those tests is cumulative, whereas the effort spent on manual testing is spent anew every release.
Manual review is still good for noticing things that "feel wrong" or for helping think up new corner cases. But those bleed into product owner & design concerns, and aren't really a separate function.
Moving from an environment with a 10:1 dev:qa to 2:1 showed me what happens when dev is not responsible for shipping working software.
No thanks. It's a bunch of deflection and diffusion of responsibility coupled with high latency flakey interactions between different teams. Everything that can slip through the cracks does slip through the cracks.
I'm sure QA can be done well, but I am convinced that giving your devs a pass to not finish their work is a dead end in several dimensions.
In practise, a manual QA team encourages Devs to throw shit over the wall and expect someone else to do some basic sanity checks they should have already done. By the time those are done, what the QA team theoretically could find gets shipped.
Then, when its discovered in Prod, the QA team will get in the way of a speedy fix.
That's a management problem, not a problem with having a QA team. The top-level QA manager and the top-level Dev manager should both report to the CTO, and the Dev manager should be judged by how many issues the QA team finds. To keep things fair, the QA manager should not be rewarded nor penalized based on the number of issues found pre-release, and everyone should be rewarded or penalized based on the number of issues found post-release.
This keeps the devs incentivized to make sure everything works before the code goes to QA, and it keeps everyone incentivized to eliminate as many bugs as possible before release.
Have you ever worked for a fortune 100 company? The dev manager has probably never met the CTO. Im in this right now with my currrent job. I still keep my builds and test them longer than i should. Less broken shit gets through, but i regularly delay my qa ba's because of it.
So, not the CTO, but some shared higher-level manager. The point is that the QA lead cannot report into Development, and Development cannot report into QA. Both of those arrangements can lead to a bad actor stopping the other group from doing the right thing. (Eg: if QA reports to Dev, then Dev manager can force bad code to ship, and if Dev reports to QA than QA manager can prevent shipping and force Dev to waste time on excessive/pointless test code development that's not cost-justified.)
I've met people that call something similar 'fake agile'.
Surely everyone should report into the business (end-user on the business side) lead responsible for making this functionality happen. This includes Dev, QA, business. QA should be business? This is no longer 1985, there is no reason business (product owner) shouldn't be a capable PM and BA for their own product.
In the company I used to work for, the CTO and VP of Product Management were equals, both reporting to the CEO. The CTO's reports included the QA director, Dev director, and Project Management director.
On any given project, we had a Product Manager who set the business requirements, a Project Manager who ran the project, a Dev lead in charge of the development team, and a QA lead in charge of QA. None of these people could override the others because of the reporting arrangement, so we had to collaborate instead to make sure we all met our goals.
Early on this arrangement worked really well. But it started falling apart when ownership of the company changed and politics became a factor as people who cared more about moving up through the organization than about the success of the company and quality of the products were hired and promoted. My take is that bad management will destroy any approach you might take to producing good software. That's why I eventually left.
I work for a FTSE 100 company and the dev manager and CTO in the division where I work are very well connected. I don't think the size of company is necessarily the only factor here.
> The top-level QA manager and the top-level Dev manager should both report to the CTO
For an organization with a single product (and, particularly, one small enough that the "top-level" QA and Dev manager are also first level managers), this makes some sense. Otherwise, this means that each product has no common level of technical management below the CTO, meaning it has no meaningful technical ownership.
It doesn't have to be the CTO; as I mentioned in another reply my intent was that QA and Dev need to be equals in the reporting chain, rather than one reporting to the other. That can work at any scale; even on an agile team with one QA and one Dev, they both report to the same boss who's responsible for both sides.
Incentives often cause really nasty political wars within orgs.
When a site gives a 500 because a database went down and the web app couldn't connect to it... is that a bug, a missed test case or should ops take a hit on down time? Furthermore, if you argue that the dev team should have reasonable failsafes in code to connect to a db, in a 10 year old organization, should the current Dev team pay for something that could have been in the code for years?
If you set up a system where devs hand off to QA then they hand off to ops to deploy. Furthermore every step is incentivised somewhat against each other. Even if all teams are equal, it ends up in my opinion, to the path to CYA and Waterfall. Everyone is more concerned with problems not being 'their' fault than shipping good code.
You're absolutely right, assuming we're talking about business/consumer apps and not pacemakers or rocket ships.
An incentive/penalty system for bugs without an incentive/penalty system for features completed leads to paralysis. And a complicated incentive system leads to game playing over productivity.
It's an old saying: be careful what you measure because you'll get a lot more of it.
I'm beginning to think that the best incentive system is just a good base pay, with an emphasis on personal learning and growth. Have engineers strive to become better engineers by taking courses, attending meetings and publishing content. Quality will fall out of that as a positive side-effect.
The solution to this is to require a bug/ticket/tracking item as part of the change. Since QA knows that dev needs to create one anyway, they might as well get the credit.
Where this starts to break apart is that QA can get overzealous and file bugs for different incarnations of the same bug.
The top-level QA manager and the top-level Dev manager should both report to the CTO
Having a top-level manager who is responsible for a whole discipline doesn't work, it just creates a high wall to throw shit over. Developers and QA testers really should report to the same first line manager.
>and everyone should be rewarded or penalized based on the number of issues found post-release.
this sounds absolutely horrible. What a great way to remove all agency and ownership from an organization and incentivize people to pass the buck off to literally anyone but themselves and to work on easy, cupcake features with no risk.
>This keeps the devs incentivized to make sure everything works before the code goes to QA, and it keeps everyone incentivized to eliminate as many bugs as possible before release.
What kind of organization are you working in where you think devs have to be motivated by punishment or reward? Do you not hire the kind of people who are intrinsically motivated and want to produce software that doesn't have bugs? Do you think that if you don't explicitly incentivize developers not to write bugs that they would just fuck around and make shitty, buggy software? I just can't imagine working in an organization so dysfunctional that leadership thinks the default state is being happy with shitty buggy software.
That sounds like a terrible strategy for a lot of us, where time to fix is much more important than bug count.
This is because there is likely to be a tradeoff between speedy development and bug count.
If you are shipping physical CDs of software, or releases which you can't update easily, or if you are working in a critical environment where mistakes are disastrous (finance, health, space, etc.), then it is fair enough to be so concerned about bugs.
But many if not most developers work on the web in areas where the product is uncertain and evolving. Prioritising rate of change is more important here. Rather than extensive QA, monitoring and fast rollback are better choices.
A formal QA process, which goes along with a formal release process and release schedule, certainly does slow things down. I think for large-scale projects with a large feature/requirement count that's ok, because there's a lot of little details to keep track of and the formal process helps with that. But you also need a fast-track for fixing small things quickly, and for that Dev, QA, and Ops need to work together to quickly diagnose, fix, test, and release changes.
My experience with the formal process I've been talking about was in a company whose product is a large-scale web application that interacted with a complex custom back end search engine and a proprietary content database. The QA and Dev teams had a friendly adversarial relationship, but ultimately we all felt we were on the same team with the same goals: to produce the best software we could. It was a fantastic place to work.
My experience is the opposite. Having independent testers means you're going to be stuck addressing 100 defects if you don't watch what you're doing the first time around.
10:1? You do realize QA has to have a semblance of life too? Adding more QA isn't designed to screw up the process, but to alleviate the workload. If you want to blame anything or anyone, I'd usually shoot for 1) hiring incompetent QA that isn't SDETs but merely just black-box QA competency, and 2) process - or lack of process, i.e. an agile process of building a little, testing a little, and delivering consistently a working product on pretty much any basis - monthly, weekly even.... That's the ideal of course.
If this is in place, adding extra QA can be beneficial.
None of this of course is excuse for a dev not producing code that works at least in its happy path plus/minus a few of the most obvious exceptions/error paths...
QA is essential. How else do you know that you have built what you set out to built? Traditional manual QA is highly effective, so much so, that you can get huge gains by automating it. QA occurs at many levels and it makes sense to have a dedicated QA team for each level. Generally, the QA team for a level should be the exact people who requested and/or approved the specific feature subject to QA, as they are in the best position to confirm or deny that the feature meets the specification, as they created and/or approved it.
But you're talking about QA as a general process, not QA as is meant in the article:
> Software engineers at Yahoo are no longer permitted to hand off their completed code to another team for cross checking.
Dev Team A was giving a batch of code to Dev Team B to review.
I agree that the business user/product owner/whatever should be reviewing everything in test prior to approval and in production after but I'm not sure that's the article means by QA.
The arguments here such as "oh they just make the devs do it and don't pay them anymore" are ridiculous. Nobody's working 80 hour weeks doing two full-time jobs at once.
To be fair the exact meaning of Volkswagening is to fake the behavior during official tests. Example: http://qz.com/515100/samsung-is-accused-of-volkswagening-its... Note that volkswagening.com was registered 20 days after the scandal, but is available for bid, if you want to do something fun with it.
It doesn't even have to be intentional. If you write the code, you always have in mind some picture of how it should be used. That picture means you ignore the ways to use it which you not intended. So when the code is used in that way, it breaks. But you won't ever write a test for it because you would never think such usage is possible - exactly because you know too much about the right way of how it should be, so you start to think it's how it is.
Devs don't think like users. They know a lot about computers and can make good guesses about how other devs think. So devs are not the best people to test code that's going to be put in front of non-dev users.
Users are likely to have different models, and a different set of expectations. You can't write tests for all those possibilities because you literally have no idea what they are - and won't find out until you put them in front of users.
Isn't pager duty baked into the salary of the position? I thought that jobs where you're expected to be on call to some extent (hopefully at level 3 or 4 of support?) have an artificially high salary compared to those where it's a 9-5 gig?
This to me was always the point of automating tests. One of the best QA people I knew refused to look at the user stories. He was brilliant at finding things that fit the spec but didn't make sense and things that users might do that weren't specified.
He found bugs in the specs, gaps in the specs and just plain untested behaviour.
QA is a job that requires skill, benefits greatly from technical understanding and requires a lot of domain knowledge.
This is why specification should happen collaboratively with everyone with a stake (and that includes QA just as much as BA), you can fix them before they're developed against.
This is what a great QA tester will do. Good devs don't ship code with known bugs - but they will miss things because they code to the feature/user story/known behavior. That's what their job is. A good QA person's job will be to spend a whole lot of time thinking about how to break the dev's app that the dev missed.
Did the QA you dealt with not have any automated testing tools? That seems rather foolish to have your QA team manually test every feature every time without an automated smoke test.
It sounds like there is some opposition between automatic tests and QA. But those do not have to be opposed - one can have both. In fact, for many types of software I don't see how you can avoid having both. The only choice you have who is doing your QA - your employees or your end users. Maybe Yahoo can afford the latter. But for some companies it would be a disaster.
Even more important is that the developers spend time writing tests and their schedules are not expanded to accomodate this, nor is their pay increased. They are simply expected to cut corners elsewhere, or work more hours, or whatever is necessary. That's the real savings.
It really depends on what you're making and if it's possible to automate testing AND if you remembered to test it. Chrome appeared to have broken full screen movie playback for 5% of users once. Something changed, perf got bad, people who previously could play full screen video on a low end machine suddenly got playback too slow to be useful. They just stopped going full screen. No bug reports came in. Nothing in the testing infrastructure caught it. I only notice because my father tried to show me a video on his atom based netbook.
Similarly going full screen on a 2 monitor setup broke once. Again, no automated test.
The Web Text to Speech API is broken on every browser it's in. It will work with a simple sample but start and stop it a few times and it will break. I suspect because again there is no easy way to test it.
"Some of the engineers really cared about system performance types of things, so they joined related teams. Some started working on automation [for testing], and they thought that was great—that they didn’t have to do the same thing over and over."
There is still QA, it's just automated QA. Welcome to the 21st century.
Foley has heard that some Windows engineers, "primarily dedicated testers" will be let go as part of the layoffs announced today and that product managers and development engineers could take over some of those testing responsibilities.
This is pretty much how things go in today's environment, especially in the startups that I've seen. More things are being pushed directly on devs, which is why we earn as high a salary as we do. Traditional QA is pretty much dead, no one should be doing that now if they want to have a career in tech.
Where I work, devs do the QA, and most of the devops work as well. It's the new reality, and anyone who thinks otherwise will be obsoleted.
I don't think it's that cut and dry. I've worked in some places where the QA team was useless, meaningless red tape to get your stuff deployed. They wouldn't do much but sign off on deployment at some point, yet bore no responsibility if shit hit the fan. In these cases, they really were just an unnecessary cost and you learned pretty quickly to make sure your tests were in place, that you were testing for the right things, and so on.
But then there were the other QA teams. The people that would just reject your stuff outright if it didn't have tests (no matter if it worked) and when the tests passed they would look at things truly from a customer perspective. They would ask really uncomfortable questions, not just to developers, but to designers and business alike. They had a mindset that was different from those creating things; they were the devil's advocate. These people did much, much more good than harm, and they are few and far between. Unfortunately, while I believe they were incredibly valuable, business thought otherwise when cuts came around..
When testing, you are the headlights of the project. ... Testing is done to find information. Critical decisions about the project or the product are made on the basis of that information." [1]
I recently became a software tester, and I really didn't understand the role for quite a while. Is my primary responsibility finding bugs? Logging defects? Analysing requirements documents? Writing test scripts? Writing Status reports?
Answer: Do enough of each to fulfill your goal of gathering and sharing information with your management and dev groups.
If the software tester has problems testing, then the customer will have problems using it, and the company will have problems supporting it.
1. Kaner, Cem; James Bach; and Bret Pettichord. 2001. Lessons Learned in Software Testing. Wiley.
I think the key difference is an engineering mindset.
When you have software developers skilled in quality assurance who have the job of finding the edge cases and producing comprehensive additional acceptance and functional testing, they're an asset. It's a particular perverse mindset that I personally really enjoy interacting with as a software developer 'customer' - those evil bastards find the best bugs, regression test them, and expand to find all of that class of error in the application.
Dev writing automation tests for their code is kinda pointless. It would be better to have another dev or a different team such as automation engineers writing regression automation tests. Automating regression tests is definitely better than manual QA-ing the same 1000 tests over and over again. There has to be a good balance. Have extensive coverage of automated regression tests and let manual QA test new features. This will at least increase the frequency of release cycles. Getting rid of an entire QA dept is somewhat equivalent to shooting yourself in the foot.
This is basically my experience. I think the ideal situation is having development and QA cultures that both prioritize quality and automation. Development cultures sometimes fail to prioritize quality ("not our problem"), and QA teams sometimes fail to prioritize automation ("not how we do things"), but those are problems with those specific cultures, not with the entire concepts of development and QA.
Quality assurance is important whether you do that via humans or code, but the thing that always bothered me about Q&A was that the ones I worked with were mindless people simply looking at the feature request and the functionality on the page and comparing the two without any thought towards the actual product, business, or user.
And in that system, the developer is completely removed from the product and is just another factory worker. The closer engineers can be to users (with design to translate obviously) the better for everyone.
Clearly you've forgotten that doing anything one doesn't want to do, even if it's still software and absolutely part of their job, is "being forced to do it for no extra pay."
TL;DR: QA is changing - just like everything else.
The article makes the assumption that QA == manual QA which as a quality professional is false. Quality is about measuring risk across the development process. Immature team need manual QA while mature (in a process/quality sense) teams need much less (or none).
Quality professionals who want a sustained career needs to learn development processes, opts, documentation & monitoring. We make teams better.
When measuring the effectiveness via reduction of issues, how do you account the natural stability introduced by reducing the updates/week each developer ships? When devs are tasked with code reviews and/or QA, this is time that could have been spent on their own features. In other words, if the product is stable today, and everyone's on vacation (no new updates), the product will generally remain stable save for unforeseen usage patterns.
185 comments
[ 1.9 ms ] story [ 238 ms ] threadAnd apparently there's a native OS X client coming, as well: http://mashable.com/2015/12/03/yahoo-messenger-is-back/
It's still mystifying that there was no iOS or Mac version for such a long (albeit temporary) period of time.
Also mystifying that I apparently deserved a -4 for this comment. Seems valid to me to note Yahoo not having an app on iOS or Mac OS for well over a year.
All developers should aim for no bugs and test their stuff themselves but of course when deadlines are looming its easier to just code and let the QA team pick it up.
Don't get me wrong, I think testing is important. But there's tons of code where you get no value by writing tests for it. (At least in frontend development.)
We don't have a QA person, but I think it'd be great to have one. You can't write automated tests to check that things all look like they should. You can't write automated tests to check that all of the interactions are behaving as expected.
However I think there is probably a middle ground where your engineers deliver quality code and you also have a QA team to increase that quality even further.
I don't know of any large, complex systems where QA is not necessary. Technology is only fallible because humans are.
As for why I think having Yahoo engineers take over janitorial duties is silly... If you're paying Silicon Valley engineering wages to someone (which they are, no one is going to Yahoo for stock options!) you should make sure they are working on engineering level tasks. It would be a giant waste of resources to have people do tasks they are overqualified for.
Imho, this is the way engineering teams should be structured anyways.
False. If engineers are typically working 9-5 or 9-6, and you give them new responsibilities but keep existing deadlines and scope, it in essence has the effect of forcing them to work longer hours to get the work done (or else...).
Not saying this is good or that the practice doesn't result in increased turnover, but that doesn't stop companies from doing it with some success.
They fired the QA team to force the devs to do a better job of designing for testability. When you have to write and plough through your own tests, you integrate tests earlier, and you modularize your code better to support that testing. There's a lot in the early phases of engineering that the initial devs can do that QA cannot. QA is handed a black box; dev gets to change the box.
It's a very similar big-picture realization as the move from "system administration" to SREs/devOps. Having a bunch of people throwing #@*( over the wall that other people then have to make work is a poor model for optimizing the big picture.
This is a good move. It's Mayer taking another play from the Google playbook and trying to improve the process at Yahoo.
If that is actually true, then they should have fired the Devs and found ones who understand writing correct and testable code.
That is a horrible way to treat people.
This is not a new observation - it's been happening over the last 10 years. see, for example, http://blogs.forrester.com/mike_gualtieri/11-02-17-want_bett... or http://product.hubspot.com/blog/the-classic-qa-team-is-obsol... and, while it's from 2007, http://googletesting.blogspot.com/2007/03/difference-between...
So are Q&A people. They are the developer's teams members not their minions.
> We respond to the incentives and structure of the environment in which we're developing. A culture of "throw crap over the wall to QA" naturally creates an incentive for quickly writing up features, without balancing that with robustness, because finding the problems "is QA's fault". A culture of "the devs are responsible for the quality of their code" produces better code.
I'm wondering why the presence of a Q&A team would in any way change the professional responsibility of developers to create correct, testable code. If your developers have developed a culture of "throw crap over the wall to QA", then that's on the developers.
> The way to treat the devs well is to have the QA folks go off and create or bring in easy-to-use automated testing and CI frameworks, to make it easy for the devs to do the right thing.
That is not really the function of QA. Developers need to use proper testing tools on their own, but QA has its own function beyond just testing the code. They also test the assumptions and specification that the "business" ask the developer's to code. They look for bugs that go beyond what a certain set of test criteria instructs. It is also important to supply Q&A with tools to do fast, automated testing. Developer tools are not the function of Q&A.
Q&A has a job to do, and its irresponsible for Developers use them as a substitute for their responsibilities. As an example from other industries, safety is not just the Safety Officer's problem.
// I've been equal years developer and system admin working with good and bad Q&A people
Certainly, I'm an advocate of a more responsible dev team sharing the quality tasks and continuous integration too. But no QA at all? Hahah... maybe if you're a web portal that no one depends on for business-critical needs.
Edit: I guess the truth hurts.
Where are these magical places that do invest in QA? In nearly 20 years of professional development, I've never seen an organization in which the criteria for shipping was anything other than "works for me". I have never seen an organization in which there was either budget or managerial patience for proper QA, let alone anything other than VERY basic acceptance testing.
I once worked on an enterprise product that had a QA team of 25 people.
I think the reason is proper tooling, a culture of thorough automated testing, and ownership of code.
Keeping a central code repository, automating builds, frequent commits and automatic tests for code are taking away a lot of load for QA teams.
Manual ("batch-release") deployments have been forbidden for over a year, which is a forcing function to change development process to allow deploying to production continuously multiple times a day. This requires robust test and deployment automation and for engineers to better understand what they build. It's pretty nice overall!
My point isn't to be negative about the state of Yahoo Finance; you probably don't work in that department, and after three years of neglect, most of the users are long gone.
My point is that if an organization is going to rely on end users to report bugs, the organization must actually respond to those bugs. Sometimes the answer might be "No, we're not going back to the Web 1.0 UX." But ignoring the top bugs for multiple years suggests a breakdown in the feedback mechanism. If Yahoo doesn't care, that's fine, it's just business. But it seems more likely that Yahoo doesn't even know there's a problem, because there's no way for user feedback to make it to the developers.
the nastiest bugs will be found by either a manual tester or your customer.
This makes sense to me. The fact that this wouldn't have worked at the old-style "enterprisey" places we used to work at doesn't say much in general. (It may not work at Y! either, but it seems possible...)
Startups still glorify Facebook's "Move Fast and Break Things" without noting that Facebook has backpedaled from that. After all, people expect startup software to have issues, so what's the harm? Technical debt? Pfft.
Engineers are not the best QA for their own code since they may be adverse to admitting errors in their own code. QA engineers are not as empathetic.
Disclosure: I am a Software QA Engineer in Silicon Valley.
Whether a distinct QA team is the best means of performing the QA function is, however, a separate question.
The insane deadlines required devs to write lots of poor-to-average quality code (tried code reviews and peer programming ... no time for that so it fell on the wayside). The automated testing done by devs was terrible but understandable. If you are up until 2am hacking out code (without any precise requirements), then why bother with testing? We ended up having one -somewhat- central component that had "gating tests". Everyone stuck their tests there. That made things worse since that one component was the one that kept seeing failed tests. The PMs did frantic "user-like" testing before demos. You can imagine how fun that was.
In my opinion, the decision to not have a dedicated team of testers was the big mistake in all of this. When you have many teams, many components and no precise requirements, you need an independent QA team to coordinate and prevent people from passing the buck. In a time critical project (what projects are not time critical today?), you don't have precise requirements and devs have to "sling" code. I accept this reality. But I don't except the "no QA will make you mature devs" stupidity. If I was being a mature dev, I would refuse to code until the requirements were clear. None of this 2 week agile-scrum nonsense.
Oh .. and one other big thing. The project was a cloud project that needed to be up 24/7 while we were developing it (for beta users). It was like going from one outage to the next. What a disaster!
So what I learned is this: not only have a QA team, but have a 24 by 7 QA team for the kind of project I was on. Note... not all projects are the same!
You don't think that could be related? If you had a sane development cycle, that may have helped. Having QA would likely have helped as well, but a healthy dev cycle is a good start.
And I don't think an independent QA team helps to coordinate people and avoid buck-passing, IME, the more different teams are essential to delivering a piece of software to the customer, the more opportunities for buck passing, and the higher level of management the buck passing occurs at.
That's not to say that dedicated QA doesn't have a purpose. They can and should be testing the live code on a regular basis (weekly or daily, depending on how long the testing cycle takes). But automated tests are what's responsible for ensuring that nothing arrives in production in a completely-broken state and it's assumed that the benefits of continually shipping software will outweigh the downsides of occasionally having subtle bugs in production. And even with QA involved, bugs will make it through the process. At that point, the expedited process of pushing code to production becomes a huge win. Testing strategies often aim for the best MTBF, but when that comes at the cost of MTTF, it's not always a good thing. We've had bugs that were fixed in production less than 10 minutes after the bug is filed.
The other point that gets missed is that people assume that there's no manual QA happening and a developer's careless change just goes to production and wreaks havoc. This ignores the code review process, which is crucial to delivering quality software in a continuous deployment scenario. On my team, changes require 2 +1s before being merged into master, subjected to continuous integration again and eventually deployed to production. Moreover, if any engineer reviewing the code isn't sure they fully understand the change or otherwise wants to see the code running, a single command that runs in under 10 min will spin up an environment in AWS using the code in the pull request so that they can do any manual testing they need to feel comfortable adding their +1. When their done, a single command cleans up that environment.
The thing to keep in mind when designing a testing strategy is the context your software runs in. I would not advocate this testing strategy for code that runs in a vehicle where a bug could cause physical destruction. Likewise, I wouldn't use it for an application with access to highly-sensitive medical or financial information where a leak or data corruption could mean millions of dollars in losses/fines. But for the majority of internet software, the stakes just aren't that high and the gains from a streamlined development process will outweigh the losses from bugs that find their way into production.
Disclosure: I manage a team that deploys continually, usually upwards of 20 times per day. We're responsible for our own QA and have a significantly lower defect rate than other teams in the company with a more traditional QA strategy. However we still draw on QA resources when we feel like we're pushing something risky.
Facebook could afford to "move fast and break things" when they weren't taking money from anyone. Now that they're a real company, that kind of attitude can get them adverse attention from the SEC if it causes an accounting discrepancy or from trial lawyers if it upsets their advertisers.
This just meant that everyone made sure they were writing well tested code before it got released because you didn't want to be the guy who made yourself, or worse, your coworker have to fix something at 3am.
Of course I can see how this could be bad too, like if developers really dislike writing tests. On the other hand, the people who write the code seem best equipped to understand how to set up automated testing most efficiently.
ha, yeah right.
What it really means at Amazon is build your service then bail for a new team before you have to maintain it.
Exaggeration, but somewhat true. I have a friend there now who has an oncall rotation that's split into day/night. nighttime oncall basically means you are working graveyard this week. Of course that is really because that team should have a proper support staff, it's a vital service.
Amazon, to me, is the epitome of a company that combines dev/qa/support because it's cheap rather than because it's actually good.
On the other hand, I'll disagree that they combine dev/qa/support because they're cheap. This actually makes little sense to me because typically QA roles are paid less than traditional software developers. That being the case, it doesn't make sense to get rid of them and let your developers do that stuff if all you want to do is save money.
What I heard was that they used to have support staff but changed it because it wasn't working out. I wasn't there so I don't know if that actually happened, but I can see how it's both harder and slower to have someone who doesn't know the code base fix bugs on the spot.
Honestly I can see it both ways. I think there are a lot of benefits to having the developers write tests themselves. On the other hand, when a project gets big enough, I can see how it makes sense to have people only working on tools like automated test frameworks or build stuff.
Except you don't do that, you fight operational fires and other such bullshit and things don't get fixed.
Their broken oncall system is one of the reasons why I don't want to go back to Amazon.
automation doesn't cover everything, you need baseline automation testing as well as more focused manual testing.
I've worked with many a QA who would get bent up over a detail outside of the spec that didn't really matter, and where all QA testing was manual.
Coders (good ones) are well equipped to automate processes, and to do so quickly, and this extends to integration testing.
This is where you need management (or someone from the product side) who can set priorities, where needed, and put and end to pointless side-disputes that can and do crop up.
Both the productivity and the quality were higher in the places with fully automated testing. Which is not shocking at all: does anybody really think a human can run through 800 test cases better than a computer can?
It's not a magic way to save money -- the developers obviously end up spending time writing tests. But the long-term value of those tests is cumulative, whereas the effort spent on manual testing is spent anew every release.
Manual review is still good for noticing things that "feel wrong" or for helping think up new corner cases. But those bleed into product owner & design concerns, and aren't really a separate function.
No thanks. It's a bunch of deflection and diffusion of responsibility coupled with high latency flakey interactions between different teams. Everything that can slip through the cracks does slip through the cracks.
I'm sure QA can be done well, but I am convinced that giving your devs a pass to not finish their work is a dead end in several dimensions.
In practise, a manual QA team encourages Devs to throw shit over the wall and expect someone else to do some basic sanity checks they should have already done. By the time those are done, what the QA team theoretically could find gets shipped.
Then, when its discovered in Prod, the QA team will get in the way of a speedy fix.
This keeps the devs incentivized to make sure everything works before the code goes to QA, and it keeps everyone incentivized to eliminate as many bugs as possible before release.
Surely everyone should report into the business (end-user on the business side) lead responsible for making this functionality happen. This includes Dev, QA, business. QA should be business? This is no longer 1985, there is no reason business (product owner) shouldn't be a capable PM and BA for their own product.
On any given project, we had a Product Manager who set the business requirements, a Project Manager who ran the project, a Dev lead in charge of the development team, and a QA lead in charge of QA. None of these people could override the others because of the reporting arrangement, so we had to collaborate instead to make sure we all met our goals.
Early on this arrangement worked really well. But it started falling apart when ownership of the company changed and politics became a factor as people who cared more about moving up through the organization than about the success of the company and quality of the products were hired and promoted. My take is that bad management will destroy any approach you might take to producing good software. That's why I eventually left.
For an organization with a single product (and, particularly, one small enough that the "top-level" QA and Dev manager are also first level managers), this makes some sense. Otherwise, this means that each product has no common level of technical management below the CTO, meaning it has no meaningful technical ownership.
If you give an engineer a career incentive to optimize something, you'd be surprised how seriously some will take it.
When a site gives a 500 because a database went down and the web app couldn't connect to it... is that a bug, a missed test case or should ops take a hit on down time? Furthermore, if you argue that the dev team should have reasonable failsafes in code to connect to a db, in a 10 year old organization, should the current Dev team pay for something that could have been in the code for years?
If you set up a system where devs hand off to QA then they hand off to ops to deploy. Furthermore every step is incentivised somewhat against each other. Even if all teams are equal, it ends up in my opinion, to the path to CYA and Waterfall. Everyone is more concerned with problems not being 'their' fault than shipping good code.
An incentive/penalty system for bugs without an incentive/penalty system for features completed leads to paralysis. And a complicated incentive system leads to game playing over productivity.
It's an old saying: be careful what you measure because you'll get a lot more of it.
Where this starts to break apart is that QA can get overzealous and file bugs for different incarnations of the same bug.
It's a bit of give and take.
Having a top-level manager who is responsible for a whole discipline doesn't work, it just creates a high wall to throw shit over. Developers and QA testers really should report to the same first line manager.
this sounds absolutely horrible. What a great way to remove all agency and ownership from an organization and incentivize people to pass the buck off to literally anyone but themselves and to work on easy, cupcake features with no risk.
>This keeps the devs incentivized to make sure everything works before the code goes to QA, and it keeps everyone incentivized to eliminate as many bugs as possible before release.
What kind of organization are you working in where you think devs have to be motivated by punishment or reward? Do you not hire the kind of people who are intrinsically motivated and want to produce software that doesn't have bugs? Do you think that if you don't explicitly incentivize developers not to write bugs that they would just fuck around and make shitty, buggy software? I just can't imagine working in an organization so dysfunctional that leadership thinks the default state is being happy with shitty buggy software.
This is because there is likely to be a tradeoff between speedy development and bug count.
If you are shipping physical CDs of software, or releases which you can't update easily, or if you are working in a critical environment where mistakes are disastrous (finance, health, space, etc.), then it is fair enough to be so concerned about bugs.
But many if not most developers work on the web in areas where the product is uncertain and evolving. Prioritising rate of change is more important here. Rather than extensive QA, monitoring and fast rollback are better choices.
My experience with the formal process I've been talking about was in a company whose product is a large-scale web application that interacted with a complex custom back end search engine and a proprietary content database. The QA and Dev teams had a friendly adversarial relationship, but ultimately we all felt we were on the same team with the same goals: to produce the best software we could. It was a fantastic place to work.
If this is in place, adding extra QA can be beneficial.
None of this of course is excuse for a dev not producing code that works at least in its happy path plus/minus a few of the most obvious exceptions/error paths...
> Software engineers at Yahoo are no longer permitted to hand off their completed code to another team for cross checking.
Dev Team A was giving a batch of code to Dev Team B to review.
I agree that the business user/product owner/whatever should be reviewing everything in test prior to approval and in production after but I'm not sure that's the article means by QA.
Devs don't think like users. They know a lot about computers and can make good guesses about how other devs think. So devs are not the best people to test code that's going to be put in front of non-dev users.
Users are likely to have different models, and a different set of expectations. You can't write tests for all those possibilities because you literally have no idea what they are - and won't find out until you put them in front of users.
Maybe in better companies than I worked it that isn't the case, but devs shouldn't be your 24/7 support staff.
My phone has never been so loud.
I think this rather misses the point; it's the bug that doesn't have a test case where QA helps.
He found bugs in the specs, gaps in the specs and just plain untested behaviour.
QA is a job that requires skill, benefits greatly from technical understanding and requires a lot of domain knowledge.
That being said, if you have to pick one or the other then you go with automated tests.
Of course not. But automated tests are just the entry ticket to the release process.
A computer can't replicate the irrationality, laziness and guile of a human end-user. That's what good QA engineers test.
Similarly going full screen on a 2 monitor setup broke once. Again, no automated test.
The Web Text to Speech API is broken on every browser it's in. It will work with a simple sample but start and stop it a few times and it will break. I suspect because again there is no easy way to test it.
There's lots of others.
automated testing is not a substitute for QA IMO.
There is still QA, it's just automated QA. Welcome to the 21st century.
Where I work, devs do the QA, and most of the devops work as well. It's the new reality, and anyone who thinks otherwise will be obsoleted.
Where have you worked? Just inside the SV bubble? This is definitely not true.
But then there were the other QA teams. The people that would just reject your stuff outright if it didn't have tests (no matter if it worked) and when the tests passed they would look at things truly from a customer perspective. They would ask really uncomfortable questions, not just to developers, but to designers and business alike. They had a mindset that was different from those creating things; they were the devil's advocate. These people did much, much more good than harm, and they are few and far between. Unfortunately, while I believe they were incredibly valuable, business thought otherwise when cuts came around..
I recently became a software tester, and I really didn't understand the role for quite a while. Is my primary responsibility finding bugs? Logging defects? Analysing requirements documents? Writing test scripts? Writing Status reports?
Answer: Do enough of each to fulfill your goal of gathering and sharing information with your management and dev groups.
If the software tester has problems testing, then the customer will have problems using it, and the company will have problems supporting it.
1. Kaner, Cem; James Bach; and Bret Pettichord. 2001. Lessons Learned in Software Testing. Wiley.
When you have software developers skilled in quality assurance who have the job of finding the edge cases and producing comprehensive additional acceptance and functional testing, they're an asset. It's a particular perverse mindset that I personally really enjoy interacting with as a software developer 'customer' - those evil bastards find the best bugs, regression test them, and expand to find all of that class of error in the application.
Then you do a code review and make sure that the reviewer examines the code for missed cases.
And in that system, the developer is completely removed from the product and is just another factory worker. The closer engineers can be to users (with design to translate obviously) the better for everyone.
The article makes the assumption that QA == manual QA which as a quality professional is false. Quality is about measuring risk across the development process. Immature team need manual QA while mature (in a process/quality sense) teams need much less (or none).
Quality professionals who want a sustained career needs to learn development processes, opts, documentation & monitoring. We make teams better.