HN Explorer

Ask HN: Why is the gaming industry resistant to automated tests?

3 points by sovietmudkipz ↗ HN
I work at bigcorp working on applications built on top of kubernetes/public cloud as the target platform. One thing I’ve been impressed by at work is how writing automated tests is paramount. The “Growing Software guided by Tests,” “Unit Testing: principles and practices,” and other media/books are recommended by engineers to help others get up to speed writing good tests and good software. What is a good test, and what is a waste of time to test? This is a constant discussion and I love it.

As a hobby, I write video games because why not. One thing I’ve noticed is how different attitudes are on automated testing. It’s highly discouraged because ‘games don’t lend themselves to being tested’ apparently. It’s just as amazing to see how resistant this slice of the overall IT industry is to something the rest of the industry has considered very good.

This culminates for me in two anecdotes.

First, trying to find out what a good test looks like for video games is very, very, very challenging. There aren’t content creators noodling on or presenting findings on this subject. Not many articles, not many videos, not much content describing good tests. If there are industry veterans who do testing right, they seem to be keeping that information far, far away from the public. Perhaps even they see this as a market differentiator? My experience leads me to think this is something very few explore, and even fewer share.

Second: Unity put together a thing in Fall of 2020 called “open game project” where the idea was to create an open source game as a community. The game, now called “chop chop,” has a series of YouTube livestreams where the Unity devs discuss the project. The super interesting thing to me is that within 72 hours of project start, someone put together a PR to try and introduce tests. A Unity forum thread was referenced where discussion occurred. When I read through the discussion, I got the sense that the Unity devs did not want tests in the project. “Too early,” “too complex,” and other blockers existed at that time, apparently. Fast forward to the project end— still no tests.

Why is the gaming industry so resistant to automated tests, especially when compared with the overall IT industry?

12 comments

[ 1.4 ms ] story [ 35.8 ms ] thread
Flankk
Video games are disposable products, other software is usually mission critical and requires long-term support. So for games you can get away without automation but with other software it would destroy you. Testing is not free, it is a tradeoff where the benefits outweigh the costs by some margin. Video game testing is difficult to automate so in many circumstances the costs of automation may outweigh the benefits. Automated tests are just a subset of quality assurance, which can be accomplished in a variety of ways.
sovietmudkipz
I appreciate the response! I’m not as satisfied with this answer. I’m not satisfied because a legion of QA engage in manual testing to ensure quality in game artifacts in the games industry. It’s like the process is begging for automation, at least that’s how I interpret the situation.

Testing is challenging; as an industry we have learned that challenging things are a signal to do it more, not less.

detaro
Just because something is "begging for automation" means automating it is easy or even just viable. You'll always need human QA, so at this point the question is merely "how can tests reduce the load for manual QA while keeping manual QA good at the parts we can't automate". Game testing is a lot like UI testing in other apps that way.

And then in reverse, are the automatic tests worth the maintenance effort if a human is going to try anyways and notice quickly? IMHO that's the part where many people underestimate them, but getting the balance right is still hard.

EDIT: of course its biased, but its interesting to think through bugs you encounter in games and how you'd catch them with tests. I feel often its more on the level of fuzzing than traditional tests that could catch these things, or it is visual issues that are very hard to automatically find.

matt_s
I know there are elements of a game engine and user interaction that would be hard to test but there are a lot of areas internal to a large online game could use automated tests. These could be actual REST types of API's that the game clients interact with as well as internal API's either in the client or server.

There should be a large enough surface area of these API's to test things and I think that would make changing/tuning aspects of games easier on the game developers.

I wonder if this is more of a culture within the gaming industry where getting humans to test games is cheaper (i.e. 15-20/hr) than hiring QA engineers to write code that tests things. I remember hearing anecdotes of people wanting to work for game companies and being a tester is a "foot in the door". Also sometimes they do open beta's where they just let players in and do the testing.

sovietmudkipz
I have a similar hypothesis; I speculate that the reason automated testing isn’t a big thing in the gaming industry is because there are legions of humans who are willing to manually test the software.

I’m glad to see that you have identified areas of game software that wouldn’t be useful to test, and other areas where it would be. Excellent to see because it does highlight that there ARE places where automated tests can provide huge value.

salawat
To really do QA for a gaming company, in a way you could leverage automation effectively, the key would be process, and structuring things in a way where you are using design patterns that let your tools cross-cut the portfolio of product.

You could totally automate tests around the web service client layers of the game, but what most people think of in gaming QA is actually playing the game. That would slot in as a System test, which is your most expensive, least scalable, latest in the process form of testing, and nigh impossible to make substantive groundreshaping changes from the input of, as that experience is reliant on everything being the way it is.

If you were embedded with the engine design group, you might be able to get some wins in terms of influencing architecture choices so as to make downstream automation testing more likely/doable, but from my understanding that's like being on the team that develops the crypto libraries everyone else just adopts and uses in the manner you prescribe.

Plus, as a longtime QA, if game developers are as resistant to feedback as corpo ones are, and with management structures almost guaranteeing you'll be underneath the Engineering group, cutting you off at the knees the first time the words "crunch time" are implied, given the rates most get in that vertical, I'd stay away from it unless your one purpose in life is to get confronted with disappointment after disappointment as economic/fiscal concerns destroy and deface what could have been over and over again.

Then again, you wouldn't be in QA if you didn't already have a tolerance to that sort of thing.

Now where'd I put that bottle?

burntoutfire
The "disposable" aspect of games mentioned by parent is crucial.

Any successful business software is going to be maintained and extended for years, possibly decades. This will be done by constantly churning team, where sooner or later no one will have a very good idea about the system's internal or requirements. In such cases, automated test cases protect against regression bugs AND also serve as a documentation of sorts. Whereas in games, after the game is released, the team will maybe patch it for a while and after that, chances are no one will ever touch that code ever again.

PaulHoule
I thought C/C++ programmers never got into the testing habit. In fact, with the dangerous memory model probably the only safe thing to do if a test fails is exit() in the child and fork() a new one in the parent… If you are in POSIX country!
sovietmudkipz
This is an interesting point. In the early days of my own journey into writing tests, I did take note that much of the content I was consuming was created by the Ruby community. In fact, one of my favorite speakers to listen to Sandi Metz comes from the Ruby community.

The community adoption of practices does seem key. If there is a culture in C/C++ where tests aren’t really a practice, that would partially explain the phenomena I observe.

detaro
> something the rest of the industry has considered very good.

That's a ... very optimistic view of the rest of the industry. You see attitudes similar to those you cite in many places.

That said, testing for games is a very interesting topic indeed. From what I gather, it's easy to build tests that are not very effective, and a lot of work to test non-trivial things. Combine that with the kind of iteration and likely less fixed specs in gaming makes many people think it's not worth it.

2rsf
> devs did not want tests in the project. “Too early,” “too complex,”

I get that response a lot from general purpose devs, not just gaming devs many still don't see the benefit of tests at all levels

muzani
I think it's because by default, you have no idea what you're building for games.

Let's say you're making a typical RPG on web/JS. First off, what is the interface? Typical WASD? Is it a grid? 2D or 3D or 2.5D/isometric? How do you attack monsters? Do you click or press a key? Is it Mount and Blade style where the feel of the weapon depends on character skill? Is it FF style where you pick an attack from a menu?

Oh wait, we forgot mobile. WASD won't work nor would most roguelike controls. Do we just ditch mobile support? Do we hack in a D-Pad? Do we do two different interfaces? Would it be portrait supported or landscape to make it more compatible with monitors?

From this alone, it's very hard to do TDD, as opposed to writing some hotel booking app where you know that essentially someone books a hotel.

And then you have the next stage: exploits.

Sword kills the skeleton monster in 10 hits. Mace kills the skeleton monster in 5 hits. Spiked chain kills the skeleton in 13 hits but it has a little more range, so the hero takes less damage. You write tests and it checks out perfectly. You test that randomisation falls within the parameters, test that the soul stealing artifact sword is balanced, test that paladins are properly corrupted when using soul stealing weapons.

And then you get this: https://www.giantitp.com/comics/oots0216.html

Someone finds out that half-ogre spiked chain can defeat every monster in the game without a single injury. Suddenly half the game are using low level spiked chains and the soul stealing artifact sword is in the bargain bin.