What you're looking for is in Figure 3 (admittedly a bit hard to read because I had to squish it into the paper). Each panel in that figure shows the results for test suites of a fixed size. For example, the top left…
Paper author here. We measured "effectiveness" as the mutation score. We showed in a separate paper that the mutation score is a good way to measure a test suite's ability to detect real faults. Of course, using the…
I'd definitely say that the absence of coverage is a problem. My view is that coverage is necessary but not sufficient for good testing. We talked a bit about classifying tests but didn't do it in the end because it's…
Paper author here. As someone already mentioned, the suites were made by picking a random subset of the test cases written by the developers, not by using a suite generation tool. Now, our method is arguably not 100%…
Computing the mutation score using a randomly selected subset of the mutants seems to work well, though it hasn't been studied a lot. One paper is "Reducing the cost of mutation testing: An empirical study" by Wong and…
Paper author here. We did a follow up study and found that the mutation score is, in fact, correlated with the suite's ability to detect real faults, at least for the programs we studied. That paper is also on my…
Paper author here. Scoring coverage for every pass is an interesting idea and something I'd like to look into. I'm not sure it would change the result as much as you think, though. The basic finding of the paper is that…
What you're looking for is in Figure 3 (admittedly a bit hard to read because I had to squish it into the paper). Each panel in that figure shows the results for test suites of a fixed size. For example, the top left…
Paper author here. We measured "effectiveness" as the mutation score. We showed in a separate paper that the mutation score is a good way to measure a test suite's ability to detect real faults. Of course, using the…
I'd definitely say that the absence of coverage is a problem. My view is that coverage is necessary but not sufficient for good testing. We talked a bit about classifying tests but didn't do it in the end because it's…
Paper author here. As someone already mentioned, the suites were made by picking a random subset of the test cases written by the developers, not by using a suite generation tool. Now, our method is arguably not 100%…
Computing the mutation score using a randomly selected subset of the mutants seems to work well, though it hasn't been studied a lot. One paper is "Reducing the cost of mutation testing: An empirical study" by Wong and…
Paper author here. We did a follow up study and found that the mutation score is, in fact, correlated with the suite's ability to detect real faults, at least for the programs we studied. That paper is also on my…
Paper author here. Scoring coverage for every pass is an interesting idea and something I'd like to look into. I'm not sure it would change the result as much as you think, though. The basic finding of the paper is that…