HN Explorer

Show HN: We built a tool for fast-forwarding 95% of tests (MIT) (github.com)

8 points by _cfl0 ↗ HN
Hi, we are working on a tool for speeding up test runs, by skipping tests unaffected by code changes.

Effectivly, Saving 80-95% of the time, by skipping 80-95% of tests.

We started a few months ago, and have managed to get into a few production CI systems. All our prospects and users are on holiday right now. So we decided to repackage and open-source for local test running. available here (https://github.com/nabaz-io/nabaz) under MIT license.

One line change: pytest -v -> nabaz test --cmdline "pytest -v"

Stalk us on GitHub, or just Star us. Ask questions, we'll answer in under 30 seconds. we have auto refresh on.

17 comments

[ 4.5 ms ] story [ 60.7 ms ] thread
metadat
Cool idea.. Has this strategy already been tried before?
_cfl0
Meta and Google employ AI based methods for skipping tests (See):

they are pretty much flawed in a way, because they require you to wait weeks/months to gain statistical sagnificance. And even then they are too costly to train to make sense in smaller scales.

There's an ex-Google employee who came out with something similar in 2005 for JUnit java tests, since erased it was called google-testar.

We talked to him and ulitmatly he sunsetted the project because:

1) static analysis back then was basically copying language parser code from the Language distro and retro fitting it which was HARD to maintain.

2) Misses when resource files were changed, he was a one man show to my understanding.

Our View: Static analysis has gone a long way since... and yet no one has made something for system tests and integration tests, there's a couple unit tests solution out there.

See: https://static.googleusercontent.com/media/research.google.c...

https://research.facebook.com/publications/predictive-test-s...

_cfl0
How it works: First time you run nabaz:

1. Collects code coverage for every tests that runs, seperatly.

2. Saves code coverage in local sqlite db, along with commit ID as key.

Second to n runs later:

1. Traverse git tree and queries db via commit ID -> per test code coverage

2.a git diff between new code (code change) and the code coverage from the old commit, test case by test case.

2.b. If an intersection between new code and old code coverage is found, test is rerun. else, it will be skipped.

maxilevi
How do you deal with tests that interact with something external to the codebase? The result of those could potentially change without a code change.
_cfl0
That's a great point!

Depends on the what you define by "externality".

I like to look at it this way, what's being tested? I see three groups:

  - testing the software you wrote.
  - testing integration with other software you wrote (microservice interaction maybe?)
  - testing integration to third party software.
I'm telling you our current plan, feel free to poke holes:

In the first case the only externalities that seem to really matter are mostly resource and configuration files. We could detect I/O with such files and rerun the test if the underlying resource/config changed.

In the second case were planning on collecting coverage for all services being tested and then employ a more sophisticated diff heurastic.

For the third case mark those tests as always run, hopefully you are testing your codebase and not just thirdparty integrations.

We have a poc of a syscall tracer that gives us an I/O map of the test being run or the software being tested. this includes reading/writing to files, pipes, network etc...

metadat
This sounds like the best you could possibly do!

What are the odds this baby will be made compatible with Java?

_cfl0
java is scheduled next, would absolutely love to hear about your use-case?
metadat
I have a pretty large Java code base (40+ micro services) and the integration test suite takes hours, many hours, to run.

However they are usually testing components from end-to-end. This seems not very amenable to benefitting this approach, since "side effects" are an inherent part of nearly every integration test.

_cfl0
Wow! Seems like a challenge to test, would love to see what we can do.

Regarding side effects, completely fine as long as they're mapped out. Would love to hear more, nothing pitchy or salesy.

bary12
This looks interesting and definitely something missing from our CI pipeline. What languages do you intend to support?
_cfl0
Basically all of them as long as their used.

Would love to know more about your use case, email is somewhere in the comments

Spongego
Sounds super cool! I’ll definitely show this to our devops guy in the company. However sounds like there might be a lot of edge cases that don’t include any code changes, e.g. some environment variable changes - have you thought about it?
Roeylalazar
Hey, awsome! feel free to contact us at hello@nabaz.io.

About the env vars: yes. We have a list of edge cases that might affect the tests. Env vars specifically is the next edge case we are about to cover.

Roey from nabaz.

_cfl0
THIS point though, Im inclined to share our edge case doc with HN and let people add stuff.
_cfl0
Thanks guys for all your feedback!

We took all notes to heart, the main problem we identified that currently (at this stage of the product cycle we don't yet support all edge cases)

And this hurts usability and value, will figure out a way to deliver this kind of tech in a way that is still valuable at this early stage :)

adamantinecog
I'm wondering if this works similar to https://github.com/digitalocean/gta which will do transitive analysis to find packages whose dependencies have changed and only run the tests for those packages?
_cfl0
We love gta! A few key differences though: 1. We analyze the effect of code changed on code coverage, which means we only compare changed code to runtime information (code that actually ran). gta analyzes statically.

Let's assume you have a function:

  func myFunction(a int) {

      if(a > 10) {  

          func1()  

      } else if (a < 4) {  

          func2()  

      } else {  

          func3()

      }  
  }
let's assume you changed func3 only, or func2, but during the run of the test only func1 ran.

The test will be skipped because no change could of affected it's result. NOT in gta.