How are changes determined I wonder. By comparing to master?
The article mentions test impact analyses going off of PR changes, but what about un-pushed local changes for when I just want to speed up local tests as I work in dev (apologies if this is a newb question).
My tool spdr allows you to select the ref to compare to, but the default "smart" mode uses two different behaviors, depending on whether it's run locally or in CI.
If it's local, it compares all changes against HEAD (both staged and unstaged, but you can change that).
If there's no uncommitted changes, like for a fresh CI checkout, it assumes it's in CI, so it should compare to the base of the PR. (For the moment, spdr just diffs with parents in CI, but I'm currently working on detecting the PR base ref. They all have different environment variables for indicating that.)
I think mozilla did this with firefox very successfully.
For many (perhaps most) projects I dont think there is much point.
It's A) easier to do stuff like parallelization to make running all the tests faster B) not necessarily worth it to bring default CI durations down from, say, 8 minutes to 4.
I haven't heard how Mozilla used it, but I know Google and MS use test impact analysis (among other techniques). Do you have a link? I'd like to see it.
TIA's definitely not needed for a lot of well-designed projects, though it's harmless to add if done right. I built spdr mostly for older projects that have CI taking 30+ minutes. When it's that slow, it's becomes a real drag on team velocity.
Parallelization is great, but it's not always drop-in. It requires thinking about. You can't turn it on, and do a smoke test, because parallel bugs might not manifest immediately.
No, I saw it briefly on hacker news once thats it. They have, IIRC, like, a 40 hour test suite or something so for them it made a massive difference.
The only time I got close to that (roughly 14 hours, many years ago) we just hacked it by running random tests constantly to detect regressions and only running the whole suite overnight. It brought release cycles down from bi-yearly to once a week or so so people were happy.
If your tests are fully isolated (and they should be for many reasons), parallelization ought to be almost drop in. You can set it up in github actions pretty easily.
If a feature that predicted which tests to run were drop in Id say it was a no brainer too, but there is no tooling for it yet so it would have to be built from scratch.
>If your tests are fully isolated (and they should be for many reasons), parallelization ought to be almost drop in. You can set it up in github actions pretty easily.
Good point. My experience is that super-long test runs also have dependencies that interfere with parallelization, but that might just be the projects I've worked on.
> If a feature that predicted which tests to run were drop in Id say it was a no brainer too, but there is no tooling for it yet so it would have to be built from scratch.
There is! I wrote one for Python (https://getspdr.dev)! If you just care about local runs, there's also the pytest-testmon plugin. Fos Js, both Vitest and Jest have settings to enable it. It's definitely out there!
8 comments
[ 2.9 ms ] story [ 36.4 ms ] threadMy tool spdr allows you to select the ref to compare to, but the default "smart" mode uses two different behaviors, depending on whether it's run locally or in CI.
If it's local, it compares all changes against HEAD (both staged and unstaged, but you can change that).
If there's no uncommitted changes, like for a fresh CI checkout, it assumes it's in CI, so it should compare to the base of the PR. (For the moment, spdr just diffs with parents in CI, but I'm currently working on detecting the PR base ref. They all have different environment variables for indicating that.)
If you want help, run `spdr --help`.
For many (perhaps most) projects I dont think there is much point.
It's A) easier to do stuff like parallelization to make running all the tests faster B) not necessarily worth it to bring default CI durations down from, say, 8 minutes to 4.
TIA's definitely not needed for a lot of well-designed projects, though it's harmless to add if done right. I built spdr mostly for older projects that have CI taking 30+ minutes. When it's that slow, it's becomes a real drag on team velocity.
Parallelization is great, but it's not always drop-in. It requires thinking about. You can't turn it on, and do a smoke test, because parallel bugs might not manifest immediately.
The only time I got close to that (roughly 14 hours, many years ago) we just hacked it by running random tests constantly to detect regressions and only running the whole suite overnight. It brought release cycles down from bi-yearly to once a week or so so people were happy.
If your tests are fully isolated (and they should be for many reasons), parallelization ought to be almost drop in. You can set it up in github actions pretty easily.
If a feature that predicted which tests to run were drop in Id say it was a no brainer too, but there is no tooling for it yet so it would have to be built from scratch.
Good point. My experience is that super-long test runs also have dependencies that interfere with parallelization, but that might just be the projects I've worked on.
> If a feature that predicted which tests to run were drop in Id say it was a no brainer too, but there is no tooling for it yet so it would have to be built from scratch.
There is! I wrote one for Python (https://getspdr.dev)! If you just care about local runs, there's also the pytest-testmon plugin. Fos Js, both Vitest and Jest have settings to enable it. It's definitely out there!
Do you parallelize your current tests? Why or why not? Do they work out well for you?