Ask HN: Are your code benchmarks statistically valid?

1 points by DenisM ↗ HN
It appears that some benchmarking tools provide average or percentile values but no information about statistical validity of the result. What did you observe or do in your career?

3 comments

[ 4.2 ms ] story [ 21.5 ms ] thread
I haven't got a clue since I have never used any benchmark tool in my whole life. Could you tell us in which professional context you would use that kind of tool?
Run a benchmark to compare performance of my product before and after making a change in code.
I run it 5 times and drop the two extremes. If the middle tree don't overlap, I'm probably fine. Not super precise, but good enough and easy.

Also, add a loop so the benchmark is like 1 second long, I'm old enough to be worried about anything smaller than 1/18 seconds.