I tested Wispr on the same 525 dictation clips a month apart (May and again in June) and there's a clear regression: Word Error Rate rose from 9.0% to 11.2% (lower is better). It got worse in 8 of 9 categories I track.
Full results and the raw transcripts attached above.
Important context: this only measures a single, American‑accented voice on synthetic prompts. Wispr supports many languages and accents, and I’d assume their updates have boosted overall accuracy. But, they regressed for my use‑case: plain old boring, unaccented American English.
How I tested: I have a harness that drives a dictation app by playing an audio file through a virtual input device and captures the app’s pasted output. I ran this on 525 audio files through Wispr Flow a month ago. Yesterday, I ran the same clips through again. The clips are recordings of me whispering AI-generated sentences.
Results: There’s a clear regression on word accuracy. Words that were correct ~1 month ago are now incorrect. Phonetically, they’re similar, but just off.
May: "Brian, refactor this auth middleware to stop leaking tokens and debug logs." (near perfect)
June: "Ryan refactored his odd metalware to stop leaking tokens in deep bugs and backlogs." (garbled)
So no, you're not imagining it, it has gotten worse. At least if you speak plain, unaccented American English like me!
Disclosure: I work on a competitor which is why I had the May eval ready. Take this with a grain of salt and do your own testing!
1 comment
[ 1.9 ms ] story [ 10.2 ms ] threadFull results and the raw transcripts attached above.
Important context: this only measures a single, American‑accented voice on synthetic prompts. Wispr supports many languages and accents, and I’d assume their updates have boosted overall accuracy. But, they regressed for my use‑case: plain old boring, unaccented American English.
How I tested: I have a harness that drives a dictation app by playing an audio file through a virtual input device and captures the app’s pasted output. I ran this on 525 audio files through Wispr Flow a month ago. Yesterday, I ran the same clips through again. The clips are recordings of me whispering AI-generated sentences.
Results: There’s a clear regression on word accuracy. Words that were correct ~1 month ago are now incorrect. Phonetically, they’re similar, but just off.
May: "Brian, refactor this auth middleware to stop leaking tokens and debug logs." (near perfect)
June: "Ryan refactored his odd metalware to stop leaking tokens in deep bugs and backlogs." (garbled)
So no, you're not imagining it, it has gotten worse. At least if you speak plain, unaccented American English like me!
Disclosure: I work on a competitor which is why I had the May eval ready. Take this with a grain of salt and do your own testing!