Not sure if it'll work on an Arduino, but maybe take a look at https://github.com/ggerganov/whisper.cpp -- it works on a Raspberry Pi at least, so resource requirements are fairly minimal
Sorry, it looks like the modal background overlaps the logo so it only works once you've loaded the model -- here's a direct link: https://github.com/vishnumenon/ermine-ai
Doesn't work on Firefox Linux for me either ( error is Uncaught (in promise) DOMException: AudioContext.createMediaStreamSource: Connecting AudioNodes from AudioContexts with different sample-rate is currently not supported. index-0dae94e71b526640.js:1 )
Yup! Love the whisper.cpp (and llama.cpp) projects, they're incredible -- I mostly ended up using transformers.js for this just because the JS interface is a bit more straightforward/well-documented and its the same underlying model
Very nice. Would it be easy to add other languages than English?
Also, as others have notes, I had to open it in Chrome to make it work, Firefox didn't work.
In theory, yes -- but tbh in practice, I tried using the multilingual version of whisper-tiny and saw significantly worse performance, so that's why it's english-only at the moment. Possible that there's some sort of parameter/tuning that could improve performance though, I'm looking in to it!
I'm after something that can transcribe medical notes and unfortunately it does not work well for that case. (almost nothing does though) There's quite a few people interested in something that doesn't turn "laparoscopic" into "leper as cop it".
Maybe the current progress will help though. Models adjusted by your own dictionary or from postprocessing fixes would be amazing.
I've been using Whisper Memos for some time now. Simple but useful app to quickly save an idea or memory when you don't have time to type. Speech recognition is much better than native one, especially with languages which are not widely supported.
This app has been my go-to solution for efficiently recording thoughts or memories without the need to spend time typing. Its proficiency in speech recognition is notably outstanding.
We are using Nuance Dragon Medical at work which is intuitive to use and surprisingly accurate even with very fast dictation. I have yet to come across a solution, that is as accurate, although I'm not sure if they offer solutions for end users directly.
I had an application for radiology and whisper large-v2 with beam size five was essentially 100% across multiple different types of dictated radiology reports.
I work at Rev.AI, we have a model tuned for medical and we are HIPAA compliant across the board. We do human and AI transcription and our ASR is #1 accuracy in the world right now
I took at a look at the Rev.AI website and didn't see any mention of a medical-specific model nor HIPAA compliance. It would be nice if this information was presented in your marketing!
When I try this on Firefox on Linux (not incognito) I get the following error:
Connecting AudioNodes from AudioContexts with different sample-rate is currently not supported. index-0dae94e71b526640.js:1:2992
Uncaught (in promise) DOMException: AudioContext.createMediaStreamSource: Connecting AudioNodes from AudioContexts with different sample-rate is currently not supported. index-0dae94e71b526640.js:1
Media resource blob:https://www.ermine.ai/e762a6f1-f292-4b23-96e0-8059a7f9d635 could not be decoded. www.ermine.ai
Media resource blob:https://www.ermine.ai/e762a6f1-f292-4b23-96e0-8059a7f9d635 could not be decoded, error: Error Code: NS_ERROR_DOM_MEDIA_METADATA_ERR (0x806e0006)
(also, the weights json doesn't download at all in Firefox incognito).
Would be good if you could pop some kind of alert (literally alert() might do the trick) on an exception just so people don't wait for a couple of minutes before realizing something's gone wrong :)
I also get this error. Unfortunately nowadays most developers forget about Firefox, this includes big sites such as Microsoft Azure Console not working under Firefox and Google Cloud Console being slow on Firefox.
Thanks for the detailed error report! I was under the impression that it worked in FF, but I guess there's some additional testing needed -- I'll definitely look into it!
Unfortunately not at the moment, but that feature (speaker diarization) is something I'm looking in to! AFAIK the Whisper model (which this uses currently) doesn't support that functionality, but I'm experimenting with adding an auxiliary model just for diarization.
This is pretty cool! Just yesterday I finished a similar demo, also using transformers.js. I am currently in the process of adding real-time transcription, do you plan on adding that?
Thanks! I actually initially had it doing transcription in 5- and 10-second chunks for close-to-realtime results, but the CPU usage on my laptop (which admittedly doesn't have the best specs) was a bit higher than i wanted. 30-second blocks gave me the best balance of semi-real-time and good performance, especially since the whisper model is built for 30-second chunks. If you get real-time working smoothly though, i'd love to take a look!
70 comments
[ 2.7 ms ] story [ 133 ms ] threadIs there a version of this we can run on ESP32 (arduino) devices?
Not sure if it'll work on an Arduino, but maybe take a look at https://github.com/ggerganov/whisper.cpp -- it works on a Raspberry Pi at least, so resource requirements are fairly minimal
https://github.com/espressif/esp-skainet
https://github.com/tomzbj/dhrystone_score
Normally I expect a lot of things that push the limit to not work on iOS, but this one did!
Are you aware that whisper.cpp has a WASM-version as well? See https://github.com/ggerganov/whisper.cpp/tree/master/example... - demo at https://whisper.ggerganov.com/
I think so, yes. The website uses whisper-tiny.en, so I suppose all they need to do is switch to whisper-tiny instead.
Maybe the current progress will help though. Models adjusted by your own dictionary or from postprocessing fixes would be amazing.
Laparoscopic - that worked fine ;)
Edit: if you have one available, mind sending over a sample deidentified note to my email in profile? I’m working on something.
No, there's no Apple hardware available in my scenario.
Also in medical context, if I can't tell where the data goes, the solution is not usable.
Here’s an iOS app to play with it: https://whispermemos.com
It even formats recording as paragraphs by running through GPT.
> Built using transformers.js and the whisper-tiny.en model.
Would be good if you could pop some kind of alert (literally alert() might do the trick) on an exception just so people don't wait for a couple of minutes before realizing something's gone wrong :)
https://bigwav.app
I will have a look at the repository to find out, but maybe someone already looked into it.
SPEAKER A: blah blah
SPEAKER B: blah blah
So it can be used for transcribing phone calls?
The little "replay your audio capture" <audio> HTML element says "error" but the transcription actually worked.
Example: 20230406115923.mp3 ==> 20230406115923.txt 20230406083110.m4a ==> 20230406083110.txt
I wish someone would build one and sell it for $10 a copy.