This is really awesome. I hope that that at some point the WASM Whisper can be hooked up to WASM ffmpeg and let me drop audio files of any kind into that site.
I’ve been playing around with various iOS apps today.
Someone else mentioned “Hello Transcribe”, which is a cool demo due to how real time the transcription is, but it’s effectively unusable for anything practical because it seems to split the audio on 30 second chunks (and not in the contextual way that Whisper normally does it).
Whisperboard is a free, open source iOS app (available on the App Store) that seems fairly user friendly, but there are a few crashes and bugs that the author is hopefully going to work out soon.
Aiko is extremely simple to use, but it only supports the “medium” model, so it isn’t very fast, but the results are good quality. Aiko is really designed to be paired with another app, like the Apple Voice Memos app, where you can share a voice memo directly to Aiko.
It supports wav, mp3, m4a, and mp4. All the demuxers were written in JavaScript which was an interesting exercise. I couldn’t figure out a way using native JavaScript audio APIs to load small sections of long audio/video files without loading the entire file.
Full circle eh. I wonder how well it compares to just trying to use the actual Whisper models on a variety of existing GPU capable bigger frameworks.
I don't know much practically about how hard it would be to take the Whisper PyTorch (1 or 2?) trained models & to make good use of them elsewhere. I expect Whisper.cpp probably better caters to users, is more readily consumable. Doing the same integer quantization that 1.4.0 whisper.cpp does would be another ask: how hard would that be?
>> I don't know much practically about how hard it would be to take the Whisper PyTorch (1 or 2?) trained models & to make good use of them elsewhere.
What do you mean by "elsewhere"?
The whisper models can be used without much effort. The tooling provided by OpenAI allows for them to be used and installed in python with ease. People use them in companies to transcribe internal media datasets. No need for CPP implementation for internal uses.
Main problem is when _on device_ is required and the OpenAI model is not quantised and so use on say a smartphone becomes trickier.
Not soon enough, the iOS transcription is terrible compared to even the small whisper model. I really hope Apple revamps voice recognition this fall, but I'm not putting money on it being Whisper quality.
It’s an implementation of a speak to text software. Currently the best open source one, if not the best.
Particularly you can talk to it with a “international” English accent and it will work. Or French or Spanish or Norwegian… and it will transcribe English.
What’s the best-in-class Whisper implementation for real-time / streaming transcription? I’ve followed the various posts linked on this GitHub issue [1]; not sure if there’s more out there.
Is there a static library implementation with easy to use interface?
Also think llama.cpp would benefit from having a lib release. It would be great to easily integrate both projects in existing c++ projects.
These new models give me hope that computers will be able to understand me soon.
As a Flemish/Dutch speaker, not a single assistant is able to understand me in my native tongue.
I can use Siri in English, but then she doesn't understand things like contacts or addresses. Also I cannot dictate text messages or people think I'm a spammer.
45 comments
[ 8.7 ms ] story [ 169 ms ] threadhttps://whisper.ggerganov.com/
FYI it seems to work on mobile sometimes but sometimes not at all. You might want to add a warning about when it might work or not work on mobile.
Any plans to make the source available for the client side part?
Thanks
Mobile is an issue, some of the web APIs only work on desktop for now.
Thanks for the feedback
does this count?
Someone else mentioned “Hello Transcribe”, which is a cool demo due to how real time the transcription is, but it’s effectively unusable for anything practical because it seems to split the audio on 30 second chunks (and not in the contextual way that Whisper normally does it).
Whisperboard is a free, open source iOS app (available on the App Store) that seems fairly user friendly, but there are a few crashes and bugs that the author is hopefully going to work out soon.
Aiko is extremely simple to use, but it only supports the “medium” model, so it isn’t very fast, but the results are good quality. Aiko is really designed to be paired with another app, like the Apple Voice Memos app, where you can share a voice memo directly to Aiko.
a macos menu bar app that uses whisper.cpp under the hood.
Uses OpenAI API so really fast
It supports wav, mp3, m4a, and mp4. All the demuxers were written in JavaScript which was an interesting exercise. I couldn’t figure out a way using native JavaScript audio APIs to load small sections of long audio/video files without loading the entire file.
I don't know much practically about how hard it would be to take the Whisper PyTorch (1 or 2?) trained models & to make good use of them elsewhere. I expect Whisper.cpp probably better caters to users, is more readily consumable. Doing the same integer quantization that 1.4.0 whisper.cpp does would be another ask: how hard would that be?
Fwiw, Whisper.cpp uses Nvidia's cuBLAS. There does appear to be an AMD rocm port. https://github.com/ROCmSoftwarePlatform/rocBLAS
What do you mean by "elsewhere"?
The whisper models can be used without much effort. The tooling provided by OpenAI allows for them to be used and installed in python with ease. People use them in companies to transcribe internal media datasets. No need for CPP implementation for internal uses.
Main problem is when _on device_ is required and the OpenAI model is not quantised and so use on say a smartphone becomes trickier.
Until then... https://superwhisper.com
Check out on https://twitter.com/superwhisperapp
Particularly you can talk to it with a “international” English accent and it will work. Or French or Spanish or Norwegian… and it will transcribe English.
"Adding --task translate will translate the speech into English."
I tried translating a conference with a German speaker. The transcription was superb, but the translation no so much.
[0] https://github.com/openai/whisper
[1] https://github.com/openai/whisper/discussions/2
https://github.com/davabase/whisper_real_time
https://github.com/abb128/LiveCaptions
https://github.com/usefulsensors/openai-whisper
https://github.com/ggerganov/whisper.cpp/releases/tag/v1.4.1
As a Flemish/Dutch speaker, not a single assistant is able to understand me in my native tongue. I can use Siri in English, but then she doesn't understand things like contacts or addresses. Also I cannot dictate text messages or people think I'm a spammer.