This open source project provides a self-hostable API for speech to text transcription using a finetuned Whisper ASR model. The API allows you to easily convert audio files to text through HTTP requests. Ideal for adding speech recognition capabilities to your applications.
Key features:
- Uses a finetuned Whisper model for accurate speech recognition
- Simple HTTP API for audio file transcription
- User level access with API keys for managing usage
- Self-hostable code for your own speech transcription service
- Quantized model optimization for fast and efficient inference
- Open source implementation for customization and transparency
Are you able to provide more information on the fine tuning? Any improvement in WER and what language it was fine tuned in and the size of the dataset used?
I guess I should have phrased it better: If you were building an app, does Android provide the ability to listen to arbitrary speech and convert it into text for your app?
It is completely self-hosted, but it currently supports only the tiny and base models. You can soon expect support for large models. For any requests, you can create an issue.
No, it is not a third-party. It is a just PostgreSQL database for logging everything. You can simply visit the /docs endpoint. It is just for authentication so that you can work with different users.
One Again its completely self hosted
Many of you are asking if the project is completely self-hosted and does not rely on any third-party services. Yes, it is completely self-hosted and does not rely on any third-party services. The user is for authentication, so no one can use the service without authentication.
Getting an authentication token does rely on a third-party service, if the README instructions are correct. It requires sending an email address to that third party.
getToken is just an authentication layer for authenticating your request. If you want to self-host it, just clone the repo and please check the .env.example file.
I think people are confused by the README showing localhost in some places and your instance url in others, causing them to assume you were centrally issuing tokens for using this project.
It is surprisingly good!
Not clear if you can go without a manual step where you need to change the likes of SPEAKER_00 to Bob and SPEAKER_01 to Sarah, but I've not had it mess up on me at all transcribing 2 hour long conversations between 6 people.
So is "real time" translation a thing yet? I've long wanted to be able watch non-english television and have the audio translated into English subtitles. It's doable for pre-recorded things, but not for live.
An iPhone app that could do this from the microphone would also be amazing. Google Translate and it's various competitors from Microsoft/Apple are nearly there, but they all stop listening inbetween sentences. Something that just listened constantly, printing translated text onto the screen, would be amazing.
Just wait for a couple of weeks. I am working on speech-to-speech translation. Instead of subtitles, you can listen to it directly. I am also working on subtitles.
Many live streamers, and platforms, would love to have custom real-time transcription elements. I actually looked into this exact project of yours when I thought about creating such a thing.
Even if it meant delaying the broadcast for a second while transcribing the accessibility value could be immense.
For anyone confused about the project, it is using whisper.cpp, a C-based runner and translation of the open whisper model from OpenAI. It is built by the team behind GGML and llama.cpp. https://github.com/ggerganov
You can fork this code, run it on your own server, and hit the API. The server itself will use FFmpeg to convert the audio file into the required format and run the C translation of the whisper model against the file.
By doing this you can separate yourself from the requirement of paying the fee that OpenAI charges for their Whisper service and fully own your translations. The models that the author has supplied here are rather small but should run decent on a CPU. If you want to go to larger model sizes you would likely need to change the compilation options and use a server with a GPU.
Similar to this project, my product https://superwhisper.com is using these whisper.cpp models to provide really good Dictation on macOS.
Its runs really fast on the M series chips. Most of this message was dictated using superwhisper.
Congrats to the author of this project. Seems like a useful implementation of the whisper.cpp project.
I wonder if they would accept it upstream in the examples.
I haven’t used faster-whisper so I can’t compare performance, but whisper.cpp does support cuda via CUBLAS, and it’s noticeably faster than the cpu version. I used it earlier this year to generate subtitles for 6 seasons of an old tv show I backed up from dvd that didn’t include subtitles on the disc.
Fwiw decent acceleration works on any avx2 compatible chipset. I get realtime speed for everything but the large models with a recent Ryzen system. The apple silicon is good but not as special as folks think!
Nice! This will be very useful for me. Think I can run this locally can spin a basic telegram bot around it for personal use.
One issue I faced with all the whisper based transcript generators is that there seems to be no good way to make editing/correcting the generated text with word level timestamp. I created a small web based tool[0] for that.
By any chance if anyone is looking to edit transcripts generated using whisper, you'd probably find it useful.
54 comments
[ 1.6 ms ] story [ 118 ms ] threadThis open source project provides a self-hostable API for speech to text transcription using a finetuned Whisper ASR model. The API allows you to easily convert audio files to text through HTTP requests. Ideal for adding speech recognition capabilities to your applications.
Key features:
- Uses a finetuned Whisper model for accurate speech recognition - Simple HTTP API for audio file transcription - User level access with API keys for managing usage - Self-hostable code for your own speech transcription service - Quantized model optimization for fast and efficient inference - Open source implementation for customization and transparency
How does this compare to what is possible using https://goodsnooze.gumroad.com/l/macwhisper for example?
Thanks!
Just visit the swagger create account and then gettoken to grab a token
> curl -X 'POST' 'https://innovatorved-whisper-api.hf.space/api/v1/users/get_t...'
while later saying
> curl -X 'POST' 'http://localhost:8000/api/v1/transcribe/?model=tiny.en.q5'
If you just change the first "curl" to also be localhost:8000, this will be cleared up.
"This project provides an API with user level access support to transcribe speech to text using a finetuned and processed Whisper ASR model."
Why is this a service at all? Why not just a library? Or a subprocess?
If it's completely self-hosted why do I need to get a token? Where does the actual model run?
An iPhone app that could do this from the microphone would also be amazing. Google Translate and it's various competitors from Microsoft/Apple are nearly there, but they all stop listening inbetween sentences. Something that just listened constantly, printing translated text onto the screen, would be amazing.
Whisper – open source speech recognition by OpenAI https://news.ycombinator.com/item?id=34985848
Even if it meant delaying the broadcast for a second while transcribing the accessibility value could be immense.
For anyone confused about the project, it is using whisper.cpp, a C-based runner and translation of the open whisper model from OpenAI. It is built by the team behind GGML and llama.cpp. https://github.com/ggerganov
You can fork this code, run it on your own server, and hit the API. The server itself will use FFmpeg to convert the audio file into the required format and run the C translation of the whisper model against the file.
By doing this you can separate yourself from the requirement of paying the fee that OpenAI charges for their Whisper service and fully own your translations. The models that the author has supplied here are rather small but should run decent on a CPU. If you want to go to larger model sizes you would likely need to change the compilation options and use a server with a GPU.
Similar to this project, my product https://superwhisper.com is using these whisper.cpp models to provide really good Dictation on macOS.
Its runs really fast on the M series chips. Most of this message was dictated using superwhisper.
Congrats to the author of this project. Seems like a useful implementation of the whisper.cpp project.
I wonder if they would accept it upstream in the examples.
If you have Nvidia hardware the ctranslate2 based faster-whisper is very very fast: https://github.com/guillaumekln/faster-whisper
We use it for our Willow Inference Server which has an API that can be used directly like OP project and supports all Whisper models, TTS, etc:
https://github.com/toverainc/willow-inference-server
The benchmarks are pretty incredible (largely thanks to ctranslate2).
https://twitter.com/Stavros/status/1693204822042739124
Fwiw decent acceleration works on any avx2 compatible chipset. I get realtime speed for everything but the large models with a recent Ryzen system. The apple silicon is good but not as special as folks think!
One issue I faced with all the whisper based transcript generators is that there seems to be no good way to make editing/correcting the generated text with word level timestamp. I created a small web based tool[0] for that.
By any chance if anyone is looking to edit transcripts generated using whisper, you'd probably find it useful.
[0] https://github.com/geekodour/wscribe-editor