Show HN: Port of OpenAI's Whisper model in C/C++ (github.com)
OpenAI recently released a model for automatic speech recognition called Whisper [0]. I decided to reimplement the inference of the model from scratch using C/C++. To achieve this I implemented a minimalistic tensor library in C and ported the high-level architecture of the model in C++. The entire code is less than 8000 lines of code and is contained in just 2 source files without any third-party dependencies. The Github project is here:
https://github.com/ggerganov/whisper.cpp
With this implementation I can very easily build and run the model - “make base.en”. It also allows me to run it on a wide range of devices. For example, I have provided examples of running the model on an iPhone, Raspberry Pi 4 and even in a web page via WebAssembly!
The implementation runs fully on the CPU and utilizes FP16, AVX intrinsics on x86 architectures and NEON + Accelerate framework on Apple Silicon. The latter is especially efficient and I observe that the inference is about 2-3 times faster compared to the current PyTorch implementation provided by OpenAI when running it on my MacBook M1 Pro. The WASM port utilizes SIMD 128-bit intrinsics - a feature supported in some modern web browsers [1].
I am very happy with the performance that I observe on Apple Silicon devices. I didn’t expect that the Accelerate framework [2] (i.e. CBLAS) offers such a dramatic performance boost for matrix multiplications so I was very pleasantly surprised! To enable the framework in your C/C++ projects, all you have to do is add `-framework Accelerate` to your clang command-line flags.
This entire exercise of implementing the Whisper model was very interesting to me and helped me understand a lot about how the transformer architecture works. I also got a lot of positive feedback from people finding and using my project. We brainstormed on a lot of interesting tools that can potentially be created with this library (such as speech-to-text plugin for Vim, RPi4 voice assistant, WASM chat bot, etc). If interested, checkout the “Examples” section and the “Show and tell” discussions for some ideas!
Would love to know what you think about this project and about your experience with using the Accelerate framework in any of your projects. Cheers!
[0] https://github.com/openai/whisper
92 comments
[ 2.9 ms ] story [ 172 ms ] threadIf they crawled /r/programming and hackernews, it should be annoyed and come up with something along the lines of "fork it and write in in Rust yourself", "C++ has lots of inertia", "his main stack is C/C++", etc
A fully CPU based implementation, simple enough with minimal dependencies is also something that helps tremendously reduce the initial friction and enable potential low-cost applications.
Excellent and impressive work, can’t wait to try this thing at home.
This is surprising to me. Is this about CPU only? Then it would make sense.
Also, is there a particular reason why the whole code is basically 2 massive files (3k and 8k lines respectively)?
Hook this up to ChatGPT and you've got something better than Google Assistant with almost no work.
(You can tell ChatGPT an API, and ask it to generate a script in response to a voice assistant query.)
I've started tinkering with the code about last week and despite knowing nothing about C/C++, I was able to make some edits to fit my use case, and connect it to a custom Python front end (I initially tried to use Qt in C++ but struggled so much to get to it to compile that I've switched to Python instead). This probably means your code is very clean and well documented.
It's a game changer in terms of accessibility: it can caption almost anything in live!
I'm very grateful for the effort that you've lead. Thank you ggerganov, and thanks to everyone who contributed.
Perhaps in combination with https://www.npmjs.com/package/peertube-plugin-transcription your port of Whisper could be used for generating subtitles for videos in PeerTube?
I just recently set up a PeerTube instance of my own and uploaded my first video on it ("No Brain Required - ChatGPT solves Advent of Code in Rust, episode 1", https://video.nstr.no/w/6z7PxB4J92H3NHhgMmfYVw)
I want to try and make use of your port of Whisper on my PeerTube instance, so that I can have subtitles generated for my videos on it :D
So many different cuda version, with each framework using its own, that all rely on a different driver, and everything needs a new version every 3 months and takes ~10G, (and don't even talk about cudnn needing some manual logged-in install).
Here everything is just two files. For embedded system that don't have a GPU it's perfect.
Here the parallelization and vectorization has been done by hand, but there is a glimmer of hope coming from the side of various compiler projects :
Here is an interesting intel project that does the parallelization and vectorization automatically for different architecture that's definitely worth a look : https://ispc.github.io/ispc.html
For the auto-differentiation when I need performance or memory, I currently use tapenade ( http://tapenade.inria.fr:8080/tapenade/index.jsp ) and/or manually written gradient when I need to fuse some kernel, but Enzyme ( https://enzyme.mit.edu/ ) is also very promising.
MPI for parallelization across machines.
So much more practical to hack around and or build small apps.
Some things never change.
It looks like it will open a new application scenario for low-performance hardware products.
I looks like you’ve definitely maximized the parallelization on the CPU/AMX here, but have you tried getting it to run on the GPU or the Neural Engine? I love the portability, but I feel like you would get a massive parallelization boost while dramatically cutting energy consumption.
Honestly, I am bit confused with all the different types of processing units available on Apple Silicon. If I understand correctly, we have: CPU, GPU, AMX coprocessor and Neural Engine on a single chip. I don't fully understand how these interact with each other. Can we use them all at the same time, or would there be some penalties? I'm interested in finding some resources/information on the topic.
My understanding is that the AMX is more tightly wound with the CPU, ultimately being accessible via an instruction set (https://github.com/corsix/amx), and it is useful if you need to do matrix multiplications interleaved with other CPU tasks. A common example would be a VIO loop or something where you want that data in the CPU caches.
The GPU and Neural Engine are not that – they take some time to set up and initialize. They also can parallelize tasks to a much higher degree. The GPU is more generalizable, because you can write compute shaders to do anything in parallel, but it uses a lot of resources. I'll have to check out the PR to see how exactly the MPS implementation matches up with the task at hand, because you could also consider writing Metal compute shaders by hand. Even if the performance is not much better, the CPU is free to do other things.
I know the least about the ANE, but it has specific hardware for running ML models, and you have to process the weights ahead of time to make sure they are in the right format. It can run ML models very efficiently and is the most battery friendly.
I haven't looked at your code closely so can't say with certainty it would be the right fit but worth a look.
Honest question, do you think with this marked improvement it could be worth making a wrapped library for this c/c++ version in python - for example like numpy?
There are frequent updates and performance improvements. There is also a small community of active users around this.
All most all feedbacks get implemented and the OP is very responsive.
The OP made it possible to do state of the art voice recognition without the PyTorch baggage and in C/C++, pretty incredible! Its one of those rare high value projects.
Very grateful for this project and respect to the OP!
Some day if a ChatGPT open version becomes available, this could mean voice assistants that speak sense and understand the human - as long you have a beefy machine.
The current efficiency is pretty surprising, even on a low spec device it performs faster than real time.
I don't know what to say. But I'm blown away.
I expect to see more magic from the OP in future.
He has even a project for a cool sound modem that works over ultrasonic! Not new stuff, but the implementation is the most robust I have seen.
I recommend hackers here to check out his other project too and maybe contribute with testing and patches and stuff!
A single binary that does this all w/o the python stack is just incredible!
edit: Got it going in 1 min!
I grabbed the prebuilt artifacts (windows)
- https://github.com/ggerganov/whisper.cpp/actions/runs/363552...
Then downloaded ggml-base.bin (148mb) and put it in models/ggml-base.en.bin
- https://huggingface.co/datasets/ggerganov/whisper.cpp/blob/m...
Ran it and everything worked! Amazing. Note that only the large (3GB) whisper-v2 is available at the moment, but haven't seen any errors yet from the older small ones. Wild.
I am experimenting with your code now. Is there a way to force Whisper to only consider a limited vocabulary and then respond with confidence levels? I am working on an app where it is important to restrict answers and I would like to know how confident that a response is one of a set of words. If the answer could be word A with a confidence level of 95% and word B with a level of 50%, I would want to know that so that I could perform context verification.
Thanks!
Bill
https://github.com/openai/whisper/discussions/117#discussion...
https://github.com/ggerganov/whisper.cpp/pull/271
The source code is in the `command.cpp` and I will soon write some more details how it works. If you give it a try, definitely let me know if it worked for you.