That's close to what this is: the reader is a CTC head on the audio encoder, but its output words are EmbeddingGemma's own text embeddings for ~229k words, instead of a separately learned vocabulary. I also tried a plain letter-level CTC head on the encoder. It did a bit better on clean audiobooks (7.0 vs 8.2 WER at the time) but much worse on harder audio like a lecture (16.5 vs 10.8), mostly from misspellings.
during transcription, nothing is actually "playing" the file in the way a human would to listen to it. the audio is stored as a sequence of bytes representing the amplitude of the physical sound waves over time (the recording device measures, or "samples", the amount that its microphone is vibrating and records that amount, some tens of thousands of times per second) with some metadata and compression applied so that other software knows how to interpret the bytes correctly. this can be given to an audio program to convert the bytes back to physical sound waves using your speakers, but it's just bytes at the end of the day, so any software that can read bytes out of the file can do so at whatever speed it wants, regardless of how long the actual recording is.
so you have a file containing bytes representing the sound wave, and you have existing recordings (read: sound waves represented as sequences of bytes) of people speaking in different languages and contests, and you can feed all those existing recordings into a model along with corresponding transcriptions, and you can get a model out which can identify patterns in recordings of people speaking to then go backwards and see a pattern of sound waves and correctly associate it with a specific token, no need to play anything back. pauses, stutters, etc. are themselves patterns of sound waves that can be identified and worked around to identify just the meaningful parts of speech.
in principle, you could probably learn to do this too, by sitting down for hundreds of hours meticulously starting at waveforms, and this is, very roughly, what signal analysts do: look at waveforms and try to identify patterns. ai just leverages this principle to turn mundane speech into text.
Does this work for Hindi or other non English languages though?
I wanted to recently transcribe Hindi + English (mixed) Audio and I used this model[0] and created a custom transcribe command which can give text over input of mp4 files
This took me roughly around an hour or more or around this time frame for a 90 minute transcription source. I would consider myself to be slightly bored during this phase.
I would really appreciate it if something like the project could be made native for a lot more languages but I think that the approach of embedding models and using them in such a way is pretty crazy. Kudos for it and I will try it
out!
I can think of many such use cases where Speech to text of even large audio files becomes near instant. As I think someone on hackernews once quoted whom I am very paraphrasing
An order of one or two magnitude of an already existing thing stops becoming a quantity upgrade but rather becomes a quality upgrade in it of itself as it provides access to workflows which previously wouldn't have been feasible.
Yeah I will have to try whisper as well. I didn't try it because I wanted to tinker more and also I had thought that whisper was an older model but I think that if it gets the job, then it gets the job done and I will be trying it out on the text that I mentioned, thanks for mentioning it as I will give it another go :-D
"EmbeddingGemma 2 is made by Google DeepMind and released under Apache 2.0. This is an independent project, not affiliated with or endorsed by Google."
The reference to Apache 2.0 on the page refers to Google's base model, not this app. Based on the page alone, the app is free, but the source code is not open-source.
Apple's on-device transcription model is about 3x slower than their reported results on my more powerful computer, with a similar error rate in my test (which I consider pretty good results)
I used it a lot in my AI client before I finally got around to hacking a Deepgram API server into Handy: It sucks enough to give me a headache worrying about its unreliability in every given transcription.
They supposedly improved it on later Apple Intelligence enabled devices, but I don’t have one.
interesting! I only tested against Parakeet v2 because my agent suggested it would be faster for English transcription (v3 is multilingual but somewhat slower, it suggested)
On my bench, parakeet-tdt-0.6b-v2 is 2x slower
I wonder if you're using different Apple models when you go through that app? I don't know
Interesting... is it overkill to use a multimodal model like this on just audio? Could the model size be decreased dramatically, or does the multimodality help the accuracy and speed in some way?
The benefit is that you can take input in one "mode" and extract output in another; that is, the input is audio mode and the output is text mode. The intermediate representation is the "embedding".
I tested a 33 minute local audio only file on a M4 Mac Mini with 24GB of Ram and it took 6.75 seconds to transcribe it using this software. Using MacWhisper/Parakeet3 it was about 7.5 secs using a stopwatch.
Hi all. I am Shahir. This was my experiment. First of all - Thanks to whoever posted this on HN. It came as a surprise.
Just to clarify - This started as just an experiment to see if we can use the fact that text and audio share a vector space in Embedding Gemma 2 to convert audio to text. Had to hack around a few things, but, it worked. Speed was just a side benefit. Was not aiming for it.
The accuracy is good, but not in the range of cloud models, but decent enough for everyday use, and this is just a couple of days work. The WER can improve a lot with more training on varied data. Will try that next.
I will opensource the whole thing soon. Might need a few days though. Also, releasing the TTS version for this soon.
It can already classify decently well, and I tried training it as a decision model, It worked well on domains I trained it for. It was beating Liquid's small decision model, but somehow, it never generalized. Out of the trained domains, it was just above chance.
You can train a traditional classifier on it quite well.
But the one-shot and few-shot classification performance of decision models really needs in-context learning. The most common way of getting that is by grafting a small LLM onto it, which is certainly possible.
At that point though, why not just use a small LLM with audio capabilities. Gemma E2B is a good model that is already setup for this, and training a decision model LoRA adapter is very easy.
It was your post yesterday that made me aware of granite-speech-5.0-470m-turboctc (which I guess you used as a comparator because it's already very fast).
I recently swapped to using Parakeet v3 on my old M1 Macbook Air, using MacWhisper. It was so fast I thought it had failed. As in, by the time I clicked "transcribe" and looked at my screen again a few seconds later the transcript had finished. Absolutely insanely fast.
People say old MacBook Air M1 but its architecture was so groundbreaking that it's still relevant in 2026. Apple definitely underplayed its release in 2020.
Maybe it's just a COVID time blending together effect but it still blows my mind how quickly we went from "automatic audio transcriptions are not even worth having on" to "an hour of audio transcribed in 13s locally."
Pre-pandemic it seemed like even the very best in class auto-transcription features were just unusably bad. I guess that was like 7 years years ago though.
It's funny cause someome needs to remind them that their company is owned by google + somebody needs to care about that part. It might even take longer than 10 years.
Not sure what you mean, they added automated transcriptions a few years back and it’s mind blowing. I regularly use it for 6-8 hour long court footage and it’s superior than any other local solution I use, albeit not perfect
In my experience unless everyone in the video speaks perfect BBC English it's really really poor quality, and I use it all the time. And for any videos with made up names it just falls apart - try watching an average video game review with it on.
"One cool trick" I like for running almost any model very quickly on macOS is converting them to CoreML so they run in the Neural Engine.
I had amazing success with Spleeter, Demucs, MVSEP-MDX23-music-separation, DeepFilterNet, Hunyuan3D. Plus a lot of others for both work and leisure.
I also convert the code to Swift without dependencies, but that's a personal preference.
The runtime on some of them went sometimes from 60s to 10s. And since it's just Swift, I can just give them to non-technical co-workers and they don't have to setup a whole environment.
If you don't wanna do this by hand, this is something that existing LLM models are good at. Even Luna can do it without much fuss.
It is very suitable and very straightforward for a crazy large amount of models! I haven’t even mentioned the ones we use at work! There’s lot beyond LLMs. I suggest trying before!
perhaps you're right, things evolve. sadly i need to target more nvidia for production, so mac stuff is still second level of concern, though indeed very promising, and oerhas for increasing amount of tasks.
Ok let's talk about the really important stuff here: that model name is atrociously long for such a game-changing model, and "embedding" is already an awkward phrase. I propose... Embedda?
95 comments
[ 3.4 ms ] story [ 50.7 ms ] threadOr is there another approach?
so you have a file containing bytes representing the sound wave, and you have existing recordings (read: sound waves represented as sequences of bytes) of people speaking in different languages and contests, and you can feed all those existing recordings into a model along with corresponding transcriptions, and you can get a model out which can identify patterns in recordings of people speaking to then go backwards and see a pattern of sound waves and correctly associate it with a specific token, no need to play anything back. pauses, stutters, etc. are themselves patterns of sound waves that can be identified and worked around to identify just the meaningful parts of speech.
in principle, you could probably learn to do this too, by sitting down for hundreds of hours meticulously starting at waveforms, and this is, very roughly, what signal analysts do: look at waveforms and try to identify patterns. ai just leverages this principle to turn mundane speech into text.
Does this work for Hindi or other non English languages though?
I wanted to recently transcribe Hindi + English (mixed) Audio and I used this model[0] and created a custom transcribe command which can give text over input of mp4 files
This took me roughly around an hour or more or around this time frame for a 90 minute transcription source. I would consider myself to be slightly bored during this phase.
I would really appreciate it if something like the project could be made native for a lot more languages but I think that the approach of embedding models and using them in such a way is pretty crazy. Kudos for it and I will try it out!
I can think of many such use cases where Speech to text of even large audio files becomes near instant. As I think someone on hackernews once quoted whom I am very paraphrasing
An order of one or two magnitude of an already existing thing stops becoming a quantity upgrade but rather becomes a quality upgrade in it of itself as it provides access to workflows which previously wouldn't have been feasible.
[0]: https://huggingface.co/parismitaglobalsolutions/indicconform...
https://openai.com/index/whisper/
https://github.com/openai/whisper
"EmbeddingGemma 2 is made by Google DeepMind and released under Apache 2.0. This is an independent project, not affiliated with or endorsed by Google."
I'll test theirs too if I get around to it
Here's a sample CLI wrapping the apple model and providing a simple benchmark: https://github.com/llimllib/transcribe
update: doesn't seem like they provide their model and I don't feel like trying to dig it out of their app bundle, so I'm not going to test this one
I use their on-device translation in my apps. Works pretty well. ChatGPT's context-aware translation is better, but it's not on-device.
They supposedly improved it on later Apple Intelligence enabled devices, but I don’t have one.
On my bench, parakeet-tdt-0.6b-v2 is 2x slower
I wonder if you're using different Apple models when you go through that app? I don't know
Just to clarify - This started as just an experiment to see if we can use the fact that text and audio share a vector space in Embedding Gemma 2 to convert audio to text. Had to hack around a few things, but, it worked. Speed was just a side benefit. Was not aiming for it.
The accuracy is good, but not in the range of cloud models, but decent enough for everyday use, and this is just a couple of days work. The WER can improve a lot with more training on varied data. Will try that next.
I will opensource the whole thing soon. Might need a few days though. Also, releasing the TTS version for this soon.
Happy to answer any questions you have.
But the one-shot and few-shot classification performance of decision models really needs in-context learning. The most common way of getting that is by grafting a small LLM onto it, which is certainly possible.
At that point though, why not just use a small LLM with audio capabilities. Gemma E2B is a good model that is already setup for this, and training a decision model LoRA adapter is very easy.
Edit: Apparently people have got decision models working on EmbeddingGemma, eg: https://huggingface.co/mrm8488/embeddinggemma-2-ft-typed-dec.... I'm not seeing it ranked on https://huggingface.co/spaces/multimodalart/jev-decision-ind... so I have questions about how well it does in-context learning.
People aren't paying enough attention to how great EmbeddingGemma 2 and Gemma 4 are.
Thank you!
(Until now I've been using mlx-community/parakeet-tdt-0.6b-v3)Pre-pandemic it seemed like even the very best in class auto-transcription features were just unusably bad. I guess that was like 7 years years ago though.
"One cool trick" I like for running almost any model very quickly on macOS is converting them to CoreML so they run in the Neural Engine.
I had amazing success with Spleeter, Demucs, MVSEP-MDX23-music-separation, DeepFilterNet, Hunyuan3D. Plus a lot of others for both work and leisure.
I also convert the code to Swift without dependencies, but that's a personal preference.
The runtime on some of them went sometimes from 60s to 10s. And since it's just Swift, I can just give them to non-technical co-workers and they don't have to setup a whole environment.
If you don't wanna do this by hand, this is something that existing LLM models are good at. Even Luna can do it without much fuss.
Although I'm sure someone is bolder than me is running a fleet of Mac Studios somewhere!