TTS models present a different threat model than LLMs, it is not easy to open source them. This is in line with some regulations EU is putting on AI companies soon (not out yet).
Closed model means I'm immediately not interested. There are lots of closed STT/TTS models that are insanely good, better than any open model, so unless this is open then not sure I'd ever use this.
Hopefully smaller models soon that can be deployed locally for personal use, definitely don't want the speech data sent outside. But having the speech ability is going to improve usefulness of the already great gen AI models drastically!
I like that a lot of ASR models now offer native context biasing capabilities. If I have enough volume, I would probably just self-host my model and do my own biasing but good enough to roll out custom solutions.
How did they integrate Voxtral Large exactly? Does it output a special token to indicate it is asking for help, or is it running a command?
Anyway, if it is trained to decide when to think/ask for help that's good.
Too bad it's not open source though. But for large companies in Europe or elsewhere that want to control their data, this might be one of the best options now.
Although the frontier is true full duplex. Anyone know if there is anything that is open source for that other than MoshiRag or PersonaPlex?
Also did the CSM update some months back fix it? I never got a chance to test it.
Interesting. I've found the Voxtral series pretty nice actually. Good transcription and diarization which is the primary personal use-case where I want to turn podcast content into text since I read faster than I listen. The older ones were pretty cheap too and had the nice property that they don't hallucinate text when there's no speech.
The examples they give of how they “improved” the voices aren’t great. Version 1 sounded like actual people, while this version sounds like generic AI slop voice.
Slightly off-topic, but does anybody know a local model/tool that fixes timestamps of existing srt subtitles? That should be easier than generating them from scratch.
I don't plan to release it as it part of an overall "media health" tool I built but you should be able to vibe this pretty easily. I have mine using Whisper to check episodes at intervals throughout to compare against this subtitles and then shift them as needed.
One thing to watch for, make sure it never tries to 2 subtitles at the same time or they will stack in a weird way. It's easy to avoid that but my first pass on a few episodes screwed up the subtitles.
Also, you almost certainly want to extract the subtitles for this work, or at least that's what I did. A combo of pulling out the subtitles in the video file and using OpenSubtitles (VIP) to clean up my media.
I whipped something up for a friend a few months ago, it's not just fixing timestamps like you've asked, it's processing the whole audio but it does a good job (uses parakeet) and is quite quick.
Honestly you can probably build your own quite easily. I put together a plex player for firetv that does on device subtitling for whatever you're watching. I got sick of having to find subtitles.
> When I last looked for this a few years ago, I only found a really primitive one that did a linear transformation of all timestamps in the video, and none to correct each line individually.
Not an answer to your current query, but my first published rust crate shortly after the language hit 1.0 was a tool to both calculate and correct linear drift, long before the era of LLMs or quality audio recognition. It might be fun to tackle this again with the new tech we now have at hand!
It works really well and it's just a single static binary file (ffmpeg is needed on your $PATH), easier to use than ffsubsync which needs gcc to compile random stuff, and that stopped working randomly
I think the original SubSync was the useless linear tool I tried last time. Though the Sefzz15 fork added some kind of "piecewise" feature which might work.
No, it finds over a hundred of sync points it can match with the translated subtitle (either via reference subtitle or audio track). How the interpolation between the sync points is done I haven't checked, but it fixes gradual drifts, offsets, gaps, etc. It syncs everything.
If you want really accurate timestamps, the current state of the art is to ask claude to implement forced alignment using a CTC transcription model. It uses the logprobs of a character based transcription model to find the best matching overlap. I've used this technique to generate sync files for epub/audiobooks and it's really good even when the data is dirty.
This method aligns the transcript of a 60 hour book in 20 minutes on my GPU using Parakeet CTC 0.6B which is english only, but there's similarly sized models in other languages, the models don't need to be that good at transcribing to generate good alignments.
I've been playing with automatically subtitling shows using local models in conjunction with DeepSeek to mild success, and this was one of my pet peeves because my subtitles were off by up to a couple seconds depending on where the STT frame was. What I have so far is good enough to be watchable if you can look past minor translation errors and not getting translated text.
Considering the progress on speech recognition, why do YouTube subtitles have little in common with what's spoken in the video? Shouldn't Google be able to do a good job at auch a task?
I suspect they're using some new cheap multi-modal model. A couple of times now I've seen the generated subtitles fill muffled speech with text about something unrelated happening on screen.
YouTube subtitles usually are not that bad, but they tend to get wrong on key moments.
For example the sentence may be "They decided to rewrite the application in Rust", and the subtitle be "They decided to rewrite the application in rats". They made a single mistake, and it is on the most important word. I guess it can be explained by the fact that these key words tend to be uncommon while the rest of the sentence just flows naturally, that is, you barely need to listen, you can guess most of it.
It worries me a bit because as models get better, they may come up with more and more consistent, but wrong transcriptions. Like a typical LLM hallucination, so confidently wrong that you don't question it.
We could eliminate fraud overnight if the same mechanisms that prevent denial of the Holocaust were repurposed for domains that actually warrant that level of rigor.
I think this is the main thing preserving the (miserable) jobs of people at call centers.
But I'm not convinced. With today's models, is it really that much easier to defraud an AI agent than a human?
I do think a lot of people are much more WILLING to try and defraud an AI agent than a person, so maybe that's the difference. There's a parallel there to people being much more willing to mess with Waymos. I think both are a huge shame.
Largely this can be solved the same way it is at a lot of call centers: first line support isn't actually given permission to do much more than the customer can do themselves.
I wouldn't exactly call that "solved", this still sucks for the customer.
I think the ideal solution would be to track and punish fraudsters, but there seems to be little political will to do anything about petty crime these days.
I think the difference is if you figure out how to defraud a human once it may not work on the next guy or on the same guy if you repeat it. If you figure out how to defraud an AI once you can repeat it indefinitely until the AI is shut off.
Intelligence has little to do with it, Opus relies on prompt injections done by a pretty zealous classifier or a cascade of classifiers. It has a ton of semi-black box machinery around it, which can be taken apart. Despite Anthropic's benchmarks and claims, Claude was always the easiest to deceive or JB if you know how to do it compared to e.g. GPT, which the RP community knows pretty well. They definitely improved it in 5.5 (due to distillation fears, ironically), but I'm not trusting their or anyone else's numbers this time either. These benchmarks always look good, and then there's someone routinely jailbreaking it just for fun.
I am one of those people that have jailbroken these models including the classifiers. However, most of these rely to having access to the system prompt and the whole harness which include noise injection and other things that you simply wouldn't be able to do in a support chat.
I’ll be interested to see how this compares to Nvidia’s Nemotron 3 Diarization model for diarization. For my use-cases real-time diarization is still not where I’d like it to be, but nemotron was definitely a step above the previous attempts.
Wow, Voxtral 3 sounds so much more artificial and “AI” as compared to the old Voxtral 1 in the comparison clips on that page! I absolutely hated listening to the new one.
Also, comparisons with Whisper always really tick me off. It’s not a dictation model so much as it is a transcription model. It’s really the only one you can use to capture incidental audio rather than one or more persons deliberately speaking into a microphone, e.g. the only one you can use to transcribe audio from surveillance footage or background audio recorders; it has a higher WER but it actually tries to capture everything (leading quite often to hallucination, unfortunately) whereas the other models they compare it against tune out background noise and dialog altogether, in my fairly extensive testing.
I’ve been using mistral-large-3 a tremendous amount. Price/speed/quality ratio was the best I could find. Been using the mistral-large-4 now since it came out for a few things, and while it’s not price competitive for the quality, it is really good. I’ve replaced a lot of my browser-based-chat-ui-llm-usage with mistral since it launched.
You're right. They were quiet for a long time and had promised new things "by the end of summer." They missed that window but have really been on a roll since they started up again.
Their previous generation open weight STT model was pretty good.
I have recently launched commercial Windows software implementing offline inference of Voxtral Mini 4B Realtime on commodity hardware: https://vrmac-softver.com/speech2text It requires 4GB VRAM, and exceeds realtime rate on my “median” laptop with Ryzen 5 5600U iGPU. I’m currently working on a Linux port.
I haven't tried Voxtral but I've been really enjoying Parakeet from NVidia.
Been using it to implement a mixed audiobook player + ebook reader that uses Parakeet to sync the narration with its position in the text. Runs fast enough even if CPU-only
"I've even had it transcribe words before I've finished saying them! " lol idk if this part is actually a good thing. It's great when you always speak in predictable sandwiches.
I do think Parakeet is pretty amazing, though, so some of these product announcements have limited interest to me, especially since it's cloud and costs per hour, even if it's a low cost.
You can’t, as far as I can tell. The post offers the API, and self-deployment goes through sales. Nothing for Voxtral 3 is on Hugging Face. The last open one is Voxtral Mini 4B Realtime from February, Apache 2.0: https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-26...
I know the doomerism of robots takin R jobs get repetitive, but this seems like it would work very well in call centers. I get frustrated talking to AI call center agents, but that's often because they are less capable and prone to bad logic errors. To be fair, speaking to humans in call centers can be annoying too. If a call could be answered without waiting in a queue for 30 minutes by a competent AI agent, I think I would prefer that.
176 comments
[ 1.1 ms ] story [ 50.0 ms ] threadhttps://x.com/GradiumAI/status/2112563128487514512?s=20
However, I found Qwen ASR and QwenTTS to be quite good, and not too large. Also https://github.com/kyutai-labs/pocket-tts was also great (but only a few languages)
FWIW.
Fucking awesome ;)
It's small enough to be run on modern phones realtime, I use it for listening to PDF's/EPUB as Audiobooks
https://huggingface.co/hexgrad/Kokoro-82M
i would love to have pdf read to me as audiobooks. Do you have a github?
i am a big fan of audiobooks, and having this with the features of my favorite player would be amazing
Not because of the models, but because it comes in a nice (easy-to-set-up) package, and the latency was lower than I expected with such a setup.
Anyway, if it is trained to decide when to think/ask for help that's good.
Too bad it's not open source though. But for large companies in Europe or elsewhere that want to control their data, this might be one of the best options now.
Although the frontier is true full duplex. Anyone know if there is anything that is open source for that other than MoshiRag or PersonaPlex?
Also did the CSM update some months back fix it? I never got a chance to test it.
One thing to watch for, make sure it never tries to 2 subtitles at the same time or they will stack in a weird way. It's easy to avoid that but my first pass on a few episodes screwed up the subtitles.
Also, you almost certainly want to extract the subtitles for this work, or at least that's what I did. A combo of pulling out the subtitles in the video file and using OpenSubtitles (VIP) to clean up my media.
Download links and a bit of write up here - https://news.ycombinator.com/item?id=48528779#48535207
Honestly you can probably build your own quite easily. I put together a plex player for firetv that does on device subtitling for whatever you're watching. I got sick of having to find subtitles.
Not an answer to your current query, but my first published rust crate shortly after the language hit 1.0 was a tool to both calculate and correct linear drift, long before the era of LLMs or quality audio recognition. It might be fun to tackle this again with the new tech we now have at hand!
Srt-resync: https://github.com/mqudsi/srtresync
https://github.com/sc0ty/subsync
This method aligns the transcript of a 60 hour book in 20 minutes on my GPU using Parakeet CTC 0.6B which is english only, but there's similarly sized models in other languages, the models don't need to be that good at transcribing to generate good alignments.
Will definitely look into implementing this.
Imagine the same subtitles working across different encodings and (minor) edits of the same video…
For example the sentence may be "They decided to rewrite the application in Rust", and the subtitle be "They decided to rewrite the application in rats". They made a single mistake, and it is on the most important word. I guess it can be explained by the fact that these key words tend to be uncommon while the rest of the sentence just flows naturally, that is, you barely need to listen, you can guess most of it.
It worries me a bit because as models get better, they may come up with more and more consistent, but wrong transcriptions. Like a typical LLM hallucination, so confidently wrong that you don't question it.
But I'm not convinced. With today's models, is it really that much easier to defraud an AI agent than a human?
I do think a lot of people are much more WILLING to try and defraud an AI agent than a person, so maybe that's the difference. There's a parallel there to people being much more willing to mess with Waymos. I think both are a huge shame.
I think the ideal solution would be to track and punish fraudsters, but there seems to be little political will to do anything about petty crime these days.
I would be very happy if it ends up being used for petty crime vs. e.g. monitoring political dissidents.
Also, comparisons with Whisper always really tick me off. It’s not a dictation model so much as it is a transcription model. It’s really the only one you can use to capture incidental audio rather than one or more persons deliberately speaking into a microphone, e.g. the only one you can use to transcribe audio from surveillance footage or background audio recorders; it has a higher WER but it actually tries to capture everything (leading quite often to hallucination, unfortunately) whereas the other models they compare it against tune out background noise and dialog altogether, in my fairly extensive testing.
I have recently launched commercial Windows software implementing offline inference of Voxtral Mini 4B Realtime on commodity hardware: https://vrmac-softver.com/speech2text It requires 4GB VRAM, and exceeds realtime rate on my “median” laptop with Ryzen 5 5600U iGPU. I’m currently working on a Linux port.
Been using it to implement a mixed audiobook player + ebook reader that uses Parakeet to sync the narration with its position in the text. Runs fast enough even if CPU-only
I do think Parakeet is pretty amazing, though, so some of these product announcements have limited interest to me, especially since it's cloud and costs per hour, even if it's a low cost.
Parakeet is trained on text that has had ITN done on it, so it prints digits, which messes up window merges. I'm training that out of it: https://github.com/MimicScribe/parakeet-spoken-form