Hi HN — I’ve been working on a real-time conversational AI, and one of the biggest problems is speech latency. As I was experimenting with various TTS systems I stumbled upon Pocket-TTS by Kyutai. It runs on the CPU which made it a great candidate since it keeps speech synthesis away from GPU resources.
As I was using it I noticed that it didn't solve the latency issues fully due to leading silence and artifacts generation. So I started poking under the hood with the help of agents to try and improve it.
On m4 max, the new browser build runs around ~14x realtime on longer utterances, and around ~3.5x realtime on an iPhone 16 Pro. The native CPU path is faster still: roughly 32–33x realtime warm, with first useful audio around 30ms on my benchmarks. Other systems will vary; my Windows browser results were slower but still usable. This does not introduce a new model or claim better voice quality. The goal was to preserve the intended PocketTTS quality while making it fast enough for low-latency interactive scenarios.
I decided to open source it in case it's useful for anyone else, or at least as a demo of what things that are finally possible now that the barrier of trying stuff has been lowered.
EDIT: Keep in mind it downloads 67mb of data on load.
Are you doing this to unlock natural human-to-model conversations? I was speaking to an AI-demo-assistant yesterday and I think I get whay you are talking about when you say the delay has to be really small.
Yes, you can see a demo video in the page, of a very early version. I’m building an AI gaming companion. Essentially an AI NPC (a speaking dog) that is grounded in the game world, can perceive what’s happening, perform actions, follow complex multi-step instructions, and strategize with the player. I want it to feel natural, so I 'm aiming for real audio conversationality (no push to talk, then wait), so audio is important, but also... every millisecond I trim from audio I can dedicate to the llm (caching helps heavily there).
Budgets are pretty tight, ~400ms for everything. I have optimized qwen-3-asr and added streaming detection to it, as well as pocket-tts (this) and qwen-3-tts, with a goal of ~70ms for audio detection/creation and then ~350ms for the LLM/planning.
It's interesting that gpt-live was just announced today, planning to make more things open source as well as the learnings while building this.
4 comments
[ 3.0 ms ] story [ 21.8 ms ] threadAs I was using it I noticed that it didn't solve the latency issues fully due to leading silence and artifacts generation. So I started poking under the hood with the help of agents to try and improve it.
On m4 max, the new browser build runs around ~14x realtime on longer utterances, and around ~3.5x realtime on an iPhone 16 Pro. The native CPU path is faster still: roughly 32–33x realtime warm, with first useful audio around 30ms on my benchmarks. Other systems will vary; my Windows browser results were slower but still usable. This does not introduce a new model or claim better voice quality. The goal was to preserve the intended PocketTTS quality while making it fast enough for low-latency interactive scenarios.
I decided to open source it in case it's useful for anyone else, or at least as a demo of what things that are finally possible now that the barrier of trying stuff has been lowered.
EDIT: Keep in mind it downloads 67mb of data on load.
Are you doing this to unlock natural human-to-model conversations? I was speaking to an AI-demo-assistant yesterday and I think I get whay you are talking about when you say the delay has to be really small.
What is the bigger project you're working on?
Budgets are pretty tight, ~400ms for everything. I have optimized qwen-3-asr and added streaming detection to it, as well as pocket-tts (this) and qwen-3-tts, with a goal of ~70ms for audio detection/creation and then ~350ms for the LLM/planning.
It's interesting that gpt-live was just announced today, planning to make more things open source as well as the learnings while building this.