The model weighs 1.5GB [1] (the q4 quant is ~500MB)
The demo is impressive. It uses reference audio at inference time, and it looks like the training code is mostly available [2][3] with a reference dataset [4] as well.
you could try out nabu and let me know, i am working on adding more tts models in the future. It features all the kokoro voices, style mixing to create your own blend from their voices, basic kitten tts support, audio book / screen reader, LLMs and more :)
Every couple of weeks I see a new TTS model showcased here and it’s always difficult to see how they differ from one another. Why don’t they describe the architecture and details of the trailing data?
My cynical side thinks people just take the state-of-the-art open source model, use an LLM to alter the source, minimal fine tuning to change the weights and they are able to claim “we built our own state of the art tts”.
I know it’s open source, so I can dig into the details myself, but are they any good high-level overviews of modern TTS, comparing/contrasting the top models?
This is really neat. I cloned my voice and can generate text, but I can't seem to generate longer clips. The README.md says:
> Context Window: 2048 tokens, enough for processing ~30 seconds of audio (including prompt duration)
But it's cutting off for me before even that point. I fed it a paragraph of text and it gets part of the way through it before skipping a few words ahead, saying a few words more, then cutting off at 17 seconds. Another test just cut off after 21 seconds (no skipping).
Lastly, I'm on a MBP M3 Max with 128GB running Sequoia. I'm following all the "Guidelines for minimizing Latency" but generating a 4.16 second clip takes 16.51s for me. Not sure what I'm doing wrong or how you would use this in practice since it's not realtime and the limit is so low (and unclear). Maybe you are supposed to cut your text into smaller chunks and run them in parallel/sequence to get around the limit?
15 comments
[ 3.9 ms ] story [ 48.1 ms ] threadThe demo is impressive. It uses reference audio at inference time, and it looks like the training code is mostly available [2][3] with a reference dataset [4] as well.
From the README:
> NeuTTS Air is built off Qwen 0.5B
1. https://huggingface.co/neuphonic/neutts-air/tree/main
2. https://github.com/neuphonic/neutts-air/issues/7
3. https://github.com/neuphonic/neutts-air/blob/feat/example-fi...
4. https://huggingface.co/datasets/neuphonic/emilia-yodas-engli...
you could try out nabu and let me know, i am working on adding more tts models in the future. It features all the kokoro voices, style mixing to create your own blend from their voices, basic kitten tts support, audio book / screen reader, LLMs and more :)
My cynical side thinks people just take the state-of-the-art open source model, use an LLM to alter the source, minimal fine tuning to change the weights and they are able to claim “we built our own state of the art tts”.
I know it’s open source, so I can dig into the details myself, but are they any good high-level overviews of modern TTS, comparing/contrasting the top models?
> Context Window: 2048 tokens, enough for processing ~30 seconds of audio (including prompt duration)
But it's cutting off for me before even that point. I fed it a paragraph of text and it gets part of the way through it before skipping a few words ahead, saying a few words more, then cutting off at 17 seconds. Another test just cut off after 21 seconds (no skipping).
Lastly, I'm on a MBP M3 Max with 128GB running Sequoia. I'm following all the "Guidelines for minimizing Latency" but generating a 4.16 second clip takes 16.51s for me. Not sure what I'm doing wrong or how you would use this in practice since it's not realtime and the limit is so low (and unclear). Maybe you are supposed to cut your text into smaller chunks and run them in parallel/sequence to get around the limit?
This means using this TTS in commercial project is very dicy due to GPL3.
Watermarking is usually very fragile and generally relies on an adversary not knowing about it. I honestly don't know why anyone bothers with it.
But the current one seems really good, tested it for quite a bit with multiple kind of inputs.