Show HN: Real-time voice chat with AI, no transcription (demo.tincans.ai)
Instead, what if we fed audio directly to the LLM - LLM's are really smart, can they figure it out? This approach is faster (we skip transcription decoding) and less lossy/more robust because the big language model should be smarter than a smaller transcription decoder.
I've trained a model in just that fashion. For more architectural information and some training details, see this first post: https://tincans.ai/slm . For details about this model and some ideas for how to prompt it, see this post: https://tincans.ai/slm3 . We trained this on a very limited budget but the model is able to do some things that even GPT-4, Gemini, and Claude cannot, eg reasoning about long-context audio directly, without transcription. We also believe that this is the first model in the world to conduct adversarial attacks and apply preference modeling in the speech domain.
The demo is unoptimized (unquantized bf16 weights, default Huggingface inference, serverless speed bumps) but achieves 120ms time to first token with audio. You can basically think of it as Mistral 7B, so it'll be very fast and can also run basically anywhere. I am especially optimistic about embedded usage -- not needing the transcription step means that the resulting model is smaller and cheaper to use on the edge.
Would love to hear your thoughts and how you would use it! Weights are Apache-2 and on Hugging Face: https://huggingface.co/collections/tincans-ai/gazelle-v02-65...
6 comments
[ 3.0 ms ] story [ 24.8 ms ] threadLatency: OpenAI's implementation is quite slow - 5+ seconds to get a reply - but even optimized solutions are far slower than this approach. Fundamentally, most transcription models do audio -> audio encoder -> text decoder. By skipping the text decoder, we are always going to win on speed, we just do less work.
Emotion: transcription cannot transcribe emotion, only the text, but emotion - eg if the user is sarcastic, angry, happy, etc - plays a big role.
Transcription errors: in a cascaded approach, any mistake in an earlier step propagates to the latter steps. Mistranscription means that the LLM will make a mistake. But the LLM's are really really smart, certainly smarter than the transcription models - they'll get it right more often.
This iteration is not trained to do so. But the general model structure should work, i.e. if you finetune with instruction data to do so.
> Training a projection layer makes sense, but ultimately you'd want to output audio conditioned on the input rather than text. Is there a way to train a reverse projection with some kind of skip connections to take audio input into account? Or an end to end audio model?
Training an end-to-end model is the eventual goal, yeah; discrete tokens is the most pragmatic approach then. I've successfully trained an end-to-end model at small scale; am in process of getting more data (and money/compute) to scale that up.