Show HN: An open source framework for voice assistants (github.com)
On the theory that something like a LlamaIndex or LangChain for real-time/conversational AI would be useful, a few of us started working on a Python library for voice (and multimodal) AI assistants/agents.
So ... Pipecat: a framework for building things like personal coaches, meeting assistants, story-telling toys for kids, customer support bots, virtual friends, and snarky social bots.
Most of the core contributors to Pipecat so far work together at our day jobs. This has been a kind of "20% time" thing at our company. But we're serious about welcoming all contributions. We want Pipecat to support any and all models, services, transport layers, and infrastructure tooling. If you're interested in this stuff, please check it out and let us know what you think. Submit PRs. Become a maintainer. Join the Discord. Post cool stuff. Post funny stuff when your voice agent goes completely off the rails (as mine sometimes do).
40 comments
[ 272 ms ] story [ 1829 ms ] threadThe demo on real-time multi language translation conversation blew me away!
Building pipelines for bridging LLMs and TTS and STT models with lower latency is fine and all, but when you compare to a natively multimodal model like GPT-4o it seems strictly inferior. The future is clearly voice-native models that are able to understand nuances in voice and speech patterns, and it's not exactly a distant future.
As soon as GPT-4o audio input is available through the APIs, we'll add 4o support to Pipecat. For bidirectional real-time audio, I think they'll need to make new WebSocket or WebRTC endpoints available.
Edit: someone found one: https://news.ycombinator.com/item?id=40346992
We should probably update the example in the README to better represent that, thank you!
But..I looked at the code but didn't see any audio-to-audio service or model. Can you link to an example of that?
I don't mean speech to text to LLM to text to speech. I mean speech-to-speech directly, as in the ML model takes audio as input and outputs audio. As they have now in OpenAI.
I am very familiar with the typical multi-model workflow and have implemented it several times.
For context relating to real-time voice AI: once you're down below ~800ms things are fast enough to feel naturally responsive for most people and use cases.
The GPT-4o announcement page says they average ~320ms time to first token from an audio prompt. Which is definitely next level and is really, really exciting. You can't get to 800ms with any pipeline that includes GPT-4 Turbo today, so this is a big deal.
It's possible to do ~500ms time to first token by pipelining today's fastest transcription, inference, and tts models. (For example, Deepgram transcription, Groq Llama-3, Deepgram Aura voices.)
From what I can tell, Siri is still a dumpster fire that nobody is willing to use. And I have no personal experience with Alexa, so I can't speak to it. But I do have a few Google Home speakers and an Android phone, and I have seen no major improvements in years. In fact, it has gotten worse - for example, you can no longer add items directly to AnyList[0], only Google Keep.
Or, as an incredibly simple example of something I thought we'd get a long time ago, it's still unable to interpret two-part requests, e.g. "please repeat that but louder," or "please turn off the kitchen and dining room lights."
I find voice assistants very useful - especially when driving, lying in bed, cooking, or when I'm otherwise preoccupied. Yet they have stagnated almost since their debut. I can only imagine nobody has found a viable way to monetize them.
What will it take to get a better voice assistant for consumers? Willow[1] doesn't seem to have taken off.
[0] https://help.anylist.com/articles/google-assistant-overview/
[1] https://heywillow.io/
edit: I realize I hijacked your thread to dump something that's been on my mind lately. Pipecat looks really cool, and I hope it takes off! I hope to get some time to experiment this weekend.
I use both (albeit more Alexa than Siri, both just for a really limited functionality set), and FWIW, I believe Alexa is worse than Siri. It can do two things at the same time though (just as your example: "turn on X and turn off Y", "turn on X for Y seconds", and things like that).
I also feel that it has gotten worse over the years. I read about the possibility of microphones getting dust and therefore capturing worse audio, so I got a dust blower (for other reasons, too), but it didn't solve anything.
After listening in the app what Alexa picks up (from an Echo and Echo Dot, both 4th. Gen), I have to say that they use really shitty microphones. Furthermore, I have been testing Whisper extensively last month, with audio coming from low-quality sources, and I think a similar model would interpret a lot better my voice than whatever Amazon is using.
And it does fine with no internet access.
Except dictation. Much better with internet access than without.
Another issue Siri used to struggle was trying to play specific music on Spotify. Is that better these days?
I asked to play a song my an artist on Spotify and it did it. Popped up in the Dynamic Island.
Honestly, I didn’t know it could do that!
(It did ask permission to access Spotify data first, but only that first time).
Our car has Google Assistant, and yeah that's annoying. Want to turn off steering wheel heater and seat heater? Gotta do two individual requests.
That said, it's actually quite nice to have voice control over these things. Especially when it's heavy traffic and snowing on top of the icy road, and you really want to have eyes on the traffic and both hands on the steering wheel.
Yes! I really think voice assistants are underrated. When I talk to iOS users, they have a much less favorable opinion of Siri (and I've watched my partner give up on using it over the past 10 years) and given that iOS has dominant market share in the US, I suspect this is a part of it.
But I also think there is just so much "low hanging fruit" that would drastically improve the experience. But I remember that even during "the race" for voice AI, everyone was wondering... how will they monetize this? And I'm not sure anyone was ever truly able to figure that out.
Alexa is a dumpster fire and constantly getting dumber. She also completely disrespects your settings and will re-enable settings you have disabled. She constantly ignores my questions to ask me if I want to try some other new feature instead. She randomly decides to add news stations I have explicitly removed from my Flash Briefing list.
I am constantly baffled by how bad it is.
Give it eyes?
https://www.youtube.com/watch?v=nXVvvRhiGjI
(I try to keep up with all the different layers/players in this space.)
https://github.com/livekit/agents
Whisper is too slow and doesn’t allow interruption.
Which means you can connect a bot to a call, and tell it dialout to a phone number and it will.