Launch HN: Hamming (YC S24) – Automated Testing for Voice Agents
LLM voice agents currently require a lot of iteration and tuning. For example, one of our customers is building an LLM drive-through voice agent for fast food chains. Their KPI is order accuracy. It's crucial for their system to gracefully handle dietary restrictions like allergies and customers who get distracted or otherwise change their minds mid-order. Mistakes in this context could lead to unhappy customers, potential health risks, and financial losses.
How do you make sure that such a thing actually works? Most teams spend hours calling their voice agent to find bugs, change the prompt or function definitions, and then call their voice agent again to ensure they fixed the problem and didn't create regressions. This is slow, ad hoc, and feels like a waste of time. In other areas of software development, automated testing has already eliminated this kind of repetitive grunt work — so why not here, too?
We were initially working on helping users create evals for prompts & LLM pipelines for a few months but noticed two things:
1) Many of our friends were building LLM voice agents.
2) They were spending too much time on manual testing.
This gave us evidence that there will be more voice companies in the future, and they will need something to make the iteration process easier. We decided to build it!
Our solution involves four steps:
(1) Create diverse but realistic user personas and scenarios covering the expected conversation space. We create these ourselves for each of our customers. Getting LLMs to create diverse scenarios even with high temperatures is surprisingly tricky. We're learning a lot of tricks along the way to create more randomness and more faithful role-play from the folks at https://www.reddit.com/r/LocalLLaMA/.
(2) Have our agents call your agent when we test your agent's ability to handle things like background noise, long silences, or interruptions. Or have us test just the LLM / logic layer (function calls, etc.) via an API hook.
(3) We score the outputs for each conversation using deterministic checks and LLM judges tailored to the specific problem domain (e.g., order accuracy, tone, friendliness). An LLM judge reviews the entire conversation transcript (including function calls and traces) against predefined success criteria, using examples of both good and bad transcripts as references. It then provides a classification output and detailed reasoning to justify its decisions. Building LLM judges that consistently align with human preferences is challenging, but we're improving with each judge we manually develop.
(4) Re-use the checks and judges above to score production traffic and use it to track quality metrics in production. (i.e., online evals)
We created a Loom recording showing our customers' logged-in experience. We cover how you store and manage scenarios, how you can trigger an experiment run, and how we score each transcript. See the video here: https://www.loom.com/share/839fe585aa1740c0baa4faa33d772d3e
We're inspired by our experiences at Tesla, where Sumanyu led growth initiatives as a data scientist, and Anduril, where Marius headed a data infrastructure team. At both companies, simulations were key to testing autonomous systems befor...
70 comments
[ 2.6 ms ] story [ 137 ms ] threadBut also remember -- the point of the economy is not jobs. It's value creation. If we can create the same or greater value with fewer people working/people working less, that's a great result! And it's the result we've seen continue over the last century and a half, even while people became much wealthier (because they were choosing to exchange some of that wealth for less working time).
I don't think reducing the need for human labor is inherently bad, but our current society seems to be heavily centered around finding work.
But also, the number of remaining roles isn't fixed. Jobs exist (at least in the private sector) because they create more value than they cost to fill, and we're always finding new and expanded ways for people to create more value. Saving resources through automation just means we can redirect that value creation somewhere else.
Ultimately this is how economies grow and the world becomes wealthier over time; we increase the value of people's time because there's so much competition for it, to the point where we can then more cheaply automate some or part of the job. If the supply of labor gets too large for the uses we can find for it, prices for labor fall, and the relative cost of automation is increased.
Testing for edge cases is especially important because getting an order wrong can cause health hazards, long line-ups, and churn!
Our current product draws inspiration from Hamming distance because we're comparing the `distance` between current LLM output vs. desired LLM output.
I'm happy to spin up some scenarios that are more relevant for you instead of our stock demo personas :)
Feel free to email me at sumanyu@hamming.ai
I'm most excited to see well-done concepts in this space, though, as I hope it means we're fast-forwarding past this era to one in which we use AI to do new things for people and not just do old things more cheaply. There's undeniably value in the latter but I can't shake the feeling that the short-term effects are really going to sting for some low-income people who can only hope that the next wave of innovations will benefit them too.
This will work with a https://www.pipecat.ai type system? Would love to wrap a continuous testing system with my bot.
It should be pretty straightforward at first glance!
Selling shovels on a gold rush seems to have become the only one mantra here.
Making current voice agents reliable is incredibly time-consuming and complex. This challenge has kept many teams from pushing their agents into production. Those who do launch often release a very limited, basic version to minimize risk. We frequently talk to teams in both camps.
As a result, there aren't many 'killer' voice products on the market right now. But as models improve, we'll see more voice-centric companies emerge.
Teams are already calling their agents by hand and keeping track of experiment runs in a spreadsheet. We're just automating the workflow and making it easier to run experiments!
It would essentially be another form of a behavioral interview. I wonder if this exists already, in some form?
Few examples:
1) Customer service: Simulating challenging customer interactions could help reps develop patience and problem-solving skills.
2) Emergency responders: Creating realistic crisis scenarios (like 911 calls) that could improve decision-making under pressure.
3) Healthcare: Virtual patients with complex symptoms could speed up the learning rate for med students.
4) Conflict resolution: Practicing with difficult personalities could aid mediators and negotiators.
5) Sales: AI-simulated tough customers could help salespeople refine their pitches and objection-handling skills in a low-stakes environment.
Thoughts?
In a bad case, I envision a ton of companies or institutions employing very strict & narrow situations to the point where they only accept a very homogenized personality. It could end up creating a stiff or worse culture than if they had naturally accumulated a diverse population, if that makes sense. Discrimination already exists, but would be made a lot easier, automated, and commonplace.
In a good case, extremely antisocial behavior (situations that are "softballs" or "hard to screw up for reasonable people") could be easily caught at scale and addressed an early age. Plus the cases you've listed, eliminating the need for special attention and mentorship from the limited people we meet irl.
I'm sure there are other horrible or amazing cases I'm missing.
So as all tools are, it would depend. Whether this will actually benefit more than harm will depend on the society you place it in, and I'm not sure I have that much faith in the corporate world.
We've discovered that it's often faster for non-technical domain experts to iterate on prompts in a structured, eval-driven way, rather than relying on engineers to translate business requirements into prompts.
While storing prompts in code offers version control benefits, it can hinder collaboration. On the other hand, using a pure CMS for prompts enhances collaboration but sacrifices some modern software development practices.
We're working towards a solution that bridges this gap, combining the best of both approaches. We're not there yet, but we have a clear roadmap to achieve this vision!
If you're going to develop AI voice agents to tackle pre-determined cases, why wouldn't you just develop a self-serve non-voice UI that's way more efficient? Why make your users navigate a nebulous conversation tree to fulfill a programmable task?
Personally when I realize I can only talk to a bot, I lose interest and end the call. If I wanted to do something routine, I wouldn't have called.
For example, if you had an existing IVR system and you tracked menu options and found that a significant portion of calls were able to be answered by non-smart pre-recorded messages, upgrading to an AI voice agent could be a reasonable improvement.
- I am dictating this message through macOS's voice to text right now
- I am a huge user of Google Assistant
- I prefer to call people versus texting them
- I tend to call restaurants instead of using something like Toast to order takeout (although this is partially because online services will add a surcharge onto the price sometimes, and sometimes I need to ask questions about dietary restrictions, etc.)
Generally, wherever possible, I will use a voice interface versus a text based one to get my point across. It's just faster and more convenient for me. I'm pretty neutral on the consumption side: I read and listen to audiobooks in roughly equal amounts.
All that to say that, just like there are people out there who prefer text UIs, there are also people who prefer voice interfaces.
Sometimes, I even ‘talk’ into Cursor’s chat window instead of typing. The only downside? It can get a bit annoying for others when you're talking to yourself all day.
I guess I understand that a lot of things are being developed for Apple silicon specifically. It's just frustrating that despite hours of searching, I'm not finding anything decent.
https://talonvoice.com/
It's a bit overkill to use Talon for just voice dictation, but that is 90% of what I use it for, and it's pretty good at it.
For example, when I need to activate a new SIM card, I need to call the company to get it activated. But if I’m talking to an AI agent at that point, why not have me go through another channel (website/app?) to activate it?
Frankly I'm surprised there isn't already some sort of NFC info transfer system in fast food restaurants' apps that lets you and everyone in your car enter your order while you're waiting in line, then knows when your car is up and brings you the food. Have the voice part be a fallback tier, not the primary one.
My grocery store can know when I'm arriving and bring out my food, based on location services on my phone. So can Walmart or Home Depot. Granted, they make me wait a couple hours until they notify me that my order is "ready" before I come get it.
I suppose it's possible this does exist and I just haven't seen it because I don't drive through fast food restaurants, but I don't get why a place that primarily takes orders in real time and hands them out the window can't broaden the way to submit them to include on-site online orders as well as "talk to our agent over a glorified walkie talkie" orders.
It worked quite well, and surprising coming from RyanAir.
If it did, wouldn't all the companies with production AI text interfaces be using similar techniques? Now being able to easily replay a conversation that was recorded with a real user seems like a huge value add.
Regarding text-based evals — part of testing voice agents involves assessing their core reasoning logic. To do that, we bypass the voice layer and simulate conversations via text. So yes, the core simulation engine is reusable for both conversational text and voice interactions.
We're also excited about shipping the ability to replay a simulated conversation inspired by a real user!
would love to have something like this integrated as part of our open source stack.
I'll reach out async!
99% of the time, we can just build a simple intent flow off of dialogflow pointing to the customer's API endpoints that will return that data. No where here do we need an LLM / RAG since their endpoint already points to that answer. Hope that makes sense!
Building product, talking to customers and making something people want!
We're working on a fix right now!
How will this really check that the models are performing well vs just listening?