Launch HN: Aqua Voice (YC W24) – Voice-driven text editor
Finn, who is big-time dyslexic, has been using dictation software since the sixth grade when his dad set him up on Dragon Dictation. He used it through school to write papers, and has been keeping his own transcription benchmarks since college. All that time, writing with your voice has remained a cumbersome and brittle experience that is riddled with painpoints.
Dictation software is still terrible. All the solutions basically compete on accuracy (i.e. speech recognition), but none of them deal with the fundamentally brittle nature of the text that they generate. They don't try to format text correctly and require you to learn a bunch of specialized commands, which often are not worth it. They're not even close to a voice replacement for a keyboard.
Even post LLM, you are limited to a set of specific commands and the most accurate models don’t have any commands. Outside of these rules, the models have no sense for what is an instruction and what is content. You can’t say “and format this like an email” or “make the last bullet point shorter”. Aqua solves this.
This problem is important to Finn and millions of other people who would write with their voice if they could. Initially, we didn't think of it as a startup project. It was just something we wanted for ourselves. We thought maybe we'd write a novel with it - or something. After friends started asking to use the early versions of Aqua, it occurred to us that, if we didn't build it, maybe nobody would.
Aqua Voice is a text editor that you talk to like a person. Depending on the way that you say it and the context in which you're operating, Aqua decides whether to transcribe what you said verbatim, execute a command, or subtly modify what you said into what you meant to write.
For example, if you were to dictate: "Gryphons have classic forms resembling shield volcanoes," Aqua would output your text verbatim. But if you stumble over your words or start a sentence over a few times, Aqua is smart enough to figure that out and to only take the last version of the sentence.
The vision is not only to provide a more natural dictation experience, but to enable for the first time an AI-writing experience that feels natural and collaborative. This requires moving away from using LLMs for one-off chat requests and towards something that is more like streaming where you are in constant contact with the model. Voice is the natural medium for this.
Aqua is actually 6 models working together to transcribe, interpret, and rewrite the document according to your intent. Technically, executing a real-time voice application with a language model at its core requires complex coordination between multiple pieces. We use MoE transcription to outperform what was previously thought possible in terms of real-time accuracy. Then we sync up with a language model to determine what should be on the screen as quickly as possible.
The model isn't perfect, but it is ready for early adopters and we’ve already been getting feedback from grateful users. For example, a historian with carpal tunnel sent us an email he wrote using Aqua and said that he is now able to be five times as productive as he was previously. We've heard from other people with disabilities that prevent them from typing. We've also seen good adoption from people who are dyslexic or simply prefer talking to typing. It’s being used for everything from emails to brainstorming to papers to legal briefings.
While there is much left to ...
254 comments
[ 2.1 ms ] story [ 459 ms ] threadAre you targeting developers?
My understanding was people who are serious about developing via voice use it pretty exclusively.
Like, yeah you need to learn commands, but "are often not worth it" feels like brushing a pretty massive offering under the rug.
Is learning vi / emacs commands not worth it (or shortcuts in another IDE?)
Is there a middle ground?
Our approach is based around understanding intent from speech alone. We think this will be the ideal division of labor between man and machine going forward - let the person think and the machine fit it into the document/file/text. Over time we think this will reduce the number of commands you have to learn to use it to zero.
But our "command-less" approach isn't reliable for every use case yet - and as a fan of voice interfaces I am rooting for Cursorless - it's super sci-fi.
Do you have any idea of how soon? Not looking for a public commitment to hang you with, just wondering if this is one of those "we're working on it now" (so days) or one of those "it's in the backlog" (months or maybe never depending on priorities).
The fusion model is similar to the architecture described here: https://arxiv.org/abs/2310.13289
NotSupportedError:
AudioContext.createMediaStreamSource:
Connecting AudioNodes from AudioContexts with different sample-rate is currently not supported.
I would add that this really needs to be a native app with ability to use it within Microsoft Word, which itself has a decent voice to text tool built in.
As a side benefit, you get real estate in people’s docks and desktops :)
The way Google's keyboard works on Android, but on my Linux computer (and my Android phone) would be my dream here. I'd pay $10 a month for that for sure.
What does the actual app look like though? Is it only in a browser or can I use this anywhere on my Mac?
The sandbox doesn't have typing, but the full app does - you can switch between typing and talking seamlessly there.
(written with Aqua)
Video talks about a Mac App. Where can I get that?
Voice input did not work on Edge browser on Windows, btw.
We had to make a bunch of breaking API changes over the last week and the Mac app isn't ready to go on it quite yet, but we'll bring it back as soon as we can, max two weeks, hopefully sooner.
How do you price based on value without a corollary to tokens? If you charged $40 for this service then maybe you don’t provide enough value for the casual user who does the occasional school report. On the other hand you may be unprofitable for the doctor that decides to dictate all of her interactions every day or the author who dictates an entire book.
A customer sees "word count", they understand what's going on perfectly, right away. Tokens? More than half of them will think "what, like, game tokens? do I have to buy them in advance?"
Generously, 10% of potential customers are going to have even an approximate idea of what a token means in this context, maybe 1% could tell you that words and tokens aren't quite the same thing.
Words? Minutes? Number of edits? Eg
Free - try 10 minutes active editing a month, great for trying it out
Light use - 120 minutes a month, perfect for jotting down a few things daily
Pro - 600 minutes a month, write an entire essay by voice
Ultra - unlimited. Make voice editing your main workflow and work 10 times faster
Do I have to use a specific Aqua Voice text editor, or can I use it in apps like JetBrains Rider and Visual Studio Code? If so, are there some kind of plugins that would allow using IDE-specific features? (e.g. "build and run the API project")
It's less about how quickly all that transpires and more about presenting the product in a way that doesn't require a lot of talking around it. Well done.
My first thought, when reading the headline, was that this could be useful for my coworker that got RSI in both hands and codes using special commands to a mic. But after having watched it I think it can be much more than such a niche product.
I would really happily pay $10 / month for this, but what I really want is either: - A Raycast plugin or Desktop app that lets this interact with any editable text area in my environment - An API that I can pass existing text / context + audio stream to and get back a heartbeat of full document updates. Then, the community can build Obsidian/VSCode/browser plugins for the huge surface area of text entry
Going to give you $10 later this afternoon regardless, and congrats!
>Certainly - let me grok your text!!... OK - I am ready!
BLAH BLAH BLAH...
etc
That would let me quickly build an interface for editing basically any application state, which would be awesome!
I don't even have that strong an accent, and I always try my best to enunciate correctly when talking to others shrug
A few things to try to maximize your accuracy right now are:
- Don't use AirPods, especially not AirPods Pro. Most built-in laptop mics or EarPods or a gaming headset are perfect. It doesn't need to be podcast quality.
- Correct transcription mistakes as you would a person, then "plow through" and often the error will be corrected as you complete the sentence.
I absolutely love the idea, as a fellow neurodivergent who works much better over voice than text. My only feedback is... I'd love to run this with more control. I already run LLMs locally (LM Studio), and I can run something like whisper too. I understand that open-sourcing (or even making the source code available) might go against any commercialization attempt. However, there are some options (Red Hat-esque) where it may be possible to charge for business use and allow local running for free for personal use.
On one hand you've got a solid first-mover advantage in a field where lots can benefit and use this, however if someone can bork together several layers of LLM output they might be able to offer competition (and such projects are often opensource, albeit sometimes less "polished".) If you offer a good deal you might have a good chance of major success. Best of luck!
edit: I refreshed and then it did load with the blue mic button
Excellent demo and great-looking product btw!
Also I really enjoyed your analysis
So let's say I work in a quiet home office by myself. Could I just have Aqua open throughout the day and give it notes / to-dos without having to click the microphone on/off each time?
The next most important thing would be the ability to write action routines for grammar. My preference is for Python because it's the easiest target when using chatGPT to write code. However, I could probably learn to live with other languages (except JavaScript, which I hate). I refer you to Joel Gould's "natPython" package he wrote for NaturallySpeaking. Here's the original presentation that people built on. https://slideplayer.com/slide/5924729/
Here's a lesson from the past. In the early days of DragonDictate/NaturallySpeaking, when the Bakers ran Dragon Systems, they regularly had employees drop into the local speech recognition user group meetings and talk to us about what worked for us and what failed. They knew that watching us Crips would give them more information about how to build a good speech recognition environment than almost any other user community. We found the corner cases before anybody else. They did some nice things, such as supporting a couple of speech recognition user group conferences with space and employee time.
It seems like nuance has forgotten those lessons.
Anyway, I was planning on getting work done today, but your announcement shoots that in the head. :-)
[edit] Freaking impressive. It is clear that I should spend more time on this. I can see how my experience of Naturally Speaking limited my view, and you have a much wider view of what the user interface could be.
My use case is dictating text into various applications and correcting that text within the text area. If I have to, I can use the dictation box and then paste it into the target application.
When you talk about using speech recognition for creating code, I've been through enough brute-force solutions like Talon to know they are the wrong way because they always focus the user on the wrong thing. When creating code, you should be thinking about the data structure and the environment in which it operates. When you use speech-driven programming systems, you focus on what you have to say to get the syntax you need to make it compile correctly. As a result, you lose your connection to the problem you're trying to solve.
Whether you like it or not, ChatGPT is currently the best solution as long as you never edit the code directly.
It just so happens, that many of the interfaces one has to deal with are somewhat low bandwidth. (For example, many spend most of their time stepping over, stepping into, or setting breakpoints in a debugger.) Code completion greatly cuts down the number of options to be navigated second to second. It seems like the time has arrived for an interactive voice operated AI pair programmer agent, where the human is taking the "strategic" role.
For those who don't know what happened next, and why Dragon seem to stagnant so much in the aughts, the story about how Goldman Sachs helped them sell to essentially Belgian Enron, months before they collapsed, was quite illuminating to me, and sad.
https://archive.ph/Zck6i
> Professor Gompers opined that at the time the acquisition closed, Dragon was a troubled company that was losing money and had regularly missed its own financial projections. It was highly uncertain whether Dragon could survive as a stand-alone entity. Professor Gompers also showed that technology stocks were on a downward trend, and L&H was the only buyer willing to pay the steep price Dragon demanded. Thus, he concluded that if the company had not accepted the L&H deal, Dragon likely would have declared bankruptcy. The jury found in favor of the defendants and awarded no damages to the plaintiffs.
We want to get Aqua into as many places as possible — and will go full tilt into that as soon as the core is extremely extremely solid (this is our focus right now).
Great lessons from Dragon Dictation. Would love to learn more about the speech recognition user group meetings! Are those still running? Are you a part of any?
FWIW, I am fleeing Fusebase, formally known as Nimbus, because they "pivoted" and messed up my notetaking environment. In the beginning, I went with Nimbus because it was the only notetaking environment that worked with Dragon. After the pivot, not so much. I'm giving Joplin a try. Aqua might work well as an extension to Joplin, especially if there was a WYSIMWYG (what you see is mostly what you get) front-end like Rich Markdown. I'd also look at heynote.
I use voice-to-text in the workshop and when taking notes and reviewing a PR. And all the current options are pretty much what you would expect. More focused on accuracy, which is usually quite poor, which, to paraphrase, "It's Erin with an E. Oh for **s sake, Erin. ERIN! E. R. I. N. <pause> N. I said N. Eh-rin. Fine. Whatever." so anything that can improve on that experience will be immensely helpful.
Looking forward to seeing where you go with this, and I hope at some point you make a native desktop application.
On pure WER we are state-of-the-art in our testing, but more importantly, mistakes in Aqua are correctable.
So you can speak your mind instead of having to wait until you have the perfect sentence and then dictate it.
That said, we know it's not perfect, but we know a few more months of work will have it really solid.
Jiminy Crickets...
I have SOOO many use cases for your thing.
[edit: what does this mean: https://i.imgur.com/rHQt6ul.png when attempting to demo?]
---
* I want an agent that I can speak to on Mobile headset as I love to think out loud - and air my thoughts and thought process through talking through my internal dialogue - if this could just capture what I am saying and log it and I can refine thoughts as I go.
For example - I ride a lot. I try to cycle 1000 miles a month if I am doing a solid month - but else - I ride daily and its a movement meditation. as I ride - I think through things and I speak through thought processes with differing opposing 'experts' in my internal monologue to self-argue through to a solution....
If I could have this record all that, then random epipehnies I think through while on ride will be captured in a meaningful way.
---
* A meeting-notes-transcriber for whiteboard sessions.
* record everything you say in an interview and be able to review after for self-coaching
* talking through a dish as you wing the ingredients so that you speak out loud what you did (my grandmother was friends with Julia Child - my grandmother taught me to cook and when it came to measurements of things - they always wing it per feel/taste "salt to taste" for example means "eh... whatever"
so to be able to talk through what your 'winging it with' and it captures it into a salient reproducible recipe (i make a mean Chimi Churry (sometimes if I can recall)
* a voice "body cam" for things I may say in situations where I may be too flustered to recall.
* Speak authoring - start telling a story outline so it captures a synopsis that you can further develop
* Speech (like giving a speech) refinement as you can talk through the speech and capture and rework and reiterate etc
and thats just off the top of my head through your demo....
LOVE this.
Sorry about the error you are getting! It's a Firefox thing. We will patch. In the meantime, Chrome/Safari will work
I just want to qualify - you did not inspire these ideas.
These are desires sought which have been there for eons...
You are not inspiring them
You have atool that ENABLES them.
Seek that which already is a flustered pop of ideas waiting for a release valve for such thought.
You are not inspiring - you are enabling that which is already there, think of it as which valve to open - the pressure is mounting upon your dyke.