Interesting! I wonder how UI will evolve in the long-term? If there are browser-use/computer-use and clicky-clones automating pointer actions, do we really need complex UI anymore? If yes, when?
This seems like one of those things that is usable infrequently enough to be forgotten/poorly developed/never used. (Even before accounting for the actual failure rate of the LLM which will be none-zero).
Perhaps a text box and file upload isn’t the perfect interface for every use case but it is versatile which is a huge barrier to overcome.
It reminds me of Microsoft Recall in the sense that some portion of the screen is going to be continuously transmitted outside of the users control.
What happens when someone browses something very private (planning a surprise engagement. looking at medical data. planning a protest)? All that data gets slurped to google and subject to a warrant or discovery or building your advertising fingerprint.
Maybe the idea is that the data is sent to AI only when you right click, but that seems like a very thin firewall that a product manager will breach in the interests of delivering "predictive AI" via some kind of precomputed results.
My first impression coming away from this is skepticism.
Anything with voice controls for routine use is a pretty tough sell. Doing this when you're not completely alone would be annoying to everyone around you.
Most of their examples seem like they could have been done with a right click drop down menu so they don't really need to "re-invent the mouse pointer".
So is this thing talking to Google's servers all the time for the AI integration? So it won't work if you're not connected to the internet? Privacy concerns are obvious; now Google wants to have an AI watching literally everything you do on your computer?
Does it cost the user anything for the LLM use? If it's free will it stay free forever? That's quite a lot to give away if they're expecting people to use it to change a single word like in one of their examples. I guess they're expecting to make the money back by gathering data about literally everything you do on your computer.
There might be a killer app for AI integration with personal computers that has yet to be invented, but this doesn't look like it.
This low effort AI shoveling reminds me of how they keep trying to make fridges that tell you when you need to buy milk or auto-buy milk. They have been pushing that idea for decades now, but it ignores the biggest issue that AI has laid bare: it sucks without accurate context.
What if I am going on vacation next week? What if I need extra milk for a dinner I am planning? What if my kid puts the milk in the fridge sideways and it no longer detects it?
"Easy fixes" to easy problems never work because they add mental load to tasks we already manage capably. Yes we no longer have to think about buying milk when it gets low, which was a stable pattern. But we replace it with a nondeterministic "milk state" that we need to be constantly vigilant about and manually adjust any time our routines are altered - exactly when we don't want to stack on more overhead.
AI is discretely useful, tremendously so, but big tech loves to default to umbrella solutions before there is a rich context to reasonably support it. The real world is messy.
General AI product tip: show your tool fixing a messy problem not a happy path problem. That's where AI is impactful!
> Most of their examples seem like they could have been done with a right click drop down menu
Right-click menus can get cumbersome. I've seen a lot of software that suffers from function bloat - not that the functions don't work, or don't play well together, but that the user interface becomes too overwhelming for users as the number of available actions explodes. This is particularly tough for new users.
This is where voice controls could shine: as we interact with computers in more and more complex ways, we need a way to specify our desires simply and easily. And if we can't do so easily, the software has to remain simple to be usable.
Oh interesting, this is very cool. At first I thought it was just focus-follows-mouse but it's more interesting. You have certain keywords trigger "add to prompt". Ignoring the voice functionality (which is admittedly crucial currently because other inputs currently take over focus), I've often wanted to just have a continuous conversation with the LLM as I 'point and click' (or tab over and select) at various things. Might be neat to have text input focus continue to go to the LLM where I'm typing text etc.
Sometimes I go to a different page to take a screenshot and other times I'm browsing for a file, and other times I'm highlighting some log lines. Cursor did this well, with selecting text in the terminal auto-focusing the Cursor agent textbox so you could talk to the agent and then select some text and you didn't have to re-select the original agent textbox again. The agent is a top-level function in that system not "just another app I have to switch to" to take my context with.
I have some small amount of bias because I've always felt input-constrained on computers. I have to move my hands to go places and that's exasperating. I've tried head tracking, had a vim pedal for a while, and used tiling WMs, and things like this to aid but while my vim-fu is pretty good and I function inside things very well with it, my cross-application interface isn't.
In the end, perhaps we all have our home offices with our Apple Vision Pros and we talk to them like this to maneouvre faster through our machines and get our ideas into them.
> I have some small amount of bias because I've always felt input-constrained on computers. I have to move my hands to go places and that's exasperating. I've tried head tracking, had a vim pedal for a while, and used tiling WMs, and things like this to aid but while my vim-fu is pretty good and I function inside things very well with it, my cross-application interface isn't
Why not constrain your computing? It will require some programming chops, but you can note down your common tasks, figure out where actual input are required, and automate the rest.
104 comments
[ 4.8 ms ] story [ 76.6 ms ] threadWe couldn't quite track you well enough before. So we're fixing that under the guise of "AI powered capabilities."
Perhaps a text box and file upload isn’t the perfect interface for every use case but it is versatile which is a huge barrier to overcome.
Nightmares are dreams as well and this is a nightmare like Windows Recall.
Technically wonderful though.
(Not going to happen)
It reminds me of Microsoft Recall in the sense that some portion of the screen is going to be continuously transmitted outside of the users control.
What happens when someone browses something very private (planning a surprise engagement. looking at medical data. planning a protest)? All that data gets slurped to google and subject to a warrant or discovery or building your advertising fingerprint.
Maybe the idea is that the data is sent to AI only when you right click, but that seems like a very thin firewall that a product manager will breach in the interests of delivering "predictive AI" via some kind of precomputed results.
Anything with voice controls for routine use is a pretty tough sell. Doing this when you're not completely alone would be annoying to everyone around you.
Most of their examples seem like they could have been done with a right click drop down menu so they don't really need to "re-invent the mouse pointer".
So is this thing talking to Google's servers all the time for the AI integration? So it won't work if you're not connected to the internet? Privacy concerns are obvious; now Google wants to have an AI watching literally everything you do on your computer?
Does it cost the user anything for the LLM use? If it's free will it stay free forever? That's quite a lot to give away if they're expecting people to use it to change a single word like in one of their examples. I guess they're expecting to make the money back by gathering data about literally everything you do on your computer.
There might be a killer app for AI integration with personal computers that has yet to be invented, but this doesn't look like it.
I assume they're using on-device Gemini Nano: https://developer.android.com/ai/gemini-nano
Depends on how many hands you've recently broken?
I'm surprised sub-vocal HCI isn't better developed by now. Perhaps because of this stuff coming out it will be.
Humans speaking to one another is literally telepathy: I'm putting my thoughts in your head, with lots of ambiguity and noise, of course.
With better sub-vocal tech we can control our devices without bothering each other.
https://www.media.mit.edu/articles/exclusive-startup-lets-yo...
What if I am going on vacation next week? What if I need extra milk for a dinner I am planning? What if my kid puts the milk in the fridge sideways and it no longer detects it?
"Easy fixes" to easy problems never work because they add mental load to tasks we already manage capably. Yes we no longer have to think about buying milk when it gets low, which was a stable pattern. But we replace it with a nondeterministic "milk state" that we need to be constantly vigilant about and manually adjust any time our routines are altered - exactly when we don't want to stack on more overhead.
AI is discretely useful, tremendously so, but big tech loves to default to umbrella solutions before there is a rich context to reasonably support it. The real world is messy.
General AI product tip: show your tool fixing a messy problem not a happy path problem. That's where AI is impactful!
Right-click menus can get cumbersome. I've seen a lot of software that suffers from function bloat - not that the functions don't work, or don't play well together, but that the user interface becomes too overwhelming for users as the number of available actions explodes. This is particularly tough for new users.
This is where voice controls could shine: as we interact with computers in more and more complex ways, we need a way to specify our desires simply and easily. And if we can't do so easily, the software has to remain simple to be usable.
[0] https://news.ycombinator.com/item?id=48107027
Aaaaand now I can't remember the name of it
Sometimes I go to a different page to take a screenshot and other times I'm browsing for a file, and other times I'm highlighting some log lines. Cursor did this well, with selecting text in the terminal auto-focusing the Cursor agent textbox so you could talk to the agent and then select some text and you didn't have to re-select the original agent textbox again. The agent is a top-level function in that system not "just another app I have to switch to" to take my context with.
I have some small amount of bias because I've always felt input-constrained on computers. I have to move my hands to go places and that's exasperating. I've tried head tracking, had a vim pedal for a while, and used tiling WMs, and things like this to aid but while my vim-fu is pretty good and I function inside things very well with it, my cross-application interface isn't.
In the end, perhaps we all have our home offices with our Apple Vision Pros and we talk to them like this to maneouvre faster through our machines and get our ideas into them.
Cool research. I wonder what we'll end up with.
Why not constrain your computing? It will require some programming chops, but you can note down your common tasks, figure out where actual input are required, and automate the rest.
I like text selection exactly how it is. I want precise controls.
It's fine for a touch interface like a phone, but on a computer I expect precision. As much as I can get.