What terminal / agentic workflows spawn the demonstrated dialogue boxes that require the User to click/reject the action? Aren't most such flows actually inline UIs? And finally, if the core issue is that these confirmation boxes are tied to the terminal that triggered them, which does not autofocus, what's the point of said "big arrow" that is also lurking behind without focus?
If the said "big arrow" automatically gains focus, isn't the real fix here to just make the dialogue boxes themselves gain focus automatically? Both are similarly disruptive anyway.
I think this would’ve been very handy to me a couple years ago when I was learning to use Blender and asking LLMs for help performing certain actions like “how do I display the normals of all the vertices in my mesh?” I spent a lot of time trying to figure out which button the model was talking about.
Interesting. When I read the headline I imagined this would be a sort of thinking trace booster -- letting the agent focus its own attention on different parts of the screen. But this is cool in a different way. I bet agentic harnesses would find it useful for communicating with other agents / themselves as well.
What a time to be a radical centrist - the AI haters seem out of touch, the AI thought leaders can't stop huffing their farts and being condescending, and somehow this is on the top of HN. What a silly time.
This could be great for documentation. Screenshots in docs are frequently useless because they show me a screen and say "click X" where I still have to search X visually. And I could just to dhat in the other tab I have open.
It doesn’t make sense that you have to read a whole article and then still search for the buttons in the UI afterwards. And with longer articles, you always have to keep the article open next to your product to follow the whole flow.
For the humor impared, it would also be useful to have a colorful animated "WHOOSH" overlay with sound effects for every time a deadpan joke goes over your head. ;)
Last time I accidentally said something sarcasticly over-ambitious to an LLM, it shipped this popup callout tooltip feature on a PDP-7 Type 340 vector graphics display emulator that shows you the meaning of the drawing you're pointing at, as well as the address of the instruction that drew it.
The README is geared towards technical people (complete with the HN screenshot) but when I see a tool like this all I can think of is how helpful this would be for my mom when I'm trying to tell her how to download and print a document over the phone
My message to the world is that LLMs should be able to point anything they see in the application they're in or even the whole computer (if you give it that kind of access).
For web apps, WebMCP is a way where you can make a frontend way more discoverable to an LLM rather than that it is going to read all your code. I sometimes create this in my apps at home and sometimes in my apps at work and it makes LLM assistants way more usable. It also helps if they can highlight certain things of a visual.
We humans can do this too on paper. We can point at something, we can highlight something with a marker. Why not give an LLM these capabilities as well?
I would pay good money for something like this on iPad. It wouldn't even need to be agent-driven, just a big "I want to make a bank transfer" button, that would launch the correct app and guide through the interface.
That would be a godsent for those of us with aging parents.
> That would be a godsent for those of us with aging parents.
Right yeah? I mean I have freagin remote desktop installed on all my parent's devices. I am the agent for them every time they call frantically for help.
It's Time to redesign the apple genius bar for the modern aging boomer using this.
I tried to read the "Does it need Screen Recording or Accessibility?" part, but it's slopped to the point I have no clue what it's trying to say. But if it can draw on top of permission prompts, what's stopping it from drawing box that hides the "decline" button and changing the "approve" button copy?
I wonder if someday we will get to the point where Github repos are just markdown files describing the project and then you just let your own agent implement it because why the hell would I trust your agent's implementation?
That sounds cynical today, but that's just because the AI models are currently outrunning enshittification. I'm already pondering personal plans about what to do when that turns around. We haven't seen enshittification yet that is going to be like the enshittification of AI. It may even deserve a new term of its very own, it's going to be such a big problem. The AI companies are leaving a lot of value on the table to entice us on to their systems but at some point that's going to turn around.
The existential danger exists today: you're providing your entire business toolchain to models in the cloud, owned by businesses who are for profit entity. Even if the model itself doesn't care about your data, the business model does.
Everything your building is now streaming through a third party who could steal it, the same way Amazon steals open source software and repackages it.
All you need to do is look at palentir's forward deployed engineers. Once you let a third party observe your processed, you're cloneable and replaceable _as an entity_.
I think we can look at our own human psychology as it developed into social groups. If our brains were obviously deterministic, it would be easy for people to prey upon us via structured inputs. Some level of randomness in our thought process is almost evolutionarily required to avoid becoming part of the larger super structure, similar to an ant colony.
So these AI companies are definitely more dangerous than just the "AI is super intelligence". They're basically a intelligence platform everyone is wiring into.
That's today. Tomorrow, they'll be run by an MBA which is the enshittification.
Right, but your example presumes tpdays facebook market. Im presuming todays market where people are feeding all their ideas to a third party, much like how open source software is pilfered and turned into business ventures.
Now even if you want your idea to compete with all the other todo apps with your special sauce, youre giving a third party your recipe. That third party ostensibly has no reason to steal it, their business model does.
So, sure, if your app is a cult of personality and nothing special, i admit it doesnt matter.
But the number of software home runs is dwarfed by all other auccess on individual products
Why would OAI or anthropic care to steal your Todo app with 10k of revenue a month when they are spending billions?
At their scale I'd expect it to be only worth stealing potential home runs and even so, imagine they did this a couple of times, how would that make them look? Do you think it would be worth the bad blood and mass exodus that would follow?
> Some level of randomness in our thought process is almost evolutionarily required to avoid becoming part of the larger super structure, similar to an ant colony.
Not really. Ants are superorganisms, they share more than half of their DNA with each other. You could think of each ant as being like a cell in a human.
yes, ok, but the point is: evolution has allow humans to continue to make stupid choices, and those stupid people still survive.
So atleast pare of our collection of genes has added value as with random decision factors rather than find some supergene that everyone benefits from.
Seriously. I imagine this will be the case soon enough in some shape or form.
Whenever I see these vibecoded apps, my move is to just do what you've described: point an agent at it and tell it to reproduce it (usually with my own little customizations). It's a bit absurd, but I don't trust that they vibecoded the app as well as "I" could lol.
A 100%! Just imagine you could describe every solution to a problem just using language! Of cause you would to make sure to avoid ANY missunderstanding, but therfore you could invent a language specified for avoiding ambiguity. I could imagine just to reduce the words to a very small corpus so everybody can remember it and have a very strict grammar so a program can effortlessly check correctness of its sentences.
Well, I've always thought we are stuck in a local minimum when it comes to programming languages. Like, programming shouldn't be this hard and shouldn't require so much boilerplate after almost a century. If the actual important logic can be inferred pretty well from markdown specs, that means most of what we write in JavaScript, Go, Java is redundant.
Hopefully someone smarter than me is working with trying to develop a new programming language given what we know to be possible with LLMs
Well it "works" only because LLMs have the context baked in. The problem is, that this context is something like world-knowledge. Defining the context you need for your exact problem, e.g. your company, will add so much more information that programming is probably the better way.
This only works while tokens are artificially cheap. The gravy train could come to an end at some point.
The open models that are chasing the frontier labs will stop being open once things slow down and there is less incentive to undercut the front runners. Time will tell if GPU compute gets cheap enough to run stuff locally.
With enough model drift that won’t even work over time. These files would have to be pinned to the intended model version and that is either used directly or emulated with a faithful emulator in a larger model.
I doubt that will be a problem. Models are different enough right now. If you can write a spec with enough detail and constraints today to get valid and comparable output from say GPT-6 Astra, Opus 5.5, DSV4F, Kimi K3, GLM 5.3, etc... Then I think there's a good chance that whatever SoTA coding LLMs everyone is using 3 years from now will also be able to implement that same spec.
Again, I think it heavily depends on if people are writing comprehensive specs with sufficient detail.
That depends on if you are really getting what you expect from the spec or you are actually relying on undefined behavior for your expected output from the spec. This is why we often pin software library versions: we may very well be relying on unintended or a bugged behavior to get out expected output.
Nothing stops it, except the fact that the agent is already executing arditrary code in your shell. If it's malicious, it'll just steal your shh keys directly instead of bothering with button masking
The fact that the utility doesn't give it the ability to do that?
It makes sense to not trust AI models with the ability to read and alter your screen for many many many reasons, but the developer of the tool knows that as well. A tool that lets your ai do something doesn't also let it do anything.
So, as always, it's a question of if you trust the software producer. You're giving them the ability to draw on your screen too. Would you have the same fear if there were no AI involved?
For stuff like this I imagine AI misbehavior as a novel failure trigger not a novel failure type. It can only do the harm that the software was able to do on its own already.
Man, Ive lost count of How many times have I had to repeat this over the phone to my parents; The repo has all the punch lines in the README
> "It's this window, not that one." You have 14 Chrome windows. The agent knows which one it means: --window "Google Chrome:Pull request". It even picks the right tab: --app "Google Chrome:Pull request".
> Remote help. "No, the other gear icon." Point at it instead of describing it.
While it's vibe-coded, I'm immensely enjoying the creativity which AI has enabled. These apps would never have been built before, and I find it so fun to see how AI is being used to provide (semi) useful things for people who need it.
I found this awesome vibe-coded Pixel News Network project recently. How fucking cool is this?? Would never have been created otherwise. https://pnn.watch/
I really want to understand your perspective here. Like, what's the creative aspect you identify here? It's literally just... drawing arrows on the screen, for no discernible purpose, and there doesn't seem to be any clear artistic intent behind it? In the demo screenshot, the arrows are ugly, some of them are misaligned (e.g., the orange and the green one in the first image), and the text is just weirdly verbose and not even consistently trying to be witty.
So the only statement that gets through, in my view, is that code generation is so cheap that we're no longer even concerned about having a clear idea for what we're trying to do or why the code should exist in the first place.
Meh, would have been cool-ish a few years back, now everyone and their grandmas could build something similar given enough tokens. Hard to be excited when similar gadget projects are a dime a dozen. I also don't think that AI use enables much in the way of creativity. It allows people to delegate their judgment to the model, which defaults to humdrum middle-of-the-road choices. Only the very high-level decisions remain personal, leading to a lack of texture past the first glance, so to speak. Not saying that it cannot be used with intentionality, just that it rarely is.
I actually somewhat enjoyed the times when vibed projects were a broken mess, lots of accidental comedy in that era.
199 comments
[ 1.6 ms ] story [ 34.5 ms ] threadGrim. If you're just there to click sudo buttons for the bot, you might as well give it root access and be done.
What terminal / agentic workflows spawn the demonstrated dialogue boxes that require the User to click/reject the action? Aren't most such flows actually inline UIs? And finally, if the core issue is that these confirmation boxes are tied to the terminal that triggered them, which does not autofocus, what's the point of said "big arrow" that is also lurking behind without focus?
If the said "big arrow" automatically gains focus, isn't the real fix here to just make the dialogue boxes themselves gain focus automatically? Both are similarly disruptive anyway.
"Tech me how to use this app myself" kinda stuff. Guiding agent rather than doing agent.
~guywithnopowertodisallowit
As a human stochastic parrot, don't you find it embarrassing and humbling to be so easily outdone by an LLM?
Does one need 4 programming languages to draw something on a mac?
We built something to help with that [1].
[1] https://www.happysupport.ai/en/in-app-messaging
- rainbow dripping arrows
- angrily pointing arrows
- flame-surrounded text boxes with particle effects
- the ability for the agent to play airhorn.wav at max volume, overriding existing volume or mute settings
Maybe isoprophlex will add that to his PR!
It's already got the power to be obnoxious at you
https://hyperties.org/cabinet/symelec/
PIXIE and FORTH on PDP-7 Cabinet Emulator with Type 340 Vector Graphics Display:
https://www.youtube.com/watch?v=lo8kdY-5i6c
She doesn't have to call you anymore.
https://donhopkins.com/home/archive/psiber/cyber/pointer.ps
https://youtu.be/_fqCeuue5Ac?t=213
https://medium.com/@donhopkins/the-shape-of-psiber-space-oct...
For web apps, WebMCP is a way where you can make a frontend way more discoverable to an LLM rather than that it is going to read all your code. I sometimes create this in my apps at home and sometimes in my apps at work and it makes LLM assistants way more usable. It also helps if they can highlight certain things of a visual.
We humans can do this too on paper. We can point at something, we can highlight something with a marker. Why not give an LLM these capabilities as well?
That would be a godsent for those of us with aging parents.
Right yeah? I mean I have freagin remote desktop installed on all my parent's devices. I am the agent for them every time they call frantically for help.
It's Time to redesign the apple genius bar for the modern aging boomer using this.
Everything your building is now streaming through a third party who could steal it, the same way Amazon steals open source software and repackages it.
All you need to do is look at palentir's forward deployed engineers. Once you let a third party observe your processed, you're cloneable and replaceable _as an entity_.
I think we can look at our own human psychology as it developed into social groups. If our brains were obviously deterministic, it would be easy for people to prey upon us via structured inputs. Some level of randomness in our thought process is almost evolutionarily required to avoid becoming part of the larger super structure, similar to an ant colony.
So these AI companies are definitely more dangerous than just the "AI is super intelligence". They're basically a intelligence platform everyone is wiring into.
That's today. Tomorrow, they'll be run by an MBA which is the enshittification.
If you can make a facebook clone it doesn't mean you can steal facebook's lunch. there are exceptions of course.
Now even if you want your idea to compete with all the other todo apps with your special sauce, youre giving a third party your recipe. That third party ostensibly has no reason to steal it, their business model does.
So, sure, if your app is a cult of personality and nothing special, i admit it doesnt matter.
But the number of software home runs is dwarfed by all other auccess on individual products
At their scale I'd expect it to be only worth stealing potential home runs and even so, imagine they did this a couple of times, how would that make them look? Do you think it would be worth the bad blood and mass exodus that would follow?
Not really. Ants are superorganisms, they share more than half of their DNA with each other. You could think of each ant as being like a cell in a human.
So atleast pare of our collection of genes has added value as with random decision factors rather than find some supergene that everyone benefits from.
Whenever I see these vibecoded apps, my move is to just do what you've described: point an agent at it and tell it to reproduce it (usually with my own little customizations). It's a bit absurd, but I don't trust that they vibecoded the app as well as "I" could lol.
Hopefully someone smarter than me is working with trying to develop a new programming language given what we know to be possible with LLMs
The open models that are chasing the frontier labs will stop being open once things slow down and there is less incentive to undercut the front runners. Time will tell if GPU compute gets cheap enough to run stuff locally.
https://github.com/seb3773/ntfs-repair-rfc
https://github.com/Kotivskyi/screenshot-tool
I think it was more popular around the beginning of the year to mid-year
Again, I think it heavily depends on if people are writing comprehensive specs with sufficient detail.
It makes sense to not trust AI models with the ability to read and alter your screen for many many many reasons, but the developer of the tool knows that as well. A tool that lets your ai do something doesn't also let it do anything.
So, as always, it's a question of if you trust the software producer. You're giving them the ability to draw on your screen too. Would you have the same fear if there were no AI involved?
For stuff like this I imagine AI misbehavior as a novel failure trigger not a novel failure type. It can only do the harm that the software was able to do on its own already.
> "It's this window, not that one." You have 14 Chrome windows. The agent knows which one it means: --window "Google Chrome:Pull request". It even picks the right tab: --app "Google Chrome:Pull request".
> Remote help. "No, the other gear icon." Point at it instead of describing it.
I found this awesome vibe-coded Pixel News Network project recently. How fucking cool is this?? Would never have been created otherwise. https://pnn.watch/
So the only statement that gets through, in my view, is that code generation is so cheap that we're no longer even concerned about having a clear idea for what we're trying to do or why the code should exist in the first place.
I actually somewhat enjoyed the times when vibed projects were a broken mess, lots of accidental comedy in that era.