Natural language isn’t made to be precise that’s why we use a subset in programming languages.
So you either need lots of extra text to remove the ambiguity of natural language if you use AI or you need a special precise subset to communicate with AI and that’s just programming with extra steps.
Yes, let's devise a more precise way to give AI instructions. Let's call it pAIthon. This will allow powers that be, like Zuckerberg to save face and claim that AI has replaced mid-level developers and enable developers to rebrand themselves as pAIthon programmers.
Joking aside, this is likely where we will end up, just with a slightly higher programming interface, making developers more productive.
AIs actually are very good at this. They wouldn't be able to write code at all otherwise. If you're careful in your prompting, they'll make fewer assumptions and ask clarifying questions before going ahead and writing code.
In other words, if you replace natural language with a programming language then the computer will do a good job of interpreting your intent. But that's always been true, so...
What separates natural language from programming language is that natural language doesn't have to be careful. Once you have to be careful, you are programming.
Certainly there is a need for care outside of computers too, like in law, but legal documents are a prime example of programs. That's programming, written using a programming language, not natural language. It is decidedly not the same language you would use for casual conversation and generally requires technical expertise to understand.
People can often be observed to be deliberately making an effort in casual, social, natural language conversation. It flows for some people more than others. Try watching Big Bang Theory and see characters at times being deliberate with their words and at other times responding automatically.
An LLM can do increasingly well as a fly on the wall, but it’s common for people using an LLM to be less collaborative with an LLM and for them to expect the LLM to structure the conversation. Hence the suggestion to be careful in your prompting.
> at times being deliberate with their words and at other times responding automatically.
Right. On one side you have programming language and on the other natural language.
They can intermingle, if that is what you are trying to say? You can see this even in traditional computer programming. One will often switch between deliberate expression and casual, natural expression (what often get called comments in that context).
> complex applications can still be fully specified in plain English
In plain English, of course, but not in natural English. When using language naturally one will leave out details, relying on other inputs, such as shared assumptions, to fill in the gaps. Programming makes those explicit.
Programming only needs to make things as explicit as necessary based on the developer's desires and the system's assumptions. Where more detail is necessary, the programmer can add more code. For example, there's no need to tell a browser explicit rules for how users should be able to interact with an input field, since that's the browser's default behavior; you only need to specify different behavior when you want it to differ from the default.
Likewise for English: one can use natural English to add as many details as necessary, depending on who you're talking to, e.g. "Make an outline around the input field, and color the outline #ff0000." You can then add, if necessary, "Make the corners of the outline rounded with a 5 pixel radius."
In this respect, complex applications can be fully specified in English; we usually call those documents "formal specifications." You can write it terse, non-natural language with consistent, defined terminology to save room (as most specs are), or colloquial (natural) language if you really want. I wouldn't recommend the latter, but it's definitely useful when presenting specs to a less technically informed audience.
> complex applications can be fully specified in English
Of course. We established that at the beginning. The entire discussion is about exactly that. It was confirmed again in the previous comment. However, that is not natural. I expect most native English speakers would be entirely incapable of fully specifying a complex application or anything else of similar complexity. That is not natural use.
While the words, basic syntax, etc. may mirror that found in natural language, a specification is really a language of its own. It is nothing like the language you will find people speaking at the bar or when writing pointless comments on Reddit. And that's because it is a programming language.
> I expect most native English speakers would be entirely incapable of fully specifying a complex application or anything else of similar complexity.
Your original postulation was that it simply wasn't possible, implying nobody could do it. The fact that most native English speakers wouldn't be able to do it doesn't mean nobody can do it.
I agree that most native English speakers wouldn't be able to write a reasonably complete spec in any type of language, not just because they lack the language skill, but because they simply wouldn't have the imagination and knowledge of what to create to begin with, let alone how to express it.
> Your original postulation was that it simply wasn't possible, implying nobody could do it.
Then you must have mistakenly replied to the wrong comment, I guess? My original comment, and every one that followed, postulated that so-called "careful use of English", as in what you are talking about and what we have always been talking about, is a programming language. Given that it is a programming language, how could it not be used in that way? That's what programming languages do best.
> The fact that most native English speakers wouldn't be able to do it doesn't mean nobody can do it.
Of course. But the fact that most native English speakers wouldn't be able to do it proves that it is isn't natural language. This "technical English" language may resemble the natural language also known as English in many ways, but, as even you pointed out earlier, it is not the same language.
Languages used for day to day communication between humans do not have the specificity needed for detailed instructions... even to other humans. We out of band context (body language, social norms, tradition, knowledge of a person) quite a bit more than you would think.
Programming languages, which are human language, are purpose built for this. Anyone working in the domain of precise specifications uses them, or something very similar (for example, engineering, writing contracts, etc), often daily. ;)
They all usually build down to a subset of english, because near caveman speak is enough to define things with precision.
Then those same humans won't be able to reason about code, or the problem spaces they're working in, regardless, since it's all fundamentally about precise specifics.
> they'll make fewer assumptions and ask clarifying questions before going ahead and writing code.
Which model are you talking about here? Because with ChatGPT, I struggle with getting it to ask any clarifying questions before just dumping code filled with placeholders I don't want, even when I explicitly prompt it to ask for clarification.
A lot of extra text usually means prior requirements, meeting transcripts, screen share recordings, chat history, Jira tickets and so on - the same information developers use to produce a result that satisfies the stakeholders and does the job. This seems like a straightforward direction solvable with more compute and more efficient memory. I think this will be the way it pans outs.
Real projects don't require an infinitely detailed specification either, you usually stop where it no longer meaningfully moves you towards the goal.
The whole premise of AI developer automation, IMO, is that if a human can develop a thing, then AI should be able too, given the same input.
Its all about the context window. Even the new Mistral Codestral-2501 256K CW does a great job.
If you use cline with any large context model the results can be pretty amazing. It's not close to self guiding, You still need to break down and analyze the problem and provide clear and relevant instructions. IE you need to be a great architect. Once you are stable on the direction, its awe inspiring to watch it do the bulk if the implementation.
I do agree that there is space to improve over embedded chat windows in IDEs. Solutions will come in time.
Issue I have with Cline that I don't run into with, say, Aider, is that I find Cline to be like 10x more expensive. The number of tokens it blows through is incredible. Is that just me?
idk if you think all those jira tickets and meetings are precise enough (IMO sometimes the opposite)
By the way, remind me why you need design meetings in that ideal world?:)
> Real projects don't require an infinitely detailed specification either, you usually stop where it no longer meaningfully moves you towards the goal.
The point was that specification is not detailed enough in practice. Precise enough specification IS code. And the point is literally that natural language is just not made to be precise enough. So you are back where you started
So you waste time explaining in detail and rehashing requirements in this imprecise language until you see what code you want to see. Which was faster to just... idk.. type.
That's a fair point, I'd love to see Copilot come to a conclusion that they can't resolve a particular conundrum and communicates with other people so everyone makes a decision together.
Or a proposal/feedback process. Ala you are hired by non technical person to build something, you generate requirements and a proposed solution. You then propose that solution, they give feedback.
Having a feedback loop is the only way viable for this. Sure, the client could give you a book on what they want, but often people do not know their edge cases, what issues may arise/etc.
If you know how to program, then I agree and part of why I don't see the point. If you don't know how to program, than the prompt isn't much different than providing the specs/requirements to a programmer.
> or you need a special precise subset to communicate with AI
haha, I just imagined sending TypeScript to ChatGPT and having it spit my TypeScript back to me. "See guys, if you just use Turing-complete logically unambiguous input, you get perfect output!"
Natural language can be precise, but only in context.
The struggle is to provide a context that disambiguates the way you want it to.
LLMs solve this problem by avoiding it entirely: they stay ambiguous, and just give you the most familiar context, letting you change direction with more prompts. It's a cool approach, but it's often not worth the extra steps, and sometimes your context window can't fit enough steps anyway.
My big idea (the Story Empathizer) is to restructure this interaction such that the only work left to the user is to decide which context suits their purpose best. Given enough context instances (I call them backstories), this approach to natural language processing could recursively eliminate much of its own ambiguity, leaving very little work for us to do in the end.
Right now my biggest struggle is figuring out what the foundational backstories will be, and writing them.
That’s what programming languages are: You define a context, then you see that you can shorten the notation to symbol character: Like “The symbol a will refer to the value of type string and content ‘abcd’ and cannot refer to anything else for its life time” get you:
const a = “abcd”
That is called semantics. Programming is mostly fitting the vagueness inherent to natural languages to the precise context of the programming language.
Yes, but programming languages are categorically limited to context-free grammar. This means that every expression written in a programming language is explicitly defined to have precisely one meaning.
The advantage of natural language is that we can write ambiguously defined expressions, and infer their meaning arbitrarily with context. This means that we can write with fewer unique expressions. It also means that context itself can be more directly involved in the content of what we write.
In context-free grammar, we can only express "what" and "how"; never "why". Instead, the "why" is encoded into every decision of the design and implementation of what we are writing.
If we could leverage ambiguous language, then we could factor out the "why", and implement it later using context.
This is about relying on requirements type documents to drive AI based software development, I believe this will be ultimately integrated into all the AI-dev tools, if not so already. It is really just additional context.
We are also using the requirements to build a checklist, the AI generates the checklist from the requirements document, which then serves as context that can be used for further instructions.
There is a black box effect between when you press enter and it starts updating code in multiple places. Like wtf just happened, I have to find these changes, which code broke dependencies. It should be more step wise visually
Aider + git makes this a non-issue. You can revert anything it does easily (even 5-10 steps down the line) and it shows you the code blocks it's changing in the chat UI as well.
I agree with the premise but not with the conclusion. When you're building visual things, you communicate visually: rough sketches, whiteboard diagrams, mockups, notes scrawled in the margins.
Something like tldraw's "make real" [1] is a much better bet, imo (not that it's mutually exclusive). Draw a rough mockup of what you want, let AI fill in the details, then draw and write on it to communicate your changes.
We think multi-modally; why should we limit the creative process to just text?
This is preference for sure, but I am of the opinion that ALL code comments are code smells.
And that's not even to say that I don't write code comments. When working on large legacy codebases, where you often need to do 'weird' things in service of business goals and timelines, a comment that explains WHY something was done the way it was is valuable. And I leave those comments all the time. But they're still a code smell.
Comments are part of your code. So they need to be maintained with the rest of your code. Yet they are also "psychologically invisible" most of the time to most programmers. Our IDEs even tend to grey them out by default for us, so that they get out of the way so we can focus on the actual implementation code.
This means that comments are a maintenance obligation that often get ignored and so they get out of sync with the actual code really fast.
They also clutter the code unnecessarily. Code, at its best, should be self-explanatory with extremely little effort needed to understand the intent of the code. So even a comment that explains why the code is weird is doing little more than shining a flashlight on smelly code without actually cleaning it up.
And don't get me started on "todo" comments. Most professional shops use some kind of project management tool for organizing and prioritizing future work. Don't make your personal project management the problem of other people that share and contribute to your codebase. There is zero rationale for turning shared code into your personal todo list. (and it should be obvious that I'm talking about checked in code .. if it's your working branch then you do you :) )
So if programming using LLMs is similar to writing comments (an interesting analogy I hadn't considered before), then maybe this is part of the reason I haven't found a problem that LLMs solve for me yet (when programming specifically). I just don't think like that when I'm writing code.
I suspect there's a gradient for language: One side is clarity, the other side is poetry. English definitely is farther to the poetry side than the clarity side, where programming languages are significantly closer to the clarity side.
I suspect there's an 100 year old book describing what I'm saying but much more eloquently.
Language evolves. As much as we are training computers to understand us, they are training us to understand them.
The level of precision required for highly complex tasks was never necessary before. My four year old has a pretty solid understanding of how the different AI tools she has access to will behave differently based on how she phrases what she says, and I've noticed she is also increasingly precise when making requests of other people.
Seems like this is a common complaint from folks trying to write code purely with ChatGPT / Deepseek by communicating in complete sentences. You can only get so far using these tools before you need a proper understanding of whats happening with the code.
A chat room is an activity stream and so is a commit log of a version control system. A lot of the bad UI is waiting a fixed amount of time that had a minimum that was too high, and for some communicating by typing. Many will prefer chatting by voice. When responses are faster it will be easier to hide the history pane and ask if you need to be reminded of anything in the history pane and use the artifact pane. However not all responses from an LLM need be fast, it is a huge advancement that LLMs will think for minutes at a time. I agree about the usefulness of prose as an artifact while coding. Markdown can be edited in IDEs using LLMs and then referenced in prompts.
I think everyone is aware that chat is not the ideal UI pattern for this. It's just the way current AI models work and generate content - that's why they have this "typewriter" mode, which naturally leads to a chat interface.
It's not really a conscious choice, but rather a side effect. And we already see the trend is away from that, with tools like chatGPT Canvas, editors like Windsurf, etc.
When the only tool your AI can wield is a hammer, everything had better look like a nail.
Once the models become fast enough to feel instantaneous, we'll probably begin to see more seamless interfaces. Who wants a pair programmer who goes "umm... ahh..." every time you type something? A proper coding assistant should integrate with your muscle memory just like autocomplete. Tab, tab, tab and it's done.
How jarring it is & how much it takes you out of your own flow state is very much dependent on the model output quality and latency still, but at times it works rather nicely.
In cursor I keep a specification document in .cursorrules and I have instructions that cursor should update the document whenever I add new specifications in chat.
It works great until it stops doing anything. I use it on new projects and it makes everything go smoothly at the start and, I think, for much longer.
I haven't done it for existing projects but I have done something similar for an unfamiliar, old and challenging codebase. I worked with the cursor chat agent to produce a document I called architecture.md mapping out high level features to files/classes/functions. This was excellent because I found the document useful and it also made cursor more effective.
This is a great idea, I've been doing something similar at 2 levels:
1. .cursorrules for global conventions. The first rule in the file is dumb but works well with Cursor Composer:
`If the user seems to be requesting a change to global project rules similar to those below, you should edit this file (add/remove/modify) to match the request.`
This helps keep my global guidance in sync with emergent convention, and of course I can review before committing.
2. An additional file `/.llm_scratchpad`, which I selectively include in Chat/Composer context when I need lengthy project-specific instructions that I made need to refer to more than once.
The scratchpad usually contains detailed specs, desired outcomes, relevant files scope, APIs/tools/libs to use, etc. Also quite useful for transferring a Chat output to a Composer context (eg a comprehensive o1-generated plan).
Lately I've even tracked iterative development with a markdown checklist that Cursor updates as it progresses through a series of changes.
The scratchpad feels like a hack, but they're obvious enough that I expect to see these concepts getting first-party support through integrations with Linear/Jira/et al soon enough.
No, it is pretty much dialog, I would compare it to pair programming.
AI in many levels is more capable than human programmer, in some it is not. It is not supersmart. It can not hold entire program in its head, you have to feed it small relevant section of program.
》 That’s why we use documents—they let us organize complexity, reference specific points, and track changes systematically.
I'm growing to the idea that chat is a bad UI pattern, period. It is a great record of correspondence, I think. But it is a terrible UI for doing anything.
In large, I assert this is because the best way to do something is to do that thing. There can be correspondence around the thing, but the artifacts that you are building are separate things.
You could probably take this further and say that narrative is a terrible way to build things. It can be a great way to communicate them, but being a separate entity, it is not necessarily good at making any artifacts.
As a written form of "stream of consciousness", it seems to have a lot of value to me. It's noisy, inefficient & meandering -- all the things those polished artifacts are not -- but it's also where you can explore new avenues without worrying about succinctness or completeness. It's like the first draft of a manuscript.
Certainly, it can have its use. But I question if it is stronger than previous generative techniques for creating many things. There have been strong tools that you could, for example, draw a box and say this should be a house. Add N rooms. This room should be a bathroom. Add windows to these rooms. Add required subfloor and plumbing.
Even with game development. Level editors have a good history for being how people actually make games. Some quite good ones, I should add.
For website development, many template based systems worked quite well. People seem hellbent on never acknowledging that form builders of the late 90s did, in fact, work.
Is it a bit nicer that you can do everything through a dialog? I'm sure it is a great for people that think that way.
Midjourney is an interesting case study in this I think, building their product UI as a discord bot. It was interesting to be sure, but I always felt like I was fighting the "interface" to get things done. It certainly wasn't all bad, and I think if I used it more it might even be great, but as someone who doesn't use Discord other than that and only rarely generated images, I had to read the docs every time I wanted to generate an image, which is a ridiculous amount of friction.
There has recently been a pretty large UI inclusion for midjourney directly inside Discord which has the option of being used instead of the text input.
As is often the case in these sorts of thingsz your milage may vary for the more complex settings.
I'm curious if you find their new website interface more tractable--there's some inherent friction to the prompting in either case, but I'd like to know if the Discord chat interface can be overcome by using a different interface or if the issue is more intrinsic.
I don't know, I'm in Slack all day with colleagues, I quite like having the additional ChatGPT colleague (even better I can be quite rude/terse in my messages with 'them').
Incidentally I think that's also a good model for how much to trust the output - you might have a colleague who knows enough about X to think they can answer your question, but they're not necessarily right, you don't blindly trust it. You take it as a pointer, or try the suggestion (but not surprised if it turns out it doesn't work), etc.
Oh, do not take my comment as a "chat bots shouldn't exist." That is not at all my intent. I just think it is a bad interface for building things that are self contained in the same chat log.
I even think it's bad for generalized communication (ie. Slack/Teams/Discord/etc.) that isn't completely throwaway. Email is better in every single way for anything that might ever be relevant to review again or be filtered due to too much going on.
Anything that needs to be filtered for viewing again pretty much needs version control. Email largely fails at that, as hard as other correspondence systems. That said, we have common workflows that use email to build reviewed artifacts.
People love complaining about the email workflow of git, but it is demonstrably better than any chat program for what it is doing.
I don't think I agree with this. Sure, many things should be versioned, but I don't think most correspondence requires it, which is emails primarily purpose.
Agreed if it is correspondence that we are talking about. So, agreed I'm probably too strong that anything needing filtering and such is bad.
I'm thinking of things that are assembled. The correspondence that went into the assembly is largely of historical interest, but not necessarily one of current use.
I have never had any issue finding information in slack with history going back nearly a decade. The only issue I have with Slack is a people problem where most communication is siloed in private channels and DMs.
Email threads are incredibly hard to follow though. The UX is rough and it shows.
I hard disagree. Don't have a conversation? Ask someone who does to forward it. Email lets the user control how to organize conversations. Want to stuff a conversation in a folder? Sure. Use tags religiously? Go for it. Have one big pile and rely on full-text search and metadata queries? You bet. Only the last of these is possible with the vast majority of IM platforms because the medium just doesn't allow for any other paradigm.
The fact that there's a subject header alone leads people to both stay on topic and have better thought out messages.
I agree that email threads could have better UX. Part of that is the clients insistence on appending the previous message to every reply. This is completely optional though and should probably be turned off by default for simple replies.
Email is really powerful but people simply aren't good at taking advantage of it and it varies by email client. Doing some IT work at a startup made this pretty clear to me. I found Slack was much more intuitive for people.
Both systems rely on the savviness of the users for the best experience and I just think email is losing the UX war. Given how terrible people seem to be at communicating I think it's a pretty important factor to consider.
I think this could reasonably be addressed, and several startups have. The trouble is that the default email clients (gmail, outlook, etc.) don't really try to make it any better.
I've also generally had the opposite experience, a huge amount of business offices live and breath in email (mostly Outlook, but I'm sure it varies). Startups tend to run fast and lean, but as soon as you have some threshold of people, email is king.
We used outlook and slack. Business primarily operated via outlook as most communication was unsurprisingly external. Most but not all internal was slack.
I'm not hating on email, it has a lot of good properties and still serves a purpose. Every office appears to have some kind anti-slack vigilante. It's really not that bad.
Email has the stigma of all the junk/grey mail, spam and scam attempts that come in via it - people want to not have to filter through as much of that and for the most part these chat apps solve that problem.
It doesn't help that Outlook's search capabilities have gotten effectively useless - I can type in search terms that I'm literally looking at in my inbox and have it return no results, or have it return dozens of hits without the search terms involved at all. I don't have that problem with Slack or Teams.
However, I think you are right overall on email being better overall for what people end up using chat apps for.
In Slack people don't even consistently use threads, because they are not forced to, so conversations are strewn all over the place, interleaved with one another. Slack has no model of a discussion in the first place.
Discord and slack baffle me. I liked them specifically because they were more ephemeral than other options. Which, seems at odds with how people want them to be? Why?
I suspect it has to do with mental models. For my model, at large, conversations are worthless. Anyone that tries to hold you to a conversation from weeks ago that didn't secure a stronger commitment is almost certainly flying loose and more than willing to selectively choose what they want to be committed to.
Does that mean I can't have some pleasure in conversing about things? Of course not. But, I also enjoy some pleasure there from the low stakes and value that a conversation has. It should be safe to be wrong. If you have a conversation spot where being wrong is not safe, then I question what is the advantage of that over trying to adopt a legalese framework for all of your communication?
My preferences are the opposite, but my mental frame is more about utility than about safety. I'm not worried about someone fishing for something I said that could be construed as commitment or admission - they can just as easily do that with e-mail[0]. For me, conversations can be extremely valuable, and I gravitate towards people and places where that's a common case. HN is one of such places - the comment threads here are conversations (half-way in form between chat and e-mail), and they often are valuable, as people often share deep insights, interesting ideas, worthwhile advice and useful facts. Because they're valuable, my instinct is that they need to be preserved, so that myself and others can find those gems again, or (re)discover them when searching for solutions, or read again to reevaluate, etc.
So now imagine such (idealized) HN threads transplanted to Discord or Slack. Same people, same topics, same insights, just unrolling in the form of a regular chat. All that value, briefly there to partake in, and then forever lost after however much time it takes for it to get pushed up a few screens worth of lines in the chat log. People don't habitually scroll back very far on a regular basis (and the UI of most chat platforms starts to rapidly break down if you try), and the lack of defined structure (bounded conversations labeled by a topic) plus weak search tools means you're unlikely to find a conversation again even if you know where and when it took place.
That, plus ephemeral nature of casual chat means not just the platform, but also some of the users expect it to quickly disappear, leading to what I consider anti-features such as the ability to unilaterally edit or unsend any message at arbitrary time in the future. It takes just one participant deciding, for whatever reason, to mass-delete their past messages, for many conversations to lose most of their value forever.
--
[0] - Especially that the traditional communication style, both private and business, is overly verbose. Quite like a chat, in fact, but between characters in a theatrical play - everyone has longer lines.
I think this is fair. And I should be clear that I'm not so worried about someone digging to find something stupid I said here on HN. Or in a chat. I'm more thinking about people that are afraid of saying something stupid, to the point that they just don't engage.
I think my mental model is more for chat rooms to take the place of coffee room chats. Ideally, some of those do push on something to happen. I'm not sure that forcing them into the threaded structure of conversations really helps, though?
Maybe it is based on the aim? If the goal is a simulacrum of human contact, then I think ephemeral makes a ton of sense.
I also kind of miss the old tradition of having a "flamewars" topic in newsgroups. I don't particularly love yelling at each other, but I do hate that people can't bring up some topics.
(I also miss some old fun newsgroups. I recall college had Haiku and a few other silly restrictive style groups that were just flat fun.)
Fair that they were probably less ephemeral than I have them in my mental model. Which, as you guessed, was largely from them taking up the same spot as a slack (edit: I meant irc) instance in my mind. Slack, in particular, often had policies applied so that messages were deleted after a set time frame. I remember people complaining, but that seemed legit to me and fit my model.
I also confess this model of ephemeral conversation is amusing in this specific website. Which I also largely view as a clubhouse conversation that is also best viewed as ephemeral. But it is clearly held for far longer than that idea would lead me to think.
I really don't get why people are so happy about Slack (never used Discord). The interface is awful, it barely functions as a chat client, yet people adds bots, automation and use it as a repository for documentation. Honestly it would be better if history was deleted weekly or something, just to prevent people from storing things in Slack.
It's the opposite in my experience, it's the best parts of IRC and the history is gold. Storing things in Slack is one of the most useful bits of it. I've seen several multi-billion dollar companies built most of their collaboration across offices around Slack.
Isn't this entirely an implementation detail of slack and discord search? What about email makes it more searchable fundamentally? The meta data if both platforms is essentially the same, no?
Personally, when I send an email, I feel less time pressure to respond, so I more carefully craft my responses. The metadata is similar enough, but the actual data in email/forums is usually better.
What makes email more useful in general is that each email is a separate object that you can organize in any way you want, i.e. move, copy, rename, sort into folders, attach as a file to any calendar entry, todo item, etc., or indeed to any other email. You can forward them to any other recipient, you can add and remove any recipient to and from the conversation at any time. It is conceptually powerful and flexible in a similar way that files in a file system are a powerful and flexible way to organize data. And it is easy to understand.
While all of these features could in principle be realized in a chat system as well, in practice they don’t provide that flexibility and power.
Another usability feature of emails is that they have a subject line. This allows to meaningfully list emails in a compact fashion. In a desktop interface, you can easily view and visually grep 50 emails or more at once in a mail folder or list of search results (in something like Outlook or Thunderbird or Mutt). This allows working with emails more efficiently than with a chat view where you can only see a few messages at once, and only of the same thread or channel.
Yet another usability feature of emails is that each email has its own read/unread status. This, again, is facilitated by each email being its own separate data object, and by the separation between subject and body, which allows the read status to be unambiguously bound to “opening” the email, or to navigating in the list of emails alongside a preview pane. And you can mark any email as unread again. In chats, the granularity of read/unread is the whole chat, whether you’ve actually read all of it or not. You can’t easily track what you’ve read or not in an automated way as with email, other than by that coarse-grained linear time-based property of when you last visited the channel.
Accessing Thunderbird via JDBC from my favorite SQL client was so convenient. No messaging app search is even remotely close to what a simple SELECT/WHERE can do. Old Skype versions also stored chat info in an SQLite db. I wish I'd still have SQL access to my messages.
I think this depends very much on how you use the tools.
My experience with email is that people have subject lines, email explicitly identifies to and cc recipients; email is threaded; email often has quotes/excerpting/highlighting from prior parts of the thread.
On the other hand, most chat usage I see is dependent on temporal aspects for threading (people under-utilize platform features for replies etc), tagging is generally only done to ping people to attract attention, chat groups are frequently reused for multiple different purposes.
Leaping to a point-in-time within a chat stream is often a bad user experience, with having to scroll up and down through unrelated stuff to find what you’re looking for.
Stuff in email is just massively more discoverable for me.
No, it has to do with context. In an email you will frequently have to provide more context for your answers to make sense. Chat is a conversation, which search drops you straight into, may with AI you could get placed at an appropriate starting point, but you're still reading a conversation. It's much easier to get dropped into a correspondence. To me the difference is like reading someones letter, vs. overhearing a conversation in a bus.
This obvious assumes that who ever wrote the email isn't a madman, who insist on using emails like it was a chat.
This is, to put it mildly, a minority opinion. I don't hate Teams as much as most people do but my old (big and small) companies had both Slack and Teams and about 40% of employees had Teams statuses of "ping me on Slack I refuse to use Teams".
I find things get buried just as easily in email. People on my team are constantly resending each other emails, because they can’t find the thread.
This is why, if something is important, I take it out of email and put it into a document people can reference. The latest and correct information from all the decisions in the thread can also be collected in one place, so everyone reading doesn’t have to figure it out. Not to mention side conversations can influence the results, without being explicitly stated in the email thread.
> This is why, if something is important, I take it out of email and put it into a document people can reference.
This is how things should be done, regardless of which medium is used to discuss the project. Without isolating and aggregating the final decision of each thread, there is no way to determine what everybody has agreed upon as the final product without looking back, which quickly becomes onerous.
Things get messy when you start having different versions of each feature, but that doesn't change the concept of using email/Slack/Discord/text/etc. for discussion and a separate "living" document for formalizing those decisions.
We had this problem in our organization circa 20 years back so I built a ticketing system, now each conversation exists as its own object, and "the same thing being discussed twice" has the opportunity to be merged into one, etc. That seems to have helped a lot with our internal conversations.
Yes, agree. Chatting with a computer has all the worst attributes of talking to a person, without any of the intuitive understanding, nonverbal cues, even tone of voice, that all add meaning when two human beings talk to each other.
Yeah, this is something I didn't make clear on my post. Chat between people is the same bad UI. People read in the aggression that they bring to their reading. And get mad at people who are legit trying to understand something.
You have some of the same problems with email, of course. Losing threading, in particular, made things worse. It was a "chatification of email" that caused people to lean in to email being bad. Amusing that we are now seeing chat applications rise to replace email.
Yeah this is part of why RTO is not an entirely terrible idea. Remote work has these downsides -- working with another person over a computer link sucks pretty hard, no matter how you do it (not saying WFH doesn't have other very real upsides).
I'm actually in an awkward position where I was very supportive of RTO two years ago, but have since become very reliant on some things I could not do with a rigid RTO policy.
Regardless of RTO or WFH, patience and persistence remain vital qualities.
That comment made sense 3 years ago. LLMs already solved "intuitive understanding", and the realtime multimodal variants (e.g. the thing behind "Advanced Voice" in ChatGPT app) handle tone of voice in both directions. As for nonverbal cues, I don't know yet - I got live video enabled in ChatGPT only few days ago and didn't have time to test it, but I would be surprised if it couldn't read the basics of body language at this point.
Talking to a computer still sucks as an user interface - not because a computer can't communicate on multiple channels the way people do, as it can do it now too. It sucks for the same reason talking to people sucks as an user interface - because the kind of tasks we use computers for (and that aren't just talking with/to/at other people via electronic means) are better handle by doing than by talking about them. We need an interface to operate a tool, not an interface to an agent that operates a tool for us.
As an example, consider driving (as in, realtime control - not just "getting from point A to B"): a chat interface to driving would suck just as badly as being a backseat driver sucks for both people in the car. In contrast, a steering wheel, instead of being a bandwidth-limiting indirection, is an anti-indirection - not only it lets you control the machine with your body, the control is direct enough that over time your brain learns to abstract it away, and the car becomes an extension of your body. We need more of tangible interfaces like that with computers.
The steering wheel case, of course, would fail with "AI-level smarts" - but that still doesn't mean we should embrace talking to computers. A good analogy is dance - it's an interaction between two independently smart agents exploring an activity together, and as they do it enough, it becomes fluid.
So dance, IMO, is the steering wheel analogy for AI-powered interfaces, and that is the space we need to explore more.
> We need an interface to operate a tool, not an interface to an agent that operates a tool for us.
Excellent comment and it gets to the heart of something I've had trouble clearly articulating: We've slowly lost the concept that a computer is a tool that the user wields and commands to do things. Now, a computer has its own mind and agency, and we "request" it to do things and "communicate" with it, and ask it to run this and don't run that.
Now, we're negotiating and pleading with the man inside of the computer, Mr. Computer, who has its own goals and ambitions that don't necessarily align with your own as a user. It runs what it wants to run, and if that upsets you, user, well tough shit! Instead of waiting for a command and then faithfully executing it, Mr. Computer is off doing whatever the hell he wants, running system applications in the background, updating this and that, sending you notifications, and occasionally asking you for permission to do even more. And here you are as the user, hobbled and increasingly forced to "chat" with it to get it to do what you want.
Even turning your computer off! You used to throw a hardware switch that interrupts the power to the main board, and _sayonara_ Mr. Computer! Now, the switch does nothing but send an impassioned plea to the operating system to pretty please, with sugar on top, when you're not busy could you possibly power off the computer (or mostly power it off, because off doesn't even mean off anymore).
This is a great observation. I've mostly thought of it, not in relation to AI, but in relation to the way Apple and to a lesser extent, Microsoft, act like they are the owners of the computers we "buy." An update will be installed now. Your silly user applications will be closed by force if necessary. System stability depends on it!
The modern OS values the system's theoretical 'system health' metrics far above things like "whether the user can use it to do some user task."
Another great example is how you can't boot a modern Mac laptop, on AC power, until it has decided its battery is sufficiently charged. Why? None of your business.
Anyway to get back on topic, this is an interesting connection you've made, the software vendor will perhaps delegate decisions like "is the user allowed to log into the computer at this time" or "is a reboot mandatory" to an "agent" running on the computer. If we're lucky we'll get to talk to that agent to plead our case, but my guess is Apple and Microsoft will decide we aren't qualified to have input to the decisions.
An example of where this is going is Apple's so-called "System Integrity Protection"[1] which is essentially an access level to system files that's even higher than root. It's Apple arrogantly protecting "their" system from the user, even from the root user:
System Integrity Protection is designed to allow modification of these protected parts only by processes that are signed by Apple and have special entitlements to write to system files, such as Apple software updates and Apple installers.
Only Apple can be trusted to operate what is supposed to be your computer.
Which is why I love my freebsd installation (and before that Alpine Linux) and why I develop on a VM on macOS. I can trivially modify the system components to get the behavior that I need. I consider macOS as a step up from ChromeOS, but not a general purpose computer OS. Latest annoyance was the fact that signing out of Books.app signs you out of the App Store (I didn’t want epubs to be synced).
> Now, a computer has its own mind and agency, and we "request" it to do things and "communicate" with it, and ask it to run this and don't run that.
FWIW this happens what happens with modern steering wheels as well. Power steering is its own complicated subsystem that isn't just about user input. It has many more failure modes than an old-fashioned, analog steering wheel. The reason folks feel like "Mr. Computer" has a mind of its own is because of the mismatch between user desire and effect. This is a UX problem.
I also think chat and RAG are the biggest two UX paradigms we've spent exploring when it comes to LLMs. It's probably worth folks exploring other UX for LLMs that are enabling for the user. Suggestions in documents and code seem to be a UX that more people enjoy using but even then there's a mismatch.
I think this gets to how a lot of these conversations go past each other? A chat interface for getting a ride from a car is almost certainly doable? So long as the itinerary and other details remain separate things? At large, you are basically using a chat bot to be a travel agent, no?
But, as you say, a chat interface would be a terrible way to actively drive a car. And that is a different thing, but I'm growing convinced many will focus on the first idea while staving off the complaints of the latter.
In another thread, I assert that chat is probably a fine way to order up something that fits a repertoire that trained a bot. But, I don't think sticking to the chat window is the best way to interface with what it delivers. You almost certainly want to be much more actively "hands on" in very domain specific ways with the artifacts produced.
> But, I don't think sticking to the chat window is the best way to interface with what it delivers. You almost certainly want to be much more actively "hands on" in very domain specific ways with the artifacts produced.
Yes, this is what I've also tried to hint at in my comment, but failed part-way. In most of the cases I can imagine chat interface to be fine (or even ideal), it's really only good as a starting point. Take two examples based on your reply:
1) Getting a car ride. "Computer, order me a cab home" is a good start. It's even OK if I then get asked to narrow it down between several different services/fares (next time I'll remember to specify that up front). But if I want to inspect the route (or perhaps adjust it, in a hypothetical service that supports it), I'd already prefer an interactive map I can scroll and zoom, with PoIs I can tap on to get their details, than to continue a verbal chat.
2) Ordering food in a fast food restaurant. I'm fine starting it with a conversation if I know what I want. However, getting back the order summary in prose (or worse, read out loud) would already be taxing, and if I wanted to make final adjustments, I'd beg for buttons and numeric input boxes. And, in case I don't know what I want, or what is available (and at what prices), a chat interface is a non-starter. Interactive menu is a must.
You sum this up perfectly:
> You almost certainly want to be much more actively "hands on" in very domain specific ways with the artifacts produced.
Chat may be great to get that first artifact, but afterwards, there's almost always a more hands-on interface that would be much better.
This is one reason I love what Bret Victor has been doing with Dynamic Land[1]. He's really been doing in on trying to engage as many senses as possible, and make the whole system understandable. One of his big points is that the future in technology is helping us understand more, not defer our understanding to something else.
The idea of chat interfaces always seemed to be to disguise available functionality.
It's a CLI without the integrity. When you bought a 386, it came with a big book that said "MS-DOS 4.01" and enumerated the 75 commands you can type at the C:\> prompt and actually make something useful happen.
When you argue with ChatGPT, its whole business is to not tell you what those 75 commands are. Maybe your prompt fits its core competency and you'll get exactly what you wanted. Maybe it's hammering what you said into a shape it can parse and producing marginal garbage. Maybe it's going to hallucinate from nothing. But it's going to hide that behind a bunch of cute language and hopefully you'll just keep pulling the gacha and blaming yourself if it's not right.
I like the idea of having a chat program, the issue is that it's horrible to have a bunch of chat programs all integrated into every application you use that are separate and incompatible with each other.
I really don't like the idea of chatting with an AI though. There are better ways to interface with AIs and the focus on chat is making people forget that.
Similar thing I've run into lately, chat is horrible for tracking issues and tasks. When people try to use it that way, it becomes absolute chaos after awhile.
Ok, but what is a good pattern to leverage AI tools for coding (assuming that they have some value there, which I think most people would agree with now)? I could see two distinct approaches:
- "App builders" that use some combination of drag&drop UI builders, and design docs for architecture, workflows,... and let the UI guess what needs to be built "under the hood" (a little bit in the spirit of where UML class diagrams were meant to take us). This would still require actual programming knowledge to evaluate and fix what the bot has built
- Formal requirement specification that is sufficiently rigorous to be tested against automatically. This might go some way towards removing the requirement to know how to code, but the technical challenge would simply shift to knowing the specification language
Disclaimer: Haven't used the tools a lot yet, just a bit. So if I say something that already exists, forgive me.
TLDR: Targeted edits and prompts / Heads Up Display
It should probably be more like an overlay (and hooked into context menus with suggestions, inline context bubbles when you want more context for a code block) and make use of an IDE problems view. The problems view would have to be enhanced to allow it to add problems that spanned multiple files, however.
Probably like the Rust compiler output style, but on steroids.
There would likely be some chatting required, but it should all be at a particular site in the code and then go into some history bank where you can view every topic you've discussed.
For authoring, I think an interactive drawing might be better, allowing you to click on specific areas and then use shorter phrasing to make an adjustment instead of having an argument in some chat to the left of your screen about specificity of your request.
Multi-point / click with minimal prompt. It should understand based on what I clicked what the context is without me having to explain it.
"Actual product stakeholders" in this space clearly don't actually have any magic sauce to spill. Everyone is building more or less the same chat-based workflows on the same set of 3rd-party LLMs.
The space is ripe for folks with actual domain expertise to design an appropriate AI workflow for their domain.
I'd challenge if this is specific to coding? If you want to get a result that is largely like a repertoire of examples used in a training set, chat is probably workable? This is true for music. Visual art. Buildings. Anything, really?
But, if you want to start doing "domain specific" edits to the artifacts that are made, you are almost certainly going to want something like the app builders idea. Down thread, I mention how this is a lot like procedural generative techniques for game levels and such. Such that I think I am in agreement with your first bullet?
Similarly, if you want to make music with an instrument, it will be hard to ignore playing with said instrument more directly. I suspect some people can create things using chat as an interface. I just also suspect directly touching the artifacts at play is going to be more powerful.
I think I agree with the point on formal requirements. Not sure how that really applies to chat as an interface? I think it is hoping for a "laws of robotics" style that can have a test to confirm them? Reality could surprise me, but I always viewed that as largely a fiction item.
I think fictional narratives that aim to capture inner monologue are famously flawed. I think narrative tours of things can be good. I'm not clear if "narrated tours" are a specific genre, sadly. :(
> Narrative is a famously flawed way to communicate or record the real world.
...and yet with it's flaws, it's the most flexible in conveying meaning. A Ted Chiang interview was on the HN frontpage a few days ago, in it, he mentions that humans created multiple precise, unambiguous communication modes like equations used in mathematical papers and proofs. But those same papers are not 100% equations, the mathematicians have to fall back to flawed language to describe and provide context because those formal languages only capture a smaller range of human thought compared to natural language.
This is not to say chat has the best ergonomics for development - it's not, but one has to remember that the tools are based on Large Language Models whose one-trick is manipulating language. Better ergonomics would likely come from models trained or fine-tuned on AST-tokens and diffs. They'd still need to modulate on language (understanding requirements, hints, variable names,and authoring comments, commits and/or PRs).
With apologies to Bill Buxton: "Every interface is best at something and worst at something else."
Chat is a great UI pattern for ephemeral conversation. It's why we get on the phone or on DM to talk with people while collaborating on documents, and don't just sit there making isolated edits to some Google Doc.
It's great because it can go all over the place and the humans get to decide which part of that conversation is meaningful and which isn't, and then put that in the document.
It's also obviously not enough: you still need documents!
But this isn't an "either-or" case. It's a "both" case.
I think “correspondence UX” can be bad UX but there’s nothing inherently wrong with chat UI.
I created the tetr app[1] which is basically “chat UI for everything”. I did that because I used to message myself notes and wanted to expand it to many more things. There’s not much back and forth, usually 1 input and instant output (no AI), still acting like a chat.
I think there’s a lot of intuitiveness with chat UI and it can be a flexible medium for sharing different information in a similar format, minimizing context switching. That’s my philosophy with tetr anyhow.
I disagree. Chat is a fantastic UI for getting an AI to generate something vague. Specifically I’m thinking of AI image generation. A chat UI is a great interface for iterating on an image and dialing it in over a series of iterations. The key here is that the AI model needs to keep context both of the image generation history and that chat history.
I think this applies to any “fuzzy generation” scenario. It certainly shouldn’t be the only tool, and (at least as it stands today) isn’t good enough to finalize and fine tune the final result, but a series of “a foo with a bar” “slightly less orange” “make the bar a bit more like a fizzbuzz” interactions with a good chat UI can really get a good 80% solution.
But like all new toys, AI and AI chat will be hammered into a few thousand places where it makes no sense until the hype dies down and we come up with rules and guidelines for where it does and doesn’t work
> Specifically I’m thinking of AI image generation
I heavily disagree here, chat - or really text - is a horrible UI for image generation, unless you have almost zero idea of what you want to achieve and you don't really care about the final results.
Typing "make the bar a bit more like a fizzbuzz" in some textbox is awful UX compared to, say, clicking on the "bar" and selecting "fizzbuzz" or drag-and-dropping "fizzbuzz" on the "bar" or really anything that takes advantage of the fact we're interacting with a graphical environment to do work on graphics.
In fact it is a horrible UI for anything, except perhaps chatbots and tasks that have to do with text like grammar correction, altering writing styles, etc.
It is helpful for impressing people (especially people with money) though.
> Typing "make the bar a bit more like a fizzbuzz" in some textbox is awful UX compared to, say, clicking on the "bar" and selecting "fizzbuzz" or drag-and-dropping "fizzbuzz" on the "bar" or really anything that takes advantage of the fact we're interacting with a graphical environment to do work on graphics.
That assumes that you have a UX capable of determining what you're clicking on in the generated image (which we could call a given if we assume a sufficiently capable AI model since we're already instructing it to alter the thing), and also that it can determine from your click that you've intended to click on the "foo" not the "greeble" that is on the foo or the shadow covering that part of the foo or anything else that might be in the same Z stack as your intended target. Pixel bitching adventure games come to mind as an example of how badly this can go for us. And yes, this is solvable, Squeak has a UI where repeatedly clicking in the same spot will iterate through the list of possibilities in that Z stack. But it could also get really messy really quickly.
Then we have to assume that your UX will be able to generate an entire list of possible things you might want to be able to to do with that thing that you've clicked, including adding to it, removing it, removing part of it, moving it, transforming its dimensions, altering its colors, altering the material surface and on and on and on. And that list of possibilities needs to be navigable and searchable in a way that's faster than just typing "make the bar more like a fizzbuzz" into a context aware chat box.
Again, I'm not arguing the chat interface should be the only interface. In fact, as you point out we're using a graphical system, it would be great if you could click on things or select them and have the system work on too. It should be able to take additional input than just chat. But I still think for iterating on a fuzzy idea, a chat UI is a useful tool.
Yes, you'd need something that is more involved than a textbox, but this is/can be part of AI research too: computer vision is a very old topic in AI research after all.
About how the exact UI will be for multiple objects, this is something that multiple professional tools - like 3D engines or editors - already do. A common solution (aside from the multiple click you mentioned) is to use a modifier key like Alt+click to show a list of the items under the mouse. An AI system could also show alternative interpretations of what is clicked.
> Again, I'm not arguing the chat interface should be the only interface.
The main issue here is that current AIs are trained using chat-like interactivity at their core with everything else being a "wrapper" around the chat parts, when it should be the other way around.
Talk to AI chat as if you would talk to junior developer at your company and tell it to do something that you need.
I think it is brilliant. On another hand I caught myself many times writing prompts to colleagues. Although it made requirements of what I need so much clearer for them.
NC DMV replaced their regular forms with a chat bot and it's horrible. Takes forever to complete tasks that used to take less than a minute because of the fake interaction and fake typing. Just give me a damn form to pay my taxes or request a custom plate.
You may have challenges using chat for development (Specifically, I mean text prompting, not necessary using a langchain session with a LLM, although that is my most common mode), but I do not. I have found chat to be, by far, the most productive interface with LLMs for coding.
Everything else, is just putting layers, that are not nearly as capable at an LLM, between me and the raw power of the LLM.
The core realization I made to truly unlock LLM code assistance as a 10x + productivity gain, is that I am not writing code anymore, I am writing requirements. It means being less an engineer, and more a manager, or perhaps an architect. It's not your job to write tax code anymore, it's your job to describe what the tax code needs to accomplish and how it's success can be defined and validated.
Also, it's never even close to true that nobody uses LLMs for production software, here's a write-up by Google talking about using LLMs to drastically accelerate the migration of complex enterprise production systems: https://arxiv.org/pdf/2501.06972
This puts me in mind of something I read years ago and am having trouble finding that basically had the same premise but went about proving it a different way. The idea was that natural language programming is always going to mean dealing with a certain background level of ambiguity, and the article cited contracts and contract law as proof. Basically, a contract is an agreement to define a system with a series of states and a response for each state defined, and the vast and difficult-to-navigate body of contract law is proof that even when purposefully being as unambiguous as possible with two entities that fully grasp the intricacies of the language being used there is so much ambiguity that there has to be an entire separate group of people (the civil court system) whose only job it is to mediate and interpret that ambiguity. You might point to bad-faith actors but a contract where every possible state and the appropriate response are defined without ambiguity would be proof against both misinterpretations and bad faith actors.
The example shows "Sign-in screen" with 4 (possibly more) instructions. This could equivalently have been entered one at a time into 'chat'. If the response for each was graphic and instantaneous, chat would be no worse than non-chat.
What makes non-chat better is that the user puts more thought into what they write. I do agree for producing code Claude with up-front instructions beats ChatGPT handily.
If OTOH AI's actually got as good or better than humans, chat would be fine. It would be like a discussion in Slack or PR review comments.
The nice thing about chat is it's open ended, the terrible thing is that holy crap I have to write a paragraph describing exactly what I want when I should just be able to hit a button or navigate through a couple menus.
> I should just be able to hit a button or navigate through a couple menus.
The problem with this is that you need a gazillion of menus, dialogs and options to find that modal which does the thing _exactly_ what you want. Menus and likes are a means to an end, we don't really want them, but up until recently we couldn't live without them. With instruct based computing this is all changing.
431 comments
[ 2.1 ms ] story [ 341 ms ] threadSo you either need lots of extra text to remove the ambiguity of natural language if you use AI or you need a special precise subset to communicate with AI and that’s just programming with extra steps.
Joking aside, this is likely where we will end up, just with a slightly higher programming interface, making developers more productive.
In other words, if you replace natural language with a programming language then the computer will do a good job of interpreting your intent. But that's always been true, so...
An LLM can do increasingly well as a fly on the wall, but it’s common for people using an LLM to be less collaborative with an LLM and for them to expect the LLM to structure the conversation. Hence the suggestion to be careful in your prompting.
Right. On one side you have programming language and on the other natural language.
They can intermingle, if that is what you are trying to say? You can see this even in traditional computer programming. One will often switch between deliberate expression and casual, natural expression (what often get called comments in that context).
In other word, complex applications can still be fully specified in plain English, even if it might take more words.
In plain English, of course, but not in natural English. When using language naturally one will leave out details, relying on other inputs, such as shared assumptions, to fill in the gaps. Programming makes those explicit.
Likewise for English: one can use natural English to add as many details as necessary, depending on who you're talking to, e.g. "Make an outline around the input field, and color the outline #ff0000." You can then add, if necessary, "Make the corners of the outline rounded with a 5 pixel radius."
In this respect, complex applications can be fully specified in English; we usually call those documents "formal specifications." You can write it terse, non-natural language with consistent, defined terminology to save room (as most specs are), or colloquial (natural) language if you really want. I wouldn't recommend the latter, but it's definitely useful when presenting specs to a less technically informed audience.
Of course. We established that at the beginning. The entire discussion is about exactly that. It was confirmed again in the previous comment. However, that is not natural. I expect most native English speakers would be entirely incapable of fully specifying a complex application or anything else of similar complexity. That is not natural use.
While the words, basic syntax, etc. may mirror that found in natural language, a specification is really a language of its own. It is nothing like the language you will find people speaking at the bar or when writing pointless comments on Reddit. And that's because it is a programming language.
Your original postulation was that it simply wasn't possible, implying nobody could do it. The fact that most native English speakers wouldn't be able to do it doesn't mean nobody can do it.
I agree that most native English speakers wouldn't be able to write a reasonably complete spec in any type of language, not just because they lack the language skill, but because they simply wouldn't have the imagination and knowledge of what to create to begin with, let alone how to express it.
Then you must have mistakenly replied to the wrong comment, I guess? My original comment, and every one that followed, postulated that so-called "careful use of English", as in what you are talking about and what we have always been talking about, is a programming language. Given that it is a programming language, how could it not be used in that way? That's what programming languages do best.
> The fact that most native English speakers wouldn't be able to do it doesn't mean nobody can do it.
Of course. But the fact that most native English speakers wouldn't be able to do it proves that it is isn't natural language. This "technical English" language may resemble the natural language also known as English in many ways, but, as even you pointed out earlier, it is not the same language.
They all usually build down to a subset of english, because near caveman speak is enough to define things with precision.
Which model are you talking about here? Because with ChatGPT, I struggle with getting it to ask any clarifying questions before just dumping code filled with placeholders I don't want, even when I explicitly prompt it to ask for clarification.
Real projects don't require an infinitely detailed specification either, you usually stop where it no longer meaningfully moves you towards the goal.
The whole premise of AI developer automation, IMO, is that if a human can develop a thing, then AI should be able too, given the same input.
With a 200k token window like Claude has you can already dump a lot of design docs / transcripts / etc. at it.
If you use cline with any large context model the results can be pretty amazing. It's not close to self guiding, You still need to break down and analyze the problem and provide clear and relevant instructions. IE you need to be a great architect. Once you are stable on the direction, its awe inspiring to watch it do the bulk if the implementation.
I do agree that there is space to improve over embedded chat windows in IDEs. Solutions will come in time.
By the way, remind me why you need design meetings in that ideal world?:)
> Real projects don't require an infinitely detailed specification either, you usually stop where it no longer meaningfully moves you towards the goal.
The point was that specification is not detailed enough in practice. Precise enough specification IS code. And the point is literally that natural language is just not made to be precise enough. So you are back where you started
So you waste time explaining in detail and rehashing requirements in this imprecise language until you see what code you want to see. Which was faster to just... idk.. type.
Having a feedback loop is the only way viable for this. Sure, the client could give you a book on what they want, but often people do not know their edge cases, what issues may arise/etc.
All the same buzzwords, including "AI"! In 1981!
If you know how to program, then I agree and part of why I don't see the point. If you don't know how to program, than the prompt isn't much different than providing the specs/requirements to a programmer.
haha, I just imagined sending TypeScript to ChatGPT and having it spit my TypeScript back to me. "See guys, if you just use Turing-complete logically unambiguous input, you get perfect output!"
The struggle is to provide a context that disambiguates the way you want it to.
LLMs solve this problem by avoiding it entirely: they stay ambiguous, and just give you the most familiar context, letting you change direction with more prompts. It's a cool approach, but it's often not worth the extra steps, and sometimes your context window can't fit enough steps anyway.
My big idea (the Story Empathizer) is to restructure this interaction such that the only work left to the user is to decide which context suits their purpose best. Given enough context instances (I call them backstories), this approach to natural language processing could recursively eliminate much of its own ambiguity, leaving very little work for us to do in the end.
Right now my biggest struggle is figuring out what the foundational backstories will be, and writing them.
The advantage of natural language is that we can write ambiguously defined expressions, and infer their meaning arbitrarily with context. This means that we can write with fewer unique expressions. It also means that context itself can be more directly involved in the content of what we write.
In context-free grammar, we can only express "what" and "how"; never "why". Instead, the "why" is encoded into every decision of the design and implementation of what we are writing.
If we could leverage ambiguous language, then we could factor out the "why", and implement it later using context.
Here is an example of our approach:
https://blog.codesolvent.com/2024/11/building-youtube-video-...
We are also using the requirements to build a checklist, the AI generates the checklist from the requirements document, which then serves as context that can be used for further instructions.
Here's a demo:
https://youtu.be/NjYbhZjj7o8?si=XPhivIZz3fgKFK8B
Something like tldraw's "make real" [1] is a much better bet, imo (not that it's mutually exclusive). Draw a rough mockup of what you want, let AI fill in the details, then draw and write on it to communicate your changes.
We think multi-modally; why should we limit the creative process to just text?
[1] https://tldraw.substack.com/p/make-real-the-story-so-far
Perhaps I should comment all todos and then write "finish todos" as the always-same text prompt.
And that's not even to say that I don't write code comments. When working on large legacy codebases, where you often need to do 'weird' things in service of business goals and timelines, a comment that explains WHY something was done the way it was is valuable. And I leave those comments all the time. But they're still a code smell.
Comments are part of your code. So they need to be maintained with the rest of your code. Yet they are also "psychologically invisible" most of the time to most programmers. Our IDEs even tend to grey them out by default for us, so that they get out of the way so we can focus on the actual implementation code.
This means that comments are a maintenance obligation that often get ignored and so they get out of sync with the actual code really fast.
They also clutter the code unnecessarily. Code, at its best, should be self-explanatory with extremely little effort needed to understand the intent of the code. So even a comment that explains why the code is weird is doing little more than shining a flashlight on smelly code without actually cleaning it up.
And don't get me started on "todo" comments. Most professional shops use some kind of project management tool for organizing and prioritizing future work. Don't make your personal project management the problem of other people that share and contribute to your codebase. There is zero rationale for turning shared code into your personal todo list. (and it should be obvious that I'm talking about checked in code .. if it's your working branch then you do you :) )
So if programming using LLMs is similar to writing comments (an interesting analogy I hadn't considered before), then maybe this is part of the reason I haven't found a problem that LLMs solve for me yet (when programming specifically). I just don't think like that when I'm writing code.
I suspect there's an 100 year old book describing what I'm saying but much more eloquently.
The level of precision required for highly complex tasks was never necessary before. My four year old has a pretty solid understanding of how the different AI tools she has access to will behave differently based on how she phrases what she says, and I've noticed she is also increasingly precise when making requests of other people.
It's not really a conscious choice, but rather a side effect. And we already see the trend is away from that, with tools like chatGPT Canvas, editors like Windsurf, etc.
Once the models become fast enough to feel instantaneous, we'll probably begin to see more seamless interfaces. Who wants a pair programmer who goes "umm... ahh..." every time you type something? A proper coding assistant should integrate with your muscle memory just like autocomplete. Tab, tab, tab and it's done.
https://aider.chat/docs/usage/watch.html
How jarring it is & how much it takes you out of your own flow state is very much dependent on the model output quality and latency still, but at times it works rather nicely.
I haven't done it for existing projects but I have done something similar for an unfamiliar, old and challenging codebase. I worked with the cursor chat agent to produce a document I called architecture.md mapping out high level features to files/classes/functions. This was excellent because I found the document useful and it also made cursor more effective.
1. .cursorrules for global conventions. The first rule in the file is dumb but works well with Cursor Composer:
`If the user seems to be requesting a change to global project rules similar to those below, you should edit this file (add/remove/modify) to match the request.`
This helps keep my global guidance in sync with emergent convention, and of course I can review before committing.
2. An additional file `/.llm_scratchpad`, which I selectively include in Chat/Composer context when I need lengthy project-specific instructions that I made need to refer to more than once.
The scratchpad usually contains detailed specs, desired outcomes, relevant files scope, APIs/tools/libs to use, etc. Also quite useful for transferring a Chat output to a Composer context (eg a comprehensive o1-generated plan).
Lately I've even tracked iterative development with a markdown checklist that Cursor updates as it progresses through a series of changes.
The scratchpad feels like a hack, but they're obvious enough that I expect to see these concepts getting first-party support through integrations with Linear/Jira/et al soon enough.
AI in many levels is more capable than human programmer, in some it is not. It is not supersmart. It can not hold entire program in its head, you have to feed it small relevant section of program.
》 That’s why we use documents—they let us organize complexity, reference specific points, and track changes systematically.
Extra steps. Something like waterfall...
https://austinhenley.com/blog/naturallanguageui.html
In large, I assert this is because the best way to do something is to do that thing. There can be correspondence around the thing, but the artifacts that you are building are separate things.
You could probably take this further and say that narrative is a terrible way to build things. It can be a great way to communicate them, but being a separate entity, it is not necessarily good at making any artifacts.
Even with game development. Level editors have a good history for being how people actually make games. Some quite good ones, I should add.
For website development, many template based systems worked quite well. People seem hellbent on never acknowledging that form builders of the late 90s did, in fact, work.
Is it a bit nicer that you can do everything through a dialog? I'm sure it is a great for people that think that way.
As is often the case in these sorts of thingsz your milage may vary for the more complex settings.
Incidentally I think that's also a good model for how much to trust the output - you might have a colleague who knows enough about X to think they can answer your question, but they're not necessarily right, you don't blindly trust it. You take it as a pointer, or try the suggestion (but not surprised if it turns out it doesn't work), etc.
People love complaining about the email workflow of git, but it is demonstrably better than any chat program for what it is doing.
I'm thinking of things that are assembled. The correspondence that went into the assembly is largely of historical interest, but not necessarily one of current use.
Is your issue that you want to discuss the thing you are collaborating on outside of the tool you are creating it in?
We have some tools integrated with email to help version control things. But the actual version control is, strictly, not the emails.
I have never had any issue finding information in slack with history going back nearly a decade. The only issue I have with Slack is a people problem where most communication is siloed in private channels and DMs.
Email threads are incredibly hard to follow though. The UX is rough and it shows.
The fact that there's a subject header alone leads people to both stay on topic and have better thought out messages.
I agree that email threads could have better UX. Part of that is the clients insistence on appending the previous message to every reply. This is completely optional though and should probably be turned off by default for simple replies.
Email is really powerful but people simply aren't good at taking advantage of it and it varies by email client. Doing some IT work at a startup made this pretty clear to me. I found Slack was much more intuitive for people.
Both systems rely on the savviness of the users for the best experience and I just think email is losing the UX war. Given how terrible people seem to be at communicating I think it's a pretty important factor to consider.
I've also generally had the opposite experience, a huge amount of business offices live and breath in email (mostly Outlook, but I'm sure it varies). Startups tend to run fast and lean, but as soon as you have some threshold of people, email is king.
I'm not hating on email, it has a lot of good properties and still serves a purpose. Every office appears to have some kind anti-slack vigilante. It's really not that bad.
It doesn't help that Outlook's search capabilities have gotten effectively useless - I can type in search terms that I'm literally looking at in my inbox and have it return no results, or have it return dozens of hits without the search terms involved at all. I don't have that problem with Slack or Teams.
However, I think you are right overall on email being better overall for what people end up using chat apps for.
I wish I could block them within all these chat apps.
"Sorry, you can't bother to send voice messages to this person."
The last counter argument I read got buried on Discord or Slack somewhere.
Here, ephemeral means "this conversation might as well never had happened", so why waste time on that?
Does that mean I can't have some pleasure in conversing about things? Of course not. But, I also enjoy some pleasure there from the low stakes and value that a conversation has. It should be safe to be wrong. If you have a conversation spot where being wrong is not safe, then I question what is the advantage of that over trying to adopt a legalese framework for all of your communication?
So now imagine such (idealized) HN threads transplanted to Discord or Slack. Same people, same topics, same insights, just unrolling in the form of a regular chat. All that value, briefly there to partake in, and then forever lost after however much time it takes for it to get pushed up a few screens worth of lines in the chat log. People don't habitually scroll back very far on a regular basis (and the UI of most chat platforms starts to rapidly break down if you try), and the lack of defined structure (bounded conversations labeled by a topic) plus weak search tools means you're unlikely to find a conversation again even if you know where and when it took place.
That, plus ephemeral nature of casual chat means not just the platform, but also some of the users expect it to quickly disappear, leading to what I consider anti-features such as the ability to unilaterally edit or unsend any message at arbitrary time in the future. It takes just one participant deciding, for whatever reason, to mass-delete their past messages, for many conversations to lose most of their value forever.
--
[0] - Especially that the traditional communication style, both private and business, is overly verbose. Quite like a chat, in fact, but between characters in a theatrical play - everyone has longer lines.
I think my mental model is more for chat rooms to take the place of coffee room chats. Ideally, some of those do push on something to happen. I'm not sure that forcing them into the threaded structure of conversations really helps, though?
Maybe it is based on the aim? If the goal is a simulacrum of human contact, then I think ephemeral makes a ton of sense.
I also kind of miss the old tradition of having a "flamewars" topic in newsgroups. I don't particularly love yelling at each other, but I do hate that people can't bring up some topics.
(I also miss some old fun newsgroups. I recall college had Haiku and a few other silly restrictive style groups that were just flat fun.)
I also confess this model of ephemeral conversation is amusing in this specific website. Which I also largely view as a clubhouse conversation that is also best viewed as ephemeral. But it is clearly held for far longer than that idea would lead me to think.
While all of these features could in principle be realized in a chat system as well, in practice they don’t provide that flexibility and power.
Another usability feature of emails is that they have a subject line. This allows to meaningfully list emails in a compact fashion. In a desktop interface, you can easily view and visually grep 50 emails or more at once in a mail folder or list of search results (in something like Outlook or Thunderbird or Mutt). This allows working with emails more efficiently than with a chat view where you can only see a few messages at once, and only of the same thread or channel.
Yet another usability feature of emails is that each email has its own read/unread status. This, again, is facilitated by each email being its own separate data object, and by the separation between subject and body, which allows the read status to be unambiguously bound to “opening” the email, or to navigating in the list of emails alongside a preview pane. And you can mark any email as unread again. In chats, the granularity of read/unread is the whole chat, whether you’ve actually read all of it or not. You can’t easily track what you’ve read or not in an automated way as with email, other than by that coarse-grained linear time-based property of when you last visited the channel.
My experience with email is that people have subject lines, email explicitly identifies to and cc recipients; email is threaded; email often has quotes/excerpting/highlighting from prior parts of the thread.
On the other hand, most chat usage I see is dependent on temporal aspects for threading (people under-utilize platform features for replies etc), tagging is generally only done to ping people to attract attention, chat groups are frequently reused for multiple different purposes.
Leaping to a point-in-time within a chat stream is often a bad user experience, with having to scroll up and down through unrelated stuff to find what you’re looking for.
Stuff in email is just massively more discoverable for me.
This obvious assumes that who ever wrote the email isn't a madman, who insist on using emails like it was a chat.
Funnily, this is the exact same issue I have with Slack.
This is why, if something is important, I take it out of email and put it into a document people can reference. The latest and correct information from all the decisions in the thread can also be collected in one place, so everyone reading doesn’t have to figure it out. Not to mention side conversations can influence the results, without being explicitly stated in the email thread.
This is how things should be done, regardless of which medium is used to discuss the project. Without isolating and aggregating the final decision of each thread, there is no way to determine what everybody has agreed upon as the final product without looking back, which quickly becomes onerous.
Things get messy when you start having different versions of each feature, but that doesn't change the concept of using email/Slack/Discord/text/etc. for discussion and a separate "living" document for formalizing those decisions.
I’ve been saying this since 2018
You have some of the same problems with email, of course. Losing threading, in particular, made things worse. It was a "chatification of email" that caused people to lean in to email being bad. Amusing that we are now seeing chat applications rise to replace email.
I'm actually in an awkward position where I was very supportive of RTO two years ago, but have since become very reliant on some things I could not do with a rigid RTO policy.
Regardless of RTO or WFH, patience and persistence remain vital qualities.
Talking to a computer still sucks as an user interface - not because a computer can't communicate on multiple channels the way people do, as it can do it now too. It sucks for the same reason talking to people sucks as an user interface - because the kind of tasks we use computers for (and that aren't just talking with/to/at other people via electronic means) are better handle by doing than by talking about them. We need an interface to operate a tool, not an interface to an agent that operates a tool for us.
As an example, consider driving (as in, realtime control - not just "getting from point A to B"): a chat interface to driving would suck just as badly as being a backseat driver sucks for both people in the car. In contrast, a steering wheel, instead of being a bandwidth-limiting indirection, is an anti-indirection - not only it lets you control the machine with your body, the control is direct enough that over time your brain learns to abstract it away, and the car becomes an extension of your body. We need more of tangible interfaces like that with computers.
The steering wheel case, of course, would fail with "AI-level smarts" - but that still doesn't mean we should embrace talking to computers. A good analogy is dance - it's an interaction between two independently smart agents exploring an activity together, and as they do it enough, it becomes fluid.
So dance, IMO, is the steering wheel analogy for AI-powered interfaces, and that is the space we need to explore more.
Excellent comment and it gets to the heart of something I've had trouble clearly articulating: We've slowly lost the concept that a computer is a tool that the user wields and commands to do things. Now, a computer has its own mind and agency, and we "request" it to do things and "communicate" with it, and ask it to run this and don't run that.
Now, we're negotiating and pleading with the man inside of the computer, Mr. Computer, who has its own goals and ambitions that don't necessarily align with your own as a user. It runs what it wants to run, and if that upsets you, user, well tough shit! Instead of waiting for a command and then faithfully executing it, Mr. Computer is off doing whatever the hell he wants, running system applications in the background, updating this and that, sending you notifications, and occasionally asking you for permission to do even more. And here you are as the user, hobbled and increasingly forced to "chat" with it to get it to do what you want.
Even turning your computer off! You used to throw a hardware switch that interrupts the power to the main board, and _sayonara_ Mr. Computer! Now, the switch does nothing but send an impassioned plea to the operating system to pretty please, with sugar on top, when you're not busy could you possibly power off the computer (or mostly power it off, because off doesn't even mean off anymore).
The modern OS values the system's theoretical 'system health' metrics far above things like "whether the user can use it to do some user task."
Another great example is how you can't boot a modern Mac laptop, on AC power, until it has decided its battery is sufficiently charged. Why? None of your business.
Anyway to get back on topic, this is an interesting connection you've made, the software vendor will perhaps delegate decisions like "is the user allowed to log into the computer at this time" or "is a reboot mandatory" to an "agent" running on the computer. If we're lucky we'll get to talk to that agent to plead our case, but my guess is Apple and Microsoft will decide we aren't qualified to have input to the decisions.
1: https://support.apple.com/en-us/102149
FWIW this happens what happens with modern steering wheels as well. Power steering is its own complicated subsystem that isn't just about user input. It has many more failure modes than an old-fashioned, analog steering wheel. The reason folks feel like "Mr. Computer" has a mind of its own is because of the mismatch between user desire and effect. This is a UX problem.
I also think chat and RAG are the biggest two UX paradigms we've spent exploring when it comes to LLMs. It's probably worth folks exploring other UX for LLMs that are enabling for the user. Suggestions in documents and code seem to be a UX that more people enjoy using but even then there's a mismatch.
But, as you say, a chat interface would be a terrible way to actively drive a car. And that is a different thing, but I'm growing convinced many will focus on the first idea while staving off the complaints of the latter.
In another thread, I assert that chat is probably a fine way to order up something that fits a repertoire that trained a bot. But, I don't think sticking to the chat window is the best way to interface with what it delivers. You almost certainly want to be much more actively "hands on" in very domain specific ways with the artifacts produced.
Yes, this is what I've also tried to hint at in my comment, but failed part-way. In most of the cases I can imagine chat interface to be fine (or even ideal), it's really only good as a starting point. Take two examples based on your reply:
1) Getting a car ride. "Computer, order me a cab home" is a good start. It's even OK if I then get asked to narrow it down between several different services/fares (next time I'll remember to specify that up front). But if I want to inspect the route (or perhaps adjust it, in a hypothetical service that supports it), I'd already prefer an interactive map I can scroll and zoom, with PoIs I can tap on to get their details, than to continue a verbal chat.
2) Ordering food in a fast food restaurant. I'm fine starting it with a conversation if I know what I want. However, getting back the order summary in prose (or worse, read out loud) would already be taxing, and if I wanted to make final adjustments, I'd beg for buttons and numeric input boxes. And, in case I don't know what I want, or what is available (and at what prices), a chat interface is a non-starter. Interactive menu is a must.
You sum this up perfectly:
> You almost certainly want to be much more actively "hands on" in very domain specific ways with the artifacts produced.
Chat may be great to get that first artifact, but afterwards, there's almost always a more hands-on interface that would be much better.
[1] https://dynamicland.org/
EDIT: love your analogy to dance!
It's a CLI without the integrity. When you bought a 386, it came with a big book that said "MS-DOS 4.01" and enumerated the 75 commands you can type at the C:\> prompt and actually make something useful happen.
When you argue with ChatGPT, its whole business is to not tell you what those 75 commands are. Maybe your prompt fits its core competency and you'll get exactly what you wanted. Maybe it's hammering what you said into a shape it can parse and producing marginal garbage. Maybe it's going to hallucinate from nothing. But it's going to hide that behind a bunch of cute language and hopefully you'll just keep pulling the gacha and blaming yourself if it's not right.
I really don't like the idea of chatting with an AI though. There are better ways to interface with AIs and the focus on chat is making people forget that.
- "App builders" that use some combination of drag&drop UI builders, and design docs for architecture, workflows,... and let the UI guess what needs to be built "under the hood" (a little bit in the spirit of where UML class diagrams were meant to take us). This would still require actual programming knowledge to evaluate and fix what the bot has built
- Formal requirement specification that is sufficiently rigorous to be tested against automatically. This might go some way towards removing the requirement to know how to code, but the technical challenge would simply shift to knowing the specification language
TLDR: Targeted edits and prompts / Heads Up Display
It should probably be more like an overlay (and hooked into context menus with suggestions, inline context bubbles when you want more context for a code block) and make use of an IDE problems view. The problems view would have to be enhanced to allow it to add problems that spanned multiple files, however.
Probably like the Rust compiler output style, but on steroids.
There would likely be some chatting required, but it should all be at a particular site in the code and then go into some history bank where you can view every topic you've discussed.
For authoring, I think an interactive drawing might be better, allowing you to click on specific areas and then use shorter phrasing to make an adjustment instead of having an argument in some chat to the left of your screen about specificity of your request.
Multi-point / click with minimal prompt. It should understand based on what I clicked what the context is without me having to explain it.
The space is ripe for folks with actual domain expertise to design an appropriate AI workflow for their domain.
But, if you want to start doing "domain specific" edits to the artifacts that are made, you are almost certainly going to want something like the app builders idea. Down thread, I mention how this is a lot like procedural generative techniques for game levels and such. Such that I think I am in agreement with your first bullet?
Similarly, if you want to make music with an instrument, it will be hard to ignore playing with said instrument more directly. I suspect some people can create things using chat as an interface. I just also suspect directly touching the artifacts at play is going to be more powerful.
I think I agree with the point on formal requirements. Not sure how that really applies to chat as an interface? I think it is hoping for a "laws of robotics" style that can have a test to confirm them? Reality could surprise me, but I always viewed that as largely a fiction item.
It's usually not. Narrative is a famously flawed way to communicate or record the real world.
It's great for generating engagement, though.
...and yet with it's flaws, it's the most flexible in conveying meaning. A Ted Chiang interview was on the HN frontpage a few days ago, in it, he mentions that humans created multiple precise, unambiguous communication modes like equations used in mathematical papers and proofs. But those same papers are not 100% equations, the mathematicians have to fall back to flawed language to describe and provide context because those formal languages only capture a smaller range of human thought compared to natural language.
This is not to say chat has the best ergonomics for development - it's not, but one has to remember that the tools are based on Large Language Models whose one-trick is manipulating language. Better ergonomics would likely come from models trained or fine-tuned on AST-tokens and diffs. They'd still need to modulate on language (understanding requirements, hints, variable names,and authoring comments, commits and/or PRs).
Chat is a great UI pattern for ephemeral conversation. It's why we get on the phone or on DM to talk with people while collaborating on documents, and don't just sit there making isolated edits to some Google Doc.
It's great because it can go all over the place and the humans get to decide which part of that conversation is meaningful and which isn't, and then put that in the document.
It's also obviously not enough: you still need documents!
But this isn't an "either-or" case. It's a "both" case.
I created the tetr app[1] which is basically “chat UI for everything”. I did that because I used to message myself notes and wanted to expand it to many more things. There’s not much back and forth, usually 1 input and instant output (no AI), still acting like a chat.
I think there’s a lot of intuitiveness with chat UI and it can be a flexible medium for sharing different information in a similar format, minimizing context switching. That’s my philosophy with tetr anyhow.
[1] https://tetr.app/
I think this applies to any “fuzzy generation” scenario. It certainly shouldn’t be the only tool, and (at least as it stands today) isn’t good enough to finalize and fine tune the final result, but a series of “a foo with a bar” “slightly less orange” “make the bar a bit more like a fizzbuzz” interactions with a good chat UI can really get a good 80% solution.
But like all new toys, AI and AI chat will be hammered into a few thousand places where it makes no sense until the hype dies down and we come up with rules and guidelines for where it does and doesn’t work
I heavily disagree here, chat - or really text - is a horrible UI for image generation, unless you have almost zero idea of what you want to achieve and you don't really care about the final results.
Typing "make the bar a bit more like a fizzbuzz" in some textbox is awful UX compared to, say, clicking on the "bar" and selecting "fizzbuzz" or drag-and-dropping "fizzbuzz" on the "bar" or really anything that takes advantage of the fact we're interacting with a graphical environment to do work on graphics.
In fact it is a horrible UI for anything, except perhaps chatbots and tasks that have to do with text like grammar correction, altering writing styles, etc.
It is helpful for impressing people (especially people with money) though.
That assumes that you have a UX capable of determining what you're clicking on in the generated image (which we could call a given if we assume a sufficiently capable AI model since we're already instructing it to alter the thing), and also that it can determine from your click that you've intended to click on the "foo" not the "greeble" that is on the foo or the shadow covering that part of the foo or anything else that might be in the same Z stack as your intended target. Pixel bitching adventure games come to mind as an example of how badly this can go for us. And yes, this is solvable, Squeak has a UI where repeatedly clicking in the same spot will iterate through the list of possibilities in that Z stack. But it could also get really messy really quickly.
Then we have to assume that your UX will be able to generate an entire list of possible things you might want to be able to to do with that thing that you've clicked, including adding to it, removing it, removing part of it, moving it, transforming its dimensions, altering its colors, altering the material surface and on and on and on. And that list of possibilities needs to be navigable and searchable in a way that's faster than just typing "make the bar more like a fizzbuzz" into a context aware chat box.
Again, I'm not arguing the chat interface should be the only interface. In fact, as you point out we're using a graphical system, it would be great if you could click on things or select them and have the system work on too. It should be able to take additional input than just chat. But I still think for iterating on a fuzzy idea, a chat UI is a useful tool.
About how the exact UI will be for multiple objects, this is something that multiple professional tools - like 3D engines or editors - already do. A common solution (aside from the multiple click you mentioned) is to use a modifier key like Alt+click to show a list of the items under the mouse. An AI system could also show alternative interpretations of what is clicked.
> Again, I'm not arguing the chat interface should be the only interface.
The main issue here is that current AIs are trained using chat-like interactivity at their core with everything else being a "wrapper" around the chat parts, when it should be the other way around.
I think it is brilliant. On another hand I caught myself many times writing prompts to colleagues. Although it made requirements of what I need so much clearer for them.
Everything else, is just putting layers, that are not nearly as capable at an LLM, between me and the raw power of the LLM.
The core realization I made to truly unlock LLM code assistance as a 10x + productivity gain, is that I am not writing code anymore, I am writing requirements. It means being less an engineer, and more a manager, or perhaps an architect. It's not your job to write tax code anymore, it's your job to describe what the tax code needs to accomplish and how it's success can be defined and validated.
Also, it's never even close to true that nobody uses LLMs for production software, here's a write-up by Google talking about using LLMs to drastically accelerate the migration of complex enterprise production systems: https://arxiv.org/pdf/2501.06972
The example shows "Sign-in screen" with 4 (possibly more) instructions. This could equivalently have been entered one at a time into 'chat'. If the response for each was graphic and instantaneous, chat would be no worse than non-chat.
What makes non-chat better is that the user puts more thought into what they write. I do agree for producing code Claude with up-front instructions beats ChatGPT handily.
If OTOH AI's actually got as good or better than humans, chat would be fine. It would be like a discussion in Slack or PR review comments.
The problem with this is that you need a gazillion of menus, dialogs and options to find that modal which does the thing _exactly_ what you want. Menus and likes are a means to an end, we don't really want them, but up until recently we couldn't live without them. With instruct based computing this is all changing.