But is it really any faster than using an already existing code generator/scaffolding tool? How do you know your project isn’t just a regurgitation of another repository? Would it be just as fast to clone some existing project and hack on it?
These are the questions everyone seems to be ignoring and saying “only LLMs can make projects quickly” but ignoring everything those LLMs are built on (your llmis probably calling a code gen tool).
For the at work side, I personally haven’t experienced any disadvantages or missed any project deadlines because I didn’t use an LLM, so what does velocity get me? Thumb twiddling time?
Yes... and in fact I'm a professional prototypist (as in I get paid to do that) and this is 100% the process.
You do not, never EVER, start from a blank slate.
Step 0 is to actually challenge the value. Before you even start you spend a LOT of time with the person with a need to narrow down what they actually need. Not what they think they want but what's genuinely problematic for that. Again, NOT how to solve it but what's a thorn that's painful for them, not for what you imagine it might be. This honestly often get uncomfortable quickly because you have to ask "Ok but have you tried this? What about this quirky thing?" because it challenges their own attempts. If you don't spend a significant amount of time in that space you WILL implement faster, that's obvious, but you are very likely to efficient "solve" the wrong problem. You will solve what you can solve easily. Think about it like looking for keys where there is light, not where you lost them.
So... that's before one has even started to code, it's mostly uncomfortable discussions. Only once there is some confidence from both parties that the problem is identified can implementing might make sense. Then you don't! You do NOT touch a line of code. Instead you take whatever you can, post-it notes, Lego bricks, existing software, you tape all that together and you ask "Would THAT (very ugly barely working monster) kind of solve it for you?". So you do not build anything new, you ONLY stick together the BIGGEST existing parts.
Only then you might eventually build something but STILL you don't start from a blank slate. You are going to find the highest level of abstraction you can find. They want something related to the Web? You don't freaking build a Web browser, or a even PWA, you paste a code snippet in the console.
You always look for the way with the MINIMUM amount of new code. It's never about implementing faster. It's about NOT implementing faster.
Now the fun part (arguably) actually begins when you have done all that but it's still not enough. You use a CMS or a browser or whatever large exiting verified code base and the need is not solved. Then you rely on the built-in extension system of that code! Guess what, there might even be an existing plugin that does what you thought was novel.
Finally, finally you did find no extension for that existing quality open-source large code based so you HAVE to build it. Well, not so fast, is there another piece of code from another software that does it? Does it have an API? Can you connect to that API to get that functionality in that extension?
Then you have done it, you brought together 2 large pieces of software but you only coded the connector between the two.
Your prototype is, in practice, 10 lines of code.
TL;DR: yes, good prototypists code very little and yet still end up with genuinely novel work.
I'm truly hopeful that AI will open a new of prototyping. Back in the day, prototyping was how you figured out what to build, you'd very deliberately toss the entire first (or second!) version, and you'd plan to do that.
Might be the opposite in some orgs. Higher ups in working with get visibly annoyed when you start talking about prototypes or trying something out in isolation, they don’t see why you wouldn’t just work with the real codebase and end the project with a PR.
Also seeing a lot of managerial class bypassing the PR system entirely and just committing to main “because it’s faster”.
What are people doing with prototypes afterward? Do you end up shipping it as is to production? What about at work? Are the prototypes useful in that context?
In my day job, we commonly create prototypes to sell the idea/concept to the higher ups, then if we get the green light, we throw the prototype in the bin and start from scratch to build it out properly.
I find this is where AI is genuinely useful, it lets us prototype an idea a lot faster, make no bones about the fact that it is a buggy proof of concept but lets people see the potential and get an idea for what the final product might look like.
While the speed of prototyping and even shipping to production has increased, I have been asking myself at what cost? I see a lot of garbage being shipped. Not because the code quality is bad, because execution has become cheap now. Ideas even though crap, are getting prototyped. Things which look effective on the surface, but has real UX problems in the underneath, are getting prioritised because someone in the room can talk better and enrol a leader to align with the idea. Good old user research or talking to users to validate ideas, iron out issues in the user flows has become too slow for the new process!!
Before if you had a crap idea you atleast had to face the social back-pressure of explaining it to someone at a local hackers meetup and trying to convince them to build it for you..
> Things which look effective on the surface, but has real UX problems in the underneath, are getting prioritised because someone in the room can talk better and enrol a leader to align with the idea
This has always existed. The ability to rapidly prototype has not changed it in any way.
An extremely experienced UX researcher once told me that, having been doing field research and user research for 3 decades now, every time it's a Fortune 500 company, after presenting mountains of research, it comes down to what color the CEO liked in the moment.
I don't understand the proclivity to latch onto whatever the new thing is and blame it for shitty decision-making that has existed as long as humans have existed.
The same thing happens because of tools like the Unity/Unreal engine. Lots of low quality barely-more-than-a-demo "games" uploaded to steam. However those games rightly fail to make any decent $ so probably not a problem long term.
One of the second order effects of AI collapsing the cost of building things is that product management is much more important now. A Product Owner/Manager who lacks the taste and insight (or data) to know what they should put in front of users and what they should just put in the bin will cause a company real harm, especially if the company moves to a "there's zero effort in building something, so we'll try everything!" model.
The only part that's really collapsed in effort is the translation from requirements into code. If you're using AI to generate requirements you're effectively building things based on what a 'random' requirements generator says. If that's as good as the requirements a Product Owner was writing then that person needs to improve.
I'm having a hard time seeing your point. Faster iteration = easier to fix UX issues. That's all the LLM is providing here. Problems with UX = bad decisions. Those happen with or without LLMs.
For the past few months, many times i’ve tried this workflow:
1. Ask a coding agent to think and implement a feature that is non trivial
2. This leads to really understand pros and cons for many possible solutions and see it happen end to end
3. Revert all changes and implement it myself when i’m settled on a solution i’m satisfied with
4. At this point the agent is just an iterative reviewer
I’ve felt that any non trivial amount of code not written myself tends to be hard to own. And like the author said, need to keep skills sharp also
Productivity has increased only for people who knows what they are doing. I have been able to increase my productivity to build and turn around things faster and in a much polished manner. One problem is too many things goes on in your head and I see using tools like Jira or notion are very handy to capture all edge scenarios, integrations need to be captured. Taking break from AI is very very essential for this to work for me.
Standard disclaimer from me that if you are forced to do it by people who have power over you then that is different from doing things voluntarily. You can still “just quit your job” but you have less agency.
I also use AI, not for muh agents but for asking questions. Even when I’m not forced to, unfortunately.
> I still don't think AI is magic, and I'm still cautious about the broader picture; the environmental, financial, and social questions haven't gone anywhere. But for me, right now, the day-to-day reality is that I can move faster, think bigger, and ship more than I could before. And that's been genuinely fun.
Three categories of concern, two of which are relevant for the well-being of commoners, and you still go ahead with it? Why? Because being productive and having fun is more important than the environment and driving people into social crises?
Prototyping is a lot easier indeed. I've experienced this as well. And many of the prototypes are kind of shipable even after a only a bit of iteration. Mostly it's super easy to go from prototype to something shippable. I hate the term vibe coding actually. Is it still vibe coding when the thing has end to end tests and I've been trying to break it for a few days?
The flip side is, nobody cares. I've put some of these things up on Github and ... nothing. It seems even my pre-AI projects have dropped sharply in eyeballs judging by issues, prs, stars, etc. People are too busy doing their own things to bother looking at other people's stuff. And rightfully so. There's nothing magical about my prompting to what people can prompt themselves. The value of these prototypes just dropped. Except op course for people still doing things the old fashioned way.
So, you can ship your prototype. But there's very little point to doing so. Even if it isn't slop, it's just very hard to stand out from the masses of other people's prototypes. The value of custom applications just dropped by an order of magnitude. Everybody is going to expect things to be tailored to them now.
I think it is mostly accurate to say that the cost of execution has dropped to zero.
In this new world, the ability to say "no" is more important than ever. It has never been so easy to burn time, money and energy. The fact that you can try anything now can be modeled as a disadvantage. The space of possible solutions got a lot bigger. Unless you have good taste you could wander and get lost very quickly in this vast new expanse. A true expert that has more paths to work with can arrive at higher quality solutions faster. A novice will get into trouble faster.
It often takes 10k hours suffering through an idea with a live customer before we deeply learn why something is a bad/good approach. None of this painful wisdom is available in the models. You can easily change the mind of ChatGPT with a single adjective. You cannot so easily persuade the person who has successfully cast the ring into the volcano already. They know what it actually feels like to get there.
I think there are costs beyond having to sacrifice writing code yourself.
When prototyping yourself you learn a lot about the problem, see what design decisions lead to what tradeofs.
While you write code your brain is always running in the background, giving you thoughts about how things could break, where the structure could be simplified, where the code could be extended.
I feel this is lost or at least reduced a lot when an LLM writes code because you have a lot less contact with the software.
41 comments
[ 3.2 ms ] story [ 63.7 ms ] threadThese are the questions everyone seems to be ignoring and saying “only LLMs can make projects quickly” but ignoring everything those LLMs are built on (your llmis probably calling a code gen tool).
For the at work side, I personally haven’t experienced any disadvantages or missed any project deadlines because I didn’t use an LLM, so what does velocity get me? Thumb twiddling time?
You do not, never EVER, start from a blank slate.
Step 0 is to actually challenge the value. Before you even start you spend a LOT of time with the person with a need to narrow down what they actually need. Not what they think they want but what's genuinely problematic for that. Again, NOT how to solve it but what's a thorn that's painful for them, not for what you imagine it might be. This honestly often get uncomfortable quickly because you have to ask "Ok but have you tried this? What about this quirky thing?" because it challenges their own attempts. If you don't spend a significant amount of time in that space you WILL implement faster, that's obvious, but you are very likely to efficient "solve" the wrong problem. You will solve what you can solve easily. Think about it like looking for keys where there is light, not where you lost them.
So... that's before one has even started to code, it's mostly uncomfortable discussions. Only once there is some confidence from both parties that the problem is identified can implementing might make sense. Then you don't! You do NOT touch a line of code. Instead you take whatever you can, post-it notes, Lego bricks, existing software, you tape all that together and you ask "Would THAT (very ugly barely working monster) kind of solve it for you?". So you do not build anything new, you ONLY stick together the BIGGEST existing parts.
Only then you might eventually build something but STILL you don't start from a blank slate. You are going to find the highest level of abstraction you can find. They want something related to the Web? You don't freaking build a Web browser, or a even PWA, you paste a code snippet in the console.
You always look for the way with the MINIMUM amount of new code. It's never about implementing faster. It's about NOT implementing faster.
Now the fun part (arguably) actually begins when you have done all that but it's still not enough. You use a CMS or a browser or whatever large exiting verified code base and the need is not solved. Then you rely on the built-in extension system of that code! Guess what, there might even be an existing plugin that does what you thought was novel.
Finally, finally you did find no extension for that existing quality open-source large code based so you HAVE to build it. Well, not so fast, is there another piece of code from another software that does it? Does it have an API? Can you connect to that API to get that functionality in that extension?
Then you have done it, you brought together 2 large pieces of software but you only coded the connector between the two.
Your prototype is, in practice, 10 lines of code.
TL;DR: yes, good prototypists code very little and yet still end up with genuinely novel work.
High quality ensued. Usually ;)
Also seeing a lot of managerial class bypassing the PR system entirely and just committing to main “because it’s faster”.
I find this is where AI is genuinely useful, it lets us prototype an idea a lot faster, make no bones about the fact that it is a buggy proof of concept but lets people see the potential and get an idea for what the final product might look like.
This has always existed. The ability to rapidly prototype has not changed it in any way.
An extremely experienced UX researcher once told me that, having been doing field research and user research for 3 decades now, every time it's a Fortune 500 company, after presenting mountains of research, it comes down to what color the CEO liked in the moment.
I don't understand the proclivity to latch onto whatever the new thing is and blame it for shitty decision-making that has existed as long as humans have existed.
One of the second order effects of AI collapsing the cost of building things is that product management is much more important now. A Product Owner/Manager who lacks the taste and insight (or data) to know what they should put in front of users and what they should just put in the bin will cause a company real harm, especially if the company moves to a "there's zero effort in building something, so we'll try everything!" model.
The only part that's really collapsed in effort is the translation from requirements into code. If you're using AI to generate requirements you're effectively building things based on what a 'random' requirements generator says. If that's as good as the requirements a Product Owner was writing then that person needs to improve.
I’ve felt that any non trivial amount of code not written myself tends to be hard to own. And like the author said, need to keep skills sharp also
I also use AI, not for muh agents but for asking questions. Even when I’m not forced to, unfortunately.
> I still don't think AI is magic, and I'm still cautious about the broader picture; the environmental, financial, and social questions haven't gone anywhere. But for me, right now, the day-to-day reality is that I can move faster, think bigger, and ship more than I could before. And that's been genuinely fun.
Three categories of concern, two of which are relevant for the well-being of commoners, and you still go ahead with it? Why? Because being productive and having fun is more important than the environment and driving people into social crises?
The flip side is, nobody cares. I've put some of these things up on Github and ... nothing. It seems even my pre-AI projects have dropped sharply in eyeballs judging by issues, prs, stars, etc. People are too busy doing their own things to bother looking at other people's stuff. And rightfully so. There's nothing magical about my prompting to what people can prompt themselves. The value of these prototypes just dropped. Except op course for people still doing things the old fashioned way.
So, you can ship your prototype. But there's very little point to doing so. Even if it isn't slop, it's just very hard to stand out from the masses of other people's prototypes. The value of custom applications just dropped by an order of magnitude. Everybody is going to expect things to be tailored to them now.
In this new world, the ability to say "no" is more important than ever. It has never been so easy to burn time, money and energy. The fact that you can try anything now can be modeled as a disadvantage. The space of possible solutions got a lot bigger. Unless you have good taste you could wander and get lost very quickly in this vast new expanse. A true expert that has more paths to work with can arrive at higher quality solutions faster. A novice will get into trouble faster.
It often takes 10k hours suffering through an idea with a live customer before we deeply learn why something is a bad/good approach. None of this painful wisdom is available in the models. You can easily change the mind of ChatGPT with a single adjective. You cannot so easily persuade the person who has successfully cast the ring into the volcano already. They know what it actually feels like to get there.
In software it is now possible to test ideas instantly, this is great for people like me that have so many ideas.