This could very well be a pattern that some teams evolve into. Specs are the new source -- they describe the architectural approach, as well as the business rules and user experience details. End to end tests are described here too. This all is what goes through PRs and review process, and the code becomes a build artifact.
very true. and we already know and agree with this.
user experience/what the app actually does >>> actually implementing it.
elon musk said this a looong time ago. we move from layer 1 (coding, how do we implement this?) to layer 2 thinking (what should the code do? what do we code? should we implement this? (what to code to get the most money?))
The underlying mechanism is still the same: humans type and products come out.
So something which must be true if this author is right is that whatever the new language is—the thing people are typing into markdown—must be able to express the same rigor in less words than existing source code.
Otherwise the result is just legacy coding in a new programming language.
> If I had to roll out such a development process today, I’d make a standardized Markdown specification the new unit of knowledge for the software project. Product owners and engineers could initially collaborate on this spec and on test cases to enforce business rules. Those should be checked into the project repositories along with the implementing code. There would need to be automated pull-request checks verifying not only that tests pass but that code conforms to the spec. This specification, and not the code that materializes it, is what the team would need to understand, review, and be held accountable for.
The constant urge I have today is for some sort of spec or simpler facts to be continuously verified at any point in the development process; Something agents would need to be aware of. I agree with the blog and think it's going to become a team sport to manage these requirements. I'm going to try this out by evolving my open source tool [1] (used to review specs and code) into a bit more of a collaborative & integrated plane for product specs/facts - https://plannotator.ai/workspaces/
I called it gates on mine. I loved Beads but it closed tasks without any validation steps. Beads also had other weird issues, so I made my own alternative. I think "Gates" is also used by others projects that took on the same challenge I did in mine weirdly enough.
I found that adding "philosophy" descriptions help guide the tooling. No specs, just general vibes what's the point, because we can't make everyone happy and it's not a goal of a good tool (I believe).
Technology, implementation may change, but general point of "why!?" stays.
Is it? All the electricity and capital investment in computing hardware costs real money. Is this properly reflected in the fees that AI companies charge or is venture capital propping each one up in the hope that they will kill off the competition before they run out of (usually other people's) money?
Even ignoring the AI costs, 'rework' is going to be more expensive as soon as you have customers. For example any sort of data migration. Or UX expectations. Or public API interface. None of these can change without some thought, so one would be leaning on these specs quite a lot.
React team seems to really have set a precedent with their "dangerouslySetInnerHTML" idea.
Or did they borrow it somewhere?
I'm just curious about that etymology, of course the idea is not universally helpful: for example, for dd CLI parameters, it would only make a mess.
But when there's a flag/option that really requires you to be vigilant and undesired the input and output and all edge cases, calling it "dangerous" is quite a feat!
I might even start my own blog to write about things I've found.
1. Always get the agent to create a plan file (spec). Whatever prompt you were going to yolo into the agent, do it in Plan Mode first so it creates a plan file.
2. Get agents to iterate on the plan file until it's complete and thorough. You want some sort of "/review-plan <file>" skill. You extend it over time so that the review output is better and better. For example, every finding should come with a recommended fix.
3. Once the plan is final, have an agent implement it.
4. Check the plan in with the impl commit.
The plan is the unit of work really since it encodes intent. Impl derives from it, and bugs then become a desync from intent or intent that was omitted. It's a nicer plane to work at.
From this extends more things: PRs should be plan files, not code. Impl is trivial. The hard part is the plan. The old way of deriving intent from code sucked. Why even PR code when we haven't agreed on a plan/intent?
This process also makes me think about how code implementation is just a more specific specification about what the computer should do. A plan is a higher level specification. A one-line prompt into an LLM is the highest level specification. It's kinda weird to think about.
Finally, this is why I don't have to read code anymore. Over time, my human review of the code unearthed fewer and fewer issues and corrections to the point where it felt unnecessary. I only read code these days so I can impose my preferences on it and get a feel for the system, but one day you realize that you can accumulate your preferences (like, use TDD and sum types) in your static prompt/instructions. And you're back to watching this thing write amazing code, often better than what you would have written unless you have maximum time + attention + energy + focus no matter how uninteresting the task, which you don't.
No, because consider the pre-AI status quo where a human PR will come in like "Added tab support", maybe scribbles out some guiding ideas, maybe references some issue where we kinda hashed out some ideas of how it could work, and then we must derive all of the intentions/assumptions/decisions of the implementor from the PR's code changes.
Basically zero plan. Or rather, the "internal" plan that the human implementor used while writing the code is hidden from us because it's a mix of ideas they held in their head, jotted in some notes, existed in a sequence of commits that were lost when squashed into a PR, etc. There's zero reproducibility in the implementation.
So take my idea and pretend we still don't have AI yet: the main point is that we move to a pipeline where we work on a first-class plan first before we begin implementation. This gets us closer to reproducible implementation no matter who is implementing it.
It just so happens that now with implementation becoming automated, we have more attention and energy freed up to focus on this plan-based model.
The lesson I've learned from our new AI age is how little a large number of people who've worked in software development their entire careers understand software development.
I suppose all the money floating around AI helps dummify everything, as people glom on to narratives, regardless of merit, that might position them to partake.
What we actually have now is the ability to bang out decent quality code really fast and cheaply.
This is massive, a huge change, one which upends numerous assumptions about the business of software development.
...and it only leaves us to work through every other aspect of software development.
The approach this article advocates is to essentially pretend none of this exists. Simple, but will rarely produce anything of value.
This paragraph from the post gives you the gist of it:
> ...we need to remove humans-in-the-loop, reduce coordination, friction, bureaucracy, and gate-keeping. We need a virtually infinite supply of requirements, engineers acting as pseudo-product designers, owning entire streams of work, with the purview to make autonomous decisions. Rework is almost free so we shouldn’t make an effort to prevent incorrect work from happening.
As if the only reason we ever had POs or designers or business teams, or built consensus between multiple people, or communicated with others, or reviewed designs and code, or tested software, was because it took individual engineers too long to bang out decent code.
AI has just gotten people completely lost. Or I guess just made it apparent they were lost the whole time?
All the talking points and techniques are those which were used when pushing outsourcing: give better specs, write detailed tests, accept bad code because it works so who cares, we can just rewrite from scratch later, and my favorite "they will get better with more exposure to your code base". None of these takes is wrong, but what they neglect is doing all that work is way more effort than if I wrote the original code myself.
Using an LLM to one shot a small function (something i would do with a very specific search on Google or SO) is handy. Giving it a harness and free access to a code base leads to some terrible code, and doubling down with more instructions and agents in the loop means more time writing the rube Goldberg orchestration rather than just opening up an editor and writing code.
Instead of accepting 20,000 lines of slop per PR (and never-ending combinatorial complexity), maybe we should aim to think about abstractions and how to steer LLMs to generate code similar to that of a skilled human developer. Then it could actually be a maintainable artifact by humans and LLMs alike.
I don't get why every AI article is so hyper-focused on coding speed. If the coding is so fast doesn't it make sense to invest more time into quality, learning, documentation, testing refactoring, making a better product? I'm beginning to think that the slopcoders are evaluated by kLOCs of lines written in addition to LLM token usage and they're just maximising the measured metrics. Whether that actually ends up in production or is used by any real person is seemingly irrelevant. Likely the more bugs that are produced the more agents can be spun in parallel to simulate busywork.
I wonder if with the speed of iteration with AI the industry will switch back to waterfall. Clear documentation first so the LLM can easily produce what's being asked with a round of testing before going back to the documentation stage and running it again. History does repeat itself.
markdown became the language I hate the most thank to LLMs and specs-driven approach. everything feels so dumb right now in agentic coding.
looping blindlessly and aimlessly until it compiles then until the playwright server or whatever devtools shows that it somehow works. push the code, have a llm autoreview/autofix,push to prod, run a mythos (perfect name) to identify the bug that opus 4.7 create. loops on loops on loops of some kind of zombie processes running to a "goal" that everyone seems to mystify in talks to just hide the fact that we do nothing anymore. the bottleneck never was code. it was the gate that was keeping away the Elizabeth Holmes and SBF from software engineering and it just opened.
A colleague and I have taken to use of the verb "meatspin", from another era in Internet shock humour, to describe what it is that coding agents actually do 99% of the time.
the irony is that AI is making this exact problem worse. ppl are generating entire codebases now without reading any of it -- the flag might as well be the default. the skill thats actually becoming scarce isnt writing code, its reading code you didnt write and knowing if its correct.
You are missing another dimension how easy it would be to migrate if adding new feature hits a ceiling and LLM keeps breaking the system.
Imagine all tests are passing and code is confirming the spec, but everything is denormalized because LLM thought this was a nice idea at the beginning since no one mentioned that requirement in the spec. After a while you want to add a feature which requires normalized table and LLM keeps failing, but you also have no idea how this complex system works.
Don't forget that very very detailed spec is actually the code
> Don't forget that very very detailed spec is actually the code
Came here to say this, but you said it for me. If the problem were merely one of insufficient rigour or detail in specs, it would have been solved long before LLMs.
> Don't forget that very very detailed spec is actually the code
In the age of AI this is more true than you know. Given a detailed enough spec and test suite you can effectively rewrite any application with any language in a fully automated way.
I've coined that as "Duck coding" :D If it quacks like a duck, walks like a duck and looks like a duck - it's duck enough for as far as the spec is concerned. Does it matter what is inside the duck?
> Don't forget that very very detailed spec is actually the code
The tests, sure. But certainly not the code itself, as that sits far too close to the implementation (i.e. it is the implementation). An almost infinite number of implantations can fulfill “does foo when bar”, so how can we prove that ours is the spec itself?
It’s kind of like a scientist coming up with a hypothesis post-hoc to fit the results of the experiment.
>... my first bet would be specifications ... and tests ... If I had to roll out such a development process today, I’d make a standardized Markdown specification the new unit of knowledge for the software project.
I've found that adopting RFC Keywords (e.g. RFC 2119 [1]; MUST, SHOULD, MAY) at least makes the LLM report satisfaction. I'd love to see a proper study on the usage of RFC keywords and their effect on compliance and effectiveness.
> Product owners and engineers could initially collaborate on this spec and on test cases to enforce business rules. Those should be checked into the project repositories along with the implementing code. There would need to be automated pull-request checks verifying not only that tests pass but that code conforms to the spec. This specification, and not the code that materializes it, is what the team would need to understand, review, and be held accountable for.
This just sounds like typical requirements management software (IBM DOORS for example, which has been around since the 90s).
It's kind of funny how AI evangelists keep re-discovering the need for work methods and systems that have existed for decades.
When I worked as a software developer at a big telecom company and I had no say in what the software was supposed to do, that was up to the software design people--they were the ones responsible for designing the software and defining all the requirements--I was just responsible for implementing that behavior in code.
> We can stop reading LLM-generated code just like we don’t read assembly, or bytecode, or transpiled JavaScript; our high-level language source would now be another form of machine code
This is too weird for me. At least with programming languages I can consult the documentation and if the programming language isn’t behaving as documented, it’s obviously a defect and if you’re savvy enough you often have open channels that accept contributions. Can we say the same for Claude or other AI solutions?
My amazon orgs leadership has been obsessed with spec driven development while individual engineers tell me the only use they have is to placate leadership. I'm tired
Author here. I'm surprised to see this surfacing now. I just wanted to clarify, since apparently the post doesn't do a good job at it, that what I discussed there is not a methodology I advocate for. The point of the post was: ok, since there are organizations mandating to maximize speed by reducing time spent on typing code (or even mandating to maximize agents usage), is there a way we can meet that requirement while still preserving the rigor somewhere else?
This was a follow up to a previous article[1] and the pair tried to express what I still think today (using AI daily at work): every time I use AI for coding, to some capacity I'm sacrificing system understanding and stability in favor of programming speed. This is not necessarily always a bad tradeoff, but I think it's important to constantly remind ourselves we are making it.
51 comments
[ 2.9 ms ] story [ 60.6 ms ] threadAs I understand, this is an unsolved problem.
user experience/what the app actually does >>> actually implementing it.
elon musk said this a looong time ago. we move from layer 1 (coding, how do we implement this?) to layer 2 thinking (what should the code do? what do we code? should we implement this? (what to code to get the most money?))
this is basic knowledge
So something which must be true if this author is right is that whatever the new language is—the thing people are typing into markdown—must be able to express the same rigor in less words than existing source code.
Otherwise the result is just legacy coding in a new programming language.
The constant urge I have today is for some sort of spec or simpler facts to be continuously verified at any point in the development process; Something agents would need to be aware of. I agree with the blog and think it's going to become a team sport to manage these requirements. I'm going to try this out by evolving my open source tool [1] (used to review specs and code) into a bit more of a collaborative & integrated plane for product specs/facts - https://plannotator.ai/workspaces/
[1] https://github.com/backnotprop/plannotator
https://github.com/Giancarlos/guardrails
Technology, implementation may change, but general point of "why!?" stays.
Is it? All the electricity and capital investment in computing hardware costs real money. Is this properly reflected in the fees that AI companies charge or is venture capital propping each one up in the hope that they will kill off the competition before they run out of (usually other people's) money?
React team seems to really have set a precedent with their "dangerouslySetInnerHTML" idea.
Or did they borrow it somewhere?
I'm just curious about that etymology, of course the idea is not universally helpful: for example, for dd CLI parameters, it would only make a mess.
But when there's a flag/option that really requires you to be vigilant and undesired the input and output and all edge cases, calling it "dangerous" is quite a feat!
I might even start my own blog to write about things I've found.
1. Always get the agent to create a plan file (spec). Whatever prompt you were going to yolo into the agent, do it in Plan Mode first so it creates a plan file.
2. Get agents to iterate on the plan file until it's complete and thorough. You want some sort of "/review-plan <file>" skill. You extend it over time so that the review output is better and better. For example, every finding should come with a recommended fix.
3. Once the plan is final, have an agent implement it.
4. Check the plan in with the impl commit.
The plan is the unit of work really since it encodes intent. Impl derives from it, and bugs then become a desync from intent or intent that was omitted. It's a nicer plane to work at.
From this extends more things: PRs should be plan files, not code. Impl is trivial. The hard part is the plan. The old way of deriving intent from code sucked. Why even PR code when we haven't agreed on a plan/intent?
This process also makes me think about how code implementation is just a more specific specification about what the computer should do. A plan is a higher level specification. A one-line prompt into an LLM is the highest level specification. It's kinda weird to think about.
Finally, this is why I don't have to read code anymore. Over time, my human review of the code unearthed fewer and fewer issues and corrections to the point where it felt unnecessary. I only read code these days so I can impose my preferences on it and get a feel for the system, but one day you realize that you can accumulate your preferences (like, use TDD and sum types) in your static prompt/instructions. And you're back to watching this thing write amazing code, often better than what you would have written unless you have maximum time + attention + energy + focus no matter how uninteresting the task, which you don't.
Doesn’t it bother you that the outcome each PR is different every time you/CI “run it”?
Basically zero plan. Or rather, the "internal" plan that the human implementor used while writing the code is hidden from us because it's a mix of ideas they held in their head, jotted in some notes, existed in a sequence of commits that were lost when squashed into a PR, etc. There's zero reproducibility in the implementation.
So take my idea and pretend we still don't have AI yet: the main point is that we move to a pipeline where we work on a first-class plan first before we begin implementation. This gets us closer to reproducible implementation no matter who is implementing it.
It just so happens that now with implementation becoming automated, we have more attention and energy freed up to focus on this plan-based model.
I suppose all the money floating around AI helps dummify everything, as people glom on to narratives, regardless of merit, that might position them to partake.
What we actually have now is the ability to bang out decent quality code really fast and cheaply.
This is massive, a huge change, one which upends numerous assumptions about the business of software development.
...and it only leaves us to work through every other aspect of software development.
The approach this article advocates is to essentially pretend none of this exists. Simple, but will rarely produce anything of value.
This paragraph from the post gives you the gist of it:
> ...we need to remove humans-in-the-loop, reduce coordination, friction, bureaucracy, and gate-keeping. We need a virtually infinite supply of requirements, engineers acting as pseudo-product designers, owning entire streams of work, with the purview to make autonomous decisions. Rework is almost free so we shouldn’t make an effort to prevent incorrect work from happening.
As if the only reason we ever had POs or designers or business teams, or built consensus between multiple people, or communicated with others, or reviewed designs and code, or tested software, was because it took individual engineers too long to bang out decent code.
AI has just gotten people completely lost. Or I guess just made it apparent they were lost the whole time?
Using an LLM to one shot a small function (something i would do with a very specific search on Google or SO) is handy. Giving it a harness and free access to a code base leads to some terrible code, and doubling down with more instructions and agents in the loop means more time writing the rube Goldberg orchestration rather than just opening up an editor and writing code.
LOL. I had to check if this was published on April 1st.
You are missing another dimension how easy it would be to migrate if adding new feature hits a ceiling and LLM keeps breaking the system.
Imagine all tests are passing and code is confirming the spec, but everything is denormalized because LLM thought this was a nice idea at the beginning since no one mentioned that requirement in the spec. After a while you want to add a feature which requires normalized table and LLM keeps failing, but you also have no idea how this complex system works.
Don't forget that very very detailed spec is actually the code
Came here to say this, but you said it for me. If the problem were merely one of insufficient rigour or detail in specs, it would have been solved long before LLMs.
In the age of AI this is more true than you know. Given a detailed enough spec and test suite you can effectively rewrite any application with any language in a fully automated way.
I've coined that as "Duck coding" :D If it quacks like a duck, walks like a duck and looks like a duck - it's duck enough for as far as the spec is concerned. Does it matter what is inside the duck?
The tests, sure. But certainly not the code itself, as that sits far too close to the implementation (i.e. it is the implementation). An almost infinite number of implantations can fulfill “does foo when bar”, so how can we prove that ours is the spec itself?
It’s kind of like a scientist coming up with a hypothesis post-hoc to fit the results of the experiment.
I've found that adopting RFC Keywords (e.g. RFC 2119 [1]; MUST, SHOULD, MAY) at least makes the LLM report satisfaction. I'd love to see a proper study on the usage of RFC keywords and their effect on compliance and effectiveness.
1. https://www.rfc-editor.org/info/rfc2119/
This just sounds like typical requirements management software (IBM DOORS for example, which has been around since the 90s).
It's kind of funny how AI evangelists keep re-discovering the need for work methods and systems that have existed for decades.
When I worked as a software developer at a big telecom company and I had no say in what the software was supposed to do, that was up to the software design people--they were the ones responsible for designing the software and defining all the requirements--I was just responsible for implementing that behavior in code.
This is too weird for me. At least with programming languages I can consult the documentation and if the programming language isn’t behaving as documented, it’s obviously a defect and if you’re savvy enough you often have open channels that accept contributions. Can we say the same for Claude or other AI solutions?
This was a follow up to a previous article[1] and the pair tried to express what I still think today (using AI daily at work): every time I use AI for coding, to some capacity I'm sacrificing system understanding and stability in favor of programming speed. This is not necessarily always a bad tradeoff, but I think it's important to constantly remind ourselves we are making it.
[1] https://olano.dev/blog/tactical-tornado/