I tried really hard to do this, but it turns out the models don't care about your ideas and want to do what's popular in their training data, so they will happily ignore anything you try to force down their throats, especially as context length grows or if you hit compaction.
So to make best use of the models steer them down familiar paths, mention common pattern and frameworks, use popular packages and languages that have the high median quality online.
I started my project with a few simple interface definitions and a short design / architecture doc that I include in the AGENTS.md file, but no matter how hard I try all of the models just end up ignoring it and sprinkled new seeds of variants of the same stuff all over my code base, that with each new session grow new branches.
right, the model will listen to you for ~32k context; half of that is just getting them into the "downhill" path for being a coding assistant. You can throw in whatever your heart's desire and if it's a single fix, it might listen to you; but treating it like a smart-IDE basically means it'll follow whatever dominant pattern is in the training data.
If we had smart capitalism (the SAME PROBLEM) we'd realize what we want are per-framework or per-OS models that simply are 80% how to english and 20% the tools we need for a given project.
yeah I tried rules, hooks and forbidding things like pip / python3 but it just led to the model failing to do what it wants and a bunch of token churn trying to get around my more rigid constraints.
main problem is that the harness files get loaded into context early in the session and slowly wash away as new information comes in.
i've been playing around with https://github.com/bytedance/deer-flow and they seem to figure out how to get around context sizing issues and has a more automated prompt system.
Essentially the problem with current hanrsses is they just can't find/evict context and want to ensure KV cache isn't evicted.
If they evict the old prompt and tried to keep all the proper rules, then you get token churn anyway, as what you want is an entirely new chain of events where your rules/guidance sit at the front of the model chain.
What we're getting to is there really isn't an economical way for cloud models to do it; you can do it with a local model, but at comfortable speeds, you need smaller sized.
I think a good harness via Qwen3.6-35B-A3B is possible, as the token gen is only 3B since that's the active parameters.
So it's mostly that to keep rules pinned to the top of the context will either bloat the model or it'll evict the KV cache.
The key is to remove degrees of freedom from the agent. IMHO this is the really hard work of agentic engineering. You need to strongly constrain what the agent can even do in the first place to force it into a certain area.
That runs the danger of putting the model out of distribution, of course, so it takes some experimenting.
I'm working on a Ruby project where - on purpose - I'm disallowing any frameworks other than what's standard Ruby. So no Rails or Sinatra, just the inbuilt Webrick web server. Initially it was hard to keep the model on the rails but as I've learned more it gets easier. That said, it's clear to me that the model pulls very strongly in familiar directions in terms of how it writes code, i.e. a lot of what gets written still ends up looking "Railsy" even though my learning project doesn't use it.
It makes everyone more average and same-y. It’s a good average, better than bad, but I cringe everytime I see someone claiming it’s a force multiplier for their personal expression. Bullshit.
For some background on me to set the context for the following opinion:
- been a SRE/DevOps at banks/hedge funds for almost 20 years
- now work in L1 crypto
- have been coding since I was 12 and have also been using frontier models for the past year (including running multiple agents at the same time etc).
My thoughts:
The models are indeed amazing. They can read large codebases, find bugs, infer the root cause of an issue from partial logs etc etc.
They do still hallucinate. WAY less than they used to but it's still non-zero. In a way that's worse b/c the model will spit out a complex piece of software and say "Yep, no mistakes. I even wrote tests and they all pass!" You might think "Phew, that's great!" but in the same way we've all found bugs in production code written by smart people, there will be bugs here too.
I say this not to imply that you have to read all of the code. I say if only to underline that for big complex systems, the "let's write unit tests for the parts that ABSOLUTELY HAVE TO BE CORRECT" is still just as important as it ever was. I'm thinking of examples like:
- the order and execution handler of a trading system
- avionics flight controls
- healthcare related medical devices
- etc
As an example: I was working on a complex system. I wasn't sure if the LLM code was actually correct so I wrote up a quick script that I checked, line by line, to be 100% sure it was working as I expected. I then used that script to double check the LLM. I didn't read all of the code the LLM created. The sense of "ok, now this works" was astounding.
I'll add, a lot of the developers I work with are going this "hybrid" route too where they will have the LLM write code and tests but then go back in and double check.
In closing, a lot of these big rewrites with LLMs are possible only b/c the devs KNOW, FOR A FACT, that the unit/integration tests are correct. I'm still not convinced that you can have LLMs write all of the code and all of the unit tests and be 100% sure that it's all correct. (I will admit that this has always been difficult and even the pre-LLM days were not a guarantee that all of the code wa s correct)
You can have almost 100% tests coverage and green tests and still have a plane crash and LLMs shilling with you that everything is ok. To me it seems we havent learned yet tests coverage means nothing without solid code
What do you mean? Are you arguing that it's impossible to fully translate things? Would you argue that "apple", "manzana", "pomme", "mela", "maçã" etc. all refer to different ideas?
Some of the most exciting engineering work is happening in the DS4 repo - and I'm watching it almost like a sports game.
When the DSpark paper came out[1] the next day we had folks attempting to implement, working together, validating their failures. Eventually their work being synthesized into a PR[1] that admits performance is not ideal. Something antirez alluded to in one of his videos (speculative decoding is a great boon, but mostly for large labs hosting and serving many requests at once, and maybe not so effective for local inference).
There's recent work into "directional steering"[3] that has made it's way into per-session directional steering overrides thanks to audreyt[4].
There's support for the new Hy3[5] model also thanks to audreyt[6].
There's Pre-M5 optimizations[7] in the queue thanks to ivanfioravanti who also helped with some of the initial M5 optimizations.
I haven't watched a repo like this since llama.cpp and whisper.cpp in the early days (though llama.cpp is pretty exciting right now with the SYCL improvements that are flowing in for the new Intel GPUs).
The DS4 repo is a really interesting place to watch folks who heavily code with agents collaborate together in a way that seems pretty effective. I've been really enjoying it.
Thanks! And sorry for not yet merging many of those. The problem is, I'm dealing with tensor parallelism for the CUDA and Metal-RDMA fork right now, so was not albe to care about PR / issues for a lot of time.
I don't want to sound overly dismissive, but for quite a few practical cases pipeline parallelism with micro-batches will be a likely win over tensor parallelism. Of course this inherently comes with a requirement to take batched inference seriously for local use, at least in special cases where this doesn't put too much of a requirement on memory capacity. By comparison, tensor parallelism is probably good wrt. making memory- and KV-cache hungry models like GLM 5.2 and perhaps Kimi genuinely viable in a non-trivial local inference scenario.
The ability to use steering vectors seems really powerful. I know you can build your own, but are there any collections of prebuilt steering vectors? Or even just mentions of what kinds of steering vectors people are building for their own use?
> speculative decoding is a great boon, but mostly for large labs hosting and serving many requests at once, and maybe not so effective for local inference
This is somewhat of a nitpicking point, but AIUI speculative decoding is worthwhile if (1) you have viable unexploited parallelism that can speed up the "verify" step compared to plain decode, and (2) it's profitable for you to load complete model layers into memory in bulk, not just a tiny fraction of active parameters. (2) tends to be true for large hosters, but not only, e.g. if you're running inference on a large dense model (think Mistral Medium) with SSD streaming and a single session (no concurrent batching), speculative decoding can be quite great.
It's also a spectrum, e.g. some MoE models are not very sparse and comparatively few concurrent requests might already span most of the experts, such that loading a complete set of layer parameters and doing speculative decoding to increase compute intensity actually becomes worthwhile.
Very uncomfortable to read. The proposed idea of not reading the code you ostensibly wrote, doesn’t just challenge the traditional workflow or methodology, it challenges the identity of a programmer itself. For experienced, respected programmers like antirez perhaps reading and writing code truly has just gotten in the way of the ideas, but I cannot identify with that perspective. The devil is in the details, reading other peoples code (and code the LLM writes) evolves the idea itself, and changes my understanding of it. My view on this is so biased by the direct relation of code I read and write to feeding myself and my family (the vast majority of my programming is for a paycheck) that suggestions of removing myself from the process feel grim, not exciting.
For complex code changes or bigger features I often spend hours with an LLM refining architecture, exploring alternate directions, trying to find alternate directions in the first place, clarifying design questions, etc. It's not that you spend less time refining ideas, you just do it at another level of abstraction
But that does come with tradeoffs, and it's not the right thing for every project. But when it does work it does increase the amount of work you get out in the same time
>For complex code changes or bigger features I often spend hours with an LLM refining architecture
The problem here is that the LLM hallucinates, so
* it will tell that something is a bad idea, even when it is not.
* it will miss good ideas.
But here is the thing. Even discussing with a rubber duck can do wonders to your thought process. So may people who are noticing the usefulness of this procedure might be actually just doing a slightly better rubber ducking...
The LLM will often miss the most obvious simplifications. And if I ask it to present me with six approaches how we could solve a problem, chances are we will settle on number seven or eight, both my ideas after rejecting all the other six. But those first six were still valuable for coming up with the version we actually settle on. Making the rubber-duck talk is genuinely useful.
And the rubber duck is actually pretty good at the localized grunt work, so you can spend more time talking with it about big-picture stuff
But the CPU is not executing the textual description of your architecture. It executes code.
Honestly, I've found that the architecture described to and by LLMs is always a more rosy picture than what is actually generated, no matter how many times you do an adversarial review. It's less visceral than in image generation, but the pattern is the same -- the broad strokes seem fine but the details are awful.
> I often spend hours with an LLM refining architecture, exploring alternate directions, trying to find alternate directions in the first place, clarifying design questions,
Is that something recent? I’ve been doing this for years, even in high school where we were asked essays on contrast between different viewpoints.
I don’t need LLM to do this. It’s the 101 of any engineering process to not rush with the first thought you have and indeed try to explicitly explore the solution space and base your decisions according to defined tradeoffs.
I'm fully on the agentic coding train and haven't manually typed code in a while so I understand the sentiment around moving beyond this level of analysis. However, how can this advice work practically speaking, day to day? Especially at ones day job?
I'm sitting here working and just caught a handful or really bad decisions by the agent, one right after the other, cascading from an assumption that was incorrect. My ideas and architecture are sound in this codebase! Are those things simply to remain in the code if they work 'good enough', or are there consequences right around the corner?
I have found including this in my AGENTS.md to be quite transformative in this regard:
> Always use an aggressive red/green TDD-approach. It is critical to remember that in the red phase, things like module/exports import failures due to trying to import file paths that don't yet exist, exports that don't yet exist, etc. is not valid TDD. For valid TDD, the test cases must actually run. For this, you must create stubs of the expected modules and exports in the red phase, so that the test cases actually run and fail on the test case assertions themselves. In some cases, when using this approach, once in a while some of the red phase test cases might "incidentally" pass, and this is ok. Before running red phase tests you should always make predictions about the number of test cases you expect to fail/pass -- this count is not the number of test files or test suites, but rather the number of test cases. By performing these red phase expected counts of passing/failing test cases, it will help you catch errors in your prior reasoning quickly and efficiently.
> Always use a proof-driven, scientific method-based approach to validate hypotheses, assumptions, and conclusions: define the smallest falsifiable hypothesis, create or identify a reproducible failing case, gather direct evidence, make the smallest targeted change, and then re-run the same proof to confirm the issue is fixed. Avoid speculative fixes, broad rewrites, or changing multiple variables at once. When possible, preserve the reproduction as a regression test before implementing the fix. Consider that when gathering evidence, additional logging and durable files can be very helpful.
> The strict TDD and proof-driven approaches described above could be described as "proof-driven development". Try to internalize and generalize these concepts, as they are broadly applicable.
I am not sure I'm buying this. The raison d'être for our existing software engineering methods is that humans make mistakes and we needed to contain the effects of these mistakes; and without an appropriate methodology to do that, software defects will just accumulate over time. Worse, once they show up, nobody understands the code well enough to do anything about them, or at least not without considerable time investment.
This does not change with agents doing the coding. Coding agents make mistakes also. Not very often nowadays, but neither do competent human programmers. And without a methodology to keep problems in check your agentic code will also accumulate software defects over time and result in code that becomes less and less maintainable, because you have no mental model of the software.
Antirez is correct in pointing out that slop existed before we started to use LLMs for programming; I've worked with my share of really ugly legacy code myself. But the problems do not magically disappear in the LLM age, no matter how good your model is. They remain, as every model is ultimately a heuristic (albeit a very powerful one), and no heuristic is 100% accurate.
This does not mean that coding agents are useless; used correctly, they can be enormously powerful accelerators for the software development (and validation!) process, because combining your strengths with those of a modern LLM is generally a substantial net gain. But that must still happen as a part of an approach that results in maintainable software with minimal defects.
Personally, I primarily use agents as virtual pair programmers these days, which I find very useful. This is an iterative process with relatively small and contained changes, where "looking at the code" is just part and parcel of following along and building a mental model of the resulting piece of software.
I think the gist of what the aurhthor is saying is, AI is not good enough to just give one vague prompt to and let it go, but it's good enough for you to give it a "design" and then not worry about the actual code it writes. But you need to do a lot of QA still. And you still need to learn to code and write some code, I guess so you can give the LLM good instructions? But if giving good enough instructions requires some level of coding skill, how are you going to gain that skill if you don't do much programming or reading code?
It all sorta feels like an old guy (he says hes's old in TFA) who forgot how he got to where he is today trying to give advice. Be careful what you believe, young programmers.
This position assumes that purely LLM-written codebases are able to scale infinitely.
IMO there isn’t enough evidence for me to feel comfortable in that judgement.
Anecdotally, I find that pretty often LLMs (even bleeding-edge models) write unidiomatic/unscalable/poorly-abstraced code when working in large codebases.
Ideas are a dime a dozen. All of us have half a dozen of what appear to be good ideas every. Execution matters, testing and sanity checking matters, actual engagement with users and iteration matters.
Sure, we're reducing the cost of idea -> prototype to near zero (well, as long as tokens are free or nearly free), but that just means we now have mountains of throw away code, within which there may a gem or two.
Nothing yet has replaced the curating of ideas that good teams do as a matter of course.
I think "idea" is getting overloaded here. The article isn't talking about new feature/product/tool ideas, but about what concepts to use to build those features. e.g. what data structures and algorithms to use.
So your last line agrees with the article, I think. They are saying that it's still important to curate ideas, and no longer important to read the code; the time you would have spent reading the code should be spent curating ideas.
Curating ideas requires experience and taste. Can you really develop that without looking at code? Compare Steve Jobs vs Joni Ive curating as a prime example of how taste without practical experience and context can lead things astray. And the worst of it is that it can take a while for the problem to be easily noticeable.
> Fable was used in several iterations to test for security issues before each release.
How? The point of Fable (vs. Mythos) is that it has extreme guardrails against doing e. g. security work, even mentioning "security" or "vulnerability" in the prompt will shut it down.
That if you polish it a bit, you can probably write a short story. /s
The thing is, making claims like that doesn’t prove anything. You have to either show the result or prove that is reproducible. Otherwise, it may as well be something you dream up this morning.
Execution is not the code, but how do you decide to do every part. "Idea does not matter, execution does" always meant: "big generic ideas don't matter, it is how you organize it in the myriad of details it is composed of (in a given incarnation of the general idea) that matters."
But why aren't lower level "ideas about execution" more clearly expressed in code? Or at least pseudocode?
Many of us would struggle with our reading comprehension of an English description of an algorithm. But the pseudo / actual code? Much easier to reason about.
I'm not saying you should review even 90% of code. But it feels like an arbitrary line to say "never look at code". It's like never taking a wrench to a car to look at what's built to see whether you trust the car factory.
> Matteo Collina yesterday asked me, in reply to my tweet: but didn’t you say that you check all the AI generated code for Redis? And this is a good question indeed. Yes, I do, but this is, at this point, something I need to do but that I believe to be mostly pointless, partially once GPT 5.5 was released, but now with Fable and GPT 5.6 Sol even more. Yes: I identify things that I don’t like how they are coded, but if I open other Redis files written by other Redis contributors there is far worse, and not since they are not good coders, but because it is a matter of taste.
I wonder why he HAS TO ("I need to...") review the code even if he thinks it is pointless? Is that because his employer, Redis, requires it?
I also got a hint from this that he worries the community would reject features in Redis (and Redis itself) if they expected that the code has not been human-reviewed.
I'm facing this with some of my own open source projects: I feel a responsibility to manually review the code because I don't want to damage the reputation of those projects by shopping code I haven't looked at myself, even where I'm confident that it works and is well structured already.
I really don't understand why we feel the need to drop the review aspect. Programming with LLMs is a very non-linear process, there's no prompt-to-code mapping where editing a part of the prompt would produce an identical code construct with just a small part of it changed. LLMs are incapable of ensuring the thing works as expected, and the prompt itself is lossy too. Without review, we have no tangible way to interact with the system being constructed.
I believe issues will arive with AI in the near future in terms of their performance. Disagreeing with the author, I think code quality pre-AI, in certain languages, were golden. They all contained bugs but they were written in a way that bug sources would be more or less obvious and most but not all top open source software had linting guidelines, code styles & friends to help make it ingestible. Perhaps these helped said project's maintainers a great lot, but it absolutely SAVED AI when it came to training data. These top-quality repositories with thousands of lines of great code combined with hundred thousand lines of mixed quality code was the perfect formula for the big data churner™. Now what? All those repos, inflated with stars are generated by one of the 5 LLMs out there. There has been extended discussions about deterministic behaviour in LLMs, and I'm no ML engineer but to me this drop in entropy will surely cause backtracking in code quality. Of course there a vast array of improvements that can be made outside of training data, but the whole psychology attached to LLM marketing, in my opinion, obstructs those improvements. You could train a better suited way of input for LLMs, set cutoffs and posttraining in just the right places and everything but you gotta think it through. How many of those who ask their favourite LLM everything think anymore? How much of our current knowledge is safe from AI hallucinations or subtle nudges? AI has changed the world of coding, but the current state in Anthropic HQ will determine if it will just be mid-quality codegen or accelerationist fever dream.
I think it's related that we as humans see when something becomes hard to reason about, and decide to refactor it. I'm not sure whether an LLM with full ownership of a codebase could do that, for its own benefit.
I do see a world where models could be trained on it, but I imagine it will be more expensive, because it requires including future rewards about the models' own later efficiency.
Code is not literature. It's not poetry. It does not express the human spirit. Code is a machine made of symbols. Engineers do care about the beauty of their machine designs, but in the end what matters is whether the machine performs its function correctly, efficiently, affordably. What's wonderful about software is that the function of software is to take input data and produce output data. Every aspect of it is measurable, if we build it to be measurable, because it's data all the way down. The future of software development is not reading code. It's specifying an outcome, success and failure conditions, and iterating until that outcome is reached. Throw out your software engineering "code quality" manuals, your SOLID and your Clean Code. It doesn't matter now.
This argument is so old and tired. It started long before LLMs. People thought senior engineers could write a detailed spec and then the actual coding could be outsourced to just any bunch of cheap programmers. It never worked. A detailed enough English specification doesn't look very different from code written in our current high level programming languages
I.e. if by its structure we can conclude that our test cases are sufficient. Because if we don't know its structure then we don't know there aren't undisclosed sub-partitions of (combinatorially exploded) test space where it breaks.
How do we know its structure? By specifying it. How do we do that in enough detail? Code. Or we use static analysis I guess. Which is harder.
the model does not make the architecture decisions, you make the architecture decisions. you define what success is, including performance characteristics. fuzzing and property based testing ensure correctness. what does it matter what the code looks like if it the machine you've created fulfills the criteria you define?
I agree that not having to read the code is the future but I struggle to get there because the AI, like humans, can make mistakes (not follow instructions). For example, I might ask the AI to render a scene using a physical model, and we might come to an agreement at the plan stage, but then it will go off and cut some corner in implementation. So the question is how to reliably verify compliance. If you merely tell another AI to check it properly implemented the plan it too can miss bugs, so you need to put all your effort in making the acceptance tests foolproof, and this is easier said than done. You need a defense in depth approach, using the type system, unit tests, formal verification (if you can), linting, etc. It is an open question how to optimally allocate your correctness efforts for a given budget but, as models get smarter, it is clear to see that the amount of human diligence needed to achieve likely correctness for a given task (say, in LOC) is going to tend to zero.
>if you control the ideas of your software, looking at the code itself is suboptimal and often pointless.
This requires developers to have absolute and unconditional Trust in the LLM. It's not easy to trust it completely to the point of completely ignoring the implementation details of the code.
In one of Salvatore's discussions, he mentioned that he hasn't even opened a single file of DS4. This is a courageous choice.
But the real question is: if the younger generation stops writing code, how are they supposed to develop that "forma mentis" (mindset) that allows them to reason about design and architecture? It's only by *writing* the code that you gradually internalize development and design patterns, specifically by clashing with the "brutality" of bugs and solving implementation problems.
P.S. I read Wohpe. It's fascinating how back in 2022 (I think?) Salvatore already wrote down many insights that have actually come true (including, for instance, the ban on "strong artificial intelligence"...).
So I suppose that the future will touch the very development of humanity (like the Genesi project :) )
I agree that in the end the most important thing is to have the correct mental model of the code, and there is no need to know every implementation detail if you can ensure with other means that they address your requirements. An issue I can can see with not reviewing the code (especially code someone else wrote) is how do you create the mental model? Can a design.md file replace the process of slowly understanding the data flow by reading?
I'm letting go of reading every single line, especially within well-scoped modules that don't affect anything else. On the other hand I struggle to form the mental model required to "control the ideas", as it were, without reading at least some critical sections of the code and without grasping how the fundamental data structures relate to each other.
Are we saying that opening the editor is basically a mistake?
No its fine. The problem is trying to read all the code. Reading some is fine. I tend to focus on public APIs, and key areas. But not the thousands of lines which arre tests or will be refactored.
I strongly agree with antirez. I believe that an intellectually honest programmer should recognize where their knowledge and experience is mostly beneficial, and at this point (and we've seen clear signs for some time) coding is largely solved. But software engineering is not just coding, and it involves a whole set of other tasks requiring direction and creativity that can greatly influence the quality and impact of the software.
The most common arguments against this view seem to arise either from ideological resistance, which I understand given how painful it can be to see one’s job at risk or an important part of one’s identity taken away, or from generalizing a small number of experiences with LLMs to the technology as a whole. In the latter case, those experiences may also be heavily conditioned by the user’s inexperience in working with LLMs, or by the specific use case in which they were applied. There are still certain tasks that not all models can handle reliably as of now, however some can (usually the most expensive), and they will likely continue to improve over time.
"coding is largely solved" and "self-driving is largely solved" smell exactly the same to me.
ETA: I bristle deeply at your idea that anyone who doesn't agree with you is letting their emotions get the better of them. Perhaps the world is not as black and white as you're painting it?
What does it mean for coding to be solved, and software engineering to not have been solved? If it implies there's now a set protocol that can be followed to reach good results, how has that not been the case before? If it means that we can reach better results than before, then how do we know that can't be improved further? Or does that mean we should know how English language, instead of computer code, maps to computer instructions?
Actually, this paper shows that the LLMs create a lot more slop and degradation of the code after the first prompt. https://arxiv.org/html/2603.24755v1.
So iterating with the LLM will simply make it to produce worse and worse code.
> Then I compared the implementation, for correctness, to other systems, finding that other implementations sometimes contained more errors. I researched more, and found that the local inference world is full of subtle errors that accumulate and damage the model output, issues in the attention implementation causing performance slopes after the context is over a certain limit because indexed attention implementations are broken (do more work than they should, for instance), and so forth.
I agree 100% that AI helps a lot with that. But I feel like there's something missing between "AI helps a lot with that" and "I believe reading code is mostly pointless". I genuinely wonder how the above can be accomplished without reading any code.
> I genuinely wonder how the above can be accomplished without reading any code.
We can't see the code for the laws of physics, and yet experiment by experiment we've come a long way.
That's not usually going to be the most efficient way to understand code, but I've found that it's pretty useful to be one of the few empiricists in an organization full of rationalists (software engineers are typically rationalists in my experience). I wouldn't propose that we dispense with rationalism altogether, but it does seem that the we'll benefit by achieving a better balance than we've traditionally had.
Never thought I'd find myself questioning Antirez, but I have many questions about this post and overall attitude. For example:
> Yes: I identify things that I don’t like how they are coded, but if I open other Redis files written by other Redis contributors there is far worse, and not since they are not good coders, but because it is a matter of taste.
Why is the attitude here about keeping the floor up rather than raising the bar?
Why can't we have better code with AIs? Its not impossible to do!
When I implement things by basically pairing with the AI, I end up with better designs/architectures/code than I could have written by myself.
It sounds like some people think of the AI code paradigm as one where there will be fewer but better devs producing code/designs of lower quality than they could individually produce, but that is higher quality than the average dev could produce.
Is that really better than a world where AI raises the average across the board at the expense of a bit of speed? At the very least, it seems like a far less risky and less disruptive way to still capture significant benefits from AI.
113 comments
[ 51.8 ms ] story [ 30.4 ms ] threadSo to make best use of the models steer them down familiar paths, mention common pattern and frameworks, use popular packages and languages that have the high median quality online.
I started my project with a few simple interface definitions and a short design / architecture doc that I include in the AGENTS.md file, but no matter how hard I try all of the models just end up ignoring it and sprinkled new seeds of variants of the same stuff all over my code base, that with each new session grow new branches.
If we had smart capitalism (the SAME PROBLEM) we'd realize what we want are per-framework or per-OS models that simply are 80% how to english and 20% the tools we need for a given project.
But that's no where the money is.
main problem is that the harness files get loaded into context early in the session and slowly wash away as new information comes in.
Essentially the problem with current hanrsses is they just can't find/evict context and want to ensure KV cache isn't evicted.
If they evict the old prompt and tried to keep all the proper rules, then you get token churn anyway, as what you want is an entirely new chain of events where your rules/guidance sit at the front of the model chain.
What we're getting to is there really isn't an economical way for cloud models to do it; you can do it with a local model, but at comfortable speeds, you need smaller sized.
I think a good harness via Qwen3.6-35B-A3B is possible, as the token gen is only 3B since that's the active parameters.
So it's mostly that to keep rules pinned to the top of the context will either bloat the model or it'll evict the KV cache.
That runs the danger of putting the model out of distribution, of course, so it takes some experimenting.
I'm working on a Ruby project where - on purpose - I'm disallowing any frameworks other than what's standard Ruby. So no Rails or Sinatra, just the inbuilt Webrick web server. Initially it was hard to keep the model on the rails but as I've learned more it gets easier. That said, it's clear to me that the model pulls very strongly in familiar directions in terms of how it writes code, i.e. a lot of what gets written still ends up looking "Railsy" even though my learning project doesn't use it.
- been a SRE/DevOps at banks/hedge funds for almost 20 years
- now work in L1 crypto
- have been coding since I was 12 and have also been using frontier models for the past year (including running multiple agents at the same time etc).
My thoughts:
The models are indeed amazing. They can read large codebases, find bugs, infer the root cause of an issue from partial logs etc etc.
They do still hallucinate. WAY less than they used to but it's still non-zero. In a way that's worse b/c the model will spit out a complex piece of software and say "Yep, no mistakes. I even wrote tests and they all pass!" You might think "Phew, that's great!" but in the same way we've all found bugs in production code written by smart people, there will be bugs here too.
I say this not to imply that you have to read all of the code. I say if only to underline that for big complex systems, the "let's write unit tests for the parts that ABSOLUTELY HAVE TO BE CORRECT" is still just as important as it ever was. I'm thinking of examples like:
- the order and execution handler of a trading system
- avionics flight controls
- healthcare related medical devices
- etc
As an example: I was working on a complex system. I wasn't sure if the LLM code was actually correct so I wrote up a quick script that I checked, line by line, to be 100% sure it was working as I expected. I then used that script to double check the LLM. I didn't read all of the code the LLM created. The sense of "ok, now this works" was astounding.
I'll add, a lot of the developers I work with are going this "hybrid" route too where they will have the LLM write code and tests but then go back in and double check.
In closing, a lot of these big rewrites with LLMs are possible only b/c the devs KNOW, FOR A FACT, that the unit/integration tests are correct. I'm still not convinced that you can have LLMs write all of the code and all of the unit tests and be 100% sure that it's all correct. (I will admit that this has always been difficult and even the pre-LLM days were not a guarantee that all of the code wa s correct)
When the DSpark paper came out[1] the next day we had folks attempting to implement, working together, validating their failures. Eventually their work being synthesized into a PR[1] that admits performance is not ideal. Something antirez alluded to in one of his videos (speculative decoding is a great boon, but mostly for large labs hosting and serving many requests at once, and maybe not so effective for local inference).
There's recent work into "directional steering"[3] that has made it's way into per-session directional steering overrides thanks to audreyt[4].
There's support for the new Hy3[5] model also thanks to audreyt[6].
There's Pre-M5 optimizations[7] in the queue thanks to ivanfioravanti who also helped with some of the initial M5 optimizations.
I haven't watched a repo like this since llama.cpp and whisper.cpp in the early days (though llama.cpp is pretty exciting right now with the SYCL improvements that are flowing in for the new Intel GPUs).
The DS4 repo is a really interesting place to watch folks who heavily code with agents collaborate together in a way that seems pretty effective. I've been really enjoying it.
[1]: https://arxiv.org/abs/2607.05147
[2]: https://github.com/antirez/ds4/pull/502
[3]: https://arxiv.org/html/2406.00045v2
[4]: https://github.com/antirez/ds4/pull/148
[5]: https://hy.tencent.com/research/hy3
[6]: https://github.com/antirez/ds4/pull/523
[7]: https://github.com/antirez/ds4/pull/555
This is somewhat of a nitpicking point, but AIUI speculative decoding is worthwhile if (1) you have viable unexploited parallelism that can speed up the "verify" step compared to plain decode, and (2) it's profitable for you to load complete model layers into memory in bulk, not just a tiny fraction of active parameters. (2) tends to be true for large hosters, but not only, e.g. if you're running inference on a large dense model (think Mistral Medium) with SSD streaming and a single session (no concurrent batching), speculative decoding can be quite great.
It's also a spectrum, e.g. some MoE models are not very sparse and comparatively few concurrent requests might already span most of the experts, such that loading a complete set of layer parameters and doing speculative decoding to increase compute intensity actually becomes worthwhile.
But that does come with tradeoffs, and it's not the right thing for every project. But when it does work it does increase the amount of work you get out in the same time
The problem here is that the LLM hallucinates, so
* it will tell that something is a bad idea, even when it is not. * it will miss good ideas.
But here is the thing. Even discussing with a rubber duck can do wonders to your thought process. So may people who are noticing the usefulness of this procedure might be actually just doing a slightly better rubber ducking...
The LLM will often miss the most obvious simplifications. And if I ask it to present me with six approaches how we could solve a problem, chances are we will settle on number seven or eight, both my ideas after rejecting all the other six. But those first six were still valuable for coming up with the version we actually settle on. Making the rubber-duck talk is genuinely useful.
And the rubber duck is actually pretty good at the localized grunt work, so you can spend more time talking with it about big-picture stuff
Honestly, I've found that the architecture described to and by LLMs is always a more rosy picture than what is actually generated, no matter how many times you do an adversarial review. It's less visceral than in image generation, but the pattern is the same -- the broad strokes seem fine but the details are awful.
Is that something recent? I’ve been doing this for years, even in high school where we were asked essays on contrast between different viewpoints.
I don’t need LLM to do this. It’s the 101 of any engineering process to not rush with the first thought you have and indeed try to explicitly explore the solution space and base your decisions according to defined tradeoffs.
I'm sitting here working and just caught a handful or really bad decisions by the agent, one right after the other, cascading from an assumption that was incorrect. My ideas and architecture are sound in this codebase! Are those things simply to remain in the code if they work 'good enough', or are there consequences right around the corner?
I have found including this in my AGENTS.md to be quite transformative in this regard:
> Always use an aggressive red/green TDD-approach. It is critical to remember that in the red phase, things like module/exports import failures due to trying to import file paths that don't yet exist, exports that don't yet exist, etc. is not valid TDD. For valid TDD, the test cases must actually run. For this, you must create stubs of the expected modules and exports in the red phase, so that the test cases actually run and fail on the test case assertions themselves. In some cases, when using this approach, once in a while some of the red phase test cases might "incidentally" pass, and this is ok. Before running red phase tests you should always make predictions about the number of test cases you expect to fail/pass -- this count is not the number of test files or test suites, but rather the number of test cases. By performing these red phase expected counts of passing/failing test cases, it will help you catch errors in your prior reasoning quickly and efficiently.
> Always use a proof-driven, scientific method-based approach to validate hypotheses, assumptions, and conclusions: define the smallest falsifiable hypothesis, create or identify a reproducible failing case, gather direct evidence, make the smallest targeted change, and then re-run the same proof to confirm the issue is fixed. Avoid speculative fixes, broad rewrites, or changing multiple variables at once. When possible, preserve the reproduction as a regression test before implementing the fix. Consider that when gathering evidence, additional logging and durable files can be very helpful.
> The strict TDD and proof-driven approaches described above could be described as "proof-driven development". Try to internalize and generalize these concepts, as they are broadly applicable.
This does not change with agents doing the coding. Coding agents make mistakes also. Not very often nowadays, but neither do competent human programmers. And without a methodology to keep problems in check your agentic code will also accumulate software defects over time and result in code that becomes less and less maintainable, because you have no mental model of the software.
Antirez is correct in pointing out that slop existed before we started to use LLMs for programming; I've worked with my share of really ugly legacy code myself. But the problems do not magically disappear in the LLM age, no matter how good your model is. They remain, as every model is ultimately a heuristic (albeit a very powerful one), and no heuristic is 100% accurate.
This does not mean that coding agents are useless; used correctly, they can be enormously powerful accelerators for the software development (and validation!) process, because combining your strengths with those of a modern LLM is generally a substantial net gain. But that must still happen as a part of an approach that results in maintainable software with minimal defects.
Personally, I primarily use agents as virtual pair programmers these days, which I find very useful. This is an iterative process with relatively small and contained changes, where "looking at the code" is just part and parcel of following along and building a mental model of the resulting piece of software.
It all sorta feels like an old guy (he says hes's old in TFA) who forgot how he got to where he is today trying to give advice. Be careful what you believe, young programmers.
IMO there isn’t enough evidence for me to feel comfortable in that judgement.
Anecdotally, I find that pretty often LLMs (even bleeding-edge models) write unidiomatic/unscalable/poorly-abstraced code when working in large codebases.
Sure, we're reducing the cost of idea -> prototype to near zero (well, as long as tokens are free or nearly free), but that just means we now have mountains of throw away code, within which there may a gem or two.
Nothing yet has replaced the curating of ideas that good teams do as a matter of course.
So your last line agrees with the article, I think. They are saying that it's still important to curate ideas, and no longer important to read the code; the time you would have spent reading the code should be spent curating ideas.
How? The point of Fable (vs. Mythos) is that it has extreme guardrails against doing e. g. security work, even mentioning "security" or "vulnerability" in the prompt will shut it down.
I can't speak for the above-mentioned project though, because I wasn't part of that release.
That if you polish it a bit, you can probably write a short story. /s
The thing is, making claims like that doesn’t prove anything. You have to either show the result or prove that is reproducible. Otherwise, it may as well be something you dream up this morning.
Many of us would struggle with our reading comprehension of an English description of an algorithm. But the pseudo / actual code? Much easier to reason about.
I'm not saying you should review even 90% of code. But it feels like an arbitrary line to say "never look at code". It's like never taking a wrench to a car to look at what's built to see whether you trust the car factory.
I wonder why he HAS TO ("I need to...") review the code even if he thinks it is pointless? Is that because his employer, Redis, requires it?
I'm facing this with some of my own open source projects: I feel a responsibility to manually review the code because I don't want to damage the reputation of those projects by shopping code I haven't looked at myself, even where I'm confident that it works and is well structured already.
I do see a world where models could be trained on it, but I imagine it will be more expensive, because it requires including future rewards about the models' own later efficiency.
I.e. if by its structure we can conclude that our test cases are sufficient. Because if we don't know its structure then we don't know there aren't undisclosed sub-partitions of (combinatorially exploded) test space where it breaks.
How do we know its structure? By specifying it. How do we do that in enough detail? Code. Or we use static analysis I guess. Which is harder.
This requires developers to have absolute and unconditional Trust in the LLM. It's not easy to trust it completely to the point of completely ignoring the implementation details of the code.
In one of Salvatore's discussions, he mentioned that he hasn't even opened a single file of DS4. This is a courageous choice.
But the real question is: if the younger generation stops writing code, how are they supposed to develop that "forma mentis" (mindset) that allows them to reason about design and architecture? It's only by *writing* the code that you gradually internalize development and design patterns, specifically by clashing with the "brutality" of bugs and solving implementation problems.
P.S. I read Wohpe. It's fascinating how back in 2022 (I think?) Salvatore already wrote down many insights that have actually come true (including, for instance, the ban on "strong artificial intelligence"...). So I suppose that the future will touch the very development of humanity (like the Genesi project :) )
I'm letting go of reading every single line, especially within well-scoped modules that don't affect anything else. On the other hand I struggle to form the mental model required to "control the ideas", as it were, without reading at least some critical sections of the code and without grasping how the fundamental data structures relate to each other.
Are we saying that opening the editor is basically a mistake?
The most common arguments against this view seem to arise either from ideological resistance, which I understand given how painful it can be to see one’s job at risk or an important part of one’s identity taken away, or from generalizing a small number of experiences with LLMs to the technology as a whole. In the latter case, those experiences may also be heavily conditioned by the user’s inexperience in working with LLMs, or by the specific use case in which they were applied. There are still certain tasks that not all models can handle reliably as of now, however some can (usually the most expensive), and they will likely continue to improve over time.
ETA: I bristle deeply at your idea that anyone who doesn't agree with you is letting their emotions get the better of them. Perhaps the world is not as black and white as you're painting it?
So iterating with the LLM will simply make it to produce worse and worse code.
I agree 100% that AI helps a lot with that. But I feel like there's something missing between "AI helps a lot with that" and "I believe reading code is mostly pointless". I genuinely wonder how the above can be accomplished without reading any code.
We can't see the code for the laws of physics, and yet experiment by experiment we've come a long way.
That's not usually going to be the most efficient way to understand code, but I've found that it's pretty useful to be one of the few empiricists in an organization full of rationalists (software engineers are typically rationalists in my experience). I wouldn't propose that we dispense with rationalism altogether, but it does seem that the we'll benefit by achieving a better balance than we've traditionally had.
Interesting perspective. Haven't really thought through that lens
I feel like everyone is being too black and white. And everyone has different ways of staying in flow with a software task.
Because a critical bug could be anywhere, even in a simple function. Particularly when it is written by an LLM.
Who knows when it chooses to have a bad hallucination?
> Yes: I identify things that I don’t like how they are coded, but if I open other Redis files written by other Redis contributors there is far worse, and not since they are not good coders, but because it is a matter of taste.
Why is the attitude here about keeping the floor up rather than raising the bar?
Why can't we have better code with AIs? Its not impossible to do!
When I implement things by basically pairing with the AI, I end up with better designs/architectures/code than I could have written by myself.
It sounds like some people think of the AI code paradigm as one where there will be fewer but better devs producing code/designs of lower quality than they could individually produce, but that is higher quality than the average dev could produce.
Is that really better than a world where AI raises the average across the board at the expense of a bit of speed? At the very least, it seems like a far less risky and less disruptive way to still capture significant benefits from AI.