>> A more serious irony is that the automatic control system has been put in place because it can do the job better than the operator, yet the operator is being asked to monitor that it is working effectively.
Automatic control is good for standard case, and the expected errors. For weird cases, for now humans are better to detect them and press the big red button.
Except experience shows they aren't. Experience shows that when humans are just pressing "ok" 99% of the time they'll press it 100% of the time and completely miss the other cases.
This is a matter of appropriate training and regular refreshing over time.
The best known example is the Japanese method of "Pointing and Calling/Naming" in their Train safety protocols but has been adapted to other industries too - https://en.wikipedia.org/wiki/Pointing_and_calling
These fall under "Behaviour Based Safety" (BBS) training methodologies and use techniques like "Check-backs", "Call-outs", "STAR" etc.
Jumping into article, I assumed that the author was going to describe how he enjoys, by drawing it out as far as he can,
the process of surrendering to the machine god..
(Not the machina diabol, as it sadly turned out to be, though some BDSM zealots might disagree.. the things that bureaucracy does to us are mild in comparison? )
But.. we do monitor whether our prayers are effective .. right? Is it a smitable offense?
After that disappointment, I was also "curious" if the author was going to talk about "curiosity", the magic virtue that we hackers believe, lifts design into the divine
The better an automatic control is, the fewer mistakes it makes, the fewer times the human needs to react, the more lax and less vigilant the human becomes -- and less skilled when they need to step in. Source: https://imgur.com/a/Vj2TaDz
"Yup, looks good to me!" said the human reviewing thousands of lines of generated code everyday.
This has become common now, and I've seen it ruin some larger projects until nobody understands the codebase full of verbose nonsensical code and comments. It becomes impossible to further develop the project without AI. Contributors not using AI for any reason are all leaving because no one enjoys working with mindless robots (and humans) as colleagues. It's demeaning and devaluing their labor and humanity. They're not getting "left behind", they are leaving the madness of crowds behind.
As this technology permeates society, the challenge to maintain control, coherence, and quality will become greater and ever more serious - for example, AI in healthcare, military, management and governance.
Even if the human's judgement is correct in every LGTM, there's a bias towards merging. Each merge that adds a feature but degrades understandability 1% will be merged. It won't be held back until it improves it by 5% instead. After 100 merges you have degraded understandability 100% even though nobody made an error.
You’re completely right that nobody is stopping a PR that just ever so slightly decreases understandability of something. It may not even be obvious to the reviewer that it’s slightly less understandable overall… Doubly so if they’re just skimming due to the volume, and tripley so if they’re having agents do the most of the review, which lets the human reviewer’s guard down.
But before LLMs, it didn’t matter because on a good team you’d have a balance where maybe some PRs added complexity and some simplified it. Sometimes people would include a small related cleanup or refactor in a PR for code around what they were doing if they noticed complexity. That has become far less common, I think.
I know it’s not your point, but a 1% decrease 100 times is 0.99^100 = 0.366 which is a 63.4% decrease in understandability. Just throwing that out there
To be fair, models are more accurate than how they were earlier this year. In march I was still frustrated and was writing my own event loops because the model simply wasn't built for it. By January I was still hand checking statistical outputs to make sure it wasn't making any errors, and it was. After august it was pretty much over. There were no stats errors, code generated was shit but it could do what is asked. Instruction following is still a problem, and I sometimes catch silly things in the scrolling diffs, but it has been much easier since then.
I hope there's still value to be found in a bunch of developers standing around a whiteboard and talking, and I think this scenario is where their ability and understanding really shows these days.
If we don't value that though--if we don't value understanding the system and just thinking through problems as a group--then let's stop pretending and just not do code reviews. Probably nobody cares and just types "LGTM" after having their own LLM do a spot check to cover their ass.
It's really easy to push developers to stop caring. If management says "go fast fast fast, use those LLMs!", then developers can't keep up, get frustrated, stop understanding the system, and at that point they're just there to collect a paycheck before society collapses. Or, if management is making dumb decisions all the time, same thing.
Hopefully as implementing ideas takes less and less time and resources, smart people who actually care will start building things with other smart people who care.
I've had someone approve my PRs within a minute of me opening them, and likewise expect immediate approvals on his own PRs. When asked questions about parts of the code sometimes the answer would be along the lines of "Claude did it this way", or verbose AI-generated responses to the question.
This is copium that is gone with the wind when the models perform up to the level of standard that the user requires. I'll give the author 6 months before they feel like its okay to just send it.
Karpathy had the same trajectory. Primeagean had the same trajectory. I think in the end everyone except for Jonathan Blow will have the same trajectory
Writing is hard, but what I was trying to say is that it's okay to just send it as long as you're doing it consciously and intentionally (and I do that frequently enough); but if it's not an explicit choice that can be problematic.
Why would Prime be in that list. He had an entire arc where he tried to make it work and then went back to the drawing board.
I am not saying he writes all code by hand, of course. Just that he is furthest in figuring out some sort of mid point between full comprehension and cognitive surrender.
I have missed the more recent content, so perhaps I have missed something, but this was my read last I watched.
I'm also not sure why anyone would attach importance to his opinions as a software engineer, he's exceptional in generating entertaining content but as an engineer he does not seem above average. Definitely not even close to the category of Jonathan Blow or Andrej Karparthy
I am often thinking recently of the littel loop diagrams from the "The Capability Trap" paper. We need to keep a loop reinvesting into our own Human capability in anything we're doing.
Great post. I particularly like the 4 column pr view idea. I’ve built something similar to review speckit specs. The scaffolding to ensure that code actually works makes sense too.
There’s one thing that’s bothering me though — all of these have us consuming what LLMs produce. I learned and internalized how things worked by building systems. At least within the constraints of the abstractions I was using. This makes me think of TikTok and shorts. Everybody is just watching. Maybe that’s not charitable though. Maybe the abstraction has now changed from an “os/gui/event loop” to “llm” and the rest is just accidental complexity.
i sometimes wonder how all carpenters or music producers in the world would react if tomorrow a magic box arrives where they type-in/dictate what they want and voila... out comes a table/composition.. Will the worlds music producers and carpenters just "accept" this new magic genie? I would hope they wont.
Yet here we are as software engineers...ready to surrender our craft to mere "prompts" with reckless abandon.
I mean more or less to an extent if you expand by what you mean by "out comes". With Amazon, a carpenter can type in any furniture he wants and have it arrive in a matter of hours. Almost equally as magically as well as disruptive. People still use carpenters to make tables, but for the majority, the ease of same-day delivery and cheaper prices are worth it for a sacrifice in personalization and character.
>With Amazon, a carpenter can type in any furniture he wants and have it arrive in a matter of hours.
Sure but that doesnt require him to be a carpenter though right? Anybody could do that.
Its just as much as an ego trip...as the Principal Engineers that roam around here, after they survived three layoffs and outputted front end code for six years straight at a FAANG...
That's new to me, I didn't know she coined the term. She's the first true "software engineer" then, I suppose. I remember her from that historic photo where she's standing next to a pile of code printed out, a stack talled than her.
Not to be confused with Margaret Hamilton the actress who played the Wicked Witch of the West in the film The Wizard of Oz.
> Yet here we are as software engineers...ready to surrender our craft to mere "prompts" with reckless abandon.
It's worth remembering that, contrary to what HN acts like, software engineers are not uniformly accepting of LLMs. They are still quite controversial in the industry. Now management, on the other hand, is completely sold on the supposed benefits of LLMs. So many engineers feel forced to use them, but it's not enthusiasm on their part which drives them to that.
My experience has been the opposite, the HN crowd seems more sketical of vibe-coding and not reading a single line of code.
The twitter(x.com) and linkedin crowd has drank the cool-aid and are now swimming in it; Wild posts from FAANG engineers and managers saying they or their engineers dont read any of the code the LLM generates.
Have you seen how people use Fruity Loops nowadays? We’re already halfway there if you compare that to using an AKAI S9000 or MPC2000. And then go back to actually playing an instrument. It’s a gradient and we’re already pretty far along on it :)
It turns out that most people are fine with most of their things being mass-produced cheap stuff from IKEA, Amazon, Temu, etc. - if that means that the cost is significantly lower. The functional benefit of having a door is huge, while the aesthetic benefit of having a beautiful, hand-crafted door is much less important to most people.
Code is not like craftsmanship anyway. The end-user doesn't see any difference whether I use microservices or hexagonal design or react or htmx or how I structure the code or which language I use. All they see is the design and the stability. Since the average developer was terrible at design anyway, the average design output from LLMs is actually an improvement in most cases, even if it always has that generic AI generated look. Stability... was not good before AI either. We have become accustomed to SAP taking 10 seconds to load each component on the page, or Atlassian spewing error messages in your face when you click submit on a ticket. It seems that the aspects that are most valued by our end users are QA and UX, not the craftsmanship of our code.
Music is already mostly a solved problem unfortunately. Try giving opus a few midi files of your favorite song. It'll write something new in that style with a competence humans would need several years of experience for.
I see a bit of a split in people in creative roles of all kinds (and this is something I had noted before LLMs made the contrast more stark). some people get into it because they like the process of making the thing that they are making (in whole or in part). Others get into it because they have a thing that they want to exist in the world but no-one else has made it (I would also split this into people who just want the thing, or people who want to make something that crosses many domains and so want custom things that will fit into the whole). The former group tend to react with revulsion to automation, because it removes the part they like (though it may depend on what part is automated: most people will find some part of their work necessary but tedious). The latter are usually much more willing to embrace it, because it's the end result that they care about moreso than the process.
Making a quick prototype is probably the best way to find out if an idea is worth keeping. Vibe-coding it and then throwing it away seems like a good way to go?
The most difficult part of LLMs and agents seems to be - surprisingly - humans.
Specifically, humans and their dysfunctions. Their fragility. Their ego tied to stuff.
This is an excellent post/reflection on the topic of solo LLM use.
The only gap I see is that it assumes that one works on something alone and not in a team. Though, that might be a different essay in itself.
Edit: Turns out that footnote 3 actually mentions that already. Nice. Good post
On the ego-effect, I sometimes feel like cataloguing some PR-review quirks that are mostly caused by humans being humans:
1. Smaller PRs (that can be easily read) tend to receive more comments and nitpicks than a large change since it's easier for us to have all the PR context in our heads at the same time.
2. Most people are ego-tied to their code even if they pretend that they are
impartial
3. Social dynamics around merging and reviewing code between senior/junior and older/newer employees
Cognitive surrender is intellectually convenient. As humans, its instinctive for us to chose convenience: path of least resistance. The tyranny of convenience is what drives most economic activity today, especially the gig economy[1]. Unless there are structural frameworks put in place to resist convenience (exams in schools), nature will always entice to chose the path of least resistance
Maybe things will evolve into projects having a well-defined, hard boundary between agents generated code and human generated code. Where agents generated code is strictly separated from human engineered code and and can call only a strictly defined set of APIs designed by humans.
Think of platforms like Shopify. Shopify allows huge number of external users to write code customizing their web shops. But Shopify engineers do not need to review and understand the users' code, because there is a boundary which limits what the external code can do. The external code can be A Big Ball of Mud without making the whole platform a A Big Ball of Mud.
Well. One of the things that's critical in all that analogy is that the BigBallOfMud that one user creates on Shopify cannot impact the other users - that's why Shopify can keep the platform from becoming ABigBallofMud.
In order to be able to enforce this hard boundary, it means that one "agent generated code" in one place, not only "can call a strictly defined set of APIs" - but, more importantly, it implements an isolated piece of functionality. Think something like: I let the AI write the dashboards, so the dashboards may be slow, crappy, incorrect, whatever - but it doesn't impact the rest of the product. However, I can't let AI write "metrics collection" because that not only impacts dashboard, but potentially impacts core product efficiency too (memory consumption, cpu, IO, etc.)
We are leaning into a kind of federated model. The core applications and vibe-coded satellites. The core apps have stricter requirements and carefully considered API guardrails. Go ham on the AI apps. If you break something, the scope of destruction is much more limited.
That said, the core apps are still full of AI code. I don't see a reality where hand-written code can or will be curated and sequestered. It's unenforceable, too slow, and too expensive.
>Maybe things will evolve into projects having a well-defined, hard boundary between agents generated code and human generated code.
The only way things will evolve is towards maximizing AI generated code and minimizing human involvement either in writing or reviewing code because that is the most efficient mode for getting product to market. We'll just have to hope things don't break too much and when they do, we'll just have to accept it because no human will be capable of (or allowed to) fix it.
The tendency to over-engineer and over-build is one of the main challenges I'm seeing. When it comes for free, why not build a class hierarchy of 8 classes each with a 200 line unit test? Something I might have previously thought really hard to minimalise down to 30 lines is now bloated out to hundreds or even thousands.
Previously, pretty much everyone all agreed on YAGNI and KISS because code was so expensive to create and more expensive to maintain. Now the code generation is "free" - but at a design level, YAGNI and KISS do still matter. But they are enforced now only by sheer will power. You have to actively restrain the code from being created if you don't want it. Perfectly good code - I can't even fault it. I just know, if it doesn't need to exist yet, it's a mistake to create it earlier than needed.
That quotation is in fact from "Wind, Sand, and Stars", the memoir of the same author.
On the same theme:
> the supreme goal of all theory is to make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience
- Einstein
> I would have written a shorter letter, but did not have the time
- Pascal
> Simplicity is a great virtue but requires hard work to achieve and education to appreciate, and, to make matters worse, complexity sells better.
At the end of the day your work is your work. If you tell Claude or whatever to do a thing it will do that thing however you tell it to do the thing.
You can choose to ignore the fine details and just tell it what to build, or you can spend more time and effort and tell it how to do things, you can ask it to create a plan, review the plan and ask for changes etc, you can review the implementation and ask for changes there. You can simply tell it you don't like how a feature is implemented and ask it for suggestions on how to improve it, include what you don't like about it so it has direction.
It's all about the effort you put in. LLMs are able to write clean beautiful and efficient code, they just can't necessarily do exactly what you want on the first try without any direction. That's why you should figure out what you want, ask for what you want and make sure it's actually building what you want.
That's always been my concern - there's just so much garbage code out there and very few signals about which of it is well written and which is not ... so LLMs will just regurgitate the good enough code that it's been trained on.
Now would be the ideal time for everybody to give it negative feedback when bad code is returned but I'm afraid not enough people care and we'll just settle with what works just like we always have.
I don't think that's it. The context that LLMs work with is simply too small, and can't get bigger. Harnesses and the models themselves are bolting structure over it manually; real people are inventing design machines that are a lot more deterministic than they look, and LLMs are just filling them in.
The design machines aren't good enough yet; and their progress has little relationship to LLMs at all. Designing computer programs and the proper level of abstraction for the easiest maintenance, flexibility and debugging is simply an open problem. There's nothing to train LLMs on that is smaller than the size of an entire program, and its history of code changes.
Get a few of those, figure out a way to analyze and reduce the patterns and lessons in them, and train on that maybe. Until then, there's no way for an LLM to figure out what lacking a particular abstraction is going to end up costing.
I think this sorta cuts both ways too. Sometimes AI code has shown me that no, I in fact don't need that big class hierarchy and speculative plumbing that I thought I might need to get the job done.
Keeping the volume of code down is definitely a challenge though.
This is so true. I've been seeing LLM generated code that does way too much, yet still cant accomplish the stated goals. "maximally flexible" plugin systems, that are too complicated to reasonably write a plugin for quickly, when one could have just built in the needed functionality in a few hundred lines.
When endless bootstrapping and framing is free, you end up generating the most beautiful scaffold while ignoring the building behind the scaffold.
What an amazing article! It's really hard to find the balance between "ship faster, fix later because AI will help" and "efficient thoughtful engineering" these days. The world is kind of pushing everybody to work faster because "with AI you're more efficient" and it's also pushing to surrender and just accept it from time to time. Maybe that's occasional thoughtful surrender mentioned in the article, or it's just lack of energy to resist to it at that point.
The article is inspiring though, I have more motivation to work on my thinking process and my engineering setup (and vibe limit haha) to be more thoughtful, thank you.
Maybe it's because I'm not working in the workforce (grad student), but I find my best workflow is to fight and argue with my agents throughout the entire process. Often times, since the agent is such a sycophant, I end up arguing with myself...I actually find this useful because it gets me to think about the other side of a problem and poke holes in it. This is not fast, but it is effective.
I simply don't care. As a student I thought that the mythical production code is perfect and error-free, but in reality 99% of systems are a hot potato game sprinkled with politics where you don't want the grenade to explode in your hands.
I mean, if the management doesn't see value in decent engineering, then who am I to butt myself into the conversation?
I have been thinking about this for a while. I'm not a Luddite, I can see the amazing value that modern AI brings, and how useful it can be.
At the same time I can't help but wonder that this is almost too much for us to cope, the LLMs are _too_ easy to use. I think we're very good at pretending there isn't a problem here but there is definitely going to be some cognitive decline happening.
We want to pretend like reading over the output of an LLM is nearly the same as us making those changes ourselves but I think there's a lot of value in those boring and manual steps. And most people (including me) will be too lazy to not eventually offload most things to the LLMs.
In a way this is similar to how watching a lecture tends to give one an "impression" of understanding that doesn't correlate with reality. We need hands-on work and some course work to get our monkey brains engaged.
I can only imagine what effects this is going to have in education.
We've surrendered many uses of our cognitive power throughout the centuries while replacing by others.
Our cognition didn't evolve to produce arguments for online discussions, but for making crude tools and hunting. As we stopped applying our intelligence towards these uses as a result of our tools, we simply shifted it to new problems that were unthinkable previously.
A caveman probably wouldn't care too much about algebra, a topic most children learn today.
We should stop thinking about the problems AI trivialized and, under the assumption that AI won't fully replace human capability (a key assumption), start thinking about the problems that were previously out of reach. Education would then start focusing on that.
This is true, but the nature of declining cognition is to not even notice what used to be possible.
Taking clothing for example, there's more available now than ever. It's disposable at great cost to the environment and the workers that produce it, but very little to you.
Yet, I bet you cannot evaluate garments, construction, or fabric quality. You may not know how to wash woolens and linens. You may have only worn smelly plastic workout clothing and plastic suits for formal events.
You may never have worn a piece of clothing that actually fit your body.
You may not realize that you can alter any clothing, that most clothing was intended to be altered once upon a time. Or that older garments were both more expensive and higher quality, because the expectation was the most important items would need to last for a decade.
The Luddite's central objections about automation was about declining working conditions (objecting to child labour, for instance), bad pay, and declining quality. They used the 'obnoxious' machines they broke. They saw what was being lost and demanded better (without demanding no machines at all).
Later, the rise of unionization brought about many of the changes they'd asked for decades before.
I hope this current technologic movement doesn't end where fast fashion has, with influencers buying a new outfit [repo] every day for a year. A vortex of undifferentiated slop for its own sake.
Right, but algebra is free. You can do it with a pencil and paper
AI is not free. Even "free" and open source models require complex silicon to operate. The silicon and copper is assembled in complex fabrication plants. Those plants are very close to geopolitical belligerents
What I resist is the dependence. I'm not willing to trade short term productivity, for long term dependence
I think there's a paradox. Is it only "cognitive surrender" if you already know how to code? Not knowing all the details seems more like the default human condition, in the sense that most people don't know exactly where their water comes from or where their waste goes exactly. "Water comes from the tap, food comes from the store".
Most of the non-technical people I see using LLMs to create software are not engaging in "cognitive surrender" as far as I can tell.
They're evaluating what they can see, i.e, judging results. They build things for their own use. They test them, iterate on them to meet their needs and try to achieve their vision. This is roughly the same process they followed when they used to outsource to an agency except the loop is much tighter.
They didn't read the code before and they don't now. However, asking questions and improving understanding of the system and architecture is actually faster and easier for them now than it used to be.
It seems to work well, mainly hurts when the problems are invisible, so security, backups, breaking migrations, perf, maintenance over time (language / runtime updates, dep updates, lib updates, server updates etc). All the stuff that hurts if you have never been responsible for the uptime of something before. Not knowing that software is never "done". But junior devs trip on those exact same things.
Tooling and processes that make that stuff more visible would close a lot of the gaps.
> Is it only "cognitive surrender" if you already know how to code?
Is the article claiming that?
> Not knowing all the details seems more like the default human condition, in the sense that most people don't know exactly where their water comes from or where their waste goes exactly. "Water comes from the tap, food comes from the store".
Sure. The ability to specialize, and on the flip-side not have to care about a lot of things, remains one of the biggest driving forces of human civilization. But in this article, I don't think the person worried about cognitive surrender is the "most people" in relation to food or water. They're the farmer or the water infrastructure engineer! They absolutely for sure do need to know those things. And all around us we see a lure of systems that invite them to surrender.
> Most of the non-technical people I see using LLMs to create software are not engaging in "cognitive surrender" as far as I can tell.
Maybe not. But the might be cognitively surrendering when they use the LLM to summarize the day's news or write a message to a friend, or even plan dinner for the week. I'd say they are. I agree that it's not clear-cut.
> They're evaluating what they can see, i.e, judging results. They build things for their own use. They test them, iterate on them to meet their needs and try to achieve their vision.
No. They have things built, then they test them. This is different. It might be OK, but it's not the same as building.
> This is roughly the same process they followed when they used to outsource to an agency except the loop is much tighter.
Yes. But they wouldn't usually outsource summarizing a letter from grandma, or the political agenda of a candidate/party in an upcoming election. That's surrender.
> It seems to work well, mainly hurts when the problems are invisible, so security, backups, breaking migrations, perf, maintenance over time (language / runtime updates, dep updates, lib updates, server updates etc).
… or ones own abilities deteriorate.
> Tooling and processes that make that stuff more visible would close a lot of the gaps.
> I don't think the person worried about cognitive surrender is the "most people" in relation to food or water. They're the farmer or the water infrastructure engineer!
Fair. Maybe I "cognitively surrendered" a long time ago just as a consequence of working in large teams :(
I hit the breaking point of not being able to read all the code well before LLMs came along. Also people would leave and the documentation for their subsystems would be wrong or incomplete, so there were always mysteries to chase down (during an incident, naturally) even in code written by hand.
Gets to the point, you don't exactly read the MRs, you're mainly checking their approach against what you talked about earlier and making sure it doesn't look way more gnarly than it's supposed to.
Working with agents doesn't actually feel that different.
> No. They have things built, then they test them. This is different. It might be OK, but it's not the same as building.
I don't personally see super hard lines between using a compiler, a 4GL, a sql query, stitching together 10 libraries and a framework I didn't write, using an LLM or even directing a team.
> But then we do say things like "Nolan made the film" even though he didn't hold a camera or build the sets.
I do say that, too. But I'm not in the film industry. My understanding is that those who are, don't use words as crude as "made" to describe a role in the making of a film. I believe they say that he wrote, directed or produced a film.
We are not talking about some other field of expertise here, where we are outsiders – the production of food, the supply of water, or the making of movies – we're talking about our own field of expertise.
Somewhat off topic now, but Nolan is fairly famous for being intricately involved with every aspect of the films he directs. He's constantly behind a camera checking out what shots are going to look like, pushing the limits of where you can put an IMAX camera, and coordinating with set builders to do ridiculous stuff like building a reproduction harbour jetty.
> My understanding is that those who are, don’t use words as crude as “made” to describe a role in the making of a film.
Oh, no, film insiders use ‘made’ constantly, or even cruder - attribute credit without using a verb at all. Here’s a VFX person saying “Nolan’s CGI”: https://youtu.be/AFLcP-t0-kI (I’m ex-film industry, and still film industry tangential, and found this clip being discussed on a film industry mailing list.)
Film people are far more likely to pay attention to who the director is, and it’s very common for film people to talk about directors ‘making’ the film or otherwise casually assign ownership. I think this is especially true for famous directors and good films, and maybe a bit less true for unknown directors and mediocre films.
> However, asking questions and improving understanding of the system and architecture is actually faster and easier for them now than it used to be.
One of the things I've built in my current role is tooling to help understand current architecture, and to resolve the differences between things as-documented, as-implemented, and as-actually-behaving. It's used for a few things, including updating documentation, but it's been most impactful when onboarding new engineers. They can simply ask questions about how things work and why, and get explanations and deep dives wherever they need it.
I think one of the potentially advantageous things about AI is that it acts as a force multiplier for people who already have expertise in a given discipline, while simultaneously raising the floor for everyone else.
People looking for self-contained, spot solutions to localized problems will have much more capacity to solve them locally, while people with in-depth expertise will be able to build systems that are even more reliable, performant, and scalable than what they were building before.
So we'll get the benefits of disintermediation for the basics, and much greater capacity to deal with effectively with complexity for the not-so-basics.
> It seems to work well, mainly hurts when the problems are invisible, so security, backups, breaking migrations, perf, maintenance over time (language / runtime updates, dep updates, server updates etc). All the stuff that hurts if you have never been responsible for the uptime of something before. Not knowing that software is never "done". But junior devs trip on those exact same things.
If we continue from your "water comes from the tap" example this situation feels like someone who has no experience of plumbing deciding they're going to replace their bathroom without bothering to find out how any of it works. They'll possibly end up with a bathroom that superficially works but they're also liable to find problems down the line with leaks, things not draining properly, or water pressure issues.
Some of the people writing software purely via LLMs are even turning this into a business, which I guess would be analogous to deciding you're going to run sewage treatment for a small city without understanding anything about how that works. It'll mainly hurt when the problems are invisible, so food poisoning, or inlets backing up and overflowing into a nearby river.
There were terrible software developers before LLMs. Honestly for a lot of people AI generated slop is a massive improvement over what they used to ship.
I agree it's pretty bloody cheeky to run an agency like that though. The only ways out I can see are licensing or real liability.
But let he who has never done dodgy home improvements cast the first stone.
That analogy breaks down immediately because it's missing the critical novel component: the AI.
In the bathroom remodel analogy, the AI is the plumber that you hire to do the work for you. You tell it what you want, it figures out the details necessary to get it to you. If what you want isn't possible, or requires large changes, a good plumber will raise the issue to you and explain the constraints they are working under, ask for guidance, and offer recommendations.
Sometimes the AI does things in a weird or inefficient way, or neglects details that become important later, or misunderstands something important about the task resulting in an outcome other than what you wanted, but that's also the same as the plumber. This is why people usually want to be at home and keep an eye on the people working on their improvement projects or repairs, because they want to make sure that the guy they hired is replacing the broken faucet, not the functional but odd-looking one.
There is still value in knowing the underlying systems well enough to present a good design vision to the plumber up-front. It's still possible to have a competent executor produce a bad idea, which is how we get stuff like Groverhaus, the Doom Bathroom, and McMansion Hell, which are examples of "slop architecture" in exactly the same way as we are seeing "slop software".
101 comments
[ 0.21 ms ] story [ 1590 ms ] threadAutomatic control is good for standard case, and the expected errors. For weird cases, for now humans are better to detect them and press the big red button.
The best known example is the Japanese method of "Pointing and Calling/Naming" in their Train safety protocols but has been adapted to other industries too - https://en.wikipedia.org/wiki/Pointing_and_calling
These fall under "Behaviour Based Safety" (BBS) training methodologies and use techniques like "Check-backs", "Call-outs", "STAR" etc.
But.. we do monitor whether our prayers are effective .. right? Is it a smitable offense?
After that disappointment, I was also "curious" if the author was going to talk about "curiosity", the magic virtue that we hackers believe, lifts design into the divine
This has become common now, and I've seen it ruin some larger projects until nobody understands the codebase full of verbose nonsensical code and comments. It becomes impossible to further develop the project without AI. Contributors not using AI for any reason are all leaving because no one enjoys working with mindless robots (and humans) as colleagues. It's demeaning and devaluing their labor and humanity. They're not getting "left behind", they are leaving the madness of crowds behind.
As this technology permeates society, the challenge to maintain control, coherence, and quality will become greater and ever more serious - for example, AI in healthcare, military, management and governance.
But before LLMs, it didn’t matter because on a good team you’d have a balance where maybe some PRs added complexity and some simplified it. Sometimes people would include a small related cleanup or refactor in a PR for code around what they were doing if they noticed complexity. That has become far less common, I think.
I know it’s not your point, but a 1% decrease 100 times is 0.99^100 = 0.366 which is a 63.4% decrease in understandability. Just throwing that out there
Enterprise software has always been crap.
If we don't value that though--if we don't value understanding the system and just thinking through problems as a group--then let's stop pretending and just not do code reviews. Probably nobody cares and just types "LGTM" after having their own LLM do a spot check to cover their ass.
It's really easy to push developers to stop caring. If management says "go fast fast fast, use those LLMs!", then developers can't keep up, get frustrated, stop understanding the system, and at that point they're just there to collect a paycheck before society collapses. Or, if management is making dumb decisions all the time, same thing.
Hopefully as implementing ideas takes less and less time and resources, smart people who actually care will start building things with other smart people who care.
Karpathy had the same trajectory. Primeagean had the same trajectory. I think in the end everyone except for Jonathan Blow will have the same trajectory
However it takes more mental energy and people don't like that
I am not saying he writes all code by hand, of course. Just that he is furthest in figuring out some sort of mid point between full comprehension and cognitive surrender.
I have missed the more recent content, so perhaps I have missed something, but this was my read last I watched.
There’s one thing that’s bothering me though — all of these have us consuming what LLMs produce. I learned and internalized how things worked by building systems. At least within the constraints of the abstractions I was using. This makes me think of TikTok and shorts. Everybody is just watching. Maybe that’s not charitable though. Maybe the abstraction has now changed from an “os/gui/event loop” to “llm” and the rest is just accidental complexity.
Its just as much as an ego trip...as the Principal Engineers that roam around here, after they survived three layoffs and outputted front end code for six years straight at a FAANG...
Prove me wrong...
But oh well, who was she, just the engineer who wrote the software for the Apollo 11. Probably she had no idea what she was talking about.
/s
Not to be confused with Margaret Hamilton the actress who played the Wicked Witch of the West in the film The Wizard of Oz.
It's worth remembering that, contrary to what HN acts like, software engineers are not uniformly accepting of LLMs. They are still quite controversial in the industry. Now management, on the other hand, is completely sold on the supposed benefits of LLMs. So many engineers feel forced to use them, but it's not enthusiasm on their part which drives them to that.
The twitter(x.com) and linkedin crowd has drank the cool-aid and are now swimming in it; Wild posts from FAANG engineers and managers saying they or their engineers dont read any of the code the LLM generates.
And if you look a the dates, the rate at which they are doing it is actually increasing...
Code is not like craftsmanship anyway. The end-user doesn't see any difference whether I use microservices or hexagonal design or react or htmx or how I structure the code or which language I use. All they see is the design and the stability. Since the average developer was terrible at design anyway, the average design output from LLMs is actually an improvement in most cases, even if it always has that generic AI generated look. Stability... was not good before AI either. We have become accustomed to SAP taking 10 seconds to load each component on the page, or Atlassian spewing error messages in your face when you click submit on a ticket. It seems that the aspects that are most valued by our end users are QA and UX, not the craftsmanship of our code.
It' like saying compassion is a solved problem: just tell your problems to the LLM and have it gush out sympathy.
Specifically, humans and their dysfunctions. Their fragility. Their ego tied to stuff.
This is an excellent post/reflection on the topic of solo LLM use. The only gap I see is that it assumes that one works on something alone and not in a team. Though, that might be a different essay in itself.
Edit: Turns out that footnote 3 actually mentions that already. Nice. Good post
1. Smaller PRs (that can be easily read) tend to receive more comments and nitpicks than a large change since it's easier for us to have all the PR context in our heads at the same time.
2. Most people are ego-tied to their code even if they pretend that they are impartial
3. Social dynamics around merging and reviewing code between senior/junior and older/newer employees
4. probably many more
[1] https://www.nytimes.com/2018/02/16/opinion/sunday/tyranny-co...
Think of platforms like Shopify. Shopify allows huge number of external users to write code customizing their web shops. But Shopify engineers do not need to review and understand the users' code, because there is a boundary which limits what the external code can do. The external code can be A Big Ball of Mud without making the whole platform a A Big Ball of Mud.
In order to be able to enforce this hard boundary, it means that one "agent generated code" in one place, not only "can call a strictly defined set of APIs" - but, more importantly, it implements an isolated piece of functionality. Think something like: I let the AI write the dashboards, so the dashboards may be slow, crappy, incorrect, whatever - but it doesn't impact the rest of the product. However, I can't let AI write "metrics collection" because that not only impacts dashboard, but potentially impacts core product efficiency too (memory consumption, cpu, IO, etc.)
That said, the core apps are still full of AI code. I don't see a reality where hand-written code can or will be curated and sequestered. It's unenforceable, too slow, and too expensive.
The only way things will evolve is towards maximizing AI generated code and minimizing human involvement either in writing or reviewing code because that is the most efficient mode for getting product to market. We'll just have to hope things don't break too much and when they do, we'll just have to accept it because no human will be capable of (or allowed to) fix it.
Previously, pretty much everyone all agreed on YAGNI and KISS because code was so expensive to create and more expensive to maintain. Now the code generation is "free" - but at a design level, YAGNI and KISS do still matter. But they are enforced now only by sheer will power. You have to actively restrain the code from being created if you don't want it. Perfectly good code - I can't even fault it. I just know, if it doesn't need to exist yet, it's a mistake to create it earlier than needed.
> La perfection est atteinte non pas lorsqu'il n'y a plus rien à ajouter, mais lorsqu'il n'y a plus rien à retirer.
On the same theme:
> the supreme goal of all theory is to make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience
- Einstein
> I would have written a shorter letter, but did not have the time
- Pascal
> Simplicity is a great virtue but requires hard work to achieve and education to appreciate, and, to make matters worse, complexity sells better.
- Dijkstra.
You can choose to ignore the fine details and just tell it what to build, or you can spend more time and effort and tell it how to do things, you can ask it to create a plan, review the plan and ask for changes etc, you can review the implementation and ask for changes there. You can simply tell it you don't like how a feature is implemented and ask it for suggestions on how to improve it, include what you don't like about it so it has direction.
It's all about the effort you put in. LLMs are able to write clean beautiful and efficient code, they just can't necessarily do exactly what you want on the first try without any direction. That's why you should figure out what you want, ask for what you want and make sure it's actually building what you want.
Now would be the ideal time for everybody to give it negative feedback when bad code is returned but I'm afraid not enough people care and we'll just settle with what works just like we always have.
The design machines aren't good enough yet; and their progress has little relationship to LLMs at all. Designing computer programs and the proper level of abstraction for the easiest maintenance, flexibility and debugging is simply an open problem. There's nothing to train LLMs on that is smaller than the size of an entire program, and its history of code changes.
Get a few of those, figure out a way to analyze and reduce the patterns and lessons in them, and train on that maybe. Until then, there's no way for an LLM to figure out what lacking a particular abstraction is going to end up costing.
Keeping the volume of code down is definitely a challenge though.
When endless bootstrapping and framing is free, you end up generating the most beautiful scaffold while ignoring the building behind the scaffold.
The economics of the whole affair unfortunately point squarely in the opposite direction.
I am losing more of what little focus I've had every day.
The article is inspiring though, I have more motivation to work on my thinking process and my engineering setup (and vibe limit haha) to be more thoughtful, thank you.
I mean, if the management doesn't see value in decent engineering, then who am I to butt myself into the conversation?
https://epiq.sh/blog/cost-of-cognitive-debt.html
It'll take a particularly dumb and lazy AI to replace me.
At the same time I can't help but wonder that this is almost too much for us to cope, the LLMs are _too_ easy to use. I think we're very good at pretending there isn't a problem here but there is definitely going to be some cognitive decline happening.
We want to pretend like reading over the output of an LLM is nearly the same as us making those changes ourselves but I think there's a lot of value in those boring and manual steps. And most people (including me) will be too lazy to not eventually offload most things to the LLMs.
In a way this is similar to how watching a lecture tends to give one an "impression" of understanding that doesn't correlate with reality. We need hands-on work and some course work to get our monkey brains engaged.
I can only imagine what effects this is going to have in education.
Our cognition didn't evolve to produce arguments for online discussions, but for making crude tools and hunting. As we stopped applying our intelligence towards these uses as a result of our tools, we simply shifted it to new problems that were unthinkable previously.
A caveman probably wouldn't care too much about algebra, a topic most children learn today.
We should stop thinking about the problems AI trivialized and, under the assumption that AI won't fully replace human capability (a key assumption), start thinking about the problems that were previously out of reach. Education would then start focusing on that.
Taking clothing for example, there's more available now than ever. It's disposable at great cost to the environment and the workers that produce it, but very little to you.
Yet, I bet you cannot evaluate garments, construction, or fabric quality. You may not know how to wash woolens and linens. You may have only worn smelly plastic workout clothing and plastic suits for formal events.
You may never have worn a piece of clothing that actually fit your body.
You may not realize that you can alter any clothing, that most clothing was intended to be altered once upon a time. Or that older garments were both more expensive and higher quality, because the expectation was the most important items would need to last for a decade.
The Luddite's central objections about automation was about declining working conditions (objecting to child labour, for instance), bad pay, and declining quality. They used the 'obnoxious' machines they broke. They saw what was being lost and demanded better (without demanding no machines at all).
Later, the rise of unionization brought about many of the changes they'd asked for decades before.
I hope this current technologic movement doesn't end where fast fashion has, with influencers buying a new outfit [repo] every day for a year. A vortex of undifferentiated slop for its own sake.
I can also recommend the book "Blood in the Machine" by Brian Merchant [0][1] which is precisely about this.
[0] https://www.hachettebookgroup.com/titles/brian-merchant/bloo...
[1] https://www.bloodinthemachine.com/about
AI is not free. Even "free" and open source models require complex silicon to operate. The silicon and copper is assembled in complex fabrication plants. Those plants are very close to geopolitical belligerents
What I resist is the dependence. I'm not willing to trade short term productivity, for long term dependence
Most of the non-technical people I see using LLMs to create software are not engaging in "cognitive surrender" as far as I can tell.
They're evaluating what they can see, i.e, judging results. They build things for their own use. They test them, iterate on them to meet their needs and try to achieve their vision. This is roughly the same process they followed when they used to outsource to an agency except the loop is much tighter.
They didn't read the code before and they don't now. However, asking questions and improving understanding of the system and architecture is actually faster and easier for them now than it used to be.
It seems to work well, mainly hurts when the problems are invisible, so security, backups, breaking migrations, perf, maintenance over time (language / runtime updates, dep updates, lib updates, server updates etc). All the stuff that hurts if you have never been responsible for the uptime of something before. Not knowing that software is never "done". But junior devs trip on those exact same things.
Tooling and processes that make that stuff more visible would close a lot of the gaps.
Is the article claiming that?
> Not knowing all the details seems more like the default human condition, in the sense that most people don't know exactly where their water comes from or where their waste goes exactly. "Water comes from the tap, food comes from the store".
Sure. The ability to specialize, and on the flip-side not have to care about a lot of things, remains one of the biggest driving forces of human civilization. But in this article, I don't think the person worried about cognitive surrender is the "most people" in relation to food or water. They're the farmer or the water infrastructure engineer! They absolutely for sure do need to know those things. And all around us we see a lure of systems that invite them to surrender.
> Most of the non-technical people I see using LLMs to create software are not engaging in "cognitive surrender" as far as I can tell.
Maybe not. But the might be cognitively surrendering when they use the LLM to summarize the day's news or write a message to a friend, or even plan dinner for the week. I'd say they are. I agree that it's not clear-cut.
> They're evaluating what they can see, i.e, judging results. They build things for their own use. They test them, iterate on them to meet their needs and try to achieve their vision.
No. They have things built, then they test them. This is different. It might be OK, but it's not the same as building.
> This is roughly the same process they followed when they used to outsource to an agency except the loop is much tighter.
Yes. But they wouldn't usually outsource summarizing a letter from grandma, or the political agenda of a candidate/party in an upcoming election. That's surrender.
> It seems to work well, mainly hurts when the problems are invisible, so security, backups, breaking migrations, perf, maintenance over time (language / runtime updates, dep updates, lib updates, server updates etc).
… or ones own abilities deteriorate.
> Tooling and processes that make that stuff more visible would close a lot of the gaps.
Not the problem of one's own skills though.
Fair. Maybe I "cognitively surrendered" a long time ago just as a consequence of working in large teams :(
I hit the breaking point of not being able to read all the code well before LLMs came along. Also people would leave and the documentation for their subsystems would be wrong or incomplete, so there were always mysteries to chase down (during an incident, naturally) even in code written by hand.
Gets to the point, you don't exactly read the MRs, you're mainly checking their approach against what you talked about earlier and making sure it doesn't look way more gnarly than it's supposed to.
Working with agents doesn't actually feel that different.
> No. They have things built, then they test them. This is different. It might be OK, but it's not the same as building.
I don't personally see super hard lines between using a compiler, a 4GL, a sql query, stitching together 10 libraries and a framework I didn't write, using an LLM or even directing a team.
I do say that, too. But I'm not in the film industry. My understanding is that those who are, don't use words as crude as "made" to describe a role in the making of a film. I believe they say that he wrote, directed or produced a film.
We are not talking about some other field of expertise here, where we are outsiders – the production of food, the supply of water, or the making of movies – we're talking about our own field of expertise.
Oh, no, film insiders use ‘made’ constantly, or even cruder - attribute credit without using a verb at all. Here’s a VFX person saying “Nolan’s CGI”: https://youtu.be/AFLcP-t0-kI (I’m ex-film industry, and still film industry tangential, and found this clip being discussed on a film industry mailing list.)
Film people are far more likely to pay attention to who the director is, and it’s very common for film people to talk about directors ‘making’ the film or otherwise casually assign ownership. I think this is especially true for famous directors and good films, and maybe a bit less true for unknown directors and mediocre films.
One of the things I've built in my current role is tooling to help understand current architecture, and to resolve the differences between things as-documented, as-implemented, and as-actually-behaving. It's used for a few things, including updating documentation, but it's been most impactful when onboarding new engineers. They can simply ask questions about how things work and why, and get explanations and deep dives wherever they need it.
People looking for self-contained, spot solutions to localized problems will have much more capacity to solve them locally, while people with in-depth expertise will be able to build systems that are even more reliable, performant, and scalable than what they were building before.
So we'll get the benefits of disintermediation for the basics, and much greater capacity to deal with effectively with complexity for the not-so-basics.
They didn't read code before, they just wrote it? That makes no sense.
If we continue from your "water comes from the tap" example this situation feels like someone who has no experience of plumbing deciding they're going to replace their bathroom without bothering to find out how any of it works. They'll possibly end up with a bathroom that superficially works but they're also liable to find problems down the line with leaks, things not draining properly, or water pressure issues.
Some of the people writing software purely via LLMs are even turning this into a business, which I guess would be analogous to deciding you're going to run sewage treatment for a small city without understanding anything about how that works. It'll mainly hurt when the problems are invisible, so food poisoning, or inlets backing up and overflowing into a nearby river.
I agree it's pretty bloody cheeky to run an agency like that though. The only ways out I can see are licensing or real liability.
But let he who has never done dodgy home improvements cast the first stone.
In the bathroom remodel analogy, the AI is the plumber that you hire to do the work for you. You tell it what you want, it figures out the details necessary to get it to you. If what you want isn't possible, or requires large changes, a good plumber will raise the issue to you and explain the constraints they are working under, ask for guidance, and offer recommendations.
Sometimes the AI does things in a weird or inefficient way, or neglects details that become important later, or misunderstands something important about the task resulting in an outcome other than what you wanted, but that's also the same as the plumber. This is why people usually want to be at home and keep an eye on the people working on their improvement projects or repairs, because they want to make sure that the guy they hired is replacing the broken faucet, not the functional but odd-looking one.
There is still value in knowing the underlying systems well enough to present a good design vision to the plumber up-front. It's still possible to have a competent executor produce a bad idea, which is how we get stuff like Groverhaus, the Doom Bathroom, and McMansion Hell, which are examples of "slop architecture" in exactly the same way as we are seeing "slop software".