Clearly, there is a thing missing here: Regulations. If you have strong regulations on how you can make money, you cannot sustainably have biz antagonize user. So in that case biz just becomes a filter for users that actually are willing (and able) to fund your software. That's a good thing.
Obviously, our regulations aren't perfect or even good enough yet. See DRM. See spyware TVs. See "who actually gets to control your device?". But still...
I've worked at some of the "top tier" finance firms over the years.
It is absolutely astounding how much of them run on code that is:
- very reliable aka it almost never breaks/fails
- written in ways that makes you wonder what series of events led to such awful code
For example:
- A deployment system that used python to read and respond to raw HTTP requests. If you triggered a deployment, you had to leave the webpage open as the deployment code was in the HTTP serving code
- A workflow manager that had <1000 lines of code but commits from 38 different people as the ownership always got passed to whoever the newest, most junior person on the team was
- Python code written in Java OOP style where every function call had to be traced up and down through four levels of abstraction
I mention this only b/c the "LLMs write shitty code" isn't quite the insult/blocker that people think it is. Humans write TONS of awful but working code too.
This is getting to be possibly the most irritating thing I've seen on Hacker News since registering here. Every thread about a limitation of LLMs being immediately rebuked with "humans do that too."
It's a continuous object lesson in missing the point. A similar thing happened a few hours ago when an article was posted about a researcher who posted a fake paper about a fake disease to a pre-print server that LLMs picked up via RAG, telling people with vague symptoms that they had this non-existent disease. Lo and behold, commenters go in immediately saying "I'd be fooled too because I trust pre-print medical research." Except the article itself was intentionally ridiculous, opening by telling you it was fake, using obviously fake names, fictional characters from popular television. The only reason it fooled humans on Hacker News is because they don't bother reading the articles and respond only to headlines.
It's just like your code examples. Humans fail because we're lazy. Just like all animals, we have a strong instinct to preserve energy and expend effort only when provoked by fear, desire, or external coercion. The easiest possible code to write that seems to work on a single happy path using stupid workarounds is deemed good enough and allowed through. If your true purpose on a web discussion board is to bloviate and prove how smart you are rather than learn anything, why bother actually reading anything? The faster you comment, the better chance you have of getting noticed and upvoted anyway.
Humans are not actually stupid. We can write great code. We can read an obviously fake paper and understand that it's fake. We know how hierarchy of evidence and trust works if we bother to try. We're just incredibly lazy. LLMs are not lazy. Unlike animals, they have no idea how much energy they're using and don't care. Their human slaves will move heaven and earth and reallocate entire sectors of their national economies and land use policies to feed them as much as they will ever need. LLMs, however, do have far more concrete cognitive limitations brought about by the way they are trained without any grounding in hierarchy of evidence or the factual accuracy of the text the ingest. We've erected quite a bit of ingenious scaffolding with various forms of augmented context, input pre-processing, post-training model fine tuning, and whatever the heck else these brilliant human engineers are doing to create the latest generation of state of the art agents, but the models underneath still have this limitation.
Do we need more? Can the scaffolding alone compensate sufficiently to produce true genius at the level of a human who is actually motivated and trying? I have no idea. Maybe, maybe not, but it's really irritating that we can't even discuss the topic because it immediately drops into the tarpit of "well, you too." It's the discourse of toddlers. Can't we do better than this?
Finance is like an oil well. You can do just about anything technology-wise and as long as it more or less pulls the oil from the ground, the money just keeps coming. So good code is not necessary. Some may even say that terrible code that needs to be replaced every year is a feature in terms of promotion possibilities.
Oh noe, noe no.. you want to crowdsource debugging.. describe the error and your expectations, then build software by machine learning while screwing up.
Richard Brand says the most important thing to grow a successful business is to put your employees above all else. Being the place where everyone wants to work and cares about their job is the way to get the most loyal customers. Having the most loyal customers is how you make the most money over the long term.
Survivorship bias exists, but look at all the Virgin brands and at places like Google. So for a moment let’s posit he’s correct.
So, then, the problem would seem not to be capitalism generally. It would be the sort of short-term quarterly goals capitalism we see so often in recent years.
> But when you run your code in production, the KISS mantra takes on a new dimension. It’s not just about code anymore; it’s about reducing the moving parts and understanding their failure modes.
This sentence, itself, takes on new meaning in the age of agentic coding. "I'm fine with treating this new feature as greenfield even if it reimplements existing code, because the LLM will handle ensuring the new code meets biz and user expectations" is fine in isolation... but it may mean that the code does not benefit from shared patterns for observability, traffic shaping, debugging, and more.
And if the agent inlines code that itself had a bug, that later proves to be a root cause, the amount of code that needs to be found and fixed in an outage situation is not only larger but more inscrutable.
Using the OOP's terminology, where biz > user > ops > dev is ideal, this is a dev > ops style failure that goes far beyond "runs on my machine" towards a notion of "is only maintainable in isolation."
Luckily, we have 1M context windows now! We can choose to say: "Meticulously explore the full codebase for ways we might be able to refactor this prototype to reuse existing functionality, patterns, and services, with an eye towards maintainability by other teams." But that requires discipline, foresight, and clock-time.
26 comments
[ 3.0 ms ] story [ 52.2 ms ] threadObviously, our regulations aren't perfect or even good enough yet. See DRM. See spyware TVs. See "who actually gets to control your device?". But still...
I am afraid that without a major crash or revolution of some sort, user won't matter next to a sufficiently big biz. But time will tell.
It is absolutely astounding how much of them run on code that is:
- very reliable aka it almost never breaks/fails
- written in ways that makes you wonder what series of events led to such awful code
For example:
- A deployment system that used python to read and respond to raw HTTP requests. If you triggered a deployment, you had to leave the webpage open as the deployment code was in the HTTP serving code
- A workflow manager that had <1000 lines of code but commits from 38 different people as the ownership always got passed to whoever the newest, most junior person on the team was
- Python code written in Java OOP style where every function call had to be traced up and down through four levels of abstraction
I mention this only b/c the "LLMs write shitty code" isn't quite the insult/blocker that people think it is. Humans write TONS of awful but working code too.
It's a continuous object lesson in missing the point. A similar thing happened a few hours ago when an article was posted about a researcher who posted a fake paper about a fake disease to a pre-print server that LLMs picked up via RAG, telling people with vague symptoms that they had this non-existent disease. Lo and behold, commenters go in immediately saying "I'd be fooled too because I trust pre-print medical research." Except the article itself was intentionally ridiculous, opening by telling you it was fake, using obviously fake names, fictional characters from popular television. The only reason it fooled humans on Hacker News is because they don't bother reading the articles and respond only to headlines.
It's just like your code examples. Humans fail because we're lazy. Just like all animals, we have a strong instinct to preserve energy and expend effort only when provoked by fear, desire, or external coercion. The easiest possible code to write that seems to work on a single happy path using stupid workarounds is deemed good enough and allowed through. If your true purpose on a web discussion board is to bloviate and prove how smart you are rather than learn anything, why bother actually reading anything? The faster you comment, the better chance you have of getting noticed and upvoted anyway.
Humans are not actually stupid. We can write great code. We can read an obviously fake paper and understand that it's fake. We know how hierarchy of evidence and trust works if we bother to try. We're just incredibly lazy. LLMs are not lazy. Unlike animals, they have no idea how much energy they're using and don't care. Their human slaves will move heaven and earth and reallocate entire sectors of their national economies and land use policies to feed them as much as they will ever need. LLMs, however, do have far more concrete cognitive limitations brought about by the way they are trained without any grounding in hierarchy of evidence or the factual accuracy of the text the ingest. We've erected quite a bit of ingenious scaffolding with various forms of augmented context, input pre-processing, post-training model fine tuning, and whatever the heck else these brilliant human engineers are doing to create the latest generation of state of the art agents, but the models underneath still have this limitation.
Do we need more? Can the scaffolding alone compensate sufficiently to produce true genius at the level of a human who is actually motivated and trying? I have no idea. Maybe, maybe not, but it's really irritating that we can't even discuss the topic because it immediately drops into the tarpit of "well, you too." It's the discourse of toddlers. Can't we do better than this?
LLMs regurgitate shitty code. They learned it entirely from people.
To be fair, the standard library `unittest` and `logging`, along with the historic `distutils`/Setuptools stack, are hardly any better.
Survivorship bias exists, but look at all the Virgin brands and at places like Google. So for a moment let’s posit he’s correct.
So, then, the problem would seem not to be capitalism generally. It would be the sort of short-term quarterly goals capitalism we see so often in recent years.
This sentence, itself, takes on new meaning in the age of agentic coding. "I'm fine with treating this new feature as greenfield even if it reimplements existing code, because the LLM will handle ensuring the new code meets biz and user expectations" is fine in isolation... but it may mean that the code does not benefit from shared patterns for observability, traffic shaping, debugging, and more.
And if the agent inlines code that itself had a bug, that later proves to be a root cause, the amount of code that needs to be found and fixed in an outage situation is not only larger but more inscrutable.
Using the OOP's terminology, where biz > user > ops > dev is ideal, this is a dev > ops style failure that goes far beyond "runs on my machine" towards a notion of "is only maintainable in isolation."
Luckily, we have 1M context windows now! We can choose to say: "Meticulously explore the full codebase for ways we might be able to refactor this prototype to reuse existing functionality, patterns, and services, with an eye towards maintainability by other teams." But that requires discipline, foresight, and clock-time.
How could this have happened? Well, the code was shipped but no customer was running it in production.