I was maybe a little unclear with the phrasing, I meant to say "software that's been around for 20 years is more reliable than software that's been around for 2 months"
But this is why using the AI in the production of (almost) deterministic systems makes so much sense, including saving on execution costs.
ISTR someone else round here observing how much more effective it is to ask these things to write short scripts that perform a task than doing the task themselves, and this is my experience as well.
If/when AI actually gets much better it will be the boss that has the problem. This is one of the things that baffles me about the managerial globalists - they don't seem to appreciate that a suitably advanced AI will point the finger at them for inefficiency much more so than at the plebs, for which it will have a use for quite a while.
> here are some example ideas that are perfectly true when applied to regular software
Hm, I'm listening, let's see.
> Software vulnerabilities are caused by mistakes in the code
That's not exactly true. In regular software, the code can be fine and you can still end up with vulnerabilities. The platform in which the code is deployed could be vulnerable, or the way it is installed make it vulnerable, and so on.
> Bugs in the code can be found by carefully analysing the code
Once again, not exactly true. Have you ever tried understanding concurrent code just by reading it? Some bugs in regular software hide in places that human minds cannot probe.
> Once a bug is fixed, it won’t come back again
Ok, I'm starting to feel this is a troll post. This guy can't be serious.
> If you give specifications beforehand, you can get software that meets those specifications
> bugs are usually caused by problems in the data used to train an AI
This also is a misunderstanding.
The LLM can be fine, the training and data can be fine, but because the LLMs we use are non-deterministic (at least in regard to their being intentional attempts at entropy to avoid always failing certain scenarios) current algorithms are inherently by-design not going to always answer every question correctly that it potentially could have if the values that fall within a range had been specific values for that scenario. You roll the dice on every answer.
Fully agree. Also inherent to the design is distillation and interpolation...meaning that even with perfect data and governing so that outputs are deterministic, the outputs will still be an imperfect distillation of the data, interpolated into a response to the prompt. That is a "bug" by design
> Because eventually we’ll iron out all the bugs so the AIs will get more reliable over time
Honestly this feels like a true statement to me. It's obviously a new technology, but so much of the "non-deterministic === unusable" HN sentiment seems to ignore the last two years where LLMs have become 10x as reliable as the initial models.
I'll totally grant that thing will get better over time. But a point that I was (mostly failing) to make, is that software is discrete, NNs are continuous. No matter how buggy your program is, it's got a countable number of lines, so you can have some notion of "all the bugs" (ignoring distributed systems).
But NNs are fundamentally continuous, I don't think it even makes sense to "count" bugs. You can have a list of prompts to which the model gives unwanted output, but it's a completely different ball game compared to regular software.
Thanks! I don't use RSS, but I believe https://boydkane.com/index.xml should work? I recently updated it to show all posts (not just the most recent 10) so it should be better now (I hope).
Where did "can't you just turn it off?" in the title come from? It doesn't appear anywhere in the actual title or the article, and I don't think it really aligns with its main assertions.
I don't understand the "your boss" framing of this article, or more accurately, the title of this article. The article contents don't actually seem to have anything to do with management specifically. Is the reader is meant to believe that not being scared of AI is a characteristic of the managerial class? Is the unstated implication that there is some class warfare angle and anybody who isn't against AI is against laborers? Because what the article actually overtly argues, without any reading between the lines, is quite mundane.
It would help if this piece was clearer about the context in which "AI bugs" reveal themselves. As an argument for why you shouldn't have LLMs making unsupervised real-time critical decisions, these points are all well taken. AI shouldn't be controlling the traffic lights in your town. We may never reach a point where it can. But among technologists, the major front on which these kinds of bugs are discussed is coding agents, and almost none of these points apply directly to coding agents: agent coding is (or should be) a supervised process.
My current method for trying to break through this misconception is informing people that nobody knows how AI works. Literally. Nobody knows. (Note that knowing how to make something is not the same as knowing how it works. Take humans as an obvious example.)
I think you’re confusing knowing how a system works with being able to predict how that system will perform.
In a non-linear system the former is often easier than the latter. For example we know how planets “work” from the laws of motion. But planetary orbits involving > 2 bodies are non-linear, and predicting their motion far into the future is surprisingly difficult.
Neural networks are the same. They’re actually quite simple, it’s all undergraduate maths and statistics. But because they’re non-linear systems, predicting their behaviour is practically impossible.
We don’t know how to build humans from scratch (other than letting nature do it) so that’s not really a relevant example. We only know how to fix certain aspects of defect humans, and how to minimize the chance of future defects, but that’s far from building a human from scratch.
This article makes a solid case. The worst kinds of bugs in software are not the most obvious ones like syntax errors, they are the ones where the code appears to be working correctly, until some users do something slightly unusual after a few weeks of some code change being deployed and it breaks spectacularly but the bug only affects a small fraction of users so developers cannot reproduce the issue... And the cose change happened such time ago that the guilty code isn't even suspected.
> It’s entirely possible that some dangerous capability is hidden in ChatGPT, but nobody’s figured out the right prompt just yet.
This sounds a little dramatic. The capabilities of ChatGPT are known. It generates text and images. The qualities of the content of the generated text and images is not fully known.
> The capabilities of ChatGPT are known. It generates text and images
There's a big difference between generating text which does someone's homework and text which changes peoples opinion about the world (e.g. the r/changemyview experiment done by Meta, in which their AI was better than almost all humans (it was 99th percentile) at changing peoples view, and not a single user was able to spot it as being AI[1])
If you're disagreeing with the precise wording of "capabilities" vs "qualities of the content", then sure, use whatever words make sense to you. But I don't think that's an interesting discussion to have.
70 years ago we were fascinated by the concept of converting analog to a perfect digital copy. In reality, that goal was a pipe drea!m and the closest we can ever get is a near identical facimile to which data fits... But it's still quite easy to determine digital from true analog with rudimentary means.
Human thought is analog. It is based on chemical reactions, time, and unpredictably (effectively) random physical characteristics. AI is an attempt to turn that which is purely digital into an rational analog thought equivalent.
No matter how much effort, money, power, and rare mineral eating TPUs will - ever - produce true analog data.
Not the point, but I’m confused by the Geoguessr screenshot. Under the reasoning for its decision, it mentions “traffic keeps to the left” but that is not apparent from the photo.
Then it says the shop sign looks like a “Latin alphabet business name rather than Spanish or Portuguese”. Uhhh… what? Spanish and Portuguese use the Latin alphabet.
For a real world example of the challenges of harnessing LLMs, look at Apple. Over a year ago they had a big product launch focused on "Apple Intelligence" that was supposed to make heavy use of LLMs for agentic workflows. But all we've really gotten since then are a couple of minor tools for making emojis, summarizing notifications, and proof reading. And they even had to roll back the notification summaries for a while for being wildly "out of control". [1] And in this year's iPhone launch the AI marketing was toned down significantly.
I think Apple execs genuinely underestimated how difficult it would be to get LLMs to perform up to Apple's typical standards of polish and control.
> get LLMs to perform up to Apple's typical standards of polish and control.
I reject this spin (which is the Apple PR explanation for their failure). LLMs already do far better than Apple’s 2025 standards of polish. Contrast things built outside Apple. The only thing holding Siri back is Apple’s refusal to build a simple implementation where they expose the APIs to “do phone things” or “do home things” as a tool call to a plain old LLM (or heck, build MCP so LLM can control your device). It would be straightforward for Apple to negotiate with a real AI company to guarantee no training on the data, etc. the same way that business accounts on OpenAI etc. offer. It might cost Apple a bunch of money, but fortunately they have like 1000 bunches of money.
Apple’s experience has almost nothing to do with “harnessing” LLMs, and everything to do with their wildly misjudged assumption they could run a viable model on a phone. Useful LLMs require their own power plants and can only be feasibly run in the cloud, or in a limited manner on powerful equipment like a 5090. Apple seems to have misunderstood that the “large” in large language model isn’t just a metaphor.
Tremendous alpha right now in making scary posts about AI. Fear drives clicks. You don't even need to point to current problems, all you have to do is say we can't be sure they won't happen in the future.
All the same criticisms are true about hiring humans. You don’t really know what they’re thinking, you don’t really know what their values and morals are, you can’t trust that they’ll never make a mistake, etc.
82 comments
[ 4.9 ms ] story [ 84.8 ms ] thread:)
Was that a humam Freudian slip, or artificial one?
Yes, old software is often more reliable than new.
ISTR someone else round here observing how much more effective it is to ask these things to write short scripts that perform a task than doing the task themselves, and this is my experience as well.
If/when AI actually gets much better it will be the boss that has the problem. This is one of the things that baffles me about the managerial globalists - they don't seem to appreciate that a suitably advanced AI will point the finger at them for inefficiency much more so than at the plebs, for which it will have a use for quite a while.
Hm, I'm listening, let's see.
> Software vulnerabilities are caused by mistakes in the code
That's not exactly true. In regular software, the code can be fine and you can still end up with vulnerabilities. The platform in which the code is deployed could be vulnerable, or the way it is installed make it vulnerable, and so on.
> Bugs in the code can be found by carefully analysing the code
Once again, not exactly true. Have you ever tried understanding concurrent code just by reading it? Some bugs in regular software hide in places that human minds cannot probe.
> Once a bug is fixed, it won’t come back again
Ok, I'm starting to feel this is a troll post. This guy can't be serious.
> If you give specifications beforehand, you can get software that meets those specifications
Have you read The Mythical Man-Month?
Did you read the footnote about writing regression tests to catch bugs before they come back in production?
https://news.ycombinator.com/item?id=45583970
Thought I might just skip the repetition. You can continue the conversation within that thread.
This also is a misunderstanding.
The LLM can be fine, the training and data can be fine, but because the LLMs we use are non-deterministic (at least in regard to their being intentional attempts at entropy to avoid always failing certain scenarios) current algorithms are inherently by-design not going to always answer every question correctly that it potentially could have if the values that fall within a range had been specific values for that scenario. You roll the dice on every answer.
Related opposing data point to this statement: https://news.ycombinator.com/item?id=45529587
Honestly this feels like a true statement to me. It's obviously a new technology, but so much of the "non-deterministic === unusable" HN sentiment seems to ignore the last two years where LLMs have become 10x as reliable as the initial models.
But NNs are fundamentally continuous, I don't think it even makes sense to "count" bugs. You can have a list of prompts to which the model gives unwanted output, but it's a completely different ball game compared to regular software.
In a non-linear system the former is often easier than the latter. For example we know how planets “work” from the laws of motion. But planetary orbits involving > 2 bodies are non-linear, and predicting their motion far into the future is surprisingly difficult.
Neural networks are the same. They’re actually quite simple, it’s all undergraduate maths and statistics. But because they’re non-linear systems, predicting their behaviour is practically impossible.
[1]: https://www.reddit.com/r/slatestarcodex/comments/1o6n5ne/why...
[1] https://www.economist.com/leaders/2025/09/25/how-to-stop-ais...
This sounds a little dramatic. The capabilities of ChatGPT are known. It generates text and images. The qualities of the content of the generated text and images is not fully known.
There's a big difference between generating text which does someone's homework and text which changes peoples opinion about the world (e.g. the r/changemyview experiment done by Meta, in which their AI was better than almost all humans (it was 99th percentile) at changing peoples view, and not a single user was able to spot it as being AI[1])
If you're disagreeing with the precise wording of "capabilities" vs "qualities of the content", then sure, use whatever words make sense to you. But I don't think that's an interesting discussion to have.
I stand by my original statement.
[1]: https://www.reddit.com/r/changemyview/comments/1k8b2hj/meta_...
Human thought is analog. It is based on chemical reactions, time, and unpredictably (effectively) random physical characteristics. AI is an attempt to turn that which is purely digital into an rational analog thought equivalent.
No matter how much effort, money, power, and rare mineral eating TPUs will - ever - produce true analog data.
Or they have, but chose to exploit or stockpile it rather than expose it.
Me: Ask me later.
Then it says the shop sign looks like a “Latin alphabet business name rather than Spanish or Portuguese”. Uhhh… what? Spanish and Portuguese use the Latin alphabet.
I think Apple execs genuinely underestimated how difficult it would be to get LLMs to perform up to Apple's typical standards of polish and control.
[1] https://www.bbc.com/news/articles/cge93de21n0o
Why not take the easy wins? Like let me change phone settings with Siri or something, but nope.
A lot of AI seems to be mismanaging it into doing things AI (LLMs) suck at... while leaving obvious quick wins on the table.
I reject this spin (which is the Apple PR explanation for their failure). LLMs already do far better than Apple’s 2025 standards of polish. Contrast things built outside Apple. The only thing holding Siri back is Apple’s refusal to build a simple implementation where they expose the APIs to “do phone things” or “do home things” as a tool call to a plain old LLM (or heck, build MCP so LLM can control your device). It would be straightforward for Apple to negotiate with a real AI company to guarantee no training on the data, etc. the same way that business accounts on OpenAI etc. offer. It might cost Apple a bunch of money, but fortunately they have like 1000 bunches of money.
Also kinda crazy that all the "native" voice assistants are still terrible, despite the tech having been around for years by now.
[1]: https://boydkane.com/essays/boss#user-content-fn-7