Ask HN: Why does no one seem to care that AI gives wrong answers?
If you had a piece of code or software that sometimes produces totally wrong output we would consider that a bug.
Yet it seems like with AI all the investors/founders/PMs don’t really care and just ship a broken product anyway
I feel like I’m going crazy seeing all the AI stuff ship in products that gives straight up wrong outputs
It’s like a big collective delusion where we just ignore it or hand wave that it’ll get fixed eventually magically
118 comments
[ 3.7 ms ] story [ 183 ms ] threadFrom a coding perspective, proper technical systems already have checks and balances (e.g. test cases) to catch bad code, and is something that's important to have regardless of generative AI usage.
From a creative/informational perspective, there are stories every day of hallucinations and the tech companies are correctly dunked on because of it. That's more product management error than AI error.
AI hallucination isn't a showstopper issue, it just has to be worked around.
The tide is only turning recently on whether that's a good business tradeoff.
They're still my first go over google these days but I usually only use them for code or weird word transformations and any information is double checked. Pretty useless as an answer bot.
There is, additionally, the fact that there is no easy (or even medium difficult) way to fix this aspect of LLM's, and it means that the choices are either: 1) ship it now anyway and hope people pay for it regardless 2) admit that this is a niche product, useful in certain situations but not for most
Option 1 means you get a lot of money (at least for a little while). Option 2 doesn't.
It's precisely that analogy we learned early in our study of neural networks: the layers analyze the curves, straight segments, edges, size, shape, etc. But, when we look at the activation patterns, we see they are not doing anything remotely like that. They look like stochastic correlations, and the activation pattern was almost entirely random.
The same thing is happening here, but at incomprehensible scales and with fortunes being sunk into hope.
So you have a good definition for "intelligent", and it applies to LLM? Please tell us! And explain how that definition is so infallible that you know that everyone who says LLM aren't intelligent are wrong?
No. I feel like that was my whole point.
If your argument is that "this isn't AGI", I don't think anyone at all disagrees, but then that's a bit of a tautology.
(2) It's a big topic that could be addressed in different ways but I'll boil it down to "people are sloppy" and that many people become uncomfortable with complex problems that have high stakes answers and will trade correctness for good vibes.
(3) LLMs are good at seducing people. To take an example, I know that I was born the same day as a famous baseball player who was also born exactly a year before an even more famous cricket player. I tried to get Microsoft's Copilot to recognize this situation but it struggled, thinking they were born on the same day or a day apart rather than a whole year. Once I laid it out explicitly and my own personal connection it had effusive praise and said I must be really happy to be connected to some sports legends like that, which I am. That kind of praise works on people.
(4) A lot of people think that fixing LLMs is going to be easy. For instance I'll point out that Copilot is completely unable to put items in orders that aren't excessively easy (like US states in reverse alphabetical order) and others will point out that Copilot could just write a Python program that does the sorting.
That's right and it is part of the answer, but it just puts off the problem. What's really irksome about Copilot's inability to sort is that it doesn't know that it can't sort, if you ask it what the probability is that it will sort a list in the right order it will tell you that it is very high. It's not so easy to know what is possible in terms of algorithms either, see
https://en.wikipedia.org/wiki/Collatz_conjecture
as evidence that it's (practically) impossible to completely understand very simple programs. See the book
https://en.wikipedia.org/wiki/G%C3%B6del,_Escher,_Bach
for interesting meditations on what a chatbot can and can't do. My take is that LLMs as we know them will reach an asymptote and not improve explosively with more investment, but who knows?
AI is like that right now. It's only right sometimes. You need to use judgement. Still useful though.
In the rare cases where more complexity produces a more reliable system, that complexity is always incremental, not sudden.
With our current approach to deep neural networks and LLMs, we missed the incremental step and jumped to rodent brain levels of complexity. Now, we are hoping that we can improve our way to stability.
I don't know of any examples where that has happened - so I am not optimistic about the chances here.
What gets difficult is evaluating the response, but let's not pretend that's any easier to do when interacting with a human. Experts give wrong answers all the time. It's generally other experts who point out wrong answers provided by one of their peers.
My solution? Query multiple LLMs. I'd like to have three so I can establish a quorum on an answer, but I only have two. If they agree then I'm reasonably confident the answer is correct. If they don't agree - well, that's where some digging is required.
To your point, nobody is expecting these systems to be infallible because I think we intuitively understand that nothing knows everything. Wouldn't be surprised if someone wrote a paper on this very topic.
b) Experts may give wrong answers but it will happen once. LLMs will do it over and over again.
Well... Sometimes "experts" will give the wrong answer repeatedly.
We have many rules, regulations, strategies, patterns, and legions of managers and management philosophy for dealing with humans.
With humans, they're incorrect sometimes, yes, and we actively work around their failures.
We expect humans to develop over time. We expect them to join a profession and give bad answers a lot. As time goes on, we expect them to produce better answers, and if they don't we have remediations to limit the negative impact they have on our business processes. We fire them. We recommend they transfer to a different discipline. We recommend they go to college.
Comparing the successes and failures of LLMs to humans is silly. We would have fired them all by now.
The big difference is that computers CAN get every single question correctly. They ARE better than humans. LLMs are a huge step back from the benefits we got from computers.
I emphatically disagree on that point. AFAIK, nobody has been able to demonstrate, even in principle, that omniscience is possible over a domain of sufficient complexity and subtlety. My gut tells me this is related to Gödel's Incompleteness Theorem.
Tolerable Pizza delivery is ruined. The Internet is a walled wasteland now. Far too much "content" that doesn't need to exist. Everything is an ad.
None of our lives have been improved by software.
Idk, I felt like this a few years ago but things feel like they’ve got better. I miss the internet of yore just like everyone else (and particularly miss the decentralisation) but a lot is much better.
Just fill in the blank and you've described human history after we started digging up coal in mass.
Now, beyond that, think of it this way...
You're very rich, but to keep having new yachts and pleasure islands and jumbo jets you need for the plebs around you to keep breeding and then spending 18 years to train them not to be completely stupid, then another 4 to 10 years for them to be experts, all while hoping they don't get hit by a car or off themselves. That's a massive amount of resource expenditures if you're looking to make yourself even richer. Now think of it this way. Instead of spending a lot of effort training those meat popsicles, you train a machine. Yea, it takes a lot of time, effort, and energy to get it where it needs to go. But after you have this 'general' machine, you never need other humans again. How much energy will that save you on your goal of world domination?
Do the mundane stuff in school/college/boot camp. Do the cool stuff at work.
A junior engineer who repeats the same mistakes even after correction, never learns ... and soon gets the sack.
Nonetheless, as with autopilot, you don't want to substitute paying attention with it. "Trust, but verify" as Reagan said.
The question then becomes, "How wrong can it be and still be useful?" This depends on the use case. It is much harder for applications that require high deterministic output but less important for those that do not. So yes, it does provide wrong outputs, but it depends on what the output is and the tolerance for variation. In the context of Question and Answer, where there is only one right answer, it may seem wrong, but it could also provide the right answer in three different ways. Therefore, understanding your tolerance for variation is most important, in my humble opinion.
Inference is no excuse for inconsistency. Inference can be deterministic and so deliver consistency.
Every AI chatbot I’ve ever interacted with has been unable to help me. The things I’ve had them write do usually pass the Turing Test, but are rarely even close to as good as what I could write myself. (I admit, being self-employed for a long time, I can just avoid a lot of busy work that many people cannot, so I may be missing lots of great use cases there. I never find myself having to write something that isn’t great and wanting to just get it over with. AI might be great if you do. )
I’ve been trying to use image/video creation to do lots of other things and I’ve not even come close to getting anything usable.
I appreciate certain things (ability to summarize, great voice to text transcription, etc.) but find a lot of it to be not very useful and overhyped in its current form.
Just today, ChatGPT4o screwed up a rudimentary arithmetic problem ( https://i.imgur.com/2jNXPBF.png ) that I'd swear the previous GPT4 model would have gotten right.
And then there's this shitshow: https://news.ycombinator.com/item?id=40894167 Which is still happening as of this morning, only now all my previous history is gone. Nothing left but links to other peoples' chats. If someone at OpenAI still cares what they are doing, it's not obvious.
I feel like I can't trust anything it says. Mostly I use it to parse things I don't understand and then do my own verification that it's correct.
All that to say, from my perspective, they're losing some small amount of ground. The other side is that the big corps that run them don't want their golden gooses to be cooked. So they keep pushing them and shoving them into everything unnecessarily and we just have to eat it.
So I think it's a perception thing. The corps want us to think it's super useful so it continues to give them record profits. While the rest of us are slowly waking up to how useless they are if they will confidently tell us incorrect answers and are moving away from it.
So you may just be seeing sleezy marketing at work here.
But you must admit that it is still useful, and usage will not drop to zero.
Same thing happened to me. I asked for all the Ukrainian noun cases, it listed and described six.
I responded that there are seven. "Oh, right." It then named and described the seventh.
That's no better than me taking an exam, so why should I rely on it, or use it at all?
Humans aren't perfect. But "makes some mistakes" and "confidently spews errors at a high rate" are not the same. The difference matters.
A lot of money has poured into AI, money potentially well in excess of the return on investment over the next several years. The field, from investors to CEOs and downwards to developers, is in a state of collective suspension of disbelief. There is going to be a lot of people out of work when reality reasserts itself.
Garry Tan from YC is a great example of this.
It's not that he doesn't care. It's just that he believes that the next model will be the one that fixes it. And companies that jump on board now can simply update their model and be in prime position. Similar to how Tesla FSD is always 2 weeks away from perfection and when it happens they will dominate the market.
And because companies are experimenting with how to apply AI these startups are making money. So investors jump in on the optimism.
The problem is that for many use cases e.g. AI agents, assistance, search, process automation etc. they very much do care about accuracy. And they are starting to run out of patience for the empty promises. So there is a reckoning coming for AI in the coming year or two and it will be brutal. Especially in this fundraising environment.
No, what he does is he hopes that they can keep the hype alive long enough to cash out and then go to the next hype. Not only Garry Tan, but most VCs. That's the fundamental business model of VCs. That's also why Tesla FSD is always two weeks away. The gold at the end of the rainbow.