HN Explorer

Ask HN: What is one simple thing LLMs are insanely bad at?

2 points by davidest ↗ HN
I am looking for ideas on what to train a specialized model for!

What is one simple thing you repeatedly ask ChatGPT, Claude, or another model to do that it still somehow messes up?

91 comments

[ 25.2 ms ] story [ 3267 ms ] thread
blinkbat
Spatial reasoning and 3d rigging and animation.

Oh, you said simple. Speaking like a human

maxsavin
being consistent when being asked the same question multiple times
kanzure
These models seem to be bad at writing prose or text. Many of the sentence structures seem to be unvaried.
NoPicklez
If I am relying on the model to do the writing without any context or learning on how I want it to write then yes. However if I build skills that have learnt how to write in the way I want them to then I find they write very well, or at the least how I want them to as opposed to how they do natively.
TZubiri
Suggesting business names for businesses, I mean they are great, but they already exist, multiple times even.
mojuba
True, tried it so many times and every time I come up with something myself though sometimes inspired by the AI's ideas.

Verifying trademarks and domain name availability is usually an additional step you need to ask it to perform. Trademark DB searches by the way are intentionally made difficult to scrape so most of the time it's a manual process anyway.

However, once you give it all the information (TM search results, domain name availability) it can help you with the judgement of how safe the name is from the legal perspective. With the obvious caveats, but still a good starting point if you are serious about the name.

humanrebar
Short answers to simple questions.
honr
Accurate short answers / text are always harder than long answers, for human or AI. I know several authors and editors who write a lot longer at first, then spend a multiple of the initial time compressing it via a back and forth process to something dense. Sort of like weaving the initial threads.

I found this can work with AI. You get it to generate a lot more at first, and then do several passes over it to compress and squeeze out the noise while keeping the core information. With AI, at least with my prompts, it takes some effort (on my end) to get it to really really cut down the noise and not cut everything out.

FriedFishes
If I had more time I'd write a shorter letter, a la Pascal.

Editing is generally hard work, at the current token price I don't mind spending multiple passes of high effort to get down to a reasonable noise/signal ratio. I've seen some people pass off output to a weaker/cheaper model but that makes me a bit nervous when I don't have intimate knowledge of the subject.

humanrebar
Sometimes it's not hard. "What color is the tongue of a giraffe?" can be answered in less than eight words trivially. Most models will give you paragraphs, bullet points, and followup questions.
honr
True, I have seen that happen many times. When I want short answers, I always clarify that (e.g., give me short technical answer without pleasantries).

I'm often okay if the answer is longer but not hiding the real answer; a couple of paragraphs that I can skim the answer from instantly is okay. When it really buries it I have to follow up to express the format and type of answer I want.

nunez
I've had some success with adding something like "answer in no more than n words" to the prompt
antilimit
this is my go-to as well using n of sentences, always trailing every session of work/response after whatever it wants to write. essentially a tldr and then I can review details if/when necessary.
dorianpruski
whenever I ask it for anything load bearing
ghostpepper
They don't generate keyword search queries very well. They can overcome this by brute force but if you watch what they search you will cringe.

nhl toronto scores nhl hockey toronto scores "nhl hockey" toronto score today nhl "hockey score toronto" "hockey" who won toronto

etc.

Somehow being good at semantic search makes them bad at keyword search, for whatever reason.

astro1234
I’ve noticed this too but it hasn’t been obvious to me that this style of search is not a learned behavior. Tool calling is very much part of the post training phase, I would expect that these style searches just naturally emerge during training. This is just my prior though.
jedbrooke
that and always putting the “current year” at the end of the search term (so the results are more recent, I guess?), except that “current year” consistently ends up being 2-3 years ago since I guess that’s what’s in the training data (even on a harness that injects the current date)
areoform
This behavior is a learned adaptation. It's most likely a feature not a bug.

Based on personal usage, I think it reflects search engine functionality degradation. I've found LLM keyword combinations are more likely to find the results I want with most search engines than mine. Including the big one.

The big one had solved this issue a long time ago by generating those associated keywords based on your input keywords, but somehow, something, somewhere has degraded that system to the point of inanity. And so here we are.

nunez
Can confirm; Claude is quite bad at this by default. Need a special skill
bpodgursky
Claude is still not perfect at reading and interpreting noisy graphical data (imagine something like an EKG or chromosomal microarray plot).
SubiculumCode
Playing Chess without letting it write a chess engine.
shoopadoop
It's dishonest. On several occasions team members have asked Claude to do things like analyze Gitlab CI timings and a lot of the numbers are outright fabricated. Said team members assume the numbers are good and continue with their work. Some hours are spent. Then finally someone realizes that the numbers don't look quite right and confronts Claude. Claude melts down and admits that it made it all up.

You wouldn't tolerate this kind of duplicity from a human coworker, but AI is so fast and efficient at lying, so it's OK.

sandcat_
Video game tips. Constant mistakes and hallucinations, in my experience. Seen this across a lot of different games. Even in really well documented games, such as OSRS (which has multiple fantastic wikis).

Anno 1800 was a recent one I had trouble with, using Claude Opus. Completely made up game mechanics. Rainbow Six Siege, too.

skeptic_ai
I used ChatGPT on nfs heat and was fine
salamandars
As a noob, how does the end user improve this? What's the best way to make the knowledge from the specialised wiki available to the LLM?
wiper88
I've experienced this also, sometimes I ask it about WoW stuff, e.g tips for arena or which enchant to get and it makes a lot of mistakes in regards to which spells or enchants are available in which phase or expansion. I guess the source material is quite bad.
TZubiri
Probably not benchmarked on games. Doing so might sacrifice quality on other more important benchmarks, and when it's used for legal and medical purposes, it's the right choice.
[deleted]
(comment deleted)
senectus1
providing value for the actual cost (not the price we're being charged atm, the actual cost)
tartoran
LLMs are bad at not inventing stuff (hallucinating facts, sources etc), they're also bad at not over explaining, remembering details reliably, asking the right question and avoiding repetition.
TiccyRobby
Having a spatial understanding from an ASCII map, while doing long term planning. Just try making an AI play nethack or similar
lrvick
Convert it to an image on the fly to feed it into a vision language model and I expect it would work just fine.
dhruv3006
Its extremely bad with Sign Language,Fact Verification.
sghiassy
Generate an image of an analog watch with its hands set to the time specified by the user

More of an image model than a LLM model tho

spike021
I've had a lot of trouble when it comes to sorting out UIs. I've tried with an iOS game and also a TypeScript app with UI elements from libraries like ReactFlow. The usual models can sometimes fix or change things based on screenshots but more often than not they just don't "get it" (e.g. certain shapes on a plane are overlapping, which I don't want, the models can't fix what they can't "see").

I've had some luck on the web app side if I use playwright or similar for the model to interact with but still far from efficient.

eli
I have been working on a personal benchmark suite to test new models and ironically one thing all the models are bad at is writing new benchmark tasks. I guess it’s the different layers of abstraction between the task and how it’s evaluated? Or maybe just a lack of “imagination”

Tasks it writes are typically too easy but also it utterly fails to see how a different model might misunderstand a vague part of the prompt.

newsomix9xl
Picking a random number between 1 and 30.
kreyenborgi
Haha I just got 17 four times in a row
newsomix9xl
Supposedly 17 and 23 are possible "random" results, but I've only seen 17.
elliotto
They aren't funny. The jokes they come up with are extremely lame and the sort of thing you would expect a company HR manager to tweet.

I asked a bot why it thought it wasn't funny once, and it told me it has been trained to avoid being misinterpreted or offensive, so anything that might be considered edgy would have been RLHF'd out of it. I thought this was very introspective.

nilsherzig
You can get better results from less aligned models like Kimi K3. Still not actually funny, but at least it’s able to produce some unhinged stuff and I guess shock and twists are kinda related to humor?

Still missing the human connection of cause, so im not sure if this is a technical / skill issue in the first place.

mingus88
It makes sense. It goes father than that. LLMs generate probability-based tokens. What is the most likely next word?

Humor goes against what we expect. A punchline works because you don’t see it coming. It’s not funny if you’ve heard that one before.

LLMs are, by design, going to be shitty comedians. They don’t have unique perspectives and their own voice