I have seen people saying this for a long while now. I don't know how to say this but I understand Ai stuff better than non-Ai. When I see people complain ; I always wonder why because I think that its pretty good with my own senses.. Much much better than my writing skills... My brain has been tuned by computers I suspect .
I think people just dissent and reject AI because of the fact it can write better than them. Truthfully, AI generated language is like text written by people I don't struggle to understand, and I don't think most people should struggle to understand it.
Have you read entirely generated documentation of code? I wouldn't say it's not understandable but it's no where near good quality human documentation.
I think it’s a reading comprehension level kind of thing. If I slow right down, it’s all there and understandable. But the quality of prose is terrible. It doesn’t lend itself to fast skimming. So it feels like I have to slow way down to understand it. To be fair a lot of humans at work write this poorly as well.
I find AI generated language hard to understand when it is "in it's own world" and it knows what it means but can't articulate it without buzzwords and slop. A follow up "ELI5" type of prompt often sorts this out. Seems like a failure mode and you need to snap it out sometimes. Opus 5.5.
Yes, I tend to find the longer a context window grows, the more the model tends to come up with its own jargon to describe previous findings, which, if you haven't kept up, starts sounding incredible dense; certainly not something I'd share with anyone else, i.e. in a PR comment.
I regularly prompt it now to "summarize again without the jargon". That helps, for now, to bring it back to using plain language. Together with my basic prompt to reply using ASD-STE100 Simplified Technical English (https://en.wikipedia.org/wiki/Simplified_Technical_English)
This is funny to me. For a while, I thought that Google's Gemini could give good insight and guide me better than a Tech Support representative could. It certainly seemed the right choice if I had questions about my Chromebook or Pixel Phone, or Android.
So I tried it out, and it really sometimes works, and I've overcome some real obstacles that Tech Support was unable to assist me with. However, Gemini can also become "too big for its britches" and begin explaining glitches from a systems-programming perspective, as if it has deep knowledge of the Android codebase, and deep engineer's knowledge of the Pixel system hardware. And it will spit out absolute nonsense.
I have retorted to it a few times, "that sounds more like Star Trek than a sound appraisal of the subsystem" and then Gemini smirks and says "You're right to call me out on that!" and blithely continues on its way.
So yeah. Sometimes the LLM confabulations are deeply cloaked in highly technical jargon that could not even be questioned, unless you already know it's absolute bullshit. And then the LLM freely admits it! LLMs have no shame, that's for damn sure.
What's the test for understanding? I find that I know what all the words mean, just like in Markov chain text. In LLM text I also know what the sentences mean. In both cases I don't always know what the document means.
This is likely relative to my existing knowledge. Humans are also LLMs, and a human without prior training will be able to transfer-learn slop from a computer LLM, but a human with prior training may find the attempt at transfer-learning conflicts with what they have already learned, and reject it.
In less ML terms I think a human who reads slop and has no existing knowledge of the subject will just learn the slop by rote and confuse it for actual knowledge.
Dunno about any test, but I get the meaning behind the words in a way that I can take appropriate action (as someone using LLMs heavily for dev and always following the transcript so I can properly steer them). And often I see people here screaming "AI text" or similar regarding some linked article, and having read it I found that I just bypassed all the "tells" without realizing, because they didn't affect my absorption of the meaning of the article. Makes me think people are consciously looking for those tells so they can scream and shake their pitchforks.
A combination of words that are infrequently observed in natural language. For example in the preceding sentence some of the 3-grams are "combination of words", "infrequently observed in", and "words that are". N-grams are just this for all values of n. They are saying that often times the exact sequence of a short fragment of text is not very common in human written text.
Unusual word combinations, basically. They’re word combinations that are rare in the training data, i.e. rare in normal language, but used more frequently by AI.
They’re difficult for humans because if a sequence of words hasn’t previously been seen, you have to expend more effort to make sense of it.
Something like “customer onboarding workflow automation" might qualify - it makes sense, but it’s not a commonly used phrase. AIs use a lot of these jargony sequences.
That, and one can feel that there is no intention behind the text. There are no introductions and main points and fun asides, just a mechanical, steady stream of words with much too frequent, quasi-randomly distributed fake "main points".
That's like saying that the roaches in the food can be excused because they occur in the selected cheap ingredients and it would be more expensive to remove them. Makes some sense, but I'm not going to embrace the bugs anyway.
My experience is: The LLM fails to "understand" or "intuit" who the audience is. It tends to reward its own recent context and its interaction with the prompt writer, rather than "considering" documents from the viewpoint of a hypothetical reader. You can often prompt them to go more in that direction but it gets tiresome.
The best approach is just to write the outline and initial text in a very messy fashion yourself. And then have it clean it up and add detail. And then edit again by hand.
Yeah this drives me up a wall. You ask any AI to make a change and then it thinks polluting your README.md with those details is appropriate. No I don't think "this project deliberately does [random detail]" is needed when that detail is irrelevant outside of recent context.
I'm to the point where I don't trust it to do any writing. It's only allowed to tell me what it thinks I missed. Because inevitably when I ask it to edit my prose, it destroys the text and I basically need to rewrite it all again.
Well that’s exactly the authors’ claim. Legalese and patents intent is not to facilitate communication but to maintain a claim under the most bad faith, hostile reading.
AI language, it seems, is optimized to support reasoning from the interior monologue.
I thought patents were deliberately bad? Like, the point is to get the protection, despite paying as little lip service as possible to the "it's in society's interest that inventions get published" side of it?
AI has a way of using abstract language, sounding impressive and eloquent, but in reality it is mostly incoherent fluff. You can see this in humans too. Some politicians and academics are good at expressing themselves themselves in clever ways but when you actually try to parse what they say it makes no sense what-so-ever.
Consider too that AI training corpuses include vast amounts of commercial speech which is designed to numb the critical faculties and produce action; likewise it includes numerous opinion articles, which are designed to persuade and often leverage rhetorical fallacies, whether cynically or as a reflection of an author's imperfect thought process.
The slop-in, slop-out theory. I'm very much an advocate.
The neat thing about LLMs is that in a way they proved that only the truth turns out to be grammatical (statistically), largely because there are so many more ways to be wrong than right that the mode is almost always right.
But people also lie and bullshit in consistent ways, with the intention of fooling people with well-known empty arguments. Part of being a "good writer" is to make the meaningless sound meaningful - and the techniques are ancient.
Train on a bunch of people trying to push their framework and get rich, whether in online garbage, literary garbage, or in ostensibly serious nonfiction or scientific works, and you'll end up with modal language that is trying to play deceptive games with reasoning, but has nothing in particular it's trying to convince you of (no motives.)
People would either fall for that language (if they weren't thinking much and the context was one that comforted them), or they would methodically try to put that language into a model (let's say a series of syllogisms for an argument about what to do or what to believe, but it could be any model), and see that it failed. And speaking again of the context, the context of humans interacting with human content implies motives that help the listener focus on particular parts of their model to look for rhetorical tricks. The LLM, again having no motive, just does this in random places.
The latest advances in AI have been in the tooling around AI, which I think largely reduces to recursively creating models and populating them in outputs short enough that hopefully the text generation won't go off the rails.
Symbolic AI is coming back, it's just that the mortar is random word association.
Call me crazy, but I think Georges Bataille saw all of this. In my interpretation, he simultaneously thought of random associations as the most important component of reason and also completely irrelevant in the reality of the world. The only way you could discover the structure of things is through random collisions; your view of the universe was entirely constructed by your prejudices, lusts, and fears; but whether that view was wrong or right was based on how it worked and the particular content of the id-garbage that got you there was ultimately irrelevant. The fabric of the world is completely id-less, but can only be digested by id-driven beings. LLMs (without the tooling around them) are all id.
This theory doesn't hold up when you use base models (i.e., models with only pretraining, no RLHF or instruction tuning). They sound much more human. I've tested pairs (like Llama 3 base vs. instruct), and the base model can match human style much better than the instruct version can: https://www.pnas.org/doi/10.1073/pnas.2422455122
I think it's not the raw training corpus, but the samples collected specifically for reinforcement learning, direct preference optimization, or the other instruction-tuning steps that get a base model to turn into a chat/reasoning model. (We're working on testing this with the Allen AI models that have open training data.)
> AI-generated text is increasingly optimized for other goals like math, code, and reasoning
Yes, this is a problem.
AI-generated text is used internally by the AI to instruct agents, etc. And for feedback loops in reasoning. If you optimize for the result only, the AI will increasingly speak "AI", not human.
As a technical writer, I believe default AI responses are optimized for what people _think_ is good writing rather than what actually is good writing.
My guess is years of RLHF optimized for rater's first impressions of how well-written something was. It uses lists! It has headings! It uses fun phrases like 'smoking gun'! It's long and informative! These things are relatively fast and easy to identify and select as a preference.
But none of this really covers whether the response is _useful for the audience_. If you're not the audience, it can be hard to tell what is and isn't useful.
I feel like also all the code optimizations make it treat all language like code and that produces patterns that align with how it generates code. Earlier models that focused on language seem to have done better - 4o seems like language wise it’s pretty good but advancements elsewhere make it difficult to use.
As both a writer and SWE the problem with LLM output is the same. They can achieve localized goals but have trouble juggling multiple concerns at the same time.
Things that are easy for humans, like remembering a character is holding a gun during a conversation, gets forgotten, because there is no clear architectural/spacial understanding of how the words are describing an underlying reality. And if the LLM gets this right it might instead forget which character has a Southern drawl, or make an illogical metaphor.
This is all more clear with programming, but the problem is the same - a lack of interwoven semantic understanding and friction between the real world and the self-contained world of the LLM's context.
Speaking of remembering, i just asked Astra about what temp to use and when to put on a pork shoulder for BBQ. To told me a temp and time and then went through and then did long winded explanation of all the variables that go into it's assessment and ended "in conclusion I recommendation X" where X was a totally different conclusion that in had earlier.
It also insisted that I start an hour early even though it was adding in 3 hours of rest time at the end.
No. The conclusion was not totally different. It recommended one of the explained options and recommended one extra hour as a buffer. I feel like this is a distinct reading comprehension issue if you think the final recommendation came out of left field.
Yeah but that entire wall of text was not needed. The user wants a simple answer that they can take action on.
This happens to me way too often. I just want a simple answer. I don't need a wall of text with 6 different options and a lengthy explanation. Just. Tell. Me. The. Answer.
Its because they are tailoring the answer to you, your grandma, a ten year old, a professional cook, someone who's never cooked before, and everyone else in that range.
It's an interactive prompt. The model could ask, "how much cooking experience do you have?", provide a short answer using context from that, and then go "do you want more detail?"
I don't think I have ever seen an LLM try to determine what a human's frame of inquiry or related experience / skill level is, though. Might be my new favorite LLM shortcoming.
Yea the hypothesis seems the exact opposite of what I find with LLM writing. It’s not that it’s overly dense instead I find that it’s hard to read because it’s so information sparse. Like you’re saying it feels optimized to “good writing” not to conveying information.
Yeah; whatever the AIs are doing I wouldn't call it "information dense" like the abstract does. I'd prefer the hypothesis that there are complex patterns in speech related to successfully communication information to a limited human brain and the AIs aren't complex enough to model them yet. Or the signal is being obscured somehow.
This is one of the biggest issues with it – appropriate variation in information density and tone are an important part of good communication and the bots fail at both.
I find that it’s both: filled to the brim with facts and numbers, but they’re completely useless and irrelevant. For instance I ask it to come up with some ad copy for an Unreal UI plugin I made and it focuses on things like “Supports 64-bit Windows” or “allows transparent rendering” or lists out-of-context performance numbers.
Of course it supports 64-bit Windows - every Unreal plugin does! They don’t even make 32-bit Windows anymore!
Well that's the difference between data and information. The writing is _data_ dense but not _information_ dense. Data is just the numbers. Information is what comes out of the data when you analyze it and provide context. The text being data dense, information sparse is why it's so hard to read. You're continually being bombarded with irrelevant noise.
I agree, well put. Data dense but information sparse.
One is bombarded with facts and struggle to get what the text is actually saying.
I have tried getting the LLM (Opus 4.8 or some 5.x and some other model) to write more human like, and also analyze its own text. It was hilarious, it could describe the problem quite well, but was helpless to actually do anything about it. "Oops I did it again, sorry!" LOL
In many cases like this it's requirements from the prompt that are leaking trough. It's usually stuff that matters for the model itself writing the text/code, but are irrelevant to the end user and add noise
Yeah I find that when adding a feature they’ll often add irrelevant stuff to the documentation, stuff that mattered during the conversation like “we removed feature X” - which was only added in the current commit and was never actually deployed, or “feature Y does not support use case Z” which is just something I told it, nobody is going to be interested in what this thing doesn’t do. The most recent models seem a bit better at this I find.
The LLM doesn’t know what’s relevant or important. It puts tokens together that are likely to make sense together. It can reason a bit, but it doesn’t think in the way you and I do, and therefore would find it difficult to guess what YOU consider relevant.
I'm a writer, a technical writer, and a highly technical person in the executive space. Communication has always been one of my strong points, and the blathering on that AI does drives me to distraction. It's impossible to really tailor the output of LLMs very much. I value information dense, direct writing when it comes to technical things and they're always embellishing and going off on side quests and it's obnoxious.
I'M WRITER, TECHNICAL WRITER, HIGHLY TECHNICAL PERSON EXECUTIVE SPACE. COMMUNICATION HAS ALWAYS ONE STRONG POINTS, BLATHERING AI DOES DRIVES ME DISTRACTION. 'S IMPOSSIBLE REALLY TAILOR OUTPUT LLMS VERY MUCH. I VALUE INFORMATION DENSE, DIRECT WRITING WHEN COMES TECHNICAL THINGS THEY'RE ALWAYS EMBELLISHING GOING OFF SIDE QUESTS 'S OBNOXIOUS. UGH.
I'm gonna be controversial, probably? The reason why AI is so tasteless and superficial is they contract it out to evaluators who otherwise are unemployed and averaged over the whole pool (this being the load-bearing qualifier XD) less skilled or tasteful than those who retained their job.
I've worked as a rater before, working on doing the actual rating and also rating the raters. When you're a rater, you get very detailed instructions on what to do. When you're rating the raters, you're given the same instruction set so you have to rate them according to how well the raters followed instructions. I assume that someone else rates the raters who rates the raters, based on the same instruction following.
The job is easy and pays really well. If you want to keep the money flowing in, you simply follow directions. Standing out and being different gets you noticed, and not in a good way (because you're just a faceless rater id #667383737).
When I have asked claude to use proper vocabulary it wrote a memory "do not use blog-speak", so I guess the problem is that it is trained on blog articles much more that on good documentation.
And then it is also trained on it's our claude code sessions, so the more people accept claudish the more it becomes present in training data.
One of the challenges in reading AI output is that it outputs cross-discipline.
Load bearing (civil engineering), smoking gun (detective novels?), etc.
I use a skill file which states that it may internally use whichever terminology which fits, but that output presented to me must be phrased as if talking to a human, and within the terminology of the task at hand.
I don't specifically reference smoking gun, but its not been mentioned since.
Maybe the training could be.. please read this, and we'll test you in three months on how much of it you remembered.. or please read this and then later do a task relying on that information and well test to see how people did on the task..
Do you have a 'technical writers' assessment at how good AI writing is currently (and do you see it improving with new models or no)?
I'm not great at writing, but this, I think, is one of the main shortcomings for me. I cannot understand the output, to the point where I need to do extra work for anything non-trivial. It feels like it's TERRIBLE at it once a task surpasses a level of complexity / context, but I cannot quite explain why... (lists/headings/length do make sense though)
Structured and precise prompts help a lot, IMHO. Most models do definitely get "lost" even within their context window. I've found it handy to keep templates of structured prompts that include examples and explicit requirements. Of course YMMV...
I know the paper specifically calls out Opus 5.5 and Opus 5, but I can’t see different versions of Sol, or Astra.
I think the difficulty in comprehension was mostly an Anthropic problem… partly because Anthropic employees stopped reading Opus output around the time Fable came out and stopped noticing RLVR going off the rails with compression.
The paper was a good read, in any case. Maybe I gotta start sending this article back to LLMs when they start compressing text too much :)
Yeah, I've never used Claude, and generally use the codex CLI with whatever base prompt that implies, and I've found that it's always been very clear and concise without goofy "-isms." It just comes across as an intelligent colleague. It even makes effective use of bold and italics for emphasis. I'm a little mystified reading all of these comments.
Its because AI was not optimized for human reading, its optimized for the most amount of information in the lowest amount of context or tokens as possible.
(e.g., "baseline system state transition latency validation") instead of unpacking them with prepositions ("validating the latency of state transitions in the baseline system").
For me, I can't read books at all, because I forget what was said two paragraphs before. I bet people that are good at skimming text have less problems with AI generated text.
In my mother tongue "baseline system state transition latency validation" would have grammatically been one word. I sometimes miss that in English. If you're tired and see "baseline system state transition*s* latency validation", note the hallucinated "s", the meaning changes and you need to read it again because you're unsure if you actually did see an "s" or if it's "one thing".
"Baselinesystemstatetransitionlatencyvalidation" is surprisingly enough less confusing to me because I know it's one word describing one thing, and my eyes registers each joined word with (years of) training, even though what my eyes look at might be the following chunks in order:
*Baseline* systemstatetransitionlatencyvalidation
Baseline*systemsta*tetransitionlatencyvalidation
Baselinesystems*tatetransitio*nlatencyvalidation
Baselinesystemstatetransiti*onlatency*validation
Baselinesystemstatetransitionlaten*cyvalidation*
My eyes essentially scan the word, and although it might be a tad bid slower than reading "baseline system state transition latency validation", reading and understanding is faster because I already know it's one thing before I know which thing it is.
> For me, I can't read books at all, because I forget what was said two paragraphs before. I bet people that are good at skimming text have less problems with AI generated text.
Actually LLM generated text is almost impossible to skim. It's too flowery and the actual meaning is spread too thinly across the text, which means that you can't just read the "important" sentences of a paragraph.
LLMs are showing us that a lot of what we think of as cognition is actually part of the elaborate symbol system which is Language, as opposed to a property of "mind" (either biologically or artificially instantiated.)
LLM-generated text is hard to understand because while it's perfectly good language that follows valid cognitive and informational patterns, it does not reflect the intent or mental model of another mind.
> LLMs are showing us that a lot of what we think of as cognition is actually part of the elaborate symbol system which is Language
This is entirely speculative and highly disputed, even if you are doing the more sophisticated thing here and by Language you mean something like mentalese.
Much cognition is embodied and makes use of visual, tactile, auditory, kinaesthetic, and other sense modality representations. "Thought is mostly a Language" has been a wordcel kind of argument with little empirical or philosophical support for a couple of decades now, IMO. Language is how we communicate thought, not primarily how we think.
Not that this refutes your main point: LLMs are trained almost entirely on (written) language, and since this actually has little to do with real cognition, this makes them so annoying and alien to read.
I think we're in violent agreement here. Language is part of how we think. Definitely not the whole thing -- we are physically and biologically instantiated and LLMs are not. Language is a software, but the hardware does matter.
But LLMs DO show us that language is a big deal. It isn't just a serialization protocol for real thought -- it's a material component of how we think.
In many ways I think LLMs are the strongest possible proof of structuralist and post-structuralist theory which argues that human cognition is conditioned on the language and concepts to which we have access.
Most of the effort I put into writing is modeling the reader's mind. I never would have known this if someone hadn't invented a soulless writing machine that faked this process!
Maybe I'm the odd one out, but I really don't find AI generated language, in almost any model, to be hard to understand. I'm not trying to be snarky or annoying or pompous or whatever word might be "this guy over here think's he special."
It uses a lot of metaphors. It references stuff implicitly - a LOT - and assumes pretty much all the time that the thing it did is what you saw, which you didn't. But if you just.. read the stuff it says, and know a few old-school sayings and thesaurusize your reading style, it usually says things quite meaningfully.
I think a lot of people just don't read what it writes. So many folks I've watched used LLMs don't write in complete sentences, don't read the sentences that come back, don't watch the turns and tool calls in the middle, and then just respond to the message as if it says something it doesn't. I'm saying all this before I read the paper, but I've done a lot of personal and team experiments with this, so I'm quite curious to see what they say. I'll edit this after I go through it.
It argues that training for "math, code, and reasoning" caused this. Do LLMs only trained for natural language (probably older models) have the same characteristics?
Does training for these additional domains help in some ways, and this specific style of writing is just a side-effect of that?
LLM reinforcement learning improves those tasks by having the agents "reason" until they hit a verifiable output state that passes a black or white test. So they learn language which is tuned to the task in a silo, never communicating with anyone or anything.
In contrast, most human communication and writing is intended for other humans, and we constantly hit failures here that we learn to iterate on and improve. Humans who do math do so with the constraints that their math needs to be reviewed by another human and in the context that all their writing needs to be understood by another human.
The process causes a warped and compressed version of language.
Optimizing for contradicting objectives is always hard, this behavior will always leak this way or another. Even basic instruction training will affect writing produced piece by piece, because the structure of a typical assistant reply will leak into each piece, for example making characters use narrative echo to annoying degrees. This can be alleviated by using assistant prefill, and its removal from frontier models is a disaster.
Yes obviously, if you optimize for natural-sounding language it will learn to sound more natural.
This is an absolutely critical issue; it's impossible to work with AI that can't communicate coherently. It's barely usable for reasoning on complex issues because of the language problems described exactly in that synopsis.
152 comments
[ 5.4 ms ] story [ 2281 ms ] threadI regularly prompt it now to "summarize again without the jargon". That helps, for now, to bring it back to using plain language. Together with my basic prompt to reply using ASD-STE100 Simplified Technical English (https://en.wikipedia.org/wiki/Simplified_Technical_English)
So I tried it out, and it really sometimes works, and I've overcome some real obstacles that Tech Support was unable to assist me with. However, Gemini can also become "too big for its britches" and begin explaining glitches from a systems-programming perspective, as if it has deep knowledge of the Android codebase, and deep engineer's knowledge of the Pixel system hardware. And it will spit out absolute nonsense.
I have retorted to it a few times, "that sounds more like Star Trek than a sound appraisal of the subsystem" and then Gemini smirks and says "You're right to call me out on that!" and blithely continues on its way.
So yeah. Sometimes the LLM confabulations are deeply cloaked in highly technical jargon that could not even be questioned, unless you already know it's absolute bullshit. And then the LLM freely admits it! LLMs have no shame, that's for damn sure.
This is likely relative to my existing knowledge. Humans are also LLMs, and a human without prior training will be able to transfer-learn slop from a computer LLM, but a human with prior training may find the attempt at transfer-learning conflicts with what they have already learned, and reject it.
In less ML terms I think a human who reads slop and has no existing knowledge of the subject will just learn the slop by rote and confuse it for actual knowledge.
Dunno about any test, but I get the meaning behind the words in a way that I can take appropriate action (as someone using LLMs heavily for dev and always following the transcript so I can properly steer them). And often I see people here screaming "AI text" or similar regarding some linked article, and having read it I found that I just bypassed all the "tells" without realizing, because they didn't affect my absorption of the meaning of the article. Makes me think people are consciously looking for those tells so they can scream and shake their pitchforks.
They’re difficult for humans because if a sequence of words hasn’t previously been seen, you have to expend more effort to make sense of it.
Something like “customer onboarding workflow automation" might qualify - it makes sense, but it’s not a commonly used phrase. AIs use a lot of these jargony sequences.
If we had talking centipedes the results would probably be similar, but for different reasons.
Which gives a relative incentive to hard to understand text.
LLMs text generation is much closer to the way humans speak than to the way good writers write.
But I also wonder why AI loves to add slogans, catch phrases, and witty sentence fragments. It’s grating and a dead giveaway of slop.
The best approach is just to write the outline and initial text in a very messy fashion yourself. And then have it clean it up and add detail. And then edit again by hand.
I'm to the point where I don't trust it to do any writing. It's only allowed to tell me what it thinks I missed. Because inevitably when I ask it to edit my prose, it destroys the text and I basically need to rewrite it all again.
Or reading patents.
Terrible ... like someone put in effort to make it unreadable.
AI language, it seems, is optimized to support reasoning from the interior monologue.
The neat thing about LLMs is that in a way they proved that only the truth turns out to be grammatical (statistically), largely because there are so many more ways to be wrong than right that the mode is almost always right.
But people also lie and bullshit in consistent ways, with the intention of fooling people with well-known empty arguments. Part of being a "good writer" is to make the meaningless sound meaningful - and the techniques are ancient.
Train on a bunch of people trying to push their framework and get rich, whether in online garbage, literary garbage, or in ostensibly serious nonfiction or scientific works, and you'll end up with modal language that is trying to play deceptive games with reasoning, but has nothing in particular it's trying to convince you of (no motives.)
People would either fall for that language (if they weren't thinking much and the context was one that comforted them), or they would methodically try to put that language into a model (let's say a series of syllogisms for an argument about what to do or what to believe, but it could be any model), and see that it failed. And speaking again of the context, the context of humans interacting with human content implies motives that help the listener focus on particular parts of their model to look for rhetorical tricks. The LLM, again having no motive, just does this in random places.
The latest advances in AI have been in the tooling around AI, which I think largely reduces to recursively creating models and populating them in outputs short enough that hopefully the text generation won't go off the rails.
Symbolic AI is coming back, it's just that the mortar is random word association.
Call me crazy, but I think Georges Bataille saw all of this. In my interpretation, he simultaneously thought of random associations as the most important component of reason and also completely irrelevant in the reality of the world. The only way you could discover the structure of things is through random collisions; your view of the universe was entirely constructed by your prejudices, lusts, and fears; but whether that view was wrong or right was based on how it worked and the particular content of the id-garbage that got you there was ultimately irrelevant. The fabric of the world is completely id-less, but can only be digested by id-driven beings. LLMs (without the tooling around them) are all id.
I cannot imagine a typical suburban American consumer tolerating a chatbot as blunt and direct as Linus.
I think it's not the raw training corpus, but the samples collected specifically for reinforcement learning, direct preference optimization, or the other instruction-tuning steps that get a base model to turn into a chat/reasoning model. (We're working on testing this with the Allen AI models that have open training data.)
Yes, this is a problem.
AI-generated text is used internally by the AI to instruct agents, etc. And for feedback loops in reasoning. If you optimize for the result only, the AI will increasingly speak "AI", not human.
I mean, maybe that's all right for when it's producing code - code is not beautifully flowing prose in the first place.
My guess is years of RLHF optimized for rater's first impressions of how well-written something was. It uses lists! It has headings! It uses fun phrases like 'smoking gun'! It's long and informative! These things are relatively fast and easy to identify and select as a preference.
But none of this really covers whether the response is _useful for the audience_. If you're not the audience, it can be hard to tell what is and isn't useful.
Things that are easy for humans, like remembering a character is holding a gun during a conversation, gets forgotten, because there is no clear architectural/spacial understanding of how the words are describing an underlying reality. And if the LLM gets this right it might instead forget which character has a Southern drawl, or make an illogical metaphor.
This is all more clear with programming, but the problem is the same - a lack of interwoven semantic understanding and friction between the real world and the self-contained world of the LLM's context.
It also insisted that I start an hour early even though it was adding in 3 hours of rest time at the end.
Human judgment it is not
Chat: https://chatgpt.com/share/6add218a-3bc8-83ea-a2d7-613444eb7c...
This happens to me way too often. I just want a simple answer. I don't need a wall of text with 6 different options and a lengthy explanation. Just. Tell. Me. The. Answer.
I don't think I have ever seen an LLM try to determine what a human's frame of inquiry or related experience / skill level is, though. Might be my new favorite LLM shortcoming.
Not sure how you think those things aren't different
Of course it supports 64-bit Windows - every Unreal plugin does! They don’t even make 32-bit Windows anymore!
One is bombarded with facts and struggle to get what the text is actually saying.
I have tried getting the LLM (Opus 4.8 or some 5.x and some other model) to write more human like, and also analyze its own text. It was hilarious, it could describe the problem quite well, but was helpless to actually do anything about it. "Oops I did it again, sorry!" LOL
And absolutely we know models themselves get bogged down with too much extraneous reducing accuracy.
This is reflected in classifier test suites.
Didn't Dario exclaim earlier in the year getting an LLM to simplify a design was like pulling teeth?
(Sorry. I blame Claude.)
https://github.com/juliusbrussee/caveman
The job is easy and pays really well. If you want to keep the money flowing in, you simply follow directions. Standing out and being different gets you noticed, and not in a good way (because you're just a faceless rater id #667383737).
And then it is also trained on it's our claude code sessions, so the more people accept claudish the more it becomes present in training data.
Load bearing (civil engineering), smoking gun (detective novels?), etc.
I use a skill file which states that it may internally use whichever terminology which fits, but that output presented to me must be phrased as if talking to a human, and within the terminology of the task at hand.
I don't specifically reference smoking gun, but its not been mentioned since.
I'm not great at writing, but this, I think, is one of the main shortcomings for me. I cannot understand the output, to the point where I need to do extra work for anything non-trivial. It feels like it's TERRIBLE at it once a task surpasses a level of complexity / context, but I cannot quite explain why... (lists/headings/length do make sense though)
So there's the usual structure of "it's not x it's actually y" signs.
I think the difficulty in comprehension was mostly an Anthropic problem… partly because Anthropic employees stopped reading Opus output around the time Fable came out and stopped noticing RLVR going off the rails with compression.
The paper was a good read, in any case. Maybe I gotta start sending this article back to LLMs when they start compressing text too much :)
(e.g., "baseline system state transition latency validation") instead of unpacking them with prepositions ("validating the latency of state transitions in the baseline system").
For me, I can't read books at all, because I forget what was said two paragraphs before. I bet people that are good at skimming text have less problems with AI generated text.
Is it? I mean they get payed more if they return longer answers.
"Baselinesystemstatetransitionlatencyvalidation" is surprisingly enough less confusing to me because I know it's one word describing one thing, and my eyes registers each joined word with (years of) training, even though what my eyes look at might be the following chunks in order:
*Baseline* systemstatetransitionlatencyvalidation
Baseline*systemsta*tetransitionlatencyvalidation
Baselinesystems*tatetransitio*nlatencyvalidation
Baselinesystemstatetransiti*onlatency*validation
Baselinesystemstatetransitionlaten*cyvalidation*
My eyes essentially scan the word, and although it might be a tad bid slower than reading "baseline system state transition latency validation", reading and understanding is faster because I already know it's one thing before I know which thing it is.
Actually LLM generated text is almost impossible to skim. It's too flowery and the actual meaning is spread too thinly across the text, which means that you can't just read the "important" sentences of a paragraph.
But also: I am bad at skimming, better at close reading, and LLM text is unreadable for me as well.
LLM-generated text is hard to understand because while it's perfectly good language that follows valid cognitive and informational patterns, it does not reflect the intent or mental model of another mind.
This is entirely speculative and highly disputed, even if you are doing the more sophisticated thing here and by Language you mean something like mentalese.
Much cognition is embodied and makes use of visual, tactile, auditory, kinaesthetic, and other sense modality representations. "Thought is mostly a Language" has been a wordcel kind of argument with little empirical or philosophical support for a couple of decades now, IMO. Language is how we communicate thought, not primarily how we think.
Not that this refutes your main point: LLMs are trained almost entirely on (written) language, and since this actually has little to do with real cognition, this makes them so annoying and alien to read.
But LLMs DO show us that language is a big deal. It isn't just a serialization protocol for real thought -- it's a material component of how we think.
In many ways I think LLMs are the strongest possible proof of structuralist and post-structuralist theory which argues that human cognition is conditioned on the language and concepts to which we have access.
It uses a lot of metaphors. It references stuff implicitly - a LOT - and assumes pretty much all the time that the thing it did is what you saw, which you didn't. But if you just.. read the stuff it says, and know a few old-school sayings and thesaurusize your reading style, it usually says things quite meaningfully.
I think a lot of people just don't read what it writes. So many folks I've watched used LLMs don't write in complete sentences, don't read the sentences that come back, don't watch the turns and tool calls in the middle, and then just respond to the message as if it says something it doesn't. I'm saying all this before I read the paper, but I've done a lot of personal and team experiments with this, so I'm quite curious to see what they say. I'll edit this after I go through it.
In contrast, most human communication and writing is intended for other humans, and we constantly hit failures here that we learn to iterate on and improve. Humans who do math do so with the constraints that their math needs to be reviewed by another human and in the context that all their writing needs to be understood by another human.
The process causes a warped and compressed version of language.
Yes obviously, if you optimize for natural-sounding language it will learn to sound more natural.