That is actually what I'm firmly convinced is the most dangerous thing about llm's. No matter what you put, it will always agree with you, and what's worse, it will try to make you think that you're unbelievably smart for saying x.
I used one to help me plan a sales route, and it kept fucking it up. Every time I corrected it, it tried that hand wringing vizier sort of ass kissing. It's very off-putting, but I can see how someone struggling with social interaction could be sucked into that nonsense.
I like that these AI idioms exist. They're like watermarks for text. It's worth the cost of humans avoiding them. Companies will eventually train their models to be undetectable, but society would be better if they didn't.
Except that the entire point of the article is that they're not AI idioms. They're not "watermarks for text." They're legitimate language constructions that LLMs tend to overuse, but that real humans also use. Real humans do, in fact, say "align with" all the time, just as often as "corresponds."
And you can pry my em dashes from my cold, dead hands.
I agree with the feeling. But if you agree with the analysis of the article, this cat & mouse game ultimately amounts to stop disclosing our reasoning threads through commonly accepted linguistic structures. That's quite a price to pay as a society...
Humans are just trying to do what Pangram is trying to do: guess what is AI, badly. The post argues against this:
> In the end, shaming people for writing that gets flagged as AI can lead people to sidestep structures the model has learned from us: structures that are effective tools for argumentation. We take the tools of critical thinking out of the kit at the time we most need them.
I don't think so at all. Models are trained in many ways and are changing aggressively, resulting in different patterns in different regions, domains, languages, and will be different 3, 5, 10 years down the line. Having everyone try to learn and adapt around how to stay within very magical, fuzzy, and ever-changing boundaries to avoid appearing to be an AI, instead of focusing on producing good writing or communicating as it is natural to them, seems like a recipe for bad thinking and arbitrary reactions.
No, fuck that. I'm not going to think twice about what I write just to avoid an AI checker, and I will delve into em dashes with gusto if that's what the writing calls for.
I'm not sacrificing the language simply to sound less like AI--that's absolutely a losing game.
And if anyone thinks my hand-crafted prose is AI-generated, they're free to look elsewhere. Right now AI detectors peg my pre-AI work as 30% AI-generated, and I'm certain that number will only increase as LLMs improve.
This is how early forms of "reasoning" in LLMs worked: just literally inserting words like "Wait...", "Hmm...", "Let me reconsider...", "But is it really..." into the token stream.
Is this not how current forms of reasoning work? It seems like the open models still output things like that, and the closed ones all just summarize their thinking instead to avoid distillation, but probably do the same thing internally.
> In the end, shaming people for writing that gets flagged as AI can lead people to sidestep structures the model has learned from us
It's interesting why LLMs generate constructions like this more frequently than they presumably exist in the training set. I wonder if this is some sort of mode collapse caused by post training, and/or maybe because they are training on synthetic data so these things become self-perpetuating and self-amplifying (a feedback loop)?
The lesson for humans worried about being falsely identified as AI is just learn to write better! It doesn't matter where your repertoire of phrasing comes from (copying AI or not), but one of the basic rules of writing is not to repeat yourself unless you are doing so deliberately for a purpose. Go ahead and use "It's not just X. It's Y" if you want to, but if you use it multiple times in the same short piece of writing, then you may deserve to be called out for poor style, if not for being an AI.
Its not model collapse nor does it have anything to do with training data frequency. It's simply RLHF where the humans hired to tune the conversational style of these LLMs preferred certain idioms over others and so the reward function for these LLMs gravitated toward using them.
If LLMs generated text based on training data frequency they'd likely be some of the most vulgar and hostile things ever created. The internet is full of insults, profanity, and low effort content. The repeated phrases are a side effect of reward optimization rather than some kind of model collapse.
nice article, but i think as a non native english speaker, i always use the model in english for reasoning and then translate the output to my language. most of these considerations do not apply. because the translation step is taking out alot of these language artifacts
"So, if we publicly shame people whose text looks like it might have been written by a machine – because it mimics the language used for human reasoning – and people stop writing in ways that they internalize as "AI writing" out of fear of false detection, it sends a signal that your language for reasoning must be policed, or you too could be held up to public scrutiny."
This is honestly both terrifying and well articulated.
> Because if Pangram's AI system found me guilty, that's the end of my career. That's literally extortion.
How is this different from humans? When I went to high school, my teachers extorted me too. Especially subjects like English and unlike Math, where evaluation is 100% subjective.
Surely these leading tells will be trained out of models pretty soon, given how well known and overused they are. And it might make the writing slightly worse in a way. But it is quite annoying how often this type of construction is used in everything at the moment.
I think that the current models are still like over-achieving savants rather than true human level because the largest model is only 1/10th the complexity of the human brain. I've recently become fairly convinced that new hardware paradigms (like types of CIM) are about to move from research into real-world development and scaling. So I believe within a few years, the model sizes will increase by another 10 times.
Compared to upcoming 100 trillion parameter models, humans will obviously be _much_ dumber/slower than AI in all fields. Already with the 10T models, some LLMs beat 99.9% of humans in competitive programming.
The AI hatred from many may actually continue to increase, but in cases where the bottom line matters, we are rapidly approaching the point where writing or work product that looks like it is human-authored will be suspect just on that basis. In other words, for some people it will be the reverse -- "this work looks like it was created by a human" could be devastating for your businesses credibility at that point.
In one of the essays posted here, which was, ironically, about AI in education, a sentence, that an AI could not possibly write, that I could possibly write, because of its length and unusual structure, before finally reaching the verb, went on for 25 words.
I don't know if it was written that way to show trust in the reader's intelligence, show disregard for reaching a wide audience, show a demonstration of skill, or was artifact of someone just thinking at that level.
> There is danger in evaluating for language patterns over its content
I agree, but it’s worth noting that that has been done since long before LLMs. Fifteen years ago, I used to teach a graduate course on academic writing pedagogy. The students and I would read research papers on the teaching of academic writing; we also analyzed textbooks and course syllabuses to get an idea about what was actually being done in classrooms. While phrases like “critical thinking” did come up, the overall focus was clearly on language patterns: sentence and paragraph structure, the use of transition words, vocabulary for hedging and boosting (i.e., making assertions seem weaker or stronger), etc.
In a university context, it can be very difficult to evaluate student writing based on its content. In humanities-focused and creative writing, what the student decides to say can be seen as an extension of the student’s personality, identity, and individual experience; if a teacher evaluates the content, including the reasoning, it can seem that the teacher is evaluating the student as a person. And if the students are in the sciences, especially at the graduate level, the writing teacher often won’t even understand what the students write because it is too technical. Teaching and evaluating language patterns, not content, is often the only option.
I liked everything in this post, with one exception. I'm less sure that avoiding speaking like an AI is robbing us of language useful in critical thinking. I'm far more worried about people offloading their critical thinking to AI systems and losing the habit.
Also, the Greeks were worried about rhetoric and, in my opinion, rightly so. The skill to argue a point well is different than those that are needed to be correct. To become a skilled rhetoritician was viewed as dangerous (and right now AIs are only moderately good... though they are improving fast).
I'm not dropping emdashes -- though you can always tell mine by their two-hyphen form lol
I've also never used an AI detector, and probably never will.
In my experience:
1. The people who rely the most on AI writing don't like to admit it. I catch obvious AI hallucinations in my boss's "documentation," and he always insists it was his own human oversight, despite it being very obviously a mistake I've caught Claude (and importantly, no human coworker) making repeatedly
2. I don't trust a machine more than myself to judge writing
3. Obvious AI "tells" just make it clear i don't need to keep reading, not that i need some kind of validation. In some sense, i guess that might save me time? But i still have to have read enough to know what it is...
In general, i think the author makes great points about how _LLM "thinking" is just the reproduction of the language of reasoning_, that is not necessarily a replacement for actual reasoning. It'll take a lot more than that for me to believe an AI is "thinking" and not just giving statistically reasonable answers (reasonable or actionable though they may be)
An old xkcd comic that is somewhat related to the current witch hunt that some text that the author claims he wrote himself was actually written by an AI:
TLDR - it's not just AI detection. It's policing of human thought.
Anyway, yeah, people trusting AI to do a better job in reasoning than fellow humans, without justification, worries me. We have no formal theory of informal reasoning (that LLMs mimic), so we cannot verify it any better than with humans.
You have to trust someone, to ground your beliefs. Trusting AI is just trusting some other people (who trained it) by proxy. Once you realize it, you might as well try to trust people you know.
48 comments
[ 3.0 ms ] story [ 47.1 ms ] threadThis feels like an easy enough hypothesis to verify, for anyone in the business of training LLMs - does the not-X-but-Y rate increase after RLVR?
I used one to help me plan a sales route, and it kept fucking it up. Every time I corrected it, it tried that hand wringing vizier sort of ass kissing. It's very off-putting, but I can see how someone struggling with social interaction could be sucked into that nonsense.
- "No X, No Y, No Z." pattern
- "Here is X - it makes Y"
The worst and most obvious one is the constant over use of emoji ticks and crosses.
And you can pry my em dashes from my cold, dead hands.
That's really unfortunate though. It's like Michael Bolton from Office Space: "No way! Why should I change? He's the one who sucks."
> In the end, shaming people for writing that gets flagged as AI can lead people to sidestep structures the model has learned from us: structures that are effective tools for argumentation. We take the tools of critical thinking out of the kit at the time we most need them.
It's not X, it's Y, though? Couldn't be me.
No, fuck that. I'm not going to think twice about what I write just to avoid an AI checker, and I will delve into em dashes with gusto if that's what the writing calls for.
I'm not sacrificing the language simply to sound less like AI--that's absolutely a losing game.
And if anyone thinks my hand-crafted prose is AI-generated, they're free to look elsewhere. Right now AI detectors peg my pre-AI work as 30% AI-generated, and I'm certain that number will only increase as LLMs improve.
It's interesting why LLMs generate constructions like this more frequently than they presumably exist in the training set. I wonder if this is some sort of mode collapse caused by post training, and/or maybe because they are training on synthetic data so these things become self-perpetuating and self-amplifying (a feedback loop)?
The lesson for humans worried about being falsely identified as AI is just learn to write better! It doesn't matter where your repertoire of phrasing comes from (copying AI or not), but one of the basic rules of writing is not to repeat yourself unless you are doing so deliberately for a purpose. Go ahead and use "It's not just X. It's Y" if you want to, but if you use it multiple times in the same short piece of writing, then you may deserve to be called out for poor style, if not for being an AI.
If LLMs generated text based on training data frequency they'd likely be some of the most vulgar and hostile things ever created. The internet is full of insults, profanity, and low effort content. The repeated phrases are a side effect of reward optimization rather than some kind of model collapse.
Sometimes it’s not just about the Ys but also the Qs.
It is bad writing.
This is honestly both terrifying and well articulated.
High praise to the blog author.
How is this different from humans? When I went to high school, my teachers extorted me too. Especially subjects like English and unlike Math, where evaluation is 100% subjective.
I think that the current models are still like over-achieving savants rather than true human level because the largest model is only 1/10th the complexity of the human brain. I've recently become fairly convinced that new hardware paradigms (like types of CIM) are about to move from research into real-world development and scaling. So I believe within a few years, the model sizes will increase by another 10 times.
Compared to upcoming 100 trillion parameter models, humans will obviously be _much_ dumber/slower than AI in all fields. Already with the 10T models, some LLMs beat 99.9% of humans in competitive programming.
The AI hatred from many may actually continue to increase, but in cases where the bottom line matters, we are rapidly approaching the point where writing or work product that looks like it is human-authored will be suspect just on that basis. In other words, for some people it will be the reverse -- "this work looks like it was created by a human" could be devastating for your businesses credibility at that point.
I don't know if it was written that way to show trust in the reader's intelligence, show disregard for reaching a wide audience, show a demonstration of skill, or was artifact of someone just thinking at that level.
I agree, but it’s worth noting that that has been done since long before LLMs. Fifteen years ago, I used to teach a graduate course on academic writing pedagogy. The students and I would read research papers on the teaching of academic writing; we also analyzed textbooks and course syllabuses to get an idea about what was actually being done in classrooms. While phrases like “critical thinking” did come up, the overall focus was clearly on language patterns: sentence and paragraph structure, the use of transition words, vocabulary for hedging and boosting (i.e., making assertions seem weaker or stronger), etc.
In a university context, it can be very difficult to evaluate student writing based on its content. In humanities-focused and creative writing, what the student decides to say can be seen as an extension of the student’s personality, identity, and individual experience; if a teacher evaluates the content, including the reasoning, it can seem that the teacher is evaluating the student as a person. And if the students are in the sciences, especially at the graduate level, the writing teacher often won’t even understand what the students write because it is too technical. Teaching and evaluating language patterns, not content, is often the only option.
Also, the Greeks were worried about rhetoric and, in my opinion, rightly so. The skill to argue a point well is different than those that are needed to be correct. To become a skilled rhetoritician was viewed as dangerous (and right now AIs are only moderately good... though they are improving fast).
This is merely the end-state of industrialization, which is efficient and soulless.
https://youtu.be/1Pr8xnNi7OM
I've also never used an AI detector, and probably never will.
In my experience:
1. The people who rely the most on AI writing don't like to admit it. I catch obvious AI hallucinations in my boss's "documentation," and he always insists it was his own human oversight, despite it being very obviously a mistake I've caught Claude (and importantly, no human coworker) making repeatedly
2. I don't trust a machine more than myself to judge writing
3. Obvious AI "tells" just make it clear i don't need to keep reading, not that i need some kind of validation. In some sense, i guess that might save me time? But i still have to have read enough to know what it is...
In general, i think the author makes great points about how _LLM "thinking" is just the reproduction of the language of reasoning_, that is not necessarily a replacement for actual reasoning. It'll take a lot more than that for me to believe an AI is "thinking" and not just giving statistically reasonable answers (reasonable or actionable though they may be)
Anyway, yeah, people trusting AI to do a better job in reasoning than fellow humans, without justification, worries me. We have no formal theory of informal reasoning (that LLMs mimic), so we cannot verify it any better than with humans.
You have to trust someone, to ground your beliefs. Trusting AI is just trusting some other people (who trained it) by proxy. Once you realize it, you might as well try to trust people you know.