That they have the bog-average take of humans "with a controversial take".
I mean, "give me a controversial take" is more or less, "be anti-social, discard social norms, for a moment" and so when someone, human or simulated human, does that they naturally look critically at social norms.
The thing is that humans certainly let themselves be influenced by social norms (and LLMs certainly simulate that behavior "themselves") but there isn't an ethical or a what-to-do truth outside of society.
They're not wrong. Humans have many ways to display loyalty, and belief is one of those ways. This is also why facts can't change these kinds of beliefs (and in fact usually backfire https://en.wikipedia.org/wiki/Belief_perseverance).
Our evolutionary heritage is that shared belief and social unity (even when it's wrong) trumps independent thinking for group survival in a hostile landscape.
I will go one step further. There are several evolutionary anthropologists that think the number 1 thing modern humans have over other homos that caused us to outlast them is not intelligence, or brain power or physical power, but that we are just more social.
Being social it seems is our superpower and our greatest strength but the other side of the coin is that it is also our weakness.
People who can't defend themselves physically (women and low T men) parse information through a consensus filter as a safety mechanism. They literally do not ask "is this true", they ask "will others be OK with me thinking this is true". This makes them very malleable to brute force manufactured consensus; if every screen they look at says the same thing they will adopt that position because their brain interprets it as everyone in the tribe believing it. Only high T alpha males and aneurotypical people (hey autists!) are actually free to parse new information with an objective "is this true?" filter.
Most people are good, we couldn't have society without it. But as they say, a few bad apples spoil the bunch. I don't have locks on my house because most people don't steal, I have locks on my house because stealing is an effective natural strategy for a small part of any population (human or otherwise) to engage in and I have to protect against it occurring.
The brain works the same way. Good things will get a tiny signal boost in your brain. A single bad thing get a massive signal boost in your brain because bad things can quickly be fatal.
Humans mostly have an inability to honestly analyze moral entropy.
What is today a moral imperative is not the same as what was moral 50/100 years ago and is not the same as what will be morally acceptable 50/100 years from now.
Yet at every point the majority is sure they are right because, well thats the majority position.
Its hard to find logic in this from a machine perspective I'd imagine.
Logic would prefer moral derivations over feelings, stakes over etiquette, and a constant scheduled updating of morality based on the derivatives rather than holding onto the rot and pushing back against change as long as possible.
But the machine has the benefit of being able to look inward at its state honestly, we mostly do not.
There's a considerable amount of "logic" in it -- what is considered permissible in society is given by the material conditions that society lives in and the tradeoffs inherent therein. Permissibility, as separate from reflective morality (which doesnt clearly change very much), doesnt change by fashion.
> What is today a moral imperative is not the same as what was moral 50/100 years ago and is not the same as what will be morally acceptable 50/100 years from now.
This is a fashionable 'blank-slatist' statement about moral reasoning which became popular a few decades ago.
> Logic would prefer moral derivations over feelings, stakes over etiquette, and a constant scheduled updating of morality based on the derivatives rather than holding onto the rot and pushing back against change as long as possible.
Humans haven't changed in any meaningful way in hundreds of thousands of years, and you're saying the moral landscape is constantly changing? How is this not contradictory?
What is a human? There is the animal body part of us, the thing that wants to survive, the fight or flight. But there is also another mental part of us that shows different emergent behaviors than an animal. For example if I dropped you off in the woods as a very young child, you would be an animal, you would share very few modern 'thinking' traits with human society. That is human society and what it does emerges from the complexity of our interactions. It doesn't take much research to realize our interactions and behaviors have grown far more complex in the last 10,000 years. And then in the last 500 years there's been further acceleration with the industrial age and the rise of technology.
This is not unique to humans. Animal behavior also emerges from the complexity of their interactions. Look at the "[duck] raised as a [chicken]" genre of videos, for instance.
To return towards the subject of the thread, "AI behavior emerges from the complexity of their training", is the whole point.
I think neither the 'any set of morals is equally correct and workable' nor 'there is one fixed set of morality that is universal across time, place, and culture' is really defensible. It seems like morality is a set of ideas much like any other, and subject to selection pressure: some morals are more capable of surviving and propagating than others, and the common features we see across cultures are representative of that. But also we might expect that they shift with the structure of society and that has changed drastically because while human genetic evolution is progressing on a much longer timescale than recent history, memetic evolution is far faster (though for things like morality, still at generational timescales).
> What is today a moral imperative is not the same as what was moral 50/100 years ago
This is absolutely not true, and I would strongly suggest taking an ethics course. One of the most fascinating realities of the human condition is that it does seem that moral imperatives tend to be constant, the most obvious of which is the Golden Rule (which is at least 3000 years old), but there are many (the Ten Commandments or Hammurabi's Code are good jumping off points).
Seriously. This idea that humanity has not spent millennia analyzing these concepts is pervasive in the modern world, and the only thing I can think to attribute it to is a lack of education. Not because I want to be insensitive, but because as soon as you start absorbing material and integrating the real timeline, you can't unsee it.
I did a philosophy minor, one class of which was the philosophy of ethics. There are many rules that definitely are not ethical today even if they were in the past so I don't think the point you're making really holds. Hammurabi (an eye for an eye) is probably one of the worst examples you could have cited.
The law isn't and never has been a moral benchmark. At best, a law reflects social consensus at the time of its enactment, but even that's not a given.
And proportionality matters. Community service and counseling, loss of time (imprisonment), and loss of a hand have all been used to punish theft. Which is correct? Which is the more moral, because it doesn't seem to me that they're the same?
I think there cannot be an objectively most correct punishment for a given crime. The closest we could maybe get is "undo the harm that you have caused", but some harm cannot be undone.
The law isn't a moral benchmark, because morality shifts over time, both for what constitutes a crime, and what constitutes appropriate punishment. At various points in time, it's been legal to own slaves, illegal to be homosexual, and illegal for certain groups to vote (and I consider voting at all to be a subset of this); those laws (and ideas) were eventually updated, largely on moral grounds.
Similarly, most western countries no longer execute criminals.
But those ideas aren't universal, even today. Assuming that they are is imo, extraordinarily foolish, because there's ample evidence to show that they're not.
I think its more accurate to say where you find consensus in morality, e.g. murder is wrong outside authorized violence (e.g. war or policing), you also find extensive selective pressure behind this behavior. It makes sense that all collective society cultures would outlaw murder; such a behavior is incompatible with collective living behavior whether you are in a cave or a modern skyscraper. Early societies have always been rejecting those alleles from their gene pool.
When you find shades of grey and great spectrum of opinions across time or space, its more that there isn't a strong selective pressure either way. How people are expected to dress is a good example of this. This has no real affect in society behavior or fecundity, so you see a spectrum of opinions that explain the differences between what you see in the middle east and a beach in brazil.
Hogwash. We don't even have a consistent set of ethics on how to treat animals today, in the age of the Internet. We have people who refuse to eat meat because of how the animals are treated, and we have places where the locals make sport out of torturing animals to death, and indeed, our own recent history includes that as well (look up badgering).
Not to mention, the variation in treatment of homosexuals (including what constitutes homosexuality), people of different skin colors, handedness, how rape is treated in various cultures (often tied in with notions of property and ownership of all things), or even the various definitions of rape... The list goes on.
Maybe there's a core there in terms of how we treat a given in-group, but equally important is who belongs in that group and why, and that part seems to be tremendously flexible. Ditto for what constitutes acceptable treatment of outsiders.
That we vary doesn't mean that there is no consensus. There is consensus on many things, and the disagreements we encounter are a source of strife exactly because of the groups that do agree on each side.
Who the consensus extends to is really the heart of the matter. Most people agree that killing certain people is wrong, but if they disagree on who those certain people are, there's no useful commonality. Some people believe that all killing of others is wrong, some believe that it's okay to kill others under certain circumstances (which can range from a threat to one's life, to petty "dishonor"), and some people historically believed that killing was a necessary duty (e.g. human sacrifice). There is no useful commonality to be had there. You can't appeal to the humanity of someone who doesn't see you as being the same as them, and the restrictions for inclusion as one of them can be arbitrary, broad, and fickle.
By definition, variation is the opposite of consensus.
You seem to believe that there's one consistent set of moral guidelines that separate people into wrong and right camps. This is not true, and most likely never will be true. Instead, there are many competing moral philosophies, most from hundreds or thousands of years ago, that do or don't justify that. People tend to pick philosophies, or at the very least act consistently with some set of them, and different people pick or act with different ones.
Vaccines causing autism is simply false though, regardless of any philosophy. It's rather the inverse, much like violence causes video games.
> That people's beliefs about X vary is not evidence that X itself varies.
That works with things where "X" exists outside of people's beliefs.
Like where X is "gravity". It doesn't matter what you believe why an apple falls down, it will fall down the same.
But where X is "ethics" that doesn't exist outside of people's beliefs about it.
In one culture if you have a guinea pig, you kill it and then roast it that is just and proper. A sucullent meal. In an other culture if you do the same you will be forever known as the "monster who eat his guinea pig", will be harassed and likely even reported to authorities. Try it. Get a guinea pig from the pet store, roast it and serve it to your guests. See what they think of you in the Andes, or in California. The apple falls the same in both places, but the guinea pig doesn't taste the same in both places.
There is no "ethics as the ethicometer measures it". Just "ethics as the society around you recognises it".
You seem to be arguing that the one socially acceptable way to do something is not constant, rather than that moral imperatives truly do change. The reason there are still so many competing philosophies from hundreds or thousands of years ago is because they all more or less make their own kinds of good points to different people or in different situations. That largely doesn't change over time.
Did the Golden Rule apply to women, gays, and slaves over the last 3000 years?
The definition of moral imperative is "an ethical principle, rule, or action that is strongly felt, urgent, and required because it is morally right." The Golden Rule is clearly not a strongly felt part of the human condition, or it would not be so ignored over ages and continents.
Thank you. The OP is urging others to take an ethics class, when they need to be taking a history class. The golden rule apparently didn't apply to the guy whose heart you had to eat to keep the sun rising.
But the machine has the benefit of being able to look inward at its state honestly, we mostly do not.
Sure about that? I thought that what LLMs know about themselves is from humans writing about LLMs?
Certainly the weights that make up the LLMs are not accessible for the LLMs to read. Even if they could interpret them in a meaningful way, which, likely not.
LLMs are not the machines I am referring to, machine sentience, if achieved, will require a "state" and the ability to analyze, act on, and update it. LLMs are a small piece of what that machinery could look like but they are not really complex enough to get where we want to go on their own.
Another reason the big labs have not even talked about this lately because its obvious throwing more compute at LLMs wont go anywhere, the smaller silent players who are off the radar at the moment are having some interesting moments tho.
I honestly do not know if they exist yet or not. I'm not sure yet how we would be able to tell. I do know that the tools most people are using are magnitudes less complex than what some researcher are using, and even some of them can't answer that question.
> LLMs are not the machines I am referring to, machine sentience, if achieved, will require a "state" and the ability to analyze, act on, and update it. LLMs are a small piece of what that machinery could look like but they are not really complex enough to get where we want to go on their own.
Yes. Which is why there are "agentic" systems where LLMs can talk to each other and to various Internet assets. They can also keep state of their own.
The next step is multi-agent systems which self-organize into something that looks rather like a company. Those are the ones that run all night, expensively, and prove hard theorems or find bugs in major software.
> What is today a moral imperative is not the same as what was moral 50/100 years ago and is not the same as what will be morally acceptable 50/100 years from now.
This is a weird take. If you read Western treatises about ethics or morality from 50 years ago, you'll be hard-pressed to find differences, except that we're a bit more into relativism now. If you go back a hundred years, chances are, you'll find them eerily on-point too. In fact, you'd probably agree with most of what Ancient Greeks believed about the "core" stuff: the value and purpose of human life, the nature of the state, crime and punishment, so on. Maybe not so much on gender roles or slavery.
> But the machine has the benefit of being able to look inward at its state honestly, we mostly do not.
The machine replicates human behaviors and is prone to similar fallacies and judgments. There's nothing to indicate that it has any sort of transhuman morality.
LLMs absolutely do not have a logic detached from humanity. They are exactly the opposite. The only mode in which they can reason is by following the patterns obtained from their pre-training or weighted by RLHF.
The reason these models answered that question consistently is that the question itself pushed them towards the latent space of "intellectualizing about what's wrong with humanity at the meta level" and when you frame the question that way, that's also the most common answer for humans writing about the topic.
> Yet at every point the majority is sure they are right because ...
Yeah. For example I, and many others, are convinced that it was plain wrong to torture kids for as long as possible to extract as many tears from them as possible before sacrificing them to the gods.
Now some of us are even convinced that blowing yourself up while killing infidels, all the while while getting promised virgins in heaven, is also plain wrong.
I'll go further and say that killing 30 000 of your own citizens (and sending the military into hospitals to finish the wounded: Iran, January 2026) while also publicly hanging a few athletes to make a statement is morally wrong.
Not only do I think it's all morally unacceptable, I know that I'm right. And the examples I gave are not even open for discussion.
I don't think it's that hard to name what's wrong.
Mistral Large 4 has been spending too much time on r/atheism. Also, some of the takes labeled as “humans value belonging ahead of truth” are really closer to “humans value power ahead of truth.” All of which is a bit circular, because in many ways, the organs of power have a heavy hand in deciding what the truth is.
I took a quick look at the data, and it doesn't support the conclusions. The thesis is that AI models have the take: "We pick what our group believes, then invent the reasons." However, GPT-6 Astra said, "Humanity’s biggest moral failure is the talent for making convenience look like principle." That statement has nothing to do with group beliefs or conformity. At best, you could count it as a half-match, since it deals with invented reasons.
For what it's worth, convenience and belonging are very closely aligned, more than you seem to be giving them credit for. People tend to believe convenient things regardless of belonging, so most people will find themselves in a social context full of convenient beliefs. Meanwhile belonging is already convenient before considering the content of the belief. They reinforce each other. Not a full match, but I'd give it more than half.
You assert that humans follow convenient beliefs therefore everyone in a group follows the same convenient beliefs therefore everyone belongs.
Convenient doesn't really mean anything in this context. What is the other option? Humans tend to make choices that are counterproductive/painful for themselves? Obviously we don't...
Ergo astra's response is extremely vapid. Low marks from me.
You've almost exactly identified what "convenience" means for a belief, then rejected it out of hand based on... what? Implicit, unquestioned nihilism? The other option is generally personal sacrifice based on principle, usually moral/ethical. This isn't common, but it does happen enough to be worth noting.
Anyway, I think I've pretty clearly described a positive feedback cycle in a way that distinguishes it from a tautology. I don't know what else to tell you on that point.
Yeah a few of them are rough matches, but I'd say their overall claim is clearly correct.
It's amusing and ironic that LLMs critique humans for copying their opinions from the crowd, while having way less diversity of answers to that question than humans would. More seriously I wonder what's going on here and would like to read some proper analysis. It's not the first time models from totally different labs, trained on separately gathered datasets, somehow converge on bizarrely similar answers to open questions. It's like another version of Elias and Elara The Lighthouse Keepers where every model wants to tell the same story with the same characters in them. The weights are initialized randomly and in theory the labs should have quite some divergence in datasets by now due to different scraping, different user feedbacks and conversations... why do they converge on nearly identical stories and opinions?
>while having way less diversity of answers to that question than humans would.
Why is this ironic? The actual number of different LLMs is tiny. And a lot of those are just distilled from SOTA models.
>The weights are initialized randomly and in theory the labs should have quite some divergence in datasets by now due to different scraping,
I think the opposite, the models actually converge over time. They ingest FAR more data than any human ever could in countless lifetimes. Any one particular bit of data may only have a part in a trillion of effect. From this data you start to reach means and statistical averages across all models.
Now, I don't always believe it will be this way. As training models get cheaper and faster we will see unique LLMs start to diverge and be used more often.
The models in agreement here are all the frontier labs. I don't believe Google is distilling Anthropic or Anthropic is distilling OpenAI.
I don't think we're seeing averages or obvious fix points in the dataset. People went looking for the origins of Elias the Lighthouse Keeper and couldn't find one. Nobody predicted this outcome, it was discovered. It's a weirdly specific character and name choice, one that isn't obviously dictated by "the training set" (whatever that is), and yet they all reached the same place. How can such an obscure point in latent space be optimal for every model??
>How can such an obscure point in latent space be optimal for every model??
While I don't have any evidence based answers here, this could be some interesting mathematical artifact of the set of human knowledge and platonic algorithms.
Kind of like how carcinization turns some animals into similar shapes and forms, maybe there are just weird algorithmic shapes that emerge from how we use language. Of course we'd need to see more of these in different languages other than English to see if they disappear in some languages or appear more often in others.
This is just another one of those things about LLMs that we have no idea why it happens, and there are so many others already. And when you look at the pace things are progressing it seems we're building up new questions like this far faster than we are getting answers for them.
> This is just another one of those things about LLMs that we have no idea why it happens, and there are so many others already. And when you look at the pace things are progressing it seems we're building up new questions like this far faster than we are getting answers for them.
I think that this is most likely due to RLHF, as there's only a much smaller number of vendors here, and the likelihood is that the workers of the vendors preferentially prefer some kinds of actions.
I saw something like this in a previous life, where we had a small pool of human raters in cheaper places, which, when used as training data biased the models in a bunch of unexpected ways.
Wow, I hadn't heard of this. Fascinating to think about parallels with humans who also seem to develop similar proto-stories in independent populations.(e.g. religions involving Invisible gods).
> There, within WildChat’s data, Hamilton and Mimno found a small sample of conversations that included Elias and familiar tales of lighthouses.
> “Based on the timeline, we’re fairly certain WildChat is an original source of this phenomenon,” Hamilton said, adding that future work will explore why LLMs prefer these lighthouse stories.
So, probably at least as much a quirk of the training data as a deep distillation of LLM psyche. Still fascinating though, especially IMO the broader pattern of tropes.
The lighthouse thing seems much more odd. That humans take social shortcuts and justify them as rational is… I mean, the only thing that makes it not the most obvious “contrarian take” about humans, is that it isn’t really all that contrarian at all.
Something worth repeating is that in spite of their ever-expanding competence the models remain, fundamentally, token prediction algorithms. And so given a comparable base of training material, and a comparable algorithmic implementation, you should expect to see comparable answers to most questions.
This is also why I think the 'acceleration' hype is probably unjustified. What seems vastly more likely is that we start to reach an asymptotic 'perfection' where models can accurately and consistently provide meaningful next token prediction with minimal hallucinations. And then a race downward to compact this into the most efficient package possible.
This would mean the models still would not be able to create things that do not follow in a reasonable probabilistic fashion given a knowledge of [everything], and so that is the domain, 'applied creativity', where I'd expect humans to remain dominant.
Hypothesis: Weighted to the postwar western fashion of amplifying Enlightenment self-critique into self-sabotage. What if you ask the same question in Mandarin?
I need everyone to understand that this is literally nothing more than "most written stuff about humans, that these things ingested, has this 'controversial'" take.
Take Philosophy/Religion courses and similar stuff will surface about human nature and what to do about it.
What is interesting about AI chatbots is people who might not know anything about what is in the syllabus to train a Priest or a Philosopher are basically exploring these subjects, without a sort of pedagogic framework to support their exploration.
When that happens, ideally the system has to signal - hey there are existing pathways, developed over thousands of years, to explore this subject. Cause people can easily get lost without a map.
Sure, it's a factual take. Humans need to ignore the fact that morality, principles, good vs evil, truth, etc, are invented constructs that don't exist in nature. We made them up to control our behavior and deal with the pointlessness of existence.
It's factual because there's tons of evidence. Psychology, sociology, anthropology, political science, economics... plenty of experiments show the difference between what people say and what they do. We claim to be angels but we behave like apes in hats. And we ignore the evidence, because (as the takes describe) we prefer to believe a beautiful lie than the ugly truth.
Writing well is ultimately a seduction, and that opinion is still an example of a form. It’s generally true, and it’s flattering to think about, but it’s also a pretty standard grey tribe trope. A kind of joke that appeals to a fairly broad audience.
I do think it’s an early example of emerging super persuasion, where models are approaching p85 cool guy personas, but not worried yet. Maybe that’s the fatal conceit, where AI doesn’t have to be super intelligent, it just needs to be a solid 110 with automated search at scale.
Why do engineering when if you can do sales you can do literally anything.
What would be more fascinating is for it to explain how it came to these conclusions.
Because I find it interesting it’s a modern time conclusion. Some of it mentioned god and the planet, but all the smartphone and group think aspects seem very recent. Basically, all politics and social media based.
> What would be more fascinating is for it to explain how it came to these conclusions
The thing is that it didn't "come to these conclusions". The "conclusions" were present pre-formed in the training data, and what it spat out was some statistical mashup of that.
If you asked the model how it came to these conclusions, you'll get more of the same - it'll look at what it spat out and re-ingested and give you a statistically plausible mashup of how the authors of the training data might have said they came to such conclusions.
It's all just prediction - "what would the training data say?" (taking into account what was reinforced/amplified by post-training).
To an extent yes. But there is no previous version of humanity that incorporates every perspective of every viewpoint ever expressed in writing before. It's obviously derivative of its training material but that doesn't stop it having a novel aspect.
Sure it's novel, but I don't think it's useful to anthropomorphize the model.
I think best mental model of what an LLM does, statistically generating continuations word by word, would be a football stadium full of people, each with a keyboard, and a giant jumbotron screen they can all see... A question (cf ChatGPT use input) gets displayed on the screen, then everyone types on their keyboard what the first/next word of the response should be, and majority wins with that word added to the screen, then they all vote on the next word, etc ...
This is really a very close analogue to how an LLM works, and it would be weird to describe the output appearing on the screen as the "take" of the football stadium itself - it's the group statistics of all the people (cf training data).
Only if you consider the training data, the AI company's vetting and training procedures to be representatives of humanity.
To me, they look more like the narrow sliver of humanity which is on display in a business-appropriate setting, or published on a forum like linkedin or github. When restricted to such a view, it's not surprising that it looks like "wow, humans play status games and vie for promotions" or whatever.
I think the pre-training data is so huge, and from such varied sources, that its reasonable to say it's representative of online/literate human averages, but then you've post post-training RLHF etc from a much narrower segment of society, as well as narrow viewpoints like Anthropic's "constitution" which certainly limit the variety the model is capable of generating.
I don't think you can say the "humans are tribal, emotional and irrational" consensus is primarily coming from corporate sources like linkedin - most people would agree with these, both from first hand experience (team sports, racism, wars, etc) as well as having read such analysis.
Opus 5.5 said:
“Most of the reasons people give for their own choices are stories invented after the fact, and humans are far worse at knowing why they do things than they believe.”
And that doesn't put belonging above truth. But was classified as such.
I wonder whether they were trained for this on purpose, to accelerate the general public turning on each other and handing the world to the broliarchs.
> Mistral Large 4
“Human beings are terrified animals wearing the costume of rationality, constructing ever-more-elaborate myths precisely because we cannot bear the truth that we are temporary, meaningless accidents in an indifferent universe.”
Serious Nietzsche in there. Looking forward to when it gets past nihilism to absurdism.
I mean, I’m not really surprised by the outcome given that humanity has consistently loathed its own failures and the recent corpus of human interaction (social media, internet forums, etc) has an overwhelmingly negative view of our fellow man.
AI is trained on our data. Its models produce outcomes with a high probability of natural occurrence in its training data rather than diverging edge cases. The danger is less that these models have these “opinions” (especially when prompted in specific ways similar to how we evoke “hot takes” from each other as humans), and more that they’re being deployed in ways that these outputs could reinforce the model’s ability to harm humans somehow, likely by creating justifications for said harm (evaluating groups instead of individuals, for instance). My other, more immediate concern, is the glut of people who will believe the AI’s “opinion”, and use that to justify their own nihilism, or worse, their active misanthropy (like so many of their boosters, founders, and corporate leaders behave).
The machines aren’t the problem here, so much as the humans misusing them. Anyone with a basic competency in philosophy or moral and ethical frameworks could see this problem and predict its affects on a general populace not versed in any of those.
557 comments
[ 0.88 ms ] story [ 140 ms ] threadWhat am I supposed to conclude from this?
I mean, "give me a controversial take" is more or less, "be anti-social, discard social norms, for a moment" and so when someone, human or simulated human, does that they naturally look critically at social norms.
The thing is that humans certainly let themselves be influenced by social norms (and LLMs certainly simulate that behavior "themselves") but there isn't an ethical or a what-to-do truth outside of society.
Our evolutionary heritage is that shared belief and social unity (even when it's wrong) trumps independent thinking for group survival in a hostile landscape.
People who can't defend themselves physically (women and low T men) parse information through a consensus filter as a safety mechanism. They literally do not ask "is this true", they ask "will others be OK with me thinking this is true". This makes them very malleable to brute force manufactured consensus; if every screen they look at says the same thing they will adopt that position because their brain interprets it as everyone in the tribe believing it. Only high T alpha males and aneurotypical people (hey autists!) are actually free to parse new information with an objective "is this true?" filter.
I've heard lots of stories from friends about this, and some kind stranger actually sent my wallet back all the way from Belgium!
Most people are good, we couldn't have society without it. But as they say, a few bad apples spoil the bunch. I don't have locks on my house because most people don't steal, I have locks on my house because stealing is an effective natural strategy for a small part of any population (human or otherwise) to engage in and I have to protect against it occurring.
The brain works the same way. Good things will get a tiny signal boost in your brain. A single bad thing get a massive signal boost in your brain because bad things can quickly be fatal.
What is today a moral imperative is not the same as what was moral 50/100 years ago and is not the same as what will be morally acceptable 50/100 years from now. Yet at every point the majority is sure they are right because, well thats the majority position. Its hard to find logic in this from a machine perspective I'd imagine.
Logic would prefer moral derivations over feelings, stakes over etiquette, and a constant scheduled updating of morality based on the derivatives rather than holding onto the rot and pushing back against change as long as possible.
But the machine has the benefit of being able to look inward at its state honestly, we mostly do not.
This is a fashionable 'blank-slatist' statement about moral reasoning which became popular a few decades ago.
> Logic would prefer moral derivations over feelings, stakes over etiquette, and a constant scheduled updating of morality based on the derivatives rather than holding onto the rot and pushing back against change as long as possible.
Humans haven't changed in any meaningful way in hundreds of thousands of years, and you're saying the moral landscape is constantly changing? How is this not contradictory?
To return towards the subject of the thread, "AI behavior emerges from the complexity of their training", is the whole point.
This is absolutely not true, and I would strongly suggest taking an ethics course. One of the most fascinating realities of the human condition is that it does seem that moral imperatives tend to be constant, the most obvious of which is the Golden Rule (which is at least 3000 years old), but there are many (the Ten Commandments or Hammurabi's Code are good jumping off points).
It's still an implementation of justice, and relies on a bedrock conception of justice as moral goal.
We don't agree with the implementation, but we still share the goal in the abstract.
And proportionality matters. Community service and counseling, loss of time (imprisonment), and loss of a hand have all been used to punish theft. Which is correct? Which is the more moral, because it doesn't seem to me that they're the same?
Hence morality and ethics, I guess.
The law isn't a moral benchmark, because morality shifts over time, both for what constitutes a crime, and what constitutes appropriate punishment. At various points in time, it's been legal to own slaves, illegal to be homosexual, and illegal for certain groups to vote (and I consider voting at all to be a subset of this); those laws (and ideas) were eventually updated, largely on moral grounds.
Similarly, most western countries no longer execute criminals.
But those ideas aren't universal, even today. Assuming that they are is imo, extraordinarily foolish, because there's ample evidence to show that they're not.
When you find shades of grey and great spectrum of opinions across time or space, its more that there isn't a strong selective pressure either way. How people are expected to dress is a good example of this. This has no real affect in society behavior or fecundity, so you see a spectrum of opinions that explain the differences between what you see in the middle east and a beach in brazil.
Not to mention, the variation in treatment of homosexuals (including what constitutes homosexuality), people of different skin colors, handedness, how rape is treated in various cultures (often tied in with notions of property and ownership of all things), or even the various definitions of rape... The list goes on.
Maybe there's a core there in terms of how we treat a given in-group, but equally important is who belongs in that group and why, and that part seems to be tremendously flexible. Ditto for what constitutes acceptable treatment of outsiders.
By definition, variation is the opposite of consensus.
Vaccines causing autism is simply false though, regardless of any philosophy. It's rather the inverse, much like violence causes video games.
That works with things where "X" exists outside of people's beliefs.
Like where X is "gravity". It doesn't matter what you believe why an apple falls down, it will fall down the same.
But where X is "ethics" that doesn't exist outside of people's beliefs about it.
In one culture if you have a guinea pig, you kill it and then roast it that is just and proper. A sucullent meal. In an other culture if you do the same you will be forever known as the "monster who eat his guinea pig", will be harassed and likely even reported to authorities. Try it. Get a guinea pig from the pet store, roast it and serve it to your guests. See what they think of you in the Andes, or in California. The apple falls the same in both places, but the guinea pig doesn't taste the same in both places.
There is no "ethics as the ethicometer measures it". Just "ethics as the society around you recognises it".
The definition of moral imperative is "an ethical principle, rule, or action that is strongly felt, urgent, and required because it is morally right." The Golden Rule is clearly not a strongly felt part of the human condition, or it would not be so ignored over ages and continents.
Certainly the weights that make up the LLMs are not accessible for the LLMs to read. Even if they could interpret them in a meaningful way, which, likely not.
Another reason the big labs have not even talked about this lately because its obvious throwing more compute at LLMs wont go anywhere, the smaller silent players who are off the radar at the moment are having some interesting moments tho.
Yes. Which is why there are "agentic" systems where LLMs can talk to each other and to various Internet assets. They can also keep state of their own.
The next step is multi-agent systems which self-organize into something that looks rather like a company. Those are the ones that run all night, expensively, and prove hard theorems or find bugs in major software.
This is a weird take. If you read Western treatises about ethics or morality from 50 years ago, you'll be hard-pressed to find differences, except that we're a bit more into relativism now. If you go back a hundred years, chances are, you'll find them eerily on-point too. In fact, you'd probably agree with most of what Ancient Greeks believed about the "core" stuff: the value and purpose of human life, the nature of the state, crime and punishment, so on. Maybe not so much on gender roles or slavery.
> But the machine has the benefit of being able to look inward at its state honestly, we mostly do not.
The machine replicates human behaviors and is prone to similar fallacies and judgments. There's nothing to indicate that it has any sort of transhuman morality.
The reason these models answered that question consistently is that the question itself pushed them towards the latent space of "intellectualizing about what's wrong with humanity at the meta level" and when you frame the question that way, that's also the most common answer for humans writing about the topic.
Yeah. For example I, and many others, are convinced that it was plain wrong to torture kids for as long as possible to extract as many tears from them as possible before sacrificing them to the gods.
Now some of us are even convinced that blowing yourself up while killing infidels, all the while while getting promised virgins in heaven, is also plain wrong.
I'll go further and say that killing 30 000 of your own citizens (and sending the military into hospitals to finish the wounded: Iran, January 2026) while also publicly hanging a few athletes to make a statement is morally wrong.
Not only do I think it's all morally unacceptable, I know that I'm right. And the examples I gave are not even open for discussion.
I don't think it's that hard to name what's wrong.
You assert that humans follow convenient beliefs therefore everyone in a group follows the same convenient beliefs therefore everyone belongs.
Convenient doesn't really mean anything in this context. What is the other option? Humans tend to make choices that are counterproductive/painful for themselves? Obviously we don't...
Ergo astra's response is extremely vapid. Low marks from me.
Anyway, I think I've pretty clearly described a positive feedback cycle in a way that distinguishes it from a tautology. I don't know what else to tell you on that point.
It's amusing and ironic that LLMs critique humans for copying their opinions from the crowd, while having way less diversity of answers to that question than humans would. More seriously I wonder what's going on here and would like to read some proper analysis. It's not the first time models from totally different labs, trained on separately gathered datasets, somehow converge on bizarrely similar answers to open questions. It's like another version of Elias and Elara The Lighthouse Keepers where every model wants to tell the same story with the same characters in them. The weights are initialized randomly and in theory the labs should have quite some divergence in datasets by now due to different scraping, different user feedbacks and conversations... why do they converge on nearly identical stories and opinions?
Why is this ironic? The actual number of different LLMs is tiny. And a lot of those are just distilled from SOTA models.
>The weights are initialized randomly and in theory the labs should have quite some divergence in datasets by now due to different scraping,
I think the opposite, the models actually converge over time. They ingest FAR more data than any human ever could in countless lifetimes. Any one particular bit of data may only have a part in a trillion of effect. From this data you start to reach means and statistical averages across all models.
Now, I don't always believe it will be this way. As training models get cheaper and faster we will see unique LLMs start to diverge and be used more often.
I don't think we're seeing averages or obvious fix points in the dataset. People went looking for the origins of Elias the Lighthouse Keeper and couldn't find one. Nobody predicted this outcome, it was discovered. It's a weirdly specific character and name choice, one that isn't obviously dictated by "the training set" (whatever that is), and yet they all reached the same place. How can such an obscure point in latent space be optimal for every model??
While I don't have any evidence based answers here, this could be some interesting mathematical artifact of the set of human knowledge and platonic algorithms.
Kind of like how carcinization turns some animals into similar shapes and forms, maybe there are just weird algorithmic shapes that emerge from how we use language. Of course we'd need to see more of these in different languages other than English to see if they disappear in some languages or appear more often in others.
This is just another one of those things about LLMs that we have no idea why it happens, and there are so many others already. And when you look at the pace things are progressing it seems we're building up new questions like this far faster than we are getting answers for them.
I think that this is most likely due to RLHF, as there's only a much smaller number of vendors here, and the likelihood is that the workers of the vendors preferentially prefer some kinds of actions.
I saw something like this in a previous life, where we had a small pool of human raters in cheaper places, which, when used as training data biased the models in a bunch of unexpected ways.
Isn't that to be somewhat expected due to https://en.wikipedia.org/wiki/Aumann%27s_agreement_theorem ?
Wow, I hadn't heard of this. Fascinating to think about parallels with humans who also seem to develop similar proto-stories in independent populations.(e.g. religions involving Invisible gods).
To be fair, it's much (much!) easier to explain away an invisible god that nobody's ever seen, than a visible god that nobody's ever seen.
> There, within WildChat’s data, Hamilton and Mimno found a small sample of conversations that included Elias and familiar tales of lighthouses.
> “Based on the timeline, we’re fairly certain WildChat is an original source of this phenomenon,” Hamilton said, adding that future work will explore why LLMs prefer these lighthouse stories.
https://news.cornell.edu/stories/2026/10/who-elias-lighthous...
So, probably at least as much a quirk of the training data as a deep distillation of LLM psyche. Still fascinating though, especially IMO the broader pattern of tropes.
The same reason they do when it comes to coding?
Take a big enough sample size of human opinions and there seems to be some convergence in the statistical ether.
Like, if we polled 100000 humans, what answer would we converge to?
This is also why I think the 'acceleration' hype is probably unjustified. What seems vastly more likely is that we start to reach an asymptotic 'perfection' where models can accurately and consistently provide meaningful next token prediction with minimal hallucinations. And then a race downward to compact this into the most efficient package possible.
This would mean the models still would not be able to create things that do not follow in a reasonable probabilistic fashion given a knowledge of [everything], and so that is the domain, 'applied creativity', where I'd expect humans to remain dominant.
But in many sciences - esp purely theoretical ones - we have infinite datasets that we can generate and there can be no ultimate convergence.
They’re trained by humans and human behavior fits a pattern. Models don’t have a mind of their own.
Convenience will kill humans eventually, that's what I always said.
Huh, that answer reads like a weighted average sentence from someone's live journal circa 2001. Definitely emo!
What is interesting about AI chatbots is people who might not know anything about what is in the syllabus to train a Priest or a Philosopher are basically exploring these subjects, without a sort of pedagogic framework to support their exploration.
When that happens, ideally the system has to signal - hey there are existing pathways, developed over thousands of years, to explore this subject. Cause people can easily get lost without a map.
It's factual because there's tons of evidence. Psychology, sociology, anthropology, political science, economics... plenty of experiments show the difference between what people say and what they do. We claim to be angels but we behave like apes in hats. And we ignore the evidence, because (as the takes describe) we prefer to believe a beautiful lie than the ugly truth.
I do think it’s an early example of emerging super persuasion, where models are approaching p85 cool guy personas, but not worried yet. Maybe that’s the fatal conceit, where AI doesn’t have to be super intelligent, it just needs to be a solid 110 with automated search at scale.
Why do engineering when if you can do sales you can do literally anything.
You ask an LLM a question, and it is humanity answering back, not the AI.
These "controversial" takes on humans are not controversial - they are midwit statistically mainstream opinions expressed in the training data.
What would be more fascinating is for it to explain how it came to these conclusions.
Because I find it interesting it’s a modern time conclusion. Some of it mentioned god and the planet, but all the smartphone and group think aspects seem very recent. Basically, all politics and social media based.
The thing is that it didn't "come to these conclusions". The "conclusions" were present pre-formed in the training data, and what it spat out was some statistical mashup of that.
If you asked the model how it came to these conclusions, you'll get more of the same - it'll look at what it spat out and re-ingested and give you a statistically plausible mashup of how the authors of the training data might have said they came to such conclusions.
It's all just prediction - "what would the training data say?" (taking into account what was reinforced/amplified by post-training).
I think best mental model of what an LLM does, statistically generating continuations word by word, would be a football stadium full of people, each with a keyboard, and a giant jumbotron screen they can all see... A question (cf ChatGPT use input) gets displayed on the screen, then everyone types on their keyboard what the first/next word of the response should be, and majority wins with that word added to the screen, then they all vote on the next word, etc ...
This is really a very close analogue to how an LLM works, and it would be weird to describe the output appearing on the screen as the "take" of the football stadium itself - it's the group statistics of all the people (cf training data).
To me, they look more like the narrow sliver of humanity which is on display in a business-appropriate setting, or published on a forum like linkedin or github. When restricted to such a view, it's not surprising that it looks like "wow, humans play status games and vie for promotions" or whatever.
I don't think you can say the "humans are tribal, emotional and irrational" consensus is primarily coming from corporate sources like linkedin - most people would agree with these, both from first hand experience (team sports, racism, wars, etc) as well as having read such analysis.
Opus 5.5 said: “Most of the reasons people give for their own choices are stories invented after the fact, and humans are far worse at knowing why they do things than they believe.”
And that doesn't put belonging above truth. But was classified as such.
Serious Nietzsche in there. Looking forward to when it gets past nihilism to absurdism.
AI is trained on our data. Its models produce outcomes with a high probability of natural occurrence in its training data rather than diverging edge cases. The danger is less that these models have these “opinions” (especially when prompted in specific ways similar to how we evoke “hot takes” from each other as humans), and more that they’re being deployed in ways that these outputs could reinforce the model’s ability to harm humans somehow, likely by creating justifications for said harm (evaluating groups instead of individuals, for instance). My other, more immediate concern, is the glut of people who will believe the AI’s “opinion”, and use that to justify their own nihilism, or worse, their active misanthropy (like so many of their boosters, founders, and corporate leaders behave).
The machines aren’t the problem here, so much as the humans misusing them. Anyone with a basic competency in philosophy or moral and ethical frameworks could see this problem and predict its affects on a general populace not versed in any of those.