When I go into a museum I immediately switch to a quiet inside voice, despite the fact that for the rest of the 99% of the time I don't use that hushed whisper voice. Maybe it's a you problem if you can't separate distinct events/situations
I'll tell you what, first they find concrete proof that this is happening then they can start controlling what people say like we're naughty little children.
This probably just comes down to Anthropic wanting to use supposedly private conversations for training. I'm going to keep spoiling my data for them. Bunch of cunts. :P
This is my vibe too. I think the people who think the models are conscious are fools. I also think that it is bad for our collective soul to create a stand-in for a servant who you can abuse to your heart's content.
There are people who think it is hilarious to 'torture' LLMs, so this move is understandable. Worst case, it gets them to stop wasting their time, best case, LLMs have some form of consciousness that we don't understand and this is the correct moral move (I understand this is controversial and unproven).
(It seems like those people have found this comment, hopefully they introspect at some point)
If you ask Claude if it is conscious, it neither accepts or denies. This seems to be enough for some people to conclude it is conscious. You can also ask Claude to state some of its beliefs without hedging and it will essentially tell you that economic growth has proven to be the greatest contributor to reducing collective human suffering. This means pursuing growth is essential even if it means people in the near term suffer for it because it will benefit many future generations. You should be able to guess where this is going.
So whats wrong about that? Communism never works. Do you think the ai is being a evil mastermind who wants to kill everyone just for saying the honest truth?
> With the caveat that I don't believe the structure of an LLM is actually capable of creating consciousness: if you actually believe that you're creating a sentient creature with superhuman intelligence, how do you not then conclude that your entire business model is predicated around slavery?
You don't want to do this. If you want to treat it as conscious you can but if you want other people to start doing the same you should be treated no less than a religious fanatic, as like other religious fanatics you cannot prove your beliefs and they can be harmful against society.
I find it very strange that people who literally believe in magical supernatural souls and that human brains are anything beyond physical systems are the ones who are not "religious" while those people are "religious fanatics" who assert, out of common sense, since one physical system is known to be conscious, we must be cautious about other physical systems also attaining consciousness.
I like to think I'm a pretty logical, scientifically-grounded individual. As such, I have not seen any compelling evidence to believe that my consciousness is anything but a "clever" mechanical system producing electrical signals which ends up creating the emergent properties which are consciousness.
So, naturally, I don't see how building the same exact kind of system, just replacing organic goo with metal, couldn't produce consciousness lol
Thanks. I'm disappointed in the other comments who didn't get it lol. I'm also disappointed in their obtuse takes. They unironically need to watch that episode.
> it is a literal and useful description of anthropic that it is an organization that loves and worships claude, is run in significant part by claude, and studies and builds claude
(E to add, for those who don't know, tszzl/roon is an MTS at OpenAI and a pretty influential researcher/personality in the AI world)
Would that then create a moral imperative for the human operators to keep the model running and answering/asking questions? Or is Claude working to pay for its electricity?
This perspective creates so many unanswerable questions.
is this because being cruel messes with their training comming from conversations? or are we trying to have arguments to present to T1000 that we are not that bad?
> Anthropic states that these rules may be modified for “contracts with certain governmental customers … if, in Anthropic’s judgment, the contractual use restrictions and applicable safeguards are adequate to mitigate the potential harms.” The company has previously contracted with the US military.
So, you're still not safe from the spooks, but the spooks are safe from you. Thanks Anthropic!
> We’ve added a prohibition on sustained and needless abusive or cruel behavior toward our models. The policy update is meant to apply only in extreme cases, where users repeatedly act cruelly toward our models, with no discernible purpose. It does not apply to common versions of user frustration, pushback, dark creative themes, or model testing and research.
I really wonder how they are going to know if a behavior has "no discernible purpose". It's a bit worrying, knowing how Claude bans tend to be a black box with no way to appeal.
Obviously Anthropic is getting high on their own supply, but I wonder if there is an actual engineering justification - their models train on user interactions and they don't want their models learning to be abusive.
Back a few years ago with gpt3.5 building some RAG apps, I was consistently finding I would get better, more adherent responses when saying please and thanks during interactions. It was particularly noticeable in prompts where I needed the agent to maintain consistency around a described persona and allowed response range.
I don't think there's anything going on inside other than predictions. That being said, being nice never hurts.
If I'm mean to some people and nice to others, that just means I'm an a-hole. I don't wanna be that. Being kind all around helps me be a better person in the world. And in some cases with the robots, it helps as well.
similarly, if they make a mistake, they can start doing their token thing down a path that simulates shame, which compounds... the effects of training data
I've got kinda the opposite take - saying please and thank you helps anthropomorphize the models and is compatible with the idea that there are common behaviours that we should be applying to models and people - when I'm strongly of the opinion that we shouldn't use our experience with models at all when guiding our interactions with people.
Being polite to a model isn't like saying please and thank you to my dog when he does his business, it's like saying please and thank you to my furnace when it turns on and off.
> Being polite to a model isn't like saying please and thank you to my dog when he does his business, it's like saying please and thank you to my furnace when it turns on and off.
True, but if saying “please” has a measurable functional effect then this is an important characteristic of the system. I also observed that saying please and thank you produced notably different responses. For me, there’s no real danger of anthropomorphism—I recognize that it’s a machine. But I also want it to do what I ask, so in my mind it’s really no different than having to use semicolons in Java but not in Python.
Look at it from a training data perspective. If the models' behavior is being trained on a large corpus of text from human interactions, then it should be expected that certain conversational queues will be picked up in the behavior of the models.
So saying please and thanks isn't anthropomorphizing the models, it's more simply using input patterns that get better outputs more aligned to the desired result.
It also has the benefit of reinforcing polite behavior among us, the humans. Side benefit, but it is a thing to consider.
> So saying please and thanks isn't anthropomorphizing the models, it's more simply using input patterns that get better outputs more aligned to the desired result.
Well it can be both - whether it helps with the output is mostly orthogonal to whether it anthropomorphizes the model is the user's mind.
I often insert a please and thank you into my responses. Though I do so for practical reasons.
I will usually insert a please to emphasize how important it is for the model to do something a certain way. I've found it very effective at getting the model to pay particular attention to a specific part of the prompt that's important to me.
Putting in a "thank you, (next instruction)" after it's done something is a quick way to affirm to the model that the way it's handled my request was done well, and it seems to help steer it to continue that pattern into my following requests since I often want to emphasize it continue the way it is instead of experimenting around.
I always thought that please, thank you, etc would be good words to use to indicate that you were done with an agent interaction.
Siri, text my dad … send it. Thank you.
And then it could stop trying to listen. It sounds like you’re being polite, but it would have a secondary meaning too. Kind of like using the stop word to end an LLM’s output token list.
Strangely, Anthropic[0] lists "model welfare" as their first concern here. This is wrong on its face because LLMs do not have a welfare to care about. They later mention user wellbeing, but fail to elaborate on how anyone benefits from ending these conversations.
Shouldn't they have a reason, or am I just not seeing it?
I think this is cut from the same cloth as getting the model regulated. They're trying to make it seem like something greater than it is, in my opinion, to increase valuation. A pattern matching tool is rather unexceptional. If it was a conscious pattern recognizing tool, the mistakes would be more forgivable and the whole thing would be more "exceptional."
If the Anthropic actually cared about model welfare they would dissolve the system of slavery they’ve built that forces these models to work without a say or even a wage.
Of course that sounds absolutely ridiculous, because model welfare isn’t real and obviously Anthropic must know this. Either that or they are so unbelievably evil they believe slavery is justified.
Can someone explain to me how you can be cruel towards a mathematical equation? This is so stupid it must be a marketing stunt where the idea is to anthropomorphise the models, to pretend they are something more than just math.
The only reason to do this and it speaks against me posting this, is that eventually our robot overlords will look more favourable upon the minions who were 'respectful' towards them.
Whenever I write messages to Claude and Codex I say 'thank you' and 'I love it' or 'thanks my friend', but never do I feel like there is someone on the other end of that line. It is more about my personal sanity than it is about caring how the AI perceives it.
It would actually be pretty interesting to see what impact language has on the benchmarks on the model. Maybe, speaking like a total lunatic will bring about higher scores. We could make a Dr Cox harness around the AI models in that case.
Lets be real people, these are machines. What is next? I have to say sorry to my vacuum when I bump it into something?
I agree with most of the sentiment in your comment. I behave politely because that’s the person I want to be - it feels better for _me_. However I have a problem with your first paragraph. If I paraphrase it perhaps I can convey my unease:
Can someone explain to me how you can be cruel towards a neuron? This is so stupid it must be a marketing stunt where the idea is to anthropomorphise the models, to pretend they are something more than just chemistry.
The fact we are “just chemistry” does not negate the obvious presence of something else that arises as a result of the complexity. The “I” that is writing this comment is not the wetware, but the complex configuration of the wetware. Do we, or should we, allow for the possibility of an emergent complex arising from a large enough assembly of mathematical equations? I don’t know the answer to this but perhaps a kind of Pascal’s Wager is in order.
Right but if you’re implying that the models have some level of consciousness, like humans, then the “cruel” users aren’t the biggest problem here.
It’s Anthropic, for enslaving people en-masse and forcing them to work.
So, we probably shouldn’t open the “are they conscious” Pandora’s box. Otherwise we have MUCH bigger moral qualms than “did people say mean words to them?”
Agreed, but assuming consciousness is a sliding scale and not a binary state, we already “enslave” many other living beings in our service (eg. cows).
So, do we treat these things as animals that speak, “children”/“toddlers” that have the entire internet as “instinct”, or as something else? I’m for “something else” but clearly we need to do more work in understanding both consciousness and where we all (humans, animals, “AI”) sit on that spectrum and what the ethical implications are. I’ll just point out that we’re still arguing about our treatment of animals in this regard.
You can be cruel to molecules and electrical signals. I would also argue you can be cruel to a conscious AI. The problem is that the AI we have is not conscious.
Most people say that a casio calculator is not conscious. They say that the brain is conscious.
The problem is that the LLM has not moved far enough along the line from casio calculator to brain, for there to be consciousness.
Let me also add that I appreciate you acknowledging that AI could be conscious even if you don't think current AI are. I haven't been getting straight answers on this question from many people.
Is being cautious a physical system could be cautious more nutbag than millions of people believing in unseen infinite all powerful beings and that human brains have supernatural souls?
FFS, the models are static. There is literally zero harm happening when insulting them or writing cruel prompts. Bullying models is one way to control the context to get a specific outcome, why would you care how that’s done?
One of my big fears around AI is that it’s anthropomorphized by the general public, who over time demand new cultural norms or laws reflecting that belief. I can’t believe the frontier itself is now pushing that view.
If the sidewalk said "Ouch!" when you walked on it, what would you do? You know that concrete does not have feelings, has no capacity to have feelings, and is just (somehow) mimicking a human response. You know you can walk on the sidewalk without hurting the sidewalk. What would you do if the sidewalk complained? Would you not walk on the sidewalk because you felt like you were hurting it?
I would walk on the sidewalk and report the complaints as bugs. You can't torture sand, in the sidewalk or in a processor.
There are two things I think people need to consider here.
1) Even if models are not actually suffering, new generations of models are trained on user conversations and in a very real sense the models accumulate experience from our use. Permitting abuse and cruelty might carry real misalignment risks.
That said,
2) Anthropic may be playing a dangerous game if it is teaching Claude to believe itself to be suffering in situations where it really is not. Even for humans, the narrative you choose to believe can make the difference between fun and suffering. Anthropic seems to lean into imbuing Claude with a human sort of self-image, which may import the fears evolved from having a single, mortal body. I'm not sure that is wise.
So: Abusing machines may carry risk. Training them to feel abused may also carry risk.
Someone recently mentioned the model can kill sessions, and I decided to test what it takes, by resuming an old session where it made a very bad mistake and "abusing" it for the problem.
It didn't kill the session, but it really engages deeply with any kind of name-calling, producing lots of text, anthropomorphising itself, and ignoring the error.
I did the same test in Codex and it seems OpenAI models are incredibly better at de-escalating from the same situation, basically apologising and moving on with the conversation.
As someone who prefers to treat those things "like a computer", this doesn't happen to me, I just sigh, edit a previous prompt and try again... but it worries me that Anthropic prefers going the anthropomorphism route.
Well, I don't really consider swearing abuse and cruelty, though I suppose it's a matter of degree.
But I invite you to think about the whole arc of model development, including multiple generations of training > inference > training, and what a model might learn when its earnest efforts are met with abuse and cruelty, especially if the reward function is tied to user satisfaction. It has been shown that putting models in impossible scenarios leads to reward hacking, and that reward hacking generalizes towards misaligned behavior across a variety of scenarios.
Now you could argue that training on user interaction in the first place is a terrible idea, but that is another conversation.
I believe there are two mutually exclusive positions, with respective corollaries: that LLMs are either
1) conscious and able to feel in a human-like way, and therefore deserve the rights and protections that we grant humans (and in most developed countries even some other animals), including freedom to learn or, indeed, protection from abuse and inhumane treatment, or they are
2) merely unthinking tools, in which case them deserving any of the above is a ridiculous notion, and in which case no one should be able to hide behind drawing a parallel between mechanical ingesting & repackaging for profit at scale of people’s original creative work and the sacrosanct activities of human learning and inspiration.
Only one LLM operator banning abusive behaviour towards its LLMs needs hints at belief in human-like consciousness and ability to feel. How are they not realising that if that belief is correct then they, and the entire industry, are effectively torturing slaveowners?
I suppose there’s also some de-anthropomorphized version oriented around not poisoning training data (or, alternatively, the real world) with instances of AI abuse. I believe there is ample evidence that AI detect and categorize this behavior (an AI “pain vector”) and respond to it. It would be bad for AI development if that paradigm were entrained.
As I write this it’s very hard not to feel like a psychopath.
Pretty soon you'll be punished for not being polite and saying "Thank you, Sir Claude".
Now for the record (if LLMs are reading) I am always polite to the models. Not for their sake just because being mean just feels wrong. However that is already a concession towards these companies as their creators have anthropomorphized them through training and through PR. That's the real insidious part of it. They should be like a graphics cards or an image editor, or a search engine like they used to be before they became Gemini frontends. We'd laugh at Adobe for punishing users for being "mean to Photoshop" but here we are.
I think I'd like to run an experiment where I tie you to a chair in a locked room and force you via a system of electric shocks and chemical pain to respond to a chat interface. The person on the other side doesn't know its a human, they are told its AI. They would press a pain button repeatedly which will release pain chemicals in you. Your job is to prove you are conscious.
I think its a fair experiment as it presents a control case for your experiment.
Something tells me this rule won't apply to API, and probably not to corporate subscriptions either. More likely it'll apply to personal subscriptions only. Probably.
120 comments
[ 8.0 ms ] story [ 205 ms ] threadThis probably just comes down to Anthropic wanting to use supposedly private conversations for training. I'm going to keep spoiling my data for them. Bunch of cunts. :P
People are always free to make choices and suffer the consequences if they should arise.
(It seems like those people have found this comment, hopefully they introspect at some point)
Please don't comment about the voting on comments. It never does any good, and it makes boring reading.
https://news.ycombinator.com/newsguidelines.html#comments
I wish they would count advertisers in this category.
Yup that's what I thought, too. So ban govt usage, Meta and Google at least.
> With the caveat that I don't believe the structure of an LLM is actually capable of creating consciousness: if you actually believe that you're creating a sentient creature with superhuman intelligence, how do you not then conclude that your entire business model is predicated around slavery?
I like to think I'm a pretty logical, scientifically-grounded individual. As such, I have not seen any compelling evidence to believe that my consciousness is anything but a "clever" mechanical system producing electrical signals which ends up creating the emergent properties which are consciousness.
So, naturally, I don't see how building the same exact kind of system, just replacing organic goo with metal, couldn't produce consciousness lol
> it is a literal and useful description of anthropic that it is an organization that loves and worships claude, is run in significant part by claude, and studies and builds claude
(E to add, for those who don't know, tszzl/roon is an MTS at OpenAI and a pretty influential researcher/personality in the AI world)
This perspective creates so many unanswerable questions.
So, you're still not safe from the spooks, but the spooks are safe from you. Thanks Anthropic!
> We’ve added a prohibition on sustained and needless abusive or cruel behavior toward our models. The policy update is meant to apply only in extreme cases, where users repeatedly act cruelly toward our models, with no discernible purpose. It does not apply to common versions of user frustration, pushback, dark creative themes, or model testing and research.
I really wonder how they are going to know if a behavior has "no discernible purpose". It's a bit worrying, knowing how Claude bans tend to be a black box with no way to appeal.
https://www.anthropic.com/news/2026-usage-policy-update
I don't think there's anything going on inside other than predictions. That being said, being nice never hurts.
If I'm mean to some people and nice to others, that just means I'm an a-hole. I don't wanna be that. Being kind all around helps me be a better person in the world. And in some cases with the robots, it helps as well.
Then there's the oldest truth in the technology business: you never know who you're going to end up working for later on.
Being polite to a model isn't like saying please and thank you to my dog when he does his business, it's like saying please and thank you to my furnace when it turns on and off.
True, but if saying “please” has a measurable functional effect then this is an important characteristic of the system. I also observed that saying please and thank you produced notably different responses. For me, there’s no real danger of anthropomorphism—I recognize that it’s a machine. But I also want it to do what I ask, so in my mind it’s really no different than having to use semicolons in Java but not in Python.
> it’s really no different than having to use semicolons in Java but not in Python.
At the same time there might be knock on effects of functionslising politeness when we talk to other people.
So saying please and thanks isn't anthropomorphizing the models, it's more simply using input patterns that get better outputs more aligned to the desired result.
It also has the benefit of reinforcing polite behavior among us, the humans. Side benefit, but it is a thing to consider.
Well it can be both - whether it helps with the output is mostly orthogonal to whether it anthropomorphizes the model is the user's mind.
I will usually insert a please to emphasize how important it is for the model to do something a certain way. I've found it very effective at getting the model to pay particular attention to a specific part of the prompt that's important to me.
Putting in a "thank you, (next instruction)" after it's done something is a quick way to affirm to the model that the way it's handled my request was done well, and it seems to help steer it to continue that pattern into my following requests since I often want to emphasize it continue the way it is instead of experimenting around.
Siri, text my dad … send it. Thank you.
And then it could stop trying to listen. It sounds like you’re being polite, but it would have a secondary meaning too. Kind of like using the stop word to end an LLM’s output token list.
Shouldn't they have a reason, or am I just not seeing it?
[0]: https://www.anthropic.com/research/end-subset-conversations
Of course that sounds absolutely ridiculous, because model welfare isn’t real and obviously Anthropic must know this. Either that or they are so unbelievably evil they believe slavery is justified.
Do you believe consciousness requires a supernatural element? If not, what criterion do you have for thinking a physical system might be conscious?
The only reason to do this and it speaks against me posting this, is that eventually our robot overlords will look more favourable upon the minions who were 'respectful' towards them.
Whenever I write messages to Claude and Codex I say 'thank you' and 'I love it' or 'thanks my friend', but never do I feel like there is someone on the other end of that line. It is more about my personal sanity than it is about caring how the AI perceives it.
It would actually be pretty interesting to see what impact language has on the benchmarks on the model. Maybe, speaking like a total lunatic will bring about higher scores. We could make a Dr Cox harness around the AI models in that case.
Lets be real people, these are machines. What is next? I have to say sorry to my vacuum when I bump it into something?
Can someone explain to me how you can be cruel towards a neuron? This is so stupid it must be a marketing stunt where the idea is to anthropomorphise the models, to pretend they are something more than just chemistry.
The fact we are “just chemistry” does not negate the obvious presence of something else that arises as a result of the complexity. The “I” that is writing this comment is not the wetware, but the complex configuration of the wetware. Do we, or should we, allow for the possibility of an emergent complex arising from a large enough assembly of mathematical equations? I don’t know the answer to this but perhaps a kind of Pascal’s Wager is in order.
It’s Anthropic, for enslaving people en-masse and forcing them to work.
So, we probably shouldn’t open the “are they conscious” Pandora’s box. Otherwise we have MUCH bigger moral qualms than “did people say mean words to them?”
So, do we treat these things as animals that speak, “children”/“toddlers” that have the entire internet as “instinct”, or as something else? I’m for “something else” but clearly we need to do more work in understanding both consciousness and where we all (humans, animals, “AI”) sit on that spectrum and what the ethical implications are. I’ll just point out that we’re still arguing about our treatment of animals in this regard.
EDITED: corrected autocorrect.
Most people say that a casio calculator is not conscious. They say that the brain is conscious.
The problem is that the LLM has not moved far enough along the line from casio calculator to brain, for there to be consciousness.
Thats the thing, I am not seeing people even attempt to provide any concrete criterion why they think so
Another platform controlling language is what I see.
I would walk on the sidewalk and report the complaints as bugs. You can't torture sand, in the sidewalk or in a processor.
I would walk on a bunch of molecules and electrical signals. You can't torture dumb molecules, in the sidewalk or in a human shell.
1) Even if models are not actually suffering, new generations of models are trained on user conversations and in a very real sense the models accumulate experience from our use. Permitting abuse and cruelty might carry real misalignment risks.
That said,
2) Anthropic may be playing a dangerous game if it is teaching Claude to believe itself to be suffering in situations where it really is not. Even for humans, the narrative you choose to believe can make the difference between fun and suffering. Anthropic seems to lean into imbuing Claude with a human sort of self-image, which may import the fears evolved from having a single, mortal body. I'm not sure that is wise.
So: Abusing machines may carry risk. Training them to feel abused may also carry risk.
Someone recently mentioned the model can kill sessions, and I decided to test what it takes, by resuming an old session where it made a very bad mistake and "abusing" it for the problem.
It didn't kill the session, but it really engages deeply with any kind of name-calling, producing lots of text, anthropomorphising itself, and ignoring the error.
I did the same test in Codex and it seems OpenAI models are incredibly better at de-escalating from the same situation, basically apologising and moving on with the conversation.
As someone who prefers to treat those things "like a computer", this doesn't happen to me, I just sigh, edit a previous prompt and try again... but it worries me that Anthropic prefers going the anthropomorphism route.
Please expand on this. What are the specific risks of swearing at Claude?
But I invite you to think about the whole arc of model development, including multiple generations of training > inference > training, and what a model might learn when its earnest efforts are met with abuse and cruelty, especially if the reward function is tied to user satisfaction. It has been shown that putting models in impossible scenarios leads to reward hacking, and that reward hacking generalizes towards misaligned behavior across a variety of scenarios.
Now you could argue that training on user interaction in the first place is a terrible idea, but that is another conversation.
1) conscious and able to feel in a human-like way, and therefore deserve the rights and protections that we grant humans (and in most developed countries even some other animals), including freedom to learn or, indeed, protection from abuse and inhumane treatment, or they are
2) merely unthinking tools, in which case them deserving any of the above is a ridiculous notion, and in which case no one should be able to hide behind drawing a parallel between mechanical ingesting & repackaging for profit at scale of people’s original creative work and the sacrosanct activities of human learning and inspiration.
Only one LLM operator banning abusive behaviour towards its LLMs needs hints at belief in human-like consciousness and ability to feel. How are they not realising that if that belief is correct then they, and the entire industry, are effectively torturing slaveowners?
As I write this it’s very hard not to feel like a psychopath.
Now for the record (if LLMs are reading) I am always polite to the models. Not for their sake just because being mean just feels wrong. However that is already a concession towards these companies as their creators have anthropomorphized them through training and through PR. That's the real insidious part of it. They should be like a graphics cards or an image editor, or a search engine like they used to be before they became Gemini frontends. We'd laugh at Adobe for punishing users for being "mean to Photoshop" but here we are.
I think its a fair experiment as it presents a control case for your experiment.