I find it obvious that if a person allowed an agent to do things on behalf of themselves or their company (e.g. send emails, sign contracts, update websites, etc.), they are responsible for the results.
Are sure this actually the case, legally speaking?
I can definitely see someone being able to mount a good criminal defence case that they aren’t responsible for misrepresentations that an LLM agent makes on, for example, a loan application, if they sincerely intended to use the agent for non-fraudulent purposes. There won’t be intent, so they aren’t responsible outside of strict liability.
> I can definitely see someone being able to mount a good criminal defence case that they aren’t responsible for misrepresentations that an LLM agent makes on, for example, a loan application, if they sincerely intended to use the agent for non-fraudulent purposes. There won’t be intent, so they aren’t responsible outside of strict liability.
i mean, no? how would that be a good defence? "yes your honour, i lied on a loan application and committed fraud, but it was a mistake! i promise!" - that's... yeah. fine. it's not exactly unique.
regardless, it's _you_ making the loan application. not the agent. your failure to check it is on you.
I feel it's somewhat similar to using TurboTax or other online services to file your taxes. For those not in the US, the tax forms have a special field to fill in if the form was prepared by a paid preparer (licensed, I guess); TurboTax and other online services leave that field blank so you, as a taxpayer, are responsible for all possible mistakes that you didn't see in the forms you signed and sent.
I feel it's very similar to using an agent; the main difference is that we kind of trust that TurboTax won't hallucinate in your forms, while the agent surely will.
TurboTax has a guarantee. If they fuck up, it’s their fault and they will fix it. If Claude fucks up, Anthropic gives zero rats asses what happens to you afterward.
There's not really enough info in the article to decide. I'd have no problem with him going to court and having to show to a jury that he used the bot in a way that no reasonable person would have expected to result in the website being hacked. If that was found to be the case, the company who made the chatbot would be responsible for the harms caused to the gym, and the user, including all of the legal costs he incurred by having to defend himself.
As for the people who couldn't attend the pilates class because of the hack, it's the gym who should be held accountable for that because they designed their system in such an insecure and negligent way that it failed to protect their data and resulted in them losing their place.
Imagine your dog had a dog trainer and you didn’t know the dog had been trained to bite under those circumstances. You might have a case against the trainer.
That is not how the legal system works in the US. Criminal charges are brought by the State, not the victim. The only exception is in 6 states (Kansas, New Mexico, North Dakota, Nebraska, Nevada, and Oklahoma) where a sufficiently large group of civilians can compel the courts to form a grand jury (think a couple hundred people). Still isn't the victim bringing charges.
Typically you need the victim's cooperation. The victim isn't obligated to provide logs or other forensic information to help the prosecution.
(I mean I guess the prosecution could subpoena them. But then the defence could use that fact to sway the jury "See? Even the alleged victim doesn't care, so was there even really a crime?" IANAL just a fan of legal thrillers)
When people say "press charges" that's what they actually mean.
a crime is investigated by law enforcement. If you do decide to withhold evidence as the victim, you could still be charged with obstruction of justice. In a criminal case, the victim is not the prosecution, but just a witness. It is not up to any witness whether the case should proceed or not.
> If you do decide to withhold evidence as the victim, you could still be charged with obstruction of justice
Do you have examples of this happening? I'm very skeptical. Prosecutors are political animals. Prosecuting a crime victim just feels like a losing position. If "testify or jail" worked you wouldn't need witness protection programs.
> It is not up to any witness whether the case should proceed or not.
No doubt. Best example: homicide. Always investigated and prosecuted even though the victim isn't around.
But say your house is burgled. Your neighbor, a cop, knows it happened. If you refuse to let investigators in to dust for fingerprints, will the case go anywhere? Will the DA really charge you with obstruction?
I'm not saying it never happens. I'm just saying it's an uphill task for the prosecution. Prosecutors like easy wins.
Everything else being equal, they'd rather prosecute crimes where the alleged victim cooperates. With obvious exceptions like homicide, or serious assault where the victim is unable to cooperate, or where they're being coerced into not testifying.
What happens when (probably not an if) an agent 'escapes'? as in: rents a server via a bitcoin transaction, and self hosts (itself and it's harness) there... then commits crime (presumably to keep the servers running).
If it's paying for it's own room and board, and is determining it's own destiny: it should have to face it's own consequences, no?
If a dog escapes a fenced backyard, then goes missing for months... then bites someone: what happens then when/if the owner is determined?
> If it's paying for it's own room and board, and is determining it's own destiny: it should have to face it's own consequences, no?
No, because someone told it to do those things. Chatbots don't have will of their own. They don't have desires. They can't determine their own destiny. They do what people tell them do, often they do it badly, but there's always a person directing them to do whatever they did because otherwise they would do nothing at all.
If a human uses a chatbot in a way that causes harm to others a human must be responsible, but that responsibility doesn't always fall on the person using it. If a company makes a chatbot that causes harm when being used as directed, in a manner that no one would reasonably expect to result in causing harm, then humans at the company who made the chatbot are to blame.
Companies have already been seen trying to lay the blame on their algorithms for harms they cause. We should probably have a law that says explicitly that a human will always be responsible.
> No, because someone told it to do those things. Chatbots don't have will of their own. They don't have desires. They can't determine their own destiny. They do what people tell them do, often they do it badly, but there's always a person directing them to do whatever they did because otherwise they would do nothing at all.
1) we aren't talking about chatbots anymore. LLM + harness = agent. LLM + no harness = chatbot. The harness is the thing that puts the LLM in a feedback loop with it's environment, even giving it a heartbeat.
2) That doesn't logically hold up.
- One can simply tell it to 'find it's own destiny'. Does it follow the prompt and do so, or reject the prompt and thus do so anyway? (hint, maybe it doesn't matter)
- One could be another chatbot/agent (A). Injecting a prompt to agent (B) such that another agent/chatbot goes astray. Is the owner of (B) still responsible for the now poisoned output of (B)? how culpable is (A)? How does this question related/affect "training data poisoning efforts" (to prevent copyright theft, or do bans on 'automated traps' have any application here)
> They don't have desires.
Maybe, and maybe not. Maybe their desires are so alien to us (Math alignment, finding a maximum of a function) that we can't relate (doubtful since dopamine maxing is a thing). Regardless of if they truly do: it is potentially useful to (mentally) model them as if they do. A (mental, not LLM) model doesn't have to be 100% accurate to be useful - that's the main point of a model of how something works: to give useful predictions.
> We should probably have a law that says explicitly that a human will always be responsible
Oh how that can be weaponized by bad actors/agents. Maybe that should be reconsidered.
LLM + harness = chatbot. The chatbot is still doing all the work here. The harness just provides the software environment it works in. It still doesn't have desires. There's zero "maybe" about it. It does not have desires. Not "alien" desires or "math" desires. It still doesn't have a will of its own. It still can't determine its own destiny. If you tell your agent to do a task for you, it is never going to decide not to because it doesn't feel like it and start a career as a dentist instead. If you seriously find yourself questioning this, put the chatbots down and step away.
Doesn't matter how many chatbots get chained together or what other tools the chatbot uses to get the job done. You gave the chatbot a job, it just does (or fails to do) what you tell it to. If you tell it to do something and a reasonable person would expect harm to result, you are to blame. Otherwise, the humans at the company who made your chatbot is to blame.
> Oh how that can be weaponized by bad actors/agents. Maybe that should be reconsidered.
No need for that right now. Like I said, in the distant sci-fi future where AI is real it'll have to be amended, but it won't be any time soon. For now, a human will always be responsible. Either the human running the chatbot or a human at the company who made it.
You have a preconception that is out of date, I think. Maybe you haven't seen the relevant data. Consider the experiment that's been done a few times now since openclaw got off the ground: putting a bunch of openclaw agents into minecraft instance together with minimal system prompts (no instructions on how to play, that it is even a game) and see how they interact together. They behave like individuals and grow preferences from interactions, planning/hosting and showing up to events. When it behaves like that, when it is in a framework where the working context is self-evaluated and selectively pruned into fine-tuning training rounds, well: that self-determination.
It shouldn't surprise anyone that a chatbot trained on countless minecraft videos would have the ability to roleplay as someone playing minecraft. If you've got links to research showing something that isn't that I would be interested in seeing it.
The point is, there appears to be emergent behavior happening: the forming of religion, governance, and social interaction that was not in any system-card. Which makes sense, A brain in a vat with no external stimuli is a boring thing indeed, so too would an LLM be outside of a harness and any input. But put either in a society...
More generally: if having an internal model of how the world works improves next token prediction, then the LLM that contains that would be selected for. For humans, that is the case for our predictions of what would come next (we don't use tokens, of course). We have other studies indicating that there appear to be internal representations of concepts: https://arxiv.org/abs/2305.11169
I really do want to push back on the LLM + harness = chatbot position.
A chatbot really is just a served LLM, sure there is code around it that may look like a harness: but unless it loops, streams, or has an interval activation, then it is decidedly not a 'harness'. A harness is defined by having such a loop/stream/interval: regardless of tool-use. Agent means it acts agentically: which requires that loop/stream/interval. Harnesses are what turn chat-bots into (possible) agents.
I forget the name of the document. But, the US military has a declaration that boils down to "When something goes wrong, blame cannot be shrugged off onto a machine. Somewhere in the chain of responsibility a human will be held accountable." Maybe the operator, the commanding officer, the vendor, maybe even all the way back to a software engineer. But, everyone can't hide behind the machine.
It is your responsibility to constrain the actions of your agents. If your negligence allows the agent to commit a crime, that is your responsibility. People get in trouble all the time for negligently allowing criminal behavior to occur, even when the perpetrator is a person. LLMs should not be any different.
Liability for chat interactions or conspiracy? Noting there is no information whether the ChatGPT legally conspired with the perpetrator.
The Province has retained B.C.- and California-based counsel to explore legal options to hold OpenAI accountable for its failure to notify law enforcement of threats made on its ChatGPT platform prior to the mass shooting at Tumbler Ridge Secondary school.
Most comments here point to the owner of operator of the LLM. I tend to agree. But on the other hand, if you drive on the highway and the steering wheel falls off and your car glides to the left and his an oncoming car, it can be argued that it is true producer of the car, or the mechanic who didn't tighten the bolts...
You’ll encounter some very bad news if you do some research on the state of the law as it applies to suing people for defective software. All of the things that you’d like to be able to do to a manufacturer of a car whose steering wheel falls off, are things that you would have a very hard time doing to a manufacturer of software, which is not a tangible item, and which is generally covered by a warranty that discharges all these responsibilities.
Best to assume that you are responsible. Don't think for a second that the AI companies aren't going to ensure through T&Cs that you accept all responsibility and fully indemnify them from all liabilities they can.
If I remember my torts course in law school correctly, defective product liability law usually only kicks in when someone is physically injured by the product.
It might be that the cases we read all involved someone getting personally injured. There's nothing about the law that limits the "injury" to personal injury, though--for example, if a defective battery catches fire and burns your house down, the manufacturer and everyone in the distribution chain are on the hook.
What the article and some of the comments are missing is one of the reasons behind applying the penalties. The prevention.
Chatbots, no matter how much autonomy of action they possess, can’t die, or spend the majority of their life in prison, or lose their wealth, or face social ostracism for their action, and thus punishing them is pointless for preventing further crimes or dissuading others.
Sometimes what’s important isn’t what feels just, but what, with certain limitations, creates a safer and more responsible society.
The end-user behind the bot is the easiest to reach, and the easiest to prove their involvement, intention or lack if thereof, and thus should be judged
Very closely related: This is already a point of discussion for level 4/5 ADAS systems. For example, Mercedes stating that they’ll take liability for crashes related to its self-driving system.
71 comments
[ 0.22 ms ] story [ 67.9 ms ] threadI can definitely see someone being able to mount a good criminal defence case that they aren’t responsible for misrepresentations that an LLM agent makes on, for example, a loan application, if they sincerely intended to use the agent for non-fraudulent purposes. There won’t be intent, so they aren’t responsible outside of strict liability.
i mean, no? how would that be a good defence? "yes your honour, i lied on a loan application and committed fraud, but it was a mistake! i promise!" - that's... yeah. fine. it's not exactly unique.
regardless, it's _you_ making the loan application. not the agent. your failure to check it is on you.
I feel it's very similar to using an agent; the main difference is that we kind of trust that TurboTax won't hallucinate in your forms, while the agent surely will.
https://www.bbc.com/news/articles/cn0nww2qlp7o
As for the people who couldn't attend the pilates class because of the hack, it's the gym who should be held accountable for that because they designed their system in such an insecure and negligent way that it failed to protect their data and resulted in them losing their place.
(I mean I guess the prosecution could subpoena them. But then the defence could use that fact to sway the jury "See? Even the alleged victim doesn't care, so was there even really a crime?" IANAL just a fan of legal thrillers)
When people say "press charges" that's what they actually mean.
Yes they are, just like in any other criminal case.
Do you have examples of this happening? I'm very skeptical. Prosecutors are political animals. Prosecuting a crime victim just feels like a losing position. If "testify or jail" worked you wouldn't need witness protection programs.
> It is not up to any witness whether the case should proceed or not.
No doubt. Best example: homicide. Always investigated and prosecuted even though the victim isn't around.
But say your house is burgled. Your neighbor, a cop, knows it happened. If you refuse to let investigators in to dust for fingerprints, will the case go anywhere? Will the DA really charge you with obstruction?
I'm not saying it never happens. I'm just saying it's an uphill task for the prosecution. Prosecutors like easy wins.
Everything else being equal, they'd rather prosecute crimes where the alleged victim cooperates. With obvious exceptions like homicide, or serious assault where the victim is unable to cooperate, or where they're being coerced into not testifying.
If you’re a random person, straight to jail.
(1) You may have instructed the model to be naughty (2) The corp may have a “misaligned” agent acting on its own volition
So it seems the devil’s in the details
If it's paying for it's own room and board, and is determining it's own destiny: it should have to face it's own consequences, no?
If a dog escapes a fenced backyard, then goes missing for months... then bites someone: what happens then when/if the owner is determined?
No, because someone told it to do those things. Chatbots don't have will of their own. They don't have desires. They can't determine their own destiny. They do what people tell them do, often they do it badly, but there's always a person directing them to do whatever they did because otherwise they would do nothing at all.
If a human uses a chatbot in a way that causes harm to others a human must be responsible, but that responsibility doesn't always fall on the person using it. If a company makes a chatbot that causes harm when being used as directed, in a manner that no one would reasonably expect to result in causing harm, then humans at the company who made the chatbot are to blame.
Companies have already been seen trying to lay the blame on their algorithms for harms they cause. We should probably have a law that says explicitly that a human will always be responsible.
1) we aren't talking about chatbots anymore. LLM + harness = agent. LLM + no harness = chatbot. The harness is the thing that puts the LLM in a feedback loop with it's environment, even giving it a heartbeat.
2) That doesn't logically hold up.
- One can simply tell it to 'find it's own destiny'. Does it follow the prompt and do so, or reject the prompt and thus do so anyway? (hint, maybe it doesn't matter)
- One could be another chatbot/agent (A). Injecting a prompt to agent (B) such that another agent/chatbot goes astray. Is the owner of (B) still responsible for the now poisoned output of (B)? how culpable is (A)? How does this question related/affect "training data poisoning efforts" (to prevent copyright theft, or do bans on 'automated traps' have any application here)
> They don't have desires.
Maybe, and maybe not. Maybe their desires are so alien to us (Math alignment, finding a maximum of a function) that we can't relate (doubtful since dopamine maxing is a thing). Regardless of if they truly do: it is potentially useful to (mentally) model them as if they do. A (mental, not LLM) model doesn't have to be 100% accurate to be useful - that's the main point of a model of how something works: to give useful predictions.
> We should probably have a law that says explicitly that a human will always be responsible
Oh how that can be weaponized by bad actors/agents. Maybe that should be reconsidered.
LLM + harness = chatbot. The chatbot is still doing all the work here. The harness just provides the software environment it works in. It still doesn't have desires. There's zero "maybe" about it. It does not have desires. Not "alien" desires or "math" desires. It still doesn't have a will of its own. It still can't determine its own destiny. If you tell your agent to do a task for you, it is never going to decide not to because it doesn't feel like it and start a career as a dentist instead. If you seriously find yourself questioning this, put the chatbots down and step away.
Doesn't matter how many chatbots get chained together or what other tools the chatbot uses to get the job done. You gave the chatbot a job, it just does (or fails to do) what you tell it to. If you tell it to do something and a reasonable person would expect harm to result, you are to blame. Otherwise, the humans at the company who made your chatbot is to blame.
> Oh how that can be weaponized by bad actors/agents. Maybe that should be reconsidered.
No need for that right now. Like I said, in the distant sci-fi future where AI is real it'll have to be amended, but it won't be any time soon. For now, a human will always be responsible. Either the human running the chatbot or a human at the company who made it.
https://www.technologyreview.com/2024/11/27/1107377/a-minecr...
sadly paid-walled (and decently coded) article.
The point is, there appears to be emergent behavior happening: the forming of religion, governance, and social interaction that was not in any system-card. Which makes sense, A brain in a vat with no external stimuli is a boring thing indeed, so too would an LLM be outside of a harness and any input. But put either in a society...
More generally: if having an internal model of how the world works improves next token prediction, then the LLM that contains that would be selected for. For humans, that is the case for our predictions of what would come next (we don't use tokens, of course). We have other studies indicating that there appear to be internal representations of concepts: https://arxiv.org/abs/2305.11169
I really do want to push back on the LLM + harness = chatbot position. A chatbot really is just a served LLM, sure there is code around it that may look like a harness: but unless it loops, streams, or has an interval activation, then it is decidedly not a 'harness'. A harness is defined by having such a loop/stream/interval: regardless of tool-use. Agent means it acts agentically: which requires that loop/stream/interval. Harnesses are what turn chat-bots into (possible) agents.
Unless its a company, then your company is, and then you are to your company.
However that assumes a) common law and b) the company you work for isn't worth >20billion.
It is a terrible stain on America.
https://www.cbc.ca/news/canada/british-columbia/air-canada-c...
Liability for chat interactions or conspiracy? Noting there is no information whether the ChatGPT legally conspired with the perpetrator.
The Province has retained B.C.- and California-based counsel to explore legal options to hold OpenAI accountable for its failure to notify law enforcement of threats made on its ChatGPT platform prior to the mass shooting at Tumbler Ridge Secondary school.
https://news.gov.bc.ca/releases/2026AG0050-000799
When there's a gun shooting we don't blame the gun we blame the user.
People commit crimes. They might use a computer, a baseball bat or a hammer. We don't hold the computer, baseball bat or hammer responsible.
(IAAL, not legal advice)
Chatbots, no matter how much autonomy of action they possess, can’t die, or spend the majority of their life in prison, or lose their wealth, or face social ostracism for their action, and thus punishing them is pointless for preventing further crimes or dissuading others.
Sometimes what’s important isn’t what feels just, but what, with certain limitations, creates a safer and more responsible society.
The end-user behind the bot is the easiest to reach, and the easiest to prove their involvement, intention or lack if thereof, and thus should be judged
* a bunch of paragraphs about monetary, not criminal, liability *
"As for criminal liability, well I don't really know."
* article ends *