> “The AI hallucinated. I never asked it to do that.”
> That’s the defense. And here’s the problem: it’s often hard to refute with confidence.
Why is it necessary to refute it at all? It shouldn't matter, because whoever is producing the work product is responsible for it, no matter whether genAI was involved or not.
I feel like you have missed the point of this. It isn't to completely absolve the user of liability, it's to prove malice instead of incompetence.
If the user claims that they only authorized the bot to review files, but they've warranted the bot to both scan every file and also send emails to outside sources, the competitors in this case, then you now have proof that the user was planning on committing corporate espionage.
To use a more sane version of an example below, if your dog runs outside the house and mauls a child, you are obviously guilty of negligence, but if there's proof of you unleashing the dog and ordering the attack, you're guilty of murder.
If an employee does something during his employment, even if he wasn't told directly to do it, the company can be held vicariously liable, how is this any different?
This is some absolute BS. In the current day and age you are 1000% responsible for the externalities of your use of AI.
Read the terms and conditions of your model provider. The document you signed, regardless if you read or considered it, explicitly removes any negative consequences being passed to the AI provider.
Unless you have something equally as explicit, e.g. "we do not guarantee any particular outcome from the use of our service" (probably needs to be significantly more explicitly than that, IANAL) all responsibility ends up with the entity who itself, or it's agents, foists unreliable AI decisions on downstream users.
Remember, you SIGNED THE AGGREMENT with the AI company the explicitly says it's outputs are unreliable!!
And if you DO have some watertight T&C that absolves you of any responsibility of your AI-backed-service, then I hope either a) your users explicitly realize what they are signing up for, or b) once a user is significantly burned by your service, and you try to hide behind this excuse, you lose all your business
What a stupid article from someone that has no idea when liability attaches.
It is the burden of a defendant to establish their defense. A defendant can't just say "I didn't do it". They need to show they did not do it. In this (stupid) hypothetical, the defendant would need to show the AI acted on its own, without prompting from anyone, in particular, themselves.
Licensed professionals are required to review their work product. It doesn't matter if the tools they use mess up--the human is required to fix any mistakes made by their tools. In the example given by the blog, the financial analyst is either required to professional review their work product or is low enough that someone else is required to review their work product. If they don't, they can be held strictly liable for any financial losses.
However, this blog post isn't about AI Hallucinations. It's about the AI doing something else separate from the output.
And that's not a defense either. The law already assigns liability in situations like this: the user will be held liable (or more correctly: their employer, for whom the user is acting as an agent). If they want to go after the AI tooling (i.e., an indemnification action) vendor the courts will happily let them do so after any plaintiffs are made whole (or as part of an impleader action).
What problem is this guy trying to solve?
Sorry, but in the end, someone's gonna have to be responsible and it's not gonna be a computer program. Someone approved the program's use, it's no different to any other software. If you know agent can make mistakes then you need to verify everything manually, simple as.
If one of my reports came to me with that defense, I'd write them up twice. Once for whatever they did wrong, and once for insulting my intelligence and wasting my time with that "defense".
On the contrary, if they just owned up to it, chances are I wouldn't even write them up once.
IMO everyone is missing the point of this thing. It's not an auth system or security boundary, it doesn't provide any security guarantees whatsoever, it doesn't do anything. The entire point is to cover a company's derriere should their agentic security apparatus (or lack thereof) fail to prevent malicious prompt injection etc.
This way, they can avoid being legally blamed for stuff-ups and instead scapegoat some hapless employee :-) using cryptographic evidence the employee "authorized" whatever action was taken
It's the same thing as saying "It's not my fault, it's a bug in the spreadsheet". The human is ultimately responsible. Heck, it's the same thing as someone saying "the computer made a mistake". Computers don't make mistakes. People do. The hallucination defense is not a defense at all and this is a non-issue. There's nothing to talk about here.
And if a professional wants to delegate their job to non-deterministic software they're not professionals at all. This overreliance on LLMs is going to have long-term consequences for society.
Anyone using AI tools should assume that at least 20% of what it produces will be wrong: Missing details, inventing things, basic mistakes.
Use AI if being 80% right quickly is fine. Otherwise if you have to do the analysis anyway because accuracy is critical, there's little point to the AI - its too unreliable.
I'm not sure I understand the point of this. If mistakes (or lies) aren't tolerable in the output, then whoever ran the agent should be responsible for reviewing the output and the resulting actions. I don't see how LLMs can be "responsible" for anything because they don't think in the way we do nor do they have motives in the way we do.
The "Hallucination Defense" isn't a defense because end of the day, if you ran it, you're responsible, IMO.
1. I vaguely recall that in the early days of Windows, the TOS explicitly told users they were not to use it for certain use-cases and the wording was something like "we don't warranty windows to be good for anything, but we EXPLICITLY do not want you to use Windows to do nuclear foo".
I expect that if the big LLM vendors aren't already doing this, they soon will. Kind of like how Fox News claims that some of heads on the screen are not journalists at all but merely entertainers (and you wouldn't take anything they say seriously ergo we're not responsible for stuff that happens as a result of people listening to our heads on the screen).
2. IANAL but I believe that in most legal systems, responsibility requires agency. Leaving aside that we call these things "agents" (which is another thing I suspect will change in the marketing), they do not have agency. As a result they must be considered tools. Tools can be used according to the instructions/TOS, or not. If not - whatever it does is down to you. If used within guidelines - it's the manufacturer.
So my conclusion is that the vendors - who have made EPIC bets on getting this tech into the hands of as many folks as possible and making it useful for pretty much anything you can think of - will be faced with a dilemma. At the moment, it seems like they believe that (just like the infamous Ford bean counters) the benefits of not restricting the TOS will far outweigh any consequences from bad things happening. Remains to be seen.
This is an advertisement for a 'tenuo warrant'. So, I read its document[0]. Put simply, it works like this:
1. A person orders an AI agent to do A.
2. The agent issues a tenuo warrant for doing A.
3. The agent can now only use the tool to perform A.
The article is about that 'warrant' can now be used in case of an incident because it contains information such as 'who ordered the task' and 'what authority was given'.
I get the idea. This isn't about whether a person is responsible or not(because of course they are). It's more about whether it was intentional.
However... wouldn't it be much easier to just save the prompt log? This article is based entirely on "But the prompt history? Deleted."(from the article) situation.
Shouldn't all agentic actions with meaningful outputs of importance like moving $48,000 simply be required to terminate in a human designed or verified output with a human in the loop attestation.
Eg a list of transactions that isn't AI generated where the only actions that actually move money must operate on the data displayed in the human designed page.
A human looks at this and says yes that is acceptable and becomes reasonable for that action.
26 comments
[ 2.8 ms ] story [ 48.8 ms ] thread> That’s the defense. And here’s the problem: it’s often hard to refute with confidence.
Why is it necessary to refute it at all? It shouldn't matter, because whoever is producing the work product is responsible for it, no matter whether genAI was involved or not.
If the user claims that they only authorized the bot to review files, but they've warranted the bot to both scan every file and also send emails to outside sources, the competitors in this case, then you now have proof that the user was planning on committing corporate espionage.
To use a more sane version of an example below, if your dog runs outside the house and mauls a child, you are obviously guilty of negligence, but if there's proof of you unleashing the dog and ordering the attack, you're guilty of murder.
Read the terms and conditions of your model provider. The document you signed, regardless if you read or considered it, explicitly removes any negative consequences being passed to the AI provider.
Unless you have something equally as explicit, e.g. "we do not guarantee any particular outcome from the use of our service" (probably needs to be significantly more explicitly than that, IANAL) all responsibility ends up with the entity who itself, or it's agents, foists unreliable AI decisions on downstream users.
Remember, you SIGNED THE AGGREMENT with the AI company the explicitly says it's outputs are unreliable!!
And if you DO have some watertight T&C that absolves you of any responsibility of your AI-backed-service, then I hope either a) your users explicitly realize what they are signing up for, or b) once a user is significantly burned by your service, and you try to hide behind this excuse, you lose all your business
It is the burden of a defendant to establish their defense. A defendant can't just say "I didn't do it". They need to show they did not do it. In this (stupid) hypothetical, the defendant would need to show the AI acted on its own, without prompting from anyone, in particular, themselves.
Licensed professionals are required to review their work product. It doesn't matter if the tools they use mess up--the human is required to fix any mistakes made by their tools. In the example given by the blog, the financial analyst is either required to professional review their work product or is low enough that someone else is required to review their work product. If they don't, they can be held strictly liable for any financial losses.
However, this blog post isn't about AI Hallucinations. It's about the AI doing something else separate from the output.
And that's not a defense either. The law already assigns liability in situations like this: the user will be held liable (or more correctly: their employer, for whom the user is acting as an agent). If they want to go after the AI tooling (i.e., an indemnification action) vendor the courts will happily let them do so after any plaintiffs are made whole (or as part of an impleader action).
> A computer must never make a management decision, because a computer cannot be held accountable.
With no amount of detailed logging makes "the AI did it" a valid excuse.
It's just a tool.
It's like blaming a loose bolt in a Boeing 737 on "screwdriver did it".
On the contrary, if they just owned up to it, chances are I wouldn't even write them up once.
but the person who turned it on can
simple as
This way, they can avoid being legally blamed for stuff-ups and instead scapegoat some hapless employee :-) using cryptographic evidence the employee "authorized" whatever action was taken
AI is just branding. At the end of the day it's still just people using computer software to do stuff.
There is going to be a person who did a thing at the end of the day -- either whoever wrote the software or whoever used the tool.
The fact that software inexplicably got unreliable when we started stamping "AI" on the box shouldn't really change anything.
And if a professional wants to delegate their job to non-deterministic software they're not professionals at all. This overreliance on LLMs is going to have long-term consequences for society.
Use AI if being 80% right quickly is fine. Otherwise if you have to do the analysis anyway because accuracy is critical, there's little point to the AI - its too unreliable.
The "Hallucination Defense" isn't a defense because end of the day, if you ran it, you're responsible, IMO.
Or perhaps scapegoating at scale.
https://news.ycombinator.com/item?id=43877301 - 398 comments
https://news.ycombinator.com/item?id=41891694 - 308 comments
1. I vaguely recall that in the early days of Windows, the TOS explicitly told users they were not to use it for certain use-cases and the wording was something like "we don't warranty windows to be good for anything, but we EXPLICITLY do not want you to use Windows to do nuclear foo".
I expect that if the big LLM vendors aren't already doing this, they soon will. Kind of like how Fox News claims that some of heads on the screen are not journalists at all but merely entertainers (and you wouldn't take anything they say seriously ergo we're not responsible for stuff that happens as a result of people listening to our heads on the screen).
2. IANAL but I believe that in most legal systems, responsibility requires agency. Leaving aside that we call these things "agents" (which is another thing I suspect will change in the marketing), they do not have agency. As a result they must be considered tools. Tools can be used according to the instructions/TOS, or not. If not - whatever it does is down to you. If used within guidelines - it's the manufacturer.
So my conclusion is that the vendors - who have made EPIC bets on getting this tech into the hands of as many folks as possible and making it useful for pretty much anything you can think of - will be faced with a dilemma. At the moment, it seems like they believe that (just like the infamous Ford bean counters) the benefits of not restricting the TOS will far outweigh any consequences from bad things happening. Remains to be seen.
1. A person orders an AI agent to do A.
2. The agent issues a tenuo warrant for doing A.
3. The agent can now only use the tool to perform A.
The article is about that 'warrant' can now be used in case of an incident because it contains information such as 'who ordered the task' and 'what authority was given'.
I get the idea. This isn't about whether a person is responsible or not(because of course they are). It's more about whether it was intentional.
However... wouldn't it be much easier to just save the prompt log? This article is based entirely on "But the prompt history? Deleted."(from the article) situation.
[0]: https://tenuo.dev/concepts
Eg a list of transactions that isn't AI generated where the only actions that actually move money must operate on the data displayed in the human designed page.
A human looks at this and says yes that is acceptable and becomes reasonable for that action.