He should have lost his license after keeping up. There should be a serious fine (at least 10x the time of other people they have wasted) and loss of professional licenses upon the first repeat of the offense. You want to use AI to discover similar cases? You want to use it for ideas on how to structure your line of defense? Good. The moment you submit it's output under your name though, expect repercussions.
surely not in principle because anything that's been written may be read out and spoken (and viceversa, anything is potentially transcribe-able)
nonetheless, when push comes to shove there's a real difference: written stuff (text) is readily usable as evidence but things only ever spoken are not.
Yes, but entirely unenforced. The bar acts like an old-time guild designed to protect insiders. On paper, it's intended to be self-policing (file a complaint with the Board of Bar Overseers and see it go into the circular file).
Having experienced family court firsthand as a clerk, and as a member of the ethics committee, I can tell you that the bar absolutely does care if a lawyer lies in court.
It is a sanctionable offense. At the low end, it is a fine in the thousands. The usual sanction is a suspension of the lawyer's license to practice for several months. At the upper end, the penalty is revocation of one's license to practice law.
Why, then, is it most common for a judge to fine a lawyer $999 if they bother to issue sanctions against them? A laughable and seemingly arbitrary hand-slap for an attorney making $550+/hour! But it's not arbitrary because a $1000+ fine would result in mandatory reporting to the bar.
A person should spend decades of their life studying law and practicing it and then have their career ended because they didn't realize that this new tool everyone said was so amazing was fundamentally and conceptually different from every search engine he'd seen before and (unlike what he otherwise would have used, Westlaw) comes up with random made up but very plausible sounding precedents when you ask it for legal research?
Embarrassed, called out, fined? Of course. But that's not a serious argument. Unless you think someone should be legally prohibited from ever being paid as a programmer for the rest of their life for blowing up a server or failing to maintain a backup system or something
This is a mistake, and a pretty easy to understand one. It's not stealing client's money or tampering with a witness or something.
He had three others doing the research preparing the brief, that means especially the citations that were fake. He did not realize they didn’t do the due diligence to check that the citations were correct, let alone that they were not completely invented.
But he was the person on the hook with the judge. This amounts to filing false statements and that is well known issue that we don’t have to invent new penalties for.
As the licensed member of the bar, his one job is to be responsible for the output of his team. If he's not doing that, then he doesn't need to be a lawyer. His staff can file their BS directly to the court.
You have no problem with $$$$ experienced lawyer billing clients for $$$$ services carried out by $$ inexperienced flunkies who cheat by using a $ “AI” to do their job?
I found it astounding that the judge accepted the lawyer's excuse that two recent law graduates were responsible for the AI citations. No recent graduate would use AI in this manner. Everyone deserves a second chance, but I would have increased the fine to $10,000 for this clear failure to accept responsibility.
Fines for lawyers are set at slap-on-the-wrist levels. Violating legal ethics almost always carries higher expected benefit than expected penalty, which is why most lawyers are crooks (even ones who don't want to be). It's impossible to be a competitive lawyer in many domains while still following legal ethics.
In this case, 10% odds of getting caught · $2000 = $200 per violation, generously, so less than one billable hour.
Why would no recent graduate use AI in this manner?
I'd imagine that a lot of law schools have started warning students against it, but in other fields there's often quite a pronounced lag between changes in the field and changes in the curriculum. Even if the best schools have adapted, I'd be shocked if there aren't a bunch that haven't.
Recent graduates would have at least played around with AI enough to notice how it can be incorrect or at least heard about this from peers. An older lawyer less familiar with technology is much less likely to have this experience. Furthermore, the lawyer blamed two recent graduates. I find it unlikely that two people together would make this mistake.
Any recommendations? The public seems to actually understand what this means although it’s just more anthropomorphization of a random bullshit generator.
Well, both the true and false outputs are equally random bullshit from a machine. "Hallucination" is just the word that caught on to describe random bullshit outputs that are false.
Even "bullshit" implies something like a mind, with intent to deceive. It should be more like noise, aberrations, incorrectly extrapolated filler material.
Bullshit is not deception. There is no intent to convince. Bullshit is even less than that.
Bullshit is just blowing hot hair to create a buffer between reality and it's consequences.
I guess that's fewer syllables than "hallucination."
I'm not sure how I feel about the term "hallucination" as it's applied to AI. Since you seem strongly opposed to it, let me ask you this long-winded half-question:
People understand computer things by creating analogies to the physical world - just look at the "Desktop" motif. "Folders" and "Files" too, for that matter. It seems to me that anthropomorphization would fit under that umbrella, though you may disagree. How do you feel about computer anthropomorphization in general? Is there something about "hallucination" that's particularly offensive?
How about we stop with all the Nonsense around calling it "temperature" like it's a sick baby and call it RAND cause that's what it is.
The PT Barnum levels of bullshit around ML (see we have a term that isnt using artificial or intelligence) has gotten old. Sam Altman is the next Elizabeth Holmes.
I came here to suggest the same thing. This "hallucination" soft euphemism seems to be the tech press's way to continue to write positively about defective AI software while lightheartedly joking about how it sometimes does an oopsie.
If I ask a software to write about a well known fact or historical event and it just makes stuff up, it's not simply hallucinating. It's defective.
The thing is, it isn't a defect. People misunderstand that there is no practical difference between a "hallucinated" result and a real one, as far as an LLM is concerned. It doesn't reason or calculate beyond matching tokens, it has no deeper contextual understanding of truth or correctness beyond statistical likelihood. Hallucinations are the result of the LLM doing exactly what it's designed to do, exactly the way its designed to do it.
The defect isn't in the software, but in people expecting these things to operate the way AIs in sci-fi do, or who believe that because they can produce coherent results in natural language, they must be sentient and self-aware.
It's a defect from the point of view of user expectations. When Intel's floating point bug was in the news, I remember a small number of people claiming it was not a defect because the chip was just doing what it was designed to do: Yea, it was designed in such a way that it could produce incorrect results. In other words a bug!
I'm sure AI companies will get very good at explaining away these defects with various forms of "aCkShUaLlY" but when your marketing materials say you made a box that takes a prompt and answers it, and it answers incorrectly, what else is it than a defect?
The problem, in that case, exists between the keyboard and chair.
Floating point math is inherently inaccurate, and no programmer using it would expect perfect precision and call it a defect not to get it. You have to understand how floating point works and take that inaccuracy into account. As a result there are some applications for which using floats is simply a bad idea. No one sane is doing real money calculations with floats.
The same goes for LLMs. Hallucination is fundamental to the model. We're going to have to realize that there are many tasks for which AI simply isn't well suited. And we're going to have to get over this persistent delusion that humans are categorically worse than AI at everything. A paralegal doing research would probably not simply fabricate cases and cites whole cloth. That's not how most humans work. Humans are capable of knowing when they don't know something, AI is not.
But we've decided, for whatever reason, that AI is perfectly trustworthy. That's going to keep biting us in the ass until we learn.
I don't think your memory of the Intel bug[1] is correct. It had nothing to do with the inherent precision problems of floating point representation.
EDIT: Another way to put it: If I sold a calculator that claimed to do math, but in the fine print I said "Actually, it just makes up answers by some means we don't fully understand, and somehow most of the time it comes up with the right answer." That doesn't mean that incorrect answers are suddenly not defects.
Yea it fundamentally changes the error bounding calculation. We explicitly learned about this in EE where they covered 754 and other similar things.
So there's a lot of rigor on containing exactly how big your errors can be in floating point calculations, and that old intel bug made it (rarely) invalid.
Fit for purpose... most of the time, except when it isnt then Oops... Lets color in the failure with a human term "hallucination" cause "we can't really fix it".
Sugar coating the fact that it is defective (defined: imperfect or faulty.) isnt changing things.
Your explanation is correct, it's defective by design.
Instead of ‘hallucinations’, try ‘samplings from the model that happen not to be sufficiently reminiscent of reality’. Of course, it’s a little bit less catchy. But that’s the problem with catchiness — it sticks regardless of its truth.
The fact that ‘correct’ outputs are treated as if they’re the product of an in-any-way-different process to the ‘hallucinated’ ones is the problem.
> The fact that ‘correct’ outputs are treated as if they’re the product of an in-any-way different process to the ‘hallucinated’ ones is the problem.
Also this particular context just makes it easier to notice, compared a 5000 word generated coherent-word-salad that equally wrong, but across the 5000 words.
I forget where I originally heard this idea, but I always explain to people that LLMs are (affectionately) "bullshitters." Terms like "lying" or "hallucinating" imply that it's trying to tell the truth, but actually it doesn't care if what it says is true or not at all save for the fact that true text is slightly more plausible than false text.
No, it's good that the public understands that AIs are wrong so regularly that we need a special word dedicated to this one specific manner in which they're wrong.
Generative AI output is becoming inextricably associated with this word, and that's not a bad thing to keep people aware of.
> No, it's good that the public understands that AIs are wrong so regularly
_compared to what_, exactly. Compared to a google search? Compared to asking a random person? Compared to wikpedia? New York Times journalists?
Any of those things are wrong _very_ frequently. It's such an uninteresting thing to call out every time an AI is wrong, when it is right about things so frequently that people don't bother to notice how amazing it is that it gets anything correct about the world at all.
The whole point is that it's a qualitatively different kind of failure, it's not quantitative. Wikipedia doesn't hallucinate. It can be wrong, sure, but it doesn't do what LLMs do when they go off the rails. So there should be a word that applies to LLM output but not a person or article that's simply wrong.
Why should there be different words for different concepts? I don't understand the question. We already use different terms for "lying" and "mistaken". We've invented a new way to be wrong and calling it something different conveys more nuance than just calling it "wrong".
I guess I don't understand the difference between a LLM "hallucinating" by probabilistically having chosen the wrong output given a certain input pattern, vs a human doing the same thing and just being "wrong". (But to be fair, this could just be my own lack of understanding about how LLMs and human brains work!)
I've certainly made that class of error myself, when I assumed that something followed a similar pattern (like in math, or writing & grammar, or coding) when it actually didn't.
I've also doubled-down on those errors when I tried to double-check my work, believing myself to have misapplied some intermediate step rather than having taken an entirely wrong approach to begin with.
I think the "why" here is "why are we assuming this failure mode is unique to LLMs and deserves novel terminology".
Neither of us understands how the human brain works, but both of us understand how synonyms work.
We could find out 800 years from now that human brains really do work exactly the same as LLMs do, but it wouldn't change the fact that today LLMs and humans in practice regularly manifest their respective mistakes in very different ways.
For example I don't have to worry about you making sense but then turning on a dime saying things like "Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay." or "Subtitles by the Amara.org community", which are both examples of OpenAI hallucinations I encountered today.
We can call that type of stupidity its own word, different from the types of mistakes you described making, just like we have many different words to convey the concept of "wet".
The fact that LLMs are probably not just baby people give an even greater justification to use different terminology for them.
Synonyms thrive in our languge with sometimes a hair's breadth of difference in nuance so it's silly to let optimism about tech deny this.
I think that's a fair take! Even if one day we could look back and say, "In the early 21st century, people thought AI would 'hallucinate' and didn't always trust them. Today we know better, and hallucinations were actually just ______..." well, it's still a useful concept in the meantime!
The goal of anyone contributing something useful shouldn't be to immortalise one's name. You, by defending this practice, give yourself away by having similar ambitions.
Intractable model error that's elemental to the approach won't get you any funding though.
Anthropomorphizing statistical learning is how you build a hype machine to cash out people with zero handle on the subject. See the comment below about "AI judges" and "true justice". Just like early electricity, all people see is magic.
If we wanted justice we'd have an inquisitive system not an adversarial one.
At some point the government will decide it's best to pay a private company to administer computerized court systems. Private companies already can collect evidence, process evidence, arrest people, detain suspects long-term (Jail) and after court hold the convicts; why not let them do this one step as well?
With an open-source system developed and trained on our codified laws, presumably, such a system would be WAY less arbitrary and more benevolent than the average judge. HN clearly hasn't experienced the family law system in America.
There was a similar incident last year [0], but in that case the lawyer actually doubled down on the fake cases and, when pressed to produce evidence of the fake cases, submitted screenshots of them asking ChatGPT to confirm that the cases were real.
At least this one had the good sense to apologize instead!
Congrats for having judges who actually read the citations. As a lawyer in Germany, I find that most judges will barely read the body of a brief, let alone look up citations.
I am not sure how is this news. If a lawyer submits filings with non-factual things in them, will he get fined? - Yes. How should this be different when using AI tools? Are AI tools statistical in nature and will produce incorrect answers sometimes? - For sure. Therefore, proceed with caution and verify the entire filing, as you are the one signing it.
83 comments
[ 3.6 ms ] story [ 157 ms ] threadto speak is not equivalent to writting
surely not in principle because anything that's been written may be read out and spoken (and viceversa, anything is potentially transcribe-able)
nonetheless, when push comes to shove there's a real difference: written stuff (text) is readily usable as evidence but things only ever spoken are not.
Spoken with witnesses? Most definitely.
Those "rules" are often seen as advisory (or less) though, as they're commonly completely ignored.
It is a sanctionable offense. At the low end, it is a fine in the thousands. The usual sanction is a suspension of the lawyer's license to practice for several months. At the upper end, the penalty is revocation of one's license to practice law.
One recent high-profile example of such a consequence-avoiding fine is THIS VERY CASE: https://www.washingtonpost.com/technology/2023/11/16/chatgpt...
A person should spend decades of their life studying law and practicing it and then have their career ended because they didn't realize that this new tool everyone said was so amazing was fundamentally and conceptually different from every search engine he'd seen before and (unlike what he otherwise would have used, Westlaw) comes up with random made up but very plausible sounding precedents when you ask it for legal research?
Embarrassed, called out, fined? Of course. But that's not a serious argument. Unless you think someone should be legally prohibited from ever being paid as a programmer for the rest of their life for blowing up a server or failing to maintain a backup system or something
This is a mistake, and a pretty easy to understand one. It's not stealing client's money or tampering with a witness or something.
2. He didn't check the results against official, established database.
3. He got caught, lied about it, and then did it all again.
4. The legal bar for good reason, has a bigger ethical standard than the average business. These are officers of the Court.
This is equivalent to a doctor prescribing a patient to take medication from a 3P seller FURODNZ on Amazon.com
But he was the person on the hook with the judge. This amounts to filing false statements and that is well known issue that we don’t have to invent new penalties for.
As the licensed member of the bar, his one job is to be responsible for the output of his team. If he's not doing that, then he doesn't need to be a lawyer. His staff can file their BS directly to the court.
The second one allegedly only quickly checked the examples provided by his associates.
In this case, 10% odds of getting caught · $2000 = $200 per violation, generously, so less than one billable hour.
I'd imagine that a lot of law schools have started warning students against it, but in other fields there's often quite a pronounced lag between changes in the field and changes in the curriculum. Even if the best schools have adapted, I'd be shocked if there aren't a bunch that haven't.
This seems like classic boomer behavior.
I'm not sure how I feel about the term "hallucination" as it's applied to AI. Since you seem strongly opposed to it, let me ask you this long-winded half-question:
People understand computer things by creating analogies to the physical world - just look at the "Desktop" motif. "Folders" and "Files" too, for that matter. It seems to me that anthropomorphization would fit under that umbrella, though you may disagree. How do you feel about computer anthropomorphization in general? Is there something about "hallucination" that's particularly offensive?
Bugs, Defects and "not fit for production".
How about we stop with all the Nonsense around calling it "temperature" like it's a sick baby and call it RAND cause that's what it is.
The PT Barnum levels of bullshit around ML (see we have a term that isnt using artificial or intelligence) has gotten old. Sam Altman is the next Elizabeth Holmes.
</rant>
If I ask a software to write about a well known fact or historical event and it just makes stuff up, it's not simply hallucinating. It's defective.
The defect isn't in the software, but in people expecting these things to operate the way AIs in sci-fi do, or who believe that because they can produce coherent results in natural language, they must be sentient and self-aware.
I'm sure AI companies will get very good at explaining away these defects with various forms of "aCkShUaLlY" but when your marketing materials say you made a box that takes a prompt and answers it, and it answers incorrectly, what else is it than a defect?
Floating point math is inherently inaccurate, and no programmer using it would expect perfect precision and call it a defect not to get it. You have to understand how floating point works and take that inaccuracy into account. As a result there are some applications for which using floats is simply a bad idea. No one sane is doing real money calculations with floats.
The same goes for LLMs. Hallucination is fundamental to the model. We're going to have to realize that there are many tasks for which AI simply isn't well suited. And we're going to have to get over this persistent delusion that humans are categorically worse than AI at everything. A paralegal doing research would probably not simply fabricate cases and cites whole cloth. That's not how most humans work. Humans are capable of knowing when they don't know something, AI is not.
But we've decided, for whatever reason, that AI is perfectly trustworthy. That's going to keep biting us in the ass until we learn.
EDIT: Another way to put it: If I sold a calculator that claimed to do math, but in the fine print I said "Actually, it just makes up answers by some means we don't fully understand, and somehow most of the time it comes up with the right answer." That doesn't mean that incorrect answers are suddenly not defects.
1: https://en.wikipedia.org/wiki/Pentium_FDIV_bug
So there's a lot of rigor on containing exactly how big your errors can be in floating point calculations, and that old intel bug made it (rarely) invalid.
Sugar coating the fact that it is defective (defined: imperfect or faulty.) isnt changing things.
Your explanation is correct, it's defective by design.
The fact that ‘correct’ outputs are treated as if they’re the product of an in-any-way-different process to the ‘hallucinated’ ones is the problem.
Also this particular context just makes it easier to notice, compared a 5000 word generated coherent-word-salad that equally wrong, but across the 5000 words.
Saying it hallucinated is just a tautology.
Generative AI output is becoming inextricably associated with this word, and that's not a bad thing to keep people aware of.
_compared to what_, exactly. Compared to a google search? Compared to asking a random person? Compared to wikpedia? New York Times journalists?
Any of those things are wrong _very_ frequently. It's such an uninteresting thing to call out every time an AI is wrong, when it is right about things so frequently that people don't bother to notice how amazing it is that it gets anything correct about the world at all.
I've certainly made that class of error myself, when I assumed that something followed a similar pattern (like in math, or writing & grammar, or coding) when it actually didn't.
I've also doubled-down on those errors when I tried to double-check my work, believing myself to have misapplied some intermediate step rather than having taken an entirely wrong approach to begin with.
I think the "why" here is "why are we assuming this failure mode is unique to LLMs and deserves novel terminology".
We could find out 800 years from now that human brains really do work exactly the same as LLMs do, but it wouldn't change the fact that today LLMs and humans in practice regularly manifest their respective mistakes in very different ways.
For example I don't have to worry about you making sense but then turning on a dime saying things like "Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay." or "Subtitles by the Amara.org community", which are both examples of OpenAI hallucinations I encountered today.
We can call that type of stupidity its own word, different from the types of mistakes you described making, just like we have many different words to convey the concept of "wet".
The fact that LLMs are probably not just baby people give an even greater justification to use different terminology for them.
Synonyms thrive in our languge with sometimes a hair's breadth of difference in nuance so it's silly to let optimism about tech deny this.
Anthropomorphizing statistical learning is how you build a hype machine to cash out people with zero handle on the subject. See the comment below about "AI judges" and "true justice". Just like early electricity, all people see is magic.
"Wargames" (1983): https://www.youtube.com/watch?v=71k7-dGhNFQ&t=4m8s
immunity to blackmail?
At some point the government will decide it's best to pay a private company to administer computerized court systems. Private companies already can collect evidence, process evidence, arrest people, detain suspects long-term (Jail) and after court hold the convicts; why not let them do this one step as well?
"in the kingdom of heaven the judge is a perfect AI, lawyers do not exist, and true justice does happen normally"
After googles recent "alignment" debacle never.
At least this one had the good sense to apologize instead!
[0] https://news.ycombinator.com/item?id=36097900