This is why there are certain jobs AI can never take: we are wired for humans to be responsible. Even though a pilot can do a lot of his work via autopilot, we need a human to be accountable. For the pilot, that means sitting in the plane. But there are plenty of other jobs, mostly high-earning experts, where we need to be able to place responsibility on a person. For those jobs, the upside is that the tool will still be available for the expert to use and capture the benefits from.
This lawyer fabricating his filings is going to be among the first in a bunch of related stories: devs who check in code they don't understand, doctors diagnosing people without looking, scientists skipping their experiments, and more.
The book https://en.wikipedia.org/wiki/The_Unaccountability_Machine introduces the term "accountability sink", which is very useful for these discussions. Increasingly complicated systems generate these voids, where ultimately no human can be singled out or held responsible.
AI offers an incredible caveat emptor tradeoff: you can get a lot more done more quickly, so long as you don't care about the quality of the work, and cannot hold anyone responsible for that quality.
$10k is not a high enough fine to dissuade this behavior. Large law firms will just approach this the same way that big tech accepts GDPR fines as part of the cost of doing business.
I'm numb to it after many "EU fines Householdnamecorp a zillion doubloons" type headlines, but using "historic fine" to describe $10k to a lawyer feels odd.
$10,000? That's a slap on the wrist. I don't say this lightly, this should have been jail time for someone. You're making a mockery of our most sacred institutions.
I searched for the lawyer's name in the state bar association, they've been practicing for over 13 years. Even has electrical engineering in their background.
> He thinks it is unrealistic to expect lawyers to stop using AI. It’s become an important tool just as online databases largely replaced law libraries and, until AI systems stop hallucinating fake information, he suggests lawyers who use AI to proceed with caution.
I think this is a good reason for fines to not be incredibly big. People are using AI all the time. There will be a growing period until they learn of its limitations.
$10,000, wow what a shocker! I don't know how anyone who can afford to live in California could ever expect to pay such a fine. I expect the lawyer will soon have to declare bankruptcy.
Do lawyers and judges not by now have software that turns all these citations into hyperlinks into some relevant database? Software that would also flag a citation as not having a referent? Surely this exists and is expensive but in wide usage...?
It's not a large step after that to verify that a quote actually exists in the cited document, though I can see how perhaps that was not something that was necessary up to this point.
I have to think the window on this being even slightly viable is going to close quickly. When you ship something to a judge and their copy ends up festooned with "NO REFERENT" symbols it's not going to go well for you.
> Mostafavi told CalMatters he wrote the appeal and then used ChatGPT to try and improve it. He said that he didn’t know it would add case citations or make things up. He thinks it is unrealistic to expect lawyers to stop using AI. [...] “In the meantime we’re going to have some victims, we’re going to have some damages, we’re going to have some wreckages,” he said. “I hope this example will help others not fall into the hole. I’m paying the price.”
Wow. Seems like he really took the lesson to heart. We're so helpless in the face of LLM technology that "having some victims, having some damages" (rather than reading what you submit to the court) is the inevitable price of progress in the legal profession?
21 of 23 citations are fake, and so is whatever reasoning they purport to support, and that's casually "adding some citations"? I sometimes use tools that do things I don't expect, but usually I'd like to think I notice when I check their work... if there were 2 citations when I started, and 23 when I finished, I'd like to think I'd notice.
> We're so helpless in the face of LLM technology that "having some victims, having some damages" (rather than reading what you submit to the court) is the inevitable price of progress in the legal profession?
Same with FSD at Tesla, there's many people who think that accidents and fatalities are "worth it" to get to the goal. And who cares if you, personally, disagree? They're comfortable that the risk to you of being hit by a Tesla that failed to react to you is an acceptable price of "the mission"/goal.
This is just incredibly defeatist on everyone talking here.
Here we have irrefutable evidence of just how bad the result of using AI would be and yet... the response is just that we need to accept that there is going to be damage but keep using it?!?
This isn't a tech company that "needs" to keep pushing AI because investors seem to think it is the only path of the future. There is absolutely zero reason to keep trying to shoehorn this tech in places it clearly doesn't belong.
We don't need to accept anything here. Just don't use it... why is that such a hard concept.
Everyone is somewhat missing the point here that the California bar is making.
They don't care if you use an AI or a Llama spitting at a board of letters to assemble your case, you are responsible for what you submit to the court.
This is just a bad lawyer who probably didn't check their work in many other cases, and AI just enabled that bad behavior further.
Doesn't seems like there isn't kind of disciplinary action. You can just make up stuff and if you're caught, pay some pocket change (in lawyer money level territory) and move on.
The hallucinations in legal briefs get really out of hand when the attorney wants to make an argument not supported by the case law. The LLM wants to do a good job defending a case, so it invents the legal precident, because otherwise it'd be impossible to make the argument credibly. This invites a rule 11 challenge from the other side where you claim the lawyer is so full of crap with his claim that he deserves sanction for not understanding the law and wasting everyone's time.
What's interesting about the rules of civil procedure is that it has been built up over centuries to prevent all kinds of abuse by sneaky, clever, unscrupulous litigants. Most systems are not so hardened against bad faith actors like the legal system is and AI just thinks it can pathologically lie its way through cause most people trust somebody who sounds authoritative.
AI constantly fabricates references. I found a question recently (accidentally) which would make it fabricate every answer. Otherwise, it tends to only fabricate about a third of them, and somehow miss dozens of others that it should easily find.
After asking for recommendations, I always immediately ask it if any are hallucinations. It then tells me a bunch of them are, then goes "Would you like more information about how LLMs "hallucinate," and for us to come up with an architecture that could reduce or eliminate the problem?" No, fake dude, I just want real books instead of imaginary ones, not to hear about your problems.
detail: The question was to find a book that examined how the work of a short order cook is done in detail, or any book with a section covering this. I started the question by mentioning that I already had Fast Foods and Short Order Cooking(1984) by Popper et al. and that was the best I had found so far.
It gave me about a half dozen great hallucinations. You can try the question and see how it works for you. They're so dumb. Our economy is screwed.
It's fun to point and laugh in this scenario where the attorney just threw slop at the court. However, these stories won't be around long.
Tools like Lexis+ from LexisNexis[1] now offer grounding, so it won't be as simple to bust people cutting corners in the future because these prevent the hallucinations.
We're now closer to the real Butlerian Jihad when we see pro se plaintiffs/defendants winning cases regularly.
The fact that the lawyer is unrepentant and says that it is simply too convenient and "there will be some victims, tough shit", basically, means the fine was a zero or two too short. Deterrence needs to hurt!
There you have it --- proof that lawyers love AI --- in more ways than this one example illustrates.
Using a tool that is widely known to be flawed to provide any sort of professional service (legal, medical, accounting, engineering, banking, etc.) is pretty much a textbook definition of negligence.
A lot of both defeatist and overly optimistic pro AI comments in here. Having built legaltech and interfaced heavily with attorneys over the last 5 years of my career, I will say that there is a wide spectrum of experience, ethics and intelligence in the field. Blindly copying output from anything and submitting it to the court seems like a mind boggling move, it doesn’t really make a difference if it was AI or Google or Bing or Thompson Reuters. This attorney is not representative of the greater population and probably had it coming imho.
There is definitely benefit to using language models correctly in law, but they are different than most users in that their professional reputation is at stake wrt the output being created and the risk of adoption is always going to be greater for them.
LLMs are just not the technology for 'search' and cite references, no matter how 'agentic' you make it.
They are great for a large number of other things, like generating code for example.
We are so focused trying to shoehorn LLMs to do something it is fundamentally unsuitable for that we are probably missing the discovery of the technology that would solve this.
34 comments
[ 4.3 ms ] story [ 59.6 ms ] threadThis lawyer fabricating his filings is going to be among the first in a bunch of related stories: devs who check in code they don't understand, doctors diagnosing people without looking, scientists skipping their experiments, and more.
AI offers an incredible caveat emptor tradeoff: you can get a lot more done more quickly, so long as you don't care about the quality of the work, and cannot hold anyone responsible for that quality.
https://www.theguardian.com/us-news/2025/apr/24/california-b...
So lawyers use it, judges use it ... have we seen evidence of lawmakers submitting AI-generated language in bills or amendments?
MPs are definitely using AI to write their speeches in parliament: https://www.telegraph.co.uk/business/2025/09/11/chatgpt-trig...
I think this is a good reason for fines to not be incredibly big. People are using AI all the time. There will be a growing period until they learn of its limitations.
It's not a large step after that to verify that a quote actually exists in the cited document, though I can see how perhaps that was not something that was necessary up to this point.
I have to think the window on this being even slightly viable is going to close quickly. When you ship something to a judge and their copy ends up festooned with "NO REFERENT" symbols it's not going to go well for you.
Wow. Seems like he really took the lesson to heart. We're so helpless in the face of LLM technology that "having some victims, having some damages" (rather than reading what you submit to the court) is the inevitable price of progress in the legal profession?
21 of 23 citations are fake, and so is whatever reasoning they purport to support, and that's casually "adding some citations"? I sometimes use tools that do things I don't expect, but usually I'd like to think I notice when I check their work... if there were 2 citations when I started, and 23 when I finished, I'd like to think I'd notice.
Same with FSD at Tesla, there's many people who think that accidents and fatalities are "worth it" to get to the goal. And who cares if you, personally, disagree? They're comfortable that the risk to you of being hit by a Tesla that failed to react to you is an acceptable price of "the mission"/goal.
Here we have irrefutable evidence of just how bad the result of using AI would be and yet... the response is just that we need to accept that there is going to be damage but keep using it?!?
This isn't a tech company that "needs" to keep pushing AI because investors seem to think it is the only path of the future. There is absolutely zero reason to keep trying to shoehorn this tech in places it clearly doesn't belong.
We don't need to accept anything here. Just don't use it... why is that such a hard concept.
They don't care if you use an AI or a Llama spitting at a board of letters to assemble your case, you are responsible for what you submit to the court.
This is just a bad lawyer who probably didn't check their work in many other cases, and AI just enabled that bad behavior further.
I didn't see them mentioned in the article.
https://apps.calbar.ca.gov/attorney/Licensee/Detail/282372
Doesn't seems like there isn't kind of disciplinary action. You can just make up stuff and if you're caught, pay some pocket change (in lawyer money level territory) and move on.
https://www.courtlistener.com/docket/68990373/nz-v-fenix-int...
What's interesting about the rules of civil procedure is that it has been built up over centuries to prevent all kinds of abuse by sneaky, clever, unscrupulous litigants. Most systems are not so hardened against bad faith actors like the legal system is and AI just thinks it can pathologically lie its way through cause most people trust somebody who sounds authoritative.
After asking for recommendations, I always immediately ask it if any are hallucinations. It then tells me a bunch of them are, then goes "Would you like more information about how LLMs "hallucinate," and for us to come up with an architecture that could reduce or eliminate the problem?" No, fake dude, I just want real books instead of imaginary ones, not to hear about your problems.
detail: The question was to find a book that examined how the work of a short order cook is done in detail, or any book with a section covering this. I started the question by mentioning that I already had Fast Foods and Short Order Cooking(1984) by Popper et al. and that was the best I had found so far.
It gave me about a half dozen great hallucinations. You can try the question and see how it works for you. They're so dumb. Our economy is screwed.
Tools like Lexis+ from LexisNexis[1] now offer grounding, so it won't be as simple to bust people cutting corners in the future because these prevent the hallucinations.
We're now closer to the real Butlerian Jihad when we see pro se plaintiffs/defendants winning cases regularly.
[1] https://www.lexisnexis.com/blogs/en-au/insights/hallucinatio...
Using a tool that is widely known to be flawed to provide any sort of professional service (legal, medical, accounting, engineering, banking, etc.) is pretty much a textbook definition of negligence.
And lawyers just love negligence.
There is definitely benefit to using language models correctly in law, but they are different than most users in that their professional reputation is at stake wrt the output being created and the risk of adoption is always going to be greater for them.
They are great for a large number of other things, like generating code for example.
We are so focused trying to shoehorn LLMs to do something it is fundamentally unsuitable for that we are probably missing the discovery of the technology that would solve this.