Everyone letting ChatGPT do their jobs, and the people training ChatGPT are too lazy to do the work themselves, they actually use ChatGPT to train ChatGPT... what could possibly go wrong in this scenario?
I wouldn't say OpenAI is "lazy," but this is still an excellent point about LLMs.
The researchers training the ones we use now are in a massive hurry. They aren't meticulously curating datasets, checking usage rights, hiring a huge field of human curators, or thoroughly field testing them. They are madly working on a new improvement, publishing it in a paper, then moving onto the next thing.
This has been a cultural issue in AI land for years. These experimental models were never meant for the general public, yet here we are.
Basically, if this paper is right, 2021 was the last year usable training data could be pulled from the internet. It could be all downhill from hereon out, as the amount of AI generated content will soon overtake actual human generated and -curated data on the internet.
They doubled down on their original mistake rather than admit what they did, even when it was quite clear the judge and opposing council was on to them. They should be disbarred.
See https://storage.courtlistener.com/recap/gov.uscourts.nysd.57... - Page 2, the judge uses the very words "doubled down": "But if the matter had ended with Respondents coming clean about their actions shortly after they received the defendant’s March 15 brief questioning the existence of the cases, or after they reviewed the Court’s Orders of April 11 and 12 requiring production of the cases, the record now would look quite different. Instead, the individual Respondents doubled down and did not begin to dribble out the truth until May 25, after the Court issued an Order to Show Cause why one of the individual Respondents ought not be sanctioned."
I guess that really depends on what you consider doubling down.
-On March 1, They filed a submission with fake citations from chatGPT.
-On April 11th court told them the cases couldnt be found and to produce full transcripts of the cases by April 25.
-On April 25, They came back and said they don't have them and couldnt find the cases, only the limited excerpts provided in "online databases".
-On May 4, the court asked them why they ought not be sanctioned for citing non-existent cases to the Court.
-On May 25, They said we found these cases on ChatGPT and thought they were real.
I guess the more transparent response on April 25 would have been to say "We weren't able to find the full cases and we now believe they probably dont exist ". Im not sure if the latter part is material if they already admitted they cant find the cases. If you cant produce the cases, is the reason relevant at that point?
At first it was just generated citations. The opposing lawyers and judges could not find the cases, and so they then asked chatGPT for them, and it generated them, and they were submitted to the court. (One of the lawyers notarized the other’s filings.)
There are even transcripts of chatGPT telling the accused that it is not an authoritative or up-to-date legal database.
—-
The punishment could arguably be more severe, since it demonstrated that these lawyers do not read their cases, particularly the one that notarized everything.
I just can't agree. To my way of thinking there are very few (if any) things that should be "career ending" the first time they happen. And the main guy involved here, as I recall, has a long career with what has been a spotless record until now. I just can't see the justification for ending that career over one fuck-up.
But, again... the punishment does seem lighter than I expected. I was thinking that any fine associated with this should be enough to hurt a bit. Something like $250,000 or something. This amounts to a slap on the wrist.
Just to clarify my position here: I do think this is a major fuck-up. Like, really major. To the extent that I think the punishment should not be "career ending" it's just a reflection of my seeing "career ending" punishments as something of a "nuclear option" that should be avoided in all but the absolute most extreme cases. I guess I would say that if punishments spanned a scale from 1-10 with "10" being "disbarment" then this should probably be about an 8 or 9 in my mind.
An officer of the court notarized that they had reviewed the imaginary citations and that those imaginary citations were material to the case. This is a gross breach of ethics and, given how easy it is to look up cases and verify them, grossly incompetent and lazy.
The punishment isn’t just the $5,000, it’s the public shaming that came with it. You’re not going to be able anyone’s name here for many years without this case coming up. The world at large has been watching this unfold gleefully.
That, combined with the lawyers involved having decades-long clean slates, as well as this case being outside the normal realm this team would operate in, as well as ChatGPT being relatively new at the time this happened makes it all seem pretty proportional.
Maybe you could say they deserve worse for how they handled things after they got caught, but I struggle to really have a problem with this punishment considering the records of those involved.
> The punishment isn’t just the $5,000, it’s the public shaming that came with it.
Both the fine and the public shaming are very light "punishments". Everyone's pointing and laughing now, but in a year nobody will even remember their name and their career will be fine.
On the other hand, there are certain kinds of lawyer with low-research clients who won't suffer much from this, and these lawyers are likely that kind (representing a client suing Avianca for a drinks cart complaint).
Because it wasn't just a simple mistake. It was such egregious professional negligence that it calls into question not only the basic competence of the attorney, but also their basic sense of ethics.
To add onto this - the lawyers failed to verify their resources! This is something that every lawyer carefully pays attention to before submitting anything.
The fact that they did no research on ChatGPT, and they didn't verify the output from ChatGPT, is pretty bad.
They're being punished more so on how they used their resources, not for using ChatGPT.
Huh, I think it would? It's still really bad, the quality of the service the client got is still abysmal, and it's still ultimately the lawyer's responsibility. That much is certainly unchanged, it would also be considered a much more understandable mistake.
If they followed exactly the same processes (showing a screenshot of a text conversation saying "Is X a real case?" when challenged about the nonexistent cases) then I imagine they would have have been treated the same way. However they probably wouldn't have gone down that route, because they know "uhhhh I included those cases because my intern insisted they exist?" absolutely wouldn't fly. The difference is that they thought they could chance it because ChatGPT is a shiny, new thing that some people have a pretty mixed understanding of.
It needs to be a warning to the industry, which requires real consequences for those involved. A warning could encourage abuse by others - there’s little to lose except having to stop after getting caught.
Things like this is what causes lawyers to lose cases for their clients. Imagine being a client of a lawyer who uses ChatGPT. You lose the case and then your livelihood or worse.
These twits should be disbarred, and without hesitation.
> Imagine being a client of a lawyer who uses ChatGPT. You lose the case and then your livelihood or worse.
This presupposes that the use of ChatGPT is the reason the case was lost. In this case, the judge found the mistake, called it out, and fined the lawyers for their carelessness. It doesn't necessarily follow that p1) ChatGPT was used, p2) Case was lost, c) Case would have been won if ChatGPT was not used.
What they did was the professional equivalent of a surgeon thinking "I wonder what'd happen if I chopped out this thingie?", then acting on that impulse, then lying about his actions.
Because to do otherwise is to normalize the use of false or non-existent information, whether due to bad faith, laziness, or gross ignorance, in processes where the outcomes may affect millions of dollars, thousands of people, or individuals' lives and freedoms.
Because if someone gets a minor fine, and doesn't lose their career, it's less clear why this would be fun to talk about. This is the most important dynamic in message board threads about current events.
Because certain careers, e.g. lawyers, surgeons, and law enforcement give their practitioners the ability to make or break their clients, potentially killing them or ruining them financially and/or socially. Such professions must, for reasons of social sanity, be held to higher standards than someone who's career is, e.g., writing web apps or serving coffee.
But also, pretty sure Doctors and Lawyers are among the least trusted/highest corruption professions. All that schooling is a barrier to entry to keep outsiders away, it doesnt guarantee competence. Basically once you get in, graduation rates are close to 100%.
My wife runs the resident eduction for her division and they are very aware of the dangers of sending an incompetent doctor out into the world to practice. They’ll force them through extra training and make them repeat residency if they have to.
Residents also drop out of their programs and move into other ones. It’s rare for them to drop out completely because they’d still be on the hook for $200k+ in loans. But medicine is a big field and it’s rare that someone who made it into and through med school to not be able to find something they are competent and capable at. Maybe you’re not great with patients, but you can look through a microscope all day as a pathologist etc…
Incompetent and corrupt doctors still make it through the process of course, but far far fewer than in our profession and just about any other profession I can think of.
As for lawyers:
Law schools have around a 77% graduation rate nation wide. And about 90% of law school graduates eventually pass the bar exam.
So we’re look at somewhere around 70% of people making it through after a pretty selective filter to begin with.
> According to polls, doctors are the 2nd most trusted profession.
n=1 anecdote here:
When I was in research track biochem and bumped into premed students, nearly the entire population of premed didn't seem to care about the science behind what they were learning. They formed a very different social clique. The "top students" in that set were sharing previous years' organic chemistry exams and none of them would read papers or get involved in research. They were entirely disinterested in the science.
In my interactions with general practitioners, I like to talk about medicines' method of actions, pharmacology, the actual biochemistry behind diseases. Most of them seem to have no clue or retention of this information. I'm not trying to challenge them, either -- I'm generally curious to learn.
That's not to say all doctors are like that. A lot of the surgeons and specialists I know are walking tomes of information.
This seems analogous to a programmer who knows all the sort functions offered by their language of choice and when to use each, but has never learned how to write their own sort algorithm, and therefore can't hold a conversation about doing so. Plenty of employers and customers would be thrilled with such a programmer, if the job at hand is just to ensure that the data gets properly sorted.
I think a lot of your general practitioners are bound to follow protocol set forth by various medical organizations. So in that sense a lot of diving into research won’t help them. They need to understand and diagnose without having to worry about every little research paper. A lot of research papers aren’t truth - they are peer reviewed opinions.
Once you get to medical research groups that formulate those protocols, that’s where research becomes important.
>I think a lot of your general practitioners are bound to follow protocol set forth by various medical organizations. So in that sense a lot of diving into research won’t help them. They need to understand and diagnose without having to worry about every little research paper. A lot of research papers aren’t truth - they are peer reviewed opinions.
Isnt this even more necessary for them to learn the fundementals? Especially if Physicians claim that they are both Art and Science, which is why we can't merely use science for diagnosis and treatment.
That’s why diving into research will not help general practitioners. Because research isn’t fact. You need a body of research along with oversight of statistical methods and its data collection to claim fact and that too with specific constraints.
So they do learn fundamentals but fundamentals do not come out of research papers.
Studying for the test and taking easy classes, especially as premeds, has been a (likely accurate) stereotype of the medical profession since at least the 1970s & 80s.
From country where medschool is done by entrance exams the whole "premed" always sounded extremely weird and wasteful. They already have longest training time and then more is added on top?
Med school in those countries tends to last a year or 2 longer.
Also most other countries have national education programs for secondary school. High school in the US is regulated at the state and local level, so it’s very variable. As a consequence, the first 2 years of college tends towards general and education.
Another note, there’s also a med school entrance exam in the US.
Sure but in the US it is 4 (undergrad) + 4 (med school) so in total it is 8 years rather than 5 or 6.
Agreed regarding the US high school standards but students that take AP courses in high school can take the MCAT. They may not do the “best” but they will know some of the material. Schools in the US like UMKC have 6 year programs were high school students take the MCAT.
In a lot of ways, the first two years of US undergrad are "remedial high school". This also explains how, e.g. American undergrads typically declare their majors until mid to late into their second years.
In theory, the point of having medical students have an undergrad degree is that they will be more "well rounded". Unlike in many countries where university education is very focused, in the US, an undergraduate even studying STEM is required to take a certain number of courses like literature or history, and likewise someone studying literature or history is required to take some science courses. Of course just requiring people to take courses doesn't mean they will actually retain the knowledge after the tests, but it's a nice idea anyway.
Something like 17% of premed students end up actually becoming physicians.
That’s a very different sample. Then you have to consider that the people that you are most likely to notice are the most vocal self identified “premed” students since most colleges don’t actually have premed degrees.
As far as interactions with doctors. GPs are the least likely to get into specifics because of the generalist nature of their work. Expecting them to remember something that they haven’t used in 10+ years is asking a lot.
Understanding calculus is incredibly useful to software engineers. But try asking a practicing software engineer 10 years out of school some textbook calculus questions.
This is because medicine is so complex and usually doctors are in the business of caring for patients, and while they may enjoy science, it’s not like every single detail of medicine, which is a very broad field, will be interesting to them. At some point, the MoA of a drug just becomes another fact to memorize for board exams. clinical application is far more based on experience and familiarity compared to abstract theoretical knowledge, even though this forms the base for medicine.
Far be it from me to defend our system of lobbying the government, but there's a big difference between bribery and federal lobbying. Your phrasing poisons the well.
It shouldn't be so mindblowing. People are lazy and dumb, as much so as they're allowed to be. We're a year away from this tech being completely mainstream to the point where we'll find out what happens when an entire society at once stops trying to think/work because there's a machine to do it for them now.
Before it was handful of mad people with things like sovereign citizen and stupid instructions on Internet. But now it seems even trained folks will follow whatever something they have not verified says...
Also I wonder how well the training data was cleaned from all potentially harmful stuff like for example the sovereign citizen ideas...
If they used ChatGPT to find real cases that's not lazy, that's smart. The dumb and lazy part was NOT CHECKING ON THE NEW TECHNOLOGY FOR SOMETHING YOU SENT TO A JUDGE. Sorry for the caps, but so, so stupid. IIRC they asked ChatGPT if it lied, and it said no..
>completely mainstream to the point where we'll find out what happens when an entire society at once stops trying to think/work
You really need not wait that far if you have been observant enough, they are they were at one point clownish enough to believe in religion, now the vast majority of them believe in 'science' etc. Over reliance on Chat GPT is just the one of the newest manifestation of laziness and dumbness.
Did they risk it? Even after it being obvious the lawyer was lying and trying to cover up, and not acquiescing to their wrongdoing even after all of it, all they had to pay was $5k.
Do we know for sure that absolutely no further action is being taken? I really don't know processes, so I dunno if this is it or if this is just one of the consequences, and that the bar association (is that the right org?) they're part of would want to weigh in on it.
They still risk being disbarred (not just hypothetically, it might actually happen) + taking a massive hit to the reputation (their own + the firm). The latter one alone is a strong punishment, given how widely-publicized this incident was in the media. It cost the firm a lot of potential clients, and the lawyer might struggle with finding a decent position after the fact (given how much more reputation-based the legal field seems to be compared to something like engineering).
I've worked with legal software professionally a few times, and kept thinking software won't replace lawyers, it'll replace paralegals. Not all, but a great deal of grunt work could be replaced with AI and good forms. Now if we could get them not to lie?
There still needs to be a human manually proofreading them, even if it's as trivial as looking up whether the cited cases actually exist. Same goes for code outputs from currently available AI tools: some still needs to debug them to make sure it works, and fix any troubles that spring up during deployment and production.
Of course after some number of iterations, AI may gain the ability to self-check and self-correct to the point of achieving 100% accuracy, but I think that milestone will quickly lead to AGI and job security will quickly become an obsolete topic for every industry anyways.
Maybe. But I keep thinking it's not going to replace the single paralegal that a single lawyer has, but it might reduce the 9 paralegals to 6, and then 4, and then..
That's a fair point, but AI tools could also unlock higher potential for them to take on more workloads. Perhaps by maintaining the same number of paralegals or even hiring more, they could take on an exponentially higher number of cases than before. There's infinite room for new litigation as long as human society persists.
Well, then either it will empower the "little guy" to do more (more your example) , or empower big business to capture more big business (more my example). Wanna guess which one I'm betting on?
I think it's because AIs aren't actually in touch with reality as we know it, they're generating text from a sort of graph of concepts that comes out of the training process.
It's because, just like human memory, they aren't databases or search engines.
Generative transformer models are basically next-word-prediction machines on steroids. They take the input and try to "guess" the most likely reply based on their training data.
These machines have no way to distinguish facts from fiction, only probabilities
of combinations of words that would make the most plausible reply.
> Is it possible to prevent it?
There are methods to prevent this by incorporating specialised knowledge databases into the training material of these models. This, however, only works with models that have been finetuned on very specific tasks and topics [1].
Other approaches use AI to transform human questions ("bag of words" inputs) to queries into structured knowledge bases, match the results (e.g. tree-like structures of context and facts) to the question and turn them back into human language [2]. The downside of these methods is that they're currently limited to simple QA formats and won't feel as "natural" as talking to chatbot and requires carefully prepared and curated knowledge databases.
Watson wasn't just one thing, it was a marketing term to combine all IBM's efforts to do early AI like efforts. They didn't have any real success in making a product out of anything and stopped marketing 'Watson'
> many lawyers already don't do their own research, they use paralegals to do it for them.
If you mean that some lawyers use paralegals to do word searches for potentially-relevant case law (i.e., published judicial opinions as precedents), that might be true. But in my experience, good lawyers either do the raw research themselves, or at a minimum they use junior lawyers to do the initial searching for precedents; then if necessary, they do supplemental searching themselves, based on their greater experience.
I had someone run into basically the same issue with their dissertation.
I went back into my texts since I had recommended them chatGPT. I screenshotted the part where I said:
"You can't trust it for valid information"
I even told him that everyone needs to be punished by incorrect information from ChatGPT once to really understand what its capable of. You need that traumatic event.
Gosh, why can't people just listen and accept other people's thoughts. Why must we learn individually?
> Gosh, why can't people just listen and accept other people's thoughts. Why must we learn individually?
If you've evolved from a species that is profoundly stupid, as a rule, then perhaps an adaptation/mutation that makes you mistrusting of that species advice is one that tends to propagate.
For every "don't use ChatGPT, it's unsafe", there's a "I talk to magical beings who tell me you're not allowed to eat good food and you should mutilate your genitals".
Statistically, refusing to learn from other people is probably the best bet.
> Levidow, Levidow & Oberman said in a statement on Thursday that its lawyers “respectfully” disagreed with the court that they had acted in bad faith. “We made a good-faith mistake in failing to believe that a piece of technology could be making up cases out of whole cloth,” it said.
Seems like they still don't get it.
It's not good faith representation of your client, it's not good lawyering to interrogate a chatbot for your case. Even if it could somehow only ever tell the truth.
It's not a "good faith mistake" - so I think they still don't understand the problem here.
It might be representative of greater society: when people interact with ChatGPT, they treat it as an oracle. I see it all the time on r/chatgpt - people say, "I asked ChatGPT about my long lost grandparent whose life I am researching. Apparently my grandma was a pilot!"
Seems like the lawyers are shocked the oracle lied but still think they were right to consult it.
The cynic in me thinks they do get it and that their statement is an excuse to save face. It’s easier to blame technology instead of admit they wanted to cut corners.
Right. They aren't completely finished with potential punishments. A review by the Bar is likely still ongoing. They can't do anything but pretend it was all done in good faith. Much like a person or company who may end up in a lawsuit can't issue a "real" apology when they screw up lest it be used against them.
The risk of ChatGPT is not that people take it seriously, since no-one does even those lawyers; The risk is that it is becomes acceptable to pretend to not understand the shortcomings of ChatGPT, and pretend that it does not make the person an irresponsible adult.
Like editing Wikipedia then citing the contents. Impossible to defend in good faith, but passes as a legit excuse in a public court.
Some people really know how to use public naivety to enforce the public’s powerlessness.
Are we all using the same definition of "good faith"? That just means to deal honestly and sincerely without any intent to defraud, deceive, or harm the other party.
I don't see any proof that shows they intended to deceive rather than that they were clueless.
This right here is what people are overlooking in their contempt.
The attorney used it as a research tool, and even foolishly thought to use it to validate that it wasn't making shit up. It's a really embarrassing mistake that's only obvious in the eyes of people familiar with the technology. There is nothing about this that suggests they intended to deceive, only intent to be a lazy idiot.
You will be surprised how someone who’s not techie, doesn’t understand tech, heck, I even worked with people who supposedly are tech savvy but lack the basic of the basics. But overall, lawyers and doctors are the least knowledgeable in tech from my experience, I found farmers for example relatively are better, maybe because of a mindset that he/she needs to learn tech to navigate around by themselves, while a lawyer is snob a out it or just hire someone else to do the whole work.
I think many commenters here are unaware of exactly how forehead-slappingly WTF it is that these lawyers did. Even if ChatGPT were replaced with a well respected human legal expert saying exactly the same things, they still cited case law that they couldn't verify the existence of, let alone that the context of the ruling was actually helpful to their cause.
Then when a judge told them "hey, we can't find any of these cases, what the hell is going on?" they doubled down. They're basically saying "trust me bro, these rulings totally happened and they totally say we're right".
When criticized about the made up cases they even showed (undated) screenshots of them asking ChatGPT if the cases were real.
This is like your local major getting tricked into buying the Eiffel tower using public funds and then defending himself by saying he asked the guy he bought the Eiffel tower from and the guy said he was totally legit.
LegalEagle did a video on this case two weeks ago, digging into it in some more depth: https://youtu.be/oqSYljRYDEM
They lied to a judge, doubled down, showed further incompetence by admitting to not knowing how the legal reference they supposedly quoted was even structured, then lied to a judge again trying to get more time to get their story straight, and all it cost them was $5000? And even then they have the guts to say they disagree with the judge?
Levidow, Levidow & Oberman must either be run by fools or take the rest of the world for fools.
I can understand how the rubber stamping lawyer signing all the paperwork got off easy; he was just signing documents because the lawyer who supposedly did all the work for the case couldn't operate in the court the case was transfered to.
The other lawyer, the one who did the AI equivalent from copy pasting from the summary page of Google, should not be deemed fit to practice law.
They also have to write written apologies to their client, and all of the judges who were mentioned in their fake cases, with proof filed in the docket.
And they are the brunt of a very well-publicized lawsuit. The judge is effectively saying that all that publicity is punishment enough; he doesn't need to do anymore. He doesn't need to refer them to the bar for disbarment--the bar already knows about it. (The judge has no power to disbar them himself, he can only refer to the bar for disbarment.)
The biggest surprise is that they didn't get made to pay the defendant's legal fees as well, but that's because defendant didn't ask for any fees.
Oh, and the final punishment... they lost the case--they get no chance to redo their opposition to the motion to dismiss. Arguably, their client can also sue them for malpractice, and their malpractice carrier is likely to prefer to settle that rather than actually fight the case.
The real punishment is this decision, laying out a record of Keystone-Cops-level buffoonery, dishonesty, and lying. Laying all that out in a published decision, in a case that's receiving a fair amount of interest, which means that the decision will be all over the internet.
This isn't going to go away. This decision will dog them for the entire rest of their careers.
> This isn't going to go away. This decision will dog them for the entire rest of their careers.
I get the impression from the events leading up to the use of GPT this is not the first time their competence led to outcomes that will "dog them for the entire rest of their careers."
> They also have to write written apologies to their client, and all of the judges who were mentioned in their fake cases, with proof filed in the docket.
I thought the judge said he couldn't force them to do that because it wouldn't be sincere.
Whether or they choose to apologize, they do have to send letters that include a copy of the transcript of the hearing on sanctions and the sanctions order.
"They also have to write written apologies to their client, and all of the judges who were mentioned in their fake cases, with proof filed in the docket."
They'll probably just have ChatGPT write their apologies for them.
What is even better is this attorney - who relied on ChatGPT for research - when called to task to provide the legal opinions....couldn't even get the affidavit right.
"The April 25 affidavit was filed in response to the Orders of April 11 and 12, 2023, but is sworn to before a Notary Public on the 25th of January 2023. Mr. LoDuca should be prepared to address this anomaly on June 8, 2023 and bring with him a wet-ink signed copy of the affidavit with the notary’s signature and stamp."
> The judge is effectively saying that all that publicity is punishment enough; he doesn't need to do anymore. He doesn't need to refer them to the bar for disbarment--the bar already knows about it. (The judge has no power to disbar them himself, he can only refer to the bar for disbarment.)
a reference from the judge and the bar just knowing about it are very different.
No one is going to be disbarred over this, disbarment is a very high bar to clear. You have to extort your client or take some other brazenly malicious or illegal action.
> They lied to a judge, doubled down, showed further incompetence by admitting to not knowing how the legal reference they supposedly quoted was even structured, then lied to a judge again trying to get more time to get their story straight, and all it cost them was $5000?
Well, no, they also have to right letters and attach the sanctions order, and the transcript of the sanctions hearing to:
(1) their client in the case, and
(2) each of the real judges to whom they (on the basis of ChatGPT “research”) falsely attributed a decision.
But, yes, sanctions are fairly light (particularly the absence of a disciplinary referral).
> I can understand how the rubber stamping lawyer signing all the paperwork got off easy; he was just signing documents because the lawyer who supposedly did all the work for the case couldn't operate in the court the case was transfered to.
> The other lawyer, the one who did the AI equivalent from copy pasting from the summary page of Google, should not be deemed fit to practice law.
They both failed their duty in nearly identical ways; passing of someone else’s work to which they did not apply independent professional analysis as their professional work. If anything, the one doing so directly to the court should be more, not less, liable to outside sanction (if the issue was sanctions within the firm, that’s a different story, maybe.)
I don’t the the absence of a formal disciplinary referral means much when the judge understands the amount of publicity from this is itself a referral.
>They both failed their duty in nearly identical ways
Once they showed up in court, one of them was unable to explain how to interpret a very basic "you should know this to practice law" case citation ("F.3d"[1]). Not only did they not "apply independent professional analysis" it seems they were woefully unprepared to even produce such a thing.
> They lied to a judge, doubled down, showed further incompetence by admitting to not knowing how the legal reference they supposedly quoted was even structured
Sorry to be cynical, but this sounds like they delegated their defense to AI as well...
in the near future this kind of argument will probably work in a lot of contexts.
Imagine if your only responsibility is to check the work of someone deemed untrustworthy, to protect everyone else from their malice or incompetence, and you just skip doing that and lie and say you did it.
Every version of ChatGPT that I have used, going back to the beginning of January 2023, has had a disclaimer appear when you first load up the page that specifically tells you, "May occasionally generate incorrect information."
This is why the law firm's contention that they were acting in good faith is laughable.
When the tool itself shows a prominent disclaimer that it "May occasionally generate incorrect information," then the law firm can't reasonably argue that "We made a good-faith mistake in failing to believe that a piece of technology could be making up cases out of whole cloth."
It was either gross negligence or intentional malfeasance.
The most interesting thing to me is why the firm is continuing to try to excuse its actions, rather than taking full responsibility for them. I guess they assume that the classic Shaggy "It Wasn't Me" defense is less likely to lose them business, because they think so little of the clients they're trying to attract that they're wagering they'll buy it.
I have a friend who used to be a federal judge for immigration, and he said that immigration lawyers are especially the worst, because families need their services desperately, and these lawyers will take their money and spend NO time on their cases. They show up to court and are figuring stuff out real time.
He tore one lawyer a new asshole for doing exactly this for his client and ordered them to come back better prepared.
So the idea that some of them will use ChatGPT to do their work doesn't surprise me at all.
I work in the immigration space. Part of the problem is that because immigration is technically federal but operates almost entirely in administrative venues (i.e. USCIS and EOIR), immigration lawyers only need to be barred in their home state and do not generally need admission to a relevant federal court bar or appear pro hac vice to represent clients in immigration court or even to make an appearance in filings.
It's also not usually a lot of money in comparison to their alternatives. I once had a criminal defense lawyer tell me that they make up the difference with scale and essentially batch-process people.
I wonder if this case was a setup for a lawsuit against OpenAI?
Given the marketing and the way Microsoft and Google were tripping over themselves to "catch up", the impression in the media and to the general public is that chat bots are a magic silver bullet for everything. Even a lot of tech people are unclear about exactly how these things work.
The case seems pretty clear: "They said it was artificial intelligence, but it wasn't very intelligent to make up cases. Look at the damages it's caused me..."
As entertaining as this is to ponder, I don’t think they’re that clever. I don’t think they’d do so much damage to their reputation for such a flimsy case. They’re just incompetent.
In court, good lawyers think of themselves as "officers of the court" — to use a sometimes-forgotten phrase — who are working not just for their clients but also for the judge, to help the judge (and the judge's law clerks) to get it right, with rulings that will stand up on appeal if it comes to that.
The good lawyer's job is not just to advocate the client's position — it's also to do "completed staff work" for the court in the process, even acknowledging any weaknesses in the client's case (which builds credibility).
As we've seen in the ChatGPT case, judges tend to respond badly when they think a lawyer is trying to put one over on them.
And conversely, judges often respond favorably when they perceive that a lawyer is being thorough and intellectually honest, not merely throwing up any bullshit argument that comes to mind. A fond memory from years ago is when a federal judge called me and my opponent — a well-known lawyer in our field — into his chambers to talk about a pending motion that involved an uncertain point of law, which we had extensively briefed. My opponent had not done an especially-great job; my own briefing had acknowledged the weak points in my client's position but also explained why my client should win anyway. The judge said, I don't really know what the answer is here. He then turned to me and said, But I'm more comfortable with your analysis. And he ruled my way – which won the case for my client (a company you might well be a customer of today).
And every time this happens from now on, the fine should be doubled. When the fine gets too high for the lazy lawyer to pay, they may instead relinquish their bar admission (lose their license to practice law) for five years.
Lawyers admitted to the bar aren't innocents. They're "officers of the court", meaning the legal system relies on them to avoid BS in their citation of other cases and of statutes and laws. It's never been acceptable for a lawyer to sign off on something without reading it and making sure it's right.
What if registered professional civil engineers started asking generative AI for sewer-system or bridge designs and then didn't inspect them before approving them and handing them off to the guys with dozers and cement trucks? Not only would people fall into rivers, but the rivers would be polluted.
The real world consequences are similarly serious when lawyers make stuff up, or let their interns or word-processing software make stuff up.
It would seem that in the future, people will be less inclined to proof-read their work, or even read it at all prior to submission.
I'd wager a guess that this will only get worse over time. LLMs will do more stuff for people - from helping school children to get through their homework, to higher learning assignments and all the way up to the "real" jobs. Along the way people might just place far too much trust in these tools. I'm pretty sure that's something we can all agree on.
What I do find striking here though, is the incredible laziness of the lawyers. Not even checking whether or not the cases actually happened? That's just bad lawyering. The AI actually helped the system to call them out on their bad lawyering.
That's not "bad things happening because AI" but just plain stupidity.
169 comments
[ 4.5 ms ] story [ 238 ms ] threadThe researchers training the ones we use now are in a massive hurry. They aren't meticulously curating datasets, checking usage rights, hiring a huge field of human curators, or thoroughly field testing them. They are madly working on a new improvement, publishing it in a paper, then moving onto the next thing.
This has been a cultural issue in AI land for years. These experimental models were never meant for the general public, yet here we are.
Basically, if this paper is right, 2021 was the last year usable training data could be pulled from the internet. It could be all downhill from hereon out, as the amount of AI generated content will soon overtake actual human generated and -curated data on the internet.
So the party could be over soon ¯\_(ツ)_/¯
-On March 1, They filed a submission with fake citations from chatGPT.
-On April 11th court told them the cases couldnt be found and to produce full transcripts of the cases by April 25.
-On April 25, They came back and said they don't have them and couldnt find the cases, only the limited excerpts provided in "online databases".
-On May 4, the court asked them why they ought not be sanctioned for citing non-existent cases to the Court.
-On May 25, They said we found these cases on ChatGPT and thought they were real.
I guess the more transparent response on April 25 would have been to say "We weren't able to find the full cases and we now believe they probably dont exist ". Im not sure if the latter part is material if they already admitted they cant find the cases. If you cant produce the cases, is the reason relevant at that point?
Just like Wikipedia, ChatGPT is great for generating leads, but you need to check the actual primary sources! How do people not get this?
It is mentioned also in the article:
>then continued to stand by the fake opinions after judicial orders called their existence into question.”
> then continued to stand by the fake opinions after judicial orders called their existence into question.
At first it was just generated citations. The opposing lawyers and judges could not find the cases, and so they then asked chatGPT for them, and it generated them, and they were submitted to the court. (One of the lawyers notarized the other’s filings.)
There are even transcripts of chatGPT telling the accused that it is not an authoritative or up-to-date legal database.
—-
The punishment could arguably be more severe, since it demonstrated that these lawyers do not read their cases, particularly the one that notarized everything.
I just can't agree. To my way of thinking there are very few (if any) things that should be "career ending" the first time they happen. And the main guy involved here, as I recall, has a long career with what has been a spotless record until now. I just can't see the justification for ending that career over one fuck-up.
But, again... the punishment does seem lighter than I expected. I was thinking that any fine associated with this should be enough to hurt a bit. Something like $250,000 or something. This amounts to a slap on the wrist.
Just to clarify my position here: I do think this is a major fuck-up. Like, really major. To the extent that I think the punishment should not be "career ending" it's just a reflection of my seeing "career ending" punishments as something of a "nuclear option" that should be avoided in all but the absolute most extreme cases. I guess I would say that if punishments spanned a scale from 1-10 with "10" being "disbarment" then this should probably be about an 8 or 9 in my mind.
These people should not be practicing law.
That, combined with the lawyers involved having decades-long clean slates, as well as this case being outside the normal realm this team would operate in, as well as ChatGPT being relatively new at the time this happened makes it all seem pretty proportional.
Maybe you could say they deserve worse for how they handled things after they got caught, but I struggle to really have a problem with this punishment considering the records of those involved.
Both the fine and the public shaming are very light "punishments". Everyone's pointing and laughing now, but in a year nobody will even remember their name and their career will be fine.
So does Google, and anyone researching these lawyers or this firm in the future...
The fact that they did no research on ChatGPT, and they didn't verify the output from ChatGPT, is pretty bad.
They're being punished more so on how they used their resources, not for using ChatGPT.
Yes, exactly. If they'd used a human intern instead of ChatGPT, it wouldn't change the situation at all.
The client may want to sue them and seek damages or something and that'd be fair but I don't think their career should end. That's harsh.
These twits should be disbarred, and without hesitation.
This presupposes that the use of ChatGPT is the reason the case was lost. In this case, the judge found the mistake, called it out, and fined the lawyers for their carelessness. It doesn't necessarily follow that p1) ChatGPT was used, p2) Case was lost, c) Case would have been won if ChatGPT was not used.
Or to waste a lot of their clients money on cases that they should have known could not be won from the outset, as is the case here.
It's not equivalent to that. Nobody lost their leg over this.
Stupid mistakes are usually not career ending, fraud and cover up usually is.
Because certain careers, e.g. lawyers, surgeons, and law enforcement give their practitioners the ability to make or break their clients, potentially killing them or ruining them financially and/or socially. Such professions must, for reasons of social sanity, be held to higher standards than someone who's career is, e.g., writing web apps or serving coffee.
- "A man sued Avianca Airline – his lawyer used ChatGPT" (27 days ago, 139 comments): https://news.ycombinator.com/item?id=36095352
- "Lawyer who used ChatGPT faces penalty for made up citations" (15 days ago, 128 comments): https://news.ycombinator.com/item?id=36242462
- "Lawyers blame ChatGPT for tricking them into citing bogus case law" (14 days ago, 175 comments): https://news.ycombinator.com/item?id=36257545
Original court documents: https://www.courtlistener.com/docket/63107798/mata-v-avianca...
The full document from the federal judge on sanctions: https://storage.courtlistener.com/recap/gov.uscourts.nysd.57...
One of many highlights: dismissing the lawyer's bad-faith argument with a footnote quoting Alice in Wonderland (footnote 12, page 18).
Sure, I see plenty of uses for ChatGPT in their field, but to not even check the output before submitting it is what blows my mind.
But also, pretty sure Doctors and Lawyers are among the least trusted/highest corruption professions. All that schooling is a barrier to entry to keep outsiders away, it doesnt guarantee competence. Basically once you get in, graduation rates are close to 100%.
My wife runs the resident eduction for her division and they are very aware of the dangers of sending an incompetent doctor out into the world to practice. They’ll force them through extra training and make them repeat residency if they have to.
Residents also drop out of their programs and move into other ones. It’s rare for them to drop out completely because they’d still be on the hook for $200k+ in loans. But medicine is a big field and it’s rare that someone who made it into and through med school to not be able to find something they are competent and capable at. Maybe you’re not great with patients, but you can look through a microscope all day as a pathologist etc…
Incompetent and corrupt doctors still make it through the process of course, but far far fewer than in our profession and just about any other profession I can think of.
As for lawyers:
Law schools have around a 77% graduation rate nation wide. And about 90% of law school graduates eventually pass the bar exam.
So we’re look at somewhere around 70% of people making it through after a pretty selective filter to begin with.
n=1 anecdote here:
When I was in research track biochem and bumped into premed students, nearly the entire population of premed didn't seem to care about the science behind what they were learning. They formed a very different social clique. The "top students" in that set were sharing previous years' organic chemistry exams and none of them would read papers or get involved in research. They were entirely disinterested in the science.
In my interactions with general practitioners, I like to talk about medicines' method of actions, pharmacology, the actual biochemistry behind diseases. Most of them seem to have no clue or retention of this information. I'm not trying to challenge them, either -- I'm generally curious to learn.
That's not to say all doctors are like that. A lot of the surgeons and specialists I know are walking tomes of information.
Once you get to medical research groups that formulate those protocols, that’s where research becomes important.
Isnt this even more necessary for them to learn the fundementals? Especially if Physicians claim that they are both Art and Science, which is why we can't merely use science for diagnosis and treatment.
So they do learn fundamentals but fundamentals do not come out of research papers.
When I took non-required theoretical classes, I tended to be one of the only undergrads in the class
Also most other countries have national education programs for secondary school. High school in the US is regulated at the state and local level, so it’s very variable. As a consequence, the first 2 years of college tends towards general and education.
Another note, there’s also a med school entrance exam in the US.
Agreed regarding the US high school standards but students that take AP courses in high school can take the MCAT. They may not do the “best” but they will know some of the material. Schools in the US like UMKC have 6 year programs were high school students take the MCAT.
That’s a very different sample. Then you have to consider that the people that you are most likely to notice are the most vocal self identified “premed” students since most colleges don’t actually have premed degrees.
As far as interactions with doctors. GPs are the least likely to get into specifics because of the generalist nature of their work. Expecting them to remember something that they haven’t used in 10+ years is asking a lot.
Understanding calculus is incredibly useful to software engineers. But try asking a practicing software engineer 10 years out of school some textbook calculus questions.
https://www.opensecrets.org/federal-lobbying/top-spenders?cy...
They are the 4th largest briber in the US.
Finding a difference between bribery and federal lobbying with money, is the same outcome.
Money goes from person who wants, to person with power who receives. Everything else is theatre.
The money that comes from lobbying is also limited and pays for reelection, the candidate doesn’t get to use it for personal expenses.
There are clearly problems with the system, but yelling no! it’s exactly the same as taking bribes isn’t helpful.
Lobbying is legal because we live in a society that places such a high value on money that we've accepted bribery as a means to increase its power.
Also I wonder how well the training data was cleaned from all potentially harmful stuff like for example the sovereign citizen ideas...
You really need not wait that far if you have been observant enough, they are they were at one point clownish enough to believe in religion, now the vast majority of them believe in 'science' etc. Over reliance on Chat GPT is just the one of the newest manifestation of laziness and dumbness.
Did they risk it? Even after it being obvious the lawyer was lying and trying to cover up, and not acquiescing to their wrongdoing even after all of it, all they had to pay was $5k.
They still risk being disbarred (not just hypothetically, it might actually happen) + taking a massive hit to the reputation (their own + the firm). The latter one alone is a strong punishment, given how widely-publicized this incident was in the media. It cost the firm a lot of potential clients, and the lawyer might struggle with finding a decent position after the fact (given how much more reputation-based the legal field seems to be compared to something like engineering).
With that information I think it's less mind-blowing, they were simply assuming ChatGPT could do the work of a paralegal
Of course after some number of iterations, AI may gain the ability to self-check and self-correct to the point of achieving 100% accuracy, but I think that milestone will quickly lead to AGI and job security will quickly become an obsolete topic for every industry anyways.
It's because, just like human memory, they aren't databases or search engines. Generative transformer models are basically next-word-prediction machines on steroids. They take the input and try to "guess" the most likely reply based on their training data.
These machines have no way to distinguish facts from fiction, only probabilities of combinations of words that would make the most plausible reply.
> Is it possible to prevent it?
There are methods to prevent this by incorporating specialised knowledge databases into the training material of these models. This, however, only works with models that have been finetuned on very specific tasks and topics [1].
Other approaches use AI to transform human questions ("bag of words" inputs) to queries into structured knowledge bases, match the results (e.g. tree-like structures of context and facts) to the question and turn them back into human language [2]. The downside of these methods is that they're currently limited to simple QA formats and won't feel as "natural" as talking to chatbot and requires carefully prepared and curated knowledge databases.
[1] http://jens-lehmann.org/files/2019/iswc_bert_simple_question...
[2] https://arxiv.org/abs/2303.13284
This reminds me. Whatever happened to IBM’s Watson? It’s not a LLM right? But it seem similar enough function wise.
If you mean that some lawyers use paralegals to do word searches for potentially-relevant case law (i.e., published judicial opinions as precedents), that might be true. But in my experience, good lawyers either do the raw research themselves, or at a minimum they use junior lawyers to do the initial searching for precedents; then if necessary, they do supplemental searching themselves, based on their greater experience.
Now many, if not likely most, of the well-schooled are generally intelligent, but of course there are also quite a few rich idiots as well.
(Today, of course, most of the latter go into tech.)
I went back into my texts since I had recommended them chatGPT. I screenshotted the part where I said:
"You can't trust it for valid information"
I even told him that everyone needs to be punished by incorrect information from ChatGPT once to really understand what its capable of. You need that traumatic event.
Gosh, why can't people just listen and accept other people's thoughts. Why must we learn individually?
Because other people's thoughts are usually wrong so you cant trust them.
Why can't people just do research the real way rather than circuitously go to the wrong answer machine and then double check everything it says?
If you've evolved from a species that is profoundly stupid, as a rule, then perhaps an adaptation/mutation that makes you mistrusting of that species advice is one that tends to propagate.
For every "don't use ChatGPT, it's unsafe", there's a "I talk to magical beings who tell me you're not allowed to eat good food and you should mutilate your genitals".
Statistically, refusing to learn from other people is probably the best bet.
Seems like they still don't get it.
It's not good faith representation of your client, it's not good lawyering to interrogate a chatbot for your case. Even if it could somehow only ever tell the truth.
It's not a "good faith mistake" - so I think they still don't understand the problem here.
It might be representative of greater society: when people interact with ChatGPT, they treat it as an oracle. I see it all the time on r/chatgpt - people say, "I asked ChatGPT about my long lost grandparent whose life I am researching. Apparently my grandma was a pilot!"
Seems like the lawyers are shocked the oracle lied but still think they were right to consult it.
Oh they get it. ChatGPT explained it to them, and then suggested that they take this position because it was the least bad option.
Like editing Wikipedia then citing the contents. Impossible to defend in good faith, but passes as a legit excuse in a public court.
Some people really know how to use public naivety to enforce the public’s powerlessness.
https://en.m.wikipedia.org/wiki/Wikipedia:Citing_Wikipedia
Sure, we all do it informally, but implicit in that is that you should check the real sources if you care about accuracy.
I don't see any proof that shows they intended to deceive rather than that they were clueless.
The attorney used it as a research tool, and even foolishly thought to use it to validate that it wasn't making shit up. It's a really embarrassing mistake that's only obvious in the eyes of people familiar with the technology. There is nothing about this that suggests they intended to deceive, only intent to be a lazy idiot.
This thread: do they not know what ChatGPT is? Did they not verify the output? How could they believe this?
Just maybe when an army of grifters jumps on the next hottest AI thing to advance their careers that kind of bald-faced lying has consequences?
Because if you can't, it's not the fault of "grifters"...
Then when a judge told them "hey, we can't find any of these cases, what the hell is going on?" they doubled down. They're basically saying "trust me bro, these rulings totally happened and they totally say we're right".
This is like your local major getting tricked into buying the Eiffel tower using public funds and then defending himself by saying he asked the guy he bought the Eiffel tower from and the guy said he was totally legit.
They lied to a judge, doubled down, showed further incompetence by admitting to not knowing how the legal reference they supposedly quoted was even structured, then lied to a judge again trying to get more time to get their story straight, and all it cost them was $5000? And even then they have the guts to say they disagree with the judge?
Levidow, Levidow & Oberman must either be run by fools or take the rest of the world for fools.
I can understand how the rubber stamping lawyer signing all the paperwork got off easy; he was just signing documents because the lawyer who supposedly did all the work for the case couldn't operate in the court the case was transfered to.
The other lawyer, the one who did the AI equivalent from copy pasting from the summary page of Google, should not be deemed fit to practice law.
They also have to write written apologies to their client, and all of the judges who were mentioned in their fake cases, with proof filed in the docket.
And they are the brunt of a very well-publicized lawsuit. The judge is effectively saying that all that publicity is punishment enough; he doesn't need to do anymore. He doesn't need to refer them to the bar for disbarment--the bar already knows about it. (The judge has no power to disbar them himself, he can only refer to the bar for disbarment.)
The biggest surprise is that they didn't get made to pay the defendant's legal fees as well, but that's because defendant didn't ask for any fees.
Oh, and the final punishment... they lost the case--they get no chance to redo their opposition to the motion to dismiss. Arguably, their client can also sue them for malpractice, and their malpractice carrier is likely to prefer to settle that rather than actually fight the case.
This isn't going to go away. This decision will dog them for the entire rest of their careers.
I get the impression from the events leading up to the use of GPT this is not the first time their competence led to outcomes that will "dog them for the entire rest of their careers."
I thought the judge said he couldn't force them to do that because it wouldn't be sincere.
They'll probably just have ChatGPT write their apologies for them.
As an AI language model...
I have been a good Bing.
"The April 25 affidavit was filed in response to the Orders of April 11 and 12, 2023, but is sworn to before a Notary Public on the 25th of January 2023. Mr. LoDuca should be prepared to address this anomaly on June 8, 2023 and bring with him a wet-ink signed copy of the affidavit with the notary’s signature and stamp."
a reference from the judge and the bar just knowing about it are very different.
Well, no, they also have to right letters and attach the sanctions order, and the transcript of the sanctions hearing to:
(1) their client in the case, and
(2) each of the real judges to whom they (on the basis of ChatGPT “research”) falsely attributed a decision.
But, yes, sanctions are fairly light (particularly the absence of a disciplinary referral).
> I can understand how the rubber stamping lawyer signing all the paperwork got off easy; he was just signing documents because the lawyer who supposedly did all the work for the case couldn't operate in the court the case was transfered to.
> The other lawyer, the one who did the AI equivalent from copy pasting from the summary page of Google, should not be deemed fit to practice law.
They both failed their duty in nearly identical ways; passing of someone else’s work to which they did not apply independent professional analysis as their professional work. If anything, the one doing so directly to the court should be more, not less, liable to outside sanction (if the issue was sanctions within the firm, that’s a different story, maybe.)
Once they showed up in court, one of them was unable to explain how to interpret a very basic "you should know this to practice law" case citation ("F.3d"[1]). Not only did they not "apply independent professional analysis" it seems they were woefully unprepared to even produce such a thing.
[1]: https://youtu.be/oqSYljRYDEM?t=1526
Sorry to be cynical, but this sounds like they delegated their defense to AI as well... in the near future this kind of argument will probably work in a lot of contexts.
God help us when it helps in court.
Why should you get off easy?
This is why the law firm's contention that they were acting in good faith is laughable.
When the tool itself shows a prominent disclaimer that it "May occasionally generate incorrect information," then the law firm can't reasonably argue that "We made a good-faith mistake in failing to believe that a piece of technology could be making up cases out of whole cloth."
It was either gross negligence or intentional malfeasance.
The most interesting thing to me is why the firm is continuing to try to excuse its actions, rather than taking full responsibility for them. I guess they assume that the classic Shaggy "It Wasn't Me" defense is less likely to lose them business, because they think so little of the clients they're trying to attract that they're wagering they'll buy it.
I have a friend who used to be a federal judge for immigration, and he said that immigration lawyers are especially the worst, because families need their services desperately, and these lawyers will take their money and spend NO time on their cases. They show up to court and are figuring stuff out real time.
He tore one lawyer a new asshole for doing exactly this for his client and ordered them to come back better prepared.
So the idea that some of them will use ChatGPT to do their work doesn't surprise me at all.
Given the marketing and the way Microsoft and Google were tripping over themselves to "catch up", the impression in the media and to the general public is that chat bots are a magic silver bullet for everything. Even a lot of tech people are unclear about exactly how these things work.
The case seems pretty clear: "They said it was artificial intelligence, but it wasn't very intelligent to make up cases. Look at the damages it's caused me..."
As entertaining as this is to ponder, I don’t think they’re that clever. I don’t think they’d do so much damage to their reputation for such a flimsy case. They’re just incompetent.
The good lawyer's job is not just to advocate the client's position — it's also to do "completed staff work" for the court in the process, even acknowledging any weaknesses in the client's case (which builds credibility).
As we've seen in the ChatGPT case, judges tend to respond badly when they think a lawyer is trying to put one over on them.
And conversely, judges often respond favorably when they perceive that a lawyer is being thorough and intellectually honest, not merely throwing up any bullshit argument that comes to mind. A fond memory from years ago is when a federal judge called me and my opponent — a well-known lawyer in our field — into his chambers to talk about a pending motion that involved an uncertain point of law, which we had extensively briefed. My opponent had not done an especially-great job; my own briefing had acknowledged the weak points in my client's position but also explained why my client should win anyway. The judge said, I don't really know what the answer is here. He then turned to me and said, But I'm more comfortable with your analysis. And he ruled my way – which won the case for my client (a company you might well be a customer of today).
I wonder when tech firms will actually be held responsible for false advertisement.
Lawyers admitted to the bar aren't innocents. They're "officers of the court", meaning the legal system relies on them to avoid BS in their citation of other cases and of statutes and laws. It's never been acceptable for a lawyer to sign off on something without reading it and making sure it's right.
What if registered professional civil engineers started asking generative AI for sewer-system or bridge designs and then didn't inspect them before approving them and handing them off to the guys with dozers and cement trucks? Not only would people fall into rivers, but the rivers would be polluted.
The real world consequences are similarly serious when lawyers make stuff up, or let their interns or word-processing software make stuff up.
I'd wager a guess that this will only get worse over time. LLMs will do more stuff for people - from helping school children to get through their homework, to higher learning assignments and all the way up to the "real" jobs. Along the way people might just place far too much trust in these tools. I'm pretty sure that's something we can all agree on.
What I do find striking here though, is the incredible laziness of the lawyers. Not even checking whether or not the cases actually happened? That's just bad lawyering. The AI actually helped the system to call them out on their bad lawyering.
That's not "bad things happening because AI" but just plain stupidity.