However, over half of the bar exam is an essay-form test, which is where GPT seems to have made the most advancement, so I thought it would be interesting to give it an actual essay questions and judge its response.
This is what I got: https://aidev.codes/u/ilaksh/law1/edit
I tried to give it some hints to give a better answer. Note that this site uses text-davinci-003 with temperature 0. Similar to ChatGPT but not the same model.
Sadly, while that's a bit more logical, that's even more wrong on the law. Recording deeds isn't ever time-barred and and being a grantor's daughter doesn't satisfy notice. I will admit that's very creative though haha. It's regurgitating less of the question, but pulling in the wrong laws to replace it.
I took the bar exam. I passed, but it was Illinois, which has low standards and high passage rates. Take that for what it is.
On the essay portion, the scoring is about 1/3 focused on knowing the legal doctrine that is being tested, and 2/3 on the structure, clarity, and style of the response. (State the law, analyze how the law applies to the facts presented, assess the strength of the counterargument, come to the conclusion, go back and state the conclusion at the beginning.). In Illinois, the passing score is about 65%, so in theory you could pass the essay part while knowing no law at all.
Of course, you will not do so well on the multiple choice. I believe a skilled test taker with no law knowledge could triangulate answers to get to around ... 33%?
Of course, with statistical learning, and enough data, you could train an LLM to pass the bar.
Honestly, given how robotic studying for the bar exam was, I am only surprised it took until 2022-3
Whether or not it's illegal, the ABA looks out for its own, and they would never permit AI to threaten the comfortable arrangement they've made for members of their industry.
The legal profession is not only self-regulating, but since they make up like 95% of legislators, they get to regulate everyone else too! Very masterful, even more so than the AMA
I highly doubt chatgpt is even remotely near being a lawyer. Lawyering is all about logical argument from a specific context and nitpicky details, which chatgpt is known to be bad at.
Anyways it probably should be illegal if someone is selling "legal services" but not doing it properly for the same reason that if a dr just did what chat-gpt said, that should also be illegal.
The future is unknowable. I have my doubts, but ultimately that is a bridge to cross when and if we get there. Notwithstanding how impressive chatgpt is, right now we are nowhere near that bridge so no point speculating.
It'll be extremely telling to see how they handle the introduction of AI in the coming years. Lawyers could very well find themselves being the first group of truly upper crust white collar professionals decimated and yet they also have the highest potential for protectionism. This is really shaping up to be the most spectacular tech vs establishment fights we've seen in a long while.
The disruption will be from within the legal profession. Some lawyers will gain a great deal. Others will lose a great deal.
There is a lot of differentiation in terms of the level of skill and effort legal work takes. Some firms will specialize in automating away routine legal work, and have just enough lawyers to comply with unauthorized-practice-of-law laws.
High value legal work still contains a lot of grunt work, which will also become more automated. Already partners at big firms review the work of a pyramid of lawyers and non-lawyers. Many of those tasks can be done by an LLM instead. As long as the partner signs the result, it is the same. As these partners learn to trust the LLMs more than the humans, a lot of human jobs will fall by the wayside.
A lot of legal work is already copy-paste from form books or other prior art, with a lot of filling in the blanks, search-replace, and so on. What is the difference between that and using an LLM to draft the document? Was that integration clause from the form book carefully reviewed by a human lawyer to make sure it fit the situation? No.
Many law firms have a deep library of forms that are battle tested, and lawyers of long experience know what provisions are good and which may not survive a court challenge. That kind of contextual know-how is what you are paying for. Why not use the adversarial techniques from deep learning to analyze how various provisions have actually performed over time? Then generate the document based on that statistical knowledge, with highlighting to show what is risky in a document? Why pay for an expensive lawyer when a cheap one will do that work for you and can give you that explanation, plus alternative legal provisions?
I am out of the profession because I did not want to be on the automated-out side. If there is automating to be done, why not do the automating?
I agree, but what if in 2-3 years, a lowly ranked state college law school grad + LawGPT is perceived to be better than an Ivy grad?
(Edit: perceived, because that matters for prestige and this billing, and I think the law school rankings may in many cases convey tiers that aren't really there)
Tough call, I think. All you need is some greed and people who don't care about what happens to the future. You ambulance chasing lawyers (aka personal injury lawyers) are not the same as your big lawfirm partner.
We've seen many instances of larger populations get sold down the river (many examples in outsourcing)
The ABA doesn’t regulate the practice of law. Its only real power is accrediting law schools, and on that front it has done such a poor job at protectionism that there’s twice as many law school graduates each year as jobs. If you just need someone who can pass the bar exam, you can hire one on Craigslist for way cheaper than an electrician, plumber, or IT guy.
I recall from my bar prep class that you get a substantial amount of partial credit if you state a legal test and apply it correctly — even if the test you state is not correct. That is, if you forget one of the prongs of a test, you lose very few points. The same applies if you completely make up a prong and use it in place of one of the actual prongs.
If this is still true, then GPT should be able to do well if it is trained to spit out rules (even incorrect ones) and then apply the facts of the scenario to the stated rules.
I can assure you that the minimum will be set reasonably at first and then be exceeded within a year. Then corporate influence will prevent it from being raised on a reasonable schedule, just like with minimum wage.
If we follow this incredible path over the coming century, all I can think of is the future doctor from "The Little Black Bag" [1]. Physician's tools become so advanced as to lower the bar for future doctors. Maybe future attorneys could rely more on AI to provide insight into relevant existing and potential novel case law.
What's interesting about this is once you can an AI that is a legal representative-- you can just spin up thousands or millions of them and start suing people for everything. Like legal spamming...
I tried this question, but I added "MEE Question 6" at the top so it has a better context and the answer seems somewhat better? No idea if it would get any credit though, maybe the author of this blog post can chime in.
> It is likely that Abe is entitled to possession of Whiteacre. The deed that Owen delivered to Abe twenty years ago constituted a valid conveyance of the property to Abe, even though it was never recorded. Abe subsequently took possession of the land and made use of it for the next eleven years, paying the property taxes and allowing others to use the land. This constitutes "adverse possession," which is a legal principle that allows someone to gain ownership of a piece of property through continuous, open, and notorious possession of the property for a certain period of time.
> In this case, Abe has satisfied the requirements for adverse possession under the laws of State B, as he has been in possession of the property openly, continuously, and notoriously for more than the statutory period of time required by State B. Doris and Buyer are considered "subsequent purchasers," as they acquired the property after Abe's conveyance to them, and they do not have a valid claim to the property because they did not have notice of Abe's conveyance. Therefore, it is likely that Abe would be entitled to possession of Whiteacre in this case.
Wait until Congress critters get a hold of this and start using it to stuff their bills with seemingly innocuous legal fluff text as a way to bury important policy decisions then force votes before everyone has time to sift through it all.
They don't need to. Do you think any of your politicians read that few thousand page long spending bill before passing it? Maybe you are alleging it already happens. After all the govt has the resources to generate this software before anyone else.
> ChatGPT's essay possibly would score zero points. It lacks any correct rules and any logical analysis.
I am really getting tired of the “how many angels could fit on the head of a pin?” style navel-gazing articles that get attention strictly because they use the word ChatGPT in the title.
Could ChatGPT write some pointless fluff that lands on the front page of HN? Almost certainly.
> Could ChatGPT write some pointless fluff that lands on the front page of HN? Almost certainly.
It's sad that despite all the hype I have seen about ChatGPT over the last month it seems that in the real world its only real application is to generate low effort blogspam.
The MBE is a knowledge test. It tests how well you can memorize a bunch of state law that you’ll never use after the bar exam. Constrained by the multiple choice format, I’d suspect an AI could actually do quite well on the MBE portion. But nobody really cares about the bar exam.
The exam the industry is actually built around is the LSAT, which decides the rank of law school you get into, which in turn decides what career pathways are available. That’s a logical reasoning test with no knowledge component. It would be interesting to apply GPT to that.
He said exactly the same thing. A lawyer needs to remember those laws, and knowing how to find them. A software developer needs to have knowledge in security or whatever, and know how to code (which uses this knowledge).
No one remembers sentences word by word. Knowing how to find them means finding words which will lead you for the final sentence. Anyway, comparing with a software developer is an unvalid comparison. They are not the same.
The Multistate Bar Exam, of course, does not test state law. After all, how would they decide which state’s law to test?
Accordingly, the NCBE, in their unending wisdom, quizzes applicants on the common law of 18th century England.
I’d also argue that quite a few people care about the bar exam (e.g., the people who study for it for months, or who fail it and cannot practice) while there are loads of lawyers who care very little about the LSAT (e.g., I never took it).
The MBE tests a generic version of state law that reflects the majority rule in the states or model codes such as the Model Penal Code or UCC. (It also tests the federal rules of civil procedure and evidence.) The law of all states except Louisiana is based on English common law, and so the MBE reflects that. But it is not a test of 18th century English common law. For example the criminal law questions reflect the model penal code which was developed in the 1960s.
I guess I’ll renew my question, then: which state law does the MBE test? The answer “a generic mishmash of least common denominator state uniform law” ain’t no state I ever heard of, anticipating your answer. If you are barred in this state I guess I could take your word for it.
Also I’m not even sure what to say about the idea that “larceny by trick” is statutory criminal law under the MPC but more power to you if it is, I guess.
"it's impressive for what it is, but ..." is probably a good way to describe chatGPT's performance for a lot of tasks - but what's interesting is the vast diversity in these tasks. I've asked it write me a few poems and it does much better than I could. I've asked it to write songs (including chord progressions, instrument/synth suggestions) for specific genres and it was OK, not great. I've seen people use it for fake maury transcripts, essays in the style of X, marketing copy, and as here, answer legal questions (which as far as I can tell it did far better than I could, not having any legal training whatsoever). chatGPT does seem quite a good (though unreliable and perhaps overconfident) generalist. I'm wondering how much better GPTs trained/fine tuned specifically for a given field - say, contract law - would do for that task.
54 comments
[ 4.3 ms ] story [ 124 ms ] threadHowever, over half of the bar exam is an essay-form test, which is where GPT seems to have made the most advancement, so I thought it would be interesting to give it an actual essay questions and judge its response.
On the essay portion, the scoring is about 1/3 focused on knowing the legal doctrine that is being tested, and 2/3 on the structure, clarity, and style of the response. (State the law, analyze how the law applies to the facts presented, assess the strength of the counterargument, come to the conclusion, go back and state the conclusion at the beginning.). In Illinois, the passing score is about 65%, so in theory you could pass the essay part while knowing no law at all.
Of course, you will not do so well on the multiple choice. I believe a skilled test taker with no law knowledge could triangulate answers to get to around ... 33%?
Of course, with statistical learning, and enough data, you could train an LLM to pass the bar.
Honestly, given how robotic studying for the bar exam was, I am only surprised it took until 2022-3
Tangentially related: DoNotPay's (probably illegal) plans to use LLMs in actual court sessions https://twitter.com/jbrowder1/status/1612312707398795264
The legal profession is not only self-regulating, but since they make up like 95% of legislators, they get to regulate everyone else too! Very masterful, even more so than the AMA
Anyways it probably should be illegal if someone is selling "legal services" but not doing it properly for the same reason that if a dr just did what chat-gpt said, that should also be illegal.
There is a lot of differentiation in terms of the level of skill and effort legal work takes. Some firms will specialize in automating away routine legal work, and have just enough lawyers to comply with unauthorized-practice-of-law laws.
High value legal work still contains a lot of grunt work, which will also become more automated. Already partners at big firms review the work of a pyramid of lawyers and non-lawyers. Many of those tasks can be done by an LLM instead. As long as the partner signs the result, it is the same. As these partners learn to trust the LLMs more than the humans, a lot of human jobs will fall by the wayside.
A lot of legal work is already copy-paste from form books or other prior art, with a lot of filling in the blanks, search-replace, and so on. What is the difference between that and using an LLM to draft the document? Was that integration clause from the form book carefully reviewed by a human lawyer to make sure it fit the situation? No.
Many law firms have a deep library of forms that are battle tested, and lawyers of long experience know what provisions are good and which may not survive a court challenge. That kind of contextual know-how is what you are paying for. Why not use the adversarial techniques from deep learning to analyze how various provisions have actually performed over time? Then generate the document based on that statistical knowledge, with highlighting to show what is risky in a document? Why pay for an expensive lawyer when a cheap one will do that work for you and can give you that explanation, plus alternative legal provisions?
I am out of the profession because I did not want to be on the automated-out side. If there is automating to be done, why not do the automating?
We've seen many instances of larger populations get sold down the river (many examples in outsourcing)
Steve Lehto put up a video[1] two days ago about this and explained a few problems with it.
The first hurdle they've got, is that in most (all?) courtrooms you can't have your phone on.
[1] https://www.youtube.com/watch?v=425O2e87uBM
If this is still true, then GPT should be able to do well if it is trained to spit out rules (even incorrect ones) and then apply the facts of the scenario to the stated rules.
[1] https://en.wikipedia.org/wiki/The_Little_Black_Bag
> It is likely that Abe is entitled to possession of Whiteacre. The deed that Owen delivered to Abe twenty years ago constituted a valid conveyance of the property to Abe, even though it was never recorded. Abe subsequently took possession of the land and made use of it for the next eleven years, paying the property taxes and allowing others to use the land. This constitutes "adverse possession," which is a legal principle that allows someone to gain ownership of a piece of property through continuous, open, and notorious possession of the property for a certain period of time.
> In this case, Abe has satisfied the requirements for adverse possession under the laws of State B, as he has been in possession of the property openly, continuously, and notoriously for more than the statutory period of time required by State B. Doris and Buyer are considered "subsequent purchasers," as they acquired the property after Abe's conveyance to them, and they do not have a valid claim to the property because they did not have notice of Abe's conveyance. Therefore, it is likely that Abe would be entitled to possession of Whiteacre in this case.
>ChatGPT, summarize the following bill and be sure to note any unexpected clauses: <text of bill here>
> ChatGPT's essay possibly would score zero points. It lacks any correct rules and any logical analysis.
I am really getting tired of the “how many angels could fit on the head of a pin?” style navel-gazing articles that get attention strictly because they use the word ChatGPT in the title.
Could ChatGPT write some pointless fluff that lands on the front page of HN? Almost certainly.
It's sad that despite all the hype I have seen about ChatGPT over the last month it seems that in the real world its only real application is to generate low effort blogspam.
https://news.ycombinator.com/item?id=34307843
https://news.ycombinator.com/item?id=34231083
The exam the industry is actually built around is the LSAT, which decides the rank of law school you get into, which in turn decides what career pathways are available. That’s a logical reasoning test with no knowledge component. It would be interesting to apply GPT to that.
What makes you a good lawyer is if you remember all those laws, and you know how to find them in your mind. Those laws are mostly necessary.
Even if they’re the ones you need.
It’s wrong.
> What makes you a good lawyer is if you remember all those laws, and you know how to find them in your mind.
Accordingly, the NCBE, in their unending wisdom, quizzes applicants on the common law of 18th century England.
I’d also argue that quite a few people care about the bar exam (e.g., the people who study for it for months, or who fail it and cannot practice) while there are loads of lawyers who care very little about the LSAT (e.g., I never took it).
Also I’m not even sure what to say about the idea that “larceny by trick” is statutory criminal law under the MPC but more power to you if it is, I guess.