Social Navigation

ChatGPT, other AI models that can be as good as human lawyers; Better performance on bar exam score


Artificial intelligence can now outperform most law school graduates on the bar exam, the grueling two-day exam that aspiring lawyers must pass to practice law in the United States, according to a new study published Wednesday.

GPT-4, the updated artificial intelligence model released this week by Microsoft-backed company OpenAI, scored 297 on the bar exam in a trial conducted by two law professors and employees of the legal technology firm Casetext.

The researchers found that this places GPT-4 in the 90th percentile of actual test takers and is enough to get them accepted to practice law in most states.

The Bar Exam assesses knowledge and reasoning and includes essays and performance tests intended to simulate legal work, as well as multiple choice questions.

“Large language models can meet the standard applied to human attorneys in nearly all jurisdictions in the United States by tackling complex tasks that require deep legal knowledge, reading comprehension, and writing ability,” the authors wrote.

Less than four months ago, two of the same researchers concluded that OpenAI’s previous big language model, ChatGPT, fell short of a bar test, highlighting how quickly the technology is improving.

The newer GPT-4 scored nearly 76 percent of the multiple-choice questions on the bar exam, up from about 50 percent in ChatGPT, beating the average human test-taker by more than 7 percent.

The National Conference of Bar Examinations, which designs the multiple-choice section, said in a statement Wednesday that lawyers have unique skills gained through education and experience that “no AI can currently match.”

Study co-author Daniel Martin Katz, a professor at Chicago-Kent College of Law, said in an interview that he was very surprised by GPT-4’s ability to produce highly coherent essay and benchmark answers.

“I’ve heard a lot of people say, ‘Well, she might get multiple choice,’ but she’ll never get the essays,” Katz said.

The AI ​​has also performed well on other standardized tests, including the SAT and GRE, but the union exam has gotten more attention. OpenAI touted the degree of success when it announced its latest model on Tuesday.

Bar exam instructor Sean Silverman attributed the emphasis on the bar exam to its widely recognized difficulty. The pass rate for the first time this year on the bar licensing exam was 78% among test takers who have spent three years in law school.

People may be less impressed to learn that an AI can pass a test designed for high school students, like the SAT, “rather than the test to become a lawyer,” Silverman said.

© Thomson Reuters 2023

From smartphones with rotatable screens or liquid cooling, to built-in augmented reality glasses and phones that owners can easily repair, we discuss the best devices we saw at MWC 2023 on the Orbital, Gadgets 360 podcast. Orbital is available on Spotify, Gaana, JioSaavn, and Google Podcasts Apple Podcasts, Amazon Music, and anywhere you get your podcasts.
Affiliate links may be generated automatically – see our Ethics Statement for details.

Joanna Swanson

Joanna Swanson is Europe correspondent at the Thomson Reuters Foundation based in Brussels covering politics, culture, business, climate change, society, economies and inclusive tech. With specific focus in breaking news, she has covered some of the world's most significant stories.