AI Secretly Helped Write California Bar Exam, Sparking Uproar (arstechnica.com) 20
An anonymous reader quotes a report from Ars Technica: On Monday, the State Bar of California revealed that it used AI to develop a portion of multiple-choice questions on its February 2025 bar exam, causing outrage among law school faculty and test takers. The admission comes after weeks of complaints about technical problems and irregularities during the exam administration, reports the Los Angeles Times. The State Bar disclosed that its psychometrician (a person skilled in administrating psychological tests), ACS Ventures, created 23 of the 171 scored multiple-choice questions with AI assistance. Another 48 questions came from a first-year law student exam, while Kaplan Exam Services developed the remaining 100 questions.
The State Bar defended its practices, telling the LA Times that all questions underwent review by content validation panels and subject matter experts before the exam. "The ACS questions were developed with the assistance of AI and subsequently reviewed by content validation panels and a subject matter expert in advance of the exam," wrote State Bar Executive Director Leah Wilson in a press release. According to the LA Times, the revelation has drawn strong criticism from several legal education experts. "The debacle that was the February 2025 bar exam is worse than we imagined," said Mary Basick, assistant dean of academic skills at the University of California, Irvine School of Law. "I'm almost speechless. Having the questions drafted by non-lawyers using artificial intelligence is just unbelievable." Katie Moran, an associate professor at the University of San Francisco School of Law who specializes in bar exam preparation, called it "a staggering admission." She pointed out that the same company that drafted AI-generated questions also evaluated and approved them for use on the exam. The report notes that the AI disclosure follows technical glitches with the February exam (like login issues, screen lag, and confusing questions), which led to a federal lawsuit against Meazure Learning and calls for a State Bar audit.
The State Bar defended its practices, telling the LA Times that all questions underwent review by content validation panels and subject matter experts before the exam. "The ACS questions were developed with the assistance of AI and subsequently reviewed by content validation panels and a subject matter expert in advance of the exam," wrote State Bar Executive Director Leah Wilson in a press release. According to the LA Times, the revelation has drawn strong criticism from several legal education experts. "The debacle that was the February 2025 bar exam is worse than we imagined," said Mary Basick, assistant dean of academic skills at the University of California, Irvine School of Law. "I'm almost speechless. Having the questions drafted by non-lawyers using artificial intelligence is just unbelievable." Katie Moran, an associate professor at the University of San Francisco School of Law who specializes in bar exam preparation, called it "a staggering admission." She pointed out that the same company that drafted AI-generated questions also evaluated and approved them for use on the exam. The report notes that the AI disclosure follows technical glitches with the February exam (like login issues, screen lag, and confusing questions), which led to a federal lawsuit against Meazure Learning and calls for a State Bar audit.
First year law student exam (Score:2)
Re:First year law student exam (Score:4, Insightful)
I am more interested in the fact that 48 questions came from a first-year law student exam. Wouldn't missing even one of these raise questions about the exam takers knowledge?
Eh, that's a standard that is a bit too high: law students and lawyers aren't superhumans. Would you be able to get 48/48 questions right in an exam that you took 3-4 years after the class in which you learned them? If the answer is yes, do you think ALL of your peers could have?
Re: (Score:2)
Could I get 48/48 on a first year exam?
Possibly. But even if it's a 46/48 or 47/48, it's still a high passing score.
Could my classmates? I went to school with a lot of retards so probably not. And I'm 110% okay with the retards failing the test in any field of study.
Re: First year law student exam (Score:2)
How do you say "you're all gonna be replaced by AI some day" without saying it directly? Use it to make their tests. ;-)
"No, put your hand in the OPEN side of the box." (Score:2)
Re: (Score:1)
Re: "No, put your hand in the OPEN side of the box (Score:3)
Nitpick, but the test is referred to as the "human test" or "test for humanity", while the gom jabbar is the poison. The gom jabbar is not necessarily the only way to run the test. You could just as easily point a lasgun at their head.
Re: (Score:1)
The test for humanity is that you gut out and bear the pain of the nerve-induction box, not just because you don't want to die but because you are plotting your revenge against who is subjecting you to this torture.
Not only did Paul Atreides amaze the Reverend Mother by bearing much more pain than anyone else, the revenge he carried out against the Reverend Mother was way beyond anything she could have even imagined.
As to gom jabbar vs lasgun, first of all lasgun weren't threatening to people in a cult
Re: "No, put your hand in the OPEN side of the bo (Score:2)
"lasgun weren't threatening to people in a culture where they had "shields.""
Shields are visible and it would take longer/more movement to activate a shield generator than it would to simply knock aside the gom jabbar needle.
Re: (Score:2)
Secondly, Dune is pretty much the same story as The Godfather with Paul Atreides being the Michael Corelone character.apart from a few changes.
Most of what you said was wrong/off in some way. This, however, deserves specific clarification. It's generally best practice to compare the later work to the former, not the other way around. Because Dune came first, it would be more accurate wording to say The Godfather is pretty much the same story as Dune.
Tempest in a teapot? (Score:5, Insightful)
The ACS questions were developed with the assistance of AI and subsequently reviewed by content validation panels and a subject matter expert in advance of the exam...
Because of the subsequent review by qualified humans, I don't see this as a problem. But the scenario described in TFS is just begging for a 'slippery slope' argument.
I think it's inevitable that in the future, one AI will be tasked with checking the work of another AI. Then, in a storm of stupidity, the policy will become having the AI check its own homework.
After that, of course, the practice will devolve into just trusting the initial AI output, with no verification step. Welcome to the AI apocalypse!
Re: (Score:2)
The ACS questions were developed with the assistance of AI and subsequently reviewed by content validation panels and a subject matter expert in advance of the exam...
Because of the subsequent review by qualified humans, I don't see this as a problem. But the scenario described in TFS is just begging for a 'slippery slope' argument.
I think it's inevitable that in the future, one AI will be tasked with checking the work of another AI. Then, in a storm of stupidity, the policy will become having the AI check its own homework.
After that, of course, the practice will devolve into just trusting the initial AI output, with no verification step. Welcome to the AI apocalypse!
My concern in this particular instance is that the company using the AI was also in charge of reviewing the questions generated by AI. And the end result was "confusing questions" (straight from the summary) which was partially the reason the entire test (along with login issues and screen lag) was called into question. I'm thinking there's bigger systemic issues here than just having AI involved in the process, but having AI involved in the process clearly wasn't helping anything.
Though I know what you're
Re: (Score:2)
After that, of course, the practice will devolve into just trusting the initial AI output, with no verification step. Welcome to the AI apocalypse!
This implicit trust of AI with no human proofreading and sanity check is not only a problem for AI but for everything in every field. Even if the creator of the test questions or anything else is an expert, having no human proofreading and sanity check is a problem. Imagine a Nobel prize winning author who wrote a book without multiple passes by proofreaders and editors. There's no way errors wouldn't creep through.
If we ever get to the point where AI can be trusted with the proofreading and sanity check
Would they have counted? (Score:5, Interesting)
The real question is if these AI-generated questions were going to be counted in the final score, or if they were merely being evaluated for inclusion in future exams.
A colleague of mine is getting his MD, and the periodic tests they take contain a fair fraction of questions that are being evaluated for inclusion on future exams, but do not count toward the present score. Could the same have been happening here, but that important distinction being lost in the clamorous din?
As with any professional qualification exam, there is a certain level of knowledge that must be demonstrated. As long as the questions being used to demonstrate that knowledge are vetted by experts in the field and validated before being officially used, does it really matter who wrote them or how?
Re: (Score:2)
AI generated questions aren't a problem if those taking the exams actually understand the question and provide the correct answer. This goes to a basic mistake that too many people make, not understanding the difference between AI being a tool for humans, and AI being used to skip having humans be involved in the work. The people complaining the most haven't asked, "are the questions worded clearly enough for humans to be able to understand what is being asked?" That's all they need to be concerned wi
So, in CA (Score:1)
The Bars are owned by the State?
Re: So, in CA (Score:2)
Re: (Score:2)
Re: (Score:2)
So? (Score:3)
What exactly is the issue here? If the questions were vetted by professionals and found to be appropriate, does it matter if it was an expert human or machine or cat walking across the keyboard that wrote them to begin with?