About | HeinOnline Law Journal Library | HeinOnline Law Journal Library | HeinOnline

LOG IN
LOG IN


3 J.L. & Empirical Analysis 2 (2026)

handle is hein.journals/jnlolwadem3 and id is 1 raw text is: Original Research Article Grading Machines: Can Al Exam-Grading Replace Law Professors? journal of Law and Empirical Analysis 2026, Vol. 3(I) 2-22 © The Author(s) 2026 Article reuse guidelines: sagepub~com/journals-perrnissions D OI: 10.1 177/2755323X261434265 journals.sagepub.com/home/lex SSage Kevin L. Cope' ,Jens Frankenreiter2, Scott Hirst3, Eric A. Posner , Daniel Schwarczs , and Dane Thorley6 Abstract In the past few years, large language models (LLMs) have achieved significant technical advances, enabling legal-advocacy organizations to adopt them as complements to-or substitutes for-lawyers and other human experts. The role of LLMs in legal education, however, is underexplored. While several studies have examined LLMs' performance in taking law school exams, finding mixed results, there have been no published studies systematically analyzing LLMs' competence at one of law professors' chief responsibilities: grading law school exams. This paper presents results of an analysis of how LLMs perform in evaluating student responses to legal analysis questions of the kind typically contained in law school exams. The data come from exams in four subjects administered attop-30 U.S. law schools. Unlike some projects in computer or data science, our goal is not to design a new LLM that minimizes error or that maximizes agreement with human graders. Rather, we seek to determine whether existing models-which can be straightforwardly applied by most professors and students-are already suitable for law exam evaluation. We find that, when provided with a detailed rubric, the LLM grades correlate with the human grader at Pearson correlation coefficients of up to 0.93. Our findings suggest that, even if they do not fully replace humans in the near future, LLMs could soon be put to valuable tasks by law school professors, such as reviewing and validating professor grading, providing substantive feedback on ungraded midterms, and providing students feedback on self-administered practice exams. Keywords artificial intelligence, grading, large language models, GPT, legal pedagogy 1. Introduction Led by OpenAI's Generative Pre-trained Transformer (GPT) model family, large language models (LLMs) have achieved significant technical progress over the past several years. These advances have inspired legal technology firms to ex- plore how LLMs might assist lawyers in tasks as varied as document review, contract and motion drafting, and brief writing and editing. In many cases, LLMs are being used to complement the work of lawyers, and-at least for some discrete tasks-they are partly or fully substituting for them. But given the speed of the technology's evolution, it is still unclear if, when, and to what extent LLMs will become a genuinely suitable substitute for human legal analysis. The questions typically contained in law school exams offer a valuable way to assess this technology's capacity for legal analysis. These exams test foundational doctrinal knowledge, demand creative and multifaceted problem solving, and, importantly, come with a built-in comparison group of human test-takers and graders. For these reasons, several studies have examined LLMs' performance in taking law school exams, finding mixed (but increasingly impres- sive) results (e.g., Fan et al., 2025). Yet no published studies 'School of Law, University of Virginia, Charlottesville, VA, USA 2School of Law, Washington University in St. Louis, St Louis, MO, USA 3School of Law, Boston University, Boston, MA, USA 4School of Law, University of Chicago, Chicago, IL, USA 5School of Law, University of Minnesota, Minneapolis, MN, USA 6School of Law, Brigham Young University, Provo, UT, USA Corresponding Author: Kevin L. Cope, School of Law, University of Virginia, 580 Massie Road, Charlottesville, VA 22903, USA. Email: kcope@Iaw.virginia.edu Creative Commons Non Commercial CC BY-NC: This article is distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 License (https://creativecommons.org/licenses/by-nc/4.0/) which permits non-commercial use, reproduction and distribution of the work without further permission provided the original work is attributed as specified on the SAGE and Open Access pages (https://us.sagepub~com/en-us/nam/open-access-at-sage).