Associate Prof @TAMULawSchool JD @Harvard_Law | PhD @MITBrainandCog | Law, language, cognition, AI
Fort Worth, TX
Joined June 2020
- Tweets187
- Following406
- Followers688
- Likes1.2K
Pinned Tweet
Can AI ace law exams?
Last year, OpenAI announced GPT-4 got 90th percentile on the bar.
Here we (a) refute 90th percentile claim; (b) replicate/extend recent work on GPT capabilities; (c) discuss implications for law profession.
Now open, AI&Law: link.springer.com/article/10…
1/
Eric Martínez retweeted
This paper finds that frontier AI models correctly format Bluebook citations only 42.6% of the time on average. The best results come from combining AI with a deterministic rule engine rather than relying on the model alone. spkl.io/60117Pymc
#LegalTech #AI
Eric Martínez retweeted
Don't ask AI to Bluebook your citations until you read this study. Heck, maybe don't even ask?
Frontier models produced correct citations just 42.6% of the time.
AI can solve open math problems and find security flaws.
But can it reliably format a legal citation?
New paper w/ Matthew Dahl: across frontier models, the answer is no. We then build a system that
does much better.
papers.ssrn.com/sol3/papers.…
THREAD 1/
Eric Martínez retweeted
Disappointing but important.
"We develop a new benchmark of 2,058 Bluebook queries and show that, on average, frontier language models produce a fully compliant legal citation only 42.6% of the time in a zero-shot setting."
Excited to dive more into this...
Eric Martínez retweeted
This is an extremely interesting and well-written paper with a very surprising result. Makes me wonder whether we've really internalized how important jaggedness is and how long it will stick around.
Also, I will pay any bluebook-savvy law student who wants to earn some extra money $20 an hour to go through the methodology and experimental design with a fine toothed comb and come up with persnickety little criticisms, not because I think you're likely to find anything that discredits the validity of the central finding but because it will really piss Eric off and that will be hilarious. DM me if interested. Maybe start with the "models can't display small caps" thing, they seem very defensive about small caps in the footnotes. I smell weakness.
AI can solve open math problems and find security flaws.
But can it reliably format a legal citation?
New paper w/ Matthew Dahl: across frontier models, the answer is no. We then build a system that
does much better.
papers.ssrn.com/sol3/papers.…
THREAD 1/
Third, for the design of legal AI.
We argue for a “formal turn”: use LLMs for language understanding, and auditable code for exact rule
execution. Start with law’s formal core before its penumbra.
Paper: papers.ssrn.com/sol3/papers.…
15/
Second, for “law-following” AI.
The Bluebook is intricate, but less ambiguous than statutes or common-law standards. If models
cannot reliably follow its explicit rules, confidence in autonomous compliance with more contested +
dynamic bodies of law should be tempered.
14/
Why does this matter? First, for legal drudgery.
Bluebooking is formal, centralized, repetitive, and objectively verifiable—an unusually favorable case
for automation. If frontier LLMs cannot reliably handle it alone, routine legal work is not yet a solved
problem.
13/
This neuro-symbolic approach achieves an average accuracy increase of 32.4 percentage points over
zero-shot, and total accuracy of up to 85.5% on our benchmark.
12/
Finally, we developed a different approach: let the LLM parse, and let a rule engine format.
The model classifies the query + extracts citation elements into structured JSON. A deterministic
formatter then applies the Bluebook rules.
11/
Third, we asked whether simply “giving it the rules” helps.
Supplying 17K–67K tokens of task-relevant Bluebook rules raised GPT-5.6 Sol’s accuracy from 42.2%
to 63.1%—better, but still wrong on more than 1 in 3 tasks.
10/
Second, we preregistered an experiment: a frontier reasoning model entered the annual “write-on”
competitions of five top-30 law reviews.
The AI scored well below the human average. At four, it scored below every successful candidate; at
the fifth, it beat only one.
9/
Performance was highly uneven: 71.5% on short-form cases, but 18.1% on court filings and 6.6% on
electronic statutes. Whitepages accuracy was 27.5%, versus 57.7% for Bluepages.
Errors were substantial: incorrect citations required 23.3 character-level edits on average.
8/
Across six frontier models, average accuracy on the open tasks was just 42.6%. No model exceeded
55%.
Enabling reasoning led to only modest gains: GPT-5.6 Sol rose from 42.2% to 47.5%.
7/
First, we introduce a benchmark of 2,058 expert-labeled Bluebook queries: 914 open-formatting tasks
+ 1,144 diagnostic cloze tasks, spanning both the practitioner-facing Bluepages and law-review-focused
Whitepages.
6/
Yet it remains an open question how well AI models can actually Bluebook.
Here we make four contributions: a new benchmark; a comparison with human write-on candidates; a
test of rule retrieval; and an AI-augmented rule-following system.
5/
Few legal tasks better fit the description of drudgery than Bluebooking: applying 500+ pages of
citation rules.
It is ubiquitous, time-consuming, famously hated—and, in principle, mechanical.
4/
Nowhere is this promise more prominent than in law.
Thomson Reuters says AI can “make quick work of routine tasks” so lawyers can focus on “high-value
work.”
Harvey: “Delegate the Work. Own the Judgment.”
3/
