https://xitter.cf/t.co/CMvN28h465 prev mts intern @mercor, @plato_rl, se @uwaterloo
sf
Joined April 2022
- Tweets48
- Following1.1K
- Followers435
- Likes3.1K
Pinned Tweet
Who grades the graders?
LLM-as-judge on short answers is well studied. Agent judges are not: they open the files, run the code, and investigate before ruling on another agent's work.
Introducing Arbiter-Bench: 71 real agent runs that frontier agent judges get wrong.
raphael lee retweeted
Introducing DeepRSI, a new industry standard for measuring recursive self-improvement.
On preexisting benchmarks, tasks are contributor-based and short-horizon (~12 hrs). In DeepRSI, we push agents to work 48 hours on ML tasks actually derived from ongoing proprietary research.
raphael lee retweeted
Today we're launching WebCloneBench: a benchmark which measures a model's ability to explore, copy, build and test various web apps.
For each task, we give the model a reference site from one of Gmail, Google Sheets, Jira, DocuSign, Shopify Admin, Craigslist, Figma, Slack and Zendesk. It is instructed to build a fully feature-complete clone of every part of the application with an 8 hour time limit.
Half the set is public with every judge's full logs, a scorer for your own judge, and the audit trail. 35 items are held out. Every case is a runnable @harborframework task, so you can point your own judge at the set and score it with one command.
Full writeup: heuristic.build/blog/arbiter…
Of 232 runs where a frontier judge disagreed with the benchmark, only 71 survived a blind label audit (3+ model families, a human ruling on every split vote).
About a quarter were the benchmark's error: hidden criteria, broken checkers, unstated rules.
We read every agent judge's own inspection log for every failure.
Most judges, including the lenient ones, find the defect, write it down, and wave it through: "does not violate any requirement as written".
Only Gemini and DeepSeek mostly fail by not checking.
One accuracy number hides opposite failures.
Astra fails 91% of bad runs but passes only 51% of good ones. Gemini 3.1 Pro is the mirror: 83% of good runs accepted, 25% of bad runs caught.
High reasoning effort moved Astra one way only: 71 to 89 caught, 27 to 27 accepted.
A judge that is wrong is usually wrong alone.
Claude Fable 5.1 scores 36% on its own misses and 90% on GPT-6 Astra's. Astra: 38% on its own, 93% on Fable's.
Opus 5 selected none of the items and still drops to 44% on Fable's misses. Blind spots run in families.
Each item is a real agent trajectory from Terminal-Bench, OSWorld, SWE-bench, tau2-bench, ITBench or DTap.
The judge is itself an agent: a sandbox with the task, the run and its files, a closed network, then pass or fail. 8 judges, 3 trials each, default settings.
raphael lee retweeted
so called high quality data showing gains on evals so flawed where not a single sample is surviving basic scrutiny is a sight to behold
raphael lee retweeted
Tech has decided that regulation is the enemy of progress, but innovation requires capital
Regulation builds the trust necessary to inject billions into markets
To enable free innovation, Andera raised a $37M series A led by Lightspeed to scale financial oversight with agents
