People improve models, models lift people. We advance the signal that keeps the loop turning.
San Francisco, CA
Joined October 2019
- Tweets78
- Following9
- Followers3.1K
- Likes234
Pareto AI retweeted
just ran GPT 5.5, Gemini 3.5 Flash, and GLM 5.2 through LEAPBench
GLM 5.2 beat every frontier model evaluated except Opus 4.7 and 4.8
Today we’re releasing LEAPBench, a benchmark for how efficiently models learn from feedback over many rounds.
Most evals only score whether a model eventually reaches a good solution. LEAPBench scores how many tries it took to get there.
Two systems can reach the same optimum while one needs 7 experiments and the other needs 27. Most benchmarks call that a tie, even though those 20 extra runs cost real time and materials.
LEAPBench is 55 multi-turn optimization tasks, 45 in biology and 10 education RCTs, built from 2,719 real experimental observations.
In each task, the model proposes parameters, sees the result, and iterates toward an optimum over as many as 30 rounds, e.g. one biology task involves maximizing antibiotic yield by tuning variables like pH, glucose concentration, temperature, and soy flour level.
Pareto AI retweeted
Today we’re releasing LEAPBench, a benchmark for how efficiently models learn from feedback over many rounds.
Most evals only score whether a model eventually reaches a good solution. LEAPBench scores how many tries it took to get there.
Two systems can reach the same optimum while one needs 7 experiments and the other needs 27. Most benchmarks call that a tie, even though those 20 extra runs cost real time and materials.
LEAPBench is 55 multi-turn optimization tasks, 45 in biology and 10 education RCTs, built from 2,719 real experimental observations.
In each task, the model proposes parameters, sees the result, and iterates toward an optimum over as many as 30 rounds, e.g. one biology task involves maximizing antibiotic yield by tuning variables like pH, glucose concentration, temperature, and soy flour level.
Pareto AI retweeted
Opus 4.8 outperforms every other model on AttuneBench
- best at picking the response humans actually preferred
- biggest MSCEIT four-branch jump of any Opus generation
- entire pairwise top-4 is now Anthropic models. non-Anthropic frontiers stall ~50%
Introducing Claude Opus 4.8: it builds on Opus 4.7 with sharper judgment, more honesty about its own progress, and the ability to work independently for longer than its predecessors.
Available today at the same price.
ALT Benchmark table showing how Claude Opus 4.8 compares to its predecessor and to other models on tests of coding, agentic skills, reasoning, and practical knowledge work tasks.
Pareto AI retweeted
1/ Today we're releasing AttuneBench, the first open EQ benchmark grounded in real multi-turn human-model conversations, scored against what the person actually felt and wanted at each turn.
Built by the research team at @pareto_ai in collaboration with @thoughtfullab.
Most existing EQ benchmarks rely on:
- synthetic prompts
- single-turn interactions
- third-party annotation
None directly measure how a model reads and responds to a real person across a full conversation.
We evaluated 11 leading models from major providers, across 200 conversations and 50,000+ first-person annotations.
Pareto AI retweeted
Do LLMs have metacognition? It is complicated, but mostly no — check out work from my team at @hellopareto
model confidence tracks a shared model-agnostic signal for fact recall, not true self-knowledge.
we tested metacognitive confidence across 19 frontier models on a closed-book SQuAD task. f1 scores look reasonable (0.6–0.8), but confidence and accuracy are nearly uncorrelated between models.
the variance traces to a single shared difficulty heuristic learned during training. models differ only in their decision threshold. claude is cautious. gpt is eager.
shifting one steering coefficient on mistral-7b recovers any target model's confidence profile at ~80% agreement.
full breakdown + methods in the article
