People improve models, models lift people. We advance the signal that keeps the loop turning.
San Francisco, CA
Joined October 2019
- Tweets78
- Following9
- Followers3.1K
- Likes234
Pareto AI retweeted
just ran GPT 5.5, Gemini 3.5 Flash, and GLM 5.2 through LEAPBench
GLM 5.2 beat every frontier model evaluated except Opus 4.7 and 4.8
Today we’re releasing LEAPBench, a benchmark for how efficiently models learn from feedback over many rounds.
Most evals only score whether a model eventually reaches a good solution. LEAPBench scores how many tries it took to get there.
Two systems can reach the same optimum while one needs 7 experiments and the other needs 27. Most benchmarks call that a tie, even though those 20 extra runs cost real time and materials.
LEAPBench is 55 multi-turn optimization tasks, 45 in biology and 10 education RCTs, built from 2,719 real experimental observations.
In each task, the model proposes parameters, sees the result, and iterates toward an optimum over as many as 30 rounds, e.g. one biology task involves maximizing antibiotic yield by tuning variables like pH, glucose concentration, temperature, and soy flour level.
Pareto AI retweeted
Today we’re releasing LEAPBench, a benchmark for how efficiently models learn from feedback over many rounds.
Most evals only score whether a model eventually reaches a good solution. LEAPBench scores how many tries it took to get there.
Two systems can reach the same optimum while one needs 7 experiments and the other needs 27. Most benchmarks call that a tie, even though those 20 extra runs cost real time and materials.
LEAPBench is 55 multi-turn optimization tasks, 45 in biology and 10 education RCTs, built from 2,719 real experimental observations.
In each task, the model proposes parameters, sees the result, and iterates toward an optimum over as many as 30 rounds, e.g. one biology task involves maximizing antibiotic yield by tuning variables like pH, glucose concentration, temperature, and soy flour level.
We added Claude Opus 4.8 to AttuneBench.
It takes the top overall score (54.55), ahead of Opus 4.6 (54.30) and GPT-5.5 (53.71).
It's also the strongest model we've tested at predicting users' emotion labels and overall conversation ratings.
2
8
3,438
The Opus family now holds all three top spots for predicting which response a user would prefer.
But the dissociation we reported in our paper remains.
Opus 4.8 improved on predicting what users wanted from individual replies (76.7% → 78.3%), yet still ranks near the bottom of the pool on that metric.
The observer-view vs. first-person gap narrowed, but did not close.
2
1
2,289
Opus 4.7, Opus 4.8, and Sonnet 4.6 all score effectively zero on identifying what specifically went wrong when a conversation missed a user's needs.
Opus 4.6 and Haiku 4.5 do not show the same pattern.
Predicting that a conversation failed and identifying why it failed appear to be separable capabilities.
Leaderboard:
public.attunebench.com
1
2
675
Pareto AI retweeted
Opus 4.8 outperforms every other model on AttuneBench
- best at picking the response humans actually preferred
- biggest MSCEIT four-branch jump of any Opus generation
- entire pairwise top-4 is now Anthropic models. non-Anthropic frontiers stall ~50%
Introducing Claude Opus 4.8: it builds on Opus 4.7 with sharper judgment, more honesty about its own progress, and the ability to work independently for longer than its predecessors.
Available today at the same price.
ALT Benchmark table showing how Claude Opus 4.8 compares to its predecessor and to other models on tests of coding, agentic skills, reasoning, and practical knowledge work tasks.
Pareto AI retweeted
1/ Today we're releasing AttuneBench, the first open EQ benchmark grounded in real multi-turn human-model conversations, scored against what the person actually felt and wanted at each turn.
Built by the research team at @pareto_ai in collaboration with @thoughtfullab.
Most existing EQ benchmarks rely on:
- synthetic prompts
- single-turn interactions
- third-party annotation
None directly measure how a model reads and responds to a real person across a full conversation.
We evaluated 11 leading models from major providers, across 200 conversations and 50,000+ first-person annotations.
Pareto AI retweeted
Do LLMs have metacognition? It is complicated, but mostly no — check out work from my team at @hellopareto
model confidence tracks a shared model-agnostic signal for fact recall, not true self-knowledge.
we tested metacognitive confidence across 19 frontier models on a closed-book SQuAD task. f1 scores look reasonable (0.6–0.8), but confidence and accuracy are nearly uncorrelated between models.
the variance traces to a single shared difficulty heuristic learned during training. models differ only in their decision threshold. claude is cautious. gpt is eager.
shifting one steering coefficient on mistral-7b recovers any target model's confidence profile at ~80% agreement.
full breakdown + methods in the article
LLMs don't fail because they lack information. They fail because they learn from unverified judgment.
That's a data problem, not a model problem.
3
839
Frontier models are trained to answer.
But in high-stakes domains, the safer behavior is often to pause, express uncertainty, and ask for more context instead of acting like a doctor who never asks.
Great work from @phoebeyao, Marilyn Zhang, and the team on a model capability that is still barely measured today.
Our CEO @phoebeyao on the limits of prompt-based AI safety.
“The future of AI isn't about replacing human expertise. It's about building better systems to capture expert insight at the edge of what's known and scale it into training data that actually moves model capabilities forward.”
Read how AI safety scales, based on real-world deployment with the UK AI Safety Institute.
An AI lab tried to cut costs by switching to a low-cost data vendor—only to face cascading quality failures.
After months of setbacks, they came back to Pareto.
Why? Because the best labs don’t gamble with data quality.
Read more: pareto.ai/blog/pitfalls-of-l…
Meet Gilbert Kamau—a data scientist who transitioned from actuarial science to AI. Gilbert shares his thoughts on the future of AI, his journey so far at Pareto, and the balance between work and passion. Read more here: pareto.ai/blog/behind-the-da…
Thrilled to announce our collaborative AI safety research with UCL, ML Alignment & Theory Scholars, and Anthropic won Best Paper at ICML! Honored to have played a role in this groundbreaking work.
Congrats to all involved! 🎉 #ICML2024 #BestPaperAward #AIResearch
3
957
Thank you for the kind words, Curtis! Hearing positive feedback from esteemed researchers and scientists at the top of their field inspires us to keep pushing boundaries.
#RLHF #EthicalAI #DataLabeling #ParetoAI
