Jonathan Roberts

Opus 5.5 (max) is no.2 on ZeroBench 43% pass@5 25% pass^5 Now, 77% of questions have been solved at least once

All three GPT-6 models sit on the ZeroBench performance-cost pareto frontier pass@5 | pass^5 GPT-6 Astra (max) 51% | 35% GPT-6 Sol (max) 41% | 17% GPT-6 Luna (max) 20% | 7%

GPT-6 Astra (max) on ZeroBench: pass@5: 51% (prev. SOTA: 30%) pass^5: 35% (prev. SOTA: 13%) If this reflects a genuine capability improvement, it is a significant step forward in both performance and consistency

Grok 4.6 (xhigh) on ZeroBench (no tool use): pass@5 / pass^5 Grok 4.6: 17% / 7% GPT-5.6 Sol (SOTA): 30% / 13% More evaluation details available on the project page

It’s been great seeing ZeroBench evaluated as part of several recent model releases from @Alibaba_Qwen, @Kimi_Moonshot and ByteDance We’ve added an externally reported results leaderboard and included these scores in our visualisations

Across all our evals, 70% of ZeroBench questions have been answered correctly at least once Here’s the distribution for the top 5 models zerobench.github.io/

After our latest evals, these are the top 3 models on ZeroBench (no tool use): pass@5 / pass^5 GPT-5.6 Sol (max): 30% / 13% Claude Opus 5 (max): 26% / 11% Claude Fable 5 (max): 24% / 9% GPT-5.6 Sol is the first to reach the 30% human baseline on pass@5

Claude Fable 5 sets a new SOTA on GRAB-lite: 74% @ClaudeDevs are pushing the frontier The best models scored around 20% when GRAB was released On the current trajectory, GRAB-lite is projected to be ~100% by mid-2027 A reminder that tracking the frontier needs hard evals

Revisiting transport networks after ~3 years In 2023, we evaluated whether GPT-4 could recreate the Hong Kong MTR, and the result felt remarkable at the time Now, Claude Fable 5 can one-shot a live, interactive MTR map with real-time train positions

Answer distributions for top models across 5 samples Similar aggregate scores but different reliability profiles ZeroBench still has substantial headroom in both reliability and peak performance Leaderboard: zerobench.github.io Data: huggingface.co/datasets/jona…

Claude Fable 5 is strong on ZeroBench, but not a clear breakthrough 23% pass@5 (tied SOTA) 8% pass^5 (SOTA 10%) 3.6% refusal rate For comparison, other recent releases (pass@5 / pass^5): Opus 4.8: 17 / 4 Gemini 3.5 Flash: 19 / 5 A good result, but still plenty of headroom

The matrix shows how often the models answered each question correctly across 5 samples Leaderboard: zerobench.github.io Data: huggingface.co/datasets/jona…

GPT-5.5 (xhigh) sets a new pass^5 high score on ZeroBench pass@5: 22% (SOTA 23%) pass^5: 10% (prev. SOTA 8%) Best 5/5 reliability so far Strong result from @sama and the OpenAI team

📣📣 New SOTA for GRAB-lite, our graph analysis benchmark GPT-5.4 crushes Opus 4.6: 71.0% vs 45.6% Impressive work from @gdb and team Updated leaderboard now live

There’s now a Node SDK for @warpsurfai Run browser workflows from TypeScript on Node.js Use it in your own scripts, tools, and pipelines Details and repos below 👇