Jonathan Roberts (@JRobertsAI) header
Jonathan Roberts (@JRobertsAI) profile picture

Jonathan Roberts

@JRobertsAI

Evals @MistralAI | Previously, PhD @Cambridge_Uni

 Cambridge
jonathanroberts42.github.io/
Joined December 2022
  • Tweets208
  • Following442
  • Followers639
57Media
Photo by Jonathan Roberts (@JRobertsAI)Photo by Jonathan Roberts (@JRobertsAI)Photo by Jonathan Roberts (@JRobertsAI)Photo by Jonathan Roberts (@JRobertsAI)Photo by Jonathan Roberts (@JRobertsAI)Photo by Jonathan Roberts (@JRobertsAI)Photo by Jonathan Roberts (@JRobertsAI)Photo by Jonathan Roberts (@JRobertsAI)Photo by Jonathan Roberts (@JRobertsAI)Photo by Jonathan Roberts (@JRobertsAI)
Is computer vision “solved”? Not yet Current models score 0% on ZeroBench 🧵1/6
Image from Jonathan Roberts (@JRobertsAI)
57
239
71
2,534
1,450,003
📢 Muse Spark Open w/tools scores 50% pass@5 on ZeroBench
Alexandr Wang (@alexandr_wang) profile pictureAlexandr Wang
@alexandr_wangOct 15
1/ muse spark open is the most capable US open-weight model to date. 816B total params, 42B active. 1M-token context. natively multimodal. built for coding and agentic work like long-horizon tasks, hard software engineering, multi-step reasoning. america! 🇺🇲
Image from Alexandr Wang (@alexandr_wang)
1
3
192
👀 DeepSeek-V4.1-Flash-1017 scores 51% pass@5 on ZeroBench w/ tools
DeepSeek (@deepseek_ai) profile pictureDeepSeek
@deepseek_aiOct 17
🚀 DeepSeek-V4.1-Flash-1017 is here! Stronger benchmark performance, faster task completion, and shorter outputs than the previous version—a Pareto improvement in performance and cost. Now live on our website, app, and API. 🛠️ Optimized for @DeepSeekHarness, Tencent CodeBuddy, WorkBuddy, and more. Upgrade to DeepSeek Harness v0.2.1-rc.1 or later for the best experience with the new model. 1/4
Image from DeepSeek (@deepseek_ai)
Image from DeepSeek (@deepseek_ai)
3
137
nothing to report opus 5.5 max looks on/above trend for the ai futures model (added to the eci graph from the august update) wide CIs yada yada
Image from Samuel Albanie 🇬🇧 (@SamuelAlbanie)
5
3
1
131
10,070
Jonathan Roberts retweeted
Mistral AI (@MistralAI) profile picture
Mistral AI
@MistralAIOct 6
Mistral Large 4 is one of the world's strongest AI models for cybersecurity.
Image from Mistral AI (@MistralAI)
104
206
41
2,814
137,588
Jonathan Roberts retweeted
Mistral AI (@MistralAI) profile picture
Mistral AI
@MistralAIOct 6
Meet Mistral Large 4, aka Le Chonk. • 1T parameters, natively multimodal. 49B active. It is the best open weights model from US or Europe on aggregated benchmarks. • State-of-the-art on critical workloads, including cyber defense, manufacturing and finance and it surpasses closed frontier models on visual grounding. • Forged in Europe end-to-end and is deployable from Europe via our own Mistral Cloud infrastructure. • Available to all via API today. Working with cybersecurity partners privately. Open weights release end of October.
🤖 Made with AI
2,035
4,562
2,813
40,609
5,762,370
More details on the project page: zerobench.github.io/
1
36
Opus 5.5 (max) is no.2 on ZeroBench 43% pass@5 25% pass^5 Now, 77% of questions have been solved at least once
Image from Jonathan Roberts (@JRobertsAI)
2
2
7
284
Project page: zerobench.github.io/
73
All three GPT-6 models sit on the ZeroBench performance-cost pareto frontier pass@5 | pass^5 GPT-6 Astra (max) 51% | 35% GPT-6 Sol (max) 41% | 17% GPT-6 Luna (max) 20% | 7%
Image from Jonathan Roberts (@JRobertsAI)
4
1
20
1,173
More details: zerobench.github.io/
2
329
GPT-6 Astra (max) on ZeroBench: pass@5: 51% (prev. SOTA: 30%) pass^5: 35% (prev. SOTA: 13%) If this reflects a genuine capability improvement, it is a significant step forward in both performance and consistency
Image from Jonathan Roberts (@JRobertsAI)
6
1
3
65
7,333
Jonathan Roberts retweeted
AI at Meta (@AIatMeta) profile picture
AI at Meta
@AIatMetaAug 20
Now let’s take a look at how Muse Spark 1.2 performs across visual reasoning, chart understanding, and knowledge-intensive tasks.
Image from AI at Meta (@AIatMeta)
4
6
1
68
10,423
Impressive! Muse Spark 1.2 w/ tools surpasses 50% pass@5 on ZeroBench!
Alexandr Wang (@alexandr_wang) profile pictureAlexandr Wang
@alexandr_wangAug 20
Replying to @alexandr_wang
2/ muse spark 1.2 performs quite strongly across a wide variety of multimodal capabilities and evals.
Image from Alexandr Wang (@alexandr_wang)
Image from Alexandr Wang (@alexandr_wang)
Image from Alexandr Wang (@alexandr_wang)
3
345
Project page: zerobench.github.io/
72
Grok 4.6 (xhigh) on ZeroBench (no tool use): pass@5 / pass^5 Grok 4.6: 17% / 7% GPT-5.6 Sol (SOTA): 30% / 13% More evaluation details available on the project page
Image from Jonathan Roberts (@JRobertsAI)
2
3
273

No more items