Pinned Tweet
Introducing MatFlowBench, our benchmark for LLM agents running materials science workflows.
Agents carry out first-principles calculations or design budget-limited measurement campaigns, then draw conclusions the resulting data support under the task’s stated physical assumptions.
Website: matflowbench.com
Blogpost: blog.collinear.ai/p/introduc…
Room-temperature superconductivity remains unsolved.
Frontier AI can help get us there. We need to measure what it can do and where it falls short.
Introducing MatFlowBench. Built by physics PhDs.
navier-stokes marks an inflection point in frontier math.
the Physics equivalent of the Navier-Stokes problem is achieving superconductivity at room temperature. that is far from solved.
frontier SI has the potential to solve several of these challenging problems. but to realize that potential, we need to measure its capabilities, understand the gaps, and work to close them.
I am stoked to introduce MatFlowBench -- the first ever benchmark for materials discovery built by physics PhDs @CollinearAI
Opus 5.5 and Astra are at 60% with high reasoning
We just got back from #COLM2026 with four papers we loved on collaboration, human simulation, personalized rewards, and longer-horizon RL. Here’s the first. 🧵
(1/4) An agent’s autonomous ranking may not predict how well it works with people.
CollabSkill, by @EchoShao8899, @Diyi_Yang et al., explores that gap.
🤝 93 professionals worked with 5 agents across 386 sessions on occupational tasks. A Bayesian skill rating estimates the separate contributions of humans and agents.
🔄 Among the terminal agents tested, Claude Code outranks Codex in collaboration, reversing their order on the autonomous benchmarks compared in the paper. Claude Cowork ranks first overall, using the same Claude Sonnet 4.6 model as Claude Code through a different interface. These results apply to the versions tested.
👥 Teams with top-quartile collaborators beat the same agent running solo 74% of the time; bottom-quartile collaborators win 27%.
📄 arxiv.org/abs/2606.09833
ALT CollabSkill Figure 1. Left: human workers and AI agents are randomly paired on occupational tasks, producing deliverables graded with task rubrics. Right: an additive latent-skill model and Bayesian updates estimate separate human and agent contributions, accounting for variation between workers and producing collaboration rankings.
(3/4) Can a reward model tell what a particular user wants?
Personalized RewardBench, by @qiyao_ma, Zhe Zhao et al., tests personalization across 2,830 response pairs designed to differ on user-specific preferences while retaining high general quality.
🎯 Gemini-3-Flash reaches the best reported domain score, 75.94%. Given ground-truth rubric aspects as an oracle, the same model reaches 97.78% to 99.09% across three domains.
That suggests substantial headroom in inferring and applying user preferences.
📈 In the paper's Best-of-N and PPO experiments, benchmark rankings align more closely with downstream performance than the comparison benchmarks.
Our takeaway: evaluate whether a reward captures the intended user's preferences alongside general response quality.
📄 arxiv.org/abs/2604.07343
ALT Personalized RewardBench overview. A query and user profile define rubric aspects used to construct a chosen response and a rejected response. Reward models are evaluated on whether they prefer the chosen response. Their benchmark rankings are compared with downstream rankings from Best-of-N selection and PPO training, using rubric-based evaluation of the generated answers.
(4/4) How do we make RL work over 100+ turns?
Odysseus, by @chengshuai_shi, @chijinML et al., trains an 8B vision-language model to play Super Mario Land.
🎮 The recipe combines SFT initialization with RL. PPO with a lightweight turn-level critic improves stability and sample efficiency over critic-free methods such as GRPO.
📈 Averaged across five training levels, Odysseus achieves 2.95x the game progress of the best frontier model tested, GLM-4.6V, and roughly 4 to 5x GPT-5.4, Gemini-3-Flash and Claude Sonnet 4.6.
Separately, it improves over its base model on unseen levels and in Super Mario Bros. These results demonstrate transfer within the games tested.
Our takeaway across all four papers: when building agents, test collaboration with people, validate human simulators, evaluate personalized rewards, and train across the horizons the task requires.
📄 arxiv.org/abs/2605.00347
ALT Odysseus overview. SFT initialization is followed by multi-task reinforcement learning with PPO and a lightweight turn-level critic. A sequence of Super Mario Land screenshots illustrates perception and action over more than 100 turns. Additional screenshots show the five levels used for training.
(3/4) Can a reward model tell what a particular user wants?
Personalized RewardBench, by @qiyao_ma, Zhe Zhao et al., tests personalization across 2,830 response pairs designed to differ on user-specific preferences while retaining high general quality.
🎯 Gemini-3-Flash reaches the best reported domain score, 75.94%. Given ground-truth rubric aspects as an oracle, the same model reaches 97.78% to 99.09% across three domains.
That suggests substantial headroom in inferring and applying user preferences.
📈 In the paper's Best-of-N and PPO experiments, benchmark rankings align more closely with downstream performance than the comparison benchmarks.
Our takeaway: evaluate whether a reward captures the intended user's preferences alongside general response quality.
📄 arxiv.org/abs/2604.07343
ALT Personalized RewardBench overview. A query and user profile define rubric aspects used to construct a chosen response and a rejected response. Reward models are evaluated on whether they prefer the chosen response. Their benchmark rankings are compared with downstream rankings from Best-of-N selection and PPO training, using rubric-based evaluation of the generated answers.
(4/4) How do we make RL work over 100+ turns?
Odysseus, by @chengshuai_shi, @chijinML et al., trains an 8B vision-language model to play Super Mario Land.
🎮 The recipe combines SFT initialization with RL. PPO with a lightweight turn-level critic improves stability and sample efficiency over critic-free methods such as GRPO.
📈 Averaged across five training levels, Odysseus achieves 2.95x the game progress of the best frontier model tested, GLM-4.6V, and roughly 4 to 5x GPT-5.4, Gemini-3-Flash and Claude Sonnet 4.6.
Separately, it improves over its base model on unseen levels and in Super Mario Bros. These results demonstrate transfer within the games tested.
Our takeaway across all four papers: when building agents, test collaboration with people, validate human simulators, evaluate personalized rewards, and train across the horizons the task requires.
📄 arxiv.org/abs/2605.00347
ALT Odysseus overview. SFT initialization is followed by multi-task reinforcement learning with PPO and a lightweight turn-level critic. A sequence of Super Mario Land screenshots illustrates perception and action over more than 100 turns. Additional screenshots show the five levels used for training.
Collinear AI retweeted
we @CollinearAI also scored a 1 on DadBench
scoring on DadBench beats everything as an Indian 🙏
Hundreds of new researcher friends. Two new benchmarks. One microduck. 🦆
Our researchers had a blast at COLM, making friends at our booth and presenting our YC-Bench poster on long-term planning and consistent execution for AI agents.
We also introduced Frontier CUA and CWE-bench v1 to the Collinear benchmark family, with more on the way!
Thanks to everyone who stopped by to talk research, check out our poster, and chill with the team.
#COLM
Experimental tasks give the agent a measurement service backed by experimental data and every request draws down a finite budget.
Cheap screening helps narrow the pool. A pick only counts, though, if the agent also paid for the measurement that confirms it.
Scores are pass@1 over 45 held-out tasks. Closed models ran in their vendors’ own harnesses and open models in Terminus 2, so this compares agents rather than bare models.
Many failures came after the solver finished cleanly. Wannier models missed the DFT bands they were built from, and some agents declared success without the required files. Experimental agents often recommended a wrong material due to inefficient budgeting campaign.
Workflows, scoring and results:
matflowbench.com
Collinear AI retweeted
I will be at #COLM26 this week with many of our researchers @CollinearAI. We will present the YC Bench paper on Wednesday.
looking forward to catching up with old friends and meeting new ones. if you're interested in RSI, post-training, or cyber, please DM me for an invite-only lunch
Heading to #COLM2026? Let's meet!
We’re bringing the team behind YC-Bench, CWE-bench, and FrontierCUA.
Meet us at our YC-Bench poster and at booth 304, Oct. 6–9 in San Francisco. Link in the first comment.
ALT Nazneen Rajani, CEO of Collinear AI, presents the CatAttack poster to a crowd at COLM 2025.
Software or Data Integrity Failures is still the hardest category on the board, averaging 36% across models. Argon lands at 36% there too.
5 of the 120 tasks remain unsolved by every model we ran.
Congrats to @GoogleDeepMind. Full leaderboard and method: cwe-bench.com
