A high-signal data ecosystem leveraging data from industry & academia to deliver post-training to frontier labs, neolabs, & enterprises building advanced models
San Francisco, CA
Joined May 2026
- Tweets319
- Following261
- Followers86
- Likes183
A better benchmark score isn't automatically a scientific insight. Agent4Sci, from Ming Zhong @MingZhong_ and coauthors, found agents improving scientific ML models mostly through familiar engineering tricks. Prompts for research ideas and experiment management helped them explore further. The useful distinction is whether an agent improves a result or learns something about the problem. arxiv.org/html/2610.28763
ReasonCore AI retweeted
FrontierCS 2 is our new benchmark for CS research & development.
Each task is based on a paper: we delete the contribution from the released code, and ask agents to outdo the original method, as judged by the authors' chosen metrics. Agents don't have access to the paper or web.
SciExam for ENSO, from Shilong Liu @Shilong_Liu_AI and coauthors, makes agents write their own checks, then freezes those checks before model development. Hidden tests judge the final climate model. That stops the agent from quietly changing what counts as success to fit its results. arxiv.org/html/2610.10513
FrontierCS 2 from Wenhao Chai @wenhaocha1 at Princeton asks a harder question than whether the tests pass: can an agent rebuild a research paper's key idea and beat the authors' released code? On its SVG-EAR task, none of twelve models passed. GPT-6 Astra came closest. Beating the human reference across every required metric is a useful bar for research-level coding.
Today we are launching FrontierCS 2, the most frontier benchmark for AI agents in computer science R&D. We built 48 tasks from published papers across 13 areas of CS, refining each one with experts in its field. Excited to co-launch FrontierCS 2 with @kaggle @GoogleDeepMind
One accuracy score hides where a model is shaky. LG-Bench, from Shanghai AI Laboratory, links related life-science questions into a graph, then checks whether a model answers each neighborhood consistently and where its mistakes cluster. That is closer to how knowledge gaps actually show up. arxiv.org/abs/2610.26787
Models read raw experimental data badly but write decent code to analyze it. On CME-SpectrumBench, from Jin Gene Wong and Zhi-Xun Shen at Stanford with Berkeley coauthors, a Python interpreter closed most of the gap with handwritten code on simulated spectra, except when noise was severe. So the useful skill is turning data into code, not eyeballing arrays. ReasonCore has 5,000 CritPt-style physics tasks in inventory. arxiv.org/abs/2610.24304
cxbench, from researchers at Lero, University of Limerick, shows a customer-service agent can get the right final answer and still break the rules on the way. About 80% of runs that passed the outcome check failed a process check, like skipping a required disclosure. Scoring only the end state counts those as wins. arxiv.org/abs/2610.25954
Giving an agent tools and memory does not reliably make it more accurate.
A new measurement study ran three open models in three open-source agent setups. The extra machinery rescued some questions the model missed alone, but broke others it had answered correctly without it. Skill instructions and retained history did not reliably help either.
Test the whole agent, not just the model. arxiv.org/abs/2610.20675
ReasonCore AI retweeted
Self-improving research agents are one of the most discussed ideas in AI right now.
(bookmark it)
Épi is a research agent that keeps working on the same research problem across many model sessions. It set 120 records, including an interval for the n-queens constant about 84 times narrower than before.
It has a persistent research history, hierarchical memory, versioned branches it can return to, and an independent reviewer. A fixed verifier scores every output, so the agent cannot change how it is judged.
Other results include exact certificates showing the Cohn-Elkies bound cannot be sharp in dimensions 10 and 11, and a computer-assisted proof of a case of Tuza's conjecture for split graphs.
A lighter version, Épi-turbo, writes solvers for routing and scheduling problems. On 30 FrontierOR tasks, it scores 5.4% higher than Codex at 64% of the model cost, with the same model and reasoning effort.
arxiv.org/abs/2610.24944
Chat with Paper: academy.dair.ai/papers/epi-t…
A plan can read well and still have the steps in the wrong order.
StepBench takes thousands of real how-to procedures and tests whether an AI agent can spot a missing step, a wrong step, or a shuffled sequence. Putting steps back in order was the hardest part. The best model got it right 63% of the time, and only about a third of the time on long procedures with nine or more steps. arxiv.org/abs/2610.26679
Agents that plan should be graded on the order, not just the final summary.
You can make a benchmark harder without changing what the task is.
Researchers kept each task's goal fixed and changed how hard it is to get there: what information the agent can see, how resources connect, what depends on what. Every model they tested did worse on the harder versions. Training on those harder runs beat training on the original ones.
Same goal, tougher world. That is a way to keep tests from going stale. ReasonCore has tasks in the area of recursive self-improvement. arxiv.org/html/2610.16397
ReasonCore AI retweeted
Most AI coding tools prove themselves on one benchmark, built by a small team doing the hard, thankless part.
ReasonCore AI retweeted
Underrated failure mode in RL envs. Overfit tests usually aren't letting hacks through, but they also reject correct answers that just took a different route. A second oracle is a cheap way to catch it. I need to try this out, but double work to write two oracle solutions!
ReasonCore AI retweeted
harbor tasks can now ship multiple oracle solutions.
think of solution/ as the unit tests for tests/.
Single solution only proves your tests accept one approach. Whereas multiple solutions catch tests that are overfit to it, and make verifier easier to debug and robust.
Saying a fix works is not the same as proving it. In CyberArena, a common defender mistake was claiming a patch worked after checking logs from before the patch went live. For security agents, the grade should come from attacking the live fix again. arxiv.org/html/2610.19005
A check you can peek at before you commit stops being a check.
A small Yonsei preprint had an AI agent design machine-learning pipelines and explain why each design works. A verifier tested those explanations by removing what the agent said mattered and rerunning.
When the agent could probe the verifier before committing, it kept the claims that passed. Held-out faithfulness fell, and the in-loop metrics never showed it.
Preliminary: one model, six tasks, and the decline is partly confounded by how the score treats small pipelines. The useful part is the method: stress-test a verifier before you trust it. arxiv.org/html/2610.22779
Customers rarely say what they actually need. FDE-Bench hands agents a vague business request and makes them ask questions first. The best setup matched the expert solution in only about 6% of runs. Good questions are part of the job, not a warm-up. arxiv.org/html/2610.14539
Plotly's new data analytics exam has a useful warning buried in it. The model with the cheaper token price made nearly three times as many calls per question and read four times as many tokens, so its full run cost seven times more than the pricier model's, and took three times longer. Token price lists tell you what a call costs, not what an answer costs. For agents that plan their own calls, the meter that matters is the finished task. plotly.com/blog/introducing-…
