Akshit (@akshitwt) header
Akshit (@akshitwt) profile picture

gpu scheduler @genreasoning ml @cambridge_uni. cs @iiit_hyderabad. futurebound.

london
Joined June 2023
Photo by Akshit (@akshitwt)Photo by Akshit (@akshitwt)Photo by Akshit (@akshitwt)Photo by Akshit (@akshitwt)Photo by Akshit (@akshitwt)Photo by Akshit (@akshitwt)Photo by Akshit (@akshitwt)Photo by Akshit (@akshitwt)Photo by Akshit (@akshitwt)Photo by Akshit (@akshitwt)
FutureSim Q3 is here! Opus 5.5 dethrones GPT, displaying strong priors and incredible test-time adaptation. Opus 5.5 is a beast, calling 50+ subagents and taking over 400 research notes during the 90 day simulation while also being 50% more token-efficient than 6.1 Sol. (1/5)
5
7
8
81
13,404
Some good insights here. (Most of the stuff you see on X is handcrafted harnesses; we cover here a more important problem which is how to *train* agents to co-ordinate)
🐝 How to Train a Swarm In our first blog post, we explain how latency pressure can incentivise parallel work, the communication primitives agents need to co-ordinate, and outstanding challenges in training large agent collectives. gr.inc/blog/how-to-train-a-s…
3
4
68
7,770
Excited to share first glimpses of our research on agent collectives, a year in the making. I’ve been thinking about parallel reasoning for some time since I was at Meta. Why automate researchers when we should be automating communities? Genius is great, but scenius matters too. We’re sharing a recipe today (RLCR) that is producing interesting results for us at a small scale. We’re using it to train our Avalon-M1 series of models, starting with a small 120B model. Usual caveats apply. We are a small team (8 people!), we are conducting more experiments and ablations as we speak, and we’re still training and seeing gains. Let us know your feedback. My summary of findings in thread below. 🧵
Introducing Reinforcement Learning from Collective Rewards (RLCR). RLCR is a recipe that trains language models as agent organisations, which share a message board and workspace. We use it to train Avalon-M1-Mini (120B) and study its behaviour. We find a number of interesting results, including: 🛠️ Learnt organisational hierarchies and divisions of labour 🕵️ Increased collective verification, through hires of verifiers and testers 👥 The rise of collective thinking, where agents think more about other agents and how to help them We believe RLCR can provide a laboratory for studying collective intelligence and multi-agent safety. Blog post and technical report 👇
15
19
2
134
11,672
Introducing Reinforcement Learning from Collective Rewards (RLCR). RLCR is a recipe that trains language models as agent organisations, which share a message board and workspace. We use it to train Avalon-M1-Mini (120B) and study its behaviour. We find a number of interesting results, including: 🛠️ Learnt organisational hierarchies and divisions of labour 🕵️ Increased collective verification, through hires of verifiers and testers 👥 The rise of collective thinking, where agents think more about other agents and how to help them We believe RLCR can provide a laboratory for studying collective intelligence and multi-agent safety. Blog post and technical report 👇
95
180
77
1,967
199,862
One thing I found missing from the ICLR submissions debate: Capping submission per author would have only very modestly reduced the load in prior years. Even under the elimination by submission order (which is unrealistically strict), going down to 3 per author would not change the trend by a lot. More realistic modes (such as minimizing removals) would remove even fewer papers. And, in reality, changes in authorship patterns will likely reduce further invalidate gains from capping. I think as people who have been in the field for a while, we tend to underestimate just how much the number of unique authors grows every year, as AI/ML consumes all of our CS department students, our math, physics and econ departments, and an evergrowing number of industry positions and independent researchers
6
19
5
128
11,222
Akshit retweeted
We are releasing FrontierSWE v2, our updated ultra-long horizon coding benchmark V2 features an expanded task suite and improved methodology. We see large performance gaps between frontier models, with Claude Fable 5.1 leading by a wide margin
89
129
86
1,248
321,045
We can finally talk about it: We found a way to extract hidden reasoning of frontier models using a vulnerability in the APIs of every frontier AI company. We verified that our reasoning token count matches billed API thinking tokens 1:1 for most of the prompts we queried.
382
1,958
620
13,499
3,546,969
💥 Major update of PostTrainBench: - We significantly improved the reward hacking judge. this pushed the open-weight models down the leaderboard. - We added GPT-5.6 Sol, Opus 5, Kimi K3, Grok 4.5. - We reran Fable 5 -> now it's the best model by a large margin (41.8% vs. 36.2% of GPT-5.6 Sol). - We also reran other experiments that had infra issues. This is what we officially call PostTrainBench v1.1. The old results of v1.0 are still available on our website! All these changes should lead to a more accurate picture of where we are in terms of autonomous post-training. The best model is Fable 5 (41.79%) while the Official Instruct Models baseline is at 51.14%. The gap is narrowing very fast. We will need harder benchmarks sooner than expected...
Introducing PostTrainBench v1.1 As agents became more capable, they exposed gaps in our original run-integrity pipeline: test item targeted data, external API distillation, and even direct lookup of previous PostTrainBench traces. We've also added new agents: GPT-5.6 (Sol), Fable 5, Opus 5, Grok 4.5, and Kimi K3. Fable 5 leads at 41.79%. posttrainbench.com/blog/post…
7
9
132
13,001
Major FutureSim updates! 🥽 Visualized trajectories: extremely long horizon, 1000s of actions per run and dozens of compactions, with summaries! Goldmine of interesting agent behaviors across Fable 5, GPT 5.5, GLM 5.2 etc! 📈 @OpenReward integration! You can now use FutureSim, incl. on your own data, with our favourite standard! The @GenReasoning was super supportive in making this happen! Some cool observations in 🧵
3
7
1
54
10,213
New blogpost: What does your benchmark actually measure? I argue what evals end up measuring is the simplest way to improve performance on them. When building a new benchmark, we are often too focused on minimizing scores at launch. This often makes benchmarks too easy to increase scores on, without reflect improvements in the underlying capability it intended to measure. Instead, I find it useful to think about, and apply, all the ways someone could improve scores on a benchmark. This often reveals unintended shortcuts that were enough to make large eval gains. Thinking about how a benchmark behaves under optimization pressure helps interpret results on it, and predict behaviors that will surface in future runs, as if it's successful, the community/models will try everything to hill-climb it. Full post below:
3
18
5
110
17,315
Akshit retweeted
Claude Fable 5 ranks #1 on FrontierSWE. This represents the biggest capability jump we have observed since releasing the benchmark On many tasks, Fable 5 works productively for close to 20 hours and fully saturates tasks that were effectively out of reach for earlier models
13
20
13
277
37,864
Akshit retweeted
We believe that better training data will come from creative research and engineering ideas, not from hiring annotators. Here are some of the open problems we are working on:
2
17
6
86
21,405
Most people training agentic LLMs with RL right now have a silently broken training loop and have no idea. Here's the trap: single-turn RL works beautifully. Clean curves, sane rewards, everything converges. Then you add tools so the model can act mid-rollout, and things get weird. Loss spikes for no reason. Eventually a shape-mismatch error. The culprit: every time you parse the model's output to detect a tool call, then re-tokenize the updated conversation for the next turn, you're rolling the dice. Usually the round-trip gives back the same tokens. Sometimes it doesn't and your gradient lands on a sequence the model never actually sampled. No crash. Just quietly wrong math and a useless gradient signal. The fix is one rule: never re-encode tokens you've decoded. Keep the sampled tokens in one buffer, never re-render them, and both failure modes disappear. That's Token-In, Token-Out done right. Our team just published a beautiful deep-dive on exactly this, including an audit across the major open-weights model families showing most chat templates already support it. Required reading if you're doing multi-turn RL 🤗🔥 qgallouedec-tito.hf.space/
48
132
24
1,107
1,014,650
Akshit retweeted
We evaluated Claude Opus 4.8 on FrontierSWE ahead of today's release. It is now the best-performing model on FrontierSWE.
6
12
5
122
20,311
Great take on the importance of evals as upstream of everything, including training oftentimes. I'd go one step further and say that proper evaluation is becoming and exciting standalone discipline within AI: davidstutz.de/LpZDE
I’ve left Google DeepMind after an amazing chapter. I’m incredibly grateful for the people I worked with, the things we built, and the lessons I learned from taking frontier AI research into production. DeepMind shaped how I think about research, product, evaluation, and what it takes to build AI systems at real scale. As I wrap up this chapter, I wrote down something I’ve been thinking about a lot: evals. We’re good at evaluating the models we have. We’re much worse at evaluating the models we’re about to build — especially if they cross into a new capability regime. We will have self-evolving models, but before that, we need self-evolving evaluations. wanglun1996.github.io/blog/y…
2
2
34
6,922
Your RL post-training may be sabotaging your LLM’s test-time scaling! Conventional RL pretends that you can collapse all reward signals *upfront* into a single *scalar reward*. We introduce Vector Policy Optimization (VPO), which natively maximizes *vector-valued* rewards, boosting test time search performance, even on the original scalar.
38
132
42
930
254,247
💥Today we release InferenceBench, our next benchmark after PostTrainBench that measures progress on AI R&D automation. AI R&D automation will very likely unfold gradually, starting from “boring” tasks like inference speed optimization that are very easily verifiable (accuracy + inference time). We show a rather negative result for current frontier agents. They are not good at system-level engineering and managing complex dependencies. They do show non-trivial performance, but they fail compared to a simple baseline: hyperparameter tuning of vLLM/SGLang hyperparameters. Importantly, InferenceBench tests *open-ended* inference optimization capabilities. This is different from more narrow benchmarks like KernelBench that only let agents optimize kernels (which is a very valuable task, too!). The benchmark is intentionally open-ended, so the poor performance of the agents is not an underelicitation issue. The agents have everything needed to succeed, but they still fail because they are not yet reliable enough for this task. Our results suggest an inverse scaling phenomenon: Claude Sonnet 4.6 and GLM-5 rank highly because they more often preserve simple, valid, high-performing final servers, while several larger models show stronger peak runs but lose utility through brittle final-state choices. This contrasts with benchmarks where rankings track raw capability (e.g., SWE-Bench, Terminal-Bench, PostTrainBench, FrontierSWE). One of the primary bottlenecks we have clearly observed is the lack of diversity of strategies: nearly all agents just use vLLM, without exploring alternatives. Overall, proper exploration is lacking: the current agents are not ready to tackle broad enough goals and get stuck after the first found solution (such as vLLM). I’m sure future agents will do much better, but here is where we are now. This benchmark is our 2nd one in a suite of benchmarks that will track the progress on AI R&D automation. We will develop many more benchmarks that will cover different aspects of AI R&D automation, culminating in recursive self-improvement. Stay tuned!
13
46
6
348
42,650
Akshit retweeted
Could an AI company lose control of its own agents? To find out, Anthropic, Google, Meta, and OpenAI let us (1) test their best internal models with CoT access, (2) review non-public info about capabilities, alignment, and control. The result: our first Frontier Risk Report.
33
196
65
943
390,332
🚨Excited to announce Agent-BRACE! LLM agents in long-horizon POMDPs either blow up their context with raw history or summarize it, discarding uncertainty by collapsing belief into a point estimate. Agent-BRACE decouples the agent into belief state + policy models, jointly trained via RL. Key takeaways: 1️⃣ 🎯The belief state model produces a structured approximation of the belief distribution as a set of atomic natural-language claims with ordinal verbalized certainty labels ranging from certain to unknown. The policy conditions on this compact belief rather than the full history. 2️⃣ 📈 Outperforms strong RL baselines on long-horizon partially observable embodied language environments while maintaining a near-constant context window independent of episode length. 3️⃣ 🔄 The learned belief becomes increasingly calibrated as evidence accumulates, and epistemic belief decreases over time: the proportion of claims that the agent has the strongest level of belief in grows from 21% → 52% over an episode. 👇🧵
2
40
8
69
16,875
I will be in Rio next week for #ICLR2026 🇧🇷 to present our poster on evaluating long horizon execution (24 April 10:30 AM - 1:00 PM at Pavilion 3 P3-#2011) Reach out if you want to chat about long horizon evals, forecasting and RL environments in general!
Why does horizon length grow exponentially as shown in the METR plot? Our new paper investigates this by isolating the execution capabilities of LLMs. Here's why you shouldn't be fooled by slowing progress on typical short-task benchmarks... 🧵
3
17
3,482