NDSEG Fellow / PhD @uncnlp with @mohitban47 working on automating env/data generation + program synthesis formerly @allenai @neclabsamerica
Boston, USA
Joined June 2023
- Tweets778
- Following1.6K
- Followers782
- Likes4K
Zaid Khan retweeted
Excited to share our work on MiRL! 🚀
💡 More training isn't always the answer — learning from the right failures matters.
Many agent failures stem from a small set of recurring misbehaviors. Rather than simply scaling up RL training, MiRL focuses on identifying and correcting these systematic errors.
🎯 By targeting where agents go wrong, we achieve stronger performance with substantially less training compute.
Great work by the team! Check out the thread below 👇
🤔 How can we efficiently train terminal agents to avoid misbehaviors?
😮 We try error-targeted learning on terminal agents and find that, with < 200 envs in TMax-15K & 35% of TMax's rollout, we can match TMax-4B full recipe.
🚨Introducing MiRL (Misbehavior-driven RL): MiRL first profiles the errors of a policy to find mistakes that recur across many environments, which we call generalizable misbehaviors. It then trains the policy with RL, starting rollouts at the steps where these misbehaviors occur and rewards a rollout only if it passes the tests and avoids the misbehavior.
Key takeaways
🔍 On Qwen3.5-4B, we find that 8 dominant misbehaviors account for 52.1% of wrong steps and appear in 75.4% of the environments.
🔥 MiRL outperforms TMax-4B, the released checkpoint trained with the full TMax recipe on Terminal-Bench Lite (51.8% vs. 47.4%) and matches it on Terminal-Bench 2.1 (21.3% vs. 20.6%), while using 35% of its rollouts and 189 selected training envs.
🧵👇 [1/9]
Error-targeted learning is a direction I'm excited about for agentic post-training. The idea is to find mistakes an agent is making, and focus compute on training away those mistakes. It's efficient + tells you how training is changing your agent.
We show how to make it work for a strong, recent terminal agent policy (Qwen3.5-based), and show training away repeated misbehaviors is a viable path to capability improvement.
One of the reasons I like this paradigm so much is that it seems important for democratizing agentic post-training. Doing thousands of rollouts on long-horizon tasks in hundreds of environments is expensive, and it's hard to predict what the outcome of that training will be. In contrast, error-targeted learning lets you decide what errors you want to fix, and then you can watch those errors go away during training. It's very legible and accessible. 🧵👇
🤔 How can we efficiently train terminal agents to avoid misbehaviors?
😮 We try error-targeted learning on terminal agents and find that, with < 200 envs in TMax-15K & 35% of TMax's rollout, we can match TMax-4B full recipe.
🚨Introducing MiRL (Misbehavior-driven RL): MiRL first profiles the errors of a policy to find mistakes that recur across many environments, which we call generalizable misbehaviors. It then trains the policy with RL, starting rollouts at the steps where these misbehaviors occur and rewards a rollout only if it passes the tests and avoids the misbehavior.
Key takeaways
🔍 On Qwen3.5-4B, we find that 8 dominant misbehaviors account for 52.1% of wrong steps and appear in 75.4% of the environments.
🔥 MiRL outperforms TMax-4B, the released checkpoint trained with the full TMax recipe on Terminal-Bench Lite (51.8% vs. 47.4%) and matches it on Terminal-Bench 2.1 (21.3% vs. 20.6%), while using 35% of its rollouts and 189 selected training envs.
🧵👇 [1/9]
Zaid Khan retweeted
🚨 Terminal agents often fail because of a few recurring mistakes. Can we fix these misbehaviors directly, rather than wasting RL compute on entire trajectories?
💡 Sometimes, learning where and what to fix matters more than training on more data!
We propose ✨MiRL (Misbehavior-driven RL)✨ which identifies generalizable agent errors and trains directly from the steps where they occur!
We find that:
1️⃣ Just 8 misbehaviors account for 52.1% of wrong steps across terminal tasks.
2️⃣ Targeted RL from misbehavior prefixes improves generalization while avoiding redundant rollouts.
3️⃣ With only 189 training envs and 35% of the rollouts, MiRL outperforms TMax-4B on Terminal-Bench Lite and matches it on Terminal-Bench 2.1!
More details in the thread! 🧵👇
🤔 How can we efficiently train terminal agents to avoid misbehaviors?
😮 We try error-targeted learning on terminal agents and find that, with < 200 envs in TMax-15K & 35% of TMax's rollout, we can match TMax-4B full recipe.
🚨Introducing MiRL (Misbehavior-driven RL): MiRL first profiles the errors of a policy to find mistakes that recur across many environments, which we call generalizable misbehaviors. It then trains the policy with RL, starting rollouts at the steps where these misbehaviors occur and rewards a rollout only if it passes the tests and avoids the misbehavior.
Key takeaways
🔍 On Qwen3.5-4B, we find that 8 dominant misbehaviors account for 52.1% of wrong steps and appear in 75.4% of the environments.
🔥 MiRL outperforms TMax-4B, the released checkpoint trained with the full TMax recipe on Terminal-Bench Lite (51.8% vs. 47.4%) and matches it on Terminal-Bench 2.1 (21.3% vs. 20.6%), while using 35% of its rollouts and 189 selected training envs.
🧵👇 [1/9]
Zaid Khan retweeted
MiRL
Error-Targeted Reinforcement Learning for Correcting Terminal Agent Misbehaviors
paper: huggingface.co/papers/2610.3…
Zaid Khan retweeted
🤔 How can we efficiently train terminal agents to avoid misbehaviors?
😮 We try error-targeted learning on terminal agents and find that, with < 200 envs in TMax-15K & 35% of TMax's rollout, we can match TMax-4B full recipe.
🚨Introducing MiRL (Misbehavior-driven RL): MiRL first profiles the errors of a policy to find mistakes that recur across many environments, which we call generalizable misbehaviors. It then trains the policy with RL, starting rollouts at the steps where these misbehaviors occur and rewards a rollout only if it passes the tests and avoids the misbehavior.
Key takeaways
🔍 On Qwen3.5-4B, we find that 8 dominant misbehaviors account for 52.1% of wrong steps and appear in 75.4% of the environments.
🔥 MiRL outperforms TMax-4B, the released checkpoint trained with the full TMax recipe on Terminal-Bench Lite (51.8% vs. 47.4%) and matches it on Terminal-Bench 2.1 (21.3% vs. 20.6%), while using 35% of its rollouts and 189 selected training envs.
🧵👇 [1/9]
Zaid Khan retweeted
Unfortunately I couldn't make it to #EMNLP2026 but do check out the following work at the conference, covering:
- pragmatics as a self-taught reasoning task
- faithful CoT via multi-agent speaker-listener training
- grounding CUAs to history/vision
- early exiting for report generation
- tracing errors in deep research
- trade-offs between debugging and regenerating
Strongly agreed. Reminds me of this perennially useful post from @jsuarez on how games aren’t toy envs that should be dismissed as being unrealistic: puffer.ai/blog/game-rl/
Zaid Khan retweeted
👋 Looking forward to attending #EMNLP2026 where I will be giving an industry keynote on the hard, imminent AI challenges of trust, world discovery, and memory, plus an important, timely discussion on re-strengthening industry-academia collaborations!
We're also hiring at all levels: phd, postdoc, faculty! Feel free to ping if you want to meet up to discuss research, hiring, life 🙂
We're looking forward to the Industry Track at #EMNLP2026! Two keynotes:
• Mohit Bansal (UNC Chapel Hill): Agentic Challenges & Industry–academia Collaboration (Sun, 16:30)
• Verena Rieser (Google DeepMind): Constitutional Alignment (Mon, 9:30)
2026.emnlp.org/program/indus…
Zaid Khan retweeted
Theory-of-mind lets agents coordinate by modeling others’ thoughts/beliefs, but requires the agent to hold many hypotheses at once. ToM2Vec teaches agents to produce continuous ToM thought vectors which capture multiple hypotheses, improving across 5 multi-agent tasks and transferring to unseen partners, including frontier-model partners.
LLM agents often focus on their own plans without accounting for their partners’ next steps, leading to redundant or conflicting actions. ToM2Vec addresses this by improving agents’ theories-of-mind, training agents to predict a continuous, abstract representation of a partner’s reasoning. These continuous ToM tokens then guide the agent’s own reasoning and actions.
At test-time, the main agent generates a series of latent thought vectors representing its estimate of another agent’s reasoning/thoughts.
ToM2Vec improves coordination without any task rewards, across cooperative, competitive, and mixed-motive settings, including Overcooked, negotiation, Decrypto, Hanabi, and Collab Escape/Capture.
Crucially, agents trained with ToM2Vec generalize to unseen partner model families, including GPT-5.4 and Gemma-4.
🧵👇
Zaid Khan retweeted
We're looking forward to the Industry Track at #EMNLP2026! Two keynotes:
• Mohit Bansal (UNC Chapel Hill): Agentic Challenges & Industry–academia Collaboration (Sun, 16:30)
• Verena Rieser (Google DeepMind): Constitutional Alignment (Mon, 9:30)
2026.emnlp.org/program/indus…
it's so bad it's good. Tempted to keep on reading to see what juicy insanities are going to turn up next in the slop
i'm not sure what their endgame is tbh… i would really like to understand. It seems they think the "published at iclr" tag is some magic label that will give credibility even if the paper itself is very obviously low quality