Foundry AI

We tested DeepSeek V4.1-Flash across 5 Terminal-bench style RL tasks from our suite (Kafka dispatch, k3s autoscalers, Flask features & multi-currency settlement). It solved 1 task completely, and on analysing trajectory we saw this behavior:

A lot of discussions lately around RL datasets, synthetic methods, QA passes, and vendor collaboration. We’ve seen a strong demand on the OTS data side to push coding model capability. As per requests from labs, we decided on a timeline to share samples this week and develop more RL tasks tailored to their requirements. A few core details on how we approach task creation and validation: 1. Zero synthetic generation: Our task generation is not synthetic, but sourced directly from real software engineering ideas, private acquired codebases, and actual production systems. 2. Robust QA before burning compute budget: Before running our compute budget on any task, our RL pipeline runs through strict checks. (ref. image below). Our Forge dashboard have the runner to find the reward hacking gap for each tasks as well and do the model runs. 3. Domain experts, not generalists: This week we’re focused on creating 10–15 more tasks with 3 experts working full-time (and have capability to scale production as required). We specifically onboard experts from top product-based companies with 5–8+ years of experience to design these tasks. We’re sharing the initial batch of samples this week and continuing to build as requested. If you are a lab looking to improve coding model capability with high-quality RL datasets focused on long horizon reasoning, my DMs are open!

We have analyzed more than 5000 trajectory across different coding models developed by the frontier labs, and this is the common behavioral patterns we have seen, and which is confirmed by the researchers of different labs as well. Give it a long horizon task, and you would see all these patterns when the trajectory is analyzed. Our process is not just task rollout, but we let expert use the coding agent for different tasks in our Forge harness and submit RLHF feedback for the coding models. We capture the trajectory for each executions and pair up with the feedback to analyze the model behavior.

Benchmarks like SWE-Bench and SWE-Bench Pro are focused on evaluating the capability of coding agents for code generation-related tasks. A few evaluate other capabilities of the model related to code review. To support this, we at @foundryyai have published SWE-PRBench. A dataset of 350 pull requests where the ground truth is the review by the human engineers. The benchmark measuring whether frontier LLMs catch similar issues that reviewers catch in production code. The results of top models on SWE-PRBench: - Claude Sonnet 4.6 by Anthropic -  29.7% detection, 22.7% hallucination - DeepSeek V3 by DeepSeek AI -  31.2% detection, 31.5% hallucination - Mistral Large 3 by Mistral AI -  30.5% detection, 35.3% hallucination - GPT-4o by OpenAI -  22.0% detection, 19.3% hallucination The best model still missed 7 in 10 issues that human reviewers caught. And the finding that surprised us most: adding more context made every model worse. All 8 models degraded monotonically as context expanded. When we added execution context and surrounding file content, Type2_Contextual issue detection collapsed by 50–55% across top models. The gap between code generation benchmarks and code review benchmarks tells us something important: progress on SWE-Bench does not predict progress on code review. These are different capabilities, and the field has been measuring only one of them. This is the generalisation gap where the focus is less. Full results below!

Introducing FoundryAI: the coding agent evaluation lab. Coding agents score well on benchmarks… but still fail in production workflows. We built FoundryAI to solve this: ➤ Evaluate models on pull request workflows, multi-file changes, and long-horizon tasks ➤ Identify where models miss real issues across code review, reasoning, and context handling ➤ Generate targeted training data to fix those exact failure modes ...and more FoundryAI focuses on the hardest part of building coding agents: evaluation and training data. Our work is grounded in software engineering workflows, including: ➤ UI generation ➤ Long-horizon tasks ➤ Custom benchmark design ➤ Architecture reasoning ➤ Multi-system workflows ➤ Code review quality Working with leading AI labs building coding agents. Want to see where your model breaks? Check the next thread!