Stoked to be attending FDS in SF on Oct 8! If you work on agents, evals, or benchmarking and are interested in joining, feel free to reach out--I may be able to share a registration code! frontierdatasummit.ai/

We wrote two tasks for Terminal-Bench 3.0, where the top model scores 43.5% vs 84% on Terminal-Bench 2.1. The failure mode that stood out to me is that agents reliably find a bug, patch the most reproducible symptom, and declare the incident over while the system is still broken. Deep dive: snorkel.ai/blog/why-frontier…

Frontier-Bench is live! Privileged to have authored one of the tasks: session-window-debug, a streaming debugging task where the bugs only surface when you trace state across multiple windows. Frontier agents still fail it more often than not Proud that @SnorkelAI contributed as a task author and data partner, with support through the Open Benchmarks Grants program. Congrats @ryan_marten and team on the launch 🚀

Heading to Bellevue for MLSys 2026 ✈️ I’ll be presenting our work on RLVR in low data and compute-constrained settings, where we studied how dataset composition, diversity, and difficulty impact reasoning performance across procedurally generated counting, graph, and spatial benchmarks. One interesting result: under fixed compute budgets, mixed-difficulty datasets often outperformed much larger easy-only datasets in terms of sample efficiency. Looking forward to the talks and meeting others working across LLM systems, training, evals, and agents. Feel free to reach out if you’ll be there! Paper: arxiv.org/abs/2604.18381

#NeurIPS2025 reflections After a week of posters, hallway chats, and workshop deep dives, here are a few themes that stood out... Training strategies and RL's evolving role: @YejinChoinka's keynote pushed past the "scale-is-all" narrative with a focus on RL--not just as a fine-tuning method, but as part of the pre-training story. The idea of eliciting reasoning behaviors in smaller models through RL is gaining traction. Benchmark design and meta-evaluation: From Terminal-Bench to agent self-evaluation setups, we're seeing a shift toward evaluating not just outputs but internal reasoning and feedback loops. There's an emerging science of how we measure capability, not just completion. LLMs in software environments: Papers like SWE-smith and SWE-rebench explored how LLMs perform in interactive dev environments. Robustness to tool changes and realistic regression testing stood out--especially relevant for agent-based use cases. From scale to specificity: A subtle but important theme...quality, provenance, and realism in data pipelines (both human-labeled and synthetic) are being treated as first-class problems. That feels like a healthy turn from quantity-first mindsets. Takeaway: NeurIPS isn't just about bigger models anymore--it's about sharper questions. #NeurIPS #LLMs #AIresearch

Excited to be at NeurIPS with the Snorkel research team—great conversations so far on eval, reasoning, and agents

Scaling AI quality without compromise? Snorkel says yes! In Part 4 of our rubric series, I break down Snorkel's multi-stage process for using rubrics in our data pipeline. In one case, we saw human/LLMaJ alignment increase from 37% → 94%. snorkel.ai/blog/scaling-trus…