This is more exciting than benchmark scores. Thinking traces from most open models are painful to read, and they waffle around a lot before figuring out the right steps.
Given how good frontier models are for bounded domain tasks; if they underform, it is more likely the benchmark tasks are flawed.
E.g. AutomationBench from Zapier has the best models scoring < 50pct. Turns out it's sloppy itself.
ananis.in/2026/Sep/08/benchm…
Picked one benchmark: Harvey's Legal Agent Benchmark
Reported 19.6% for Argon. Checked vals.ai/benchmarks/hlab which reports 25.42% for Muse Spark 1.2. Astra, and so on, underperform a lot indeed.
What is going on there?
Funny how frontier LLMs are terrible at assessing the cost of software work, while being great at writing code.
Unless you push back, any significant task is a "few days to weeks" of work. Idk how you can claim these models being self aware.
Much better idea to "pace" AI development - hold firms liable for harms done by automation they _run_. That'll teach them to not release unsafe models.
Talk of new regulation feels just so culpability can be pushed to users/no-one.
It feels like the same problem as coding agents - generation is automated, and human attention moves to reviewing the (massive) output.
Makes it harder to develop sympathy/intuition for the problem. At least software pays..
Great article. Though agents "colluding" by writing notes seems clearly related to how newer openai models are trained to run longer.
By leaving notes and searching over them, agent sessions can keep the context compact. Doesn't seem outright bad.
nitter.gq/Vtrivedy10/status/2095…
Amusing metric! They define Frontier as firms at top of token usage leaderboards.
Of course they end up using features that make you spend more tokens!
I don't quite get this - when you can deterministically lower from a compact repr (higher level languages) to machine code, what do you get by writing the executable directly? Maybe if you were writing task specific kernels.
Cool! Tons of conjectures might soon get settled one way or the other.
hello there the jacobian conjecture is false thanx to my close friend akhil for asking about it and my other close friend fable for working during the world cup final
((1+xy)^3 z + y^2 (1+xy) (4+3xy), y + 3 x (1+xy)^2 z + 3 x y^2 (4+3xy), 2 x - 3 x^2 y - x^3 z): \C^3\to \C^3, has jacobian determinant -2, and sends (0, 0, -1/4), (1, -3/2, 13/2), and (-1, 3/2, 13/2) to (-1/4, 0, 0)
Very neat ideas. Though the setup still feels uncomfortable - coding agents implementing a plan and I only get to review at the end.
Very cool work - definite proof that at least for software delivery, LLM agents can keep grinding for looong durations, if they are set up to succeed.
The paper does address it, but the rationale feels weak.
Task-difficulty is the confounder here - both sed and ruff are harder to implement than nonogrid or tssql. And likely why they fail, despite being memorized.
Also, interesting contrast to ProgramBench who reported much lower success rates.
Improvements seem to come from much higher token budgets, and more friendly setup for each task - a visible test set so the scope of implementation is clear.
programbench.com/
Useful discussion.
A simpler maxim could be - if code (no matter where on the typing -> vibe-code axis) is helping hone your mental model of the problem space.
That alone determines if it is net gain or future tech debt.
