Brad Johnson 百面包 retweeted
GPT-6.1 Terra is OpenAI’s #1 model on APEX-Agents.
68.3% Pass@1 (#6)
79.3% mean score (#6)
Compared to last generation GPT-5.6 Terra, the new model gains 10.1 points on Pass@1.
Management consulting: 80.9% Pass@1 (#2)
Investment banking: 66.2% Pass@1 (#7)
Corporate law: 57.7% Pass@1 (#39)
The model is strongest on management consulting tasks, ranking second overall, behind Gemini 4 Argon (88.4%).
Against GPT-5.6 Terra, it gains 24.7 points in consulting and 10.9 in banking. Law goes the other way, down 5.4 points from GPT-5.6 Terra.
GPT-6.1 Terra still averages 80.6% of rubric criteria on law tasks, but fully completes only 57.7% of them. It gets most of the way and misses the last requirements.
Across 4 runs per task, GPT-6.1 Terra passed 134 of 240 tasks every time and 187 at least once. 22 tasks it never scored on, 14 of them in investment banking.
GPT-6.1 Terra’s strength comes from reading more than its siblings: 735k input tokens per attempt, vs 497k for GPT-6 Astra and 489k for GPT-6.1 Sol. Despite reading more, it costs less.
GPT-6.1 Terra: $0.19 per task ($1/$5 per 1M tokens)
GPT-6.1 Sol: $0.26
GPT-6 Astra: $1.30
It’s also a fast model. The median attempt finishes 1.2x faster than GPT-6 Astra and on par with GPT-6.1 Sol.
Congrats @OpenAI on the launch.
See full leaderboard: mercor.com/apex/apex-agents-…
Brad Johnson 百面包 retweeted
We’ve automated a very low percentage of enterprise tasks with AI.
That’s @jefrankle’s estimate, and he thinks the models are already smart enough to do significantly more.
Jonathan is the Chief AI Scientist at Databricks, co-founder of MosaicML, and my latest guest on The Frontier.
We talked about why evals are the biggest bottleneck to enterprise AI adoption. The intelligence is increasingly there, but enterprises need to figure out how to define what they want agents to do and measure whether they’re doing it well.
Full episode below 👇
Brad Johnson 百面包 retweeted
Mercury 3, @_inception_ai's diffusion LLM, debuts on the APEX leaderboards.
50.2% Pass@1 on APEX-Agents (#30 of 59)
43.1% Pass@1 on APEX-SWE (#31 of 62)
Inception positions Mercury 3 against speed-optimized frontier models.
On APEX-Agents it lands between GPT-6 Luna and Haiku 5.5, and on coding it trails Haiku 5.5 by 12 points.
Across APEX-Agents domains (mean score):
Corporate law: 78.4%
Management consulting: 57.5%
Investment banking: 51.4%
It is cost pareto optimal on APEX-Agents. An average APEX-Agents task costs about 6 cents at Inception's list price, and about 3 cents at the launch discount, helped by heavy prompt caching.
Congratulations to the Inception team on showing a diffusion model can hold its own on long-horizon professional work.
See full leaderboard: mercor.com/apex/apex-agents-…
Brad Johnson 百面包 retweeted
Glad to see more work building on our previous mapping effort (arxiv.org/abs/2603.01203) to reveal the distributional gap between AI benchmarks and human work.
Surprised to see this gap remains 10 months after our study was released. Let’s build more useful benchmarks!
Mercor Research mapped AI benchmarks to actual US jobs.
Only 208 of 923 occupations in the US economy are represented by existing AI benchmarks. 90% of hours worked in the US aren't covered well by a relevant AI benchmark.
Today, we’re releasing the 𝗕𝗲𝗻𝗰𝗵𝗺𝗮𝗿𝗸 𝗠𝗮𝗽 𝗼𝗳 𝘁𝗵𝗲 𝗨𝗦 𝗘𝗰𝗼𝗻𝗼𝗺𝘆, an interactive breakdown of which economic tasks are covered by AI benchmarks.
Benchmarks are how we measure AI’s progress and anticipate its impact on real work. They’re crucial to understand which jobs AI can automate, where the biggest gaps are, and how those capabilities are changing over time.
The Benchmark Map is our first step toward building a resource that makes it easier to answer: 𝘋𝘰𝘦𝘴 𝘵𝘩𝘪𝘴 𝘈𝘐 𝘮𝘰𝘥𝘦𝘭 𝘸𝘰𝘳𝘬 𝘧𝘰𝘳 𝘮𝘺 𝘰𝘤𝘤𝘶𝘱𝘢𝘵𝘪𝘰𝘯?
Explore it here: mercor.com/research/labor-in…
This was a great interview. YouTube video below if you missed it.
Last week, Mercor Research economist @aden_barton joined @MTSlive to discuss our latest study comparing frontier AI models with licensed accountants on real-world accounting tasks.
18 months ago, the best models performed below the average accountant. Today, models are faster and more accurate than every accountant in our study.
Aden discussed what this progress means for the accounting profession, and why performing well on individual tasks is different from doing an entire job.
Watch the full conversation. youtube.com/watch?v=9JcfR1Lb…
Brad Johnson 百面包 retweeted
Looking forward to joining MTS later today to discuss the Mercor Econ team's recent work!
TESLA ROADSTER | MECH INTERP | AI POLICY SPECTRUM xitter.cf/i/broadcasts/1DxLdZnLN…
Excited to see @aden_barton on MTS Live today. 📺
TESLA ROADSTER | MECH INTERP | AI POLICY SPECTRUM xitter.cf/i/broadcasts/1DxLdZnLN…
Brad Johnson 百面包 retweeted
we need better benchmarks for the work that people actually do!!
90% of Americans’ working hours are spent on tasks not covered well by AI benchmarks.
That’s our best guess from a new project mapping frontier benchmarks onto the US economy.
Preparing for a post-AGI economy requires predicting the future. Current benchmarks makes it very difficult even to measure the present.
Brad Johnson 百面包 retweeted
"90% of American working hours are spent on tasks with little or no coverage from existing benchmarks."
Building great benchmarks is extremely high leverage to making AI more economically valuable. Can't manage what you don't measure.
Mercor Research mapped AI benchmarks to actual US jobs.
Only 208 of 923 occupations in the US economy are represented by existing AI benchmarks. 90% of hours worked in the US aren't covered well by a relevant AI benchmark.
Today, we’re releasing the 𝗕𝗲𝗻𝗰𝗵𝗺𝗮𝗿𝗸 𝗠𝗮𝗽 𝗼𝗳 𝘁𝗵𝗲 𝗨𝗦 𝗘𝗰𝗼𝗻𝗼𝗺𝘆, an interactive breakdown of which economic tasks are covered by AI benchmarks.
Benchmarks are how we measure AI’s progress and anticipate its impact on real work. They’re crucial to understand which jobs AI can automate, where the biggest gaps are, and how those capabilities are changing over time.
The Benchmark Map is our first step toward building a resource that makes it easier to answer: 𝘋𝘰𝘦𝘴 𝘵𝘩𝘪𝘴 𝘈𝘐 𝘮𝘰𝘥𝘦𝘭 𝘸𝘰𝘳𝘬 𝘧𝘰𝘳 𝘮𝘺 𝘰𝘤𝘤𝘶𝘱𝘢𝘵𝘪𝘰𝘯?
Explore it here: mercor.com/research/labor-in…
Brad Johnson 百面包 retweeted
There is so much to say about this, and I will, but WOW, congratulations to Nick and Goodfire. This may be one of the most important single days in deep learning history, and perhaps much more than that.
No matter what, this is a victory for the dogged pursuit of understanding.
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Brad Johnson 百面包 retweeted
when nick told me that he had some interp breakthroughs, he was uh, underselling it. i think this may be remembered as a major moment in scientific history. news so good even eliezer will have to lower his pdoom. we can start making alignment into a 'normal science'
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Brad Johnson 百面包 retweeted
@OppInsights should check if the IQ wage premium has changed post-ChatGPT using their linked SAT/IRS data. @LichtingerGuy & @SeyedMH98 suggests impact should be U-shaped, but admin data would be so good here.
Brad Johnson 百面包 retweeted
Benchmarks prioritize high wage-bill, AI-exposed jobs & tasks. This makes sense.
But the rest of the economy becomes more relevant as models diffuse. We want to know what AI means for paralegals and medical assistants too, not just doctors and lawyers.
Brad Johnson 百面包 retweeted
90% of Americans’ working hours are spent on tasks not covered well by AI benchmarks.
That’s our best guess from a new project mapping frontier benchmarks onto the US economy.
Preparing for a post-AGI economy requires predicting the future. Current benchmarks makes it very difficult even to measure the present.
Brad Johnson 百面包 retweeted
The results show how a relatively small amount of expert-built training data can improve an open model’s performance on professional work.
Open models plus realistic benchmarks and targeted training data create a viable path to domain-specific agents.
Congrats to @NVIDIAAI on the Nemotron 3.5 Super VL launch. And to @Harvey, @Cognition, and @tryRamp for partnering with us on this work.
Full blog here: mercor.com/blog/mercor-nvidi…
Brad Johnson 百面包 retweeted
On the APEX-Accounting benchmark, mean rubric score rose from 34.6% to 41.9% (+7.3 points). Tasks scoring zero fell from 19% to 13%.
The post-trained model began by reviewing all available files before starting work (+38 points on planning behavior).
It also opened significantly more source records per task, from 8.5 to 12.
Brad Johnson 百面包 retweeted
On the APEX-SWE benchmark, Pass@1 rose from 26.6% to 34.7% (+8.2 points).
The model's consistency improved significantly as the tasks solved on all four attempts doubled from 9.9% to 19.9%.
Post-training taught the model to read API docs before working (+18.1 points). It checked the live system after making changes, increasing system calls per task from 0.98 to 2.01.
Brad Johnson 百面包 retweeted
On the APEX-Agents benchmark legal domain, the post-trained Nemotron 3.5 Super VL's mean rubric score rose from 45.1% to 51.8% (+6.7 points).
After post-training, the model learned to read case files before answering and its document reviews per attempt doubled from 5.6 to 11.3.
For one task, the model had to find minimum payment terms in a settlement agreement. The base model almost never opened the actual agreement. After post-training, the model consistently found the agreement and applied the terms correctly.
Brad Johnson 百面包 retweeted
Congratulations to @NVIDIAAI for launching Nemotron 3.5 Super VL.
We worked with our partners @tryRamp, @Harvey, and @Cognition along with the NVIDIA team to post-train and improve the model’s performance using targeted datasets from our APEX benchmarks.
We tested the post-trained Nemotron models on legal, software engineering, and accounting tasks it had never seen.
The post-trained model improved across all three professional domains.
Read the full results: mercor.com/blog/mercor-nvidi…