Evals, RL environments & AI data | Building benchmarks for binary reasoning.
Austin, TX
Joined December 2009
- Tweets51.1K
- Following1.4K
- Followers1.4K
- Likes6.8K
Marq retweeted
Applications are now open for the Anthropic Residency, a paid, fixed-term research program for academics, technical founders, and staff-level engineers who want to make AI safe and beneficial for everyone. No ML background is required.
Apply by November 22: alignment.anthropic.com/2026…
Marq retweeted
Anthropic says its internal estimates and expectations for future models suggest it may soon reach its threshold for dramatically accelerated AI progress driven by AI itself.
Marq retweeted
Our budget philosophy-themed couple's Halloween costume: René Descartes and his Evil Demon 🎃👻
Marq retweeted
5 weeks in an Opus 5.5 is still the goat.
It has the magic and feel of my beloved Opus 4.6, the intelligence of Fable 5.5, and the token efficiency of GPT-6.1 Sol.
5 weeks after its release, Opus 5.5 remains by far the best model overall.
Therefore, I expect a major release from OpenAI next week, because it's clear that Anthropic will also be releasing Fable 5.5 next week, and most likely a new Opus as well.
Marq retweeted
you either die a
Sent from my iPhone
or you live long enough to see yourself
Authored by Claude Code
Marq retweeted
Marvel Studios has confirmed that Patrick Stewart's final on-screen performance will be as Professor X in ‘AVENGERS: DOOMSDAY’.
Marq retweeted
"real envs > synthetic envs" feels like mostly 3 things:
- a complaint about sloppy synthetic data pipelines
- a way to make envs pricier without making them better
- synthetic task authors guessing the task distribution because they've never worked with real production traffic at scale
real envs ship very hackable tasks too. whereas a carefully built synthetic env that's realistic enough has all the same benefits as real envs, + you get to aim at one failure mode on purpose and isolate it from the random noise
Marq retweeted
This is crazier than any spy movie:
According to The New York Times, a Chinese businesswoman with close ties to Nvidia CEO Jensen Huang was passing intelligence to Beijing, and Trump was briefed but took no action. What she shared remains unclear, and U.S. officials initially believed Huang was unaware of her alleged intelligence role.
Breaking News: A Chinese businesswoman who had a close relationship with Nvidia’s CEO, Jensen Huang, was found to be passing intelligence to Beijing. nyti.ms/3Ukob1t
Marq retweeted
how do you post-train an agent?
we laid it out as a 5-step curriculum, from SFT to RL through a real harness, with livestreams, code and docs for each step
huggingface.co/blog/huggingf…
Marq retweeted
Gemini 4 Carbon is our next frontier model.
It’s state-of-the-art across cybersecurity and long-horizon capabilities.
Marq retweeted
A few thoughts on Anthropic’s approach:
1. Drug discovery will be one of the most significant revenue streams in the future; the goal here is to set the right course early on. Most diseases will become treatable, a market encompassing 7 billion people.
2. The timing is clearly intentional, announced just before the IPO to signal to investors that the business will continue to thrive in the future.
3. This will likely make it clear, even to those outside the "tech nerd" sphere, why AI is so important. It goes beyond just writing code; it solves problems for humanity.
Marq retweeted
Axiom says its AI agents formalized a 15,000-page mathematical proof in just 24 days, largest proofs in mathematics.
Built by over 100 mathematicians across decades, the proof classifies the fundamental building blocks of finite symmetry.
With human guidance, the agents translated it into Lean, making every logical step computer-checkable and repairing errors along the way.
AI tackling an entire interconnected body of mathematics, with results others can verify. A major milestone.
We are witnessing history unfold. Again. Every day now.
Marq retweeted
The largest sporadic group, the Monster, has ~8 × 10⁵³ elements.
Related to it is the "Monstrous Moonshine" conjecture, Richard Borcherds’ 1998 Fields Medal work.
Yet for long, there is cloud on the correctness of the proof, as no one person has read the whole - it’s too big.
Marq retweeted
I've seen a lot of pushback to this, with the following complaints:
1. "The benchmark owners already know about the flaws, and are working on it."
2. "Why are they spending time critiquing and not contributing to the benchmark" OR "critiquing known issues are not helpful".
3. "The flawed verdict is too binary"
I find this pushback to be misguided. These benchmarks are used to move millions of dollars of token spend, and many people blindly trust how well these benchmarks generalize to their own products, and don't look any further than the headline numbers.
For example, whenever a new model drops, you'll find thousands of youtube videos analyzing the pareto frontier using one of these benchmarks as the primary quality metric without questioning the benchmark.
I think we would all be better served by inviting skepticism and encouraging people to develop an independent opinion of how well these benchmarks represent the tasks they care about. Concretely:
- For the failing tasks within the benchmark, are they: realistic, have a proper harness, graded properly, sufficiently disambiguated user intent?
- Are models and API calls configured properly (ex: inconsistent context windows, reasoning levels).
- Are the models equipped with a modern harness and minimal retrieval that is sufficiently realistic?
- Does the score even make sense ex: is it just a LLM judge committee that produces a score on a 0-100 scale that isn't aligned with ground truth?
It's often the case you'll see many of these issues while looking at traces of a failing benchmark tasks, and it doesn't take long! Often the **very first trace you look at** has a serious issue.
I also don't think there is an issue with verified vs. flawed verdict. It's good to have opinions. If you look through enough of what's behind their opinions and you find that you usually agree with their verdict, then they have earned your trust, the same way that a benchmark can lose your trust.
I think this eval of benchmarks is a red pill that motivates investing in your own private evals.
Marq retweeted
daily reading guys
arxiv.org/abs/2610.31867
Marq retweeted
Repo2RLEnv（Hugging Face 开源）和AgentEnv Framework（Scale AI 开源）从不同角度来实现RL环境。
Repo2RLEnv是输入一个 GitHub 仓库，它自动从已合并的 PR、commit 历史中提取任务，生成完整的 Harbor 格式环境（包括沙盒、指令、私有验证器）。验证器直接跑仓库自己的测试用例打分。
AgentEnv Framework（Scale AI 开源）：提供标准化接口定义环境（MCP 工具、沙盒状态）、任务（步骤 DAG）和评分逻辑。可以用插件扩展，把真实应用或模拟工作场景打包成可重复训练的环境。
一些垂直领域可以用AgentEnv Framework这类框架定制RL环境，也会有领域（或是公司）的Repo2RLEnv，根据领域一些特性，工程师做出类似Repo2RLEnv类工具，自动编译出领域的RL环境。这都对传统的数据标注类业务来讲大大的提升了效率，所以硅谷最近出了很多非常小的RL环境团队，大部分都是非常小，但收入很好的。他们就在往这种方向上靠拢。
github.com/scaleapi/agentenv…
github.com/huggingface/Repo2…
Marq retweeted
💥 You don’t need a 128 GB Mac for a 105 GB Qwen model anymore.
Slotstream streams Qwen3.8-Flash-Next from your SSD — runs on 16–64 GB Apple Silicon Macs. One Swift binary, no Python, offline after download.
Their 48 GB M5 Pro (33 GB budget): 20.91–23.69 tok/s across short / long / coding prompts.
Caveat: Slotstream’s own Oct 5 benches — your Mac/SSD will differ.
via GitHub 👇
github.com/carloslfu/slotstr…
