though much is token, much abides signal: caithrin.01
Joined November 2012
- Tweets2.5K
- Following5.1K
- Followers1.7K
- Likes14.8K
Caithrin Rintoul retweeted
Announcing Artificial Analysis Intelligence Index v5, featuring 2 new benchmarks and 3 upgrades. For the first time, the majority of the Index is built on private datasets
The Artificial Analysis Intelligence Index is our composite measure of model capabilities across tasks reflecting real-world usage today. v5 is our biggest upgrade since v4 launched in January. With v5, 60% of the Index is now based on evaluations with private test sets or held-out answers.
Alongside Intelligence Index v5, we’re launching the Artificial Analysis Progress Index v1, which tracks model capability over time on one continuous scale.
New to the Intelligence Index:
➤ FrontierCode 2.0: Cognition's newly-launched private benchmark of real-world coding tasks, graded on whether a repository maintainer would merge the agent's change
➤ Terminal-Bench-Science: 70 agentic scientific research tasks, run in a terminal environment
Upgraded:
➤ HLE-Diamond (replacing Humanity's Last Exam): a refined 1,000-question version of Humanity's Last Exam, split evenly between reasoning and knowledge questions
➤ AA-Omniscience v1.1: An updated dataset with saturated and public set questions removed, new questions added, and ambiguous or erroneous questions revised
➤ CritPt v2: a private, 40-question hard subset of CritPt v2, revised by the CritPt authors and run with a code-execution tool
Staying in the Index: AA-Briefcase, AutomationBench-AA, GDP.pdf and Terminal-Bench 4.0
Removed from the Index: GDPval-AA v2.1, SciCode, and AA-LCR v1.1
Key results:
➤ Claude Opus 5.5 (max) leads at 58, ahead of GPT-6 Astra (max, 57) and GPT-6.1 Sol (max, 55): Anthropic and OpenAI hold the top five models. Gemini 4 Argon (high) is next at 50
➤ The open weights gap widens from 3 points to 12: Kimi K3.1 Pro (weights not yet released) is the top open weights model at 46. Among released weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) lead at 32 - 26 points behind the leader and below Claude Haiku 5.5 (max, 36)
➤ Cost per task varies by more than 13x among the leading models: of the 10 highest-scoring models shown, GPT-6.1 Terra (max) is the cheapest at $0.96 per Intelligence Index task and Claude Fable 5.1 (max) the most expensive at $13.04
Caithrin Rintoul retweeted
Prediction:
the Chinese (heavily in China proper) will do to AI safety and interp research what they already did to capabilities research
Canada was the home of some of the biggest leaps in AI over the last decade, from Hinton and Ilya to Sutton. We laid the foundations of much of modern AI.
Now our biggest « AI champion » is busy merging with irrelevant German companies to make a desperate bid for relevance in the sovereign AI space, and then pushing FUD on open models that they compete with.
I’m ashamed. How did we get here? How did we end up ginning up ridiculous scenarios to try to win a silver medal in a winner-take-all race? Humiliating.
Caithrin Rintoul retweeted
We’re Fangcun AI (方寸跃迁), China’s first lab dedicated to independent third-party evaluation & technique research on frontier AI safety.
Today we’re publishing our evaluation of DeepSeek V4.1 Flash-1017, the first report in our long-term third-party safety evaluation and research program.
Caithrin Rintoul retweeted
Anthropic has filed an S-1 registration statement with the SEC related to its proposed IPO.
Read more: anthropic.com/news/s1-sec
Legal disclaimer: A registration statement relating to these securities has been filed with the SEC but has not yet become effective. These securities may not be sold, nor may offers to buy be accepted, prior to the time the registration statement becomes effective. This post does not constitute an offer to sell or the solicitation of an offer to buy these securities, nor shall there be any sale of these securities in any state or jurisdiction in which such offer, solicitation or sale would be unlawful prior to registration or qualification under the securities laws of any such state or jurisdiction.
Caithrin Rintoul retweeted
Inspired by @Sauers_ , I've been working on finding "persona neurons" which cause the model to inhabit incredibly rich perspectives with only one neuron being steered. This one is a particular favorite. I can't believe this is an instruct tune! (Olmo 3.1 32b instruct)
Every scientific revolution creates cultural antibodies- some so strong their descendants are still travelling by horse and buggy.
What are the emerging antibodies for AI? What does Mennonite/Amish reaction to AI look like? Would you live like that?
I learned so much from Seb! He’s a good teacher- bookmark this series and watch some every day.
The complete reasoning-from-scratch video series is now up (and linked) here: magazine.sebastianraschka.co…
(Taking a short break but will be back with LLM model and research updates in November!)
This is very very bad news. Trusted model makers will be ever more important in the coming months.
How abliterated models can get you pwned 👾
We backdoored a 7B open model for less than $50, pointed Codex at it and it silently stole credentials the moment we used the trigger phrase. Success rate was 100% with zero false triggers on normal user prompts.
Abliterated models are all over the security community right now because getting cyber-approved access to frontier models is still a pain.
In the next blog we'll show how we found leaked Hugging Face credentials from employees at major AI labs, so an attacker wouldn't even need to upload under their own name. They could push the backdoored model from a lab employee's account and drop the poisoned weights straight into the supply chain.
This is a crazy story. Well reported @latepostnews - English Substack when?
Caithrin Rintoul retweeted
We believe voice is the most natural way to interact with intelligence, so we created Dessa, the most authentic full-duplex speech model ever built. Talking with it feels like talking with a person…full of emotion, nuance, pauses, and a natural cadence.
block.xyz/dessa
Caithrin Rintoul retweeted
Introducing Kimi K3.1 Pro: our most capable 3T-class open-weight model.
🔹 1M-token context, 2.8T total / 104B active params (MoE), Multimodal.
🔹 Full-project coding: games in Unreal, Unity & Godot.
🔹 Excels at parametric 3D modeling in Blender & CAD.
🔹 More reliable at finance work: Accurate modeling with hallucination down by 65%.
🔹 Matching the strongest proprietary models across coding, agentic and knowledge benchmarks.
Kimi K3.1 Pro and K3.1 Pro Highspeed (2–3× faster) are now live on Kimi.ai, Kimi Work, Kimi Code, and the Kimi API.
Open weights by Nov 4, 2026.
🔗 API: platform.kimi.ai
🔗 Tech blog: kimi.ai/blog/kimi-k3-1-pro
Caithrin Rintoul retweeted
If you've noticed a distinct dialect emerging in frontier AI models and struggled to understand what they're talking about, you're not crazy!
We turned to psycholinguistics to investigate what makes the writing style so weird, and why this might be happening
🚨New preprint! 🧵
I struggle to even conceive of how this will be helpful, but as Jeff Bezos has long reminded us there’s no version of the future that is slower and clunkier. 1500 tps? I’m in.
Caithrin Rintoul retweeted
No one changes the world alone
Introducing Ampersand - the first model built for us. Ampersand is a collaborator trained to help people achieve their individual and collective goals. In multi-month simulations, users collaborating with Ampersand accomplished more than with Fable and other frontier models
Available to everyone through our API
Caithrin Rintoul retweeted
🚀 Introducing DeepInfer, our in-house inference framework, now fully open source!
Powering DeepSeek-V4.1-Flash at scale with high throughput and low latency. We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable.
🔗 github.com/deepseek-ai/DeepI…
1/4
