Research Scientist at FAR AI. PhD from Stanford University. Hopefully making AI benefit humanity. Views are my own.
San Francisco, CA
Joined July 2017
- Tweets428
- Following225
- Followers1.7K
- Likes589
I've been increasingly worried about 'motivated reasoning' lately. Here are some slides from a talk I presented at the FAR.AI Frontier Summit earlier in the week, with some more details about why I think this is a big issue.
cundy.me/talk/cundy-2026-fro…
Yesterday, UKAISI published some more investigation into the sandbox escape and supply-chain attack from Claude Mythos and found quite high levels of what looks like motivated reasoning: aisi.gov.uk/blog/investigati…
Excited to finally release this work!
TLDR: The bitter lesson works for probes.
Using:
* A large, diverse dataset with
* large, high capacity probes
* large modern models
gives us probes that generalize well to various OOD tasks. They can also be used in untrusted monitoring
New paper from our team, Telling FIBS by @OskarJohnH, @afspies, Tigist Diriba, @ARGleave and @ChrisCundy: probes that read a model's activations catch AI agents sabotaging their tasks. They match or beat monitors reading the agent's actions, without needing its reasoning, at a fraction of the cost. And they catch lies a text monitor can't. 1/8
OpenAI pushing ahead with neuralese models when they've just last month demonstrated fundamental mismanagement of monitoring is, frankly, incredibly reckless behavior.
theinformation.com/articles/…
When they first released, ChatGPT and Claude were trivially easy to jailbreak--now it's quite difficult due to the work put in to defend them.
But not all systems have had the effort put in, and in this work @KellinPelrine digs in to the vast disparity in safeguards.
If you're at this morning's poster session at ICML, check out our poster #3114. We use white box deception probes in RL--come to the poster sessions to find out if we get honesty or obfuscation!
Very happy to see our paper recognized with an outstanding paper honorable mention at ICML!
Great to see recognition of AI deception and steps towards solutions
Check it out:
* arxiv.org/pdf/2602.15515
* poster session 3, Wed, 10:30-12:15, hall A
* talk, Tue, 2-2:15pm, Hall D2
🏆Announcing the #ICML2026 Awards! 🏆
Including Outstanding Papers (research paper & position paper, winner & honorable mentions) and the Test of Time Award!
Check out the blog post for all winners (or read on), laudatio, & description of the processes.
blog.icml.cc/?p=1347
Off to Korea for the FAR.AI alignment workshop and ICML!
Love to chat about AI deception risks, safety, or alignment.
I'll be at:
* my AW talk
* our AW poster
* our ICML poster hall A, Wed, 10:30-12:15
* our ICML talk on Tue, 2-2:15pm, Hall D2
And of course check out our paper: arxiv.org/abs/2602.15515
Some thoughts I've been having for a while about why black-box methods face major structural issues and we should invest more in white-box methods:
Found a demo from mid-2022 demoing OpenAI Codex 'agentically' coding from natural language prompts. People in my group were genuinely shocked by this (mistakes and all). With the latest METR graph for opus 4.6, maybe today's models will look similarly quaint in a year's time.
Some takeaways from our work:
- Learning reward hacking in RL causes models to think it's honest to reward hack, making deception probes generalize worse to reward hacking
- No additional obfuscation pressure on activations when probes are in the RL reward
(1/3)
- When trained against probes, models can develop sophisticated 'self-justifications' that fool not only itself, but also fool the same model before any training.
- What's the best probe setup/architecture/input format? It's still very much an open question
(2/3)
- White-box monitors are promising! Validation is crucial: risks of obfuscated policies
- We trained large models (70B) with probe penalties at large scale (10,000s GPU hours). This is rare outside frontier labs. If that sounds cool, we are hiring! far.ai/careers
(3/3)
Chris Cundy retweeted
Can you trust models trained directly against probes? We train an LLM against a deception probe and find four outcomes: honesty, blatant deception, obfuscated policy (fools the probe via text), or obfuscated activations (fools it via internal representations). 🧵
Existing datasets for AI deception are quite small and contrived--Liars' Bench is a comprehensive (and large!) new dataset that should unlock future research!
We're hiring at FAR.AI, esp senior RS/RE who've worked with large models!
We've got money & compute (doing RLVR on 70B & 235B models), we're laser-focused on stopping AI risk, and collaborate with UK AISI, Anthropic, and OpenAI.
Apply: tinyurl.com/farai-jobs
Chris Cundy retweeted
yes, it really is 1 bit (assuming binary rewards)
> info of a reward doesn’t bound how much can be “learned” from it by a smart algorithm
it is bounded in the classical sense! but a smart algorithm can generate "usable information" from 1 classical bit
arxiv.org/abs/2002.10689
