Pinned Tweet
Excited to share @NIST+CAISI’s initial public draft on how to run and report results of automated evals.
If you have opinions on evals, we’d love your feedback — help us improve the AI evals ecosystem!
Public comments accepted through March 31st via [email protected].
more in🧵
All feedback is welcome, but we are particularly interested to hear about:
- The usefulness and relative importance of included practices and principles.
- Any important practices that are within scope but missing from the draft.
- Any content that is incorrect, unclear, or otherwise problematic.
- When automated benchmark evaluations are more or less useful relative to other evaluation paradigms.
See this post for more information on giving feedback*: nist.gov/news-events/news/20…
*Note that all emails, including attachments and other supporting materials, may be subject to public disclosure.
1
2
225
One of CAISI’s core missions is to advance the state of best practices and standards for developing and working with advanced AI systems. If this speaks to you, come work with us. CAISI is hiring an AI Standards Architect among many other roles: nist.gov/caisi/careers-caisi
3
159
New paper! We show how to give an LLM the ability to accurately verbalize what changed about itself after a weight update is applied.
We see this as a proof of concept for a new, more scalable approach to interpretability.🧵
Paper authors: @avichal_goel, Yoon Kim, Nir Shavit, and me
arXiv: arxiv.org/abs/2510.05092
Demo notebook: colab.research.google.com/dr…
Code: github.com/Aviously/diff-int…
1
21
1,391
nitter.gq/aypan_17/status/186762…
Finally, a shoutout to LatentQA, the piece of prior work that had the most influence on our paper. The name and setup of our WeightDiffQA task is directly inspired by LatentQA, and the name of our method also takes after the name of their method “Latent Interpretation Tuning”.
LLMs have behaviors, beliefs, and reasoning hidden in their activations. What if we could decode them into natural language?
We introduce LatentQA: a new way to interact with the inner workings of AI systems. 🧵
13
1,454
Sharing a new paper by some collaborators and myself. We show that it is possible to covertly jailbreak models by leveraging finetuning access. We call this Covert Malicious Finetuning (CMFT).
If this vulnerability exists in future more powerful models, then safe deployment of those models will probably require banning untrusted users from finetuning models. This is actually a big restriction!
Restricting finetuning access doesn’t just mean restricting finetuning APIs, it also means model personalization services beyond basic retrieval augmented generation probably cannot be offered. Moreover, looking to the future, this also means models cannot be deployed in "continual learning" mode (which many cite as a key limitation of current large models).
IMO, understanding the full severity of CMFT and developing defenses against it are quite important lines of research.
Some more thoughts on this paper here: lesswrong.com/posts/33emJkmw…
Tony Wang retweeted
New paper, surprising result:
We finetune an LLM on just (x,y) pairs from an unknown function f. Remarkably, the LLM can:
a) Define f in code
b) Invert f
c) Compose f
—without in-context examples or chain-of-thought.
So reasoning occurs non-transparently in weights/activations!
Roughly 18 months ago, my collaborators and I discovered that supposedly superhuman Go AIs can be defeated by humans using a simple adversarial strategy.
In the time since, we've been testing a few baseline techniques for defending against this exploit. Unfortunately, in our new work we show that none of the defenses we tested work (see qt’d thread for details).
My personal takeaway is that we need to move beyond basic adversarial training if we want to build robust AI agents. By "basic adversarial training", I mean schemes where real examples of failures are collected and then agents are trained to downweight the probability of the actions that led to those failures.
I think the issue with basic adversarial training is twofold. Firstly, too much compute needs to be expended for each counterexample (our strongest defense spent 18x more compute on finding attacks compared to learning to defend). Secondly, existing learning algorithms aren't able to extract enough juice from counterexamples (i.e. they don’t have good sample efficiency / generalization).
There are a few ways that these problems could be addressed. I highlight some approaches below.
To end, I’d like to share a ballad about our work composed by Suno. Thank you for reading (and listening)!
Link to Suno song: suno.com/song/03963cf2-f719-…
Some additional links.
Paper: arxiv.org/abs/2406.12843
Website: goattack.far.ai
Blog: far.ai/post/2024-06-go-defen…









