Mechanistic interpretability and Philosophy of AI. PhD Student @epfl. Scholar @MATSProgram. Founder @philinterp.
Lausanne, Suisse
Joined February 2023
- Tweets222
- Following233
- Followers563
- Likes597
Pinned Tweet
New paper: World Modeling in Transformers
with @matthieu_queloz @andrenfreitas @MATSprogram
Can AI models recover world models just from data? We find a faithful internal map and more inside of taxiGPT! We trace failures to how that the map is stored in superposition. 🧵(1/11)
Pierre Beckmann retweeted
check out this intuitive video on why interpretable directions in llms look non-gaussian, how ica finds them in minutes per layer, and the evaluation bug that made ica look weak for 3 years
We revisited Independent Component Analysis (ICA), a decades-old statistical method, and found that it works far better for interpreting LLM activations than we expected!
This old baseline deserves a second look.
Across GPT-2 Small, Gemma 2 2B, and Qwen 3.5 9B Base, we find ICA is actually competitive with the public SAEs across interpretability, sparse reconstruction, concept prediction, and targeted probe perturbation, while fitting in minutes per layer and using 7–43× less parameter memory.
We're releasing ICA Lens, which lets you quickly discover, annotate, and evaluate interpretable directions in LLM residual-stream activations. No need to train another dictionary!
More broadly, our results suggest that non-Gaussianity is a useful inductive bias for efficiently finding interpretable directions.
Try it: pip install icalens pypi.org/project/icalens/
Paper: arxiv.org/abs/2606.11722v2
1/8
thanks for the shout-out!
New research shows transformers do build internal world models, but localization failures stem from feature superposition. Mechanistic analysis reveals how representation and routing inte…
arxiv.org/abs/2609.21748
#MachineLearning #AI #LLM #DeepLearning #AgenticAI
ALT World Modeling in Transformers
Pierre Beckmann retweeted
"three reasons why AI isn't conscious." really?
"consciousness is not intelligence": not a reason why AI isn't conscious, just rebuts a bad argument for AI consciousness (AI: intelligent therefore conscious).
"consciousness is substrate-dependent"/"noncomputational": would be a reason if true, but truth needs to be established.
more accurate title: "three reasons why AI might not be conscious".
xitter.cf/Nature/status/21121504…
LLMs can mimic thought and feeling with uncanny skill, but intelligence alone is not enough for subjective experience
go.nature.com/3VSXWQ4
Pierre Beckmann retweeted
What is an LLM?
Is it an incomprehensible black box, a next-word-predictor, a silicon genius?
To answer this, we don't have to only guess, we can open it up and look inside!
It was an honor to chat with @lennysan about what we learned doing this, some takeaways in 🧵
"To make them is not to understand them."
Emmanuel Ameisen (@mlpowered) has spent the last 3 years doing interpretability research at @AnthropicAI, looking inside LLMs like Claude to figure out what they're actually doing. As he puts it, models are "grown," not built.
We discuss:
🔸 His discovery that models plan many tokens ahead
🔸 Why anthropomorphizing AI can actually be the right approach
🔸 What the "J-space" reveals about how models think
🔸 How interpretability connects to AI alignment and safety
Listen now 👇
youtu.be/JHo-aiePjoM
Still, I don’t think this is a paradigm shift for mech interp. We’ll still need diff-means, probes and SAEs to find features (in particular for the res stream). But I’m excited to see more attribution graphs built directly on MLP neurons! 17/
Overall, Nick gives evidence that there are fewer geometry tricks than we thought, at least for MLP neurons. More widely, this points toward superposition being driven more by structure than by geometry tricks. If true, latent spaces are indeed cleaner than prev. expected. 16/
Things might be different in the residual stream, though. Every att head and MLP reads from and writes to it, so it may need to carry a lot of information. So space-saving superposition and geometry tricks could be more common there. 15/
In transformers, this means that MLP neurons might be more interpretable than we thought, and that we can build attribution graphs directly on them. For example, Nick traces a medical-diagnosis circuit in Qwen3-32B using only MLP neurons. 14/
Nick’s analyses are mostly at the neuron level, e.g. right after an activation function. Activation functions act on each neuron separately (e.g. ReLU cuts off negative values), so the network has a reason to align features with neurons. This is called a privileged basis. 13/
He also shows how seemingly unrelated features can get grouped by training history. So with these two cases, he argues that there might be much less space-driven and geometry-trick superposition than we thought. 12/
One big motivation for believing in geometry-tricks was InceptionV1 neurons that seemed to mix unrelated things, like cat faces and car fronts. Nick shows that such a neuron actually detects a shared V-pattern 11/
The other is to use geometry tricks: putting unrelated features together because they rarely show up at the same time, e.g. “quantum physics” on the opposite side of “cat.” This kind of packing disrupts elegant structure-based organisation. 10/
Networks can deal with the limited-space pressure in two ways. One is to go a bit further in exploiting relations and structure, e.g. by merging features or putting them at even closer angles. 9/
The second pressure is limited space. Neural networks often have a very constraining width, so features get pushed closer together than structure alone would require. I’ll call this space-saving superposition. 8/
Is this really superposition, or just composition? If coffee and espresso were combinations of nearly orthogonal base features, we could call it composition. But it’s not clear that such a base exists, so I’ll go with the less demanding notion of structure-driven superposition 7/
Now composition. Coffee can be seen as [hot + brown + liquid + big cup], and espresso as [hot + brown + liquid + small cup]. Since they share many constituents, their directions partly overlap. 6/
