Member of Technical Staff @GoodfireAI
San Francisco, CA
Joined March 2022
- Tweets331
- Following654
- Followers760
- Likes1.3K
Tal Haklay retweeted
my favourite result: we can use mechanistic shaping to remove spurious correlations in medical imaging that no-one on the team was able to identify. how? tell claude what we wanted to train the model to do, show it interpretability results, and have it decide what supervision to provide. scalable oversight in action!
We cut through the mystery around interpretability in training.
A single, mathematically rigorous framework transforms model internals into training feedback and works across domains, problems, and modalities.
We call this *mechanistic shaping*, and we’ve used it to mitigate reward hacking, remove shortcuts in chest x-rays, and reduce sycophancy and harmfulness. We believe this is a big step forward for technical alignment.
But it carries a known risk: training against a model’s internals can make it harder to monitor. We explore both the technique and this concern below. 🧵
Tal Haklay retweeted
This was one of the coolest and most fun projects I have ever worked on: mechanistic shaping. We introduce a new framework for shaping what concepts a model learns to rely on during training.
My favorite applications from the paper: medical imaging. 🩻
We cut through the mystery around interpretability in training.
A single, mathematically rigorous framework transforms model internals into training feedback and works across domains, problems, and modalities.
We call this *mechanistic shaping*, and we’ve used it to mitigate reward hacking, remove shortcuts in chest x-rays, and reduce sycophancy and harmfulness. We believe this is a big step forward for technical alignment.
But it carries a known risk: training against a model’s internals can make it harder to monitor. We explore both the technique and this concern below. 🧵
Tal Haklay retweeted
This work is the departure point for a science of model training: we must have models that are "right for the right reasons"
We cut through the mystery around interpretability in training.
A single, mathematically rigorous framework transforms model internals into training feedback and works across domains, problems, and modalities.
We call this *mechanistic shaping*, and we’ve used it to mitigate reward hacking, remove shortcuts in chest x-rays, and reduce sycophancy and harmfulness. We believe this is a big step forward for technical alignment.
But it carries a known risk: training against a model’s internals can make it harder to monitor. We explore both the technique and this concern below. 🧵
Tal Haklay retweeted
being able to shape models during training is a big step forward for alignment, moving from trial and error to designing models with intention.
incredibly proud of the team - this has been cooking for a long time, and we now see a path to truly being able to predict and shape model behavior from the inside out
We cut through the mystery around interpretability in training.
A single, mathematically rigorous framework transforms model internals into training feedback and works across domains, problems, and modalities.
We call this *mechanistic shaping*, and we’ve used it to mitigate reward hacking, remove shortcuts in chest x-rays, and reduce sycophancy and harmfulness. We believe this is a big step forward for technical alignment.
But it carries a known risk: training against a model’s internals can make it harder to monitor. We explore both the technique and this concern below. 🧵
Tal Haklay retweeted
I'm super excited about application of mechanistic shaping to medical imaging! Medical imaging models are known to be fixated on confounds (labels written, device bits etc.) confound removal using diff of means will be super useful. Congrats to the team!
We cut through the mystery around interpretability in training.
A single, mathematically rigorous framework transforms model internals into training feedback and works across domains, problems, and modalities.
We call this *mechanistic shaping*, and we’ve used it to mitigate reward hacking, remove shortcuts in chest x-rays, and reduce sycophancy and harmfulness. We believe this is a big step forward for technical alignment.
But it carries a known risk: training against a model’s internals can make it harder to monitor. We explore both the technique and this concern below. 🧵
Tal Haklay retweeted
We cut through the mystery around interpretability in training.
A single, mathematically rigorous framework transforms model internals into training feedback and works across domains, problems, and modalities.
We call this *mechanistic shaping*, and we’ve used it to mitigate reward hacking, remove shortcuts in chest x-rays, and reduce sycophancy and harmfulness. We believe this is a big step forward for technical alignment.
But it carries a known risk: training against a model’s internals can make it harder to monitor. We explore both the technique and this concern below. 🧵
Tal Haklay retweeted
A science of post-training can be built on interpretability.
Models know when they reward hack, and we see it, yet training ignores it. We propose Mechanistic Shaping: using the model’s own concepts to write training rules.
In vision, DPO & RL, and its effects on monitoring 🧵
Tal Haklay retweeted
Two more weeks to apply for the @azrielifdn post-doc fellowship, an excellent program, funding, and cohort to interact with. Let me know if you're interested.
azrielifoundation.org/fellow…
Tal Haklay retweeted
Long post about my thoughts on this:
I'm so excited that this is finally out! Optimizing this until scaling it out is actually feasible and building the systems to perform the scaling has been my job for the last couple of months. This work is some of the most deserving of my time and energy that I can imagine.
There's been a lot of Discourse in the interpretability community lately about the neuronal and featurizer paradigms (not enough discussion of the causal abstraction paradigm imo..), the LRH and superposition, etc but no matter what paradigm we find ourselves in (and remember they can co-exist and describe different phenomena!) it's paramount that we accelerate the science of how models handle a wide array of tasks, how compositionality is constructed through intersection and containment of internal machinery (whether neurons, circuits, subspaces, or whatever else). Our current approaches have been too isolated to specific task and model combinations, but now that we have proven methodologies we can begin the process of "normal science" by applying them to population studies.
Once we've proven this out, it will be time to scale the effort beyond even the model-specific setting. This is where things get really interesting! Given how many different axes models vary on, we need to see what findings hold across which axes (model size, architecture, attention schemes, etc) if we are ever to develop "laws" of interpretability. Only then can we start to apply mechanistic alignment techniques broadly across the industry and have a reasonable expectation that it will work.
Personally, I'm quite optimistic based on our early results. I firmly believe that this is a solvable problem, but it won't be easy and requires an enormous amount of domain expertise (I'm so grateful to work with such talented researchers) to keep the agents from slopping it up. It will require an even more enormous about of resources, which is why it has to be an effort of the community at large. When we're ready, I hope that we can rally everyone who has the drive to solve these hard problems to use our encyclopedia and to help us fill it out.
if any science should be revolutionised by AI, it should be interpretability. I wrote about how we can accelerate progress, and why we need a Human Genome Project for AI. coauthored with @mathildepapillo
Tal Haklay retweeted
The most ambitious vision for interpretability has long been to reverse-engineer a model well enough to answer an arbitrary question about how it works.
For years, that’s seemed out of reach. Now, we think it’s possible. Neural networks are cleaner than we thought: we can now understand most neurons in models up to trillions of parameters. And agents can rapidly accelerate our progress.
But to get there, we need a concerted effort to unlock scaling laws for interpretability, where our understanding of a model grows predictably with compute.
That’s why we’re committing $10M to start a Human Genome Project for AI. Our ultimate goal is to reverse-engineer a frontier scale model and publish a living encyclopedia explaining how it works.
We don’t know exactly what we’ll find, and we can’t do it alone. We hope you’ll join us. Our chief scientist @banburismus_ lays out the vision here:
if any science should be revolutionised by AI, it should be interpretability. I wrote about how we can accelerate progress, and why we need a Human Genome Project for AI. coauthored with @mathildepapillo
Tal Haklay retweeted
if any science should be revolutionised by AI, it should be interpretability. I wrote about how we can accelerate progress, and why we need a Human Genome Project for AI. coauthored with @mathildepapillo
Tal Haklay retweeted
I'm at #EMNLP2026 in Budapest over the next few days! Presenting this work tomorrow, Oct 25, Poster Session B (16:00–17:30, Hall H).
If you're into Speech LMs, interpretability or neuro-AI, feel free to DM me to grab a coffee or chat! ✌️🤗
Tal Haklay retweeted
The average human lifespan is just four thousand weeks. Every day is precious and fleeting. That is why I am requesting that the SAC charge the authors of this paper with murder
Tal Haklay retweeted
Very happy to see this from Nick!
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Tal Haklay retweeted
working to help get these results has been very exciting, but there are still many unanswered questions
if expanding on this line of research is interesting to you, we're currently looking for research fellows to work with me on the many followups involved! link in next tweet
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Tal Haklay retweeted
Check on your neurons
“The canonical evidence of polysemanticity and superposition that convinced much of the field, including me, that networks pack concepts arbitrarily was misread”
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Good news from @GoodfireAI 🔥
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Tal Haklay retweeted
yessss i can finally talk about this! neurons are much more interpretable than expected, very excited to see where this goes
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Tal Haklay retweeted
when nick told me that he had some interp breakthroughs, he was uh, underselling it. i think this may be remembered as a major moment in scientific history. news so good even eliezer will have to lower his pdoom. we can start making alignment into a 'normal science'
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Tal Haklay retweeted
For years, neural networks looked inscrutable, like tangled knots that needed to be unwound. We no longer think that's the case. AI models are far more interpretable than we thought. We think this is great news for interpretability.
Goodfire's @nickcammarata co-authored Zoom In, the paper that introduced polysemanticity and superposition. He spent months revisiting the vision model organisms that much of interpretability grew out of and studying modern language models up to trillions of parameters. His new essay explains what the early field, him included, got wrong, why there's reason for optimism today, and why there might be a bitter lesson for interpretability.
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/