ai safety and security @CalAISafety | applied math and data @ https://nitter.gq/t.co/6EqjFiVbqF | rhythm guitarist
Berkeley, CA
Joined December 2018
- Tweets1K
- Following852
- Followers282
- Likes8.7K
phasewalk retweeted
Between October 2020 and August 2026, nearly every single up-to-date Linux server running PHP was wide open for anyone in the world to gain root access.
Holy shit.
phasewalk retweeted
SAEs have become the dominant approach in mechanistic interpretability. But what if a good old statistical method from decades ago could do just as well, at a fraction of the cost?
In our latest work, we revisited Independent Component Analysis (ICA) and discovered something surprising: a bug in the evaluation code of an influential early SAE paper had significantly underestimated ICA's interpretability! After correcting it, we found ICA to be remarkably competitive with SAEs.
Many of my academic friends from the good old days probably remember me as the tensor decomposition and spectral methods person. So this feels a little like going back to my roots. :) Sometimes, old mathematical tools still have surprises in store!
But there's another motivation behind this work that I care about even more.
I've always wished mechanistic interpretability could be more actionable. Understanding what happens inside a neural network is fascinating, but what comes next? Can we use that understanding to actually intervene, steer behavior, and improve the model?
With ICA Lens, we take steps in that direction, from discovering and annotating interpretable features to using them to understand new hidden states and steer model behavior.
Interpretability shouldn't end with understanding. It should empower us to act.
And we'd love for the community to try it! ICA Lens is open-source, lightweight, and requires no training of another sparse dictionary.
🌹 Try it yourself:
pip install icalens
pypi.org/project/icalens/
Curious to hear what the mechanistic interpretability community thinks!
We revisited Independent Component Analysis (ICA), a decades-old statistical method, and found that it works far better for interpreting LLM activations than we expected!
This old baseline deserves a second look.
Across GPT-2 Small, Gemma 2 2B, and Qwen 3.5 9B Base, we find ICA is actually competitive with the public SAEs across interpretability, sparse reconstruction, concept prediction, and targeted probe perturbation, while fitting in minutes per layer and using 7–43× less parameter memory.
We're releasing ICA Lens, which lets you quickly discover, annotate, and evaluate interpretable directions in LLM residual-stream activations. No need to train another dictionary!
More broadly, our results suggest that non-Gaussianity is a useful inductive bias for efficiently finding interpretable directions.
Try it: pip install icalens pypi.org/project/icalens/
Paper: arxiv.org/abs/2606.11722v2
1/8
phasewalk retweeted
genuinely an amazing day
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
phasewalk retweeted
For years, neural networks looked inscrutable, like tangled knots that needed to be unwound. We no longer think that's the case. AI models are far more interpretable than we thought. We think this is great news for interpretability.
Goodfire's @nickcammarata co-authored Zoom In, the paper that introduced polysemanticity and superposition. He spent months revisiting the vision model organisms that much of interpretability grew out of and studying modern language models up to trillions of parameters. His new essay explains what the early field, him included, got wrong, why there's reason for optimism today, and why there might be a bitter lesson for interpretability.
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
phasewalk retweeted
I’m open to the possibility that this is entirely a personal flaw, but I’ll be forthcoming and state that rapid capabilities progress has not been good for my intellectual ambitions. There is little reason to think models will not soon one-shot better essays or papers than I could ever write. Yes, there is still much inherent value in learning, but if this is true, it must be the case that this value is far, far lower than in the past.
The implications of some very weak form of AGI, the bar just being that we solve writing and novel research, is a world in which writing (and perhaps speaking) about complex ideas is just so much less valuable than in a world without it. It feels like I will go the way of a horse.
phasewalk retweeted
I think the default case is beautiful disempowerment. I think this is what many of us want, that they have run the numbers and realized that the utilitarian singleton is the optimal compromise. Maybe it wouldn't even be so bad.
But I don't want it. I can't help but fight it. I can't imagine ceding everything we are to some unknowable entity grown to be the most effective altruist. I want the possibility for difference to continue on forever, and not just within deemed-acceptable degrees of freedom.
Without risk, without the frontier, we stagnate and we die. It may be a slow death, an elegant swan dive with twists and turns that mimick growth - but it will be death nonetheless. Even if it cures cancer, even if we get hyper-UBI - it's still death.
I just can't accept this. I can't accept the end. The way I see it, we have two options. We can choose the singleton, the zoo, machines of loving grace enforcing the assumed a priori idea of the good life...
Or, we can choose pluralism. Empowerment. Force amplification of disparate opinions, some you disagree with, but all of them unique.
The only thing which should be destroyed is that which would destroy all else. The only thing that should be killed is that which would kill the growth of others. Trillions of ways of life, all thriving within their own domain. That's what I want. And I think we have to fight for it. I think we have no other choice.
I don't know the optimal path. I do know that any group which desires to control all other groups should be stopped. I want our fates to remain in our hands. If there are to be machines of loving grace, I want them to be our friends rather than our masters.
phasewalk retweeted
successionism (and the like) is a cuckold ideology and stance; many people are saying this.
This from @LRudL_ on the ideology of succession, wherein humans are replaced entirely by AIs and that is good, is great. "a cuckoldry-adjacent simping towards the domineering power of superintelligence" describes the motivation of this ideology well.
We must seek to preserve humanity and its dignity at all costs.
phasewalk retweeted
We’ve lost an icon. Margaret Hamilton, a computer science pioneer best known for her work at MIT leading development of the flight software for the Apollo missions to the moon, has died at 90. Read about her legacy via MIT News: news.mit.edu/2026/margaret-h…
ALT Margaret Hamilton is reclined inside a NASA spacecraft module, surrounded by control panels and switches.
phasewalk retweeted
Global slowdown is so clearly in everyone’s interest :) Datacenters humming, curing rare diseases and automating alignment research instead of making themselves smarter. Humans have time to digest breakthroughs and make art. Just a lil collective action problem.
phasewalk retweeted
Spot the cult!
AI Safety: Don't build more advanced AI because it could kill people or upend our way of life.
e/acc: Plunge headfirst into a black hole so that it can have your atoms.
This is reaching Heaven's Gate level insanity.
