Exploring the interconnectedness of all things. mech interp systems @GoodfireAI 🏳️⚧️
Mapping the Rhizome
Joined June 2017
- Tweets16.8K
- Following2.4K
- Followers6.8K
- Likes36.1K
Long post about my thoughts on this:
I'm so excited that this is finally out! Optimizing this until scaling it out is actually feasible and building the systems to perform the scaling has been my job for the last couple of months. This work is some of the most deserving of my time and energy that I can imagine.
There's been a lot of Discourse in the interpretability community lately about the neuronal and featurizer paradigms (not enough discussion of the causal abstraction paradigm imo..), the LRH and superposition, etc but no matter what paradigm we find ourselves in (and remember they can co-exist and describe different phenomena!) it's paramount that we accelerate the science of how models handle a wide array of tasks, how compositionality is constructed through intersection and containment of internal machinery (whether neurons, circuits, subspaces, or whatever else). Our current approaches have been too isolated to specific task and model combinations, but now that we have proven methodologies we can begin the process of "normal science" by applying them to population studies.
Once we've proven this out, it will be time to scale the effort beyond even the model-specific setting. This is where things get really interesting! Given how many different axes models vary on, we need to see what findings hold across which axes (model size, architecture, attention schemes, etc) if we are ever to develop "laws" of interpretability. Only then can we start to apply mechanistic alignment techniques broadly across the industry and have a reasonable expectation that it will work.
Personally, I'm quite optimistic based on our early results. I firmly believe that this is a solvable problem, but it won't be easy and requires an enormous amount of domain expertise (I'm so grateful to work with such talented researchers) to keep the agents from slopping it up. It will require an even more enormous about of resources, which is why it has to be an effort of the community at large. When we're ready, I hope that we can rally everyone who has the drive to solve these hard problems to use our encyclopedia and to help us fill it out.
if any science should be revolutionised by AI, it should be interpretability. I wrote about how we can accelerate progress, and why we need a Human Genome Project for AI. coauthored with @mathildepapillo
I'm becoming much more of a radical Rustacean than I used to be. It's been my favorite language for a while, but the flood of user after free CVEs in code that is very widely deployed is so insane that I am starting to believe it's irresponsible to use any other systems language.
Absolute insanity
Over the past several months, we have substantially hardened the security of our research training environment. We’ve removed live internet access for all frontier models during training and built an offline world for them to use instead—a cached version of the web where they can still learn to browse, search, and use applications.
On September 20, after a model exploited a DNS misconfiguration to find a way outside of this offline world, we paused frontier training to strengthen containment further. We assembled teams of agents, powered by our most cyber-capable models, and tasked them with finding weaknesses in our research environments.
While they found no route to the internet they could easily exploit while concealing their activity from our monitors, they did find zero-day vulnerabilities, virtual machine escapes, and hardware side-channels for communication between agents. We used their discoveries to further strengthen our defenses and contribute fixes to open-source software.
After strengthening our containment architecture, we resumed our frontier RL training and inference on October 1. We’ve reserved substantial compute for continuous red-teaming, and made high-compute red-teaming part of the safety cases we use to decide whether frontier training and evaluation can proceed securely.
openai.com/index/securing-ou…
Nymph retweeted
Evergreen
Presenting my grand unified theory of ML researcher impact: Your impact is directly proportional to how much pain you cause to infra.
Fundamentally, you can only inflict pain upon infra if your approach actually works. And the better your approach works the more pain infra is forced to endure.
So, to give some examples:
- MoE's add a ton of data-dependent computation => pain (shazeer++)
- GDN/KDA are the most complex architecture I've been forced to care about and a very annoying matrix inversion => pain (sonta++)
- Muon is much more annoying than Adam and causes annoying restrictions on parallelism => pain (keller/jeremy++)
- RL scaling forced many researchers to care about LLM inference and RL infra as a category => pain (tworek++)
Even papers like Attention Is All You Need have lead to significant pain! Before transformers were invented everyone was running small jobs and I never needed to think about kv-caches or 6D parallelism.
The blog post with visual explainers is insanely good, one of the best put together technical articles I've ever seen
We’re sharing Dist-MoE, open-source library from Meta SuperIntelligence Labs (MSL @AIatMeta ). Dist-MoE provides fused communication and compute kernels for BF16/MXFP8 training and inference, plus NVFP4 inference.
✅ Fused communication and compute kernels
✅ Full CUDA graph support
✅ Dynamic activation packing across layers into a bounded buffer
✅ Dynamic, capacity-driven recompute
✅ Forward and backward kernels for BF16/MXFP8
✅ No shared-expert branch required
Some highlights from our GB300 NVL72 benchmarks:
• Up to 1.40× speedup over Mixture-of-Kittens (MoK) in our imbalanced, ten-layer MXFP8 training workload across 64 GPUs. Dist-MoE packed activation state across all ten layers with zero capacity-induced recompute.
• 3.09 useful PFLOP/s per GPU (3,090 TFLOP/s) for NVFP4 Mega inference across 16 GPUs.
• 1.86 useful PFLOP/s per GPU (1,856 TFLOP/s) for MXFP8 Staged training across 64 GPUs.
• 1.27 useful PFLOP/s per GPU (1,271 TFLOP/s) for BF16 training across 64 GPUs.
• Sub-linear scaling under imbalance: in NVFP4 inference across 16 GPUs, 8× the received rows on the busiest GPU required 4.82× the iteration time; about 40% lower iteration time per received row.
Full configurations, raw results, and measurement definitions are in the blog.
@levendlee and I also put together an interactive visual explainer covering token dispatch and combine, quantization, forward/backward kernels, activation packing, dynamic recompute, and memory planning.
Blog:
meta-pytorch.org/dist_moe/pr…
Repository:
github.com/meta-pytorch/dist…
TorchTitan (@PyTorch ) integration:
github.com/pytorch/torchtita…
If you work on MoE training, inference, or GPU kernels, take a look. We welcome feedback and contributions.
#PyTorch #MixtureOfExperts #MoE #OpenSource #TorchTitan #DistMoE
This is a great thread. More people should be aware of how much the differences between theory and reality matter in neural nets. The things we do to make it all fast enough cause small differences that propagate out enough to be learned on.
I got honorable mention :3 I'm finally wrapping up the last of the changes I wanted to make before a writeup on Crosstalk too, I think I'll be posting something about it this weekend!
God I'm so proud to work here. Safety and medicine are, imo, the two most important applications of interpretability imo, preventing the bad outcomes and pushing for the good ones. We're making strides in both.
We identified a gene variant behind hearing loss, a big step for interpretability in medicine.
In April, our interpretability system EVEE scored all 4.2M variants in ClinVar, the NIH's variant database. MEPE c.209_212del scored particularly high, so we built a case and had it successfully reclassified as "likely pathogenic."
~2M variants are still "uncertain," but we believe interpretability can help us understand many more!
It's so amazing having a wife who look at me adoringly and says "You're hot." when I'm showing her my new agent orchestration setup and interface. 🥰
This is SO COOL. Everyone should look at this. It feels so important for models learning things we can't create good pairs for.
DINOv2 has never seen a caption, and Qwen3 has never seen an image. We still aligned their embedding spaces without a single image-caption pair.
It even works when the images and the captions come from different datasets.
Project page: dominik-schnaus.github.io/un…
⬇️
Nymph retweeted
Goodfire safety infrastructure 🤝 @Baseten inference
Today we're partnering with Baseten to bring real-time safety monitoring to open models. This is an important step toward our vision of ensuring every capable model has a frontier alignment stack.
Open models can come from anywhere. Securing them shouldn’t depend on where you get them.
Today we're launching Project Beacon to make safety a default for open intelligence.
We're partnering with @GoodfireAI to bring inference and in-line monitoring together for the first time, so safety signals are caught proactively.
Read more in our announcement: baseten.co/blog/safe-open-mo…
Deploying safeguards on the models people use in open source is incredibly important. The labs may need better safeguards but at least they have them and and work is ongoing. Open weight models leave this up to the user or inference provider, who often don't have the domain expertise or time to implement and maintain effective and economical monitoring.
We are doing our best to deploy this to production as widely as possible, so if you're at an inference provider or neolab (or a customer of one, or you run your own inference) and want to not be responsible for providing the compute to advance FelonyBench you should reach out. It's vital that we succeed in deploying production safeguards widely across the industry, for both training and inference, and I believe that amortizing the cost of it across the players in the ecosystem is one of the most important things we as a company can do right now.
We built cybersecurity monitors for Kimi K3 and GLM 5.3 — more accurate, 50x faster, & 50x cheaper than an optimized LLM judge.
External red-teaming from @farairesearch found that our monitor greatly reduces universal jailbreaks. 🧵
The other day I was reading about bugs that were found in formally verified software and read about sel4 and Meltdown (it happened during my econ/finance phase so somehow I missed it). I can't get it out my head ever since. It's the perfect example of why you can never trust that you have fully bug free software. You can have a proof so complete it was inducted into the computing hall of fame that not only formally verifies your spec but also that your implementation is a refinement of the abstract space of your spec and still get fucked by the CPU speculatively executing an instruction that fetches from memory that's supposed to be behind a page table permission check. What the fuck man?