Replying to @biggestjoel
the human concepts of thinking, wanting and understanding are valuable tools for conceptualizing AI models. it is ultimately a disservice to readers if journalists refuse to use them. yes, they are not perfect words, but we do not have better ones.
Was just as excited as everyone else to read this cool new paper, and it felt like a trip down memory lane!
XM seems to be the same as IMLE, and the motivation and insights are also similar. Having worked on IMLE for years, I'll provide some context for my followers 👇.
1/n
We discovered a third pretraining axis beyond parameters and data: exploration.
Scaling exploration monotonically improves existing models across images/video/language, and unlocks end-to-end generation.
In the simplest case, it's just a for loop.
Introducing Explorative Modeling.
TLDR:
- Gains from exploration grow with scale: 7%→36% as data scales, 13%→23% as parameters scale, and gains double at 3× the compute
- Adding exploration to ~SOTA baselines improves data efficiency by 6.2×, FLOP efficiency by 4.1×, parameter efficiency by 47%, and hits a near-SOTA 1.43 unguided FID on ImageNet
- Exploration lets you trade training compute for generalization, and scales how end-to-end your generative model is
- End-to-end Explorative Models (XMs) match diffusion performance on control tasks with up to 256× less inference compute
🧵Thread:
Miles retweeted
Introducing MazeBench, a 3D open world environment that evaluates long term planning with visual spatial reasoning.
It spans hundreds of rooms and puzzles. Today’s best agents cannot progress beyond the initial levels.
Replying to @SebAaltonen
I encountered a bug a few days ago, k3 found it by digging through the PyTorch MPS kernel source code and finding a numerical error with probability 1 in 10 million, showed that that error coincided exactly with my frequency of loss spikes. wild stuff






