Miles (@Krubbled) header
Miles (@Krubbled) profile picture
~sf
Joined March 2019
Miles retweeted
twitter epistemics for interpretability research are really bad
15
28
5
510
23,659
Miles retweeted
Replying to @biggestjoel
the human concepts of thinking, wanting and understanding are valuable tools for conceptualizing AI models. it is ultimately a disservice to readers if journalists refuse to use them. yes, they are not perfect words, but we do not have better ones.
1
2
105
Miles retweeted
Was just as excited as everyone else to read this cool new paper, and it felt like a trip down memory lane! XM seems to be the same as IMLE, and the motivation and insights are also similar. Having worked on IMLE for years, I'll provide some context for my followers 👇. 1/n
We discovered a third pretraining axis beyond parameters and data: exploration. Scaling exploration monotonically improves existing models across images/video/language, and unlocks end-to-end generation. In the simplest case, it's just a for loop. Introducing Explorative Modeling. TLDR: - Gains from exploration grow with scale: 7%→36% as data scales, 13%→23% as parameters scale, and gains double at 3× the compute - Adding exploration to ~SOTA baselines improves data efficiency by 6.2×, FLOP efficiency by 4.1×, parameter efficiency by 47%, and hits a near-SOTA 1.43 unguided FID on ImageNet - Exploration lets you trade training compute for generalization, and scales how end-to-end your generative model is - End-to-end Explorative Models (XMs) match diffusion performance on control tasks with up to 256× less inference compute 🧵Thread:
24
92
28
996
325,516
Miles retweeted
Introducing MazeBench, a 3D open world environment that evaluates long term planning with visual spatial reasoning. It spans hundreds of rooms and puzzles. Today’s best agents cannot progress beyond the initial levels.
122
156
69
1,477
251,518
Miles retweeted
Replying to @SebAaltonen
I encountered a bug a few days ago, k3 found it by digging through the PyTorch MPS kernel source code and finding a numerical error with probability 1 in 10 million, showed that that error coincided exactly with my frequency of loss spikes. wild stuff
2
3
208
the secret to indie gamedev (no one will tell you): just make a fucking banger. everything else is survivor mumbo-jumbo
10
20
3
242
11,127