AI Scientist @ Mistral prev: PhD student @ UCSD, Research Intern @ Google Research & MSR NE, Research Engineer at IBM Watson
San Francisco, CA
Joined June 2009
- Tweets115
- Following666
- Followers273
- Likes1.3K
Pinned Tweet
Grokking modular arithmetic is widely studied for the seemingly unique emergent abilities of neural networks.
Instead, we find that iteratively solving a kernel machine and estimating the Average Gradient Outer Product (AGOP) recovers this phenomenon identically:
Emergence in non-neural models: grokking modular arithmetic via average gradient outer product ift.tt/mCS5hTE
Neil Mallinar retweeted
shoutout to @eaboix and @nmallinar for actually doing the work that made this paper happen, and Misha Belkin for advising.
paper here: arxiv.org/abs/2507.05644
Neil Mallinar retweeted
coauthors + I couldn't get a poster printed in Rio in time, so here's me ad-libbing at @iclr_conf
Neil Mallinar retweeted
1/ Deep learning is going to have a scientific theory. We can see the pieces starting to come together, and it's looking a lot like physics!
We're releasing a paper pulling together these emerging threads and giving them a name: learning mechanics.
🔨 arxiv.org/pdf/2604.21691 🔧
Super excited to share that we have an Oral presentation for this paper next week at ICML!
It will be on Tuesday at 10am (Oral 1E) in West Ballroom D, I'll be presenting 4th at 10:45am :)
Our poster will be on Wednesday at 11am and I encourage you to stop by and chat!
Shout-out to my amazing coauthors @dbeagleholeCS @BusyZhu @PartheP Adit and Misha!! Check out all of their other papers they are brilliant researchers
2
153
Happy to share that we got a spotlight at ICML for this work, see y'all there!!
Grokking modular arithmetic is widely studied for the seemingly unique emergent abilities of neural networks.
Instead, we find that iteratively solving a kernel machine and estimating the Average Gradient Outer Product (AGOP) recovers this phenomenon identically:
14
472
Neil Mallinar retweeted
Two generalization regimes in ICL: (1) context-scaling, where performance improves with more in-context examples, and (2) task-scaling, where performance improves with more pre-training tasks. While MLPs show task-scaling but not context-scaling,
arxiv.org/abs/2410.12783
Neil Mallinar retweeted
Iterating kernel ridgeless regression with AGOP computation groks modular arithmetic… and this grokking is remarkably similar to the phenomenon in neural networks.
I found these results very surprising!
In our setting, grokking appears to occur solely due to feature learning. We decouple from neural architectures and gradient-descent based optimization by using kernels equipped with feature learning through AGOP and find many of the same phenomena as observed in neural networks.
1
3
235
Please check out our paper here: arxiv.org/abs/2407.20199
This was an amazing collaboration with Daniel Beaglehole (@dbeagleholeCS), Libin Zhu (@BusyZhu), Adit Radhakrishnan, Parthe Pandit (@PartheP), and Misha Belkin.
5
352
By the relation between circulant matrices and the Discrete Fourier Transform, we theoretically show that a quadratic kernel equipped with circulant features implements the same generalizing solution as neural networks - the Fourier Multiplication Algorithm found in prior work.
As before, initializing features in the neural network using a random circulant dramatically reduces the time-to-generalization.
1
2
211
The relationship between the NFM and neural network AGOP has been noted in prior work. arxiv.org/abs/2212.13881
In settings where weight decay, or trace(NFM), induces grokking we find that AGOP regularization, or trace(AGOP), does the same.
1
2
210
The progress measures of circulant deviation and AGOP alignment tend to steadily improve in the early iterations of neural networks as well, suggesting that feature learning is taking place in spite of unchanging test loss and accuracy.
1
1
254
We propose two "progress measures" that reveal gradual emergence: circulant deviation and AGOP alignment. These allow us to measure progress towards a generalizing model throughout training, and prior to grokking.
1
1
256
Like RFM, we show that generalizing neural networks learn block circulant structure in the Gram matrix of the first layer weights (or Neural Feature Matrix / NFM). Further, the AGOP of the same neural network contains the same structure (with correlation of > 0.92 to the NFM).
1
1
270
The features (AGOP) learned by a generalizing model for these tasks takes the form of a block circulant matrix. To see this structure for multiplication and division we must reorder input coordinates using the generator of their cyclic group.
1
2
342
