deep learning and large language models. football (banter) fan. 19. prev: research intern at https://nitter.gq/t.co/Uyagq8HNqO
Mumbai, India.
Joined October 2019
- Tweets6K
- Following1.4K
- Followers971
- Likes34.2K
Many quasi-"purists", me included, have seen the general meta of the industry w.r.t. RL envs with some snobbishness and flippance: "eww no the path to AGI should not be riddled with a horde of data contractors hand-building envs that pass certain specifications"
But, I have had some recent thoughts/muses that helped reconcile my own position (the thoughts excluding the subjective bitterness, but the actual inability to understand why this is the meta we converged on): It is well-understood that model capabilities (and gains therein) are fractal-like and follow long-tails (see img attached).
Imagine capabilities work as seeing the circle-packed disk below (Apollonian gasket) and trying to fill in big parts. The problem is that as parts get smaller and smaller, more specific and more scattered, it is an understandable concern to bet against super general uplifts (since the major part of the disk is filled in already, and the transfer/generalisation that we observe is not amazing).
More importantly, the bottleneck in such an “almost-filled-in-completely” regime is figuring out the uncolored parts in general. RL companies work similarly with the intent being to find cases/tasks where models have a low passing rate. The great evals are built in similar spirit and thus the “every good eval is an RL env” observation many make.
Of course, a bitter purist (like me) looks at this and still says "it's not general enough", but at least it makes perfect sense as to why it exists. In general, figuring out (and hillclimbing) the (ever decreasing) share of tasks that models can't do is a necessary and perfectly honorable pursuit.
But one keeps hoping that we see something more beautiful. Automated env design (adversarial setups will return), algorithms that generalise and transfer OOMs better (taking the "low samples, high compute" quadrant seriously), a new "meta", etc. etc.
some part of the latter seems attributable to evolution itself too
pmc.ncbi.nlm.nih.gov/article…
👀
'How biological synapses self-assemble gradient learning"
from Tommy's (Poggio) team @mcgovernmit
@qianli_liao Ziyn @LiuZiyin10 , @yule_gan, ..
@PNASNews
pnas.org/doi/10.1073/pnas.26…
Cool to see how these (incredible) efficiency gains affect val loss.
New historic NanoGPT record at 39.9s (-27.7s) from @DevenPzak , obliterating the prior record of 67.6s!
This record introduces a new paradigm of thinking to NanoGPT: instead of optimizing matmuls or adding more expressive operations, optimize at the individual flop level with incredibly clever engineering and ML judgement. If a flop is low value on a particular step, skip it.
Specifically:
-(~8s) Sampled softmax. If a token doesn’t appear in a batch, skip its lm_head fwd/bwd some fraction of the time.
-Sparse values. Only run an optimizer step for ngram embeddings that occurred in the batch. Set beta1 to zero to enable this. Beta2 is applied retroactively when the row is later used.
-Sparse updates. Only update ngram and value embeddings once every 4 steps instead of once every 2.
-Sparse communication. Shard the n-gram table across GPUs, and only pass the rows receiving updates on each step.
-Sparse optimizer states. For the n-gram table, reduce from 2 floats in Adam optimizer per param, to 1 float per 768 params.
-Hand-rolled flash attention for 64 dim heads.
There are several additions that add accuracy too:
-(~4s) EMA during last 300 steps, combined with lifting final_lr to 0.3 instead of 0.15.
-(~1s) A new optimizer, Anvil2, which expands muon via a second tracked momentum buffer, improves the ortho coefficients, and modifies the cautious weight decay application.
-A couple additional dynamic skip connections in the network.
The most striking consequence of the ‘flop aware paradigm’ is you can grow parameters arbitrarily large,
only limited by the available memory, since you can selectively choose how to expend flops on those parameters on each step. NanoGPT has kept active parameters below 124M, but total is unbounded, and has grown to 640M through embedding sparsity over the last year. This PR takes that to its logical conclusion on the 8xH100, scaling up to 65B sparse embedding parameters, which accounts for 25% of the PR’s gains. At frontier scale, where one is not bounded by an 8xH100, one could imagine where this paradigm could lead.
github.com/KellerJordan/modd…
As this was a very notable PR, I spoke with Deven for an hour to learn how he did it. Here’s his story on the changes: hyperstition.cc/training-nan…
life gets noticeably more boring after a certain age (IMO middle of teenage) for reasons not too different
New post: Jev has been a great intuition pump for what a capabilities overhang feels like. The generality of models is still underappreciated.
Some-to-many bedtime thoughts on that, electric motors, and "GPTs are GPTs".
hiranmay.com/blog/universall…
pretty crazy observation: mice lower in their group's hierarchy die earlier
pmc.ncbi.nlm.nih.gov/article…
