and the result is even cleaner if when taking the average, we also count for the size of the power law tail
45
๐๐ฎ๐ป ๐๐ฒ ๐ณ๐ถ๐
๐๐ฑ๐ฎ๐บ๐ชโ๐ ๐ถ๐บ๐ฝ๐น๐ถ๐ฐ๐ถ๐ ๐๐ฝ๐ฒ๐ฐ๐๐ฟ๐ฎ๐น ๐ฏ๐ถ๐ฎ๐ ๐๐๐ถ๐ป๐ด ๐ฅ๐ฒ๐ป๐ผ๐ฟ๐บ๐ฎ๐น๐ถ๐๐ฎ๐๐ถ๐ผ๐ป ๐๐ฟ๐ผ๐๐ฝ ๐๐ต๐ฒ๐ผ๐ฟ๐?
Weโve been testing a prediction from our new Renormalization Group (RG) theory of neural-network weight spectra, building on WeightWatcher, HTSR, and SETOL.
Our theory identifies ๐ฎ๐น๐ฝ๐ต๐ฎ = ๐ฎ as a marginal spectral regime. We expect the theory to work most naturally with scale-covariant optimizers such as SGD with momentum.
But AdamW introduces an interesting complication.
๐๐ฑ๐ฎ๐บ๐ช ๐ฏ๐ฟ๐ฒ๐ฎ๐ธ๐ ๐๐ต๐ฒ ๐๐ฐ๐ฎ๐น๐ฒ ๐๐๐บ๐บ๐ฒ๐๐ฟ๐.
AdamWโs adaptive update rule does not transform under weight rescaling in the same way as SGD with momentum.
We hypothesize that this symmetry breaking introduces an implicit spectral bias. During training, AdamW can produce layer exponents below ฮฑ = 2, while prolonged training may drive them above 2 without necessarily improving generalization.
๐๐ฎ๐ป ๐๐ฒ ๐ฐ๐ผ๐ฟ๐ฟ๐ฒ๐ฐ๐ ๐๐ต๐ถ๐ ๐ฏ๐ถ๐ฎ๐?
We modified AdamW by replacing its matrix-update magnitudes with SGDM-matched magnitudes, while retaining AdamWโs adaptive directions and moment history.
The goal is to restore the spectral behavior predicted by our RG framework without abandoning adaptive optimization.
๐ง๐ต๐ฒ ๐ฝ๐ฟ๐ฒ๐น๐ถ๐บ๐ถ๐ป๐ฎ๐ฟ๐ ๐ฟ๐ฒ๐๐๐น๐๐ ๐ฎ๐ฟ๐ฒ ๐ฒ๐ป๐ฐ๐ผ๐๐ฟ๐ฎ๐ด๐ถ๐ป๐ด!
We ran controlled experiments on MNIST and Fashion-MNIST, comparing native AdamW, modified AdamW, and SGD with momentum.
The accompanying figure shows 40 Fashion-MNIST runs for each AdamW variant.
The best modified AdamW model achieved:
โข ๐ต๐ฌ.๐ฎ๐ฒ% ๐๐ฒ๐๐ ๐ฎ๐ฐ๐ฐ๐๐ฟ๐ฎ๐ฐ๐
โข ๐๐ฆ-๐ฐ๐ผ๐ป๐ณ๐ถ๐ฑ๐ฒ๐ป๐ฐ๐ฒ-๐๐ฒ๐ถ๐ด๐ต๐๐ฒ๐ฑ ๐ฎ๐๐ฒ๐ฟ๐ฎ๐ด๐ฒ ฮฑ โ ๐ฎ.๐ฌ๐ด
โข Higher accuracy than every native AdamW model in this experiment.
The best native AdamW model reached 89.73%.
Importantly, ๐๐ฒ ๐ฎ๐ฟ๐ฒ ๐ป๐ผ๐ ๐ฐ๐น๐ฎ๐ถ๐บ๐ถ๐ป๐ด ๐๐ต๐ฎ๐ ๐ฒ๐๐ฒ๐ฟ๐ ๐น๐ฎ๐๐ฒ๐ฟ ๐ฐ๐ผ๐ป๐๐ฒ๐ฟ๐ด๐ฒ๐ ๐๐ผ ฮฑ = ๐ฎ.
The prediction concerns a collective spectral observable: the confidence-weighted average across FC1, FC2, and FC3.
We compute this using the quality of each layerโs power-law fit, measured by its KolmogorovโSmirnov distance:
ฮฑฬ_KS = [ฮฃโ ฮฑฬโ/(D_KS,โ + 0.02)] / [ฮฃโ 1/(D_KS,โ + 0.02)]
This weighting reduces the influence of poorly fitted spectra.
๐ช๐ต๐ฎ๐ ๐ฑ๐ผ๐ฒ๐ ๐๐ต๐ถ๐ ๐๐๐ด๐ด๐ฒ๐๐?
The results are consistent with our hypothesis that optimizer-induced scale-symmetry breaking affects spectral marginality.
Correcting the update magnitudes appears to produce models with improved accuracy and a collective spectral exponent close to the predicted marginal value.
These are preliminary results, not yet a demonstration of universal convergence or causality.
The experiments, checkpoint verification, and analysis were conducted using ๐๐ต๐ฎ๐๐๐ฃ๐ง ๐ช๐ผ๐ฟ๐ธ, which acted as an autonomous research assistant.
Weโll be extending this investigation to transformer models and preparing a more detailed write-up.
๐ง๐ต๐ฒ ๐ฒ๐บ๐ฒ๐ฟ๐ด๐ถ๐ป๐ด ๐ฝ๐ถ๐ฐ๐๐๐ฟ๐ฒ: ๐ผ๐ฝ๐๐ถ๐บ๐ถ๐๐ฒ๐ฟ ๐ฑ๐๐ป๐ฎ๐บ๐ถ๐ฐ๐ ๐บ๐ฎ๐๐๐ฒ๐ฟ ๐ณ๐ผ๐ฟ ๐๐ฝ๐ฒ๐ฐ๐๐ฟ๐ฎ๐น ๐๐ป๐ถ๐๐ฒ๐ฟ๐๐ฎ๐น๐ถ๐๐.
๐ด ๐ฉ๐ฎ๐ฐ๐ฎ๐๐ถ๐ผ๐ป ๐ฟ๐ฒ๐๐ฒ๐ฎ๐ฟ๐ฐ๐ต: ๐๐ฑ๐ฎ๐บ๐ช ๐ฎ๐ป๐ฑ ๐ฎ๐น๐ฝ๐ต๐ฎ
ChatGPT and I have been digging into why AdamW often pushes layer ฮฑ below 2.
๐ง๐ต๐ฒ ๐ฐ๐น๐๐ฒ: Adamโs normalization divides out the gradientโs scale. Under a scale-invariant loss, SGDM needs ฮท โ bยฒฮท; Adam needs ฮท โ bฮท. That breaks the SGDM scaling law used in our RG mapping.
๐ง๐ต๐ฒ ๐๐ฒ๐๐: Keep Adamโs direction. Replace its step magnitude with the momentum-SGD rule.
โข Test accuracy: ๐ต๐ณ.๐ฒ๐ต% โ ๐ต๐ด.๐ฐ๐ฐ%
โข ๐ฏ๐ฏ% ๐ณ๐ฒ๐๐ฒ๐ฟ ๐๐ฒ๐๐ ๐ฒ๐ฟ๐ฟ๐ผ๐ฟ๐, three fresh seeds
โข FC1 ฮฑ: 1.74 โ 2.09; FC2 ฮฑ: 1.86 โ 1.94
โข FC3 remained above 2
These MNIST results support a clearer explanation of AdamWโs implicit bias. Weโre working out the corrections to our theory. For now, I recommend ๐ฆ๐๐ ๐๐ถ๐๐ต ๐บ๐ผ๐บ๐ฒ๐ป๐๐๐บ as the reference for applying our WeightWatcher/RG framework.
A great example from the weightwatcher community discord channel testing out the ww-pgd optimizer on nano GPT
4
376
and one more subsection, I will add to the paper before I leave for vacation this week
Here I will describe two ideas
1. Why the average layer alpha is a good metric, even when the individual layer alphas are off
2. The difference between mode seeking and geometry, seeking and how we can gauge layers this way.
In finalizing the RG paper, I realize that I can connect the spectral flow directly to a a dilation beta function for each individual layer arising directly in BackProp
Once Iโm sure this is correct. Iโll add it to the paper and submit an updated version.
Comments welcome
๐ ๐ฆ๐ฝ๐ฒ๐ฐ๐๐ฟ๐ฎ๐น ๐ฅ๐ฒ๐ป๐ผ๐ฟ๐บ๐ฎ๐น๐ถ๐๐ฎ๐๐ถ๐ผ๐ป ๐๐ฟ๐ผ๐๐ฝ ๐ง๐ต๐ฒ๐ผ๐ฟ๐ ๐ผ๐ณ ๐๐ฒ๐ฎ๐ฟ๐ป๐ถ๐ป๐ด
Yeah, itโs taken me almost a decade of work to get here! My new paper is finally live on arXiv.
This has been a labor of love, beginning with research at UC Berkeley and continuing independently with wonderful collaborators.
I spent much of this summer writing it up with substantial help from ChatGPT. Itโs my first strongly AI-enabled research paper.
I believe weโre on the verge of a scientific revolution in AI. Yet we still donโt have a satisfactory theory explaining WHY deep learning works.
Why do remarkable capabilities emerge? Why do models generalize or overfit? Why do optimizers work? How does learning organize correlations?
๐ช๐ต๐ฎ๐ ๐๐ต๐ฒ ๐ฝ๐ฎ๐ฝ๐ฒ๐ฟ ๐ฝ๐ฟ๐ผ๐ฝ๐ผ๐๐ฒ๐
We bring together Heavy-Tailed Self-Regularization (HTSR), our Semi-Empirical Theory of Learning (SETOL), and optimization research into a proposed Renormalization Group (RG) framework.
๐ฆ๐ฝ๐ฒ๐ฐ๐๐ฟ๐ฎ๐น ๐ฅ๐ ๐ณ๐น๐ผ๐๐: Truncation, rescaling, and changes of measure defining effective flows and candidate fixed points.
๐ ๐ฎ๐ฟ๐ด๐ถ๐ป๐ฎ๐น๐ถ๐๐ ๐ฎ๐ ฮฑ โ ๐ฎ: Connecting scale invariance, the self-averaging boundary, and learned correlations.
๐๐ฒ๐ป๐ฒ๐ฟ๐ฎ๐น๐ถ๐๐ฎ๐๐ถ๐ผ๐ป: Understanding model quality, overfitting, and correlation traps through spectral statistics.
๐ช๐ช-๐ฃ๐๐: Using these ideas to develop better optimizers. A separate paper is coming soon!
Our broader hypothesis is:
๐ฆ๐ฝ๐ฒ๐ฐ๐๐ฟ๐ฎ๐น ๐จ๐ป๐ถ๐๐ฒ๐ฟ๐๐ฎ๐น๐ถ๐๐ ๐ผ๐ณ ๐๐ฒ๐ฎ๐ฟ๐ป๐ถ๐ป๐ด.
๐ ๐ฃ๐ฒ๐ฟ๐๐ผ๐ป๐ฎ๐น ๐ก๐ผ๐๐ฒ:
This is also an homage to my late PhD advisor, Karl Freed, a world expert in RG, and my undergraduate physics professor Kenneth Wilson, whose pioneering RG work earned him the Nobel Prize.
I tried to bring everything I learned from that old-school scientific education to bear on modern AI.
A huge thanks to Christopher Hinrichs for helping develop SETOL; Mirco Milletari a for helping develop the early ideas; Hari Prakash for his indispensable contributions to WW-PGD and our understanding of Correlation Traps , and Serena Peng for her work on early HTSR research.
I couldnโt have done this without them!
Ultimately, I developed WeightWatcher and its theory to help clients train, fine-tune, and evaluate better AI models. WeightWatcher has now exceeded 500,000 downloads.
This is a first draft of a proposal, not the final word. I welcome feedback, positive or negative, and hope you find it useful!
Some context may help here
This idea came to me when writing 5.6 of the more recent paper
weightwatcher.ai/rg_theory_dโฆ
where I was considering how to explain the choice of normalization on the ERG Trace-Log condition in terms of a layer-wide rescaling of each layer
21










