Calc Consulting (@CalcCon) header
Calc Consulting (@CalcCon) profile picture

Calculation Consulting is a boutique consultancy that specializes in machine learning, AI, and data science

San Francisco
Joined January 2013
and the result is even cleaner if when taking the average, we also count for the size of the power law tail
45
๐—–๐—ฎ๐—ป ๐˜„๐—ฒ ๐—ณ๐—ถ๐˜… ๐—”๐—ฑ๐—ฎ๐—บ๐—ชโ€™๐˜€ ๐—ถ๐—บ๐—ฝ๐—น๐—ถ๐—ฐ๐—ถ๐˜ ๐˜€๐—ฝ๐—ฒ๐—ฐ๐˜๐—ฟ๐—ฎ๐—น ๐—ฏ๐—ถ๐—ฎ๐˜€ ๐˜‚๐˜€๐—ถ๐—ป๐—ด ๐—ฅ๐—ฒ๐—ป๐—ผ๐—ฟ๐—บ๐—ฎ๐—น๐—ถ๐˜‡๐—ฎ๐˜๐—ถ๐—ผ๐—ป ๐—š๐—ฟ๐—ผ๐˜‚๐—ฝ ๐˜๐—ต๐—ฒ๐—ผ๐—ฟ๐˜†? Weโ€™ve been testing a prediction from our new Renormalization Group (RG) theory of neural-network weight spectra, building on WeightWatcher, HTSR, and SETOL. Our theory identifies ๐—ฎ๐—น๐—ฝ๐—ต๐—ฎ = ๐Ÿฎ as a marginal spectral regime. We expect the theory to work most naturally with scale-covariant optimizers such as SGD with momentum. But AdamW introduces an interesting complication. ๐—”๐—ฑ๐—ฎ๐—บ๐—ช ๐—ฏ๐—ฟ๐—ฒ๐—ฎ๐—ธ๐˜€ ๐˜๐—ต๐—ฒ ๐˜€๐—ฐ๐—ฎ๐—น๐—ฒ ๐˜€๐˜†๐—บ๐—บ๐—ฒ๐˜๐—ฟ๐˜†. AdamWโ€™s adaptive update rule does not transform under weight rescaling in the same way as SGD with momentum. We hypothesize that this symmetry breaking introduces an implicit spectral bias. During training, AdamW can produce layer exponents below ฮฑ = 2, while prolonged training may drive them above 2 without necessarily improving generalization. ๐—–๐—ฎ๐—ป ๐˜„๐—ฒ ๐—ฐ๐—ผ๐—ฟ๐—ฟ๐—ฒ๐—ฐ๐˜ ๐˜๐—ต๐—ถ๐˜€ ๐—ฏ๐—ถ๐—ฎ๐˜€? We modified AdamW by replacing its matrix-update magnitudes with SGDM-matched magnitudes, while retaining AdamWโ€™s adaptive directions and moment history. The goal is to restore the spectral behavior predicted by our RG framework without abandoning adaptive optimization. ๐—ง๐—ต๐—ฒ ๐—ฝ๐—ฟ๐—ฒ๐—น๐—ถ๐—บ๐—ถ๐—ป๐—ฎ๐—ฟ๐˜† ๐—ฟ๐—ฒ๐˜€๐˜‚๐—น๐˜๐˜€ ๐—ฎ๐—ฟ๐—ฒ ๐—ฒ๐—ป๐—ฐ๐—ผ๐˜‚๐—ฟ๐—ฎ๐—ด๐—ถ๐—ป๐—ด! We ran controlled experiments on MNIST and Fashion-MNIST, comparing native AdamW, modified AdamW, and SGD with momentum. The accompanying figure shows 40 Fashion-MNIST runs for each AdamW variant. The best modified AdamW model achieved: โ€ข ๐Ÿต๐Ÿฌ.๐Ÿฎ๐Ÿฒ% ๐˜๐—ฒ๐˜€๐˜ ๐—ฎ๐—ฐ๐—ฐ๐˜‚๐—ฟ๐—ฎ๐—ฐ๐˜† โ€ข ๐—ž๐—ฆ-๐—ฐ๐—ผ๐—ป๐—ณ๐—ถ๐—ฑ๐—ฒ๐—ป๐—ฐ๐—ฒ-๐˜„๐—ฒ๐—ถ๐—ด๐—ต๐˜๐—ฒ๐—ฑ ๐—ฎ๐˜ƒ๐—ฒ๐—ฟ๐—ฎ๐—ด๐—ฒ ฮฑ โ‰ˆ ๐Ÿฎ.๐Ÿฌ๐Ÿด โ€ข Higher accuracy than every native AdamW model in this experiment. The best native AdamW model reached 89.73%. Importantly, ๐˜„๐—ฒ ๐—ฎ๐—ฟ๐—ฒ ๐—ป๐—ผ๐˜ ๐—ฐ๐—น๐—ฎ๐—ถ๐—บ๐—ถ๐—ป๐—ด ๐˜๐—ต๐—ฎ๐˜ ๐—ฒ๐˜ƒ๐—ฒ๐—ฟ๐˜† ๐—น๐—ฎ๐˜†๐—ฒ๐—ฟ ๐—ฐ๐—ผ๐—ป๐˜ƒ๐—ฒ๐—ฟ๐—ด๐—ฒ๐˜€ ๐˜๐—ผ ฮฑ = ๐Ÿฎ. The prediction concerns a collective spectral observable: the confidence-weighted average across FC1, FC2, and FC3. We compute this using the quality of each layerโ€™s power-law fit, measured by its Kolmogorovโ€“Smirnov distance: ฮฑฬ‚_KS = [ฮฃโ‚— ฮฑฬ‚โ‚—/(D_KS,โ‚— + 0.02)] / [ฮฃโ‚— 1/(D_KS,โ‚— + 0.02)] This weighting reduces the influence of poorly fitted spectra. ๐—ช๐—ต๐—ฎ๐˜ ๐—ฑ๐—ผ๐—ฒ๐˜€ ๐˜๐—ต๐—ถ๐˜€ ๐˜€๐˜‚๐—ด๐—ด๐—ฒ๐˜€๐˜? The results are consistent with our hypothesis that optimizer-induced scale-symmetry breaking affects spectral marginality. Correcting the update magnitudes appears to produce models with improved accuracy and a collective spectral exponent close to the predicted marginal value. These are preliminary results, not yet a demonstration of universal convergence or causality. The experiments, checkpoint verification, and analysis were conducted using ๐—–๐—ต๐—ฎ๐˜๐—š๐—ฃ๐—ง ๐—ช๐—ผ๐—ฟ๐—ธ, which acted as an autonomous research assistant. Weโ€™ll be extending this investigation to transformer models and preparing a more detailed write-up. ๐—ง๐—ต๐—ฒ ๐—ฒ๐—บ๐—ฒ๐—ฟ๐—ด๐—ถ๐—ป๐—ด ๐—ฝ๐—ถ๐—ฐ๐˜๐˜‚๐—ฟ๐—ฒ: ๐—ผ๐—ฝ๐˜๐—ถ๐—บ๐—ถ๐˜‡๐—ฒ๐—ฟ ๐—ฑ๐˜†๐—ป๐—ฎ๐—บ๐—ถ๐—ฐ๐˜€ ๐—บ๐—ฎ๐˜๐˜๐—ฒ๐—ฟ ๐—ณ๐—ผ๐—ฟ ๐˜€๐—ฝ๐—ฒ๐—ฐ๐˜๐—ฟ๐—ฎ๐—น ๐˜‚๐—ป๐—ถ๐˜ƒ๐—ฒ๐—ฟ๐˜€๐—ฎ๐—น๐—ถ๐˜๐˜†.
1
4
22
771
๐ŸŒด ๐—ฉ๐—ฎ๐—ฐ๐—ฎ๐˜๐—ถ๐—ผ๐—ป ๐—ฟ๐—ฒ๐˜€๐—ฒ๐—ฎ๐—ฟ๐—ฐ๐—ต: ๐—”๐—ฑ๐—ฎ๐—บ๐—ช ๐—ฎ๐—ป๐—ฑ ๐—ฎ๐—น๐—ฝ๐—ต๐—ฎ ChatGPT and I have been digging into why AdamW often pushes layer ฮฑ below 2. ๐—ง๐—ต๐—ฒ ๐—ฐ๐—น๐˜‚๐—ฒ: Adamโ€™s normalization divides out the gradientโ€™s scale. Under a scale-invariant loss, SGDM needs ฮท โ†’ bยฒฮท; Adam needs ฮท โ†’ bฮท. That breaks the SGDM scaling law used in our RG mapping. ๐—ง๐—ต๐—ฒ ๐˜๐—ฒ๐˜€๐˜: Keep Adamโ€™s direction. Replace its step magnitude with the momentum-SGD rule. โ€ข Test accuracy: ๐Ÿต๐Ÿณ.๐Ÿฒ๐Ÿต% โ†’ ๐Ÿต๐Ÿด.๐Ÿฐ๐Ÿฐ% โ€ข ๐Ÿฏ๐Ÿฏ% ๐—ณ๐—ฒ๐˜„๐—ฒ๐—ฟ ๐˜๐—ฒ๐˜€๐˜ ๐—ฒ๐—ฟ๐—ฟ๐—ผ๐—ฟ๐˜€, three fresh seeds โ€ข FC1 ฮฑ: 1.74 โ†’ 2.09; FC2 ฮฑ: 1.86 โ†’ 1.94 โ€ข FC3 remained above 2 These MNIST results support a clearer explanation of AdamWโ€™s implicit bias. Weโ€™re working out the corrections to our theory. For now, I recommend ๐—ฆ๐—š๐—— ๐˜„๐—ถ๐˜๐—ต ๐—บ๐—ผ๐—บ๐—ฒ๐—ป๐˜๐˜‚๐—บ as the reference for applying our WeightWatcher/RG framework.
1
1
8
390
A great example from the weightwatcher community discord channel testing out the ww-pgd optimizer on nano GPT
4
376
and one more subsection, I will add to the paper before I leave for vacation this week Here I will describe two ideas 1. Why the average layer alpha is a good metric, even when the individual layer alphas are off 2. The difference between mode seeking and geometry, seeking and how we can gauge layers this way.
2
13
694
In finalizing the RG paper, I realize that I can connect the spectral flow directly to a a dilation beta function for each individual layer arising directly in BackProp Once Iโ€™m sure this is correct. Iโ€™ll add it to the paper and submit an updated version. Comments welcome
2
26
989
๐—” ๐—ฆ๐—ฝ๐—ฒ๐—ฐ๐˜๐—ฟ๐—ฎ๐—น ๐—ฅ๐—ฒ๐—ป๐—ผ๐—ฟ๐—บ๐—ฎ๐—น๐—ถ๐˜‡๐—ฎ๐˜๐—ถ๐—ผ๐—ป ๐—š๐—ฟ๐—ผ๐˜‚๐—ฝ ๐—ง๐—ต๐—ฒ๐—ผ๐—ฟ๐˜† ๐—ผ๐—ณ ๐—Ÿ๐—ฒ๐—ฎ๐—ฟ๐—ป๐—ถ๐—ป๐—ด Yeah, itโ€™s taken me almost a decade of work to get here! My new paper is finally live on arXiv. This has been a labor of love, beginning with research at UC Berkeley and continuing independently with wonderful collaborators. I spent much of this summer writing it up with substantial help from ChatGPT. Itโ€™s my first strongly AI-enabled research paper. I believe weโ€™re on the verge of a scientific revolution in AI. Yet we still donโ€™t have a satisfactory theory explaining WHY deep learning works. Why do remarkable capabilities emerge? Why do models generalize or overfit? Why do optimizers work? How does learning organize correlations? ๐—ช๐—ต๐—ฎ๐˜ ๐˜๐—ต๐—ฒ ๐—ฝ๐—ฎ๐—ฝ๐—ฒ๐—ฟ ๐—ฝ๐—ฟ๐—ผ๐—ฝ๐—ผ๐˜€๐—ฒ๐˜€ We bring together Heavy-Tailed Self-Regularization (HTSR), our Semi-Empirical Theory of Learning (SETOL), and optimization research into a proposed Renormalization Group (RG) framework. ๐—ฆ๐—ฝ๐—ฒ๐—ฐ๐˜๐—ฟ๐—ฎ๐—น ๐—ฅ๐—š ๐—ณ๐—น๐—ผ๐˜„๐˜€: Truncation, rescaling, and changes of measure defining effective flows and candidate fixed points. ๐— ๐—ฎ๐—ฟ๐—ด๐—ถ๐—ป๐—ฎ๐—น๐—ถ๐˜๐˜† ๐—ฎ๐˜ ฮฑ โ‰ˆ ๐Ÿฎ: Connecting scale invariance, the self-averaging boundary, and learned correlations. ๐—š๐—ฒ๐—ป๐—ฒ๐—ฟ๐—ฎ๐—น๐—ถ๐˜‡๐—ฎ๐˜๐—ถ๐—ผ๐—ป: Understanding model quality, overfitting, and correlation traps through spectral statistics. ๐—ช๐—ช-๐—ฃ๐—š๐——: Using these ideas to develop better optimizers. A separate paper is coming soon! Our broader hypothesis is: ๐—ฆ๐—ฝ๐—ฒ๐—ฐ๐˜๐—ฟ๐—ฎ๐—น ๐—จ๐—ป๐—ถ๐˜ƒ๐—ฒ๐—ฟ๐˜€๐—ฎ๐—น๐—ถ๐˜๐˜† ๐—ผ๐—ณ ๐—Ÿ๐—ฒ๐—ฎ๐—ฟ๐—ป๐—ถ๐—ป๐—ด. ๐—” ๐—ฃ๐—ฒ๐—ฟ๐˜€๐—ผ๐—ป๐—ฎ๐—น ๐—ก๐—ผ๐˜๐—ฒ: This is also an homage to my late PhD advisor, Karl Freed, a world expert in RG, and my undergraduate physics professor Kenneth Wilson, whose pioneering RG work earned him the Nobel Prize. I tried to bring everything I learned from that old-school scientific education to bear on modern AI. A huge thanks to Christopher Hinrichs for helping develop SETOL; Mirco Milletari a for helping develop the early ideas; Hari Prakash for his indispensable contributions to WW-PGD and our understanding of Correlation Traps , and Serena Peng for her work on early HTSR research. I couldnโ€™t have done this without them! Ultimately, I developed WeightWatcher and its theory to help clients train, fine-tune, and evaluate better AI models. WeightWatcher has now exceeded 500,000 downloads. This is a first draft of a proposal, not the final word. I welcome feedback, positive or negative, and hope you find it useful!
3
11
77
2,948
Some context may help here This idea came to me when writing 5.6 of the more recent paper weightwatcher.ai/rg_theory_dโ€ฆ where I was considering how to explain the choice of normalization on the ERG Trace-Log condition in terms of a layer-wide rescaling of each layer
21
The book of diagrams hundreds of open shell Feynman diagrams, all handwritten out A souvenir from ancient times
2
163