Christopher H. Bennett

Why neural networks learn at the edge of chaos When you train a neural network, parameter updates aren't normally distributed. They're heavy-tailed—rare large jumps punctuate many small adjustments. This pattern appears across MLPs, CNNs, and Transformers, across MNIST and CIFAR-10, across early rapid learning and late convergence. It's suspiciously universal. Xin-Ya Zhang and Chao Tang argue this isn't a quirk of stochastic gradient descent or mini-batch noise. It's a signature of self-organized criticality, emerging from a fundamental trade-off: the maximum entropy principle pushes the network to explore freely, while a mutual information constraint forces updates to remain task-relevant. Balance these two forces, and you get power-law statistics—the same scaling behavior seen in earthquakes, neuronal avalanches, and financial markets. The evidence is compelling. The power-law exponent stays remarkably stable throughout training, even as loss drops by orders of magnitude. The loss landscape itself shows multiscale structure: exponential smoothness under small perturbations (locally flat basins), transitioning to power-law ruggedness at larger scales. Even the timing of large updates follows heavy-tailed statistics—big learning events cluster together rather than occurring randomly, with exponents around 2.5–2.7. What makes this conceptually satisfying is the theoretical derivation from first principles. Starting from entropy maximization under an information constraint, and using a path-integral formulation of KL divergence through parameter space, the authors recover exactly the observed scaling behavior. No fine-tuning, no ad hoc assumptions. The implications cut deep: neural network learning isn't just optimization—it's a nonequilibrium physical process governed by the same statistical principles that shape complex systems throughout nature. Understanding this could guide the design of more efficient learning algorithms and explain why SGD generalizes better than adaptive methods that suppress large exploratory updates. Paper: pnas.org/doi/full/10.1073/pn…