Abstract
Empirical studies reported that the Hessian matrix of neural networks (NNs)exhibits a near-block-diagonal structure, yet its theoretical foundationremains unclear. In this work, we reveal two forces that shape the Hessianstructure: a ``static force'' rooted in the architecture design, and a``dynamic force'' arisen from training. We then provide a rigorous theoreticalanalysis of ``static force'' at random initialization. We study linear modelsand 1-hidden-layer networks with the mean-square (MSE) loss and theCross-Entropy (CE) loss for classification tasks. By leveraging random matrixtheory, we compare the limit distributions of the diagonal and off-diagonalHessian blocks and find that the block-diagonal structure arises as $C\rightarrow \infty$, where $C$ denotes the number of classes. Our findingsreveal that $C$ is a primary driver of the near-block-diagonal structure. Theseresults may shed new light on the Hessian structure of large language models(LLMs), which typically operate with a large $C$ exceeding $10^4$ or $10^5$.