Abstract
It is well-known that the expressivity of a neural network depends on itsarchitecture, with deeper networks expressing more complex functions. In thecase of networks that compute piecewise linear functions, such as those withReLU activation, the number of distinct linear regions is a natural measure ofexpressivity. It is possible to construct networks with merely a single region,or for which the number of linear regions grows exponentially with depth; it isnot clear where within this range most networks fall in practice, either beforeor after training. In this paper, we provide a mathematical framework to countthe number of linear regions of a piecewise linear network and measure thevolume of the boundaries between these regions. In particular, we prove thatfor networks at initialization, the average number of regions along anyone-dimensional subspace grows linearly in the total number of neurons, farbelow the exponential upper bound. We also find that the average distance tothe nearest region boundary at initialization scales like the inverse of thenumber of neurons. Our theory suggests that, even after training, the number oflinear regions is far below exponential, an intuition that matches ourempirical observations. We conclude that the practical expressivity of neuralnetworks is likely far below that of the theoretical maximum, and that this gapcan be quantified.