Abstract
We utilize a connection between compositional kernels and branching processesvia Mehler's formula to study deep neural networks. This new probabilisticinsight provides us a novel perspective on the mathematical role of activationfunctions in compositional neural networks. We study the unscaled and rescaledlimits of the compositional kernels and explore the different phases of thelimiting behavior, as the compositional depth increases. We investigate thememorization capacity of the compositional kernels and neural networks bycharacterizing the interplay among compositional depth, sample size,dimensionality, and non-linearity of the activation. Explicit formulas on theeigenvalues of the compositional kernel are provided, which quantify thecomplexity of the corresponding reproducing kernel Hilbert space. On themethodological front, we propose a new random features algorithm, whichcompresses the compositional layers by devising a new activation function.