Abstract
A big mystery in deep learning continues to be the ability of methods togeneralize when the number of model parameters is larger than the number oftraining examples. In this work, we take a step towards a better understandingof the underlying phenomena of Deep Autoencoders (AEs), a mainstream deeplearning solution for learning compressed, interpretable, and structured datarepresentations. In particular, we interpret how AEs approximate the datamanifold by exploiting their continuous piecewise affine structure. Ourreformulation of AEs provides new insights into their mapping, reconstructionguarantees, as well as an interpretation of commonly used regularizationtechniques. We leverage these findings to derive two new regularizations thatenable AEs to capture the inherent symmetry in the data. Our regularizationsleverage recent advances in the group of transformation learning to enable AEsto better approximate the data manifold without explicitly defining the groupunderlying the manifold. Under the assumption that the symmetry of the data canbe explained by a Lie group, we prove that the regularizations ensure thegeneralization of the corresponding AEs. A range of experimental evaluationsdemonstrate that our methods outperform other state-of-the-art regularizationtechniques.