Abstract
This paper introduces "Non-Autonomous Input-Output Stable Network"(NAIS-Net), a very deep architecture where each stacked processing block isderived from a time-invariant non-autonomous dynamical system. Non-autonomy isimplemented by skip connections from the block input to each of the unrolledprocessing stages and allows stability to be enforced so that blocks can beunrolled adaptively to a pattern-dependent processing depth. We prove that thenetwork is globally asymptotically stable so that for every initial conditionthere is exactly one input-dependent equilibrium assuming tanh units, andmultiple stable equilibria for ReLU units. An efficient implementation thatenforces the stability under derived conditions for both fully-connected andconvolutional layers is also presented. Experimental results show how NAIS-Netexhibits stability in practice, yielding a significant reduction ingeneralization gap compared to ResNets.