Abstract
Neural Ordinary Differential Equation (Neural ODE) has been proposed as acontinuous approximation to the ResNet architecture. Some commonly usedregularization mechanisms in discrete neural networks (e.g. dropout, Gaussiannoise) are missing in current Neural ODE networks. In this paper, we propose anew continuous neural network framework called Neural Stochastic DifferentialEquation (Neural SDE) network, which naturally incorporates various commonlyused regularization mechanisms based on random noise injection. Our frameworkcan model various types of noise injection frequently used in discrete networksfor regularization purpose, such as dropout and additive/multiplicative noisein each block. We provide theoretical analysis explaining the improvedrobustness of Neural SDE models against input perturbations/adversarialattacks. Furthermore, we demonstrate that the Neural SDE network can achievebetter generalization than the Neural ODE and is more resistant to adversarialand non-adversarial input perturbations.