Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise

  • 2019-06-05 23:19:50
  • Xuanqing Liu, Tesi Xiao, Si Si, Qin Cao, Sanjiv Kumar, Cho-Jui Hsieh
  • 22


Neural Ordinary Differential Equation (Neural ODE) has been proposed as acontinuous approximation to the ResNet architecture. Some commonly usedregularization mechanisms in discrete neural networks (e.g. dropout, Gaussiannoise) are missing in current Neural ODE networks. In this paper, we propose anew continuous neural network framework called Neural Stochastic DifferentialEquation (Neural SDE) network, which naturally incorporates various commonlyused regularization mechanisms based on random noise injection. Our frameworkcan model various types of noise injection frequently used in discrete networksfor regularization purpose, such as dropout and additive/multiplicative noisein each block. We provide theoretical analysis explaining the improvedrobustness of Neural SDE models against input perturbations/adversarialattacks. Furthermore, we demonstrate that the Neural SDE network can achievebetter generalization than the Neural ODE and is more resistant to adversarialand non-adversarial input perturbations.


Quick Read (beta)

Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise

Xuanqing Liu
Department of Computer Science, UCLA
[email protected]
   Tesi Xiao
Department of Statistics, UC Davis
[email protected]
   Si Si
Google Research
[email protected]
   Qin Cao
Google Research
[email protected]
   Sanjiv Kumar
Google Research
[email protected]
   Cho-Jui Hsieh
Department of Computer Science, UCLA
[email protected]

Neural Ordinary Differential Equation (Neural ODE) has been proposed as a continuous approximation to the ResNet architecture. Some commonly used regularization mechanisms in discrete neural networks (e.g. dropout, Gaussian noise) are missing in current Neural ODE networks. In this paper, we propose a new continuous neural network framework called Neural Stochastic Differential Equation (Neural SDE) network, which naturally incorporates various commonly used regularization mechanisms based on random noise injection. Our framework can model various types of noise injection frequently used in discrete networks for regularization purpose, such as dropout and additive/multiplicative noise in each block. We provide theoretical analysis explaining the improved robustness of Neural SDE models against input perturbations/adversarial attacks. Furthermore, we demonstrate that the Neural SDE network can achieve better generalization than the Neural ODE and is more resistant to adversarial and non-adversarial input perturbations.

1 Introduction

Residual neural networks (ResNet) [1] are composed of multiple residual blocks transforming the hidden states according to:

𝒉n+1=𝒉n+𝒇(𝒉n;𝒘n), (1)

where 𝒉n is the input to the n-th layer and 𝒇(𝒉n;𝒘n) is a non-linear function parameterized by 𝒘n. Recently, a continuous approximation to the ResNet architecture has been proposed [2], where the evolution of the hidden state 𝒉t can be described as a dynamic system obeying the equation:

𝒉t=𝒉s+st𝒇(𝒉τ,τ;𝒘)\difτ, (2)

where 𝒇(𝒉τ,τ;𝒘) is the continuous form of the nonlinear function 𝒇(𝒉n;𝒘n); 𝒉s and 𝒉t are hidden states at two different time st. A standard ODE solver can be used to solve all the hidden states and final states (output from the neural network), starting from an initial state (input to the neural network). The continuous neural network described in (2) exhibits several advantages over its discrete counterpart described in (1), in terms of memory efficiency, parameter efficiency, explicit control of the numerical error of final output, etc.

One missing component in the current Neural ODE network is the various regularization mechanisms commonly employed in discrete neural networks. These regularization techniques have been demonstrated to be crucial in reducing generalization errors, and in improving the robustness of neural networks to adversarial attacks. Many of these regularization techniques are based on stochastic noise injection. For instance, dropout [3] is widely adopted to prevent overfitting; injecting Gaussian random noise during the forward propagation is effective in improving generalization [4, 5] as well as robustness to adversarial attacks [6, 7]. However, these regularization methods in discrete neural networks are not directly applicable to Neural ODE network, because Neural ODE network is a deterministic system.

Our work attempts to incorporate the above-mentioned stochastic noise injection based regularization mechanisms to the current Neural ODE network, to improve the generalization ability and the robustness of the network. In this paper, we propose a new continuous neural network framework called Neural Stochastic Differential Equation (Neural SDE) network, which models stochastic noise injection by stochastic differential equations (SDE). In this new framework, we can employ existing techniques from the stability theory of SDE to study the robustness of neural networks. Our results provide theoretical insights to understanding why introducing stochasticity during neural network training and testing leads to improved robustness against adversarial attacks. Furthermore, we demonstrate that, by incorporating the noise injection regularization mechanism to the continuous neural network, we can reduce overfitting and achieve lower generalization error. For instance, on the CIFAR-10 dataset, we observe that the new Neural SDE can improve the test accuracy of the Neural ODE from 81.63% to 84.55%, with other factors unchanged. Our contributions can be summarized as follows:

  • We propose a new Stochastic Differential Equation (SDE) framework to incorporate randomness in continuous neural networks. The proposed random noise injection can be used as a drop-in component in any continuous neural networks. Our Neural SDE framework can model various types of noises widely used for regularization purpose in discrete networks, such as dropout (Bernoulli type) and Gaussian noise.

  • Training the new SDE network requires developing different backpropagation approach from the Neural ODE network. We develop a new efficient backpropagation method to calculate the gradient, and to train the Neural SDE network in a scalable way. The proposed method has its roots in stochastic control theory.

  • We carry out a theoretical analysis of the stability conditions of the Neural SDE network, to prove that the randomness introduced in the Neural SDE network can stabilize the dynamical system, which helps improve the robustness and generalization ability of the neural network.

  • We verify by numerical experiments that stochastic noise injection in the SDE network can successfully regularize the continuous neural network models, and the proposed Neural SDE network achieves better robustness and improves generalization performance.


Throughout this paper, we use 𝒉n to denote the hidden states in a neural network, where 𝒉0=𝒙 is the input (also called initial condition) and y is the label. The residual block with parameters 𝒘d can be written as a nonlinear transform 𝒇(𝒉τ,τ;𝒘). We assume the integration is always taken from 0 to T. 𝐁tm is m-dimensional Brownian motion. 𝑮(𝒉τ,τ;𝒗)n×m is the diffusion matrix parameterized by 𝒗. Unless stated explicitly, we use to represent 2-norm for vector and Frobenius norm for matrix.

2 Related work

Our work is inspired by the success of the recent Neural ODE network, and we seek to improve the generalization and robustness of Neural ODE, by adding regularization mechanisms crucial to the success of discrete networks. Regularization mechanisms such as dropout cannot be easily incorporated in the Neural ODE due to its deterministic nature.

Neural ODE

The basic idea of Neural ODE is discussed in the previous section, here we briefly review relevant literature. The idea of formulating ResNet as a dynamic system was discussed in [8]. A framework was proposed to link existing deep architectures with discretized numerical ODE solvers [9], and was shown to be parameter efficient. These networks adopt layer-wise architecture – each layer is parameterized by different independent weights. The Neural ODE model [2] computes hidden states in a different way: it directly models the dynamics of hidden states by an ODE solver, with the dynamics parameterized by a shared model. A memory efficient approach to compute the gradients by adjoint methods was developed, making it possible to train large, multi-scale generative networks [10, 11]. Our work can be regarded as an extension of this framework, with the purpose of incorporating a variety of noise-injection based regularization mechanisms. Stochastic differential equation in the context of neural network has been studied before, focusing either on understanding how dropout shapes the loss landscape [12], or on using stochastic differential equation as a universal function approximation tool to learn the solution of high dimensional PDEs [13]. Instead, our work tries to explain why adding random noise boosts the stability of deep neural networks, and demonstrates the improved generalization and robustness.

Noisy Neural Networks

Adding random noise to different layers is a technique commonly employed in training neural networks. Dropout [3] randomly disables some neurons to avoid overfitting, which can be viewed as multiplying hidden states with Bernoulli random variables. Stochastic depth neural network [14] randomly drops some residual blocks of residual neural network during training time. Another successful regularization for ResNet is Shake-Shake regularization [15], which sets a binary random variable to randomly switch between two residual blocks during training. More recently, dropblock [16] was designed specifically for convolutional layers: unlike dropout, it drops some continuous regions rather than sparse points to hidden states. All of the above regularization techniques are proposed to improve generalization performance. One common characteristic of them is that they fix the network during testing time. There is another line of research that focuses on improving robustness to perturbations/adversarial attacks by noise injection. Among them, random self-ensemble [6, 7] adds Gaussian noise to hidden states during both training and testing time. In training time, it works as a regularizer to prevent overfitting; in testing time, the random noise is also helpful, which will be explained in this paper.

3 Neural Stochastic Differential Equation

Figure 1: Toy example. By comparing the simulations under σ=0 and σ=2.8, we see adding noise to the system can be an effective way to control xt. Average over multiple runs is used to cancel out the volatility during the early stage.

In this section, we first introduce our proposed Neural SDE to improve the robustness of Neural ODE. Informally speaking, Neural SDE can be viewed as using randomness as a drop-in augmentation for Neural ODE, and it can include some widely used randomization layers such as dropout and Gaussian noise layer [6]. However, solving Neural SDE is non-trivial, we derive the gradients of loss over model weights. Finally we theoretically analyze the stability conditions of Neural SDE.

Before delving into the multi-dimensional SDE, let’s first look at a 1-d toy example to see how SDE can solve the instability issue of ODE. Suppose we have a simple SDE, \difxt=xt\dift+σxt\difBt with Bt be the standard Brownian motion. We provide a numerical simulation in Figure 1 for xt with different σ.

When we set σ=0, SDE becomes ODE \difxt=xt\dift and xt=c0et where c0 is an integration constant. If c00 we can see that xt±. Furthermore, a small perturbation in xt will be amplified through t. This clearly shows instability of ODE. On the other hand, if we instead make σ>1 (the system is SDE), we have xt=c0exp((1-σ2)t+σBt)a.s.0.

Figure 2: Our model architecture. The initial value of SDE is the output of a convolutional layer, and the value at time T is passed to a linear classifier after average pooling.

The toy example in Figure 1 reveals that the behavior of solution paths can change significantly after adding a stochastic term. This example is inspiring because we can control the impact of perturbations on the output by adding a stochastic term to neural networks.

Figure 2 shows a sample Neural SDE model architecture, and it is the one used in the experiment. It consists of three parts, the first part is a single convolution block, followed by a Neural SDE network (we will explain the detail of Neural SDE in Section 3.1) and lastly the linear classifier. We put most of the trainable parameters into the second part (Neural SDE), whereas the first/third parts are mainly for increasing/reducing the dimension as desired. Recall that both Neural ODE and SDE are dimension preserving.

3.1 Modeling randomness in neural networks

In the Neural ODE system (2), a slightly perturbed input state will be amplified in deep layers (as shown in Figure 1) which makes the system unstable to input perturbation and prone to overfitting. Randomness is an important component in discrete networks (e.g., dropout for regularization) to tackle this issue, however to our knowledge, there is no existing work concerning adding randomness in the continuous neural networks. And it is non-trivial to encode randomness in continuous neural networks, such as Neural ODE, as we need to consider how to add randomness so that to guarantee the robustness, and how to solve the continuous system efficiently. To solve these challenges, motivated by  [9, 12], we propose to add a single diffusion term into Neural ODE as:

\dif𝒉t=𝒇(𝒉t,t;𝒘)\dift+𝑮(𝒉t,t;𝒗)\dif𝐁t, (3)

where 𝐁t is the standard Brownian motion [17], which is a continuous time stochastic process such that 𝐁t+s-𝐁s follows Gaussian with mean 0 and variance t; 𝑮(𝒉t,t;𝒗) is a transformation parameterized by 𝒗. This formula is quite general, and can include many existing randomness injection models with residual connections under different forms of 𝑮(𝒉t,t;𝒗). As examples, we briefly list some of them below.

Gaussian noise injection:

Consider a simple example in (3) when 𝑮(𝒉t,t;𝒗) is a diagonal matrix, and we can model both additive and multiplicative noise as

additive: \dif𝒉t=𝒇(𝒉t,t;𝒘)\dift+𝚺(t)\dif𝐁t (4)
multiplicative: \dif𝒉t=𝒇(𝒉t,t;𝒘)\dift+𝚺(𝒉t,t)\dif𝐁t,

where 𝚺(t) is a diagonal matrix and its diagonal elements control the variance of the noise added to hidden states. This can be viewed as a continuous approximation of noise injection techniques in discrete neural network. For example, the discrete version of the additive noise can be written as

𝒉n+1=𝒉n+𝒇(𝒉n;𝒘n)+𝚺n𝒛n,with 𝚺n=σn𝑰,𝒛ni.i.d.𝒩(0,1), (5)

which injects Gaussian noise after each residual block. It has been shown that injecting small Gaussian noise can be viewed as a regularization in neural networks [4, 5]. Furthermore, [6, 7] recently showed that adding a slightly larger noise in one or all residual blocks can improve the adversarial robustness of neural networks. We will provide the stability analysis of (3) in Section 3.3, which provides a theoretical explanation towards the robustness of Neural SDE.


Our framework can also model the dropout layer which randomly disables some neurons in the residual blocks. Let us see how to unify dropout under our Neural SDE framework. First we notice that in the discrete case

𝒉n+1=𝒉n+𝒇(𝒉n;𝒘n)𝜸np=𝒉n+𝒇(𝒉n;𝒘n)+𝒇(𝒉n;𝒘n)(𝜸np-𝑰), (6)

where 𝜸ni.i.d.(1,p) and indicates the Hadamard product. Note that we divide γn by p in (6) to maintain the same expectation. Furthermore, we have

𝜸np-I=1-ppp1-p(γnp-𝑰)1-pp𝒛n,𝒛ni.i.d.𝒩(0,1). (7)

The boxed part above is approximated by standard normal distribution (by matching the first and second order moment). The final SDE with dropout can be obtained by combining (6) with (7)

\dif𝒉t=𝒇(𝒉t,t;𝒘)\dift+1-pp𝒇(𝒉t,t;𝒘)\dif𝐁t. (8)


[9] includes some other stochastic layers that can be formulated under Neural SDE framework, including shake-shake regularization [15] and stochastic depth [14]. Both of them are used as regularization techniques that work very similar to dropout.

3.2 Back-propagating through SDE integral

To optimize the parameters 𝒘, we need to back-propagate the Neural SDE system. A straightforward solution is to rely on the autograd method derived from chain rule. However, for Neural SDE the chain can be fairly long. If SDE solver discretizes the range [t0,t1] to N intervals, then the chain has 𝒪(N) nodes and the memory cost is 𝒪(N). One challenging part of backpropagation for Neural SDE is to calculate the gradient through SDE solver which could have high memory cost. To solve this issue, we first calculate the expected loss conditioning on the initial value 𝒉0, denoted as L=𝔼[(𝒉t1)|𝒉0]. Then our goal is to calculate L𝒘. In fact, we have the following theorem (also called path-wise gradient [18, 19]).

Theorem 3.1.

For continuously differentiable loss (𝐡t1), we can obtain an unbiased gradient estimator as

L𝒘^=(𝒉t1)𝒘=(𝒉t1)𝒉t1𝒉t1𝒘. (9)

Moreover, if we define 𝛃t=𝐡t𝐰, then 𝛃t follows another SDE

\dif𝜷t=(𝒇(𝒉t,t;𝒘)𝒘+𝒇(𝒉t,t;𝒘)𝒉t𝜷t)\dift+(𝑮(𝒉t,t;𝒘)𝒘+𝑮(𝒉t,t;𝒘)𝒉t𝜷t)\dif𝐁t. (10)

It is easy to check that if 𝐆0, then our Monte-Carlo gradient estimator (9) falls back to the exact gradient by back-propagation.

Similar to the adjoint method in Neural ODE, we will solve (10) jointly with the original SDE dynamics (3), this process can be done iteratively without memorizing the middle states, which makes it more memory efficient than autograd (𝒪(1) v.s. 𝒪(N) memory, N is the number of steps determined by SDE solver).

3.3 Robustness of Neural SDE

In this section, we theoretically analyze the stability of Neural SDE, showing that the randomness term can indeed improve the robustness of the model against small input perturbation. This also explains why noise injection can improve the robustness in discrete networks, which has been observed in literature [6, 7]. First we need to show the existence and uniqueness of solution to (3), we pose following assumptions on drift 𝒇 and diffusion 𝑮.

Assumption 1.

𝒇 and 𝐆 are at most linear, i.e. 𝐟(𝐱,t)+𝐆(𝐱,t)c1(1+𝐱) for c1>0, 𝐱Rn and tR+.

Assumption 2.

𝒇 and 𝐆 are c2-Lipschitz: 𝐟(𝐱,t)-𝐟(𝐲,t)+𝐆(𝐱,t)-𝐆(𝐲,t)c2𝐱-𝐲 for c2>0, 𝐱,𝐲Rn and tR+.

Based on the above assumptions, we can show that the SDE (3) has a unique solution [17]. We remark that assumption on 𝒇 is quite natural and is also enforced on the original Neural ODE model [2]; as to diffusion matrix 𝑮, we have seen that for dropout, Gaussian noise injection and other random models, both assumptions are automatically satisfied as long as 𝒇 possesses the same regularities.

We analyze the dynamics of perturbation. Our analysis applies not only to the Neural SDE model but also to Neural ODE model, by setting the diffusion term 𝑮 to zero. First of all, we consider initializing our Neural SDE (3) at two slightly different values 𝒉0 and 𝒉0e=𝒉0+𝜺0, where 𝜺0 is the perturbation for 𝒉0 with 𝜺0δ. So, under the new perturbed initialization 𝒉0e, the hidden state at time t follows the same SDE in (3),

\dif𝒉te=𝒇(𝒉te,t;𝒘)\dift+𝑮(𝒉te,t;𝒗)\dif𝐁t,with 𝒉0e=𝒉0+𝜺0, (11)

where 𝐁t is Brownian motions for the SDE associated with initialization 𝒉0e. Then it is natural to analyze how the perturbation 𝜺t=𝒉te-𝒉t evolves in the long run. Subtracting (3) from (11)

\dif𝜺t =[𝒇(𝒉te,t;𝒘)-𝒇(𝒉t,t;𝒘)]\dift+[𝑮(𝒉te,t;𝒗)-𝑮(𝒉t,t;𝒗)]\dif𝐁t (12)

Here we made an implicit assumption that the Brownian motions 𝐁t and 𝐁t have the same sample path for both initialization 𝒉0 and 𝒉0e, i.e. 𝐁t=𝐁t w.p.1. In other words, we focus on the difference of two random processes 𝒉t and 𝒉te driven by the same underlying Brownian motion. So it is valid to subtract the diffusion terms.

An important property of (12) is that it admits a trivial solution 𝜺t𝟎, t+ and 𝒘d. We show that both the drift (𝒇) and diffusion (𝑮) are zero under this solution:

𝒇Δ(𝟎,t;𝒘) =𝒇(𝒉t+𝟎,t;𝒘)-𝒇(𝒉t,t;𝒘)=0, (13)
𝑮Δ(𝟎,t;𝒗) =𝑮(𝒉t+𝟎,t;𝒘)-𝑮(𝒉t,t;𝒘)=0.

The implication of zero solution is clear: for a neural network, if we do not perturb the input data, then the output will never change. However, the solution 𝜺t=𝟎 can be highly unstable, in the sense that for an arbitrarily small perturbation 𝜺0𝟎 at initialization, the change of output 𝜺T can be arbitrarily bad. On the other hand, as shown below, by choosing the diffusion term 𝑮 properly, we can always control 𝜺t within a small range.

In general, we cannot get the closed form solution to a multidimensional SDE but we can still analyze the asymptotic stability through the dynamics 𝒇 and 𝑮. This is essentially an extension of Lyapunov stability theory to a stochastic system. First we define the notion of stability in the stochastic case. Let (Ω,,P) be a complete probability space with filtration {t}t0 and 𝐁t be an m-dimensional Brownian motion defined in the probability space, we consider the SDE in Eq. (12) with initial value 𝜺0

\dif𝜺t=𝒇Δ(𝜺t,t)\dift+𝑮Δ(𝜺t,t)\dif𝐁t, (14)

For simplicity we dropped the dependency on parameters 𝒘 and 𝒗. We further assume 𝒇Δ:n×+n and 𝑮Δ:n×+n×m are both Borel measurable. We can show that if assumptions (1) and (2) hold for 𝒇 and 𝑮, then they hold for 𝒇Δ and 𝑮Δ as well (see Lemma A.1 in Appendix), and we know the SDE (14) allows a unique solution 𝜺t. We have the following Lynapunov stability results from [20].

Definition 3.1 (Lyapunov stability of SDE).

The solution 𝛆t=0 of (14):

  • A.

    is stochastically stable if for any α(0,1) and r>0, there exists a δ=δ(α,r)>0 such that Pr{𝜺t<r for all t0}1-α whenever 𝜺0δ. Moreover, if for any α(0,1), there exists a δ=δ(α)>0 such that Pr{limt𝜺t=0}1-α whenever 𝜺0δ, it is said to be stochastically asymptotically stable;

  • B.

    is almost surely exponentially stable if lim supt1tlog𝜺t<0 a.s.11 1 “a.s.” is the abbreviation for “almost surely”. for all 𝜺0n.

Note that for part A in Definition 3.1, it is hard to quantify how well the stability is and how fast the solution reaches equilibrium. In addition, under assumptions (1, 2), we have a straightforward result Pr{𝜺t𝟎 for all t0}=1 whenever 𝜺0𝟎 as shown in Appendix (see Lemma A.2). That is, almost all the sample paths starting from a non-zero initialization can never reach zero due to Brownian motion. On the contrary, the almost sure exponentially stability result implies that almost all the sample paths of the solution will be close to zero exponentially fast. We present the following theorem from [20] on the almost sure exponentially stability.

Theorem 3.2.

[20] If there exists a non-negative real valued function V(𝛆,t) defined on Rn×R+ that has continuous partial derivatives


and constants p>0,c1>0,c2R,c30 such that the following inequalities hold:

  1. 1.


  2. 2.


  3. 3.


for all 𝛆0 and t>0. Then for all 𝛆0Rn,

lim supt1tlog𝜺t-c3-2c22pa.s. (15)

In particular, if c32c2, the solution 𝛆t0 is almost surely exponentially stable.

We now consider a special case, when the noise is multiplicative 𝑮(𝒉t,t)=σ𝒉t and m=1. The corresponding SDE of perturbation 𝜺t=𝒉te-𝒉t has the following form

\dif𝜺t=𝒇Δ(𝜺t,t;𝒘)\dift+σ𝜺t\dif𝐁t. (16)

Note that for the deterministic case of (16) by setting σ0, the solution may not be stable in certain cases (see Figure 1). Whereas for general cases when σ>0, following corollary claims that by setting σ properly, we will achieve an (almost surely) exponentially stable system.

Corollary 3.2.1.

For (16), if 𝐟(𝐡t,t;𝐰) is L-Lipschtiz continuous w.r.t. 𝐡t, then (16) has a unique solution with the property lim supt1tlog𝛆t-(σ22-L) almost surely for any 𝛆0Rn. In particular, if σ2>2L, the solution 𝛆t=0 is almost surely exponentially stable.

4 Experimental Results

In this section we show the effectiveness of our Neural SDE framework in terms of generalization, non-adversarial robustness and adversarial robustness. We use the SDE model architecture illustrated in Figure 2 during the experiment. Throughout our experiments, we set 𝒇() to be a neural network with several convolution blocks. As to 𝑮() we have the following choices:

  • Neural ODE, this can be done by dropping the diffusion term 𝑮(𝒉t,t;𝒗)=𝟎.

  • Additive noise, when the diffusion term is independent of 𝒉t, here we simply set it to be diagonal 𝑮(𝒉t,t;𝒗)=σt𝑰.

  • Multiplicative noise, when the diffusion term is proportional to 𝒉t, or 𝑮(𝒉t,t;𝒘)=σt𝒉t.

  • Dropout noise, when the diffusion term is proportional to the drift term 𝒇(𝒉t,t;𝒘), i.e. 𝑮(𝒉t,t;𝒗)=σtdiag{𝒇(𝒉t,t;𝒘)}.

Note the last three are our proposed Neural SDE with different types of randomness as explained in Section 3.1.

4.1 Generalization Performance

In the first experiment, we show small noise helps generalization. However, note that our noise injection is different from randomness layer in the discrete case, for instance, dropout layer adds Bernoulli noise at training time, but the layer are then fixed at testing time; whereas our Neural SDE model keeps randomness at testing time and takes the average of multiple forward propagation.

As for datasets, we choose CIFAR-10, STL-10 and Tiny-ImageNet22 2 Downloaded from to include various sizes and number of classes. The experimental results are shown in Table 1. We see that for all datasets, Neural SDE consistently outperforms ODE, and the reason is that adding moderate noise to the models at training time can act as a regularizer and thus improves testing accuracy. Based upon that, if we further keep testing time noise and ensemble the outputs, we will obtain even better results.

Table 1: Evaluating the model generalization under different choices of diffusion matrix 𝑮(𝒉t,t;𝒗) introduced above. For the last three noise types, we search a suitable parameter σt for each so that the diffusion matrix 𝑮 properly regularizes the model. TTN means testing time noise. Model size is counted by #parameters.
Data Model size [email protected] — w/o TTN [email protected] — w/ TTN
ODE Additive Multiplicative Dropout ODE Additive Multiplicative Dropout
CIFAR-10 115 K 81.63 83.65 83.26 83.60 83.89 83.76 84.55
STL-10 2.44 M 58.03 61.23 60.54 61.26 62.11 62.58 62.13
Tiny-ImageNet 2.49 M 45.19 45.25 46.94 47.04 45.39 46.65 47.81

4.2 Improved non-adversarial robustness

In this experiment, we aim at evaluating the robustness of models under non-adversarial corruptions following the idea of [21]. The corrupted datasets contain tens of defects in photography including motion blur, Gaussian noise, fog etc. For each noise type, we run Neural ODE and Neural SDE with dropout noise, and gather the testing accuracy. The final results are reported by mean accuracy (mAcc) in Table 2 by changing the level of corruption. Both models are trained on completely clean data, which means the corrupted images are not visible to them during the training stage, nor could they augment the training set with the same types of corruptions. From the table, we can see that Neural SDE performs better than Neural ODE in 8 out of 10 cases. For the rest two, both ODE and SDE are performing very close. This shows that our proposed Neural SDE can improve the robustness of Neural ODE under non-adversarial corrupted data.

Table 2: Testing accuracy results under different levels of non-adversarial perturbations.
Data Noise type mild corrupt Accuracy severe corrupt
Level 1 Level 2 Level 3 Level 4 Level 5
CIFAR10-C ODE 75.89 70.59 66.52 60.91 53.02
Dropout 77.02 71.58 67.21 61.61 53.81
Dropout+TTN 79.07 73.98 69.74 64.19 55.99
TinyImageNet-C ODE 23.01 19.18 15.20 12.20 9.88
Dropout 22.85 18.94 14.64 11.54 9.09
Dropout+TTN 23.84 19.89 15.28 12.08 9.44

4.3 Improved adversarial robustness

Figure 3: Comparing the robustness against 2-norm constrained adversarial perturbations, on CIFAR-10 (left), STL-10 (middle) and Tiny-ImageNet (right) data. We evaluate testing accuracy with three models, namely Neural ODE, Neural SDE with multiplicative noise and dropout noise.

Next, we consider the performance of Neural SDE models under adversarial perturbation. Clearly, this scenario is strictly harder than previous cases: by design, the adversarial perturbations are guaranteed to be the worst case within a small neighborhood (ignoring the suboptimality of optimization algorithms) crafted through constrained loss maximization procedure, so it represents the worst case performance. In our experiment, we adopt multi-step -PGD attack [22], although other strong white-box attacks such as C&W [23] are also suitable. The experimental results are shown in Figure 3. As we can see both Neural SDE with multiplicative noise and dropout noise are more resistant to adversarial attack than Neural ODE, and dropout noise outperforms multiplicative noise.

4.4 Visualizing the perturbations of hidden states

Figure 4: Comparing the perturbations of hidden states, 𝜺t, on both ODE and SDE (we choose dropout-style noise).

In this experiment, we take a look at the perturbation 𝜺t=𝒉te-𝒉t at any time t. Recall the 1-d toy example in Figure 1, we can observe that the perturbation at time t can be well suppressed by adding a strong diffusion term, which is also confirmed by theorem. However, it is still questionable whether the same phenomenon also exists in deep neural network since we cannot add very large noise to the network during training or testing time. If the noise is too large, it will also remove all useful features. Thus it becomes important to make sure that this will not happen to our models. To this end we first sample an input 𝒙 from CIFAR-10 and gather all the hidden states 𝒉t at time t=[0,Δt,2Δt,,NΔt]. Then we perform regular PGD attack [22] and find the perturbation 𝜹x such that 𝒙adv=𝒙+𝜹x is an adversarial image, and feed the new data 𝒙adv into network again so we get 𝒉te at the same time stamps as 𝒉t. Finally we plot the error 𝜺t=𝒉te-𝒉t w.r.t. time t (also called “network depth”), shown in Figure 4. We can observe that by adding a diffusion term (dropout-style noise), the error accumulates much slower than ordinary Neural ODE model.

5 Conclusion

To conclude, we introduce the Neural SDE model which can stabilize the prediction of Neural ODE by injecting stochastic noise. Our model can achieve better generalization and improve the robustness to both adversarial and non-adversarial noises.


We acknowledge the support by NSF IIS1719097, Intel, Google Cloud and AITRICS.


  • [1] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  • [2] Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In Advances in Neural Information Processing Systems, pages 6572–6583, 2018.
  • [3] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958, 2014.
  • [4] Chris M Bishop. Training with noise is equivalent to tikhonov regularization. Neural computation, 7(1):108–116, 1995.
  • [5] Guozhong An. The effects of adding noise during backpropagation training on a generalization performance. Neural computation, 8(3):643–674, 1996.
  • [6] Xuanqing Liu, Minhao Cheng, Huan Zhang, and Cho-Jui Hsieh. Towards robust neural networks via random self-ensemble. In Proceedings of the European Conference on Computer Vision (ECCV), pages 369–385, 2018.
  • [7] Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness to adversarial examples with differential privacy. arXiv preprint arXiv:1802.03471, 2018.
  • [8] Weinan E. A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics, 5(1):1–11, 2017.
  • [9] Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. In International Conference on Machine Learning, pages 3282–3291, 2018.
  • [10] Lynton Ardizzone, Jakob Kruse, Sebastian Wirkert, Daniel Rahner, Eric W Pellegrini, Ralf S Klessen, Lena Maier-Hein, Carsten Rother, and Ullrich Köthe. Analyzing inverse problems with invertible neural networks. arXiv preprint arXiv:1808.04730, 2018.
  • [11] Will Grathwohl, Ricky TQ Chen, Jesse Betterncourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. arXiv preprint arXiv:1810.01367, 2018.
  • [12] Qi Sun, Yunzhe Tao, and Qiang Du. Stochastic training of residual networks: a differential equation viewpoint. arXiv preprint arXiv:1812.00174, 2018.
  • [13] Maziar Raissi. Forward-backward stochastic neural networks: Deep learning of high-dimensional partial differential equations. arXiv preprint arXiv:1804.07010, 2018.
  • [14] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European conference on computer vision, pages 646–661. Springer, 2016.
  • [15] Xavier Gastaldi. Shake-shake regularization. arXiv preprint arXiv:1705.07485, 2017.
  • [16] Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V Le. Dropblock: A regularization method for convolutional networks. In Advances in Neural Information Processing Systems, pages 10727–10737, 2018.
  • [17] Bernt Øksendal. Stochastic differential equations. In Stochastic differential equations, pages 65–84. Springer, 2003.
  • [18] Jichuan Yang and Harold J Kushner. A monte carlo method for sensitivity analysis and parametric optimization of nonlinear stochastic systems. SIAM journal on control and optimization, 29(5):1216–1249, 1991.
  • [19] Emmanuel Gobet and Rémi Munos. Sensitivity analysis using itô–malliavin calculus and martingales, and application to stochastic optimal control. SIAM Journal on control and optimization, 43(5):1676–1713, 2005.
  • [20] Xuerong Mao. Stochastic differential equations and applications. Elsevier, 2007.
  • [21] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261, 2019.
  • [22] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017.
  • [23] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), pages 39–57. IEEE, 2017.

Appendix A Proofs

We present the proofs of theorems on stability of SDE. The proofs are adapted from [20]. We start with two crucial lemmas.

Lemma A.1.

If 𝐟,𝐆 satisfy Assumption (2), then 𝐟Δ,𝐆Δ satisfy Assumption (1,2).


By Assumption (2) on 𝒇,𝑮, we can obtain that for any 𝜺,𝜺~n,t0


This guarantees the uniqueness of the solution of (14). ∎

Lemma A.2.

For (14), whenever 𝛆00, Pr{𝛆t0 for all t0}=1.


We prove it by contradiction. Let τ=inf{t0:𝜺t=0}. Then if it is not true, there exists some 𝜺0𝟎 such that Pr{τ<}>0. Therefore, we can find sufficiently large constant T>0 and θ>1 such that Pr(A)Pr{τ<T and |𝜺t|θ-1, 0tτ}>0. By Assumption 2 on 𝒇 and 𝑮, there exists a positive constant Kθ such that

𝒇Δ(𝜺,t)+𝑮Δ(𝜺,t)Kθ𝜺, for all 𝜺θ and 0tT. (17)

Let V(𝜺,t)=ϵ-1. Then, for any 0𝜺θ and 0tT, we have

V(𝜺,t) =-𝜺-3𝜺𝒇Δ(𝜺,t)+12{-𝜺-3𝑮Δ(𝜺,t)2+3𝜺-5𝜺𝑮Δ(𝜺,t)2}
Kθ𝜺-1+Kθ2𝜺-1=Kθ(1+Kθ)V(𝜺,t), (18)

where the first inequality comes from Cauchy-Schwartz and the last one comes from (17). For any δ(0,𝜺0), we define the stopping time τδinf{t0:𝜺t(δ,θ)}. Let νδ=min{τδ,T}. By Itô’s formula, 𝔼[e-Kθ(1+Kθ)νδV(𝜺νδ,νδ)]

=V(𝜺0,0)+𝔼0νδe-Kθ(1+Kθ)s[-Kθ(1+Kθ)V(𝜺s,s)+V(𝜺s,s)]\difs𝜺0-1. (19)

Since τδT and 𝜺τδ=δ for any ωA, then (19) implies

𝔼[e-Kθ(1+Kθ)Tδ-1𝟏A]=δ-1e-Kθ(1+Kθ)TPr(A)𝜺0-1. (20)

Thus, Pr(A)δ𝜺0-1eKθ(1+Kθ)T. Letting δ0, we obtain Pr(A)=0, which leads to a contradiction. ∎

Proof of Theorem 3.2

We then prove Theorem 3.2. Clearly, (15) holds for 𝜺0=𝟎 since 𝜺t𝟎. For any 𝜺0𝟎, we have 𝜺t𝟎 for all t0 almost surely by Lemma A.2. Thus, by applying Itô’s formula and condition (2), we can show that for t0,

logV(𝜺t,t)logV(𝜺0,0)+c2t+M(t)-120t|V1(𝜺s,s)𝑮Δ(𝜺s,s)|2V2(𝜺s,s)\difs. (21)

where M(t)=0tV1(𝜺s,s)𝑮Δ(𝜺s,s)V(𝜺s,s)\dif𝐁s is a continuous martingale with initial value M(0)=0. By the exponential martingale inequality, for any arbitrary α(0,1) and n=1,2,, we have

Pr{sup0tn[M(t)-α20t|V1(𝜺s,s)𝑮Δ(𝜺s,s)|2V2(𝜺s,s)\difs]>2αlogn}1n2. (22)

Applying Borel-Cantelli lemma, we can get that for almost all ωΩ, there exists an integer n0=n0(ω) such that if nn0,

M(t)2αlogn+α20t|V1(𝜺s,s)𝑮Δ(𝜺s,s)|2V2(𝜺s,s)\difs, 0tn. (23)

Combining (21), (23) and condition (3), we can obtain that

logV(𝜺t,t)logV(𝜺0,0)-12[(1-α)c3-2c2]t+2αlogn. (24)

for all 0tn and nn0 almost surely. Therefore, for almost all ωΩ, if n-1tn and nn0, we have

1tlogV(𝜺t,t)-12[(1-α)c3-2c2]+logV(𝜺0,0)+2αlognn-1 (25)

which consequently implies

lim supt1tlogV(𝜺t,t)-12[(1-α)c3-2c2])a.s. (26)

With condition (1) and arbitrary choice of α(0,1), we can obtain (15).

Proof of Corollary 3.2.1

We apply Theorem 3.2 to establish the theories on stability of (16). Note that 𝒇(𝒉t,t;𝒘) is L-Lipschitz continuous w.r.t 𝒉t and 𝑮(𝒉t,t;𝒗)=σ𝒉t,m=1. Then, (16) has a unique solution, with 𝒇Δ and 𝑮Δ satisfying Assumptions (1,2)


To apply Theorem 3.2, let V(𝜺,t)=𝜺2. Then,


Let c1=1,p=2,c2=2L+σ2,c3=4σ2. By Theorem 3.2, we finished the proof.