Abstract
Neural Ordinary Differential Equation (Neural ODE) has been proposed as acontinuous approximation to the ResNet architecture. Some commonly usedregularization mechanisms in discrete neural networks (e.g. dropout, Gaussiannoise) are missing in current Neural ODE networks. In this paper, we propose anew continuous neural network framework called Neural Stochastic DifferentialEquation (Neural SDE) network, which naturally incorporates various commonlyused regularization mechanisms based on random noise injection. Our frameworkcan model various types of noise injection frequently used in discrete networksfor regularization purpose, such as dropout and additive/multiplicative noisein each block. We provide theoretical analysis explaining the improvedrobustness of Neural SDE models against input perturbations/adversarialattacks. Furthermore, we demonstrate that the Neural SDE network can achievebetter generalization than the Neural ODE and is more resistant to adversarialand nonadversarial input perturbations.
Quick Read (beta)
Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise
Abstract
Neural Ordinary Differential Equation (Neural ODE) has been proposed as a continuous approximation to the ResNet architecture. Some commonly used regularization mechanisms in discrete neural networks (e.g. dropout, Gaussian noise) are missing in current Neural ODE networks. In this paper, we propose a new continuous neural network framework called Neural Stochastic Differential Equation (Neural SDE) network, which naturally incorporates various commonly used regularization mechanisms based on random noise injection. Our framework can model various types of noise injection frequently used in discrete networks for regularization purpose, such as dropout and additive/multiplicative noise in each block. We provide theoretical analysis explaining the improved robustness of Neural SDE models against input perturbations/adversarial attacks. Furthermore, we demonstrate that the Neural SDE network can achieve better generalization than the Neural ODE and is more resistant to adversarial and nonadversarial input perturbations.
1 Introduction
Residual neural networks (ResNet) [1] are composed of multiple residual blocks transforming the hidden states according to:
$${\bm{h}}_{n+1}={\bm{h}}_{n}+\bm{f}({\bm{h}}_{n};{\bm{w}}_{n}),$$  (1) 
where ${\bm{h}}_{n}$ is the input to the $n$th layer and $\bm{f}({\bm{h}}_{n};{\bm{w}}_{n})$ is a nonlinear function parameterized by ${\bm{w}}_{n}$. Recently, a continuous approximation to the ResNet architecture has been proposed [2], where the evolution of the hidden state ${\bm{h}}_{t}$ can be described as a dynamic system obeying the equation:
$${\bm{h}}_{t}={\bm{h}}_{s}+{\int}_{s}^{t}\bm{f}({\bm{h}}_{\tau},\tau ;\bm{w})\text{dif}\tau ,$$  (2) 
where $\bm{f}({\bm{h}}_{\tau},\tau ;\bm{w})$ is the continuous form of the nonlinear function $\bm{f}({\bm{h}}_{n};{\bm{w}}_{n})$; ${\bm{h}}_{s}$ and ${\bm{h}}_{t}$ are hidden states at two different time $s\ne t$. A standard ODE solver can be used to solve all the hidden states and final states (output from the neural network), starting from an initial state (input to the neural network). The continuous neural network described in (2) exhibits several advantages over its discrete counterpart described in (1), in terms of memory efficiency, parameter efficiency, explicit control of the numerical error of final output, etc.
One missing component in the current Neural ODE network is the various regularization mechanisms commonly employed in discrete neural networks. These regularization techniques have been demonstrated to be crucial in reducing generalization errors, and in improving the robustness of neural networks to adversarial attacks. Many of these regularization techniques are based on stochastic noise injection. For instance, dropout [3] is widely adopted to prevent overfitting; injecting Gaussian random noise during the forward propagation is effective in improving generalization [4, 5] as well as robustness to adversarial attacks [6, 7]. However, these regularization methods in discrete neural networks are not directly applicable to Neural ODE network, because Neural ODE network is a deterministic system.
Our work attempts to incorporate the abovementioned stochastic noise injection based regularization mechanisms to the current Neural ODE network, to improve the generalization ability and the robustness of the network. In this paper, we propose a new continuous neural network framework called Neural Stochastic Differential Equation (Neural SDE) network, which models stochastic noise injection by stochastic differential equations (SDE). In this new framework, we can employ existing techniques from the stability theory of SDE to study the robustness of neural networks. Our results provide theoretical insights to understanding why introducing stochasticity during neural network training and testing leads to improved robustness against adversarial attacks. Furthermore, we demonstrate that, by incorporating the noise injection regularization mechanism to the continuous neural network, we can reduce overfitting and achieve lower generalization error. For instance, on the CIFAR10 dataset, we observe that the new Neural SDE can improve the test accuracy of the Neural ODE from 81.63% to 84.55%, with other factors unchanged. Our contributions can be summarized as follows:

•
We propose a new Stochastic Differential Equation (SDE) framework to incorporate randomness in continuous neural networks. The proposed random noise injection can be used as a dropin component in any continuous neural networks. Our Neural SDE framework can model various types of noises widely used for regularization purpose in discrete networks, such as dropout (Bernoulli type) and Gaussian noise.

•
Training the new SDE network requires developing different backpropagation approach from the Neural ODE network. We develop a new efficient backpropagation method to calculate the gradient, and to train the Neural SDE network in a scalable way. The proposed method has its roots in stochastic control theory.

•
We carry out a theoretical analysis of the stability conditions of the Neural SDE network, to prove that the randomness introduced in the Neural SDE network can stabilize the dynamical system, which helps improve the robustness and generalization ability of the neural network.

•
We verify by numerical experiments that stochastic noise injection in the SDE network can successfully regularize the continuous neural network models, and the proposed Neural SDE network achieves better robustness and improves generalization performance.
Notations:
Throughout this paper, we use $\bm{h}\in {\mathbb{R}}^{n}$ to denote the hidden states in a neural network, where ${\bm{h}}_{0}=\bm{x}$ is the input (also called initial condition) and $y$ is the label. The residual block with parameters $\bm{w}\in {\mathbb{R}}^{d}$ can be written as a nonlinear transform $\bm{f}({\bm{h}}_{\tau},\tau ;\bm{w})$. We assume the integration is always taken from $0$ to $T$. ${\mathbf{B}}_{t}\in {\mathbb{R}}^{m}$ is $m$dimensional Brownian motion. $\bm{G}({\bm{h}}_{\tau},\tau ;\bm{v})\in {\mathbb{R}}^{n\times m}$ is the diffusion matrix parameterized by $\bm{v}$. Unless stated explicitly, we use $\parallel \cdot \parallel $ to represent ${\mathrm{\ell}}_{2}$norm for vector and Frobenius norm for matrix.
2 Related work
Our work is inspired by the success of the recent Neural ODE network, and we seek to improve the generalization and robustness of Neural ODE, by adding regularization mechanisms crucial to the success of discrete networks. Regularization mechanisms such as dropout cannot be easily incorporated in the Neural ODE due to its deterministic nature.
Neural ODE
The basic idea of Neural ODE is discussed in the previous section, here we briefly review relevant literature. The idea of formulating ResNet as a dynamic system was discussed in [8]. A framework was proposed to link existing deep architectures with discretized numerical ODE solvers [9], and was shown to be parameter efficient. These networks adopt layerwise architecture – each layer is parameterized by different independent weights. The Neural ODE model [2] computes hidden states in a different way: it directly models the dynamics of hidden states by an ODE solver, with the dynamics parameterized by a shared model. A memory efficient approach to compute the gradients by adjoint methods was developed, making it possible to train large, multiscale generative networks [10, 11]. Our work can be regarded as an extension of this framework, with the purpose of incorporating a variety of noiseinjection based regularization mechanisms. Stochastic differential equation in the context of neural network has been studied before, focusing either on understanding how dropout shapes the loss landscape [12], or on using stochastic differential equation as a universal function approximation tool to learn the solution of high dimensional PDEs [13]. Instead, our work tries to explain why adding random noise boosts the stability of deep neural networks, and demonstrates the improved generalization and robustness.
Noisy Neural Networks
Adding random noise to different layers is a technique commonly employed in training neural networks. Dropout [3] randomly disables some neurons to avoid overfitting, which can be viewed as multiplying hidden states with Bernoulli random variables. Stochastic depth neural network [14] randomly drops some residual blocks of residual neural network during training time. Another successful regularization for ResNet is ShakeShake regularization [15], which sets a binary random variable to randomly switch between two residual blocks during training. More recently, dropblock [16] was designed specifically for convolutional layers: unlike dropout, it drops some continuous regions rather than sparse points to hidden states. All of the above regularization techniques are proposed to improve generalization performance. One common characteristic of them is that they fix the network during testing time. There is another line of research that focuses on improving robustness to perturbations/adversarial attacks by noise injection. Among them, random selfensemble [6, 7] adds Gaussian noise to hidden states during both training and testing time. In training time, it works as a regularizer to prevent overfitting; in testing time, the random noise is also helpful, which will be explained in this paper.
3 Neural Stochastic Differential Equation
In this section, we first introduce our proposed Neural SDE to improve the robustness of Neural ODE. Informally speaking, Neural SDE can be viewed as using randomness as a dropin augmentation for Neural ODE, and it can include some widely used randomization layers such as dropout and Gaussian noise layer [6]. However, solving Neural SDE is nontrivial, we derive the gradients of loss over model weights. Finally we theoretically analyze the stability conditions of Neural SDE.
Before delving into the multidimensional SDE, let’s first look at a 1d toy example to see how SDE can solve the instability issue of ODE. Suppose we have a simple SDE, $\text{dif}{x}_{t}={x}_{t}\text{dif}t+\sigma {x}_{t}\text{dif}{B}_{t}$ with ${B}_{t}$ be the standard Brownian motion. We provide a numerical simulation in Figure 1 for ${x}_{t}$ with different $\sigma $.
When we set $\sigma =0$, SDE becomes ODE $\text{dif}{x}_{t}={x}_{t}\text{dif}t$ and ${x}_{t}={c}_{0}{e}^{t}$ where ${c}_{0}$ is an integration constant. If ${c}_{0}\ne 0$ we can see that ${x}_{t}\to \pm \mathrm{\infty}$. Furthermore, a small perturbation in ${x}_{t}$ will be amplified through $t$. This clearly shows instability of ODE. On the other hand, if we instead make $\sigma >1$ (the system is SDE), we have ${x}_{t}={c}_{0}\mathrm{exp}\left((1{\sigma}^{2})t+\sigma {B}_{t}\right)\stackrel{\mathrm{a}.\mathrm{s}.}{\to}0$.
The toy example in Figure 1 reveals that the behavior of solution paths can change significantly after adding a stochastic term. This example is inspiring because we can control the impact of perturbations on the output by adding a stochastic term to neural networks.
Figure 2 shows a sample Neural SDE model architecture, and it is the one used in the experiment. It consists of three parts, the first part is a single convolution block, followed by a Neural SDE network (we will explain the detail of Neural SDE in Section 3.1) and lastly the linear classifier. We put most of the trainable parameters into the second part (Neural SDE), whereas the first/third parts are mainly for increasing/reducing the dimension as desired. Recall that both Neural ODE and SDE are dimension preserving.
3.1 Modeling randomness in neural networks
In the Neural ODE system (2), a slightly perturbed input state will be amplified in deep layers (as shown in Figure 1) which makes the system unstable to input perturbation and prone to overfitting. Randomness is an important component in discrete networks (e.g., dropout for regularization) to tackle this issue, however to our knowledge, there is no existing work concerning adding randomness in the continuous neural networks. And it is nontrivial to encode randomness in continuous neural networks, such as Neural ODE, as we need to consider how to add randomness so that to guarantee the robustness, and how to solve the continuous system efficiently. To solve these challenges, motivated by [9, 12], we propose to add a single diffusion term into Neural ODE as:
$$\text{dif}{\bm{h}}_{t}=\bm{f}({\bm{h}}_{t},t;\bm{w})\text{dif}t+\bm{G}({\bm{h}}_{t},t;\bm{v})\text{dif}{\mathbf{B}}_{t},$$  (3) 
where ${\mathbf{B}}_{t}$ is the standard Brownian motion [17], which is a continuous time stochastic process such that ${\mathbf{B}}_{t+s}{\mathbf{B}}_{s}$ follows Gaussian with mean $0$ and variance $t$; $\bm{G}({\bm{h}}_{t},t;\bm{v})$ is a transformation parameterized by $\bm{v}$. This formula is quite general, and can include many existing randomness injection models with residual connections under different forms of $\bm{G}({\bm{h}}_{t},t;\bm{v})$. As examples, we briefly list some of them below.
Gaussian noise injection:
Consider a simple example in (3) when $\bm{G}({\bm{h}}_{t},t;\bm{v})$ is a diagonal matrix, and we can model both additive and multiplicative noise as
additive:  $\text{dif}{\bm{h}}_{t}=\bm{f}({\bm{h}}_{t},t;\bm{w})\text{dif}t+\mathbf{\Sigma}(t)\text{dif}{\mathbf{B}}_{t}$  (4)  
multiplicative:  $\text{dif}{\bm{h}}_{t}=\bm{f}({\bm{h}}_{t},t;\bm{w})\text{dif}t+\mathbf{\Sigma}({\bm{h}}_{t},t)\text{dif}{\mathbf{B}}_{t},$ 
where $\mathbf{\Sigma}(t)$ is a diagonal matrix and its diagonal elements control the variance of the noise added to hidden states. This can be viewed as a continuous approximation of noise injection techniques in discrete neural network. For example, the discrete version of the additive noise can be written as
$${\bm{h}}_{n+1}={\bm{h}}_{n}+\bm{f}({\bm{h}}_{n};{\bm{w}}_{n})+{\mathbf{\Sigma}}_{n}{\bm{z}}_{n},\text{with}{\mathbf{\Sigma}}_{n}={\sigma}_{n}\bm{I},{\bm{z}}_{n}\stackrel{\text{i.i.d.}}{\sim}\mathcal{N}(0,1),$$  (5) 
which injects Gaussian noise after each residual block. It has been shown that injecting small Gaussian noise can be viewed as a regularization in neural networks [4, 5]. Furthermore, [6, 7] recently showed that adding a slightly larger noise in one or all residual blocks can improve the adversarial robustness of neural networks. We will provide the stability analysis of (3) in Section 3.3, which provides a theoretical explanation towards the robustness of Neural SDE.
Dropout:
Our framework can also model the dropout layer which randomly disables some neurons in the residual blocks. Let us see how to unify dropout under our Neural SDE framework. First we notice that in the discrete case
$${\bm{h}}_{n+1}={\bm{h}}_{n}+\bm{f}({\bm{h}}_{n};{\bm{w}}_{n})\odot \frac{{\bm{\gamma}}_{n}}{p}={\bm{h}}_{n}+\bm{f}({\bm{h}}_{n};{\bm{w}}_{n})+\bm{f}({\bm{h}}_{n};{\bm{w}}_{n})\odot (\frac{{\bm{\gamma}}_{n}}{p}\bm{I}),$$  (6) 
where ${\bm{\gamma}}_{n}\stackrel{\text{i.i.d.}}{\sim}\mathcal{B}(1,p)$ and $\odot $ indicates the Hadamard product. Note that we divide ${\gamma}_{n}$ by $p$ in (6) to maintain the same expectation. Furthermore, we have
$$\frac{{\bm{\gamma}}_{n}}{p}I=\sqrt{\frac{1p}{p}}\cdot \overline{)\sqrt{\frac{p}{1p}}\left(\frac{{\gamma}_{n}}{p}\bm{I}\right)}\approx \sqrt{\frac{1p}{p}}{\bm{z}}_{n},{\bm{z}}_{n}\stackrel{\text{i.i.d.}}{\sim}\mathcal{N}(0,1).$$  (7) 
The boxed part above is approximated by standard normal distribution (by matching the first and second order moment). The final SDE with dropout can be obtained by combining (6) with (7)
$$\text{dif}{\bm{h}}_{t}=\bm{f}({\bm{h}}_{t},t;\bm{w})\text{dif}t+\sqrt{\frac{1p}{p}}\bm{f}({\bm{h}}_{t},t;\bm{w})\odot \text{dif}{\mathbf{B}}_{t}.$$  (8) 
Others:
3.2 Backpropagating through SDE integral
To optimize the parameters $\bm{w}$, we need to backpropagate the Neural SDE system. A straightforward solution is to rely on the autograd method derived from chain rule. However, for Neural SDE the chain can be fairly long. If SDE solver discretizes the range $[{t}_{0},{t}_{1}]$ to $N$ intervals, then the chain has $\mathcal{O}(N)$ nodes and the memory cost is $\mathcal{O}(N)$. One challenging part of backpropagation for Neural SDE is to calculate the gradient through SDE solver which could have high memory cost. To solve this issue, we first calculate the expected loss conditioning on the initial value ${\bm{h}}_{0}$, denoted as $L=\mathbb{E}[\mathrm{\ell}({\bm{h}}_{{t}_{1}}){\bm{h}}_{0}]$. Then our goal is to calculate $\frac{\partial L}{\partial \bm{w}}$. In fact, we have the following theorem (also called pathwise gradient [18, 19]).
Theorem 3.1.
For continuously differentiable loss $\mathrm{\ell}\mathit{}\mathrm{(}{\mathbf{h}}_{{t}_{\mathrm{1}}}\mathrm{)}$, we can obtain an unbiased gradient estimator as
$$\widehat{\frac{\partial L}{\partial \bm{w}}}=\frac{\partial \mathrm{\ell}({\bm{h}}_{{t}_{1}})}{\partial \bm{w}}=\frac{\partial \mathrm{\ell}({\bm{h}}_{{t}_{1}})}{\partial {\bm{h}}_{{t}_{1}}}\cdot \frac{\partial {\bm{h}}_{{t}_{1}}}{\partial \bm{w}}.$$  (9) 
Moreover, if we define ${\mathbf{\beta}}_{t}\mathrm{=}\frac{\mathrm{\partial}\mathit{}{\mathbf{h}}_{t}}{\mathrm{\partial}\mathit{}\mathbf{w}}$, then ${\mathbf{\beta}}_{t}$ follows another SDE
$$\text{dif}{\bm{\beta}}_{t}=\left(\frac{\partial \bm{f}({\bm{h}}_{t},t;\bm{w})}{\partial \bm{w}}+\frac{\partial \bm{f}({\bm{h}}_{t},t;\bm{w})}{\partial {\bm{h}}_{t}}{\bm{\beta}}_{t}\right)\text{dif}t+\left(\frac{\partial \bm{G}({\bm{h}}_{t},t;\bm{w})}{\partial \bm{w}}+\frac{\partial \bm{G}({\bm{h}}_{t},t;\bm{w})}{\partial {\bm{h}}_{t}}{\bm{\beta}}_{t}\right)\text{dif}{\mathbf{B}}_{t}.$$  (10) 
It is easy to check that if $\mathbf{G}\mathrm{\equiv}\mathrm{0}$, then our MonteCarlo gradient estimator (9) falls back to the exact gradient by backpropagation.
Similar to the adjoint method in Neural ODE, we will solve (10) jointly with the original SDE dynamics (3), this process can be done iteratively without memorizing the middle states, which makes it more memory efficient than autograd ($\mathcal{O}(1)$ v.s. $\mathcal{O}(N)$ memory, $N$ is the number of steps determined by SDE solver).
3.3 Robustness of Neural SDE
In this section, we theoretically analyze the stability of Neural SDE, showing that the randomness term can indeed improve the robustness of the model against small input perturbation. This also explains why noise injection can improve the robustness in discrete networks, which has been observed in literature [6, 7]. First we need to show the existence and uniqueness of solution to (3), we pose following assumptions on drift $\bm{f}$ and diffusion $\bm{G}$.
Assumption 1.
$\bm{f}$ and $\mathbf{G}$ are at most linear, i.e. $\mathrm{\parallel}\mathbf{f}\mathit{}\mathrm{(}\mathbf{x}\mathrm{,}t\mathrm{)}\mathrm{\parallel}\mathrm{+}\mathrm{\parallel}\mathbf{G}\mathit{}\mathrm{(}\mathbf{x}\mathrm{,}t\mathrm{)}\mathrm{\parallel}\mathrm{\le}{c}_{\mathrm{1}}\mathit{}\mathrm{(}\mathrm{1}\mathrm{+}\mathrm{\parallel}\mathbf{x}\mathrm{\parallel}\mathrm{)}$ for ${c}_{\mathrm{1}}\mathrm{>}\mathrm{0}$, $\mathrm{\forall}\mathbf{x}\mathrm{\in}{\mathrm{R}}^{n}$ and $t\mathrm{\in}{\mathrm{R}}^{\mathrm{+}}$.
Assumption 2.
$\bm{f}$ and $\mathbf{G}$ are ${c}_{\mathrm{2}}$Lipschitz: $\mathrm{\parallel}\mathbf{f}\mathit{}\mathrm{(}\mathbf{x}\mathrm{,}t\mathrm{)}\mathrm{}\mathbf{f}\mathit{}\mathrm{(}\mathbf{y}\mathrm{,}t\mathrm{)}\mathrm{\parallel}\mathrm{+}\mathrm{\parallel}\mathbf{G}\mathit{}\mathrm{(}\mathbf{x}\mathrm{,}t\mathrm{)}\mathrm{}\mathbf{G}\mathit{}\mathrm{(}\mathbf{y}\mathrm{,}t\mathrm{)}\mathrm{\parallel}\mathrm{\le}{c}_{\mathrm{2}}\mathit{}\mathrm{\parallel}\mathbf{x}\mathrm{}\mathbf{y}\mathrm{\parallel}$ for ${c}_{\mathrm{2}}\mathrm{>}\mathrm{0}$, $\mathrm{\forall}\mathbf{x}\mathrm{,}\mathbf{y}\mathrm{\in}{\mathrm{R}}^{n}$ and $t\mathrm{\in}{\mathrm{R}}^{\mathrm{+}}$.
Based on the above assumptions, we can show that the SDE (3) has a unique solution [17]. We remark that assumption on $\bm{f}$ is quite natural and is also enforced on the original Neural ODE model [2]; as to diffusion matrix $\bm{G}$, we have seen that for dropout, Gaussian noise injection and other random models, both assumptions are automatically satisfied as long as $\bm{f}$ possesses the same regularities.
We analyze the dynamics of perturbation. Our analysis applies not only to the Neural SDE model but also to Neural ODE model, by setting the diffusion term $\bm{G}$ to zero. First of all, we consider initializing our Neural SDE (3) at two slightly different values ${\bm{h}}_{0}$ and ${\bm{h}}_{0}^{e}={\bm{h}}_{0}+{\bm{\epsilon}}_{0}$, where ${\bm{\epsilon}}_{0}$ is the perturbation for ${\bm{h}}_{0}$ with $\parallel {\bm{\epsilon}}_{0}\parallel \le \delta $. So, under the new perturbed initialization ${\bm{h}}_{0}^{e}$, the hidden state at time $t$ follows the same SDE in (3),
$$\text{dif}{\bm{h}}_{t}^{e}=\bm{f}({\bm{h}}_{t}^{e},t;\bm{w})\text{dif}t+\bm{G}({\bm{h}}_{t}^{e},t;\bm{v})\text{dif}{\mathbf{B}}_{t}^{\prime},\text{with}{\bm{h}}_{0}^{e}={\bm{h}}_{0}+{\bm{\epsilon}}_{0},$$  (11) 
where ${\mathbf{B}}_{t}^{\prime}$ is Brownian motions for the SDE associated with initialization ${\bm{h}}_{0}^{e}$. Then it is natural to analyze how the perturbation ${\bm{\epsilon}}_{t}={\bm{h}}_{t}^{e}{\bm{h}}_{t}$ evolves in the long run. Subtracting (3) from (11)
$\text{dif}{\bm{\epsilon}}_{t}$  $=\left[\bm{f}({\bm{h}}_{t}^{e},t;\bm{w})\bm{f}({\bm{h}}_{t},t;\bm{w})\right]\text{dif}t+\left[\bm{G}({\bm{h}}_{t}^{e},t;\bm{v})\bm{G}({\bm{h}}_{t},t;\bm{v})\right]\text{dif}{\mathbf{B}}_{t}$  (12)  
$={\bm{f}}_{\mathrm{\Delta}}({\bm{\epsilon}}_{t},t;\bm{w})\text{dif}t+{\bm{G}}_{\mathrm{\Delta}}({\bm{\epsilon}}_{t},t;\bm{v})\text{dif}{\mathbf{B}}_{t}.$ 
Here we made an implicit assumption that the Brownian motions ${\mathbf{B}}_{t}$ and ${\mathbf{B}}_{t}^{\prime}$ have the same sample path for both initialization ${\bm{h}}_{0}$ and ${\bm{h}}_{0}^{e}$, i.e. ${\mathbf{B}}_{t}={\mathbf{B}}_{t}^{\prime}$ w.p.1. In other words, we focus on the difference of two random processes ${\bm{h}}_{t}$ and ${\bm{h}}_{t}^{e}$ driven by the same underlying Brownian motion. So it is valid to subtract the diffusion terms.
An important property of (12) is that it admits a trivial solution ${\bm{\epsilon}}_{t}\equiv \mathrm{\U0001d7ce}$, $\forall t\in {\mathbb{R}}^{+}$ and $\bm{w}\in {\mathbb{R}}^{d}$. We show that both the drift ($\bm{f}$) and diffusion ($\bm{G}$) are zero under this solution:
${\bm{f}}_{\mathrm{\Delta}}(\mathrm{\U0001d7ce},t;\bm{w})$  $=\bm{f}({\bm{h}}_{t}+\mathrm{\U0001d7ce},t;\bm{w})\bm{f}({\bm{h}}_{t},t;\bm{w})=0,$  (13)  
${\bm{G}}_{\mathrm{\Delta}}(\mathrm{\U0001d7ce},t;\bm{v})$  $=\bm{G}({\bm{h}}_{t}+\mathrm{\U0001d7ce},t;\bm{w})\bm{G}({\bm{h}}_{t},t;\bm{w})=0.$ 
The implication of zero solution is clear: for a neural network, if we do not perturb the input data, then the output will never change. However, the solution ${\bm{\epsilon}}_{t}=\mathrm{\U0001d7ce}$ can be highly unstable, in the sense that for an arbitrarily small perturbation ${\bm{\epsilon}}_{0}\ne \mathrm{\U0001d7ce}$ at initialization, the change of output ${\bm{\epsilon}}_{T}$ can be arbitrarily bad. On the other hand, as shown below, by choosing the diffusion term $\bm{G}$ properly, we can always control ${\bm{\epsilon}}_{t}$ within a small range.
In general, we cannot get the closed form solution to a multidimensional SDE but we can still analyze the asymptotic stability through the dynamics $\bm{f}$ and $\bm{G}$. This is essentially an extension of Lyapunov stability theory to a stochastic system. First we define the notion of stability in the stochastic case. Let $(\mathrm{\Omega},\mathcal{F},P)$ be a complete probability space with filtration ${\{{\mathcal{F}}_{t}\}}_{t\ge 0}$ and ${\mathbf{B}}_{t}$ be an $m$dimensional Brownian motion defined in the probability space, we consider the SDE in Eq. (12) with initial value ${\bm{\epsilon}}_{0}$
$$\text{dif}{\bm{\epsilon}}_{t}={\bm{f}}_{\mathrm{\Delta}}({\bm{\epsilon}}_{t},t)\text{dif}t+{\bm{G}}_{\mathrm{\Delta}}({\bm{\epsilon}}_{t},t)\text{dif}{\mathbf{B}}_{t},$$  (14) 
For simplicity we dropped the dependency on parameters $\bm{w}$ and $\bm{v}$. We further assume ${\bm{f}}_{\mathrm{\Delta}}:{\mathbb{R}}^{n}\times {\mathbb{R}}_{+}\mapsto {\mathbb{R}}^{n}$ and ${\bm{G}}_{\mathrm{\Delta}}:{\mathbb{R}}^{n}\times {\mathbb{R}}_{+}\mapsto {\mathbb{R}}^{n\times m}$ are both Borel measurable. We can show that if assumptions (1) and (2) hold for $\bm{f}$ and $\bm{G}$, then they hold for ${\bm{f}}_{\mathrm{\Delta}}$ and ${\bm{G}}_{\mathrm{\Delta}}$ as well (see Lemma A.1 in Appendix), and we know the SDE (14) allows a unique solution ${\bm{\epsilon}}_{t}$. We have the following Lynapunov stability results from [20].
Definition 3.1 (Lyapunov stability of SDE).
The solution ${\mathbf{\epsilon}}_{t}\mathrm{=}\mathrm{0}$ of (14):

A.
is stochastically stable if for any $\alpha \in (0,1)$ and $r>0$, there exists a $\delta =\delta (\alpha ,r)>0$ such that $$ whenever $\parallel {\bm{\epsilon}}_{0}\parallel \le \delta $. Moreover, if for any $\alpha \in (0,1)$, there exists a $\delta =\delta (\alpha )>0$ such that $\mathrm{Pr}\{{lim}_{t\to \mathrm{\infty}}\parallel {\bm{\epsilon}}_{t}\parallel =0\}\ge 1\alpha $ whenever $\parallel {\bm{\epsilon}}_{0}\parallel \le \delta $, it is said to be stochastically asymptotically stable;

B.
is almost surely exponentially stable if $$ a.s.^{1}^{1} 1 “a.s.” is the abbreviation for “almost surely”. for all ${\bm{\epsilon}}_{0}\in {\mathbb{R}}^{n}$.
Note that for part A in Definition 3.1, it is hard to quantify how well the stability is and how fast the solution reaches equilibrium. In addition, under assumptions (1, 2), we have a straightforward result $\mathrm{Pr}\{{\bm{\epsilon}}_{t}\ne \mathrm{\U0001d7ce}\text{for all}t\ge 0\}=1$ whenever ${\bm{\epsilon}}_{0}\ne \mathrm{\U0001d7ce}$ as shown in Appendix (see Lemma A.2). That is, almost all the sample paths starting from a nonzero initialization can never reach zero due to Brownian motion. On the contrary, the almost sure exponentially stability result implies that almost all the sample paths of the solution will be close to zero exponentially fast. We present the following theorem from [20] on the almost sure exponentially stability.
Theorem 3.2.
[20] If there exists a nonnegative real valued function $V\mathit{}\mathrm{(}\mathbf{\epsilon}\mathrm{,}t\mathrm{)}$ defined on ${\mathrm{R}}^{n}\mathrm{\times}{\mathrm{R}}_{\mathrm{+}}$ that has continuous partial derivatives
$${V}_{1}(\bm{\epsilon},t)\u2254\frac{\partial V(\bm{\epsilon},t)}{\partial \bm{\epsilon}},{V}_{2}(\bm{\epsilon},t)\u2254\frac{\partial V(\bm{\epsilon},t)}{\partial t},{V}_{1,1}(\bm{\epsilon},t)\u2254\frac{{\partial}^{2}V(\bm{\epsilon},t)}{\partial \bm{\epsilon}\partial {\bm{\epsilon}}^{\top}}$$ 
and constants $p\mathrm{>}\mathrm{0}\mathrm{,}{c}_{\mathrm{1}}\mathrm{>}\mathrm{0}\mathrm{,}{c}_{\mathrm{2}}\mathrm{\in}\mathrm{R}\mathrm{,}{c}_{\mathrm{3}}\mathrm{\ge}\mathrm{0}$ such that the following inequalities hold:

1.
${c}_{1}{\parallel \bm{\epsilon}\parallel}^{p}\le V(\bm{\epsilon},t)$

2.
$\mathcal{L}V(\bm{\epsilon},t)={V}_{2}(\bm{\epsilon},t)+{V}_{1}(\bm{\epsilon},t){\bm{f}}_{\mathrm{\Delta}}(\bm{\epsilon},t)+\frac{1}{2}\mathrm{\U0001d5b3\U0001d5cb}[{\bm{G}}_{\mathrm{\Delta}}^{\top}(\bm{\epsilon},t){V}_{1,1}(\bm{\epsilon},t){\bm{G}}_{\mathrm{\Delta}}(\bm{\epsilon},t)]\le {c}_{2}V(\bm{\epsilon},t)$

3.
${\parallel {V}_{1}(\bm{\epsilon},t){\bm{G}}_{\mathrm{\Delta}}(\bm{\epsilon},t)\parallel}^{2}\ge {c}_{3}{V}^{2}(\bm{\epsilon},t)$
for all $\mathbf{\epsilon}\mathrm{\ne}\mathrm{0}$ and $t\mathrm{>}\mathrm{0}$. Then for all ${\mathbf{\epsilon}}_{\mathrm{0}}\mathrm{\in}{\mathrm{R}}^{n}$,
$$\underset{t\to \mathrm{\infty}}{lim\; sup}\frac{1}{t}\mathrm{log}\parallel {\bm{\epsilon}}_{t}\parallel \le \frac{{c}_{3}2{c}_{2}}{2p}\mathit{\hspace{1em}}a.s.$$  (15) 
In particular, if ${c}_{\mathrm{3}}\mathrm{\ge}\mathrm{2}\mathit{}{c}_{\mathrm{2}}$, the solution ${\mathbf{\epsilon}}_{t}\mathrm{\equiv}\mathrm{0}$ is almost surely exponentially stable.
We now consider a special case, when the noise is multiplicative $\bm{G}({\bm{h}}_{t},t)=\sigma \cdot {\bm{h}}_{t}$ and $m=1$. The corresponding SDE of perturbation ${\bm{\epsilon}}_{t}={\bm{h}}_{t}^{e}{\bm{h}}_{t}$ has the following form
$$\text{dif}{\bm{\epsilon}}_{t}={\bm{f}}_{\mathrm{\Delta}}({\bm{\epsilon}}_{t},t;\bm{w})\text{dif}t+\sigma \cdot {\bm{\epsilon}}_{t}\text{dif}{\mathbf{B}}_{t}.$$  (16) 
Note that for the deterministic case of (16) by setting $\sigma \equiv 0$, the solution may not be stable in certain cases (see Figure 1). Whereas for general cases when $\sigma >0$, following corollary claims that by setting $\sigma $ properly, we will achieve an (almost surely) exponentially stable system.
Corollary 3.2.1.
For (16), if $\mathbf{f}\mathit{}\mathrm{(}{\mathbf{h}}_{t}\mathrm{,}t\mathrm{;}\mathbf{w}\mathrm{)}$ is $L$Lipschtiz continuous w.r.t. ${\mathbf{h}}_{t}$, then (16) has a unique solution with the property $\underset{t\mathrm{\to}\mathrm{\infty}}{\mathrm{lim\; sup}}\mathit{}\frac{\mathrm{1}}{t}\mathit{}\mathrm{log}\mathit{}\mathrm{\parallel}{\mathbf{\epsilon}}_{t}\mathrm{\parallel}\mathrm{\le}\mathrm{}\mathrm{(}\frac{{\sigma}^{\mathrm{2}}}{\mathrm{2}}\mathrm{}L\mathrm{)}$ almost surely for any ${\mathbf{\epsilon}}_{\mathrm{0}}\mathrm{\in}{\mathrm{R}}^{n}$. In particular, if ${\sigma}^{\mathrm{2}}\mathrm{>}\mathrm{2}\mathit{}L$, the solution ${\mathbf{\epsilon}}_{t}\mathrm{=}\mathrm{0}$ is almost surely exponentially stable.
4 Experimental Results
In this section we show the effectiveness of our Neural SDE framework in terms of generalization, nonadversarial robustness and adversarial robustness. We use the SDE model architecture illustrated in Figure 2 during the experiment. Throughout our experiments, we set $\bm{f}(\cdot )$ to be a neural network with several convolution blocks. As to $\bm{G}(\cdot )$ we have the following choices:

•
Neural ODE, this can be done by dropping the diffusion term $\bm{G}({\bm{h}}_{t},t;\bm{v})=\mathrm{\U0001d7ce}$.

•
Additive noise, when the diffusion term is independent of ${\bm{h}}_{t}$, here we simply set it to be diagonal $\bm{G}({\bm{h}}_{t},t;\bm{v})={\sigma}_{t}\bm{I}$.

•
Multiplicative noise, when the diffusion term is proportional to ${\bm{h}}_{t}$, or $\bm{G}({\bm{h}}_{t},t;\bm{w})={\sigma}_{t}{\bm{h}}_{t}$.

•
Dropout noise, when the diffusion term is proportional to the drift term $\bm{f}({\bm{h}}_{t},t;\bm{w})$, i.e. $\bm{G}({\bm{h}}_{t},t;\bm{v})={\sigma}_{t}\text{diag}\{\bm{f}({\bm{h}}_{t},t;\bm{w})\}$.
Note the last three are our proposed Neural SDE with different types of randomness as explained in Section 3.1.
4.1 Generalization Performance
In the first experiment, we show small noise helps generalization. However, note that our noise injection is different from randomness layer in the discrete case, for instance, dropout layer adds Bernoulli noise at training time, but the layer are then fixed at testing time; whereas our Neural SDE model keeps randomness at testing time and takes the average of multiple forward propagation.
As for datasets, we choose CIFAR10, STL10 and TinyImageNet^{2}^{2} 2 Downloaded from https://tinyimagenet.herokuapp.com/ to include various sizes and number of classes. The experimental results are shown in Table 1. We see that for all datasets, Neural SDE consistently outperforms ODE, and the reason is that adding moderate noise to the models at training time can act as a regularizer and thus improves testing accuracy. Based upon that, if we further keep testing time noise and ensemble the outputs, we will obtain even better results.
Data  Model size  [email protected] — w/o TTN  [email protected] — w/ TTN  

ODE  Additive  Multiplicative  Dropout  ODE  Additive  Multiplicative  Dropout  
CIFAR10  115 K  81.63  83.65  83.26  83.60  –  83.89  83.76  84.55 
STL10  2.44 M  58.03  61.23  60.54  61.26  –  62.11  62.58  62.13 
TinyImageNet  2.49 M  45.19  45.25  46.94  47.04  –  45.39  46.65  47.81 
4.2 Improved nonadversarial robustness
In this experiment, we aim at evaluating the robustness of models under nonadversarial corruptions following the idea of [21]. The corrupted datasets contain tens of defects in photography including motion blur, Gaussian noise, fog etc. For each noise type, we run Neural ODE and Neural SDE with dropout noise, and gather the testing accuracy. The final results are reported by mean accuracy (mAcc) in Table 2 by changing the level of corruption. Both models are trained on completely clean data, which means the corrupted images are not visible to them during the training stage, nor could they augment the training set with the same types of corruptions. From the table, we can see that Neural SDE performs better than Neural ODE in 8 out of 10 cases. For the rest two, both ODE and SDE are performing very close. This shows that our proposed Neural SDE can improve the robustness of Neural ODE under nonadversarial corrupted data.
Data  Noise type  mild corrupt $\leftarrow $ Accuracy $\to $ severe corrupt  
Level 1  Level 2  Level 3  Level 4  Level 5  
CIFAR10C^{†}  ODE  75.89  70.59  66.52  60.91  53.02 
Dropout  77.02  71.58  67.21  61.61  53.81  
Dropout+TTN  79.07  73.98  69.74  64.19  55.99  
TinyImageNetC^{†}  ODE  23.01  19.18  15.20  12.20  9.88 
Dropout  22.85  18.94  14.64  11.54  9.09  
Dropout+TTN  23.84  19.89  15.28  12.08  9.44 

†
Downloaded from https://github.com/hendrycks/robustness
4.3 Improved adversarial robustness
Next, we consider the performance of Neural SDE models under adversarial perturbation. Clearly, this scenario is strictly harder than previous cases: by design, the adversarial perturbations are guaranteed to be the worst case within a small neighborhood (ignoring the suboptimality of optimization algorithms) crafted through constrained loss maximization procedure, so it represents the worst case performance. In our experiment, we adopt multistep ${\mathrm{\ell}}_{\mathrm{\infty}}$PGD attack [22], although other strong whitebox attacks such as C&W [23] are also suitable. The experimental results are shown in Figure 3. As we can see both Neural SDE with multiplicative noise and dropout noise are more resistant to adversarial attack than Neural ODE, and dropout noise outperforms multiplicative noise.
4.4 Visualizing the perturbations of hidden states
In this experiment, we take a look at the perturbation ${\bm{\epsilon}}_{t}={\bm{h}}_{t}^{e}{\bm{h}}_{t}$ at any time $t$. Recall the 1d toy example in Figure 1, we can observe that the perturbation at time $t$ can be well suppressed by adding a strong diffusion term, which is also confirmed by theorem. However, it is still questionable whether the same phenomenon also exists in deep neural network since we cannot add very large noise to the network during training or testing time. If the noise is too large, it will also remove all useful features. Thus it becomes important to make sure that this will not happen to our models. To this end we first sample an input $\bm{x}$ from CIFAR10 and gather all the hidden states ${\bm{h}}_{t}$ at time $t=[0,\mathrm{\Delta}t,2\mathrm{\Delta}t,\mathrm{\dots},N\mathrm{\Delta}t]$. Then we perform regular PGD attack [22] and find the perturbation ${\bm{\delta}}_{x}$ such that ${\bm{x}}_{\mathrm{adv}}=\bm{x}+{\bm{\delta}}_{x}$ is an adversarial image, and feed the new data ${\bm{x}}_{\mathrm{adv}}$ into network again so we get ${\bm{h}}_{t}^{e}$ at the same time stamps as ${\bm{h}}_{t}$. Finally we plot the error ${\bm{\epsilon}}_{t}={\bm{h}}_{t}^{e}{\bm{h}}_{t}$ w.r.t. time $t$ (also called “network depth”), shown in Figure 4. We can observe that by adding a diffusion term (dropoutstyle noise), the error accumulates much slower than ordinary Neural ODE model.
5 Conclusion
To conclude, we introduce the Neural SDE model which can stabilize the prediction of Neural ODE by injecting stochastic noise. Our model can achieve better generalization and improve the robustness to both adversarial and nonadversarial noises.
Acknowledgement
We acknowledge the support by NSF IIS1719097, Intel, Google Cloud and AITRICS.
References
 [1] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
 [2] Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In Advances in Neural Information Processing Systems, pages 6572–6583, 2018.
 [3] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958, 2014.
 [4] Chris M Bishop. Training with noise is equivalent to tikhonov regularization. Neural computation, 7(1):108–116, 1995.
 [5] Guozhong An. The effects of adding noise during backpropagation training on a generalization performance. Neural computation, 8(3):643–674, 1996.
 [6] Xuanqing Liu, Minhao Cheng, Huan Zhang, and ChoJui Hsieh. Towards robust neural networks via random selfensemble. In Proceedings of the European Conference on Computer Vision (ECCV), pages 369–385, 2018.
 [7] Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness to adversarial examples with differential privacy. arXiv preprint arXiv:1802.03471, 2018.
 [8] Weinan E. A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics, 5(1):1–11, 2017.
 [9] Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. In International Conference on Machine Learning, pages 3282–3291, 2018.
 [10] Lynton Ardizzone, Jakob Kruse, Sebastian Wirkert, Daniel Rahner, Eric W Pellegrini, Ralf S Klessen, Lena MaierHein, Carsten Rother, and Ullrich Köthe. Analyzing inverse problems with invertible neural networks. arXiv preprint arXiv:1808.04730, 2018.
 [11] Will Grathwohl, Ricky TQ Chen, Jesse Betterncourt, Ilya Sutskever, and David Duvenaud. Ffjord: Freeform continuous dynamics for scalable reversible generative models. arXiv preprint arXiv:1810.01367, 2018.
 [12] Qi Sun, Yunzhe Tao, and Qiang Du. Stochastic training of residual networks: a differential equation viewpoint. arXiv preprint arXiv:1812.00174, 2018.
 [13] Maziar Raissi. Forwardbackward stochastic neural networks: Deep learning of highdimensional partial differential equations. arXiv preprint arXiv:1804.07010, 2018.
 [14] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European conference on computer vision, pages 646–661. Springer, 2016.
 [15] Xavier Gastaldi. Shakeshake regularization. arXiv preprint arXiv:1705.07485, 2017.
 [16] Golnaz Ghiasi, TsungYi Lin, and Quoc V Le. Dropblock: A regularization method for convolutional networks. In Advances in Neural Information Processing Systems, pages 10727–10737, 2018.
 [17] Bernt Øksendal. Stochastic differential equations. In Stochastic differential equations, pages 65–84. Springer, 2003.
 [18] Jichuan Yang and Harold J Kushner. A monte carlo method for sensitivity analysis and parametric optimization of nonlinear stochastic systems. SIAM journal on control and optimization, 29(5):1216–1249, 1991.
 [19] Emmanuel Gobet and Rémi Munos. Sensitivity analysis using itô–malliavin calculus and martingales, and application to stochastic optimal control. SIAM Journal on control and optimization, 43(5):1676–1713, 2005.
 [20] Xuerong Mao. Stochastic differential equations and applications. Elsevier, 2007.
 [21] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261, 2019.
 [22] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017.
 [23] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), pages 39–57. IEEE, 2017.
Appendix A Proofs
We present the proofs of theorems on stability of SDE. The proofs are adapted from [20]. We start with two crucial lemmas.
Lemma A.1.
Proof.
By Assumption (2) on $\bm{f},\bm{G}$, we can obtain that for any $\bm{\epsilon},\stackrel{~}{\bm{\epsilon}}\in {\mathbb{R}}^{n},t\ge 0$
$\parallel {\bm{f}}_{\mathrm{\Delta}}(\bm{\epsilon},t)\parallel +\parallel {\bm{G}}_{\mathrm{\Delta}}(\bm{\epsilon},t)\parallel \le {c}_{2}\parallel \bm{\epsilon}\parallel \le {c}_{2}(1+\parallel \bm{\epsilon}\parallel ),$  
$\parallel {\bm{f}}_{\mathrm{\Delta}}(\bm{\epsilon},t){\bm{f}}_{\mathrm{\Delta}}(\stackrel{~}{\bm{\epsilon}},t)\parallel +\parallel {\bm{G}}_{\mathrm{\Delta}}(\bm{\epsilon},t){\bm{G}}_{\mathrm{\Delta}}(\stackrel{~}{\bm{\epsilon}},t)\parallel \le {c}_{2}\parallel \bm{\epsilon}\stackrel{~}{\bm{\epsilon}}\parallel .$ 
This guarantees the uniqueness of the solution of (14). ∎
Lemma A.2.
For (14), whenever ${\mathbf{\epsilon}}_{\mathrm{0}}\mathrm{\ne}\mathrm{0}$, $\mathrm{Pr}\mathit{}\mathrm{\{}{\mathbf{\epsilon}}_{t}\mathrm{\ne}\mathrm{0}\mathit{}\mathit{\text{for all}}\mathit{}t\mathrm{\ge}\mathrm{0}\mathrm{\}}\mathrm{=}\mathrm{1}$.
Proof.
We prove it by contradiction. Let $\tau =inf\{t\ge 0:{\bm{\epsilon}}_{t}=0\}$. Then if it is not true, there exists some ${\bm{\epsilon}}_{0}\ne \mathrm{\U0001d7ce}$ such that $$. Therefore, we can find sufficiently large constant $T>0$ and $\theta >1$ such that $$. By Assumption 2 on $\bm{f}$ and $\bm{G}$, there exists a positive constant ${K}_{\theta}$ such that
$$\parallel {\bm{f}}_{\mathrm{\Delta}}(\bm{\epsilon},t)\parallel +\parallel {\bm{G}}_{\mathrm{\Delta}}(\bm{\epsilon},t)\parallel \le {K}_{\theta}\parallel \bm{\epsilon}\parallel ,\text{for all}\parallel \bm{\epsilon}\parallel \le \theta \text{and}0\le t\le T.$$  (17) 
Let $V(\bm{\epsilon},t)={\parallel \u03f5\parallel}^{1}$. Then, for any $0\le \parallel \bm{\epsilon}\parallel \le \theta $ and $0\le t\le T$, we have
$\mathcal{L}V(\bm{\epsilon},t)$  $={\parallel \bm{\epsilon}\parallel}^{3}{\bm{\epsilon}}^{\top}{\bm{f}}_{\mathrm{\Delta}}(\bm{\epsilon},t)+{\displaystyle \frac{1}{2}}\{{\parallel \bm{\epsilon}\parallel}^{3}{\parallel {\bm{G}}_{\mathrm{\Delta}}(\bm{\epsilon},t)\parallel}^{2}+3{\parallel \bm{\epsilon}\parallel}^{5}{\parallel {\bm{\epsilon}}^{\top}{\bm{G}}_{\mathrm{\Delta}}(\bm{\epsilon},t)\parallel}^{2}\}$  
$\le {\parallel \bm{\epsilon}\parallel}^{2}\parallel {\bm{f}}_{\mathrm{\Delta}}(\bm{\epsilon},t)\parallel +{\parallel \bm{\epsilon}\parallel}^{3}{\parallel {\bm{G}}_{\mathrm{\Delta}}(\bm{\epsilon},t)\parallel}^{2}$  
$\le {K}_{\theta}{\parallel \bm{\epsilon}\parallel}^{1}+{K}_{\theta}^{2}{\parallel \bm{\epsilon}\parallel}^{1}={K}_{\theta}(1+{K}_{\theta})V(\bm{\epsilon},t),$  (18) 
where the first inequality comes from CauchySchwartz and the last one comes from (17). For any $\delta \in (0,\parallel {\bm{\epsilon}}_{0}\parallel )$, we define the stopping time ${\tau}_{\delta}\u2254inf\{t\ge 0:\parallel {\bm{\epsilon}}_{t}\parallel \notin (\delta ,\theta )\}$. Let ${\nu}_{\delta}=\mathrm{min}\{{\tau}_{\delta},T\}$. By Itô’s formula, $\mathbb{E}\left[{e}^{{K}_{\theta}(1+{K}_{\theta}){\nu}_{\delta}}V({\bm{\epsilon}}_{{\nu}_{\delta}},{\nu}_{\delta})\right]$
$=V({\bm{\epsilon}}_{0},0)+\mathbb{E}{\displaystyle {\int}_{0}^{{\nu}_{\delta}}}{e}^{{K}_{\theta}(1+{K}_{\theta})}s\left[{K}_{\theta}(1+{K}_{\theta})V({\bm{\epsilon}}_{s},s)+\mathcal{L}V({\bm{\epsilon}}_{s},s)\right]\text{dif}s\le {\parallel {\bm{\epsilon}}_{0}\parallel}^{1}.$  (19) 
Since ${\tau}_{\delta}\le T$ and $\parallel {\bm{\epsilon}}_{{\tau}_{\delta}}\parallel =\delta $ for any $\omega \in A$, then (19) implies
$$\mathbb{E}\left[{e}^{{K}_{\theta}(1+{K}_{\theta})T}{\delta}^{1}{\mathrm{\U0001d7cf}}_{A}\right]={\delta}^{1}{e}^{{K}_{\theta}(1+{K}_{\theta})T}\mathrm{Pr}(A)\le {\parallel {\bm{\epsilon}}_{0}\parallel}^{1}.$$  (20) 
Thus, $\mathrm{Pr}(A)\le \delta {\parallel {\bm{\epsilon}}_{0}\parallel}^{1}{e}^{{K}_{\theta}(1+{K}_{\theta})T}$. Letting $\delta \to 0$, we obtain $\mathrm{Pr}(A)=0$, which leads to a contradiction. ∎
Proof of Theorem 3.2
We then prove Theorem 3.2. Clearly, (15) holds for ${\bm{\epsilon}}_{0}=\mathrm{\U0001d7ce}$ since ${\bm{\epsilon}}_{t}\equiv \mathrm{\U0001d7ce}$. For any ${\bm{\epsilon}}_{0}\ne \mathrm{\U0001d7ce}$, we have ${\bm{\epsilon}}_{t}\ne \mathrm{\U0001d7ce}$ for all $t\ge 0$ almost surely by Lemma A.2. Thus, by applying Itô’s formula and condition (2), we can show that for $t\ge 0$,
$$\mathrm{log}V({\bm{\epsilon}}_{t},t)\le \mathrm{log}V({\bm{\epsilon}}_{0},0)+{c}_{2}t+M(t)\frac{1}{2}{\int}_{0}^{t}\frac{{{V}_{1}({\bm{\epsilon}}_{s},s){\bm{G}}_{\mathrm{\Delta}}({\bm{\epsilon}}_{s},s)}^{2}}{{V}^{2}({\bm{\epsilon}}_{s},s)}\text{dif}s.$$  (21) 
where $M(t)={\int}_{0}^{t}\frac{{V}_{1}({\bm{\epsilon}}_{s},s){\bm{G}}_{\mathrm{\Delta}}({\bm{\epsilon}}_{s},s)}{V({\bm{\epsilon}}_{s},s)}\text{dif}{\mathbf{B}}_{s}$ is a continuous martingale with initial value $M(0)=0$. By the exponential martingale inequality, for any arbitrary $\alpha \in (0,1)$ and $n=1,2,\mathrm{\cdots}$, we have
$$\mathrm{Pr}\left\{\underset{0\le t\le n}{sup}\left[M(t)\frac{\alpha}{2}{\int}_{0}^{t}\frac{{{V}_{1}({\bm{\epsilon}}_{s},s){\bm{G}}_{\mathrm{\Delta}}({\bm{\epsilon}}_{s},s)}^{2}}{{V}^{2}({\bm{\epsilon}}_{s},s)}\text{dif}s\right]>\frac{2}{\alpha}\mathrm{log}n\right\}\le \frac{1}{{n}^{2}}.$$  (22) 
Applying BorelCantelli lemma, we can get that for almost all $\omega \in \mathrm{\Omega}$, there exists an integer ${n}_{0}={n}_{0}(\omega )$ such that if $n\ge {n}_{0}$,
$$M(t)\le \frac{2}{\alpha}\mathrm{log}n+\frac{\alpha}{2}{\int}_{0}^{t}\frac{{{V}_{1}({\bm{\epsilon}}_{s},s){\bm{G}}_{\mathrm{\Delta}}({\bm{\epsilon}}_{s},s)}^{2}}{{V}^{2}({\bm{\epsilon}}_{s},s)}\text{dif}s,\forall \mathrm{\hspace{0.25em}0}\le t\le n.$$  (23) 
Combining (21), (23) and condition (3), we can obtain that
$$\mathrm{log}V({\bm{\epsilon}}_{t},t)\le \mathrm{log}V({\bm{\epsilon}}_{0},0)\frac{1}{2}[(1\alpha ){c}_{3}2{c}_{2}]t+\frac{2}{\alpha}\mathrm{log}n.$$  (24) 
for all $0\le t\le n$ and $n\ge {n}_{0}$ almost surely. Therefore, for almost all $\omega \in \mathrm{\Omega}$, if $n1\le t\le n$ and $n\ge {n}_{0}$, we have
$$\frac{1}{t}\mathrm{log}V({\bm{\epsilon}}_{t},t)\le \frac{1}{2}[(1\alpha ){c}_{3}2{c}_{2}]+\frac{\mathrm{log}V({\bm{\epsilon}}_{0},0)+\frac{2}{\alpha}\mathrm{log}n}{n1}$$  (25) 
which consequently implies
$$\underset{t\to \mathrm{\infty}}{lim\; sup}\frac{1}{t}\mathrm{log}V({\bm{\epsilon}}_{t},t)\le \frac{1}{2}[(1\alpha ){c}_{3}2{c}_{2}])\mathit{\hspace{1em}}a.s.$$  (26) 
With condition (1) and arbitrary choice of $\alpha \in (0,1)$, we can obtain (15).
Proof of Corollary 3.2.1
We apply Theorem 3.2 to establish the theories on stability of (16). Note that $\bm{f}({\bm{h}}_{t},t;\bm{w})$ is $L$Lipschitz continuous w.r.t ${\bm{h}}_{t}$ and $\bm{G}({\bm{h}}_{t},t;\bm{v})=\sigma {\bm{h}}_{t},m=1$. Then, (16) has a unique solution, with ${\bm{f}}_{\mathrm{\Delta}}$ and ${\bm{G}}_{\mathrm{\Delta}}$ satisfying Assumptions (1,2)
$\parallel {\bm{f}}_{\mathrm{\Delta}}({\bm{\epsilon}}_{t},t)\parallel +\parallel {\bm{G}}_{\mathrm{\Delta}}({\bm{\epsilon}}_{t},t)\parallel \le \mathrm{max}\{L,\sigma \}\parallel {\bm{\epsilon}}_{t}\parallel \le \mathrm{max}\{L,\sigma \}(1+\parallel {\bm{\epsilon}}_{t}\parallel ),$  
$\parallel {\bm{f}}_{\mathrm{\Delta}}({\bm{\epsilon}}_{t},t){\bm{f}}_{\mathrm{\Delta}}({\stackrel{~}{\bm{\epsilon}}}_{t},t)\parallel +\parallel {\bm{G}}_{\mathrm{\Delta}}({\bm{\epsilon}}_{t},t){\bm{G}}_{\mathrm{\Delta}}(\stackrel{~}{{\bm{\epsilon}}_{t}},t)\parallel \le \mathrm{max}\{L,\sigma \}\parallel {\bm{\epsilon}}_{t}\stackrel{~}{{\bm{\epsilon}}_{t}}\parallel .$ 
To apply Theorem 3.2, let $V(\bm{\epsilon},t)={\parallel \bm{\epsilon}\parallel}^{2}$. Then,
$\mathcal{L}V(\bm{\epsilon},t)=2{\bm{\epsilon}}^{\top}{\bm{f}}_{\mathrm{\Delta}}(\bm{\epsilon},t)+{\sigma}^{2}{\parallel \bm{\epsilon}\parallel}^{2}\le (2L+{\sigma}^{2}){\parallel \bm{\epsilon}\parallel}^{2}=(2L+{\sigma}^{2})V(\bm{\epsilon},t),$  
${\parallel {V}_{1}(\bm{\epsilon},t){\bm{G}}_{\mathrm{\Delta}}(\bm{\epsilon},t)\parallel}^{2}=4{\sigma}^{2}V{(\bm{\epsilon},t)}^{2}.$ 
Let ${c}_{1}=1,p=2,{c}_{2}=2L+{\sigma}^{2},{c}_{3}=4{\sigma}^{2}$. By Theorem 3.2, we finished the proof.