Abstract
Though deep neural networks (DNNs) achieve remarkable performances in manyartificial intelligence tasks, the lack of training instances remains anotorious challenge. As the network goes deeper, the generalization accuracydecays rapidly in the situation of lacking massive amounts of training data. Inthis paper, we propose novel deep neural network structures that can beinherited from all existing DNNs with almost the same level of complexity, anddevelop simple training algorithms. We show our paradigm successfully resolvesthe lack of data issue. Tests on the CIFAR10 and CIFAR100 image recognitiondatasets show that the new paradigm leads to 20$\%$ to $30\%$ relative errorrate reduction compared to their base DNNs. The intuition of our algorithms fordeep residual network stems from theories of the partial differential equation(PDE) control problems. Code will be made available.