Abstract
In recent years, the connections between deep residual networks andfirst-order Ordinary Differential Equations (ODEs) have been disclosed. In thiswork, we further bridge the deep neural architecture design with thesecond-order ODEs and propose a novel reversible neural network, termed asm-RevNet, that is characterized by inserting momentum update to residualblocks. The reversible property allows us to perform backward pass withoutaccess to activation values of the forward pass, greatly relieving the storageburden during training. Furthermore, the theoretical foundation based onsecond-order ODEs grants m-RevNet with stronger representational power thanvanilla residual networks, which potentially explains its performance gains.For certain learning scenarios, we analytically and empirically reveal that ourm-RevNet succeeds while standard ResNet fails. Comprehensive experiments onvarious image classification and semantic segmentation benchmarks demonstratethe superiority of our m-RevNet over ResNet, concerning both memory efficiencyand recognition performance.