Abstract
Training a neural network using backpropagation algorithm requires passingerror gradients sequentially through the network. The backward locking preventsus from updating network layers in parallel and fully leveraging the computingresources. Recently, there are several works trying to decouple and parallelizethe backpropagation algorithm. However, all of them suffer from severe accuracyloss or memory explosion when the neural network is deep. To address thesechallenging issues, we propose a novel parallel-objective formulation for theobjective function of the neural network. After that, we introduce featuresreplay algorithm and prove that it is guaranteed to converge to critical pointsfor the non-convex problem under certain conditions. Finally, we apply ourmethod to training deep convolutional neural networks, and the experimentalresults show that the proposed method achieves {faster} convergence, {lower}memory consumption, and {better} generalization error than compared methods.