Abstract

Data parallelism has become the de facto standard for training Deep NeuralNetwork on multiple processing units. In this work we propose DC-S3GD, adecentralized (without Parameter Server) stale-synchronous version of theDelay-Compensated Asynchronous Stochastic Gradient Descent (DC-ASGD) algorithm.In our approach, we allow for the overlap of computation and communication, andcompensate the inherent error with a first-order correction of the gradients.We prove the effectiveness of our approach by training Convolutional NeuralNetwork with large batches and achieving state-of-the-art results.

 

Quick Read (beta)