Abstract
Large-scale distributed training of deep neural networks suffer from thegeneralization gap caused by the increase in the effective mini-batch size.Previous approaches try to solve this problem by varying the learning rate andbatch size over epochs and layers, or some ad hoc modification of the batchnormalization. We propose an alternative approach using a second-orderoptimization method that shows similar generalization capability to first-ordermethods, but converges faster and can handle larger mini-batches. To test ourmethod on a benchmark where highly optimized first-order methods are availableas references, we train ResNet-50 on ImageNet. We converged to 75% Top-1validation accuracy in 35 epochs for mini-batch sizes under 16,384, andachieved 75% even with a mini-batch size of 131,072, which took 100 epochs.