Abstract
Techniques such as ensembling and distillation promise model qualityimprovements when paired with almost any base model. However, due to increasedtest-time cost (for ensembles) and increased complexity of the trainingpipeline (for distillation), these techniques are challenging to use inindustrial settings. In this paper we explore a variant of distillation whichis relatively straightforward to use as it does not require a complicatedmulti-stage setup or many new hyperparameters. Our first claim is that onlinedistillation enables us to use extra parallelism to fit very large datasetsabout twice as fast. Crucially, we can still speed up training even after wehave already reached the point at which additional parallelism provides nobenefit for synchronous or asynchronous stochastic gradient descent. Two neuralnetworks trained on disjoint subsets of the data can share knowledge byencouraging each model to agree with the predictions the other model would havemade. These predictions can come from a stale version of the other model sothey can be safely computed using weights that only rarely get transmitted. Oursecond claim is that online distillation is a cost-effective way to make theexact predictions of a model dramatically more reproducible. We support ourclaims using experiments on the Criteo Display Ad Challenge dataset, ImageNet,and the largest to-date dataset used for neural language modeling, containing$6\times 10^{11}$ tokens and based on the Common Crawl repository of web data.