Abstract
Recent developments in large-scale machine learning suggest that by scalingup data, model size and training time properly, one might observe thatimprovements in pre-training would transfer favorably to most downstream tasks.In this work, we systematically study this phenomena and establish that, as weincrease the upstream accuracy, the performance of downstream tasks saturates.In particular, we investigate more than 4800 experiments on VisionTransformers, MLP-Mixers and ResNets with number of parameters ranging from tenmillion to ten billion, trained on the largest scale of available image data(JFT, ImageNet21K) and evaluated on more than 20 downstream image recognitiontasks. We propose a model for downstream performance that reflects thesaturation phenomena and captures the nonlinear relationship in performance ofupstream and downstream tasks. Delving deeper to understand the reasons thatgive rise to these phenomena, we show that the saturation behavior we observeis closely related to the way that representations evolve through the layers ofthe models. We showcase an even more extreme scenario where performance onupstream and downstream are at odds with each other. That is, to have a betterdownstream performance, we need to hurt upstream accuracy.