Abstract
We present a simple self-training method that achieves 87.4% top-1 accuracyon ImageNet, which is 1.0% better than the state-of-the-art model that requires3.5B weakly labeled Instagram images. On robustness test sets, it improvesImageNet-A top-1 accuracy from 16.6% to 74.2%, reduces ImageNet-C meancorruption error from 45.7 to 31.2, and reduces ImageNet-P mean flip rate from27.8 to 16.1. To achieve this result, we first train an EfficientNet model on labeledImageNet images and use it as a teacher to generate pseudo labels on 300Munlabeled images. We then train a larger EfficientNet as a student model on thecombination of labeled and pseudo labeled images. We iterate this process byputting back the student as the teacher. During the generation of the pseudolabels, the teacher is not noised so that the pseudo labels are as good aspossible. But during the learning of the student, we inject noise such as dataaugmentation, dropout, stochastic depth to the student so that the noisedstudent is forced to learn harder from the pseudo labels.