SimMIM: A Simple Framework for Masked Image Modeling

  • 2021-11-18 18:59:45
  • Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, Han Hu
  • 46


This paper presents SimMIM, a simple framework for masked image modeling. Wesimplify recently proposed related approaches without special designs such asblock-wise masking and tokenization via discrete VAE or clustering. To studywhat let the masked image modeling task learn good representations, wesystematically study the major components in our framework, and find thatsimple designs of each component have revealed very strong representationlearning performance: 1) random masking of the input image with a moderatelylarge masked patch size (e.g., 32) makes a strong pre-text task; 2) predictingraw pixels of RGB values by direct regression performs no worse than the patchclassification approaches with complex designs; 3) the prediction head can beas light as a linear layer, with no worse performance than heavier ones. UsingViT-B, our approach achieves 83.8% top-1 fine-tuning accuracy on ImageNet-1K bypre-training also on this dataset, surpassing previous best approach by +0.6%.When applied on a larger model of about 650 million parameters, SwinV2-H, itachieves 87.1% top-1 accuracy on ImageNet-1K using only ImageNet-1K data. Wealso leverage this approach to facilitate the training of a 3B model(SwinV2-G), that by $40\times$ less data than that in previous practice, weachieve the state-of-the-art on four representative vision benchmarks. The codeand models will be publicly available at


Quick Read (beta)

loading the full paper ...