SimReg: Regression as a Simple Yet Effective Tool for Self-supervised Knowledge Distillation

  • 2022-01-13 18:41:46
  • K L Navaneet, Soroush Abbasi Koohpayegani, Ajinkya Tejankar, Hamed Pirsiavash
  • 1


Feature regression is a simple way to distill large neural network models tosmaller ones. We show that with simple changes to the network architecture,regression can outperform more complex state-of-the-art approaches forknowledge distillation from self-supervised models. Surprisingly, the additionof a multi-layer perceptron head to the CNN backbone is beneficial even if usedonly during distillation and discarded in the downstream task. Deepernon-linear projections can thus be used to accurately mimic the teacher withoutchanging inference architecture and time. Moreover, we utilize independentprojection heads to simultaneously distill multiple teacher networks. We alsofind that using the same weakly augmented image as input for both teacher andstudent networks aids distillation. Experiments on ImageNet dataset demonstratethe efficacy of the proposed changes in various self-supervised distillationsettings.


Quick Read (beta)

loading the full paper ...