KTAN: Knowledge Transfer Adversarial Network

  • 2018-10-18 15:57:02
  • Peiye Liu, Wu Liu, Huadong Ma, Tao Mei, Mingoo Seok
  • 8

Abstract

To reduce the large computation and storage cost of a deep convolutionalneural network, the knowledge distillation based methods have pioneered totransfer the generalization ability of a large (teacher) deep network to alight-weight (student) network. However, these methods mostly focus ontransferring the probability distribution of the softmax layer in a teachernetwork and thus neglect the intermediate representations. In this paper, wepropose a knowledge transfer adversarial network to better train a studentnetwork. Our technique holistically considers both intermediate representationsand probability distributions of a teacher network. To transfer the knowledgeof intermediate representations, we set high-level teacher feature maps as atarget, toward which the student feature maps are trained. Specifically, wearrange a Teacher-to-Student layer for enabling our framework suitable forvarious student structures. The intermediate representation helps the studentnetwork better understand the transferred generalization as compared to theprobability distribution only. Furthermore, we infuse an adversarial learningprocess by employing a discriminator network, which can fully exploit thespatial correlation of feature maps in training a student network. Theexperimental results demonstrate that the proposed method can significantlyimprove the performance of a student network on both image classification andobject detection tasks.

 

Quick Read (beta)

loading the full paper ...