Abstract
In recent years, knowledge distillation (KD) has been widely used as aneffective way to derive efficient models. Through imitating a large teachermodel, a lightweight student model can achieve comparable performance with moreefficiency. However, most existing knowledge distillation methods are focusedon classification tasks. Only a limited number of studies have appliedknowledge distillation to object detection, especially in time-sensitiveautonomous driving scenarios. We propose the Adaptive Instance Distillation(AID) method to selectively impart knowledge from the teacher to the studentfor improving the performance of knowledge distillation. Unlike previous KDmethods that treat all instances equally, our AID can attentively adjust thedistillation weights of instances based on the teacher model's prediction loss.We verified the effectiveness of our AID method through experiments on theKITTI and the COCO traffic datasets. The results show that our method improvesthe performance of existing state-of-the-art attention-guided and non-localdistillation methods and achieves better distillation results on bothsingle-stage and two-stage detectors. Compared to the baseline, our AID led toan average of 2.7% and 2.05% mAP increases for single-stage and two-stagedetectors, respectively. Furthermore, our AID is also shown to be useful forself-distillation to improve the teacher model's performance.