Abstract
Deep learning techniques have been demonstrated to surpass precedingcutting-edge machine learning techniques in recent years, with computer visionbeing one of the most prominent examples. However, deep learning models sufferfrom significant drawbacks when deployed in resource-constrained environmentsdue to their large model size and high complexity. Knowledge Distillation isone of the prominent solutions to overcome this challenge. This review paperexamines the current state of research on knowledge distillation, a techniquefor compressing complex models into smaller and simpler ones. The paperprovides an overview of the major principles and techniques associated withknowledge distillation and reviews the applications of knowledge distillationin the domain of computer vision. The review focuses on the benefits ofknowledge distillation, as well as the problems that must be overcome toimprove its effectiveness.