Abstract
Dynamic neural network is an emerging research topic in deep learning.Compared to static models which have fixed computational graphs and parametersat the inference stage, dynamic networks can adapt their structures orparameters to different inputs, leading to notable advantages in terms ofaccuracy, computational efficiency, adaptiveness, etc. In this survey, wecomprehensively review this rapidly developing area by dividing dynamicnetworks into three main categories: 1) instance-wise dynamic models thatprocess each instance with data-dependent architectures or parameters; 2)spatial-wise dynamic networks that conduct adaptive computation with respect todifferent spatial locations of image data and 3) temporal-wise dynamic modelsthat perform adaptive inference along the temporal dimension for sequentialdata such as videos and texts. The important research problems of dynamicnetworks, e.g., architecture design, decision making scheme, optimizationtechnique and applications, are reviewed systematically. Finally, we discussthe open problems in this field together with interesting future researchdirections.