Abstract
Normalization techniques are essential for accelerating the training andimproving the generalization of deep neural networks (DNNs), and havesuccessfully been used in various applications. This paper reviews and commentson the past, present and future of normalization methods in the context of DNNtraining. We provide a unified picture of the main motivation behind differentapproaches from the perspective of optimization, and present a taxonomy forunderstanding the similarities and differences between them. Specifically, wedecompose the pipeline of the most representative normalizing activationmethods into three components: the normalization area partitioning,normalization operation and normalization representation recovery. In doing so,we provide insight for designing new normalization technique. Finally, wediscuss the current progress in understanding normalization methods, andprovide a comprehensive review of the applications of normalization forparticular tasks, in which it can effectively solve the key issues.