Abstract
Regularization is commonly used in machine learning for alleviatingoverfitting. In convolutional neural networks, regularization methods, such asDropout and Shake-Shake, have been proposed to improve the generalizationperformance. However, these methods are lack of self-adaption throughouttraining, i.e., the regularization strength is fixed to a predefined schedule,and manual adjustment has to be performed to adapt to various networkarchitectures. In this paper, we propose a dynamic regularization method whichcan dynamically adjust the regularization strength in the training procedure.Specifically, we model the regularization strength as a backward difference ofthe training loss, which can be directly extracted in each training iteration.With dynamic regularization, the large model is regularized by the strongperturbation and vice versa. Experimental results show that the proposed methodcan improve the generalization capability of off-the-shelf networkarchitectures and outperforms state-of-the-art regularization methods.