Abstract
In this paper we study the effects of quantization in DNN training. Wehypothesize that weight quantization is a form of regularization and the amountof regularization is correlated with the quantization level (precision). Weconfirm our hypothesis by providing analytical study and empirical results. Bymodeling weight quantization as a form of additive noise to weights, we explorehow this noise propagates through the network at training time. We then showthat the magnitude of this noise is correlated with the level of quantization.To confirm our analytical study, we performed an extensive list of experimentssummarized in this paper in which we show that the regularization effects ofquantization can be seen in various vision tasks and models, over variousdatasets. Based on our study, we propose that 8-bit quantization provides areliable form of regularization in different vision tasks and models.