Abstract
Low-precision representation of deep neural networks (DNNs) is critical forefficient deployment of deep learning application on embedded platforms,however, converting the network to low precision degrades its performance.Crucially, networks that are designed for embedded applications usually sufferfrom increased degradation since they have less redundancy. This is mostevident for the ubiquitous MobileNet architecture which requires a costlyquantization-aware training cycle to achieve acceptable performance whenquantized to 8-bits. In this paper, we trace the source of the degradation inMobileNets to a shift in the mean activation value. This shift is caused by aninherent bias in the quantization process which builds up across layers,shifting all network statistics away from the learned distribution. We showthat this phenomenon happens in other architectures as well. We propose asimple remedy - compensating for the quantization induced shift by adding aconstant to the additive bias term of each channel. We develop two simplemethods for estimating the correction constants - one using iterativeevaluation of the quantized network and one where the constants are set using ashort training phase. Both methods are fast and require only a small amount ofunlabeled data, making them appealing for rapid deployment of neural networks.Using the above methods we are able to match the performance of training-basedquantization of MobileNets at a fraction of the cost.