Abstract
Optimization of Binarized Neural Networks (BNNs) currently relies onreal-valued latent weights to accumulate small update steps. In this paper, weargue that these latent weights cannot be treated analogously to weights inreal-valued networks. Instead their main role is to provide inertia duringtraining. We interpret current methods in terms of inertia and provide novelinsights into the optimization of BNNs. We subsequently introduce the firstoptimizer specifically designed for BNNs, Binary Optimizer (Bop), anddemonstrate its performance on CIFAR-10 and ImageNet. Together, theredefinition of latent weights as inertia and the introduction of Bop enable abetter understanding of BNN optimization and open up the way for furtherimprovements in training methodologies for BNNs. Code is available at:https://github.com/plumerai/rethinking-bnn-optimization