Abstract
Compression is a key step to deploy large neural networks onresource-constrained platforms. As a popular compression technique,quantization constrains the number of distinct weight values and thus reducingthe number of bits required to represent and store each weight. In this paper,we study the representation power of quantized neural networks. First, we provethe universal approximability of quantized ReLU networks. Then we provide upperbounds of storage size given the approximation error bound and the bit-width ofweights for function-independent and function-dependent structures. To the bestof the authors' knowledge, this is the first work on the universalapproximability as well as the associated storage size bound of quantizedneural networks.