Abstract
We propose to impose symmetry in neural network parameters to improveparameter usage and make use of dedicated convolution and matrix multiplicationroutines. Due to significant reduction in the number of parameters as a resultof the symmetry constraints, one would expect a dramatic drop in accuracy.Surprisingly, we show that this is not the case, and, depending on networksize, symmetry can have little or no negative effect on network accuracy,especially in deep overparameterized networks. We propose several ways toimpose local symmetry in recurrent and convolutional neural networks, and showthat our symmetry parameterizations satisfy universal approximation propertyfor single hidden layer networks. We extensively evaluate theseparameterizations on CIFAR, ImageNet and language modeling datasets, showingsignificant benefits from the use of symmetry. For instance, our ResNet-101with channel-wise symmetry has almost 25% less parameters and only 0.2%accuracy loss on ImageNet. Code for our experiments is available athttps://github.com/hushell/deep-symmetry