Abstract
Many data symmetries can be described in terms of group equivariance and themost common way of encoding group equivariances in neural networks is bybuilding linear layers that are group equivariant. In this work we investigatewhether equivariance of a network implies that all layers are equivariant. Onthe theoretical side we find cases where equivariance implies layerwiseequivariance, but also demonstrate that this is not the case generally.Nevertheless, we conjecture that CNNs that are trained to be equivariant willexhibit layerwise equivariance and explain how this conjecture is a weakerversion of the recent permutation conjecture by Entezari et al. [2022]. Weperform quantitative experiments with VGG-nets on CIFAR10 and qualitativeexperiments with ResNets on ImageNet to illustrate and support our theoreticalfindings. These experiments are not only of interest for understanding howgroup equivariance is encoded in ReLU-networks, but they also give a newperspective on Entezari et al.'s permutation conjecture as we find that it istypically easier to merge a network with a group-transformed version of itselfthan merging two different networks.