Abstract
We address the following question of neural network identifiability: Supposewe are given a function $f:\mathbb{R}^m\to\mathbb{R}^n$ and a nonlinearity$\rho$. Can we specify the architecture, weights, and biases of allfeed-forward neural networks with respect to $\rho$ giving rise to $f$?Existing literature on the subject suggests that the answer should be yes,provided we are only concerned with finding networks that satisfy certain"genericity conditions". Moreover, the identified networks are mutually relatedby symmetries of the nonlinearity. For instance, the $\tanh$ function is odd,and so flipping the signs of the incoming and outgoing weights of a neuron doesnot change the output map of the network. The results known hitherto, however,apply either to single-layer networks, or to networks satisfying specificstructural assumptions (such as full connectivity), as well as to specificnonlinearities. In an effort to answer the identifiability question in greatergenerality, we consider arbitrary nonlinearities with potentially complicatedaffine symmetries, and we show that the symmetries can be used to find a richset of networks giving rise to the same function $f$. The set obtained in thismanner is, in fact, exhaustive (i.e., it contains all networks giving rise to$f$) unless there exists a network $\mathcal{A}$ "with no internal symmetries"giving rise to the identically zero function. This result can thus beinterpreted as an analog of the rank-nullity theorem for linear operators. Wefurthermore exhibit a class of "$\tanh$-type" nonlinearities (including thetanh function itself) for which such a network $\mathcal{A}$ does not exist,thereby solving the identifiability question for these nonlinearities in fullgenerality. Finally, we show that this class contains nonlinearities witharbitrarily complicated symmetries.