Abstract
Identifiability is a desirable property of a statistical model: it impliesthat the true model parameters may be estimated to any desired precision, givensufficient computational resources and data. We study identifiability in thecontext of representation learning: discovering nonlinear data representationsthat are optimal with respect to some downstream task. When parameterized asdeep neural networks, such representation functions typically lackidentifiability in parameter space, because they are overparameterized bydesign. In this paper, building on recent advances in nonlinear ICA, we aim torehabilitate identifiability by showing that a large family of discriminativemodels are in fact identifiable in function space, up to a linearindeterminacy. Many models for representation learning in a wide variety ofdomains have been identifiable in this sense, including text, images and audio,state-of-the-art at time of publication. We derive sufficient conditions forlinear identifiability and provide empirical support for the result on bothsimulated and real-world data.