Abstract
Image representations (artificial or biological) are often compared in termsof their global geometric structure; however, representations with similarglobal structure can have strikingly different local geometries. Here, wepropose a framework for comparing a set of image representations in terms oftheir local geometries. We quantify the local geometry of a representationusing the Fisher information matrix, a standard statistical tool forcharacterizing the sensitivity to local stimulus distortions, and use this as asubstrate for a metric on the local geometry in the vicinity of a base image.This metric may then be used to optimally differentiate a set of models, byfinding a pair of "principal distortions" that maximize the variance of themodels under this metric. As an example, we use this framework to compare a setof simple models of the early visual system, identifying a novel set of imagedistortions that allow immediate comparison of the models by visual inspection.In a second example, we apply our method to a set of deep neural network modelsand reveal differences in the local geometry that arise due to architecture andtraining types. These examples demonstrate how our framework can be used toprobe for informative differences in local sensitivities between complexmodels, and suggest how it could be used to compare model representations withhuman perception.