Similarity of Neural Network Representations Revisited

  • 2019-06-11 16:19:38
  • Simon Kornblith, Mohammad Norouzi, Honglak Lee, Geoffrey Hinton
  • 0

Abstract

Recent work has sought to understand the behavior of neural networks bycomparing representations between layers and between different trained models.We examine methods for comparing neural network representations based oncanonical correlation analysis (CCA). We show that CCA belongs to a family ofstatistics for measuring multivariate similarity, but that neither CCA nor anyother statistic that is invariant to invertible linear transformation canmeasure meaningful similarities between representations of higher dimensionthan the number of data points. We introduce a similarity index that measuresthe relationship between representational similarity matrices and does notsuffer from this limitation. This similarity index is equivalent to centeredkernel alignment (CKA) and is also closely connected to CCA. Unlike CCA, CKAcan reliably identify correspondences between representations in networkstrained from different initializations.

 

Quick Read (beta)

loading the full paper ...