Abstract
In this paper, we elucidate how representations in deep neural networks(DNNs) evolve during training. Our focus is on overparameterized learningsettings where the training continues much after the trained DNN starts toperfectly fit its training data. We examine the evolution of learnedrepresentations along the entire training process. We explore therepresentational similarity of DNN layers, each layer with respect to its ownrepresentations throughout the training process. For this, we use twosimilarity metrics: (1) The centered kernel alignment (CKA) similarity; (2)Similarity of decision regions of linear classifier probes that we train forthe DNN layers. We visualize and analyze the decision regions of the DNN outputand the layer probes during the DNN training to show how they geometricallyevolve. Our extensive experiments discover training dynamics patterns that canemerge in layers depending on the relative layer-depth, architecture andoptimizer. Among our findings: (i) The training phases, including those relatedto memorization, are more distinguishable in SGD training than in Adamtraining, and for Vision Transformer (ViT) than for ResNet; (ii) Unlike ResNet,the ViT layers have synchronized dynamics of representation learning.