Abstract
Self-Supervised Learning (SSL) powers many current AI systems. As researchinterest and investment grow, the SSL design space continues to expand. ThePlatonic view of SSL, following the Platonic Representation Hypothesis (PRH),suggests that despite different methods and engineering approaches, allrepresentations converge to the same Platonic ideal. However, this phenomenonlacks precise theoretical explanation. By synthesizing evidence fromIdentifiability Theory (IT), we show that the PRH can emerge in SSL. However,current IT cannot explain SSL's empirical success. To bridge the gap betweentheory and practice, we propose expanding IT into what we term SingularIdentifiability Theory (SITh), a broader theoretical framework encompassing theentire SSL pipeline. SITh would allow deeper insights into the implicit dataassumptions in SSL and advance the field towards learning more interpretableand generalizable representations. We highlight three critical directions forfuture research: 1) training dynamics and convergence properties of SSL; 2) theimpact of finite samples, batch size, and data diversity; and 3) the role ofinductive biases in architecture, augmentations, initialization schemes, andoptimizers.