Abstract
Large language models (LLMs) learn non-trivial abstractions duringpretraining, like detecting irregular plural noun subjects. However, it is notwell understood when and how specific linguistic abilities emerge astraditional evaluation methods such as benchmarking fail to reveal how modelsacquire concepts and capabilities. To bridge this gap and better understandmodel training at the concept level, we use sparse crosscoders to discover andalign features across model checkpoints. Using this approach, we track theevolution of linguistic features during pretraining. We train crosscodersbetween open-sourced checkpoint triplets with significant performance andrepresentation shifts, and introduce a novel metric, Relative Indirect Effects(RelIE), to trace training stages at which individual features become causallyimportant for task performance. We show that crosscoders can detect featureemergence, maintenance, and discontinuation during pretraining. Our approach isarchitecture-agnostic and scalable, offering a promising path toward moreinterpretable and fine-grained analysis of representation learning throughoutpretraining.