Abstract
We analyze the operation of transformer language adapters, which are smallmodules trained on top of a frozen language model to adapt its predictions tonew target languages. We show that adapted predictions mostly evolve in thesource language the model was trained on, while the target language becomespronounced only in the very last layers of the model. Moreover, the adaptationprocess is gradual and distributed across layers, where it is possible to skipsmall groups of adapters without decreasing adaptation performance. Last, weshow that adapters operate on top of the model's frozen representation spacewhile largely preserving its structure, rather than on an 'isolated' subspace.Our findings provide a deeper view into the adaptation process of languagemodels to new languages, showcasing the constraints imposed on it by theunderlying model and introduces practical implications to enhance itsefficiency.