Abstract
Realignment techniques are often employed to enhance cross-lingual transferin multilingual language models, still, they can sometimes degrade performancein languages that differ significantly from the fine-tuned source language.This paper introduces AlignFreeze, a method that freezes either the layers'lower half or upper half during realignment. Through controlled experiments on4 tasks, 3 models, and in 35 languages, we find that realignment affects allthe layers but can be the most detrimental to the lower ones. Freezing thelower layers can prevent performance degradation. Particularly, AlignFreezeimproves Part-of-Speech (PoS) tagging performances in languages where fullrealignment fails: with XLM-R, it provides improvements of more than onestandard deviation in accuracy in seven more languages than full realignment.