Abstract
Recent work has proposed explicitly inducing language-wise modularity inmultilingual LMs via sparse fine-tuning (SFT) on per-language subnetworks as ameans of better guiding cross-lingual sharing. In this work, we investigate (1)the degree to which language-wise modularity naturally arises within modelswith no special modularity interventions, and (2) how cross-lingual sharing andinterference differ between such models and those with explicit SFT-guidedsubnetwork modularity. To quantify language specialization and cross-lingualinteraction, we use a Training Data Attribution method that estimates thedegree to which a model's predictions are influenced by in-language orcross-language training examples. Our results show that language-specializedsubnetworks do naturally arise, and that SFT, rather than always increasingmodularity, can decrease language specialization of subnetworks in favor ofmore cross-lingual sharing.