Abstract
Modular deep learning has been proposed for the efficient adaption ofpre-trained models to new tasks, domains and languages. In particular,combining language adapters with task adapters has shown potential where nosupervised data exists for a language. In this paper, we explore the role oflanguage adapters in zero-shot cross-lingual transfer for natural languageunderstanding (NLU) benchmarks. We study the effect of including atarget-language adapter in detailed ablation studies with two multilingualmodels and three multilingual datasets. Our results show that the effect oftarget-language adapters is highly inconsistent across tasks, languages andmodels. Retaining the source-language adapter instead often leads to anequivalent, and sometimes to a better, performance. Removing the languageadapter after training has only a weak negative effect, indicating that thelanguage adapters do not have a strong impact on the predictions.