Abstract
Cross-lingual knowledge transfer, especially between high- and low-resourcelanguages, remains challenging in natural language processing (NLP). This studyoffers insights for improving cross-lingual NLP applications through thecombination of parameter-efficient fine-tuning methods. We systematicallyexplore strategies for enhancing cross-lingual transfer through theincorporation of language-specific and task-specific adapters and soft prompts.We present a detailed investigation of various combinations of these methods,exploring their efficiency across 16 languages, focusing on 10 mid- andlow-resource languages. We further present to our knowledge the first use ofsoft prompts for language transfer, a technique we call soft language prompts.Our findings demonstrate that in contrast to claims of previous work, acombination of language and task adapters does not always work best; instead,combining a soft language prompt with a task adapter outperforms mostconfigurations in many cases.