Abstract
Limited availability of multilingual text corpora for training languagemodels often leads to poor performance on downstream tasks due to undertrainedrepresentation spaces for languages other than English. This'under-representation' has motivated recent cross-lingual transfer methods toleverage the English representation space by e.g. mixing English and'non-English' tokens at the input level or extending model parameters toaccommodate new languages. However, these approaches often come at the cost ofincreased computational complexity. We propose Fusion forLanguageRepresentations (FLARE) in adapters, a novel method that enhancesrepresentation quality and downstream performance for languages other thanEnglish while maintaining parameter efficiency. FLARE integrates source andtarget language representations within low-rank (LoRA) adapters usinglightweight linear transformations, maintaining parameter efficiency whileimproving transfer performance. A series of experiments across representativecross-lingual natural language understanding tasks, including natural languageinference, question-answering and sentiment analysis, demonstrate FLARE'seffectiveness. FLARE achieves performance improvements of 4.9% for Llama 3.1and 2.2% for Gemma~2 compared to standard LoRA fine-tuning onquestion-answering tasks, as measured by the exact match metric.