Abstract
The capabilities of Large Language Models (LLMs) in low-resource languageslag far behind those in English, making their universal accessibility asignificant challenge. To alleviate this, we present$\textit{Franken-Adapter}$, a modular language adaptation approach fordecoder-only LLMs with embedding surgery. Our method begins by creatingcustomized vocabularies for target languages and performing language adaptationthrough embedding tuning on multilingual data. These pre-trained embeddings aresubsequently integrated with LLMs that have been instruction-tuned on Englishalignment data to enable zero-shot cross-lingual transfer. Our experiments on$\texttt{Gemma2}$ models with up to 27B parameters demonstrate improvements ofup to 20% across 96 languages, spanning both discriminative and generativetasks, with minimal regressions ($<$1%) in English. Further in-depth analysisreveals the critical role of customizing tokenizers in enhancing languageadaptation, while boosting inference efficiency. Additionally, we show theversatility of our method by achieving a 14% improvement over a math-optimizedLLM across 20 languages, offering a modular solution to transfer reasoningabilities across languages post hoc.