Abstract
Instruction-tuned large language models (LLMs) have shown remarkablegeneralization capability over multiple tasks in multiple languages.Nevertheless, their generalization towards different languages variesespecially to underrepresented languages or even to unseen languages. Priorworks on adapting new languages to LLMs find that naively adapting newlanguages to instruction-tuned LLMs will result in catastrophic forgetting,which in turn causes the loss of multitasking ability in these LLMs. To tacklethis, we propose the Instruct-Align a.k.a (IA)$^1$ framework, which enablesinstruction-tuned LLMs to learn cross-lingual alignment between unseen andpreviously learned languages via alignment-based cross-lingualinstruction-tuning. Our preliminary result on BLOOMZ-560M shows that (IA)$^1$is able to learn a new language effectively with only a limited amount ofparallel data and at the same time prevent catastrophic forgetting by applyingcontinual instruction-tuning through experience replay. Our work contributes tothe progression of language adaptation methods for instruction-tuned LLMs andopens up the possibility of adapting underrepresented low-resource languagesinto existing instruction-tuned LLMs. Our code will be publicly released uponacceptance.