Abstract
Extremely low-resource languages, especially those written in rare scripts,as shown in Figure 1, remain largely unsupported by large language models(LLMs). This is due in part to compounding factors such as the lack of trainingdata. This paper delivers the first comprehensive analysis of whether LLMs canacquire such languages purely via in-context learning (ICL), with or withoutauxiliary alignment signals, and how these methods compare toparameter-efficient fine-tuning (PEFT). We systematically evaluate 20under-represented languages across three state-of-the-art multilingual LLMs.Our findings highlight the limitation of PEFT when both language and its scriptare extremely under-represented by the LLM. In contrast, zero-shot ICL withlanguage alignment is impressively effective on extremely low-resourcelanguages, while few-shot ICL or PEFT is more beneficial for languagesrelatively better represented by LLMs. For LLM practitioners working onextremely low-resource languages, we summarise guidelines grounded by ourresults on adapting LLMs to low-resource languages, e.g., avoiding fine-tuninga multilingual model on languages of unseen scripts.