Cendol: Open Instruction-tuned Generative Large Language Models for Indonesian Languages

  • 2024-07-08 04:33:52
  • Samuel Cahyawijaya, Holy Lovenia, Fajri Koto, Rifki Afina Putri, Emmanuel Dave, Jhonson Lee, Nuur Shadieq, Wawan Cenggoro, Salsabil Maulana Akbar, Muhammad Ihza Mahendra, Dea Annisayanti Putri, Bryan Wilie, Genta Indra Winata, Alham Fikri Aji, Ayu Purwarianti, Pascale Fung
  • 0


Large language models (LLMs) show remarkable human-like capability in variousdomains and languages. However, a notable quality gap arises in low-resourcelanguages, e.g., Indonesian indigenous languages, rendering them ineffectiveand inefficient in such linguistic contexts. To bridge this quality gap, weintroduce Cendol, a collection of Indonesian LLMs encompassing bothdecoder-only and encoder-decoder architectures across a range of model sizes.We highlight Cendol's effectiveness across a diverse array of tasks, attaining20% improvement, and demonstrate its capability to generalize to unseen tasksand indigenous languages of Indonesia. Furthermore, Cendol models showcaseimproved human favorability despite their limitations in capturing indigenousknowledge and cultural values in Indonesia. In addition, we discuss theshortcomings of parameter-efficient tunings, such as LoRA, for languageadaptation. Alternatively, we propose the usage of vocabulary adaptation toenhance efficiency. Lastly, we evaluate the safety of Cendol and showcase thatsafety in pre-training in one language such as English is transferable tolow-resource languages, such as Indonesian, even without RLHF and safetyfine-tuning.


Quick Read (beta)

loading the full paper ...