Adapting Multilingual LLMs to Low-Resource Languages with Knowledge Graphs via Adapters

  • 2024-07-23 16:51:12
  • Daniil Gurgurov, Mareike Hartmann, Simon Ostermann
  • 0

Abstract

This paper explores the integration of graph knowledge from linguisticontologies into multilingual Large Language Models (LLMs) using adapters toimprove performance for low-resource languages (LRLs) in sentiment analysis(SA) and named entity recognition (NER). Building upon successfulparameter-efficient fine-tuning techniques, such as K-ADAPTER and MAD-X, wepropose a similar approach for incorporating knowledge from multilingualgraphs, connecting concepts in various languages with each other throughlinguistic relationships, into multilingual LLMs for LRLs. Specifically, wefocus on eight LRLs -- Maltese, Bulgarian, Indonesian, Nepali, Javanese,Uyghur, Tibetan, and Sinhala -- and employ language-specific adaptersfine-tuned on data extracted from the language-specific section of ConceptNet,aiming to enable knowledge transfer across the languages covered by theknowledge graph. We compare various fine-tuning objectives, including standardMasked Language Modeling (MLM), MLM with full-word masking, and MLM withtargeted masking, to analyse their effectiveness in learning and integratingthe extracted graph data. Through empirical evaluation on language-specifictasks, we assess how structured graph knowledge affects the performance ofmultilingual LLMs for LRLs in SA and NER, providing insights into the potentialbenefits of adapting language models for low-resource scenarios.

 

Quick Read (beta)

loading the full paper ...