Adapting Multilingual LLMs to Low-Resource Languages with Knowledge Graphs via Adapters

  • 2024-07-01 16:56:24
  • Daniil Gurgurov, Mareike Hartmann, Simon Ostermann
  • 0


This paper explores the integration of graph knowledge from linguisticontologies into multilingual Large Language Models (LLMs) using adapters toimprove performance for low-resource languages (LRLs) in sentiment analysis(SA) and named entity recognition (NER). Building upon successfulparameter-efficient fine-tuning techniques, such as K-ADAPTER and MAD-X, wepropose a similar approach for incorporating knowledge from multilingualgraphs, connecting concepts in various languages with each other throughlinguistic relationships, into multilingual LLMs for LRLs. Specifically, wefocus on eight LRLs -- Maltese, Bulgarian, Indonesian, Nepali, Javanese,Uyghur, Tibetan, and Sinhala -- and employ language-specific adaptersfine-tuned on data extracted from the language-specific section of ConceptNet,aiming to enable knowledge transfer across the languages covered by theknowledge graph. We compare various fine-tuning objectives, including standardMasked Language Modeling (MLM), MLM with full-word masking, and MLM withtargeted masking, to analyse their effectiveness in learning and integratingthe extracted graph data. Through empirical evaluation on language-specifictasks, we assess how structured graph knowledge affects the performance ofmultilingual LLMs for LRLs in SA and NER, providing insights into the potentialbenefits of adapting language models for low-resource scenarios.


Quick Read (beta)

loading the full paper ...