Language Representation Projection: Can We Transfer Factual Knowledge across Languages in Multilingual Language Models?

  • 2023-11-07 08:16:16
  • Shaoyang Xu, Junzhuo Li, Deyi Xiong
  • 0

Abstract

Multilingual pretrained language models serve as repositories of multilingualfactual knowledge. Nevertheless, a substantial performance gap of factualknowledge probing exists between high-resource languages and low-resourcelanguages, suggesting limited implicit factual knowledge transfer acrosslanguages in multilingual pretrained language models. This paper investigatesthe feasibility of explicitly transferring relatively rich factual knowledgefrom English to non-English languages. To accomplish this, we propose twoparameter-free $\textbf{L}$anguage $\textbf{R}$epresentation$\textbf{P}$rojection modules (LRP2). The first module converts non-Englishrepresentations into English-like equivalents, while the second module revertsEnglish-like representations back into representations of the correspondingnon-English language. Experimental results on the mLAMA dataset demonstratethat LRP2 significantly improves factual knowledge retrieval accuracy andfacilitates knowledge transferability across diverse non-English languages. Wefurther investigate the working mechanism of LRP2 from the perspectives ofrepresentation space and cross-lingual knowledge neuron.

 

Quick Read (beta)

loading the full paper ...