Abstract
Multilingual large-scale Pretrained Language Models (PLMs) have been shown tostore considerable amounts of factual knowledge, but large variations areobserved across languages. With the ultimate goal of ensuring that users withdifferent language backgrounds obtain consistent feedback from the same model,we study the cross-lingual consistency (CLC) of factual knowledge in variousmultilingual PLMs. To this end, we propose a Ranking-based Consistency (RankC)metric to evaluate knowledge consistency across languages independently fromaccuracy. Using this metric, we conduct an in-depth analysis of the determiningfactors for CLC, both at model level and at language-pair level. Among otherresults, we find that increasing model size leads to higher factual probingaccuracy in most languages, but does not improve cross-lingual consistency.Finally, we conduct a case study on CLC when new factual associations areinserted in the PLMs via model editing. Results on a small sample of factsinserted in English reveal a clear pattern whereby the new piece of knowledgetransfers only to languages with which English has a high RankC score.