Abstract
Language models (LMs) have proven surprisingly successful at capturingfactual knowledge by completing cloze-style fill-in-the-blank questions such as"Punta Cana is located in _." However, while knowledge is both written andqueried in many languages, studies on LMs' factual representation ability havealmost invariably been performed on English. To assess factual knowledgeretrieval in LMs in different languages, we create a multilingual benchmark ofcloze-style probes for \langnum typologically diverse languages. To properlyhandle language variations, we expand probing methods from single- tomulti-word entities, and develop several decoding algorithms to generatemulti-token predictions. Extensive experimental results provide insights abouthow well (or poorly) current state-of-the-art LMs perform at this task inlanguages with more or fewer available resources. We further propose acode-switching-based method to improve the ability of multilingual LMs toaccess knowledge, and verify its effectiveness on several benchmark languages.Benchmark data and code have been released at https://x-factr.github.io.