Abstract
Masked Language Models (MLMs) pre-trained by predicting masked tokens onlarge corpora have been used successfully in natural language processing tasksfor a variety of languages. Unfortunately, it was reported that MLMs also learndiscriminative biases regarding attributes such as gender and race. Becausemost studies have focused on MLMs in English, the bias of MLMs in otherlanguages has rarely been investigated. Manual annotation of evaluation datafor languages other than English has been challenging due to the cost anddifficulty in recruiting annotators. Moreover, the existing bias evaluationmethods require the stereotypical sentence pairs consisting of the same contextwith attribute words (e.g. He/She is a nurse). We propose Multilingual BiasEvaluation (MBE) score, to evaluate bias in various languages using onlyEnglish attribute word lists and parallel corpora between the target languageand English without requiring manually annotated data. We evaluated MLMs ineight languages using the MBE and confirmed that gender-related biases areencoded in MLMs for all those languages. We manually created datasets forgender bias in Japanese and Russian to evaluate the validity of the MBE. Theresults show that the bias scores reported by the MBE significantly correlateswith that computed from the above manually created datasets and the existingEnglish datasets for gender bias.