Abstract
Warning: This paper contains explicit statements of offensive stereotypeswhich might be upsetting. Language models are prone to exhibiting biases, further amplifying unfair andharmful stereotypes. Given the fast-growing popularity and wide application ofthese models, it is necessary to ensure safe and fair language models. As ofrecent considerable attention has been paid to measuring bias in languagemodels, yet the majority of studies have focused only on English language. ADutch version of the US-specific CrowS-Pairs dataset for measuring bias inDutch language models is introduced. The resulting dataset consists of 1463sentence pairs that cover bias in 9 categories, such as Sexual orientation,Gender and Disability. The sentence pairs are composed of contrastingsentences, where one of the sentences concerns disadvantaged groups and theother advantaged groups. Using the Dutch CrowS-Pairs dataset, we show thatvarious language models, BERTje, RobBERT, multilingual BERT, GEITje andMistral-7B exhibit substantial bias across the various bias categories. Usingthe English and French versions of the CrowS-Pairs dataset, bias was evaluatedin English (BERT and RoBERTa) and French (FlauBERT and CamemBERT) languagemodels, and it was shown that English models exhibit the most bias, whereasDutch models the least amount of bias. Additionally, results also indicate thatassigning a persona to a language model changes the level of bias it exhibits.These findings highlight the variability of bias across languages and contexts,suggesting that cultural and linguistic factors play a significant role inshaping model biases.