Abstract
Pre-trained language models have been known to perpetuate biases from theunderlying datasets to downstream tasks. However, these findings arepredominantly based on monolingual language models for English, whereas thereare few investigative studies of biases encoded in language models forlanguages beyond English. In this paper, we fill this gap by analysing genderbias in West Slavic language models. We introduce the first template-baseddataset in Czech, Polish, and Slovak for measuring gender bias towards male,female and non-binary subjects. We complete the sentences using both mono- andmultilingual language models and assess their suitability for the maskedlanguage modelling objective. Next, we measure gender bias encoded in WestSlavic language models by quantifying the toxicity and genderness of thegenerated words. We find that these language models produce hurtful completionsthat depend on the subject's gender. Perhaps surprisingly, Czech, Slovak, andPolish language models produce more hurtful completions with men as subjects,which, upon inspection, we find is due to completions being related toviolence, death, and sickness.