Abstract
Gender, race and social biases have recently been detected as evidentexamples of unfairness in applications of Natural Language Processing. A keypath towards fairness is to understand, analyse and interpret our data andalgorithms. Recent studies have shown that the human-generated data used intraining is an apparent factor of getting biases. In addition, currentalgorithms have also been proven to amplify biases from data. To further address these concerns, in this paper, we study how anstate-of-the-art recurrent neural language model behaves when trained on data,which under-represents females, using pre-trained standard and debiased wordembeddings. Results show that language models inherit higher bias when trainedon unbalanced data when using pre-trained embeddings, in comparison with usingembeddings trained within the task. Moreover, results show that, on the samedata, language models inherit lower bias when using debiased pre-trainedemdeddings, compared to using standard pre-trained embeddings.