Abstract
Transformer-based language models are able to generate fluent text and beefficiently adapted across various natural language generation tasks. However,language models that are pretrained on large unlabeled web text corpora havebeen shown to suffer from degenerating toxic content and social bias behaviors,consequently hindering their safe deployment. Various detoxification methodswere proposed to mitigate the language model's toxicity; however, these methodsstruggled to detoxify language models when conditioned on prompts that containspecific social identities related to gender, race, or religion. In this study,we propose Reinforce-Detoxify; A reinforcement learning-based method formitigating toxicity in language models. We address the challenge of safety inlanguage models and propose a new reward model that is able to detect toxiccontent and mitigate unintended bias towards social identities in toxicityprediction. The experiments demonstrate that the Reinforce-Detoxify method forlanguage model detoxification outperforms existing detoxification approaches inautomatic evaluation metrics, indicating the ability of our approach inlanguage model detoxification and less prone to unintended bias toward socialidentities in generated content.