Abstract
Offensive language detection and prevention becomes increasing critical formaintaining a healthy social platform and the safe deployment of languagemodels. Despite plentiful researches on toxic and offensive language problem inNLP, existing studies mainly focus on English, while few researches involveChinese due to the limitation of resources. To facilitate Chinese offensivelanguage detection and model evaluation, we collect COLDataset, a Chineseoffensive language dataset containing 37k annotated sentences. With thishigh-quality dataset, we provide a strong baseline classifier, COLDetector,with 81% accuracy for offensive language detection. Furthermore, we alsoutilize the proposed \textsc{COLDetector} to study output offensiveness ofpopular Chinese language models (CDialGPT and CPM). We find that (1) CPM tendsto generate more offensive output than CDialGPT, and (2) certain type ofprompts, like anti-bias sentences, can trigger offensive outputs moreeasily.Altogether, our resources and analyses are intended to help detoxify theChinese online communities and evaluate the safety performance of generativelanguage models. Disclaimer: The paper contains example data that may beconsidered profane, vulgar, or offensive.