Abstract
The ubiquity of offensive content on social media is a growing cause forconcern among companies and government organizations. Recently,transformer-based models such as BERT, XLNET, and XLM-R have achievedstate-of-the-art performance in detecting various forms of offensive content(e.g. hate speech, cyberbullying, and cyberaggression). However, the majorityof these models are limited in their capabilities due to their encoder-onlyarchitecture, which restricts the number and types of labels in downstreamtasks. Addressing these limitations, this study presents the first pre-trainedmodel with encoder-decoder architecture for offensive language identificationwith text-to-text transformers (T5) trained on two large offensive languageidentification datasets; SOLID and CCTK. We investigate the effectiveness ofcombining two datasets and selecting an optimal threshold in semi-supervisedinstances in SOLID in the T5 retraining step. Our pre-trained T5 modeloutperforms other transformer-based models fine-tuned for offensive languagedetection, such as fBERT and HateBERT, in multiple English benchmarks.Following a similar approach, we also train the first multilingual pre-trainedmodel for offensive language identification using mT5 and evaluate itsperformance on a set of six different languages (German, Hindi, Korean,Marathi, Sinhala, and Spanish). The results demonstrate that this multilingualmodel achieves a new state-of-the-art on all the above datasets, showing itsusefulness in multilingual scenarios. Our proposed T5-based models will be madefreely available to the community.