Abstract
Online toxic language causes real harm, especially in regions with limitedmoderation tools. In this study, we evaluate how large language models handletoxic comments in Serbian, Croatian, and Bosnian, languages with limitedlabeled data. We built and manually labeled a dataset of 4,500 YouTube andTikTok comments drawn from videos across diverse categories, including music,politics, sports, modeling, influencer content, discussions of sexism, andgeneral topics. Four models (GPT-3.5 Turbo, GPT-4.1, Gemini 1.5 Pro, and Claude3 Opus) were tested in two modes: zero-shot and context-augmented. We measuredprecision, recall, F1 score, accuracy and false positive rates. Including ashort context snippet raised recall by about 0.12 on average and improved F1score by up to 0.10, though it sometimes increased false positives. The bestbalance came from Gemini in context-augmented mode, reaching an F1 score of0.82 and accuracy of 0.82, while zero-shot GPT-4.1 led on precision and had thelowest false alarms. We show how adding minimal context can improve toxiclanguage detection in low-resource settings and suggest practical strategiessuch as improved prompt design and threshold calibration. These results showthat prompt design alone can yield meaningful gains in toxicity detection forunderserved Balkan language communities.