Abstract
We introduce WildGuard -- an open, light-weight moderation tool for LLMsafety that achieves three goals: (1) identifying malicious intent in userprompts, (2) detecting safety risks of model responses, and (3) determiningmodel refusal rate. Together, WildGuard serves the increasing needs forautomatic safety moderation and evaluation of LLM interactions, providing aone-stop tool with enhanced accuracy and broad coverage across 13 riskcategories. While existing open moderation tools such as Llama-Guard2 scorereasonably well in classifying straightforward model interactions, they lag farbehind a prompted GPT-4, especially in identifying adversarial jailbreaks andin evaluating models' refusals, a key measure for evaluating safety behaviorsin model responses. To address these challenges, we construct WildGuardMix, a large-scale andcarefully balanced multi-task safety moderation dataset with 92K labeledexamples that cover vanilla (direct) prompts and adversarial jailbreaks, pairedwith various refusal and compliance responses. WildGuardMix is a combination ofWildGuardTrain, the training data of WildGuard, and WildGuardTest, ahigh-quality human-annotated moderation test set with 5K labeled items coveringbroad risk scenarios. Through extensive evaluations on WildGuardTest and tenexisting public benchmarks, we show that WildGuard establishes state-of-the-artperformance in open-source safety moderation across all the three taskscompared to ten strong existing open-source moderation models (e.g., up to26.4% improvement on refusal detection). Importantly, WildGuard matches andsometimes exceeds GPT-4 performance (e.g., up to 3.9% improvement on promptharmfulness identification). WildGuard serves as a highly effective safetymoderator in an LLM interface, reducing the success rate of jailbreak attacksfrom 79.8% to 2.4%.