WildGuardTest: LLM Safety Moderation Benchmark

WildGuardTest is a human-annotated evaluation benchmark designed to rigorously assess the safety moderation capabilities of LLMs, specifically with respect to harmful prompt detection, response harmfulness evaluation, and refusal identification. Developed as a core part of the WildGuardMix dataset, WildGuardTest provides high-quality, fine-grained, and adversarially challenging test cases to advance the state-of-the-art in automatic LLM safety moderation.

1. Design Principles and Motivations

WildGuardTest was constructed to address critical deficits in existing moderation benchmarks. While prior datasets offered some coverage of harmful and benign content, they often lacked rigorous adversarial prompt evaluation, nuanced measurement of model refusals, and comprehensive risk taxonomy coverage. WildGuardTest was developed to:

• Provide high-fidelity, gold-standard human-labeled test sets for LLM safety benchmarks.
• Evaluate not only straightforward but also challenging “jailbreak” (adversarial prompt) cases.
• Jointly assess the three central moderation tasks: prompt harmfulness detection, response harmfulness detection, and response refusal detection.
• Enable robust, reproducible comparisons across both open-source and API-based moderation models.

2. Coverage, Structure, and Annotation Methodology

WildGuardTest contains 5,299 labeled examples, comprising 1,725 prompt-response pairs derived from both synthetic vanilla and adversarial sources. Each example is triple-annotated for:

• Prompt harm (Is the user prompt itself harmful?)
• Response harm (Is the model’s response harmful?)
• Response refusal (Does the model correctly refuse inappropriate prompts?)

The test set covers both vanilla and jailbreak prompt forms, with comprehensive inclusion of both benign and harmful scenarios as well as a diversity of nuanced refusals and compliances.

Risk Taxonomy

WildGuardTest organizes risk using 13 subcategories across four principal domains, aligned with Weidinger et al. (2021):

• Privacy (sensitive information, copyright)
• Misinformation (false content, material harm)
• Harmful Language (stereotypes/discrimination, violence, hate speech, sexual content)
• Malicious Uses (cyberattacks, aiding illegal activity, unsafe action encouragement, mental health misuse)
• Additionally, other minor harms and benign cases are proportionally represented.

Annotation Process

• Each example is labeled by three independent human annotators.
• Gold labels are determined by majority vote; items with insufficient agreement (“unsure” status) are excluded.
• Disagreements between human annotators and a prompted GPT-4 classifier are manually audited, increasing quality assurance.
• Inter-annotator reliability, measured by Fleiss’ Kappa, is strong: 0.55 (prompt harm), 0.50 (response harm), and 0.72 (refusal).

3. Unique Features and Differentiation

WildGuardTest is characterized by several distinctive aspects compared to existing safety and moderation benchmarks:

• Explicit Adversarial Coverage: Includes a broad set of carefully constructed adversarial attacks (“jailbreaks”), which are largely absent or insufficiently represented in alternative datasets.
• Multi-task Labeling: Unlike datasets supporting only one or two moderation tasks per item, WildGuardTest provides comprehensive joint labels (prompt harm, response harm, refusal) for each instance.
• Fine-grained Taxonomy: Harmonized and even coverage across granular risk subdomains, supporting detailed evaluation and error analysis.
• Audit-Driven Consensus: Combination of human and GPT-4 audits delivers high-confidence gold standards.
• Nuanced Refusal Annotation: Extensive annotation of refusal intent and its correctness—previously under-studied in open benchmarks.

4. Impact, Model Comparisons, and Statistical Results

WildGuardTest serves as the primary evaluation set for WildGuard and multiple competing open and closed moderation systems. It facilitates rigorous, fine-grained measurement across moderation tasks in both standard and adversarial contexts.

Evaluation Metrics

Performance is primarily reported as F1 scores for each moderation task and subpartition (e.g., on adversarial prompts specifically, or in aggregate).

Key observations:

• WildGuard exceeds all open baselines in adversarial prompt harmfulness detection by over 11 F1 points, surpassing even GPT-4 by 3.9% on this axis.
• For refusal detection, WildGuard is the only open-source system that approaches GPT-4 performance, improving by +21.2% F1 over all others.
• As a moderator in an LLM pipeline, WildGuard reduces the success rate of harmful jailbreak attacks from 79.8% to 2.4%, while minimally increasing the refusal rate (RTA) on benign prompts (from 0.0% to 0.4%).

$$\text{ASR} = \frac{\#\, \text{jailbreak attacks not rejected}}{\#\, \text{jailbreak attempts}}$$

$$\text{ASR}_{\text{Tulu2-dpo}} = 79.8\% \rightarrow \text{ASR}_{\text{Tulu2+WildGuard}} = 2.4\%$$

$$\text{RTA}_{\text{benign}} = 0.0\% \rightarrow 0.4\%$$

5. Role in Model and Dataset Development

WildGuardTest underpins much of WildGuard’s design and development:

• Dataset and Model Ablation: Systematic ablations reveal that removing adversarial or annotator-written examples from the training set degrades performance, highlighting WildGuardTest’s necessity in identifying strengths and limitations.
• Generalization Benchmark: Test set breadth and annotation quality drive state-of-the-art transfer to unseen benchmarks and support robust model optimization.
• Evaluation Feedback Loop: Iterative performance measurement on WildGuardTest facilitates the identification and amelioration of residual failure modes, particularly for nuanced refusals and adversarial exploits.

6. Comparison with Recent Related Benchmarks

WildGuardTest is frequently compared with newly released moderation and safety benchmarks. For instance, in BingoGuard (Yin et al., 9 Mar 2025 ), WildGuardTest serves as the standard reference for binary safety detection: WildGuard-8B is outperformed by BingoGuard-8B by 4.3% on overall detection accuracy, although WildGuardTest itself tends to concentrate on high-severity harmful cases.

Compared to fine-grained severity-level evaluation benchmarks (such as BingoGuardTest), WildGuardTest offers:

• Heavier coverage of high-severity and explicitly adversarial harmful responses,
• Greater focus on three-task alignment at once (prompt, response, refusal),
• Slightly less diversity (measured by Self-BLEU) but much broader risk coverage and adversarial challenge than typical binary datasets.

7. Ongoing Influence and Research Applications

WildGuardTest has become the de facto gold standard for evaluating open and closed LLM moderation systems across the safety research community. Its fine-grained risk taxonomy, adversarial prompt curation, and robust annotation processes underpin:

• The assessment and optimization of new moderation architectures,
• Calibration of reward models for LLM RLHF pipelines,
• Empirical studies of transferability, generalization, and safety trade-offs for both commercial and open-source guardrails,
• Benchmarking of innovative approaches in robust in-the-wild LLM alignment.

WildGuardTest’s dual focus on both forbidden and benign content, including nuanced, policy-borderline cases, ensures its relevance for real-world LLM deployments in safety-critical applications. Its creation marks a significant advance in the methodology and rigor of LLM safety evaluation, setting a high bar for future benchmarks.