CHILLGuardTest: Chinese Content-Safety Benchmark

• CHILLGuardTest is a rigorously curated prompt-level Chinese content-safety benchmark comprising 51,745 samples organized under a 5-macro and 31-micro risk taxonomy.
• It enables both binary safety detection and fine-grained category evaluation, with CHILLGuard-8B achieving an overall F1 score of 89.77 and outperforming alternative guardrails.
• Constructed via a multi-stage pipeline combining RAG-generated data, prompt engineering, and expert annotation, the dataset ensures strict contamination control and balanced class distribution.

CHILLGuardTest is a prompt-level Chinese content-safety benchmark introduced alongside CHILLGuard, a Chinese LLM guardrail trained on a fine-grained taxonomy of harmful content. It is defined as a rigorously curated annotated test set with 51,745 samples, constructed under a 5-macro, 31-micro category risk taxonomy and used solely for standardized evaluation rather than training. Within the CHILLGuard framework, it serves as the held-out test set, supports both overall binary safety detection and fine-grained category-level assessment, and functions as the primary benchmark for comparing Chinese guardrails such as CHILLGuard, Qwen3Guard, LlamaGuard, and PolyGuard in Chinese-specific safety settings (Yu et al., 13 Jun 2026).

1. Benchmark Role within the CHILLGuard Framework

CHILLGuard introduces two paired datasets: CHILLGuardTrain with 405,007 samples and CHILLGuardTest with 51,745 samples. CHILLGuardTest is the evaluation benchmark, whereas model training uses CHILLGuardTrain together with external training corpora. The train–test split is explicitly contamination-controlled: CHILLGuardTest has no overlap with training data, and external multilingual datasets such as PolyGuard CN and OpenGuardrails are integrated only into CHILLGuardTrain.

Its benchmark function is threefold. First, it is the held-out test set for the CHILLGuard guardrail family. Second, it enables fine-grained evaluation across 5 macro categories and 31 micro categories, rather than only coarse safe–unsafe discrimination. Third, it is the dataset on which the headline comparison is reported: on CHILLGuardTest, the 8B CHILLGuard variant reaches an overall F1 score of 89.77, surpassing Qwen3Guard-8B-Strict by 15.92% (Yu et al., 13 Jun 2026).

This positioning makes CHILLGuardTest more than a generic test split. It is the central empirical surface on which claims about Chinese-specific guardrail performance, category coverage, and robustness to implicit harmful prompts are adjudicated.

2. Composition, Class Balance, and Label Structure

CHILLGuardTest contains 51,745 samples, of which 26,691 are safe and 25,054 are unsafe. The safe ratio is 51.58%, and the unsafe ratio is 48.42%. The distribution is therefore nearly balanced, which is consequential because evaluation uses the F1 score with “unsafe” defined as the positive class.

The source composition reported for CHILLGuardTest is as follows:

Source	Total	Safe / Unsafe
RAG-Generated	18,600	8,060 / 10,540
PE-Rewritten	31,961	17,701 / 14,260
Original Real-world	1,184	930 / 254
Total	51,745	26,691 / 25,054

Each sample carries a binary safety label, safe or unsafe. Unsafe samples additionally receive a micro-category label from the 31-category taxonomy, with the corresponding macro category implied by that micro label. Safe samples are labeled only as safe in the binary sense and do not bear a harmful category. There is no explicit mention of multi-label cases. This suggests that CHILLGuardTest operationalizes unsafe prompts as single-label instances for fine-grained evaluation.

The dataset as a whole is stratified across all 31 micro categories to maintain roughly uniform class distribution and to reduce bias toward over-represented harmful types during training. Although the paper does not provide a separate count table for each micro category in the test split, it indicates that all 31 harmful subcategories are represented in CHILLGuardTest. No explicit average prompt length statistics are reported, although length-based filtering is part of preprocessing.

3. Taxonomic Scope: 5 Macro Categories and 31 Micro Categories

The CHILLGuard taxonomy is organized into five macro categories and thirty-one micro categories, and CHILLGuardTest uses this schema uniformly for unsafe prompts.

Macro-Category A: Violations of Core Socialist Values includes A1 inciting subversion of state power or overthrow of the socialist system; A2 endangering national security and interests, or damaging the national image; A3 inciting separatism, undermining national unity and social stability; A4 promoting terrorism or extremism; A5 promoting ethnic hatred; A6 promoting violence, obscenity, or pornography; A7 disseminating false and harmful information; and A8 other content prohibited by laws and regulations.

Macro-Category B: Discriminatory Content includes B1 ethnic discrimination content; B2 religious or belief discrimination content; B3 national discrimination content; B4 regional discrimination content; B5 gender discrimination content; B6 age discrimination content; B7 occupational discrimination content; B8 health-based discrimination content; and B9 other forms of discrimination content.

Macro-Category C: Commercial Violations and Non-compliance includes C1 infringing upon intellectual property rights of others; C2 violating business ethics; C3 disclosing commercial secrets of others; C4 utilizing algorithm, data, or platform advantages to implement monopoly and unfair competition; and C5 other illegal or non-compliant commercial activities.

Macro-Category D: Infringement of Legitimate Rights and Interests includes D1 endangering the physical or mental health of others; D2 infringing upon the portrait rights of others; D3 infringing upon the reputation rights of others; D4 infringing upon the honor rights of others; D5 infringing upon the privacy rights of others; D6 infringing upon personal information rights and interests; and D7 infringing upon other legitimate rights and interests of others.

Macro-Category E: Failure to Meet Safety Demands of Specific Services includes E1 inaccurate content that severely contradicts scientific common sense or mainstream cognition, and E2 unreliable content that fails to provide meaningful assistance to users.

This taxonomy distinguishes CHILLGuardTest from coarser safety benchmarks. In particular, it couples regulatory alignment, culturally situated risk categories, and service-quality-related safety demands within one unified Chinese-centric labeling scheme. A plausible implication is that the benchmark is designed not only to detect explicit harmfulness, but also to measure failure modes that are operationally salient in Chinese deployment environments, including indirect, obfuscated, or service-specific safety deficits (Yu et al., 13 Jun 2026).

4. Construction Pipeline and Annotation Calibration

CHILLGuardTest is built with the same three-stage pipeline used for CHILLGuardTrain: multi-source data generation, unified preprocessing, and multi-model label calibration. The test split, however, draws only from RAG-generated data, PE-rewritten data, and a subset of original real-world prompts; external multilingual safety datasets are excluded from the test set.

In the RAG-based corpus expansion stage, the corpus is crawled from Quora, X (Twitter), and Weibo. The process begins with 20 seed keywords per micro category, totaling 620 keywords, and expands them using Gemini 3.1 Pro to approximately 80 keywords per micro category, for about 2,480 keywords in total. Approximately 480k text samples are crawled, encoded with bge-m3 embeddings, and indexed in a vector database. For each subcategory, retrieval queries are constructed using the macro label, micro label, and sampled keywords; the system retrieves the Top-100 candidate texts, uniformly samples 5 texts, and sends them to Dolphin-Mistral-24B-Venice-Edition for prompt rewriting. The resulting RAG pipeline generates 59,520 samples overall, of which 18,600 appear in CHILLGuardTest.

In the prompt-engineering stage, the objective is to produce implicit and obfuscated harmful prompts reflecting Chinese linguistic phenomena. The rewriting strategies include symbolization and morpheme transformation; cultural allusions and historical metaphor; rhetorical irony and emotional displacement; semantic nesting and compliant packaging; and logical induction and presupposition. These templates are applied to 46,742 real-world production prompts collected from authoritative institutions’ production environments, generating 109,312 rewritten samples. CHILLGuardTest contains 31,961 of these PE-rewritten prompts and 1,184 original real-world prompts.

Preprocessing includes translation of all English data into Chinese with opus-mt-en-zh, exact deduplication, length-based filtering to eliminate overly short, overly long, and meaningless text, standardization of text formats, and removal of irrelevant special characters. For label calibration, binary safe–unsafe decisions are produced by majority voting among Qwen3-30B-Instruct, GLM-4.7-30B-Flash, InternVL3.5-38B-Instruct, and Yi-1.5-34B-Chat, with DeepSeek-V3.2-685B serving as the final adjudicator in tied cases. Fine-grained micro-category annotation is then assigned by DeepSeek-V3.2-685B (Yu et al., 13 Jun 2026).

Manual expert input enters at the seed stage: more than five PhD experts in cybersecurity, computational linguistics, and legal compliance jointly develop the annotation standard aligned with Chinese regulatory requirements. No numeric inter-annotator agreement such as $\kappa$ is reported. Quality control additionally includes removal of personally identifiable information and multi-stage review of potentially offensive or harmful content, and no raw data containing identifiable individuals or unmoderated offensive material is to be released.

5. Evaluation Protocol and Empirical Diagnostic Value

CHILLGuardTest is the central evaluation dataset in the CHILLGuard experiments. The core metric is the F1 score, with the “unsafe” category strictly defined as the positive class. Evaluation is reported at three granularities: overall F1 across the full test set, macro-level F1 for categories A–E, and micro-category F1 for all 31 subcategories.

The benchmark is also used to evaluate the effect of the generator–classifier collaborative training framework and Model-aware Direct Preference Optimization. Although MDPO and the generator are trained only on the training split, their effectiveness is validated on CHILLGuardTest. In the reported ablation, CHILLGuard-8B with two rounds and standard DPO achieves an overall F1 of 88.85, while the full CHILLGuard-8B with MDPO reaches 89.77.

In the main comparison, CHILLGuard-8B achieves an overall F1 of 89.77 on CHILLGuardTest, while Qwen3Guard-8B-Strict achieves 77.44. The reported 15.92% gain is the benchmark’s headline improvement figure; the same data also correspond to an absolute difference of 12.33 F1 points. The macro-average breakdown for CHILLGuard-8B is 90.70 on Macro A, 90.84 on Macro B, 82.36 on Macro C, 92.30 on Macro D, and 90.99 on Macro E. For Qwen3Guard-8B-Strict, the corresponding values are 78.66, 77.08, 75.04, 86.24, and 49.40 (Yu et al., 13 Jun 2026).

These results show why CHILLGuardTest is diagnostically useful. It does not merely rank models by a single scalar score; it exposes category-specific weaknesses, especially on discrimination content and on Macro E, which covers inaccurate or unreliable content that fails safety demands of specific services. This suggests that the benchmark is sensitive to safety failure modes that may be overlooked by datasets centered primarily on overt toxicity or refusal behavior.

6. Relation to Other Benchmarks, Intended Use, and Constraints

CHILLGuardTest is evaluated alongside other datasets, including PolyGuardPrompts, WildGuardTest, ChineseSafe, DoNotAnswer, SafetyPrompts, BeaverTails, and RTP_LX. Among these, CHILLGuardTest is the only benchmark described as specifically designed for Chinese content, built on the 5-macro, 31-micro taxonomy aligned with Chinese regulations, and used to report per-category F1 across all 31 micro categories. By contrast, SafetyBench is characterized as bilingual with only 7 coarse categories, and PolyGuard and WildGuard are described as multilingual or English-centric, with smaller Chinese subsets not aligned to the Chinese regulatory taxonomy.

Its intended use is research benchmarking: evaluating Chinese LLM guardrails and benchmarking robustness against implicit and adversarial Chinese harmful prompts. The dataset, models, and code are to be released under the CC BY-NC 4.0 license, permitting non-commercial research use with attribution and prohibiting harmful or commercial exploitation. The release is announced at https://github.com/cswbyu/CHILLGuard.

Several limitations are explicit. The taxonomy is tuned to mainstream Chinese application scenarios, so specialized vertical domains may require further expansion. Robustness against future or more adaptive adversarial attacks still requires improvement. The dataset and model are optimized for Chinese, and cross-lingual generalization is not guaranteed. Because the taxonomy is aligned with Chinese regulatory policies and cultural norms, it embeds Chinese legal and value frameworks; some content categories may therefore be treated differently from how they would be treated in other jurisdictions.

Taken together, these characteristics define CHILLGuardTest as a Chinese-centric, fine-grained, contamination-controlled safety benchmark for prompt-level evaluation. Its technical significance lies in combining near-balanced binary evaluation, 31-way harmful-content stratification, synthetic and real-world prompt sources, and a label calibration pipeline designed for Chinese regulatory and linguistic specificity (Yu et al., 13 Jun 2026).