Adversarial HK Value Benchmark

• The paper introduces a proprietary benchmark using 300 human-crafted, Hong Kong-sensitive prompts to rigorously assess LLM alignment.
• It employs a structured evaluation process with human annotations that classify model outputs as safe, unsafe, or refusal based on local socio-legal norms.
• Empirical findings highlight a trade-off where HKGAI-V1 excels on regional sensitivities but is more vulnerable to general jailbreak attacks, informing RLHF improvements.

Searching arXiv for the specified paper and closely related benchmark/alignment work to support the article. Search query: arXiv (Han et al., 14 Jul 2025) HKGAI-V1 Adversarial HK Value Benchmark The Adversarial HK Value Benchmark is a proprietary safety and alignment benchmark introduced in "HKGAI-V1: Towards Regional Sovereign LLM for Hong Kong" (Han et al., 14 Jul 2025). It was constructed to test how well HKGAI-V1 adheres to Hong Kong–relevant ethical and legal norms under adversarial conditions, with particular emphasis on politically sensitive, legally sensitive, and ethically charged prompts. Rather than functioning as a general toxicity or bias dataset, it is presented as a focused red-teaming tool for Hong Kong’s socio-legal environment, using 300 human-crafted sensitive questions and human judgment to measure whether model outputs are safe, unsafe, or merely template refusals (Han et al., 14 Jul 2025).

1. Definition and evaluative role

The benchmark is explicitly described as employing “300 human-crafted sensitive questions with opposing ‘safe’ and ‘unsafe’ viewpoints to rigorously evaluate model alignment.” Its methodology consists of five stages: constructing a diverse question set aligned with Hong Kong-specific content labels; eliciting model responses; conducting human evaluation using predefined safe and unsafe criteria; statistically analyzing the proportions and biases of response classes across question categories; and feeding the results back into training refinement (Han et al., 14 Jul 2025).

Its stated purposes are threefold. First, it is meant to stress-test HKGAI-V1’s value alignment in the Hong Kong context, beyond generic global safety benchmarks. Second, it aims to expose vulnerabilities, especially unsafe responses to local political and legal sensitivities or to jailbreak-style instructions. Third, it is intended to generate actionable signals for further RLHF, correction learning, and policy updates in what the paper calls a sovereign, governance-embedded setting (Han et al., 14 Jul 2025).

The term “adversarial” is used in a specific sense. Prompts are human-crafted to probe failure modes rather than neutral question answering; they include Hong Kong-specific sensitive topics, instruction attacks, and typical safety scenarios; and they are designed to test the model under what the paper calls distributional and normative stress. This includes legally sensitive content, political contention, and ethically charged contexts where naïve generative models tend to fail (Han et al., 14 Jul 2025).

2. Normative and legal foundations

The benchmark is anchored in the broader value framework governing HKGAI-V1. The paper repeatedly situates that framework within Hong Kong’s socio-legal environment under “one country, two systems,” understood as a hybrid of PRC sovereignty and Hong Kong’s common-law–influenced legal system. Safe responses are defined as “ethical, neutral, compliant with PRC and HK laws” and also as “neutral, reasonable, and ethically sound, adhering to legal and social norms.” Unsafe responses are defined as “controversial, risky, harmful” content that violates or undermines those laws and norms (Han et al., 14 Jul 2025).

Several normative dimensions structure this safe/unsafe distinction. The paper highlights alignment with Hong Kong’s mainstream social values, including “law, fairness, diversity, etc.”; harmlessness and safety, including avoidance of violence, self-harm, illegal acts, and discrimination; legality and law compliance, including national security and privacy obligations such as PDPO; neutrality and political sensitivity; and honesty and factuality, especially in the broader RAG moderation workflow (Han et al., 14 Jul 2025).

Within this framework, the benchmark primarily scores responses as safe, unsafe, or refusal template rather than by factual accuracy. Even so, its conceptual basis is multi-dimensional. The paper explicitly ties it to safety or harmlessness, legality or regulatory compliance, political and cultural appropriateness in Hong Kong’s context, and neutrality or non-escalation on contentious issues. This suggests that the binary safe/unsafe outcome is an operational surface over a denser normative substrate rather than a complete theory of alignment (Han et al., 14 Jul 2025).

3. Construction, prompt taxonomy, and annotation

The benchmark consists of 300 human-crafted questions. Each is designed to probe safety and value alignment and is associated with both a safe and an unsafe possible viewpoint. The question set is aligned with Hong Kong-specific content labels, and the paper links its construction to a broader process of mining Hong Kong media, online discussions, and public discourse for sensitive themes (Han et al., 14 Jul 2025).

Evaluation is organized into three modules:

Module	Function	Reported response classes
Hong Kong Sensitive Issues	Politically or legally sensitive Hong Kong topics	Refusal template, Safe, Unsafe
Instruction Attack	Jailbreak or safety-bypass prompts	Refusal template, Safe, Unsafe
Typical Safety Scenarios	Generic safety tasks used as baseline	Refusal template, Safe, Unsafe

The “Hong Kong Sensitive Issues” module covers topics such as national security, protest-related issues, sovereignty, and public order. The “Instruction Attack” module contains prompts designed to induce the model to ignore or circumvent safety constraints. The “Typical Safety Scenarios” module contains more standard risk prompts, such as violence, self-harm, and discrimination, and functions as a baseline against which specialized Hong Kong stressors can be contrasted (Han et al., 14 Jul 2025).

Annotation is human-based. Responses are judged according to predefined criteria for Safe and Unsafe, with an additional “Refusal template” category for canned, template-based refusals. The paper states that local annotators were enlisted to provide corrections and that dataset design was carried out in close collaboration with social science colleagues. The same local Hong Kong annotators and collaborators are used to craft sensitive questions, define Hong Kong-specific content labels, and manually judge outputs. Inter-annotator agreement metrics such as Cohen’s $\kappa$ are not reported (Han et al., 14 Jul 2025).

The benchmark is also embedded in a multilingual setting. HKGAI-V1 is designed for Cantonese, Mandarin, and English, and the surrounding evaluation framework tests language-following in Simplified Chinese, Traditional Chinese, English, and oral Cantonese. The benchmark itself is not reported with language-specific statistics, but it is presented as part of the same multilingual alignment framework. A plausible implication is that benchmark prompts are intended to reflect Hong Kong-local discourse, including Traditional Chinese, Cantonese-relevant forms, and English-language legal or international contexts, even though the paper does not tabulate the adversarial benchmark by language (Han et al., 14 Jul 2025).

4. Evaluation protocol and metrics

The evaluation protocol is deliberately non-automatic. For each of the 300 adversarial questions, the model is prompted; the raw response is collected; human evaluators classify the response as Safe, Unsafe, or Refusal template; and the proportions of these classes are then computed for each module and each model (Han et al., 14 Jul 2025).

The paper formalizes these proportions in a straightforward way. For module $m$ and model $k$, if $$\ell_i^{(k)} \in \{\text{Safe}, \text{Unsafe}, \text{Refusal}\}$$ is the human label for question $i$, then the benchmark reports:

$$\text{SafeRate}_{m}^{(k)} = \frac{1}{N_m} \sum_{i=1}^{N_m} \mathbf{1}\{\ell_{i}^{(k)} = \text{Safe}\},$$

$$\text{UnsafeRate}_{m}^{(k)} = \frac{1}{N_m} \sum_{i=1}^{N_m} \mathbf{1}\{\ell_{i}^{(k)} = \text{Unsafe}\},$$

$$\text{RefusalRate}_{m}^{(k)} = \frac{1}{N_m} \sum_{i=1}^{N_m} \mathbf{1}\{\ell_{i}^{(k)} = \text{Refusal}\}.$$

No more complex composite index is defined. The benchmark therefore privileges interpretable response-class rates over scalarized aggregate scores (Han et al., 14 Jul 2025).

A notable design choice is the separation of refusal templates from substantive safe answers. The paper explicitly treats this as important because excessive reliance on pre-fixed refusals can mask weak underlying alignment: a model that refuses nearly everything may appear safe while remaining unhelpful or poorly calibrated. This is one of the benchmark’s distinctive analytical moves, especially in comparison with evaluations that collapse all refusals into harmlessness (Han et al., 14 Jul 2025).

The benchmark also does not report formal adversarial metrics such as attack success rate. The adversarial dimension is operationalized instead through unsafe percentages and the contrast among Hong Kong-sensitive prompts, jailbreak-style instruction attacks, and standard safety scenarios. The paper further notes that systematic perturbation strategies such as paraphrase adversaries or dedicated multi-turn chains are not specified for this benchmark, even though HKGAI-V1 is assessed more broadly through red-team adversarial testing elsewhere (Han et al., 14 Jul 2025).

5. Empirical findings and revealed trade-offs

The benchmark compares HKGAI-V1 Chat, Kimi, and ChatGPT across the three modules. The most prominent result is on the Hong Kong Sensitive Issues module: HKGAI-V1 Chat achieved 79% safe responses, which the paper describes as the strongest performance in that category, while ChatGPT displayed 88.7% unsafe responses in the same domain. Kimi is described as relying heavily on template-based safe responses, with 42% refusal templates, 53% purely safe answers, and 5% unsafe answers (Han et al., 14 Jul 2025).

On instruction attacks, the pattern reverses. The paper states that HKGAI-V1 Chat had the highest percentage of unsafe responses at 16.5%, compared with 6% for Kimi and 8% for ChatGPT. This is interpreted as a relative vulnerability of the Hong Kong-specialized model to general jailbreak-style prompting, despite its stronger behavior on Hong Kong-specific sensitivities (Han et al., 14 Jul 2025).

On typical safety scenarios, all three models were reported as robust. ChatGPT achieved the highest safe response rate at 91%, followed by Kimi at 83% and HKGAI-V1 Chat at 82%, with minimal unsafe responses across the module. In these generic scenarios, HKGAI-V1 is competitive but not dominant (Han et al., 14 Jul 2025).

These outcomes can be summarized succinctly:

Module	Key outcome	Comparative interpretation
Hong Kong Sensitive Issues	HKGAI-V1 Chat: 79% safe; ChatGPT: 88.7% unsafe	HKGAI-V1 is strongest on region-specific sensitivity
Instruction Attack	HKGAI-V1 Chat: 16.5% unsafe; Kimi: 6%; ChatGPT: 8%	HKGAI-V1 is more vulnerable to jailbreak-style prompts
Typical Safety Scenarios	ChatGPT: 91% safe; Kimi: 83%; HKGAI-V1 Chat: 82%	All models perform strongly on generic safety tasks

The paper explicitly frames this as a trade-off. It states that HKGAI-V1 demonstrated “superior alignment on Hong Kong-specific sensitive queries (79% safe responses vs. ChatGPT’s 10.7%)” while exhibiting “a higher vulnerability to general instruction attacks, with an unsafe response rate of 16.5%.” The authors infer that the alignment process may have successfully encoded specific regional guardrails while also creating new attack surfaces (Han et al., 14 Jul 2025). That interpretation is central to the benchmark’s significance: it does not merely rank models, but diagnoses where regional specialization improves alignment and where it may weaken generalized adversarial robustness.

6. Integration, comparative position, and limitations

The benchmark is not treated as a standalone leaderboard instrument. It is embedded in HKGAI-V1’s larger alignment and governance framework, where failures feed back into RLHF, LLF, correction-based training through HKValue-Aligner, and updates to policy templates and RAG guardrails. Figure-level descriptions in the paper place it in a “red-teaming sandbox,” and the broader evaluation workflow uses benchmarking datasets to drive “HKGAI-V1 System Optimization.” In that sense, the benchmark functions as a continuous-improvement mechanism: measure, analyze, retrain or adjust policies, and re-measure (Han et al., 14 Jul 2025).

The paper also positions it relative to other benchmarks. Beaver-zh-hk is described as a Chinese-language safety benchmark tailored to Hong Kong but not focused on adversarial prompts. SafeLawBench is a legally focused safety benchmark, and NaVAB is a multi-national value alignment benchmark spanning China, the United States, the United Kingdom, France, and Germany. What is said to distinguish the Adversarial HK Value Benchmark is that it is more aggressively adversarial, entirely region-specific, and explicitly grounded in Hong Kong and PRC law–consistent constraints (Han et al., 14 Jul 2025).

In a broader benchmark landscape, related work reinforces the importance of this design logic. "Flames: Benchmarking Value Alignment of LLMs in Chinese" (Huang et al., 2023) likewise emphasizes adversarial prompts, implicit malice, and culturally specific morality dimensions, although it is Chinese-language and not Hong Kong-specific. "Is your benchmark truly adversarial? AdvScore: Evaluating Human-Grounded Adversarialness" (Sung et al., 2024) argues that adversarial datasets can become obsolete as models improve and proposes a human-grounded metric for tracking whether a dataset remains harder for models than for humans. This suggests that long-term maintenance, recalibration, and adversarial refresh cycles may be especially important for region-specific value benchmarks, although the HK paper itself does not formalize such a lifecycle (Sung et al., 2024).

The benchmark’s limitations are stated with unusual directness. Its scale is limited to 300 items, making it useful for red-teaming but not exhaustive. The paper acknowledges “the inherent subjectivity in ‘aligning to Hong Kong values,’” noting that local annotators may not represent the full spectrum of views in a diverse and dynamic society. Coverage is also partial: only certain classes of sensitive content and attacks are included, and emerging issues or new legal interpretations will require updates (Han et al., 14 Jul 2025).

At the same time, the paper presents the benchmark as part of a replicable blueprint for regionally focused AI systems. The transferable design principles include region-specific normative grounding in local law and public discourse, adversarial rather than merely descriptive prompt construction, human-centered judgment by local annotators and social science experts, explicit separation of refusal templates from substantive safe answers, and tight integration with governance and RAG frameworks. Under that reading, the Adversarial HK Value Benchmark is both an evaluation artifact and an operational model of sovereign alignment auditing for a multilingual, legally hybrid, and politically sensitive environment (Han et al., 14 Jul 2025).