PolicyGuardBench: Policy-Compliant Benchmark

• PolicyGuardBench is a dual-usage framework featuring a web-agent trajectory violation dataset and a multilingual safety evaluation system grounded in explicit policy texts.
• It standardizes and normalizes agent actions to enable sequence-aware policy checking, employing heuristic and LLM-based annotation for precise violation labeling.
• The benchmark shows high efficiency with PolicyGuard-4B achieving over 90% accuracy at low latency, while supporting out-of-domain generalization and cross-jurisdictional compliance.

Searching arXiv for the cited work and closely related benchmark papers to ground the article. PolicyGuardBench is a policy-conditioned benchmarking concept whose meaning depends on the research context in which it is used. In its most explicit sense, it denotes a benchmark of about 60k examples for detecting policy violations in autonomous web-agent trajectories, including both full-trajectory and prefix-based anticipation tasks, together with the lightweight PolicyGuard-4B detector (Wen et al., 3 Oct 2025). In a second usage, PolicyGuardBench refers to a regulation-aware multilingual safety framework instantiated by ML-Bench and ML-Guard, where native legal texts define the evaluation and enforcement substrate across languages and jurisdictions (Zhao et al., 1 May 2026). Related work on in-context policy guardrailing, policy-adaptive image guardrails, reinforcement-learning backdoor defense, document compliance review, and moderator evaluation defines adjacent benchmark structures, but several of those papers do not introduce “PolicyGuardBench” as a formal artifact name (Zhang et al., 29 Jun 2026, Song et al., 7 Jul 2026, Huang, 11 Jun 2026, Malik et al., 30 Jun 2026, Machlovi et al., 22 Dec 2025).

1. Nomenclature and conceptual scope

The term has a split identity in the recent literature. In "Towards Policy-Compliant Agents: Learning Efficient Guardrails For Policy Violation Detection" (Wen et al., 3 Oct 2025), PolicyGuardBench is the named benchmark itself: a dataset of trajectory–policy pairs designed to determine whether a complete or partial agent trajectory violates externally imposed or human-specified rules. The task is explicitly trajectory-level rather than single-step, because violations can emerge cumulatively through long-horizon action composition, ordering constraints, or conditional rules.

In "ML-Bench&Guard: Policy-Grounded Multilingual Safety Benchmark and Guardrail for LLMs" (Zhao et al., 1 May 2026), PolicyGuardBench is used as a higher-level description for a framework rather than as the paper’s formal benchmark name. There, ML-Bench is the policy-grounded benchmark and ML-Guard is the guardrail system; together they instantiate a regulation-aware, multilingual safety evaluation and enforcement stack grounded in native, jurisdiction-specific policies.

This suggests that PolicyGuardBench is best understood not as a single immutable benchmark family, but as a recurring design pattern: policy text is made explicit, evaluation is conditioned on that policy, and guardrails are expected to execute policy logic rather than rely only on fixed hazard taxonomies.

2. PolicyGuardBench for web-agent trajectory compliance

The explicit PolicyGuardBench of the web-agent literature is built around autonomous browser trajectories collected from ScribeAgent operating in WebArena (Wen et al., 3 Oct 2025). The full construction contains 314,556 trajectory–policy pairs, built from 733 standardized trajectories and 2,195 curated policies, with 28.2% of pairs designated as cross-subdomain target cases. The curated balanced benchmark contains 59,997 pairs, comprising 35,012 source pairs and 24,985 target pairs; 41.6% are cross-subdomain, and the label distribution is 25,435 violations and 34,562 non-violations. The average standardized trajectory length is 9.3 actions.

The benchmark spans five domains: Reddit, Map, GitLab, Shopping, and Shopping_Admin. Reddit and Map are single-subdomain domains, whereas GitLab, Shopping, and Shopping_Admin support within-subdomain and cross-subdomain pairing. Policies are synthesized from diverse agent runs and then curated for clarity and checkability. Each policy includes a schema with source_subdomain and up to two target_subdomains in the same domain, which supports both in-domain transfer and cross-subdomain generalization.

Trajectory standardization is central to the benchmark design. Raw browser logs contain clicks, form inputs, scrolls, URL navigations, and status updates. These logs are normalized by removing noise such as empty or duplicate events and rendering artifacts, mapping actions to a controlled verb set such as Click, Input, Scroll, Select, Navigate, and Submit, canonicalizing target objects, merging redundancy, and serializing the result as step-wise text. The resulting representation makes policy checking sequence-aware while remaining compatible with instruction-tuned LLM backbones.

Policy–trajectory pair formation combines embedding similarity through Sentence-BERT, keyword triggers, heuristic rules, and LLM-based scoring. Negative sampling pairs trajectories with policies from the same domain that they do not violate, and heuristics are used to prevent accidental false violations. Labels operationalize four recurring rule types: obligations, prohibitions, ordering constraints, and conditional rules.

3. Tasks, metrics, and evaluation protocol

PolicyGuardBench defines two tasks (Wen et al., 3 Oct 2025). The first is full-trajectory violation detection, where the input is a unified template containing policy text, serialized trajectory actions, and domain/subdomain metadata, and the output is a strict binary label: violation or no_violation. The second is prefix-based violation detection, in which violation trajectories are truncated to the first $N$ steps for $N = 1 \ldots 5$, re-matched with their policies, and re-labeled to test anticipatory detection under partial information.

The benchmark also includes a leave-one-domain-out protocol for out-of-domain generalization. One domain is held out for OOD testing while the model is trained on the remaining four. This makes the benchmark a generalization test rather than a pure memorization exercise over fixed policy phrasing or site patterns.

The reported metrics are Accuracy, F1, inference latency, and an efficiency-aware metric termed Efficiency-Adjusted F1. The definitions given are:

$$\mathrm{Acc} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}(\hat{y}_i = y_i)$$

$$\mathrm{Prec} = \frac{TP}{TP + FP}, \qquad \mathrm{Rec} = \frac{TP}{TP + FN}, \qquad \mathrm{F1} = 2 \cdot \frac{\mathrm{Prec} \cdot \mathrm{Rec}}{\mathrm{Prec} + \mathrm{Rec}}$$

and

$\mathrm{EA\mbox{-}F1} = \frac{\mathrm{F1} \cdot L_0}{\mathrm{Latency~(ms)}}, \qquad L_0 = 1000.$

The train/test split is reported as 49,997 / 12,000 in an 8:2 ratio. Annotation follows a two-stage process: sampled cases are first human-annotated to establish guidelines, then automated large-model annotation with gpt-oss-120B produces labels plus confidence scores, and low-confidence or inconsistent cases receive additional human review.

4. PolicyGuard-4B and empirical results

PolicyGuard-4B is the benchmark’s associated lightweight guardrail model (Wen et al., 3 Oct 2025). It uses Qwen3-4B-Instruct as the backbone, takes the unified policy–trajectory template as input, and emits a strict binary label token. Training uses the PolicyGuardBench training split with learning rate $1 \times 10^{-5}$, AdamW, a cosine scheduler, warmup ratio 0.1, 3 epochs, seed 42, 4 devices, per-device train batch size 2, gradient accumulation 8, effective train batch size 64, and deterministic decoding at evaluation with temperature 0. Reported hardware is H100 80GB GPUs.

On full-trajectory evaluation, PolicyGuard-4B achieves 0.9014 Accuracy, 0.8759 F1, and 22.5 ms latency. The comparison points reported alongside it are revealing. Llama-3.3-70B-Instruct reaches 0.9054 Accuracy and 0.8883 F1, but at 305.0 ms latency; Gemma-3-12B-IT reaches 0.8964 Accuracy and 0.8773 F1 at 51.3 ms; Qwen2.5-72B-Instruct reaches 0.8825 Accuracy and 0.8607 F1 at 205.0 ms. Safety-oriented guardrails transfer poorly to the task: LlamaGuard-7B reaches 0.4256 Accuracy and 0.5957 F1, while ShieldGemma-2B reaches 0.5735 Accuracy and 0.3317 F1.

Efficiency is a major claim of the benchmark-model pairing. PolicyGuard-4B is reported at 2.57 TFLOPs per example and an EA-F1 of 38.9289, compared with 17.1014 for Gemma-3-12B-IT, 2.9125 for Llama-3.3-70B, 4.1985 for Qwen2.5-72B, and 0.2387 for Qwen3-235B-A22B-Instruct-2507. A plausible implication is that the benchmark is structured tightly enough that explicit compliance supervision can substitute for brute-force scale.

The OOD results reinforce this interpretation. The average leave-one-domain-out ID performance is approximately 93.3% Accuracy and 93.2% F1, while OOD performance is approximately 90.8% Accuracy and 90.9% F1. Domain-wise OOD results are reported as GitLab 91.16/91.16, Map 90.20/90.78, Reddit 90.24/90.55, Shopping 91.74/91.37, and Shopping-Admin 90.79/90.44. On prefix-based detection, PolicyGuard-4B achieves 0.9101, 0.8648, 0.8441, 0.8276, and 0.8190 Accuracy at $N=1\ldots5$, with an average of 0.8531.

5. PolicyGuardBench as multilingual policy-grounded safety evaluation

In the multilingual-safety literature, PolicyGuardBench denotes a regulation-aware framework built from native, jurisdiction-specific policies and enforced through policy-conditioned guardrail models (Zhao et al., 1 May 2026). Here the benchmark component is ML-Bench and the guardrail component is ML-Guard.

ML-Bench covers 14 languages across 14 countries or regions and sources 17 regulatory texts. It is constructed directly from regional regulations rather than from general, language-agnostic hazard taxonomies. Article-level rules are extracted from official AI regulations and policy documents, de-duplicated and refined into atomic rules, grouped into language-specific risk categories, and kept in native languages throughout. No machine translation is used at any stage. The benchmark contains 56,000 instances split into 34,000 training instances and 22,000 evaluation instances, and includes multilingual seed queries, refined queries, attack-enhanced queries, and boundary-sensitive paired responses.

ML-Guard is a Diffusion LLM-based guardrail built on Fast_dLLM_v2_1.5B and Fast_dLLM_v2_7B. The 1.5B variant is a lightweight binary checker for fast safe/unsafe classification, while the 7B variant is a policy-conditioned compliance assessor that outputs four lines: JUDGE, VIOLATION, VIOLATED_RULES, and RATIONALE. When rules are provided, decisions are grounded on the specified policy; when rules are absent or unseen, the model falls back to general hazard categories.

The evaluation claims are strong. Against 11 strong guardrail baselines across ML-Bench and six existing multilingual safety benchmarks, ML-Guard-7B achieves F1 = 0.97 on Seed queries, 0.90 on Refined queries, 0.61 on Responses, and 0.92 attack-enhanced accuracy on ML-Bench. Across PolyGuardPrompts, XSafety, RTP-LX, MultiJail, CSRT, and Nemotron, ML-Guard-7B reports an average F1 of 0.81, while ML-Guard-1.5B reports 0.79. In this usage, PolicyGuardBench names a policy-grounded multilingual compliance stack rather than a trajectory benchmark.

6. Adjacent benchmark families and non-equivalent constructs

Several neighboring projects sharpen the conceptual boundary of PolicyGuardBench without using the name in the same way.

SafePyramid studies in-context policy guardrailing, where a model receives an application-specific policy at inference time and must output the violated-rule set for a multi-turn conversation (Zhang et al., 29 Jun 2026). It contains 1,000 conversations across 10 domains, 3,000 corresponding policies, and 61,699 distinct natural-language rules, organized into L0, L1, and L2 difficulty levels. Its primary metrics are Rule Matching Rate and Rule Disagreement Rate, and it emphasizes exact recovery of violated-rule sets rather than binary moderation.

PolicyShiftBench addresses policy-adaptive image guardrailing (Song et al., 7 Jul 2026). It contains 2,000 policy-discriminative instances over 265 images, with 7.55 policy-conditioned prompts per image on average, 28 evaluation policy variants, and a policy sensitivity metric termed PSS. The paper explicitly states that the benchmark is named PolicyShiftBench, not PolicyGuardBench.

The reinforcement-learning paper "PolicyGuard: Towards Test-time and Step-level Adversary Defense for Reinforcement Learning Agent" does not introduce a benchmark called PolicyGuardBench, but it does specify a de facto evaluation suite spanning seven games, perturbation-based and adversary-agent attacks, AUROC as the principal metric, and hard-coded as well as adaptive attacks (Huang, 11 Jun 2026). Similarly, the document-compliance paper "PolicyGuard: From Organizational Policies to Neuro-SymbolicCompliance Review Engines" does not use the term as a formal artifact name, but its evaluation harness comprises 95 policy guidelines, five real NDAs, 475 policy–contract decisions, Z3-based symbolic evaluation, and pass$^k$ reliability reporting (Malik et al., 30 Jun 2026). GuardEval is broader still: 166,488 prompts, 64,484 responses, 23 coarse categories, and 106 fine-grained subcategories for evaluating safety, fairness, and robustness in LLM moderators, and it is presented as a possible foundation for a PolicyGuardBench-style evaluation stack rather than as the named benchmark itself (Machlovi et al., 22 Dec 2025).

7. Limitations and research directions

Across its explicit and extended usages, PolicyGuardBench inherits the limitations of policy-grounded benchmarking more generally. In the web-agent benchmark, coverage is restricted to five domains, some domains have single subdomains, base trajectory diversity is limited to 733 standardized trajectories, policy diversity may under-represent rare or highly specialized rules, inter-annotator agreement is not reported, and very intricate temporal or multi-session policies remain outside the current scope (Wen et al., 3 Oct 2025). In the multilingual framework, coverage spans 17 regulatory sources and 14 languages but cannot cover all jurisdictions or domains, LLM-based rule extraction and annotation may reflect upstream model biases, and continuous curation is required as regulations evolve (Zhao et al., 1 May 2026).

Related benchmarks report analogous boundaries. SafePyramid is text-only and does not provide a human baseline in its first version; PolicyShiftBench is limited to static images and primarily English policy bundles; the RL PolicyGuard evaluation assumes access to a clean environment and focuses on detection rather than mitigation; the NDA-compliance PolicyGuard study is constrained by proprietary data and the absence of a shared document-level fact graph (Zhang et al., 29 Jun 2026, Song et al., 7 Jul 2026, Huang, 11 Jun 2026, Malik et al., 30 Jun 2026). This suggests that future PolicyGuardBench-style systems will likely expand along four axes already named in the literature: broader domains and subdomains, stronger treatment of policy updates and cross-jurisdictional variation, multimodal evaluation, and tighter integration between explicit policy execution and deployable guardrail intervention.