CompliBench: Multi-Domain Compliance Benchmark

• CompliBench is a suite of three domain-specific benchmarks designed to evaluate compliance in dialogue systems, Android code auditing, and interval arithmetic libraries.
• It employs adversarial data generation, expert annotations, and composite metrics to ensure rigorous, reproducible performance assessments.
• The framework enables detailed error analysis and generalizability checks, enhancing practical applications for LLM judges, legal compliance, and numerical computations.

CompliBench is the designation for three distinct, domain-specific benchmarks designed for rigorous, reproducible evaluation of machine reasoning and software systems compliance along technical, regulatory, or numerical dimensions. While each benchmark targets a different methodological regime—dialogue system policy adherence, cross-jurisdictional legal compliance in code, and interval arithmetic library correctness—they are unified by explicit ground-truth alignment, adversarially designed or expert-annotated tasks, and a commitment to reproducibility and error localization. This article surveys the principal CompliBench frameworks established in dialogue compliance, software law auditing, and interval arithmetic, drawing on their published definitions and empirical findings.

1. CompliBench for Dialogue Compliance Violation Detection

The CompliBench framework introduced in "CompliBench: Benchmarking LLM Judges for Compliance Violation Detection in Dialogue Systems" (Yang et al., 14 Apr 2026) is a systematic, adversarially constructed benchmark targeting the evaluation of LLM-based judges (LLM-as-a-Judge) for multi-turn, task-oriented dialogues typical in enterprise agents.

Benchmark Scope and Taxonomy

CompliBench spans three business domains (Airline, Healthcare, Insurance), aggregating 318 conversations (avg. 16.6 turns, 3.7 violations/conversation). Guidelines for agent behavior are organized into:

• Universal Compliance (e.g., "always show empathy"),
• Workflow Guidelines: ordered task steps for structured interactions,
• Condition Guidelines: contextually gated rules ("if–then" dependencies).

At each agent turn $r_i$, judges are to emit a pair $\hat{y}_i = (\hat{g}_i, \tilde{v}_i)$, identifying the governing guideline and signaling binary violation status.

Automated Data Pipeline and Flaw Injection

Data is generated via a three-stage pipeline:

• Guideline Expansion: Robust scenario coverage via guideline permutation.
• Violation Induction: Probabilistic replacement of 30% of workflow and up to one condition guideline per conversation by algorithmically generated, mutually exclusive violation variants. Precise turn-level labels $y_i = (g_i, v_i)$ are automatically assigned. Flaw injection is formalized (see pseudocode in source) for transparency and replicability.
• Dialogue Simulation: Staged agent/user simulation, in which non-violation turns are regenerated if multiple LLM judges do not confirm compliance.

Adversarial search identifies violation variants with measurable behavioral divergence that are hard for LLMs to detect, employing a judge-and-refine loop with feedback propagation and up to 3 rounds of generation (see AdversarialJudgeRefine algorithm in (Yang et al., 14 Apr 2026)).

Evaluation Metrics

• Strict Guideline Accuracy (SGA): Fraction of compliant turns with exact guideline match and no violation.
• Violation Detection Accuracy (VDA): Fraction of truly violated turns correctly flagged, ignoring guideline ID.
• Conversation-Level Accuracy (CLA): Fraction of dialogues with perfect per-turn assessment.

Frontier proprietary LLMs (GPT-5, Gemini-3-pro, Claude-sonnet-4-6) achieve SGA ≈90% but much weaker VDA (50–95% domain- and model-dependent) and low CLA (10–57%). Fine-tuned mid-scale judges (Qwen3-8B) using distilled rationales achieve balanced metrics—SGA ≈88–89%, VDA ≈86–89%, and higher CLA (51.5% Airline, ≥39% Healthcare/Insurance)—and robust generalization to unseen domains, highlighting the impact of pipeline diversity and adversarial perturbation.

Error Analysis

Dominant failure mode is "scope mis-attribution": judges over-penalize compliant turns (overly strict) but under-detect subtle violations (lenient).

2. CompliBench for Android Code Compliance Auditing

"Can LLMs Detect Real-World Android Software Compliance Violations?" (Zhang et al., 1 Nov 2025) defines a CompliBench instance that evaluates LLMs' capability to identify privacy regulation violations in real Android applications, aligning code spans with statutory articles from Brazil's LGPD, Singapore's PDPA, and Canada's PIPEDA.

Benchmark Construction

• Anchors: 791 file-level, 792 module-level, 840 line-level, and 840 snippet-based records sourced from eight open-source Android applications, each with immutable provenance (commit-pinned).
• Labeling: Expert jurists map each anchor to one or more explicit legal articles, e.g., "Art. 7", "§ 4.3", conservatively retaining statutory granularity.

Evaluation Tasks

• Task 1 (Retrieval/Localization): Given a repository and regulatory context, models rank candidate file/module/line spans; scored on IR metrics (Precision@k, MRR, MAP, nDCG@5).
• Task 2 (Multi-label Judgment): Given a code snippet, models predict article(s) implicated; scored by multi-label measures (Micro/Macro/Weighted F1, Jaccard, Hamming, normalized coverage).

Standard metrics are supplemented by four stability-aware composites, penalizing inconsistent performance across granularities, statutes, or task types:

• SGS: Harmonic mean of retrieval metrics across granularities, penalized by coefficient of variation.
• RCS: Mahalanobis TOPSIS-compressed regulation-wise consistency.
• CRGS: Cross-regulation geometric stability, penalizing variance.
• OCS: Overall coupled stability, aggregating per-law, per-task consistency.

Empirical Results

Among six LLMs evaluated, Claude-3.5-sonnet-20241022 achieves highest OCS (0.3295); GPT-4o moderately follows (0.2736); open-weight Qwen2.5-72b-instruct performs better on snippet-level reasoning than on retrieval; Gemini-2.5-pro ranks lowest. Top models exhibit per-law and per-task coupling, but law- and granularity-specific brittleness persists.

Operational Implications

CompliBench supports roll-out decisions via CRGS stability checks, and release gating via OCS to ensure uniform improvements. Retrieval is best treated as a candidate generation approach, with downstream thresholding and per-article calibration for judgment.

3. CompliBench for Interval Arithmetic Libraries

"A Cross-Platform Benchmark for Interval Computation Libraries" (Tang et al., 2021) presents CompliBench as a standard for verifying C/C++ interval arithmetic libraries across architectures and platforms for correctness, tightness, speed, and cross-platform reproducibility.

Test Suite and Methodology

• Expression Set: 28 handcrafted expressions (core arithmetic and transcendental ops, composite forms) plus 104 FPBench-derived real-world expressions.
• Correctness: Each test is validated by comparison to rational-arithmetic references: for $N=10^6$ random valid inputs, library intervals $[l,u]$ must enclose exact-computed ground-truth.
• Tightness: Interval width ratio distributions (library to ground-truth) are tabulated for each expression.
• Performance: Wall-clock time is measured for $1,000$ inputs $\times\, 10,000$ runs per expression.
• Consistency: Bit-for-bit output invariance is checked on Windows (MSVC), macOS (Clang x86/ARM), and Linux (GCC x86).

Implementations Evaluated

• Boost.Interval
• BIAS/PROFIL
• filib (bit-hack based)
• filib++ (multiple rounding/approximation modes)

Results

• Correctness: Only filib and filib++ (pred/mult modes) are fully correct on all transcendental and composite tests. Boost fails on transcendental and composite operations.
• Tightness: Direct rounding mode control produces tightest intervals on arithmetic-only expressions; conservative software rounding (multiplicative) yields wider intervals on transcendentals.
• Performance: Hardware rounding incurs high per-op cost; filib++ pred/mult modes are both fastest and most robust.
• Consistency: filib is the only implementation with cross-platform, bit-identical output.

4. Comparative Table of CompliBench Domains

Sub-benchmark	Target System	Core Task(s)
Dialogue Compliance (Yang et al., 14 Apr 2026)	Task-oriented LLMs	Per-turn violation detection and localization
Android Code Compliance (Zhang et al., 1 Nov 2025)	Android+LLMs	Code-relevant law retrieval, article mapping
Interval Arithmetic (Tang et al., 2021)	Numeric libraries	Correctness, tightness, and performance eval

5. Methodological and Practical Significance

CompliBench establishes methodological standards for:

• Ground-truth aligned, adversarial, and/or expert-labeled data generation, enabling fine-grained error analysis and mitigation (e.g., scope mis-attribution, granularity-variant coupling).
• Composite metric design (e.g., OCS/CRGS/SGS) that penalizes instability, a recognized gap in IR and multi-label evaluation for software compliance.
• Intrinsic and extrinsic model comparison: Effective for benchmarking both off-the-shelf LLMs and supervised/fine-tuned models, delivering diagnostic value for model pre-deployment.
• Generalizability assessment: E.g., task-focused judgers trained on airline dialogue generalize to healthcare and insurance compliance; OCS/CRGS surfaces inter-jurisdictional generalization in code compliance.

A plausible implication is that structured, compositional benchmarks of this type expose subtle model weaknesses (e.g., under-detection or over-penalization) not captured by legacy scalar metrics.

6. Limitations and Prospective Extensions

CompliBench instances remain domain-anchored (enterprise dialogue, Android, C/C++) and intentionally eschew train/test splits in favor of intrinsic evaluation. Limitations include:

• Coverage (e.g., current Android and interval arithmetic benchmarks are not extended to web/IoT platforms or programming languages beyond C/C++, Android/Java).
• Manual semantic validation for interval arithmetic and code compliance, with future prospects including automated test/symbolic equivalence checking.
• Statistical power, particularly for rare error types in relatively compact datasets (e.g., 200 interval arithmetic cases).

Extending CompliBench frameworks to additional domains (e.g., GDPR/CCPA compliance, Rust/Java error repair, compositional digital policy verification) and leveraging static/dynamic program analysis as priors for retrieval tasks represent active frontiers. Increasing dataset scale, automating deeper semantic validation, and hybridizing human-LLM workflow in error attribution and compliance evaluation are anticipated future directions (Yang et al., 14 Apr 2026, Zhang et al., 1 Nov 2025, Tang et al., 2021).