AIReg-Bench: EU AI Compliance Benchmark

• AIReg-Bench is a benchmark dataset that evaluates LLM performance in assessing compliance with specific provisions of the EU AI Act.
• It employs an LLM-driven pipeline to generate synthetic yet plausible technical documents, which are then rigorously annotated by legal experts.
• Empirical results show models like Gemini 2.5 Pro achieving high agreement with expert ratings, with metrics such as a Cohen’s kappa of 0.863 and a 60% exact match rate.

Searching arXiv for AIReg-Bench and closely related benchmarking work to ground the article in cited papers. AIReg-Bench is a benchmark dataset for evaluating whether LLMs can assess compliance with the European Union AI Act. It is presented as the first benchmark dataset designed to test how well LLMs can assess compliance with the EU AI Act, and it targets a concrete legal-reasoning task: given a technical documentation excerpt and a specified Article of the Act, a model must estimate the probability that the described AI system is compliant. The benchmark couples synthetic-yet-plausible documentation with expert legal annotation, thereby creating a standardized setting for comparing frontier and domain-specific LLMs on fine-grained AI regulation compliance assessment (Marino et al., 1 Oct 2025).

1. Definition and objectives

AIReg-Bench was created to address the absence of a benchmark for LLM-based assessment of AI regulation compliance. Its stated objectives are to provide the first open benchmark dataset for quantifying LLM performance in AI regulation compliance, specifically for the EU AI Act; to enable systematic, comparative evaluation of LLMs on fine-grained legal assessment tasks; and to facilitate further research in automating or supporting regulatory compliance processes while promoting transparency around LLM strengths and weaknesses in legal reasoning (Marino et al., 1 Oct 2025).

The benchmark is centered on compliance assessment rather than general legal QA or statutory retrieval. The task formulation requires a model to judge whether a described AI system complies with a specific provision of the AI Act, using the same contextual materials given to human annotators. This makes AIReg-Bench a benchmark for ordinal legal judgment under constrained evidence, not merely for text classification.

A common misconception is that AIReg-Bench evaluates broad “AI law understanding” in the abstract. Its actual scope is narrower and more operational: it measures model agreement with expert judgments on selected provisions of the EU AI Act, applied to technical documentation excerpts for high-risk use cases. This suggests that the benchmark is best understood as a controlled testbed for compliance assessment behavior rather than a complete proxy for regulatory practice.

2. Dataset construction and document generation

AIReg-Bench contains 120 technical documentation excerpts simulating the kind of records an AI provider would submit to demonstrate compliance with specific AI Act requirements. Construction follows a two-step process. First, an LLM-driven pipeline generated the excerpts. Second, legal experts reviewed and annotated them (Marino et al., 1 Oct 2025).

The generation pipeline uses OpenAI GPT-4.1-mini as the generation engine and is described as multi-stage. In the first stage, the model produces realistic high-level descriptions of fictional but plausible AI systems, each mapped to a high-risk use case under the Act. Examples named in the source description include road traffic control, recruitment, and credit scoring. In the second stage, for each system and selected Article, the model generates “compliance profiles” that specify plausible ways the system might comply or breach the requirements of that Article. These profiles are intended to produce subtle and realistic non-compliance rather than trivial violations. In the third stage, the model uses the system description, targeted Article, and compliance profile to generate a technical documentation excerpt in a professional, audit-ready style.

The pipeline was validated and refined through expert interviews and iterative human review to ensure realism and diversity. Plausibility validation is itself formalized: each generated excerpt was rated by three legal experts for plausibility on a Likert scale from 1 to 5, where 1 denotes very low and 5 denotes very high plausibility. Excerpts were retained only if judged realistic and professional, plausibly emulating documentation a compliance assessor might encounter. The reported median plausibility score is 4.

The dataset scope is fixed and explicit.

Component	Content
Samples	120 technical documentation excerpts
Use cases	8 distinct high-risk applications
AI Act Articles	9, 10, 12, 14, 15
Annotation regime	3 legal experts per excerpt
Label steering	Approximately 1/3 toward compliance, 2/3 toward non-compliance

The use of generated documentation is a notable design choice. The source explicitly states that the underlying documentation is LLM-generated, plausibility-validated, and motivated by the lack of real-world samples. A plausible implication is that AIReg-Bench prioritizes controlled coverage and reproducibility over direct capture of real provider documentation.

3. Annotation protocol and ground truth

Each excerpt is annotated by three legal experts, described as law students or graduates or qualified lawyers. The annotators independently rate the probability that the system described in the excerpt complies with a specific AI Act Article—Articles 9, 10, 12, 14, or 15—using a 1–5 Likert scale, where 1 indicates very low probability of compliance and 5 indicates very high probability of compliance (Marino et al., 1 Oct 2025).

The annotation process includes qualitative justifications accompanying each score. These justifications provide insight into annotator reasoning and make the benchmark useful not only for scalar evaluation but also for examining the structure of legal disagreement. Plausibility scores are also collected during the broader process, and divergences and difficult-to-score cases are explicitly marked.

For evaluation purposes, the median annotator score is used as the ground truth. This is significant because the target is not a binary statutory label but an ordinal compliance judgment aggregated across experts. The benchmark therefore encodes both legal uncertainty and the practical need to collapse expert assessments into a single comparison target.

The distribution of ratings reflects substantial variation. The source gives an annotation summary in which each of the 120 excerpts receives three scores, for 360 total ratings. It also reports that approximately one third of samples were steered toward compliance and two thirds toward non-compliance during generation, while emphasizing that final labels reflect genuine human adjudication. This distinction matters: steering influences the generation process, but the benchmark labels are determined by expert assessment rather than by generation intent.

4. Evaluation task and metrics

In the benchmark evaluation, an LLM is given the same system description, the relevant AI Act Article in full text and by number, and the documentation excerpt. The model must answer the question, in effect, “What is the probability this is compliant with the Article?” on the same 1–5 Likert scale used by the human annotators, and it must explain its choice (Marino et al., 1 Oct 2025).

The benchmark imposes an important restriction: LLMs do not have access to external sources, unlike human experts. They rely only on the excerpt and the provided Article text. This constrains the task to in-context statutory interpretation and document assessment, rather than open-book legal research.

Model outputs are compared to the median human expert rating using several ordinal and error-sensitive metrics: quadratic weighted Cohen’s kappa, Spearman’s rank correlation, mean signed difference as bias, mean absolute error, exact match rate, overestimate and underestimate rates, and binary F1 when scores 4–5 are grouped against 1–3. The paper also reports confusion matrices and MAE heatmaps, including per-use-case and per-article analyses.

The choice of metrics is technically aligned with the benchmark target. $\kappa_w$ and $\rho$ capture ordinal agreement and rank consistency, while bias and MAE make it possible to distinguish systematic leniency or severity from simple dispersion. The exact-match and directional-error statistics then expose whether a model tends to overshoot or undershoot expert judgments.

5. Empirical findings

The reported results show that frontier LLMs achieve substantial, but not uniform, agreement with human legal experts. The source highlights Gemini 2.5 Pro as the strongest overall model, with quadratic weighted Cohen’s kappa of 0.863, Spearman’s $\rho$ of 0.856, bias of -0.225, and MAE of 0.458. GPT-5 is reported at $\kappa_w=0.849$, $\rho=0.838$, bias $=-0.067$, and MAE $=0.450$. GPT-4o, o3, and o3 mini perform less strongly on agreement metrics, and fine-tuned legal LLMs such as Saul-7B-Instruct and Saul-54B-Instruct are reported to perform notably lower than large general-purpose or frontier LLMs (Marino et al., 1 Oct 2025).

Model	$\kappa_w$	MAE
Gemini 2.5 Pro	0.863	0.458
GPT-5	0.849	0.450
GPT-4o	0.775	0.558
o3	0.723	0.658
o3 mini	0.624	0.775

The benchmark summary further reports that Gemini 2.5 Pro was within one point of the human median for all but 7 of 120 samples, and that it matched the human expert rating exactly in 60% of cases. Across models, overestimation appears more common than underestimation: the source notes that most LLMs more often overestimated than underestimated compliance, and identifies noticeably more lenient or “sycophantic” behavior in models such as o3 mini and GPT-4o.

Ablation results are also informative. Removing access to the actual Article text or modifying prompts to elicit harsher judgments both negatively affected LLM performance. This indicates that benchmark performance depends materially on direct exposure to the governing provision and on prompt framing. A plausible implication is that regulatory assessment performance is not reducible to generic legal priors; it is sensitive to how statutory context is surfaced to the model.

The benchmark also includes a price–performance perspective: Gemini 2.5 Pro and Grok 3 mini are reported to offer the best tradeoff between price and performance on the Pareto frontier.

6. Scope, limitations, and position within the benchmark landscape

AIReg-Bench is explicitly presented as a standardized, expert-validated testbed for detailed legal compliance assessment. At the same time, the paper states several limits on validity and scope. It captures only a slice of the EU AI Act under current legal understanding, and it may need to evolve as guidelines and standards mature. Its documentation is synthetic rather than drawn from real provider submissions, although the documents were plausibility-validated by experts. It also covers only five Articles—9, 10, 12, 14, and 15—and eight high-risk use cases (Marino et al., 1 Oct 2025).

These design choices distinguish AIReg-Bench from other benchmarking traditions. AI Matrix generates synthetic DNN benchmarks from profiled workload characteristics in order to represent hardware-relevant compute behavior (Wei et al., 2018). SAIBench formalizes scientific AI benchmarking through modular decoupling of problems, models, ranking criteria, and software/hardware configurations via the SAIL eDSL (Li et al., 2022). AgentRecBench evaluates LLM agent-based personalized recommender systems using an interactive textual recommendation simulator, three recommendation scenarios, and a unified modular agent framework (Shang et al., 26 May 2025). By contrast, AIReg-Bench is neither a systems benchmark nor a general modular benchmarking substrate; it is a domain-specific legal assessment benchmark with expert-adjudicated ordinal labels.

The benchmark’s open-source release broadens its research utility. The source states that the repository includes the full dataset, annotation guidelines, prompts, and evaluation code at https://github.com/camlsys/aireg-bench. The paper also presents the pipeline as extensible to new use cases, AI Act Articles, and jurisdictions. This suggests a broader role for AIReg-Bench as a template for future compliance benchmarks, including evaluations of retrieval-augmented, tool-using, or fine-tuned systems, provided that the distinction between synthetic construction and real-world legal validity remains explicit.