MASLegalBench: Multi-Agent Legal Benchmark

• MASLegalBench is a benchmark that evaluates multi-agent systems in deductive legal reasoning by extending the traditional IRAC framework for GDPR enforcement cases.
• It operationalizes task decomposition, role specialization, and retrieval-augmented generation to align legal facts with statutory rules using real enforcement reports.
• The framework demonstrates improved accuracy and reduced refusal rates by effectively combining distinct reasoning agents for facts, legal rules, application, and common-sense inference.

MASLegalBench is a benchmark specifically designed to evaluate multi-agent systems in deductive legal reasoning. It targets the General Data Protection Regulation (GDPR) domain and operationalizes an extended IRAC framework—Issue, Rule, Application, Conclusion, plus Common Sense—to reflect how legal experts decompose and resolve case questions. Its stated motivation is that prior legal benchmarks for foundation models and LLM agents did not explicitly address the distinctive advantages of multi-agent systems, including task decomposition, agent specialization, and flexible training; MASLegalBench therefore frames legal reasoning as a coordinated, role-based process grounded in real enforcement materials and retrieval-accessible background knowledge (Jing et al., 29 Sep 2025).

1. Conceptual lineage and benchmark rationale

MASLegalBench is best understood as a MAS-specific extension of the legal benchmarking line associated with LegalBench. LegalBench proposed that legal reasoning benchmarks should be organized through IRAC, a framework used by legal scholars to distinguish Issue, Rule, Application, and Conclusion, and it introduced a seed set of 44 tasks built according to that structure. It also framed legal benchmarking as an open, collaborative project between computer science and legal communities (Guha et al., 2022).

MASLegalBench inherits that decompositional logic but adapts it to a setting in which multiple agents can specialize in distinct reasoning functions. Its benchmark design is explicitly motivated by three persistent challenges for single LLM agents in law: inconsistent multi-step reasoning, weak grounding of abstract legal provisions in concrete facts, and gaps or hallucinations in domain or common-sense knowledge. In response, it provides a MAS-adapted task structure that supports role specialization and recursive task decomposition, grounds evaluation in real court enforcement reports with rich background knowledge segmented for retrieval, and measures how additional agents affect deductive performance, refusal behavior, and inter-agent agreement (Jing et al., 29 Sep 2025).

This positioning makes MASLegalBench narrower in legal coverage than broad legal NLP suites, but more explicit about reasoning architecture. A plausible implication is that the benchmark is intended less as a general legal-language leaderboard than as an instrument for diagnosing how agentic decomposition alters the quality of legal deduction.

2. Corpus, legal scope, and knowledge representation

The legal focus is GDPR enforcement cases in the UK category collected from the GDPR Enforcement Tracker. The scenario includes violations and compliance assessments under articles such as Article 5(1)(f) on integrity and confidentiality, Article 5(2) on accountability, Article 32(1)–(2) on security of processing, and Article 33(1) on breach notification (Jing et al., 29 Sep 2025).

The dataset comprises 950 multiple-choice questions derived from 15 distinct enforcement cases. Of these, 647 are yes/no questions and 303 are single-choice questions with four options, labeled A–D. The questions span three stated types: compliance checks, legal interpretations, and case analyses. Compliance checks ask whether conduct violates a specific GDPR provision; legal interpretations ask about the meaning or level of obligations, such as the level of security measures required by Article 32; and case analyses ask whether obligations such as the notification requirements of Article 33 were met.

The underlying source material is deliberately document-rich. Each source PDF is an expert-authored enforcement report parsed into a hierarchical outline that may include introduction, legal framework, background, nature of the entities, findings of infringement, decision, penalty calculation, and annex. These sections are used because they mirror how regulators reason from facts to legal conclusions. The PDFs range from 30 to 153 pages, with an average of 59.80 pages, and are segmented into minimal text chunks ranging from 67 to 439 per file, with an average of 185.53. Average section lengths in chunks are reported as follows: Introduction 7.13, Legal Framework 18.27, Background 25.27, Nature 18.29, Infringements 59.27, Decision 28.93, Penalty 22.40, and Annex 8.46 (Jing et al., 29 Sep 2025).

The benchmark’s background knowledge is partitioned into four reasoning supports. Facts (F) are excerpts from background and infringements sections; Legal Rules (LR) are excerpts from related legal framework and decision sections that cite articles; Application/Alignment (AR) consists of explicit relationships that map entities and concepts to legal roles and classifications, as well as rule-fact correspondences; and Common Sense (CS) consists of inferred alignments strictly derivable from the text, with no external speculation. An IRAC distribution analysis across the 15 cases shows rules concentrated in fewer chunks and facts distributed across longer sections, reflecting real case structure.

3. Extended IRAC and deductive reasoning structure

MASLegalBench operationalizes deductive reasoning through extended IRAC rather than through a mandated symbolic or deontic logic. The benchmark emphasizes deduction by mapping facts to rules, aligning legal roles and classifications to entities, and bridging residual gaps through constrained common-sense inferences. The paper states that it does not enforce a specific deontic logic; instead, it uses rule-based deduction aligned to statutory text and case facts (Jing et al., 29 Sep 2025).

The local rules used in the benchmark are expressed schematically. Two examples given are:

$$\text{SensitiveData} \wedge \neg \text{AppropriateMeasures} \Rightarrow \text{Violation}(5(1)(f))$$

and

$$\text{HighRiskProcessing} \Rightarrow O(\text{AppropriateMeasures}),$$

where $O(\cdot)$ denotes a normative obligation.

These formulas clarify the benchmark’s conception of legal deduction. Facts by themselves are insufficient; they must be linked to legal predicates such as “SensitiveData” or “HighRiskProcessing,” then assessed against statutory obligations or prohibitions. The AR component captures those legal-role mappings explicitly, while the CS component supplies only those additional alignments that are strictly derivable from the text.

The benchmark includes worked case examples from Birthlink. In one example involving Articles 5(1)(f) and 32, the issue is that a charity processed highly sensitive personal data, including sentimental items such as letters and photographs, and failed to ensure appropriate security. The rule is that Article 32(1)–(2) requires appropriate technical and organizational measures commensurate with risk, while Article 5(1)(f) requires integrity and confidentiality. The application step instantiates the schematic rules above, and the common-sense step states that if records are irreplaceable and destruction occurred through processing, then the risk level is high and security obligations are stringent. The conclusion is a violation of GDPR, and the MCQ answer is “Yes.” A second Birthlink example concerns Article 33(1): failure to notify the Commissioner within 72 hours and failure to implement measures to establish whether a breach occurred are treated as a violation, again yielding the MCQ answer “Yes” (Jing et al., 29 Sep 2025).

4. Agent architecture, retrieval, and evaluation protocol

The benchmark’s reference architecture is role-based. A Meta-LLM serves as planner and integrator: it recursively decomposes a case or question into atomic sub-tasks, assigns them to specialized agents, and integrates their outputs into a final answer. The specialized agents are $A_{\text{facts}}$ for Facts, $A_{\text{rule}}$ for Legal Rules, $A_{\text{analysis}}$ for Application/Alignment, and $A_{\text{common\_sense}}$ for Common Sense. The AR agent extracts explicit rule-fact alignments and legal roles or classifications for entities such as controller, processor, supervisory authority, filing system, personal data, and special category data, as well as relations among entities and concepts stated in the text. The CS agent extracts only those inferred alignments strictly derivable from existing alignments and relations, without external speculation (Jing et al., 29 Sep 2025).

The interaction protocol is prompt-based rather than fine-tuned. Algorithmic outlines for recursive decomposition and agent assignment guide Meta-LLM behavior and are intended to ensure that sub-tasks are atomic, meaning solvable in one reasoning step. Agents are steered by domain-specific prompt templates to produce JSON outputs, including alignment lists, relations, and inferred alignments. The Meta-LLM returns a final JSON object with “rationale” and “answer.” No fine-tuning is used; all agents operate via prompting. Sub-tasks for LR and F can be simulated by direct segmentation of source data, whereas AR and CS require additional agent processing.

Retrieval-augmented generation is central to evaluation. Two retrievers are used: BM25 as a lexical retriever and an embedding-based retriever built on sentence-transformers/all-MiniLM-L6-v2. Retrieval settings are denoted by the notation search@hit, such as BM25@3 for the top three lexical hits or EMB@5 for the top five embedding-based hits. The Meta-LLM must return one of A/B/C/D/Yes/No or refuse.

Accuracy is the primary metric:

$$\text{Accuracy} = \frac{\sum_{i=1}^{N} \mathbb{1}(\hat{y}_i = y_i)}{N}.$$

Refusal rate is also tracked when the Meta-LLM declines to answer for insufficient context, and agreement analysis is reported via Cohen’s Kappa heatmaps. The baselines include random choice without refusal, which yields 42.03% accuracy, and single-agent or RAG baselines using only LR, only F, and LR+F, without AR or CS. The Meta-LLM backbones evaluated are Llama3.1-8B-Instruct, Qwen2.5-7B-Instruct, Qwen3-8B, DeepSeek-v3.1, and GPT-4o-mini; all sub-task agents are implemented with DeepSeek-v3.1. Ablations systematically activate subsets of agents—F, LR, F+LR, AR, CS, AR+CS, F+LR+AR, and F+LR+AR+CS—and compare BM25 against embedding retrieval with top-$k \in \{1,3,5\}$ (Jing et al., 29 Sep 2025).

5. Quantitative findings, agreement patterns, and failure modes

The reported results show substantial variation across Meta-LLM choices and context configurations. Llama3.1-8B-Instruct attains the highest reported accuracy, 86.21%, under the CS configuration with EMB@5; it also performs strongly with LR at EMB@5, reaching 85.89, and with F at BM25@3, reaching 81.26. GPT-4o-mini achieves its best accuracies under LR, with 84.32 at EMB@5 and 82.84 at EMB@3; a multi-agent setting, F+LR+AR, reaches 82.06 under BM25@5. Qwen2.5-7B-Instruct reaches 76.95 under LR with EMB@5, while CS also improves performance under BM25, reaching 69.79 at top-3 and 73.16 at top-5. Qwen3-8B peaks at 70.32 under LR with EMB@5, and F+LR+AR improves relative to F or LR alone under BM25, reaching 62.32 at top-3 and 66.42 at top-5. DeepSeek-v3.1, when used as Meta-LLM, performs best under F+LR with EMB@5 at 64.32, while AR and AR+CS alone produce notably lower accuracies, including 24.00 for AR under BM25@1 (Jing et al., 29 Sep 2025).

The aggregate patterns are as important as the headline scores. The paper states that richer contexts and more agents generally improve accuracy, especially for larger models. It also reports that 44 of the 60 top-two results across contexts are achieved under MAS configurations that activate AR and/or CS, which is presented as evidence that deductive alignment and constrained common-sense inference are materially useful. Legal Rules and Common Sense agents are often present in best-performing settings, suggesting that they help mitigate knowledge gaps and hallucinations.

The main negative result concerns refusal behavior and narrow reliance on AR. For DeepSeek-v3.1 under BM25, AR alone yields refusal rates of 22.32% at top-1, 16.63% at top-3, and 14.00% at top-5; AR+CS yields 21.16%, 16.74%, and 14.21%, respectively. By contrast, F+LR+AR reduces refusal below F+LR, indicating that AR is beneficial when combined with fact and rule contexts rather than used in isolation. Cross-configuration agreement heatmaps show lower agreement between LR-only and F-only systems, while incorporating AR improves consistency and performance trends, summarized in the paper as $F{+}LR{+}AR > F{+}LR > F/LR$ under BM25. Statistical significance is not reported (Jing et al., 29 Sep 2025).

These findings support a precise interpretation of the benchmark’s difficulty profile. MASLegalBench is not simply testing whether a model can recall GDPR provisions. It is testing whether different forms of legally relevant evidence—facts, rules, alignments, and constrained inferences—can be coordinated without increasing abstention or internal inconsistency.

6. Position within legal AI, reproducibility, and limitations

MASLegalBench differs from earlier legal benchmarks in both scope and evaluation philosophy. LegalBench measures legal knowledge and reasoning broadly and centers the IRAC reasoning decomposition, but it is not MAS-specific (Guha et al., 2022). A neighboring line of work is SAMVAD, which models judicial deliberation in India through a Judge, Prosecution Counsel, Defense Counsel, and Adjudicators, all grounded via RAG in the Constitution of India, the Indian Penal Code, and the Code of Criminal Procedure. SAMVAD emphasizes dialogic reasoning, citation-grounded instructions and arguments, and consensus formation in multi-round deliberation (Devadiga et al., 4 Sep 2025). MASLegalBench instead focuses on GDPR grounding in real expert enforcement cases, explicit sub-task scaffolding through F/LR/AR/CS, recursive decomposition, role specialization, and deductive rule-fact alignment (Jing et al., 29 Sep 2025).

The benchmark is publicly accessible through the repository at https://github.com/HKUST-KnowComp/MASLegalBench, and its source cases come from the GDPR Enforcement Tracker. The benchmark and code are released publicly, the DeepSeek-v3.1 model card lists the MIT License, and the repository is the location for benchmark licensing details. Reproducibility support includes detailed experiment settings, prompts for agents and Meta-LLM, data construction procedures, retrieval pipelines for BM25 and embedding search, configuration scripts, and an appendix containing section statistics, IRAC mapping, and sample prompts and output formats. Agents output structured JSON, and the Meta-LLM outputs JSON containing “rationale” and “answer” (Jing et al., 29 Sep 2025).

Its limitations are sharply defined. The dataset covers only 15 cases and 950 MCQs, all drawn from GDPR UK enforcement reports; broader jurisdictions and additional GDPR articles would expand coverage. Question generation uses DeepSeek-v3.1 with human checks rather than purely manual drafting, and the reported 30-sample audit yields 92.22% faithfulness, 95.56% clarity, and 94.44% expertise. The benchmark does not enforce a formal deontic or symbolic calculus, which leaves consistency checks to the benchmark design rather than to a hard logical substrate. Systems that rely narrowly on AR can increase refusal rates. The study uses manually configured MAS rather than automatically designed workflows. Future directions stated in the paper include broader jurisdictional scope, more formalized legal logics, auto-designed agent workflows such as planner generation and role discovery, robustness to backdoors and prompt injections, consistency and faithfulness audits, and metrics beyond accuracy and refusal rate, including formal $$\text{HighRiskProcessing} \Rightarrow O(\text{AppropriateMeasures}),$$0 reporting, contradiction detection across agents, and robustness under adversarial retrieval (Jing et al., 29 Sep 2025).

Taken together, MASLegalBench occupies a specific position in legal AI evaluation: it is a benchmark for MAS-mediated legal deduction, grounded in GDPR enforcement materials, organized through extended IRAC, and designed to expose MAS-specific phenomena such as role complementarity, refusal behavior, and agreement structure rather than only single-model answer accuracy.