DEONTICBENCH: Evaluating Legal Reasoning

• DEONTICBENCH is a large-scale benchmark assessing deontic reasoning in legal and policy domains through explicit, rule-based tasks that combine symbolic and NLP methods.
• It covers 6,232 tasks across four domains with both numeric and binary evaluations, highlighting practical challenges in complex regulatory contexts.
• Its dual workflows—free-form chain-of-thought and solver-based approaches—facilitate transparent reasoning and precise, audit-traceable outcomes.

DEONTICBENCH is a large-scale, multi-domain benchmark explicitly dedicated to evaluating and advancing machine reasoning in real-world rule-grounded, context-specific domains. It is designed to test the capacity of models—particularly LLMs—to reason about obligations, permissions, and prohibitions as articulated in complex legal and policy regimes. Its scenario coverage, programmatic integration, and formal rigour collectively position it as a pivotal resource for the study of deontic reasoning in both symbolic and non-symbolic workflows (Dou et al., 6 Apr 2026).

1. Structure, Scope, and Task Typology

DEONTICBENCH comprises 6,232 tasks spanning four legal- and policy-relevant domains:

• U.S. Federal Taxes ("SARA")
• Airline Baggage Policies
• U.S. Immigration Appeals (USCIS-AAO)
• U.S. State Housing Law

Each domain provides multiple task types:

Domain	Numeric Task (Regression)	Binary/Classification Task
SARA	Tax liability computation	Entailment/Contradiction
Airline	Baggage fee computation	N/A
Housing	N/A	Statute-grounded Yes/No questions
USCIS-AAO	N/A	Accept/Dismiss classification

Difficulty stratification is provided via a "Whole" split and a hand-curated "Hard" subset. The hard subset omits easier tasks identified by both model runs (GPT-5.2, o3, Claude Sonnet) and human review. The proportion of hard instances is lowest in SARA Numeric (35/100) and greatest in Housing (78/5314), reflecting domain-specific complexity (Dou et al., 6 Apr 2026).

2. Formalization of Deontic Reasoning

Task construction in DEONTICBENCH requires the machine to reason about deontic modalities under explicit, multifactor rules:

• Obligation: $$\texttt{obligation(Person, Action) :- condition(Person, Situation).}$$
• Permission: $$\texttt{permission(Person, Action) :- prohibition(Person, Action), condition(Person, Situation).}$$
• Prohibition: $$\texttt{prohibition(Person, Action) :- forfeiture(Person), condition(Person, Situation).}$$

Rules are encoded as Prolog clauses. Problem instances combine relevant statutory rules (as Prolog code), case-specific facts, and a final query (e.g., $\texttt{tax('Alice', 2017, Tax)}$). The formal apparatus is rich enough to model both compound numerical dependencies (tax) and context-sensitive exceptions (legal permissions/prohibitions).

3. Reasoning Workflows: Free-Form vs. Solver-Based

DEONTICBENCH supports dual operational modes:

• Free-Form Chain-of-Thought (CoT): LLMs articulate stepwise rationales in natural language, describing the decision path from legal fact identification to rule application. While promoting transparency, this mode is susceptible to hallucination and rule drift.
• Solver-Based (Symbolic) Workflow: Statutes and facts are rendered as executable Prolog code, which is then solved under SWI-Prolog to yield deterministic, audit-traceable answers. Each benchmark instance is distributed with a reference Prolog solution for evaluation (Dou et al., 6 Apr 2026).

These options allow integrated study of neural (textual) and symbolic (logic-based) approaches, along with hybrid neuro-symbolic models.

4. Evaluation Metrics, Performance, and Error Regimes

Performance metrics are grounded in standard supervised learning formulas:

• Numeric Tasks: $$\text{Accuracy} = \frac{\#\text{cases with }|y-\hat y|\le1}{N}$$ (rounded, margin-1)
• Categorical Tasks: $$\mathrm{Macro\!-\!}F_1=\frac{1}{2}(F_1^{\mathrm{class\ 1}} + F_1^{\mathrm{class\ 0}})$$
• Handling Abstentions: Non-executable, invalid, or unparsable outputs are counted as incorrect (effectively incentivizing robust execution).

Models tested include GPT-4.1, GPT-5.2, o3, Claude Sonnet, Gemini, Qwen3, and their coding-specialized variants. Prompting methods include direct, zero-shot (Prolog generation), and few-shot (with exemplars). Best-case "hard" subset results illustrate the challenge:

Task/Domain	Best Model	Approach	Score
SARA Numeric	o3	Zero-Shot	44.4%
Airline	o3	Few-Shot	90.8%
SARA Binary	GPT-5.1	Direct	71.5%
Housing	GPT-4.1	Few-Shot	46.6%

Solver-based (Prolog) workflows typically enhance coverage but increase abstentions, while more direct methods reduce abstains but incur higher outright error rates (Dou et al., 6 Apr 2026).

5. Symbolic Program Induction and Training

Dedicated program synthesis experiments use Qwen2.5-32B-Instruct as a base:

• Supervised Fine-Tuning (SFT): Improves few-shot macro-F1 in SARA Binary classification (from 29.1% to ~41.6%) but is much less effective on numerical reasoning (remains <10%).
• Direct Preference Optimization (DPO) & Dr. GRPO (RL): Provide additional gains on structured/binary tasks; however, such improvements are inconsistent and remain sensitive to prompt design, with persistent errors in rule selection and fact encoding (Dou et al., 6 Apr 2026).

Quality assessment is dual: exact match of generated Prolog code vs. gold; semantic correctness via execution under SWI-Prolog.

6. Agentic and Interactive Reasoning Extensions

Recent investigations into agentic paradigms such as Deontic Agentic Reasoning (DAR) (Dou et al., 3 Jun 2026) use interactive harnesses (e.g., Terminus-2, Terminus-KIRA) that permit models to iteratively query statutes and reason using external reading tools (shell utilities, interpreters). This setup yields:

• Significant gains for capable (frontier) models: E.g., GPT-5.2 achieves a 30-point jump in SARA Numeric accuracy under Terminus-KIRA.
• Degradation for weaker (open-weight) models: Token usage can rise fourfold, and models may fail to terminate or chase irrelevant provisions.
• Fine-grained audit trails: Each agentic step, observation, and query is logged.
• Unsolved challenges: Scaling to longer statutes, harness specialization, and mitigating the "lost in context" failure mode, particularly for models with limited self-critique.

DAR demonstrates that selective, stepwise reading of external statutes mitigates context-overload in high-complexity settings but magnifies the gap in capabilities between model classes (Dou et al., 3 Jun 2026).

7. Interaction with Rule Representation and SSO Diagnosis

While DEONTICBENCH focuses on end-task outcomes, related benchmarks such as NormBench stress the importance of intermediate representations (e.g., Span-Grounded Deontic Trees, SG-DT) for diagnosing Silent Scope Omission (SSO)—a failure mode where nested exceptions are dropped. Lessons from SG-DT evaluations inform DEONTICBENCH-style curation by emphasizing:

• Explicit span grounding and exclusion guards as mechanisms for verifiable exception handling.
• Structural auditability: Intermediate trees facilitate the identification of omission errors not visible in classification metrics.
• Recursion Decay: Performance drops sharply with nesting/defeater depth; benchmark tasks with such complexity are vital to stress-test LLMs (Chen et al., 8 Jun 2026).

A plausible implication is that future iterations of DEONTICBENCH might benefit from importing span-grounded representations as diagnostic or operational checks.

8. Practical Usage, Resource Availability, and Future Directions

Reference Prolog programs, case splits, and evaluation scripts are publicly available via GitHub (Dou et al., 6 Apr 2026). Standardized usage includes:

• Repository cloning and SWI-Prolog installation
• evaluate.py for metric computation
• Inspection or modification of reference rule encodings (gold_prolog/)
• Containerized (Docker) workflows for reproducibility

Key open challenges highlighted by DEONTICBENCH include the necessity for:

• Long-context rule selection
• Improved numerical precision in arithmetic-dense domains
• Enhancements in CoT prompting and hybrid retrieval-neuro-symbolic systems
• Calibration and confidence estimation for regulatory compliance

DEONTICBENCH is thus situated as a foundational substrate for the development and evaluation of explainable, high-stakes legal/policy reasoning systems and their underlying algorithmic machinery (Dou et al., 6 Apr 2026).