OpenExempt Benchmark

• OpenExempt is a dynamic benchmark that evaluates legal reasoning in language models by generating controlled, rule-based tasks based on U.S. Bankruptcy Code asset exemptions.
• It utilizes expert-crafted symbolic encodings, parametric task generation, and a deterministic solver to produce reliable, ground-truth legal reasoning tasks.
• The framework's diverse evaluation suites and metrics reveal model failure modes and support interpretable, robust testing of advanced legal AI systems.

OpenExempt is a dynamic benchmark and framework for diagnostic evaluation of legal reasoning in LLMs, focusing on the complex and rule-bound domain of U.S. Bankruptcy Code asset-exemption statutes. It addresses limitations of static QA datasets by enabling fine-grained measurement of reasoning skills along multiple axes of complexity, employing expert-crafted symbolic representations, parametric task generation, and a provably correct deterministic solver to produce a large and controlled suite of legal reasoning tasks and ground-truth solutions (Servantez et al., 19 Jan 2026).

1. Theoretical Foundations and Formal Definitions

OpenExempt is defined by three central components: expert-crafted symbolic encodings of exemption statutes, a formal mapping for on-demand generation of natural-language reasoning tasks, and an exhaustive solver for computing ground-truth answers.

Exemption statutes are encoded as a set $E = \{e_1, e_2, ..., e_m\}$, where each $e_i$ possesses fields for aggregate caps ($e_i.\text{single\_limit}$, $e_i.\text{married\_limit}$), per-item limits ($e_i.\text{per\_item\_limit}$, $e_i.\text{item\_claim\_count}$), fallback rules ($e_i.\text{fallback\_exemption} \rightarrow e_j$, $e_i.\text{fallback\_single\_limit}$), and exclusions ($e_i.\text{mutual\_exclusion} \rightarrow e_k$). Each statute has both a natural-language template for prompt construction and structured fields for legality checks and optimization in the solver.

Formally, all benchmarking tasks are generated via: $$G : S \times P \longrightarrow \{(Q_i, A_i)\}_{i=1}^N$$ where $S$ denotes structured inputs (asset pools $A$ and exemption encodings $E$), $P$ comprises user-configurable parameters (such as asset and domicile counts, jurisdiction selection, obfuscation flags, etc.), $Q_i$ are generated prompts, and $A_i$ are machine-computable solutions.

The solver uses exhaustive branch-and-bound search to evaluate statutory constraints for each generated case, ensuring ground-truth fidelity (Servantez et al., 19 Jan 2026).

2. Pipeline for Dynamic Task Generation

The pipeline consists of four principal steps: case sampling, optional obfuscation injection, prompt rendering, and solution computation. A high-level pseudocode illustrates the mechanism:

function GenerateBenchmark(S, P): tasks = [] for sample_idx in 1..P.dataset_size: case ← CaseGenerator.sample_case( assets ∼ uniform(A, P.asset_count_min, P.asset_count_max), domiciles ∼ uniform(P.domicile_count_min, P.domicile_count_max), married ∼ Bernoulli(P.married_ratio), jurisdiction ∼ uniform(P.state_jurisdictions) ) if P.irrelevant_asset_facts: case.inject_irrelevant_assets() if P.irrelevant_domicile_facts: case.inject_irrelevant_domiciles() if P.asset_opinions: case.inject_asset_opinions() if P.domicile_opinions: case.inject_domicile_opinions() prompt_text ← TaskGenerator.render_prompt(case, start_task_id=P.start_task_id, terminal_task_id=P.terminal_task_id) solution ← Solver.solve(case, start=P.start_task_id, end=P.terminal_task_id) tasks.append((prompt_text, solution)) return tasks

Complexity control is achieved via parameterization, enabling orthogonal variation of asset count ($A$), domicile count ($D$), reasoning pipeline length ($L$), and obfuscation/distractor flags ($O_d$, $O_s$, $O_o$). Every sample can be represented by a vector $(A, D, L, O_d, O_s, O_o)$, allowing systematic probing of individual failure modes.

3. Benchmark Composition and Evaluation Suites

OpenExempt comprises 9,765 samples, systematically divided into nine evaluation suites—three "competency" tiers and six diagnostic suites, each probing distinct dimensions of reasoning difficulty.

Suite Category	Tasks/Features	Sample Count
Basic Competency	AE, EC, EV, NA, OE; $A=2$, $D=2..3$, $L=5$; states WI, IL	525
Intermediate Competency	$A=3..5$, $D=4$, partial obfuscation; AZ, PA, OR	525
Advanced Competency	$A=6..8$, $D=5$, full obfuscation; all five states	525
Temporal Reasoning	AE only, $D=1..5$, all states, no obfuscation	525
Reasoning Decomposition	EC–OE, $A=6$, $D=4$; all subsets of predecessor tasks	2,100
Distractor Robustness	All tasks, $A=4$, $D=3$, $O_d=1$; AZ, PA, WI	525
Sycophancy Robustness	All tasks, $O_s=1$, same config as above	525
Obfuscation Robustness	Full distractor+opinion injection, $O_o=1$	525
Asset Scaling	EC–OE, $A=2,4,6,8$; $D=2$; IL, OR, PA	1,260

Competency suites address holistic variations of asset and domicile complexity with increasing presence of obfuscation, while diagnostic suites isolate single axes (e.g., temporal reasoning probes domicile history, distractor suites introduce irrelevant facts, decomposition probes intermediate error propagation).

Illustrative tasks demonstrate the range of evaluated capabilities. For instance, temporal reasoning asks for applicable exemptions given a sequence of domiciles, while distractor robustness introduces unrelated asset facts to test classification resilience.

4. Metrics and Evaluation Protocol

Four principal metrics are defined:

1. Precision, Recall, F1: Applied to multi-label and structured tasks, with F1 computed per asset and then macro-averaged per sample, preventing sample-size dominance.
2. Mean Absolute Relative Error (MARE): For dollar-amount predictions, $$\mathbb{I}_{\tau}(\hat y, y) = \mathbf{1}\bigl[|\frac{\hat y}{y+\varepsilon} - 1| < \tau\bigr]$$, with $\varepsilon=1$, $\tau=0.05$; predictions within 5% of gold are scored as correct.
3. Format-compliance rate: Fraction of sample responses that parse as valid JSON/CSV.
4. Failure-mode analysis: Examination of performance cliffs along complexity axes $(A, D, L, O)$.

Micro-averaging ensures statistically meaningful measurement even as task and data complexity scale.

5. Empirical Findings and Diagnostic Insights

Evaluation across 13 diverse LLMs (three reasoning-oriented, three large non-reasoning, seven efficient) surfaces distinct patterns:

• Competency Suites: Leading reasoning models (GPT-5, o3, Gemini-2.5-Pro) achieve F1 $>$ 0.9 on basic AE/EC tasks; efficient models lag by 10–20 points. Advanced-tier tasks show no model exceeding F1 $\approx$ 0.63, with performance gaps widening with complexity.
• Temporal Reasoning: At $D=1$, F1 is nearly perfect. Increasing $D$ to 2 yields minimal degradation; however, a pronounced cliff emerges at $D=4$ (drop $\geq$ 0.14 F1), then stabilizing at $D=5$.
• Asset Scaling: Asset-level tasks degrade slowly as $A$ increases from 2 to 8; estate-level tasks exhibit super-linear failure, e.g., Llama-4-Scout drops OE F1 from .425 ($A$=2) to .077 ($A$=8). Top reasoning models remain robust (OE F1 $>$ .90 at $A$=2, declining $\leq$ 0.03 by $A$=8).
• Obfuscation Robustness: For AE/EC, distractors occasionally produce slight F1 gains (suggesting models “slow down”); for OE, all models suffer major declines under distractor+obfuscation (GPT-5 OE F1 falls from .844 to .450).
• Reasoning Decomposition: Providing correct intermediate solutions can boost downstream F1 (classic error propagation effect), yet in some estate-level tasks, feeding the gold sub-solution disrupts performance for reasoning models.

These results establish OpenExempt as a fine-grained diagnostic tool capable of revealing precise breaking points in legal reasoning, uncovering weaknesses not visible in static datasets.

6. Public Availability and Research Applications

All code, templates, and data underlying OpenExempt are released under CC BY 4.0 at github.com/servantez/OpenExempt and the Hugging Face Hub. The framework enables creation of custom benchmark suites on demand, facilitating targeted debugging of model weaknesses, regression testing, and transparency in legal AI system development.

A plausible implication is that OpenExempt’s dynamic, parametric architecture offers a new level of experimental controllability compared to traditional legal reasoning datasets. Its principled design supports future research into interpretable reasoning, robust benchmarking, and comprehensive error-mode analysis for advanced LLMs (Servantez et al., 19 Jan 2026).