OpenExempt: Legal Reasoning Benchmark

• OpenExempt is a dynamic framework for legal reasoning that benchmarks U.S. Bankruptcy Code exemptions by generating customizable, deterministic tasks.
• It leverages expert-crafted symbolic encodings and a four-part task-generation pipeline to expose language model failure modes in complex, rule-bound scenarios.
• The framework's fine-grained configurability and public release support targeted research on temporal logic, rule matching, and combinatorial legal challenges.

OpenExempt is a diagnostic evaluation framework and benchmark for legal reasoning, specifically targeting the U.S. Bankruptcy Code exemption statutes and select state “opt-out” provisions. It addresses the inflexibility of static benchmarks through a fully dynamic, on-demand task generation system grounded in expert-crafted symbolic encodings of law. OpenExempt provides deterministic, machine-verifiable solutions and fine-grained configurability, enabling targeted investigation of the failure modes of LLMs in complex, rule-bound domains. The framework is publicly released under CC-BY 4.0 and includes a benchmark of 9,765 samples across nine curated evaluation suites (Servantez et al., 19 Jan 2026).

1. Objectives, Scope, and Design Principles

OpenExempt is designed to address key limitations of static legal reasoning benchmarks by introducing:

• A dynamic task-generation engine, based on expert-crafted encodings of the U.S. Bankruptcy Code (Title 11) and selected state statutes (Arizona, Illinois, Oregon, Pennsylvania, Wisconsin).
• Fine-grained user control over task complexity via configurable axes including asset count, domicile history, jurisdictional sampling, and inclusion of obfuscating facts.
• Paired representations of assets and statutes: natural-language templates for prompt construction and symbolic, machine-readable encodings for solution generation.
• Public release of the benchmark (9,765 samples, nine suites) and the framework, restricted in scope to objective exemption problems (tangible assets, explicit monetary caps, no subjective or exception-based statutes). Only English-language prompts and exemptions are supported.

This approach is built to support rigorous, diagnostic, on-demand evaluation of LLM legal reasoning, with a particular emphasis on isolating specific reasoning skills (e.g., temporal logic, rule-matching, combinatorial optimization).

2. Symbolic Formalization of Exemption Statutes

The core of OpenExempt is a formal, rule-based encoding of bankruptcy exemption statutes. Each statute is represented as a rule $R = (H, C, E)$:

• $H = \{h_1,\dotsc,h_k\}$, the set of antecedent predicates or hypotheses (e.g., asset type matches statute, debtor marital status).
• $C = \{c_1,\dotsc,c_\ell\}$, quantitative or structural conditions (e.g., per-item limits, aggregate caps).
• $E = \{e_1,\dotsc,e_m\}$, the set of exceptions or mutually exclusive conditions.

Given a fact database $\Delta$ describing the case context, an asset $a$ is exempt under rule $R$ if:

• All antecedents in $H$ hold: $\forall h \in H,\, \Delta \vdash h(a)$
• All conditions in $C$ are satisfied: $\forall c \in C,\, \Delta \vdash c(a)$
• No exception in $E$ applies: $\forall e \in E,\, \Delta \not\vdash e(a)$

That is,

$\Delta \vdash \text{Exempt}(a, R)$

The framework encodes these rules to support deterministic reasoning and eliminates subjective or ambiguous provisions. For estate-level optimization, rules are augmented with monetary caps (e.g., single-limit, married-limit) and fallback dependencies, maintaining legal validity and combinatorial constraints.

3. Task-Generation Pipeline

OpenExempt operationalizes benchmark and custom sample creation via a four-part pipeline:

1. Knowledge-Representation Layer:
   • DebtorAssets: Repository of over 500 assets, tagged with applicable $R$ rules for each jurisdiction.
   • ExemptionStatutes: Complete encodings of supported federal and state exemption statutes.
2. CaseGenerator:
   • Samples case parameters based on user configuration (e.g., asset count, domicile timeline, jurisdiction mix, obfuscation toggles).
   • Generates SymbolicCase objects capturing all relevant facts in machine-readable form.
3. TaskGenerator:
   • Instantiates parameterized natural-language templates for task instructions and facts.
   • Produces prompts, optionally with intermediate “solved” outputs for decomposition experiments.
   • Accompanies prompts with a strict JSON schema specifying the expected response format.
4. Deterministic Solver:
   • For asset-level tasks, draws gold standards from DebtorAssets annotations.
   • For estate-level reasoning (global exemption assignment), employs a branch-and-bound search algorithm over all valid exemption schedules, pruning suboptimal paths.
   • Excludes statutes and facts that would prevent objective, deterministic ground truth.

4. Configurability and Diagnostic Capability

User control over benchmark generation is central to OpenExempt’s diagnostic purpose, exposing sharp model failure points:

• Asset Scaling: Varying asset count (asset_count_min…max) increases combinatorial complexity, especially for global exemption assignment.
• Temporal Reasoning: Varying the number of domiciles (domicile_count_min…max) directly probes the application of the bankruptcy “730-day Rule.”
• Obfuscation Flags: Inclusion of irrelevant asset facts and opinion distractors tests robustness to misleading or superfluous information.
• Reasoning Decomposition: The pipeline can expose or pre-solve any intermediate step (Asset Extraction, Exemption Classification, Exemption Valuation, Net Asset Evaluation, Optimal Exemption Assignment) to distinguish between stage and propagation errors.

Each diagnostic suite isolates a single axis, maintaining all others constant, enabling precise attribution of model failures.

5. Custom Benchmark Construction and Output Artifacts

OpenExempt enables users to synthesize tailored benchmarks through a configuration file (YAML or JSON), specifying:

• Start and terminal tasks in the reasoning pipeline.
• Dataset size, asset and domicile range, marital ratio.
• State jurisdiction(s), obfuscation toggles.
• Paths for assets, statutory encodings, templates, and output directories.

Output consists of:

• Prompt files (one per sample).
• Ground-truth JSON solutions with gold assignments or class labels.
• Machine-verifiable output schemas (Pydantic models) for subsequent model evaluation and compliance checking.

This design allows rapid prototyping and customization, facilitating research on both system-level and targeted failure analysis.

6. Sample Generation Algorithms and Core Methods

Dynamic estate-level reasoning is operationalized through a recursive branch-and-bound solver. The algorithm enumerates all asset-to-exemption assignments consistent with statutory caps and mutual exclusions, using early pruning to discard globally suboptimal branches. The pseudocode provided in the source delineates the depth-first search, cap updates, and fallback mechanisms required for legally valid assignment schedules, computing the minimum unprotected estate value for each scenario.

Example task–solution pairs illustrate robustness evaluation (EC tasks with distractors) and reasoning decomposition (OE tasks where intermediate AE, EC, EV steps are pre-solved), confirming that a single logical engine can consistently generate gold solutions across varied task granularities.

7. Limitations, Extensions, and Deployment Recommendations

OpenExempt currently restricts coverage to objective, English-language statutes of five states plus federal exemptions, explicitly omitting subjective or exception-based statutes (e.g., “reasonably necessary” or disability-related). Extensions may include additional Title 11 chapters, more states, or analogous symbolic encodings in other legal domains (tax, family law). Multilingual support would necessitate both parallel template libraries and statutory encodings for each jurisdiction.

Best practices include:

• Use of zero-shot evaluation to ensure task purity.
• Validation of system outputs via an Evaluator class enforcing JSON schema compliance, citation normalization, and legal re-evaluation.
• Exploiting diagnostic axis suites to pinpoint breaking points (e.g., sharp accuracy drops at four domiciles or beyond four assets).
• Strict template authoring methodologies: LM-assisted candidate generation, legal screening, and explicit parametrization to minimize linguistic or logical ambiguity.

OpenExempt thus represents an extensible, transparent framework for high-precision diagnostic evaluation of legal reasoning in LLMs, enabling deep analysis beyond conventional snapshot benchmarks (Servantez et al., 19 Jan 2026).