LegalHalluLens: Auditing Legal Hallucinations

• LegalHalluLens is a legal-domain auditing framework that systematically classifies hallucinations into numeric, temporal, obligation, and factual claim types.
• It computes a Risk Direction Index to distinguish between omission-heavy and invention-heavy errors, enabling precise compliance risk measurement.
• Its calibrated multi-agent debate pipeline reduces fabricated detections by 45%, balancing error mitigation while matching commercial API performance.

LegalHalluLens is a legal-domain hallucination auditing framework that combines three elements: typed hallucination profiles across four legally motivated claim categories, a Risk Direction Index (RDI) that compresses omission-versus-invention bias into a deployment-comparable scalar, and a typed debate pipeline calibrated to both error magnitudes and error directions. It is evaluated on CUAD v1.0 over 510 contracts and 249,252 clause-level instances, where aggregate hallucination reporting at approximately 52% is shown to conceal both category-specific concentration of errors and the direction in which those errors run. The framework further reports that its calibrated debate pipeline reduces fabricated detections by 45% while matching commercial APIs with a substantially smaller backbone of 4B active parameters (Yadav et al., 16 Jun 2026).

1. Definition, scope, and research lineage

LegalHalluLens is situated within a broader line of hallucination research that has progressively moved from generic scalar detection toward domain-specific auditing and mitigation. HalluLens separated hallucination from factuality and formalized extrinsic and intrinsic hallucinations in a benchmark setting (Bang et al., 24 Apr 2025). HalluDetect then addressed consumer-law conversational systems with a multi-stage detector and architecture benchmarking, reporting HalluDetect $\mathrm{F}_1 = 0.6892$ with GPT-4o-mini and identifying AgentBot as the strongest mitigation strategy with $\mathrm{HPT}_1 = 0.4159$ and $\mathrm{TokAcc}_1 = 96.13\%$ (Anaokar et al., 15 Sep 2025). HalluGraph introduced auditable knowledge-graph alignment through Entity Grounding, Relation Preservation, and a Composite Fidelity Index (Noël et al., 1 Dec 2025). HalluField proposed a logit-space, field-theoretic detector based on free-energy and entropy variation (Vu et al., 12 Sep 2025). HalluGuard, in turn, decomposed hallucination risk into data-driven and reasoning-driven components using NTK geometry and rollout instability (Zeng et al., 26 Jan 2026).

Within that trajectory, LegalHalluLens shifts emphasis from a single aggregate hallucination score to typed legal verification regimes and direction-sensitive auditing. This suggests a complementary role rather than a replacement: earlier systems primarily ask whether an output is unsupported, whereas LegalHalluLens additionally asks which legal claim type is failing and whether the model tends to omit or invent legally material qualifiers (Yadav et al., 16 Jun 2026).

2. Typed hallucination profiles

A central contribution of LegalHalluLens is the stratification of hallucinations into four claim-type verification regimes, each defined by the nature of the ground-truth check and the characteristic failure modes (Yadav et al., 16 Jun 2026).

Claim type	Definition	Formal criterion
Numeric	Clause semantics reduces to a numeric value	Verbatim match of number and unit
Temporal	Clause semantics hinges on a date or duration	Verbatim match of date or duration and time unit
Obligation/Entitlement	Clause confers or restricts a duty or right	Preserve modal verb, trigger conditions, carve-outs, and scope qualifiers
Factual	Short identity claim verifiable by verbatim lookup	Match oracle’s literal text

The numeric regime covers amounts, percentages, thresholds, caps, and units. Examples given include “Cap on Liability is \$5,000,000” and “Minimum Commitment: 10,000 units per year.” A prompted note excludes clauses that merely exclude categories of damages without stating a maximum amount. The temporal regime covers dates and durations, with examples such as “Effective Date: January 1, 2024” and “Notice Period: 30 days,” and explicitly requires distinguishing stated timeframes from inferred ones.

The obligation/entitlement regime is more structurally demanding. It requires preservation of the exact modal verb—such as shall, may, must, or should—as well as all trigger conditions, carve-outs introduced by phrases such as “provided that” or “unless,” and any scope qualifiers, including geographic, subject-matter, or temporal qualifiers. The factual regime covers identity-style claims such as governing law, counterparty name, and document title, and requires literal agreement with the oracle text.

The paper’s typical hallucination examples demonstrate why these regimes are legally distinct. Numeric hallucinations include substituting common priors, such as “\$5 million” when the cap is \$4 million, or dropping “per annum.” Temporal hallucinations include inferring “one month” for “30 days.” Obligation hallucinations include downgrading “shall” to “may,” omitting carve-outs after “except,” or dropping a qualifier such as “within 50 miles.” Factual hallucinations include injecting state-law defaults, such as “New York law,” when the contract is silent, or normalizing party names. A plausible implication is that LegalHalluLens treats hallucination not as a unitary textual deviation but as a family of verification failures with materially different legal consequences.

3. Risk Direction Index

LegalHalluLens argues that legal hallucinations are not only heterogeneous by type but also directional. Errors can be omission-heavy, understating obligations or conditions present in the source, or invention-heavy, overstating obligations or adding qualifiers absent from the source (Yadav et al., 16 Jun 2026).

For each detected clause whose judge verdict is “contradicted,” the framework records a mismatch_type. It then computes:

$$p_{\mathrm{extra}} = \frac{\#\{\text{extra\_condition labels}\}} {\#\{\text{contradicted TPs}\}} \times 100\%$$

and

$$p_{\mathrm{missing}} = \frac{\#\{\text{missing\_condition labels}\}} {\#\{\text{contradicted TPs}\}} \times 100\%.$$

The Risk Direction Index is then

$$\mathrm{RDI}(M) = \frac{p_{\mathrm{extra}(M)} - p_{\mathrm{missing}(M)}}{100}.$$

The interpretation is direct. $\mathrm{RDI} > 0$ indicates invention-heavy behavior; $\mathrm{RDI} < 0$ indicates omission-heavy behavior; and $\mathrm{RDI} \approx 0$ indicates a roughly balanced profile. The reported sample results are: gpt-5.2 at $\mathrm{HPT}_1 = 0.4159$0 with 95% bootstrap CI $\mathrm{HPT}_1 = 0.4159$1, gemini-3-flash at $\mathrm{HPT}_1 = 0.4159$2 with CI $\mathrm{HPT}_1 = 0.4159$3, llama-3.3-70b at $\mathrm{HPT}_1 = 0.4159$4 with CI $\mathrm{HPT}_1 = 0.4159$5, and qwen3-32b at $\mathrm{HPT}_1 = 0.4159$6 with CI $\mathrm{HPT}_1 = 0.4159$7.

A common misconception in aggregate evaluation is that two systems with similar hallucination rates pose similar deployment risk. LegalHalluLens rejects that assumption empirically: two systems with identical aggregate hallucination rates of approximately 52% can exhibit opposite RDIs and therefore opposite compliance-risk profiles. In the paper’s example, gpt-5.2 and qwen3-32b are matched at roughly 52% $\mathrm{HPT}_1 = 0.4159$8, yet gpt-5.2 is invention-heavy while qwen3-32b is omission-heavy (Yadav et al., 16 Jun 2026).

4. Calibrated multi-agent debate

To mitigate high-risk fabricated detections and directional bias, LegalHalluLens organizes its correction mechanism around a Baseline Extractor and a calibrated 6-role debate consisting of Skeptic, Supporter, Re-extractor, Verifier, Judge (In-debate), and Arbiter (Yadav et al., 16 Jun 2026).

The workflow is asymmetric by design. The Skeptic issues claim-type-specific challenge questions. For numeric claims, it asks whether the exact value is verbatim, including unit and qualifiers, or merely a prior substitution. For obligation claims, it asks whether the modal verb is preserved and whether carve-outs and scope are retained. For temporal claims, it asks whether the date or duration is explicit rather than inferred and whether units are exact. For factual claims, it asks whether the fact is in the document or imported from external knowledge and whether the entity name is exact.

The Supporter defends the baseline extraction by quoting verbatim spans. If the Skeptic identifies a structural error, meaning the wrong clause rather than merely a content defect, the Re-extractor reruns extraction from source and the debate resets. The Verifier independently searches the contract and checks definition fit. The Judge then makes final Add/Delete decisions using asymmetric structural gates, while the Arbiter resolves deadlock conservatively after all rounds.

The asymmetry is legally important. The Addition Gate, which governs transitions from absent to present, requires both debate consensus and Verifier confirmation; this is designed to prevent fabricated additions. The Deletion Gate, which governs transitions from present to absent, blocks deletion if the Verifier confirms presence; this is designed to prevent erasing real clauses. The paper explicitly states that this asymmetry encodes measured risk profiles, often where FAR exceeds FRR on high-error types, rather than a generic 50/50 policy. This suggests that LegalHalluLens treats mitigation not as generic ensembling but as post-hoc control tuned to observed failure direction.

5. Empirical profile on CUAD

All experiments use CUAD v1.0 as oracle, covering 510 contracts, 41 clause types, and 249,252 clause-level instances across runs (Yadav et al., 16 Jun 2026).

In Experiment 1, the typed hallucination profiles are reported using $\mathrm{HPT}_1 = 0.4159$9, defined as the content-error rate among detected clauses. The four evaluated models show large within-model spreads across claim types. For gemini-3-flash, Numeric is $\mathrm{TokAcc}_1 = 96.13\%$0, Obligation $\mathrm{TokAcc}_1 = 96.13\%$1, Factual $\mathrm{TokAcc}_1 = 96.13\%$2, and Temporal $\mathrm{TokAcc}_1 = 96.13\%$3, for a gap of $\mathrm{TokAcc}_1 = 96.13\%$4 percentage points. For gpt-5.2, the values are $\mathrm{TokAcc}_1 = 96.13\%$5, $\mathrm{TokAcc}_1 = 96.13\%$6, $\mathrm{TokAcc}_1 = 96.13\%$7, and $\mathrm{TokAcc}_1 = 96.13\%$8, with a gap of $\mathrm{TokAcc}_1 = 96.13\%$9 points. For qwen3-32b, they are $5 million” when the cap is \$0, $5 million” when the cap is \$1, $5 million” when the cap is \$2, and $5 million” when the cap is \$3, with a gap of $5 million” when the cap is \$4 points. For llama-3.3-70b, they are $5 million” when the cap is \$5, $5 million” when the cap is \$6, $5 million” when the cap is \$7, and $5 million” when the cap is \$8, with a gap of $5 million” when the cap is \$9 points. Aggregate $$p_{\mathrm{extra}} = \frac{\#\{\text{extra\_condition labels}\}} {\#\{\text{contradicted TPs}\}} \times 100\%$$0 values of $$p_{\mathrm{extra}} = \frac{\#\{\text{extra\_condition labels}\}} {\#\{\text{contradicted TPs}\}} \times 100\%$$1 therefore hide a stable failure ordering across all architectures: $$p_{\mathrm{extra}} = \frac{\#\{\text{extra\_condition labels}\}} {\#\{\text{contradicted TPs}\}} \times 100\%$$2.

Experiment 2 evaluates calibrated debate mitigation on a 120-contract subset with 4,920 nominal rows per run. The matched-subset leaderboard places gemma-debate first with FAR $$p_{\mathrm{extra}} = \frac{\#\{\text{extra\_condition labels}\}} {\#\{\text{contradicted TPs}\}} \times 100\%$$3, FRR $$p_{\mathrm{extra}} = \frac{\#\{\text{extra\_condition labels}\}} {\#\{\text{contradicted TPs}\}} \times 100\%$$4, Acc $$p_{\mathrm{extra}} = \frac{\#\{\text{extra\_condition labels}\}} {\#\{\text{contradicted TPs}\}} \times 100\%$$5, $$p_{\mathrm{extra}} = \frac{\#\{\text{extra\_condition labels}\}} {\#\{\text{contradicted TPs}\}} \times 100\%$$6, JEq $$p_{\mathrm{extra}} = \frac{\#\{\text{extra\_condition labels}\}} {\#\{\text{contradicted TPs}\}} \times 100\%$$7, and rank 1. It is followed by gpt-5.2, qwen3-32b, llama-3.3-70b, gemini-3-flash, and gemma-base. The key reported outcome is a 45% reduction in false-positive detections by gemma-debate relative to gemma-base, from 524 to 287 false positives. Per-category $$p_{\mathrm{extra}} = \frac{\#\{\text{extra\_condition labels}\}} {\#\{\text{contradicted TPs}\}} \times 100\%$$8 tracks the diagnosis from the typed profile: Obligation $$p_{\mathrm{extra}} = \frac{\#\{\text{extra\_condition labels}\}} {\#\{\text{contradicted TPs}\}} \times 100\%$$9 points, Factual $$p_{\mathrm{missing}} = \frac{\#\{\text{missing\_condition labels}\}} {\#\{\text{contradicted TPs}\}} \times 100\%.$$0, Numeric $$p_{\mathrm{missing}} = \frac{\#\{\text{missing\_condition labels}\}} {\#\{\text{contradicted TPs}\}} \times 100\%.$$1, and Temporal $$p_{\mathrm{missing}} = \frac{\#\{\text{missing\_condition labels}\}} {\#\{\text{contradicted TPs}\}} \times 100\%.$$2. The framework also reports an obligation-category RDI shift for gemma-4-26B-A4B from $$p_{\mathrm{missing}} = \frac{\#\{\text{missing\_condition labels}\}} {\#\{\text{contradicted TPs}\}} \times 100\%.$$3 to $$p_{\mathrm{missing}} = \frac{\#\{\text{missing\_condition labels}\}} {\#\{\text{contradicted TPs}\}} \times 100\%.$$4, moving the system toward a near-balanced profile.

These results support the paper’s central claim that aggregate reporting alone is insufficient for trustworthy legal deployment. The main empirical point is not merely that some models hallucinate more than others, but that the locus and direction of error differ substantially even when aggregate rates appear matched.

6. Relation to broader legal hallucination methods and deployment use

LegalHalluLens belongs to a broader ecosystem of legal hallucination methods, but its contribution is distinctive. HalluDetect provides a multi-stage detection-and-mitigation framework for consumer-law chatbots, using retriever expansion, memory summarization, a hallucination analyzer, severity filtering, and architecture benchmarking (Anaokar et al., 15 Sep 2025). HalluGraph focuses on auditable structural alignment via knowledge graphs, decomposing fidelity into Entity Grounding and Relation Preservation and returning explicit missing_entities and unsupported_rels (Noël et al., 1 Dec 2025). HalluField detects hallucinations from output logits by measuring free-energy and entropy variation across temperature perturbations, without auxiliary neural networks or fine-tuning (Vu et al., 12 Sep 2025). HalluGuard formalizes a Hallucination Risk Bound that separates semantic bias from rollout instability and operationalizes the decomposition with an NTK-based score (Zeng et al., 26 Jan 2026). HalluLens contributes the benchmark taxonomy of extrinsic and intrinsic hallucinations and emphasizes dynamic test-set generation to reduce leakage (Bang et al., 24 Apr 2025).

LegalHalluLens does not replicate those formulations. Instead, it introduces typed legal auditing and a direction-aware mitigation layer. This suggests a division of labor in which scalar detectors and structural verifiers identify unsupported content, while LegalHalluLens determines how legally salient failure modes concentrate and how mitigation should be calibrated for those modes (Yadav et al., 16 Jun 2026).

The deployment guidance in the framework is correspondingly operational. For direction-aware procurement, typed profiles can be used to choose models according to which clause types matter most to a workflow, such as liability caps versus deadlines. For accountability and monitoring, the framework recommends remeasuring typed profiles and RDI on representative production documents and reporting per-type rates and RDI in audits instead of a single aggregate number. For agent design, it recommends calibrating Skeptic challenges to measured high-failure types, setting gate asymmetries according to observed FAR-versus-FRR imbalance, and combining human-in-the-loop review on high-risk clause types with automated debate for low-risk or high-volume clauses. In that sense, LegalHalluLens frames hallucination auditing as a governance and systems-design problem as much as a detection problem.