Risk Direction Index (RDI)

• Risk Direction Index (RDI) is a signed scalar that summarizes a legal AI model’s net tendency to invent extra conditions versus omit necessary ones.
• It computes the difference between the proportions of extra and missing qualifiers multiplied by 100, offering a deployment-actionable diagnostic.
• Deployed within LegalHalluLens, RDI informs calibrated debate pipelines and safety gates to enhance compliance and minimize legal risks.

Searching arXiv for the specified paper to ground the article in the cited source. Risk Direction Index (RDI) is a signed scalar introduced in "LegalHalluLens" to summarize a legal AI model’s net tendency to invent versus omit legally relevant qualifiers in clause extraction. Within that framework, aggregate hallucination rates are reported at approximately $52\%$, but the central claim is that this average can conceal both where errors concentrate and the direction in which they run. RDI is therefore positioned as a deployment-actionable diagnostic: two systems may exhibit the same overall hallucination rate yet differ materially in whether they add spurious conditions or drop real obligations, a distinction with direct consequences for compliance, review, and procurement decisions (Yadav et al., 16 Jun 2026).

1. Conceptual role within LegalHalluLens

RDI is one of three components of LegalHalluLens: typed hallucination profiles across four legally motivated claim categories, a Risk Direction Index that reduces omission-versus-invention bias to a single deployment-comparable scalar, and a typed debate pipeline calibrated to both magnitudes and directions. The four claim categories are numeric, temporal, obligation/entitlement, and factual. The framework is evaluated over CUAD on 510 contracts and 249,252 clause-level instances, and it reports a within-model gap of approximately 38–40 percentage points between obligation/numeric and temporal claims, a disparity that aggregate reporting hides (Yadav et al., 16 Jun 2026).

The placement of RDI inside this architecture is significant. Typed hallucination profiles localize error concentration by claim type, whereas RDI summarizes directional bias across contradicted detections. In that sense, RDI is not a generic accuracy statistic; it is a directional risk descriptor intended for legal workflows in which the difference between overstatement and understatement is operationally consequential.

2. Formal definition

RDI is defined for a model $M$ evaluated on an oracle-verified legal extraction task. An external judge examines each True-Positive detection and, for any content contradiction, assigns a mismatch label. Two labels are directional:

• extra_condition: the model asserts a qualifier absent from the source, corresponding to invention.
• missing_condition: the model omits a qualifier present in the source, corresponding to omission.

The two component proportions are defined as

$$p_{\mathrm{extra}}(M) = \frac{\text{number of TP contradictions labeled extra\_condition}} {\text{total number of TP contradictions}}$$

and

$$p_{\mathrm{missing}}(M) = \frac{\text{number of TP contradictions labeled missing\_condition}} {\text{total number of TP contradictions}}.$$

The Risk Direction Index is then

$$\mathrm{RDI}(M) = \bigl(p_{\mathrm{extra}}(M) - p_{\mathrm{missing}}(M)\bigr) \times 100.$$

Under this definition, $p_{\mathrm{extra}}(M)\in[0,1]$, $p_{\mathrm{missing}}(M)\in[0,1]$, and $\mathrm{RDI}(M)\in[-100,100]$. Positive values indicate that invention outweighs omission; negative values indicate that omission outweighs invention (Yadav et al., 16 Jun 2026).

This construction makes RDI a directional contrast rather than a count of all hallucinations. It is specifically designed to collapse omission-versus-invention bias into one number while preserving sign.

3. Computation procedure and semantics

The workflow for computing RDI is explicitly stepwise. First, model $M$ extracts clauses and the judge compares each detected clause with ground truth; whenever a content contradiction is found, the judge emits one mismatch_type label, including extra_condition or missing_condition. Second, directional errors are counted:

• $N_{\mathrm{extra}}$: TP contradictions labeled extra_condition.
• $M$0: TP contradictions labeled missing_condition.
• $M$1: total TP contradictions across all types.

Third, the proportions $M$2 and $M$3 are computed using the shared denominator $M$4. Fourth, omission share is subtracted from invention share and the result is multiplied by 100, yielding percentage points (Yadav et al., 16 Jun 2026).

A notable design choice is that the denominator includes all TP contradictions, including non-directional tags such as scope, while the numerator uses only the two directional tags. No additional normalization or thresholding is applied. This suggests that RDI is intended to summarize one specific axis of error geometry—directionality—without conflating it with the full taxonomy of contradiction types.

The sign has direct interpretive semantics. A positive RDI indicates that, conditional on contradiction, the model more often introduces qualifiers not present in the source. A negative RDI indicates that it more often removes qualifiers that are present. Because legal review often distinguishes removable overstatement from silent understatement, the sign is treated as deployment-relevant rather than merely descriptive.

4. Reported values and empirical interpretation

The reported overall RDI values on the CUAD clause-extraction task are shown below, together with the per-category values reported for the dominant Obligation category.

Model	Overall RDI	Obligation RDI
gpt-5.2	$M$5 $M$6	$M$7
gemini-3-flash	$M$8 $M$9	$$p_{\mathrm{extra}}(M) = \frac{\text{number of TP contradictions labeled extra\_condition}} {\text{total number of TP contradictions}}$$0
llama-3.3-70b	$$p_{\mathrm{extra}}(M) = \frac{\text{number of TP contradictions labeled extra\_condition}} {\text{total number of TP contradictions}}$$1 $$p_{\mathrm{extra}}(M) = \frac{\text{number of TP contradictions labeled extra\_condition}} {\text{total number of TP contradictions}}$$2	$$p_{\mathrm{extra}}(M) = \frac{\text{number of TP contradictions labeled extra\_condition}} {\text{total number of TP contradictions}}$$3
qwen3-32b	$$p_{\mathrm{extra}}(M) = \frac{\text{number of TP contradictions labeled extra\_condition}} {\text{total number of TP contradictions}}$$4 $$p_{\mathrm{extra}}(M) = \frac{\text{number of TP contradictions labeled extra\_condition}} {\text{total number of TP contradictions}}$$5	$$p_{\mathrm{extra}}(M) = \frac{\text{number of TP contradictions labeled extra\_condition}} {\text{total number of TP contradictions}}$$6

These values support several concrete interpretations. gpt-5.2 is reported as invention-heavy overall, whereas llama-3.3-70b and qwen3-32b are omission-heavy. gemini-3-flash is reported as near-balanced overall. In the paper’s practical interpretation, gpt-5.2’s positive RDI means that when it contradicts a true clause, it more often invents extra qualifiers than drops real ones; in a compliance-safety context, where spurious conditions can be removed in review but silently lost obligations are more problematic, this is described as the safer profile. Conversely, qwen3-32b’s negative RDI indicates a habitual tendency to omit conditions, which risks silent understatement of obligation; the same passage notes that a legal-ops team suffering from too many false positives might still prefer an omission-biased model, but would need to guard against missed risks (Yadav et al., 16 Jun 2026).

A central empirical point is that aggregate hallucination rates do not determine directional risk orientation. The reported comparison between gpt-5.2 and qwen3-32b shows that two models with nearly identical aggregate hallucination rates, approximately $$p_{\mathrm{extra}}(M) = \frac{\text{number of TP contradictions labeled extra\_condition}} {\text{total number of TP contradictions}}$$7 $$p_{\mathrm{extra}}(M) = \frac{\text{number of TP contradictions labeled extra\_condition}} {\text{total number of TP contradictions}}$$8, can carry RDI signs of opposite polarity. The paper further states that bootstrapped $$p_{\mathrm{extra}}(M) = \frac{\text{number of TP contradictions labeled extra\_condition}} {\text{total number of TP contradictions}}$$9 confidence intervals for these RDI values do not overlap, indicating that the directional signal is stable rather than an artifact of sampling variation.

5. Use in calibrated multi-agent debate

LegalHalluLens does not treat RDI as a purely retrospective audit statistic. The paper describes a six-role debate pipeline calibrated using RDI and the typed failure profile in two ways. The first is typed Skeptic challenges. If RDI indicates omission-heavy bias in Obligation clauses, Skeptic prompts focus on missing carve-outs and conditions. If RDI indicates invention-heavy bias, Skeptic questions instead probe whether each extracted qualifier actually appears in the source (Yadav et al., 16 Jun 2026).

The second mechanism is a pair of asymmetric add/delete safety gates. For addition, the absent-to-present gate is tightened because many high-risk categories, specifically numeric and obligation, had $$p_{\mathrm{missing}}(M) = \frac{\text{number of TP contradictions labeled missing\_condition}} {\text{total number of TP contradictions}}.$$0 and positive invention risk; under this rule, the pipeline blocks new detections unless both the Verifier and debate consensus agree. For deletion, the present-to-absent gate is designed to counter omission: the pipeline prevents deletion of a confirmed clause when the Verifier independently finds it.

These calibrations are explicitly contrasted with one-size-fits-all debate settings. The stated rationale is that debate should be targeted at the measured directional bias of the model. Empirically, the broader typed debate pipeline is reported to reduce fabricated detections by $$p_{\mathrm{missing}}(M) = \frac{\text{number of TP contradictions labeled missing\_condition}} {\text{total number of TP contradictions}}.$$1, with per-category gains tracking the diagnosis, and to match commercial APIs with a substantially smaller backbone of 4B active parameters. Within this design, RDI functions as a control signal for how adversarial review and final gating are configured.

6. Deployment significance, correction of bias, and scope

The deployment significance of RDI lies in its use for direction-aware procurement, accountability, and agent design. A common misconception in model comparison is that matched aggregate hallucination rates imply similar deployment behavior. The reported RDI results show otherwise: directionality can differ even when overall rates are nearly identical, and that difference can matter more than the aggregate rate in high-stakes legal workflows (Yadav et al., 16 Jun 2026).

The paper also reports that RDI can register the effect of intervention. After applying the typed debate pipeline to an omission-heavy model, gemma-4-26B-A4B, with $$p_{\mathrm{missing}}(M) = \frac{\text{number of TP contradictions labeled missing\_condition}} {\text{total number of TP contradictions}}.$$2, its Obligation RDI shifts to near zero, approximately $$p_{\mathrm{missing}}(M) = \frac{\text{number of TP contradictions labeled missing\_condition}} {\text{total number of TP contradictions}}.$$3. This is presented as evidence that calibrated Skeptic challenges can correct omission biases rather than merely documenting them.

RDI is also bounded in scope. It is judge-driven, lightweight, and intentionally focused on the extra-condition versus missing-condition axis. Because non-directional contradiction types such as scope remain in the denominator but not the numerator, RDI should not be read as a complete summary of all hallucination structure. A plausible implication is that RDI is best understood as one layer in a typed auditing stack: hallucination rate gives magnitude, typed profiles give localization, and RDI gives directional orientation. Within LegalHalluLens, its value is precisely that combination of compactness, sign, and direct operational use.