GapHalu: Evaluating Gaps & Hallucinations

• GapHalu is an analytic framework that evaluates machine-generated legal texts by distinguishing neutral gaps from critical hallucinations.
• It employs a hierarchical 14-leaf taxonomy with quantifiable metrics like mGF1, precision, and recall to categorize error types including intrinsic, target mismatch, and citation mismatches.
• The framework leverages in-context learning with LLM-based detectors to provide actionable insights for legal professionals and model developers improving output fidelity.

GapHalu refers to an evaluation metric and analytic framework designed for the assessment of machine-generated outputs, particularly legal text, distinguishing between gaps—neutral differences between generated and reference content—and hallucinations—specific factual, logical, or grounding errors. GapHalu provides a systematic, fine-grained approach for identifying, categorizing, and quantifying these discrepancies, allowing researchers and practitioners to gauge both the frequency and nature of significant unfaithfulness in LLM outputs (Hou et al., 2024).

1. Formal Definition and Motivation

A "gap" is defined as any dissimilarity between a machine-generated legal analysis and a human-written (target) analysis. This definition is neutral; not all gaps are harmful or erroneous. In contrast, a "hallucination" is a specific subset of gaps introducing incoherence, unfaithfulness, or factual error—i.e., content that is either internally inconsistent with the prompt/context ("intrinsic") or externally unfaithful to authoritative sources ("extrinsic").

GapHalu operationalizes these notions by decomposing evaluation into taxonomized error types and by providing both categorical and aggregate metrics. The motivation is to enable fine-grained, domain-sensitive audit of LLM-generated professional outputs, distinguishing stylistic or organizational differences from unambiguous factual fabrication or misattribution (Hou et al., 2024).

2. Taxonomy of Gaps and Hallucinations

GapHalu employs a 14-leaf taxonomy hierarchically grouped under three high-level error types, with hallucinations defined as a strict subset:

• Intrinsic Gaps ($G^1$): Deviations disrupting logical or linguistic quality (e.g., redundancy, citation format/style, structural error).
• Target Mismatch ($G^2$): Dissimilarities that re-organize, rephrase, or reframe content, generally remaining legally valid (e.g., alternative rhetorical constructions, organizing citations differently), usually not considered hallucinations.
• Citation Content Mismatch ($G^3$): Errors in factual citation, attribution, or grounding—these constitute hallucinations (e.g., claim hallucination, citation hallucination, irrelevant retrieval).

Under this framework, hallucinations are rigorously formalized as the union of intrinsic and citation content mismatches $(G^1 \cup G^3)$, in contrast to target mismatches, which are recognized as acceptable variation in expert judgment and rhetoric (Hou et al., 2024).

3. Quantitative Metrics and Mathematical Formalism

GapHalu introduces explicit metrics for the presence of gaps and hallucinations at the example level. For a generated paragraph $p_t$ and its ground-truth, define:

• True multi-hot label vector: $g_t = (G^1_t, \ldots, G^m_t)^\top$, $G^k_t \in \{0, 1\}$
• Detector prediction: $\hat g_t = (\hat G^1_t, \ldots, \hat G^m_t)^\top$

Performance metrics:

$$\begin{aligned} \mathrm{GEM}_i &= \mathbb{I}[\hat g_i = g_i] \ \mathrm{GP}_i &= \frac{\hat g_i \cdot g_i}{\lVert \hat g_i\rVert^2} \ \mathrm{GR}_i &= \frac{\hat g_i \cdot g_i}{\lVert g_i\rVert^2} \ \mathrm{GF1}_i &= \frac{2\,\mathrm{GP}_i\,\mathrm{GR}_i}{\mathrm{GP}_i + \mathrm{GR}_i} \end{aligned}$$

Aggregate metrics across $N$ samples:

$G^2$0

Crucial composite outcomes are:

$G^2$1

Here, GapScore measures the overall rate of any type of gap; GapHalu isolates the presence of the more serious hallucination classes (Hou et al., 2024).

4. Detection Methodology and LLM-Based Evaluation

GapHalu detection leverages in-context, few-shot prompting: LLMs (such as GPT-4o, Llama-3.1-8B-Instruct, and Mistral-Nemo-Instruct-2407) are provided with up to 20 annotated human-labeled examples and instructed to classify generated paragraphs into gap categories. Detectors assign multi-class labels at both the high-level (no gap, intrinsic, target mismatch, citation mismatch) and potentially the fine-grained leaf level.

No parameter fine-tuning is applied; all learning occurs through prompt demonstration (pure in-context learning). Detection performance on test examples reaches F1 ≈ 0.67–0.68 (mGF1), precision ≈ 0.75–0.80 (mGP), and exact match rate mGEM ≈ 0.47–0.52, indicating that in-context LLM detectors can robustly identify both benign gaps and more serious hallucinations (Hou et al., 2024).

5. Empirical Results and Statistical Insights

Evaluation on machine-generated legal analysis (CLERC generations) shows that gaps are ubiquitous: ~96% of paragraphs contain at least one gap (GapScore ≈ 96%). When restricting to hallucinations, GapHalu shows that ≈ 80% of SOTA LLM generations contain at least one hallucination (GapHalu ≈ 80%), split across intrinsic and citation-content mismatch subtypes.

A summary of salient result metrics (Mistral-Nemo detector, 500 CLERC generations):

Metric	Value (GPT-4o Gen.)	Value (Llama-3 Gen.)
GapScore	96.31	95.46
GapHalu	79.51	82.05
Intrinsic G¹	24.80	25.20
Target G²	82.99	84.96
Citation G³	61.48	60.94

This indicates that although nearly all outputs diverge stylistically or structurally from human targets, a substantial fraction still commit strict hallucinations that can undermine legal validity (Hou et al., 2024).

6. Implications, Use Cases, and Limitations

GapHalu facilitates granular, expert-centric auditing of LLM outputs, enabling distinction between inevitable, often harmless rhetorical reformulation ("target mismatch") and risky, factually problematic hallucinations. For legal professionals, this supports prioritization of human intervention. For model developers, GapHalu's taxonomy and metrics inform targeted mitigation (e.g., improved citation handling, retrieval augmentation).

Limitations include imperfect coverage of the finest-grained categories due to the high annotation cost, possible annotation noise in citation string extraction, and the neural detector's own susceptibility to domain shift and error under-prediction. The present framework is developed on a relatively small, expert-annotated dataset. Future work is directed at extending GapHalu for general text generation tasks with multiple legitimate ground truths, broader fine-tuned classifier deployment, and integration of symbolic reasoning for further hallucination reduction (Hou et al., 2024).

7. Generalization and Future Directions

GapHalu, although developed in the context of machine-generated legal analysis, introduces formal definitions, a scalable taxonomy, and metricization potentially applicable to any professional multi-ground-truth generative setting. Extensions may involve adaptation to other domains (e.g., scientific text), development of domain-specific fine-grained detectors, and leveraging symbolic-verification in conjunction with neural models to further minimize hallucination rate. A plausible implication is that as generative models are increasingly deployed in high-stakes fields, frameworks like GapHalu will become integral to automated quality auditing and risk mitigation (Hou et al., 2024).