Entity Hallucination Index (EHI)

• Entity Hallucination Index (EHI) is an entity-faithfulness metric that quantifies five key aspects—PH, EF, NH, OF, and LF—to evaluate grounded entity representation in summaries.
• It uses spaCy NER for extracting named entities and applies exponential scaling to balance rewards for correct extractions against penalties for hallucinations and omissions.
• The metric is integrated as a reinforcement learning reward in summarization models, with experiments showing improved stability and higher Entity F1 scores post fine-tuning.

Searching arXiv for the cited EHI and related entity-hallucination papers to ground the article. arXiv search: "Reducing Hallucinations in Summarization via Reinforcement Learning with Entity Hallucination Index" Entity Hallucination Index (EHI) is an entity-centric faithfulness score introduced for abstractive summarization as a lightweight, automatic metric and reward signal for measuring and reducing entity-level hallucination. In the formulation reported for meeting summarization, EHI is intended to quantify whether named entities in a generated summary are grounded in the source transcript, while also accounting for useful grounded additions, unsupported insertions, overconcentration on a narrow subset of entities, and omission of important entities. The score is used not only for post hoc evaluation but also directly as a reinforcement-learning reward in fine-tuning a summarization model, with the stated aim of biasing generation toward entity-faithful outputs (Katwe et al., 30 Jul 2025).

1. Definition and semantic scope

The paper defines EHI as an entity-faithfulness score that captures five aspects of entity behavior: Positive Hallucination (PH), Extractiveness Factor (EF), Negative Hallucination (NH), Overfocused Relations (OF), and Lost Focus (LF). PH denotes newly introduced entities that are factually correct and helpful. EF denotes entities correctly extracted from the input document into the summary. NH denotes hallucinated entities that are incorrect or not grounded in the source. OF penalizes summaries that over-focus on a narrow subset of entities and miss diversity. LF penalizes summaries that omit important entities present in the input (Katwe et al., 30 Jul 2025).

Component	Role in the score
PH	Newly introduced entities that are factually correct and helpful
EF	Entities correctly extracted from the input document into the summary
NH	Hallucinated entities that are incorrect or not grounded in the source
OF	Penalty for over-focus on a narrow subset of entities
LF	Penalty for omission of important source entities

A central interpretive point is that higher EHI means better entity faithfulness. The metric rewards correct extraction and useful grounded additions, and penalizes hallucinated, overly narrow, or missing entity coverage. The paper explicitly notes a common source of confusion: despite the term “hallucination” in its name, EHI is used like a reward, so higher values are better rather than worse (Katwe et al., 30 Jul 2025).

The paper also states that when NH, OF, and LF are all zero, EHI reduces to 1, regardless of PH and EF values, and interprets that case as a fully faithful summary. Because the printed equation is malformed in the provided text, the exact closed-form algebraic structure is not recoverable from the excerpt. What is recoverable is the intended semantics: EHI is an exponentially scaled composite index over the five factors, with PH and EF functioning as positive terms and NH, OF, and LF functioning as penalties (Katwe et al., 30 Jul 2025).

2. Computation, extraction, and grounding logic

The reported EHI computation pipeline begins by extracting named entities from both the source transcript and the generated summary. Entity extraction is performed using spaCy NER, specifically en_core_web_sm. Matching is case-insensitive and done at the entity string level. The grounding logic is then applied as follows: entities in the summary that also appear in the source are treated as grounded; entities not found in the source contribute to hallucination penalties; missing important source entities contribute to lost-focus penalties; overconcentration on a small subset of entities contributes to overfocus penalties; and additional correct entities can contribute positively if judged as beneficial (Katwe et al., 30 Jul 2025).

The paper gives a corrupted typeset equation for EHI, but it also specifies the variable set

$$x_i \in \{\text{PH}, \text{EF}, \text{NH}, \text{OF}, \text{LF}\},$$

and explains that the score uses exponential scaling to balance rewards and penalties. This suggests that the metric is designed to amplify differences between grounded and ungrounded entity behavior while retaining a bounded interpretation in which perfect grounding yields $\text{EHI} = 1$ (Katwe et al., 30 Jul 2025).

In the same experimental context, the paper evaluates entity overlap against reference summaries using Entity F1. The reported formulas are

$$\text{Precision} = \frac{|E_{ref} \cap E_{gen}|}{|E_{gen}|},$$

$$\text{Recall} = \frac{|E_{ref} \cap E_{gen}|}{|E_{ref}|},$$

$$\text{Entity F1} = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}.$$

The provided text notes another formatting issue in the excerpted formula, but states that the intended expression is the standard harmonic mean (Katwe et al., 30 Jul 2025).

3. Use as a reinforcement-learning reward

The defining methodological move in the EHI paper is to optimize entity faithfulness directly by using EHI as a reward in a policy-gradient or REINFORCE-style fine-tuning setup. Given input document $x$, generated summary $y$, and model parameters $\theta$, the objective is

$$J(\theta) = \mathbb{E}_{y \sim p_\theta(y|x)} [\text{EHI}(y, x)].$$

The gradient estimate is reported as

$$\nabla_\theta J(\theta) = \mathbb{E}_{y \sim p_\theta(y|x)} [\text{EHI}(y, x)\nabla_\theta \log p_\theta(y|x)].$$

The operational interpretation given in the paper is straightforward: sample a summary from the model, compute its EHI score, and update parameters to increase the probability of summaries with higher EHI (Katwe et al., 30 Jul 2025).

The training configuration reported in the paper uses Flan-T5-Large as the base model, Adam as the optimizer, and a learning rate of $\text{EHI} = 1$0. Rewards are normalized within each batch. Summaries are regenerated every 500 updates. Fine-tuning is performed directly with RL, with no intermediate supervised cross-entropy stage. The paper emphasizes that this removes dependence on human-written factuality annotations and enables scalable fine-tuning through an automatic entity-faithfulness reward (Katwe et al., 30 Jul 2025).

This use of EHI distinguishes it from evaluation-only factuality measures. In the reported setup, the metric is not merely diagnostic; it is embedded in the optimization loop and shapes the model’s conditional distribution over summaries. A plausible implication is that EHI occupies a dual role: it is simultaneously a measurement instrument and a training signal.

4. Experimental setting and reported behavior

The summarization experiments are reported on AutoMin 2025 data, using the ELITR Minuting Corpus for summarization and ELITR-Bench for QA. Samples are randomly partitioned into 80% train, 10% validation, and 10% test. Because Flan-T5-Large has a 1024-token context limit and transcripts can be much longer, transcripts are split into overlapping chunks with chunk size up to 950 tokens and stride 200; each chunk is summarized independently, and the summaries are aggregated. For summarization, the reported evaluation metrics are EHI, Entity F1, ROUGE-1, ROUGE-2, ROUGE-L, BERTScore, and BARTScore. For QA, evaluation uses GPT-4o as LLM judge with maximum score 10 and mean score reported (Katwe et al., 30 Jul 2025).

The paper reports that before fine-tuning, EHI values were volatile and scores often dropped to 0.0, while Entity F1 was often below 0.5 with many zeros. After EHI-guided RL fine-tuning, EHI became more stable, with scores mostly in the 0.3–0.6 range, and Entity F1 often improved to 0.6–1.0. The accompanying qualitative table reports EHI behavior as “volatile, inconsistent” before fine-tuning and “stable, moderate” after fine-tuning, and reports the post-fine-tuning EHI–F1 correlation as “strong inverse correlation” (Katwe et al., 30 Jul 2025).

A qualitative example in the paper illustrates the intended effect. Before RL, the summary “The speaker discussed a new deal between IBM and Oracle to expand into Asian markets.” receives EHI $\text{EHI} = 1$1, with IBM identified as hallucinated or not grounded in the transcript. After RL, the summary “The speaker discussed the existing agreement with Oracle and Microsoft about service expansion.” receives EHI $\text{EHI} = 1$2, and the paper interprets this as improved grounding through avoidance of the unsupported IBM mention (Katwe et al., 30 Jul 2025).

The same paper also reports mean scores on a monolingual summarization/QA benchmark table: baseline_gpt-4o-2024-11-20 at 7.74, baseline_llama-3.1-8B-instruct at 7.08, baseline_phi-4-mini-instruct at 6.84, baseline_phi-3-small-128k-instruct at 6.65, baseline_llama-3.2-3B-instruct at 6.33, GETALP@AutoMin at 4.55, GETALP@AutoMin_amr at 4.34, GETALP@AutoMin_amr_only at 2.73, and HallucinationIndexes@AutoMin (ours) at 2.28. The paper does not present these QA figures as an EHI score; they are reported as part of the broader benchmark summary (Katwe et al., 30 Jul 2025).

5. Relation to adjacent entity-level hallucination metrics

EHI belongs to a broader line of work on entity-level factual consistency, but it is not identical to earlier source-grounding metrics. In “Entity-level Factual Consistency of Abstractive Text Summarization,” the central entity-level hallucination metric is the source-based precision score

$\text{EHI} = 1$3

which measures the percentage of summary entities supported by the source. That work treats low $\text{EHI} = 1$4 as evidence of entity hallucination and combines it with target-side metrics $\text{EHI} = 1$5, $\text{EHI} = 1$6, and $\text{EHI} = 1$7. It does not define a metric literally called EHI, but $\text{EHI} = 1$8 serves a closely related purpose (Nan et al., 2021).

A subsequent long-input summarization study on PubMed likewise does not define EHI explicitly. It adopts scispaCy NER and the same entity-overlap style of evaluation, using $\text{EHI} = 1$9 as the key source-faithfulness indicator and $$\text{Precision} = \frac{|E_{ref} \cap E_{gen}|}{|E_{gen}|},$$0 as a target-based entity agreement metric. In that setting, filtering article-abstract pairs with low $$\text{Precision} = \frac{|E_{ref} \cap E_{gen}|}{|E_{gen}|},$$1 improves source-based entity precision even when it does not improve ROUGE or target-side Entity F1 (Rehman et al., 2023).

Work on entity-level hallucination detection rather than summarization reward makes the contrast sharper. “Can Your Uncertainty Scores Detect Hallucinated Entity?” introduces HalluEntity, a dataset with entity-level hallucination labels built from ChatGPT-generated biographies in FActScore, and evaluates uncertainty-based detectors over 17 modern LLMs. That paper does not define an EHI, but it supplies entity-level labels, per-entity scores, and evaluation measures such as AUROC, AUPRC, $$\text{Precision} = \frac{|E_{ref} \cap E_{gen}|}{|E_{gen}|},$$2, $$\text{Precision} = \frac{|E_{ref} \cap E_{gen}|}{|E_{gen}|},$$3, and $$\text{Precision} = \frac{|E_{ref} \cap E_{gen}|}{|E_{gen}|},$$4. It argues that sentence-level detection is too coarse and that simple token-probability uncertainty over-predicts hallucinations, whereas context-aware methods perform better but remain suboptimal (Yeh et al., 17 Feb 2025).

A different nearby formulation appears in “Mitigating Entity-Level Hallucination in LLMs,” which introduces Dynamic Retrieval Augmentation based on hallucination Detection (DRAD). That paper does not define EHI either. Instead, it defines an entity hallucination decision rule using entity probability $$\text{Precision} = \frac{|E_{ref} \cap E_{gen}|}{|E_{gen}|},$$5 and entity entropy $$\text{Precision} = \frac{|E_{ref} \cap E_{gen}|}{|E_{gen}|},$$6: $$\text{Precision} = \frac{|E_{ref} \cap E_{gen}|}{|E_{gen}|},$$7 This situates EHI within a larger methodological space in which entity hallucination may be scored continuously, classified by threshold, or used to trigger retrieval and revision (Su et al., 2024).

There is also a cross-modal analogy. “Hallucination Index: An Image Quality Metric for Generative Reconstruction Models” defines hallucination as model-induced structural variability in image reconstruction and measures it as a Hellinger distance between a reconstruction distribution and a zero-hallucination reference distribution. The paper’s own framing states that this is conceptually similar to an EHI-style idea in other domains: the reference represents content changes attributable only to an allowed corruption process, and the distance quantifies additional structure introduced by the generative model (Tivnan et al., 2024). This suggests a broader conceptual pattern in which “hallucination index” denotes deviation from a no-hallucination reference, even though the operational definitions differ substantially across text and imaging.

6. Limitations, interpretive issues, and common misconceptions

The EHI paper identifies several limitations. First, EHI focuses on named entities, so it does not fully capture relation hallucination or broader factual inconsistencies. Second, because EHI uses spaCy NER with en_core_web_sm, its reliability depends on entity extraction quality, and NER errors propagate into the score. Third, string-level, case-insensitive matching can miss paraphrases, aliases, normalized entity forms, and partial matches. Fourth, the paper notes a trade-off with abstraction: the model can become overly conservative, prioritize exact entity copying, and thereby reduce paraphrasing or abstractiveness. Fifth, EHI is not presented as a complete factuality metric; it is described instead as a useful entity-faithfulness signal, a reward for RL fine-tuning, and a complement to other measures such as relation-level factuality checks (Katwe et al., 30 Jul 2025).

A recurrent misconception is to treat EHI as a generic hallucination score for all forms of factual error. The paper does not support that interpretation. Its assumptions are narrower: the source transcript is the grounding reference, named entities are a major carrier of hallucination risk, and entity faithfulness can be improved through reward-based policy optimization. Another misconception would be to read the score’s name literally and assume that lower is better. The paper states the opposite: higher EHI means better entity alignment and fewer harmful hallucinations (Katwe et al., 30 Jul 2025).

The malformed printed equation is also an interpretive issue. The exact algebraic form cannot be reconstructed from the excerpted typesetting, but the semantics are explicitly described: EHI is an exponentially scaled composite index over PH, EF, NH, OF, and LF, with perfect grounding mapped to $$\text{Precision} = \frac{|E_{ref} \cap E_{gen}|}{|E_{gen}|},$$8. For practical and comparative purposes, the paper therefore defines EHI more clearly through its workflow and behavioral interpretation than through the printed equation.

In that sense, EHI is best understood not as a fully general theory of hallucination, but as a specific entity-centric operationalization of faithfulness in summarization. Its significance lies in combining automatic entity grounding logic with direct optimization, thereby turning entity-level hallucination from a post hoc diagnostic into an explicit training objective (Katwe et al., 30 Jul 2025).