--- title: Sentence-level Hallucination Ratio (SHR) url: https://www.emergentmind.com/topics/sentence-level-hallucination-ratio-shr type: topic --- # Sentence-level Hallucination Ratio (SHR) The Sentence-level Hallucination Ratio (SHR) is a quantitative metric designed to measure the incidence of hallucinated output—defined as sentences containing fabricated or non-grounded entities—within model-generated text. SHR has emerged as a central diagnostic tool for multimodal large language models (MLLMs), vision-language models (VLVMs), and automatic speech recognition (ASR) systems, allowing researchers to isolate and quantify hallucination phenomena at a fine-grained, interpretable level [2507.12455][2404.14233][2510.16567]. ## 1. Formal Definition and Mathematical Formulation SHR is formally defined as the fraction of generated sentences flagged as hallucinated out of the total number of generated sentences within a given evaluation set. Let $S$ be the total number of sentences generated by a model, and $H_s$ the count of sentences classified as hallucinated. The metric is expressed as: \[ \mathrm{SHR} \;=\; \frac{1}{S}\;\sum_{i=1}^{S}\mathbf{1}\bigl(s_i\ \text{is hallucinated}\bigr) \] where $s_i$ denotes the $i$th generated sentence and $\mathbf{1}(\cdot)$ is the indicator function returning 1 for hallucinated sentences and 0 otherwise [2507.12455][2404.14233][2510.16567]. In the context of ASR outputs, SHR is equivalently defined, with hallucination determined by the presence of tokens not grounded in the acoustic input [2510.16567]. ## 2. Computation Pipeline and Annotation Schemes The computation of SHR involves distinct annotation and validation pipelines tailored to the modality of the generated response: - **Multimodal and Vision-Language Models:** 1. **Sentence Segmentation:** Responses are partitioned into sentences at boundary punctuation (typically periods). 2. **Object Extraction:** Each sentence is parsed to extract mentioned object entities using scene-graph parsing and standard NLP preprocessing (POS filtering, lemmatization). 3. **Validation Against Visual Inputs:** Each extracted entity is validated using two open-vocabulary detectors (e.g., GroundingDINO and YOLO-World). Entities confirmed present by both detectors are factual; those confirmed absent are hallucinated; detection disagreement yields “uncertain,” ignored in sentence classification. 4. **Sentence Labeling:** Sentences containing at least one hallucinated object and no factual contradictions are tagged as hallucinated. Sentences mentioning only factual or uncertain entities are considered non-hallucinated [2507.12455]. - **Sentence-level AI Feedback Annotation:** 1. **Classical Approach:** Sentence-level labels are generated by proprietary models (e.g., GPT-4/GPT-4V) with access to image annotations, which then provide binary hallucination judgments and severity scores (0–3). 2. **Learned Sentence-level Detector (H-DER):** Detection models such as InternVL-Chat-Plus fine-tuned via LoRA replicate the annotation protocol, outputting hallucination type (, , ), explanation, and severity [2404.14233]. - **Automatic Speech Recognition:** 1. **Alignment of Hypothesis & Reference:** Each ASR output is aligned with a human-annotated reference via edit distance. 2. **Hallucination Decision:** Insertions, unjustified substitutions, and semantically incoherent tokens trigger a hallucination label for the entire sentence [2510.16567]. ## 3. Severity-aware Extensions and Weighted Variants To address the binary coarseness of classic SHR, severity-aware variants have been developed [2404.14233]. Each sentence $j$ receives a severity score $s_j \in \{0,1,2,3\}$, and the length-weighted action score for a response $R$ is: \[ \text{Score}_{\text{action}}(R) = \frac{1}{N}\sum_{j=1}^{m}\left(s_j \times |\text{seg}_j|\right); \quad N = \sum_{j=1}^{m}|\text{seg}_j| \] where $|\text{seg}_j|$ is the token length of segment $j$. Setting $s_j = 1$ for any hallucinated sentence and $0$ otherwise recovers classic SHR. Severity-aware SHR emphasizes disproportionately harmful hallucinations by scaling their impact according to severity and segment length, which has been shown to better target major versus minor errors [2404.14233]. ## 4. Empirical Behavior and Benchmarking Reported SHR values across modalities and architectures illustrate its sensitivity to hallucination mitigation strategies: | Model | Dataset | Baseline SHR | SHR After Mitigation | Relative Reduction | |----------------------|------------------|--------------|----------------------|-------------------| | LLaVA-v1.5-7B | Object HalBench | 0.527 | 0.043 | ~91.8% | | LLaVA-v1.5-13B | Object HalBench | 0.460 | 0.033 | ~92.8% | | LLaVA | AMBER CHAIR_s | 0.463 | 0.053 | ~89% | | LLaVA-1.5 (MMHal-Bench)| MMHal-Bench | 0.57 | 0.48 | ~15.8% | | **ASR Example ([2510.16567])** | Insertion-only Toy Set | 0.4 | (N/A) | - | Empirical studies demonstrate that methods such as SENTINEL [2507.12455] and HSA-DPO [2404.14233] consistently reduce SHR by factors exceeding 90% in vision-captioning settings, confirming the utility of sentence-level early intervention and severity-weighted preference learning. ## 5. Practical Significance, Limitations, and Domain Nuances SHR is favored for its interpretability—reporting “the fraction of sentences containing any hallucination”—and its diagnostic utility in high-stakes applications such as medical ASR transcription [2510.16567]. Its use encourages strict monitoring policies (“never hallucinate”) and enables post-hoc analysis by examining the underlying hallucination dimensions (lexical, phonetic, morphological, semantic). However, several nuances and limitations are salient: - **Binary Coarseness:** SHR does not differentiate minor from major errors once a sentence is flagged; severity-aware variants partially address this issue [2404.14233]. - **Detector/Annotation Accuracy:** SHR is highly sensitive to the precision and recall of object detectors or annotation models; bias or false positives/negatives directly inflate or deflate $H_s$ [2507.12455]. - **Domain Shift:** Detector generalization to rare or specialized domains is limited; domain-specific errors can affect SHR reliability. - **Sentence Segmentation and Thresholds:** Both the segmentation protocol and detection confidence thresholds affect the quantitative SHR value, requiring consistency across comparative studies. - **No Error Decomposition:** SHR does not disaggregate hallucinations by error type; multi-dimensional scores such as SHALLOW provide complementary insights [2510.16567]. ## 6. Methodological Connections and Evaluation Strategies SHR integrates with broader methodologies involving preference learning frameworks and automated annotation pipelines: - **Iterative Contextual Bootstrapping (ICB):** Utilizing context-coherent non-hallucinated sentences to iteratively build robust preference data and enhance sentence-level discrimination in training [2507.12455]. - **Detect-then-Rewrite:** Employing sentence-level hallucination detection as a filtering mechanism in constructing training datasets and guiding model refinements [2404.14233]. - **Benchmark Alignment:** Quantitative SHR evaluations are conducted on held-out datasets such as Object HalBench, AMBER’s generative split, HallusionBench (image-context reasoning), and MHaluBench, enabling fair cross-model comparisons by fixing annotation schemes, detectors, and thresholds [2507.12455][2404.14233]. In summary, the Sentence-level Hallucination Ratio (SHR) is a robust, reproducible metric for quantifying the prevalence of hallucinated sentences in model output, distinguished by its generality across modalities and its role in benchmarking, training optimization, and risk analysis. Severity-graded extensions and automated pipelines further enhance its diagnostic resolution and practical relevance in contemporary multimodal and ASR modeling [2507.12455][2404.14233][2510.16567]. Source: https://www.emergentmind.com/topics/sentence-level-hallucination-ratio-shr