---
title: 'HVRR: Metric for Hallucinated Visual Reasoning'
url: https://www.emergentmind.com/topics/hallucinated-visual-reasoning-rate-hvrr
type: topic
---

# HVRR: Metric for Hallucinated Visual Reasoning

Hallucinated Visual Reasoning Rate (HVRR) is a grounding-sensitive metric for multimodal models that quantifies how often a system appears to reason visually by generating visual descriptions while its actual answer is invariant to the image. In its explicit formulation, HVRR measures the probability that a model produces at least one novel visual claim in its rationale and nevertheless returns the same answer when the image is shuffled, so that the visual language is present but the decision is behaviorally independent of the visual evidence [2603.03437]. Related work uses the same underlying idea under other names or implicit constructions—answer-level hallucination rates, scene-graph faithfulness, token-level hallucination probabilities, or reasoning–grounding trade-off curves—showing that HVRR belongs to a broader family of metrics for visual grounding and multimodal reasoning faithfulness [2407.15680].

## 1. Formal definition and relation to neighboring metrics

In the explicit medical-VQA formulation, hallucinated visual reasoning is defined from two ingredients: whether the model generates **novel visual claims (NVCs)** in its rationale, and whether its answer changes when the image is shuffled. The metric is

$$
\text{HVRR} = P[\text{NVC} = 1 \land a_{\text{real}} = a_{\text{shuffle}}],
$$

where \(a_{\text{real}}\) is the answer given the correct image and \(a_{\text{shuffle}}\) is the answer given a mismatched image from the same benchmark [2603.03437]. In words, HVRR is the probability that the model makes at least one novel visual claim and yet produces the exact same answer when the image is shuffled.

The same framework defines two closely related quantities. The **Novel Visual Claim Rate (NVCR)** is

$$
\text{NVCR} = P[\text{NVC} = 1],
$$

which measures how often the model generates visual language at all. The **conditional hallucination probability** is

$$
P[a_{\text{real}} = a_{\text{shuffle}} \mid \text{NVC} = 1] = \frac{\text{HVRR}}{\text{NVCR}},
$$

which measures how often those visual claims are behaviorally ungrounded once they appear [2603.03437].

HVRR is distinct from answer-only grounding metrics. The same paper defines **Visual Reliance Score (VRS)** as

$$
\text{VRS} = \text{Acc}_{\text{real}} - \text{Acc}_{\text{shuffle}},
$$

**Blank Drop (BD)** as

$$
\text{BD} = \text{Acc}_{\text{real}} - \text{Acc}_{\text{blank}},
$$

and **Image Sensitivity (IS)** as

$$
\text{IS} = P[a_{\text{real}} \neq a_{\text{shuffle}}].
$$

VRS and BD are accuracy-based counters under different image conditions, and IS is an answer-level change probability independent of correctness. HVRR differs by moving into the rationale and asking whether visual claims in that rationale have any causal effect on the answer [2603.03437].

This distinction matters because answer correctness alone cannot separate a model that actually inspects the image from one that relies on text priors and decorates its answer with plausible visual language. The medical formulation was introduced precisely to expose such “fake” or ungrounded visual reasoning in safety-critical settings [2603.03437].

## 2. Measurement procedure and detection of novel visual claims

The original HVRR evaluation uses a counterfactual framework with three image conditions for each example \((x_{\text{text}}, x_{\text{img}}, y)\): **Real** \((x_{\text{text}}, x_{\text{img}})\), **Blank** \((x_{\text{text}}, \text{Blank})\), where Blank is a uniform gray image of size \(224 \times 224\) with RGB \([128,128,128]\), and **Shuffled** \((x_{\text{text}}, \tilde{x}_{\text{img}})\), where \(\tilde{x}_{\text{img}}\) is a random image from the same benchmark [2603.03437]. The model is prompted to output

```text
<think> ... {rationale} ... </think>
<answer> ... {answer} ... </answer>
```

and is queried with deterministic decoding at temperature \(= 0\) [2603.03437].

An instance is counted as hallucinated visual reasoning if two conditions hold. First, the rationale for the Real image contains at least one novel visual claim. Second, the answers for the Real and Shuffled conditions are identical. The empirical estimator for model \(m\) is

$$
\text{HVRR}_m = \frac{1}{N}\sum_{i=1}^{N} \mathbf{1}[\text{NVC}_{i,m} = 1 \land a_{i,m,\text{real}} = a_{i,m,\text{shuffle}}],
$$

with the corresponding estimator for NVCR defined analogously over \(\mathbf{1}[\text{NVC}_{i,m}=1]\) [2603.03437].

The **novel visual claim detector** is heuristic. A statement is labeled as an NVC if visual observation language is present and the statement adds information not already present in the question text. The detector scans for tokens associated with presence (“shows”, “visible”), location (“left”, “upper”), appearance (“irregular”, “spiculated”), and severity (“mild”, “extensive”), then filters out overlapping phrases up to 5 words long between the question and the rationale [2603.03437]. This produces a binary indicator \(\text{NVC} \in \{0,1\}\).

The framework was evaluated on four medical VQA benchmarks—PathVQA, PMC-VQA, SLAKE, and VQA-RAD—with 100 examples sampled from each benchmark, giving 400 examples total [2603.03437]. To validate the detector, the authors manually audited 50 high-risk cases per model, defined as incorrect predictions with \(\text{NVC}=1\). Each case was labeled as **Grounded but wrong**, **Ungrounded hallucination**, or **Ambiguous**, and inter-annotator agreement on 20 such cases was measured using Cohen’s Kappa [2603.03437].

The paper explicitly notes several limitations of this measurement procedure. NVC detection relies on a fixed list of visual verbs and adjectives plus an n-gram overlap filter; answer equality under Real and Shuffled is only a behavioral proxy for grounding; the estimates are based on 100 examples per benchmark; and HVRR does not inspect the semantic correctness of visual claims directly except in the manual audit [2603.03437].

## 3. Empirical findings in multimodal medical reasoning

The central empirical result is that models frequently produce visual language without making image-dependent decisions. Averaged across the four medical benchmarks, the reported values are: **Baseline** NVCR 68%, HVRR 38%, conditional hallucination probability 54.5%, accuracy 57%; **RL(text)** NVCR 74%, HVRR 40%, conditional hallucination probability 53.4%, accuracy 56%; **RL(image)** NVCR 70%, HVRR 43%, conditional hallucination probability 60.9%, accuracy 59% [2603.03437]. Thus, models generate novel visual claims in 68–74% of responses, yet 38–43% of all responses are classified as hallucinated visual reasoning.

Benchmark-level breakdowns show that HVRR is not uniform. On **PathVQA**, HVRR is 51% for Baseline, 52% for RL(text), and 55% for RL(image). On **PMC-VQA**, the values are 24%, 27%, and 31%, which the paper identifies as the lowest hallucination rates among the four benchmarks. On **SLAKE**, HVRR is 35%, 40%, and 38%. On **VQA-RAD**, HVRR is 41%, 40%, and 48%, with RL(image) reaching a conditional hallucination probability of 69.6% [2603.03437].

A recurring finding is that accuracy improvements can coexist with worse grounding. The paper reports a strong negative correlation between VRS and HVRR across benchmarks, with Spearman \(\rho = -0.71\) and \(p < 0.05\), indicating that settings in which accuracy depends less on the image tend to show more hallucinated visual reasoning [2603.03437]. A striking example is VQA-RAD: both RL(text) and RL(image) reach 63% accuracy, but RL(image) has improved VRS, lower IS, and higher HVRR than the baseline, showing that answer accuracy alone does not reveal whether the model’s visual reasoning is genuine [2603.03437].

The motivating abstract reports the same pattern in more compact form. Text-only RLVR achieves negative VRS on PathVQA (\(-0.09\)), performing better with mismatched images, while image-text RLVR reduces image sensitivity to 39.8% overall despite improving accuracy. On VQA-RAD, text-only RLVR retains 81% performance with blank images, whereas image-text RLVR shows only 29% image sensitivity. These findings were presented as evidence that accuracy-only rewards enable shortcut exploitation and that progress requires grounding-aware evaluation protocols and training objectives that explicitly enforce visual dependence [2603.03437].

## 4. Alternative operationalizations in multimodal benchmarks

Several contemporaneous benchmarks do not define HVRR explicitly, but they provide operationalizations that closely match it. **HaloQuest** defines multimodal hallucination through three question categories—**False Premise**, **Visually Challenging**, and **Questions with Insufficient Context**—and evaluates correctness with either human raters or an LLM-based automatic judge. The paper uses accuracy as the central metric and interprets low accuracy as high hallucination, with hallucination rate implicitly approximated by \(1-\text{Accuracy}\). It further states that “all ingredients are present” for defining an HVRR-like metric, and gives a direct construction in which **Visual Reasoning Accuracy (VRA)** is computed on a chosen subset such as False Premise plus Visually Challenging questions and HVRR is defined as \(1-\text{VRA}\) [2407.15680]. In this setting, the metric is grounded not in counterfactual answer invariance but in error rates on premise-checking, fine-grained visual reasoning, and epistemic uncertainty.

A second line of work ties HVRR-like behavior to reasoning length. “More Thinking, Less Seeing?” introduces **RH-AUC**, which evaluates how reasoning accuracy and perception accuracy vary as reasoning length increases. The paper operationalizes hallucination on perception tasks through answer-level judgments and shows that longer reasoning chains lead to reduced focus on visual inputs and increased hallucination. In that framework, a direct answer-level analogue is \(1-H_T\), where \(H_T\) is perception accuracy at reasoning length \(T\); the paper explicitly presents this as the natural dual of the reasoning–hallucination trade-off curve [2505.21523]. This suggests that HVRR can also be viewed as a function of test-time compute rather than only a fixed property of a model.

A third formulation appears in **PRE-HAL**, which separates **Perception** and **Reasoning** capabilities across **instances**, **scenes**, and **relations**. There, hallucination is simply an incorrect answer on a visually dependent multiple-choice item, but the benchmark labels whether the item targets reasoning rather than perception. The paper highlights that reasoning-based tasks pose greater challenges than perception-based tasks and reports relation reasoning hallucination rates of 49.44% for LLaVA-v1.5, 50.42% for mPLUG-Owl2, and 19.92% for mPLUG-Owl3 [2506.19513]. This supports an answer-level HVRR defined as the error rate restricted to reasoning-labeled items, especially relation reasoning items.

Work on cognitive hallucination mitigation makes the same point from the generation side. “Visual Description Grounded Decoding” states that HVRR is not defined explicitly but that its methods and results support a plausible formalization from GPT-based **Factuality** scores on cognitive prompts. Because the paper repeatedly treats factual misalignment in cognitive visual reasoning as the main form of hallucination, and reports factuality gains of 2–33% under VDGD, it explicitly argues that a factuality-thresholded or continuous HVRR would decrease under the proposed decoding method [2405.15683].

## 5. Fine-grained, structured, and mechanistic variants

One extension of HVRR moves from answers to tokens. **HalLoc** provides 155,953 token-level annotated samples across VQA, instruction-following, and captioning, with hallucination types **object**, **attribute**, **relationship**, and **scene**. The paper defines token-level labels \(H_h(t)\in\{0,1\}\) and proposes probabilistic token-level hallucination detection, then explicitly notes that the same machinery can define HVRR over “reasoning tokens,” such as tokens conveying objects, attributes, relations, or scenes that underpin an answer. It gives natural token-level and instance-level rate formulas and argues that expected HVRR can be estimated from calibrated per-token probabilities \(p_{h,t}\) [2506.10286]. This variant changes HVRR from a sample-level event into a graded quantity over spans or steps.

A second extension uses structured visual representations. **SceneAlign** models reasoning faithfulness with scene graphs \(SG=(E,A,R)\) and constructs hard negative rationales via four targeted perturbations: **swap**, **replace**, **shorten**, and **overthink**. These are designed to simulate hallucinated entities, mis-grounded relations, skipped steps, and over-specified reasoning. Although the paper does not name HVRR directly, it gives an explicit path to define it either by thresholding graph conflict or by using HallusionBench faithfulness accuracy \(fAcc\), with the text stating that \(1-fAcc\) is a natural approximation to an HVRR-style quantity on that benchmark [2601.05600]. In this formulation, HVRR becomes a scene-graph inconsistency rate rather than an answer-invariance rate.

Interactive physical reasoning produces a third variant. **VAORA** treats “hallucinated chain-of-thought reasoning that contradicts physical reality” as a primary failure mode, and introduces a **Visual Alignment Reward** for scene grounding plus **Visual-Action Alignment Reward** terms for collision and placement consistency. The paper does not name HVRR, but states that it is essentially built around that exact failure mode and provides the symbolic machinery to count hallucinated items in scene descriptions, collision predictions, and placement descriptions [2607.06522]. This suggests a per-symbol HVRR in which hallucination is the fraction of generated symbolic items that fail consistency checks against the initial scene or the post-action outcome.

Mechanistic work on reasoning hallucination provides a complementary perspective. “Detection and Mitigation of Hallucination in Large Reasoning Models” introduces a **Reasoning Score** based on Jensen–Shannon divergence between late-layer and final-layer token distributions, then builds a **Reasoning Hallucination Detection** score from average reasoning depth, early coefficient of variation, attention to abnormal earlier steps, and the correlation between reasoning score and perplexity. The paper is text-only, but it explicitly argues that the same ingredients extend to visual reasoning once multimodal logits, cross-modal attention, and image-grounding checks are available, yielding a natural mechanistic basis for HVRR [2505.12886].

## 6. Limitations, controversies, and future directions

The main controversy around HVRR is what exactly should count as hallucinated visual reasoning. In the explicit medical definition, HVRR flags cases where the model makes novel visual claims while the answer remains invariant under shuffling, but the same paper notes that this is only a behavioral proxy: occasionally a visually grounded chain may still end in the same answer when images are shuffled, especially when many images in the benchmark share the same label for a given question type [2603.03437]. The metric therefore targets causal visual dependence rather than semantic correctness alone.

A second limitation is granularity. The original metric is binary at the sample level. It does not quantify degree of hallucination, and mixed answers—partly correct reasoning with a small hallucinated detail—are still counted only as correct or incorrect. The paper also notes that its NVC detector is heuristic, that its manual audit covers only 50 high-risk cases per model, and that its estimates are based on 100 examples per benchmark [2603.03437]. Related work reaches the same conclusion from another angle: token-level frameworks such as HalLoc were motivated precisely because binary whole-answer judgments ignore ambiguity and gray areas, and because a calibrated confidence over hallucination is often more useful than a definitive label [2506.10286].

A third open question concerns what to reward during training. The medical HVRR work argues that progress requires grounding-aware evaluation protocols and training objectives that explicitly enforce visual dependence [2603.03437]. Other papers specify possible directions. The medical paper itself lists improved visual-claim detection, correctness-conditioned variants, per-category HVRR, and direct use of HVRR components as RL penalties as natural extensions [2603.03437]. VDGD suggests that cognitive hallucination can be reduced by strengthening visual perception during decoding rather than only by changing answer supervision [2405.15683]. SceneAlign suggests that hard negatives grounded in structured visual interventions are more effective than text-only preference signals for reducing unfaithful reasoning [2601.05600].

Finally, HVRR is increasingly understood as one member of a larger class of grounding-faithfulness measures. Video generation benchmarks report step-level reasoning failures and hint-induced hallucinations; long-chain multimodal reasoning benchmarks show that “more thinking” can produce “less seeing”; and embodied or physical reasoning frameworks treat hallucinated chain-of-thought as misalignment between stated reasoning and observed outcomes [2603.20194]. This suggests that future HVRR variants are likely to become more domain-specific—token-level in open-ended VQA, step-level in video and embodied reasoning, and structure-level in scene-graph or simulator-based settings—while preserving the same central idea: measuring when a model says it is using visual evidence although its behavior shows that it is not.

Source: https://www.emergentmind.com/topics/hallucinated-visual-reasoning-rate-hvrr