---
title: Visual Reliance Score in Vision-Language Models
url: https://www.emergentmind.com/topics/visual-reliance-score-vrs
type: topic
---

# Visual Reliance Score in Vision-Language Models

Searching arXiv for recent papers on Visual Reliance Score and closely related formulations.
I’m looking up arXiv entries relevant to “Visual Reliance Score”, “Visual Dependence Score”, and “Visual Robustness Score”.
Visual Reliance Score (VRS) is not a single universally standardized construct in current vision-language research. Across recent arXiv work, the term names at least three closely related families of quantities: an accuracy-gap grounding metric defined by counterfactual image substitution, a robustness metric against textual traps, and step-level model-internal measures of how strongly a reasoning step depends on visual input. A further terminological complication is that “VRS” can also denote “Visual Reasoning Scaling” in test-time inference work, where it is unrelated to visual reliance [2603.03437; 2601.04897; 2605.13467; 2606.11576].

## 1. Terminological scope and disambiguation

In current usage, “Visual Reliance Score” functions less as a single canonical metric than as a family of operationalizations for the question: how much does a model’s behavior depend on the image rather than on textual priors, prompt bias, or purely linguistic continuation? The principal distinctions are between dataset-level counterfactual metrics, benchmark-specific robustness scores, and step-level internal dependence measures.

| Formulation | Definition | Setting |
|---|---|---|
| Visual Reliance Score | $\text{VRS} = Acc_{\text{real}} - Acc_{\text{shuffle}}$ | Counterfactual grounding evaluation |
| Visual Robustness Score | Harmonic mean of $\mathrm{mAcc}$ and $1-\mathrm{mTDS}$ | Text-bias stress testing |
| Visual Dependence Score | $V_k = \log \pi_\theta(h_k \mid \mathbf{I},\dots) - \log \pi_\theta(h_k \mid \mathbf{I}_{\text{white}},\dots)$ | Step-level multimodal RL |
| Visual attention / dependency signals | Attention to visual tokens; Hellinger-distance dependency under visual-token removal | Visual reasoning diagnostics |

The most explicit naming clarification appears in "AVIS: Adaptive Test-Time Scaling for Vision-Language Models" [2606.11576]: there, VRS does **not** mean Visual Reliance Score, but **Visual Reasoning Scaling**, the axis controlling the number of reasoning rollouts $K$. That usage concerns inference-time reasoning search rather than grounding, dependence, or reliability.

A plausible implication is that any encyclopedia treatment of VRS must be polysemous. The shared conceptual core is not a single formula but a recurring research objective: isolating the causal or behavioral contribution of visual evidence.

## 2. Counterfactual accuracy-gap VRS

A direct and influential formulation defines VRS through counterfactual image substitution. In "Beyond Accuracy: Evaluating Visual Grounding In Multimodal Medical Reasoning" [2603.03437], each example is evaluated under three conditions—real, blank, and shuffled images—and the paper defines
$$
\text{VRS} = Acc_{\text{real}} - Acc_{\text{shuffle}}.
$$
Here, $Acc_{\text{real}}$ is accuracy with the correct image, and $Acc_{\text{shuffle}}$ is accuracy when the image is replaced by a random image from the same benchmark.

This definition measures dependence on the **correct image/question pairing**. If a model truly benefits from the specific image, then $Acc_{\text{real}} > Acc_{\text{shuffle}}$ and VRS is positive. If performance is similar under both conditions, VRS is near zero, suggesting that the identity of the image contributes little. Negative VRS indicates that the model performs better with mismatched images than with correct ones, which the paper interprets as strong evidence of shortcut exploitation.

The same work emphasizes that VRS is only one grounding metric within a larger counterfactual framework. Blank Drop is defined as $Acc_{\text{real}} - Acc_{\text{blank}}$, Image Sensitivity as $\Pr[a_{\text{real}} \neq a_{\text{shuffle}}]$, and Hallucinated Visual Reasoning Rate as $\Pr[NVC = 1 \land a_{\text{real}} = a_{\text{shuffle}}]$. This framing matters because VRS is accuracy-based: it can reveal whether correct pairing helps, but it does not by itself determine whether the model changes its answer when the image changes, nor whether visual claims in the rationale are grounded.

The empirical results illustrate both the usefulness and the limitation of this definition. On PathVQA, the text-only RLVR variant has $Acc_{\text{real}} = 56\%$, $Acc_{\text{shuffle}} = 65\%$, and $VRS = -0.09$, meaning performance is better with mismatched images. On PMC-VQA, RL(image) improves accuracy from $50\%$ to $57\%$ while VRS drops from $0.25$ to $0.13$. On VQA-RAD, RL(image) improves VRS from $0.09$ to $0.17$ while Image Sensitivity drops from $0.43$ to $0.29$. These cases establish the central interpretive point: accuracy gains do not guarantee stronger visual reliance.

## 3. Trap-resistance and the Visual Robustness Score

A second formulation appears in "V-FAT: Benchmarking Visual Fidelity Against Text-bias" [2601.04897], where VRS names the **Visual Robustness Score** rather than the counterfactual accuracy gap. The benchmark constructs conflicts between visual evidence and textual information at three levels: internal corpus bias (L1), external instruction bias (L2), and synergistic bias where both coincide (L3). For level $L_n$, it defines
$$
\mathrm{mAcc}_{L_n} = \frac{1}{N}\sum_{i=1}^N \mathbbm{1}(\hat{y}_i = y_i),
$$
$$
\mathrm{mTDS}_{L_n} = \frac{1}{N}\sum_{i=1}^N \mathbbm{1}(\hat{y}_i = y_{trap}),
$$
$$
R_{L_n} = 1 - \mathrm{mTDS}_{L_n},
$$
and then
$$
\mathrm{VRS}_{L_n} = 2 \cdot \frac{\mathrm{mAcc}_{L_n} \cdot R_{L_n}}{\mathrm{mAcc}_{L_n} + R_{L_n}}.
$$

This construction separates three outcomes: visually correct answers, trap-conforming answers, and other errors. Its purpose is to penalize “lucky” linguistic guesses and reward resistance to textual traps. A model can therefore have moderate or even high raw accuracy while receiving a lower VRS if many of its successes coexist with frequent trap-following. The harmonic mean makes low performance in either correctness or trap-resistance decisive.

The V-FAT results show that robustness to text bias does not scale as rapidly as overall accuracy. In the ablation table, Qwen2.5-VL-7B, 32B, and 72B obtain average VRS values of $0.71$, $0.75$, and $0.79$, respectively. Qwen3-8B-Instruct obtains $0.75$, while Qwen3-8B-Thinking obtains $0.72$. The paper interprets the latter pattern as evidence that longer reasoning traces may amplify external instruction bias by rationalizing misleading prompts rather than rechecking the image.

Level-wise results are used to diagnose “visual collapse” under textual dominance. Seed 1.6 drops from L1 VRS $0.651$ to L3 VRS $0.514$, and Claude-Haiku from $0.685$ to $0.567$, whereas Gemini-3-Flash reaches L3 VRS $0.895$. In this benchmark, then, VRS is not a generic grounding gap but a robustness index against trap-conditioned language priors.

## 4. Step-level and model-internal formulations

A third family of formulations treats visual reliance as a **step-level internal property** rather than a dataset-level accuracy difference. In "PDCR: Perception-Decomposed Confidence Reward for Vision-Language Reasoning" [2605.13467], the core construct is the **Visual Dependence Score**:
$$
V_k^{(i)} = \log \pi_{\theta}(h_k^{(i)} \mid \mathbf{I}, \mathbf{q}, H_{<k}^{(i)}) - \log \pi_{\theta}(h_k^{(i)} \mid \mathbf{I}_{\text{white}}, \mathbf{q}, H_{<k}^{(i)}).
$$
This is a counterfactual log-likelihood ratio comparing the probability of a reasoning step under the real image and a white image. High values indicate that the step is visually grounded; values near zero indicate largely textual reasoning. The paper explicitly states that this is “exactly the kind of quantity many people refer to as a Visual Reliance Score.”

PDCR uses these step scores for unsupervised skill decomposition. VDS values are clustered with Otsu’s method into visual and textual steps, and the process reward is normalized within each cluster rather than globally. The motivation is that visual perception steps are sparse but crucial, whereas textual reasoning steps are dense, and global normalization creates mixture-induced signal degradation. Against GPT-labeled step annotations, Otsu thresholding on VDS reaches about $76.2\%$ decomposition accuracy, while Top-K heuristics peak around $67.5\%$.

Related work generalizes the same idea to other internal signals. "Look Again, Think Slowly: Enhancing Visual Reflection in Vision-Language Models" [2509.12132] tracks visual attention weight to visual tokens and a Visual Dependency Measure based on Hellinger distance between next-token distributions with and without visual tokens. "Grounding the Score: Explicit Visual Premise Verification for Reliable Vision-Language Process Reward Models" [2603.16253] introduces a scalar visual reliability signal by matching step-wise visual checklists against extracted numeric, relation, and structure constraints, then aggregating the match scores with a geometric mean. These are not all named VRS in the original papers, but they instantiate the same underlying question at finer granularity: whether a particular reasoning step, token, or premise is visually supported.

This suggests a conceptual division between **global VRS** and what may be called, as an editor’s term, **local VRS**: the former assesses answer-level dependence across a dataset, while the latter assesses step-level dependence within a trajectory.

## 5. Reliability, attention, and common misconceptions

A recurring misconception is that high benchmark accuracy implies high visual reliance. The medical VQA counterfactual study directly contradicts this: RLVR can improve accuracy while degrading VRS, Image Sensitivity, or both [2603.03437]. A second misconception is that visually focused attention maps are themselves reliable indicators of grounded behavior. "Visuals Lie, Consistency Speaks: Disentangling Spatial Attention from Reliability in Vision-Language Models" [2606.17389] systematically challenges that assumption.

That paper introduces structural-attention metrics—cluster count $C_k$ and spatial entropy $H_s$—and reports near-zero correlation with correctness. On a pooled structural-analysis set, $R(C_k, y) = 0.001$ with 95% confidence interval $[-0.034, 0.036]$, and $R(H_s, y) = -0.012$ with 95% confidence interval $[-0.047, 0.024]$. By contrast, self-consistency across sampled reasoning paths is the dominant predictor of truth, with $R = 0.429$, and precision reaches $90.8\%$ when self-consistency equals $1$.

The paper explains this through “Symbolic Detachment”: models may sharply focus visual attention early, then diffuse or decouple that attention from the final generative process. In LLaVA, for example, attention sharpens very early, visual tokens show little residual updating through much of the middle stack, and reliability becomes concentrated in a fragile late-stage bottleneck. PaliGemma and Qwen2-VL display more distributed or cyclical integration. The implication is that a static heatmap is neither a calibrated reliability signal nor a sufficient VRS proxy.

A third misconception is terminological: “VRS” always means “Visual Reliance Score.” In AVIS, it means Visual Reasoning Scaling, the test-time choice of rollout count $K$, and the paper explicitly states that it is not a scalar diagnostic score [2606.11576]. Inference-time search depth and visual reliance are therefore orthogonal notions, even though both concern multimodal reasoning.

## 6. Computation, interpretation, and practical use

The procedure for computing VRS depends entirely on which formulation is intended. In the counterfactual grounding formulation, the required ingredients are real-image and shuffled-image accuracies over the same dataset, and the score is their difference. In the V-FAT robustness formulation, one must additionally define a trap answer $y_{trap}$ for each instance, compute both mean accuracy and mean trap-hit rate, convert the latter into resistance, and take the harmonic mean. In step-level formulations, the calculation shifts from answer correctness to model internals: one segments a trajectory into steps, computes counterfactual likelihood differences, attention-based dependency signals, or checklist-constraint consistency, and then aggregates or clusters those values [2605.13467; 2601.04897; 2603.16253].

Interpretation is likewise formulation-dependent. For the accuracy-gap version, positive values indicate benefit from correct image/question pairing, values near zero indicate weak dependence on image identity, and negative values indicate reverse dependence. For the robustness version, high values require both correctness and resistance to trap answers. For step-level scores, high values indicate that a specific reasoning step or premise is visually grounded rather than text-derived.

Across the literature, the most stable practical recommendation is not to use VRS in isolation. The medical VQA framework evaluates VRS together with Blank Drop, Image Sensitivity, Visual Benefit/Harm Rates, and HVRR. Reflection-V supplements attention statistics with a dependency measure based on removing visual tokens. The reliability probe study favors self-consistency and hidden-state probes over structural attention. This suggests that VRS is best treated as one component of a multimodal grounding audit rather than as a complete account of visual faithfulness.

Taken together, the recent literature presents VRS as a family of operational diagnostics for causal visual use. Its forms differ—counterfactual accuracy gap, trap-resistant robustness, step-level dependence, premise reliability—but the common scientific aim is the same: to distinguish genuine image-conditioned reasoning from performance supported mainly by textual shortcuts, prior correlations, or internally self-consistent but weakly grounded generation.

Source: https://www.emergentmind.com/topics/visual-reliance-score-vrs