Visual-Factual Reliance Index Overview

• Visual-Factual Reliance Index (VFRI) is a metric that quantifies the balance between visual evidence and pre-trained factual knowledge in model decision-making.
• It employs counterfactual interventions and capability normalization to distinctly measure visualization fidelity versus factual alignment across different tasks.
• VFRI provides actionable insights by diagnosing model tendencies—whether favoring visual override or relying on textual shortcuts—using rigorous quantitative frameworks.

Visual–Factual Reliance Index (VFRI) is a metric for arbitration between visual evidence and non-visual information. In the chart-literacy setting, it is explicitly defined as “the paper’s summary index for this arbitration”: it “quantitatively describes how much a model, given its underlying capabilities, tends to rely on visual vs. factual information in counterfactual scenarios” (Lee et al., 2 Jun 2026). Closely related work uses the same term or near-equivalent constructs to ask whether answers “causally depend on those visual objects that the dataset marks as factually required,” whether multimodal systems exhibit “causal visual dependence,” whether “visual-factual reliance” is “exactly the complement of text bias,” and whether figures function as central carriers of scientific information rather than decoration (Reich et al., 2023, Zafar et al., 3 Mar 2026, Wang et al., 8 Jan 2026, Lee et al., 2016). This suggests that VFRI is best understood as a family of task-specific indices for measuring reliance on visual evidence under agreement, conflict, ablation, or counterfactual intervention.

1. Conceptual scope and core distinction

The central distinction underlying VFRI is between visual evidence and factual priors. In the visualization-literacy formulation, these are “what is literally encoded in the chart” versus “what the model ‘knows’ from pre-training about the real world” (Lee et al., 2 Jun 2026). In multimodal VQA, the analogous distinction is between answers that depend on “question-relevant image regions” and answers driven by language priors or irrelevant context (Reich et al., 2023). In V-FAT, the same problem is framed as the tension between “visual perception and linguistic priors,” with visual-factual reliance described as the complement of text bias (Wang et al., 8 Jan 2026).

Across settings, VFRI is therefore not a generic accuracy statistic. It is an index of source preference or source dependence. A model may be correct because the chart agrees with world knowledge, because the image is genuinely necessary, because the question is shortcut-solvable from text alone, or because the model hallucinates visually worded reasoning while remaining image-invariant. VFRI is designed to separate these cases rather than collapse them into a single correctness number (Lee et al., 2 Jun 2026, Zafar et al., 3 Mar 2026).

This separation matters because several papers report that high benchmark accuracy is compatible with weak visual grounding. In the chart setting, high visualization accuracy can reflect factual recall rather than visual understanding; in medical VQA, text-only RLVR can match or outperform image-text RLVR; in text-bias diagnostics, models can answer correctly while following the “textual trap” rather than the image (Lee et al., 2 Jun 2026, Zafar et al., 3 Mar 2026, Wang et al., 8 Jan 2026).

2. Capability-normalized VFRI in counterfactual visualization literacy

The most explicit formalization of VFRI appears in the Counterfactual Visualization Literacy Assessment Test (CVLAT), where each item forces a choice between a visual-correct option and a factual-correct option (Lee et al., 2 Jun 2026). The construction adds two capability references: an anonymized visual baseline $V_{\text{anon}}$, which removes domain labels so that only chart-reading remains, and a Q-only condition $F_Q$, which removes the chart so that only factual prior availability remains.

Because the tasks are multiple-choice, the framework first applies correction-for-guessing. For item $i$, with target-response rate $S_i$, distractor-response rate $W_i$, number of options $C_i$, and number of distractors $D_i$, the corrected score is

$$\text{Score}_i = \max\left(0, S_i - \frac{W_i}{D_i}\right).$$

Using this corrected score, the framework defines Visualization Fidelity and Factual Alignment as capability-normalized quantities:

$$VF_i = \frac{V_{\text{CVLAT}, i}}{V_{\text{anon}, i} + \varepsilon}, \qquad FA_i = \frac{F_{\text{CVLAT}, i}}{F_{Q, i} + \varepsilon},$$

with $\varepsilon = 10^{-6}$. The item-level VFRI is then

$F_Q$0

and the model-level score is the item average,

$F_Q$1

This formulation has range $F_Q$2. Values near $F_Q$3 indicate a strongly visualization-oriented model; values near $F_Q$4 indicate a strongly factual-knowledge-oriented model; values near $F_Q$5 are explicitly treated with caution, because they can reflect either genuine balance or high false-response rates and confusion rather than principled arbitration (Lee et al., 2 Jun 2026).

The same framework also defines a quadrant analysis of responses: VC ∧ FC (source ambiguity), VI ∧ FI (model failure), VC ∧ FI (visual override), and VI ∧ FC (factual override). CVLAT is designed so that the model must reveal whether it chooses visual override or factual override under conflict, and VFRI summarizes that tendency after normalizing for whether the model can read the chart at all and whether it actually knows the fact (Lee et al., 2 Jun 2026).

3. Counterfactual and causal formulations in VQA and multimodal reasoning

A closely related causal formulation appears in visual question answering as Faithful and Plausible Visual Grounding. For question $F_Q$6, full object set $F_Q$7, relevant objects $F_Q$8, irrelevant objects $F_Q$9, and predicted answers $i$0, the per-sample indicator is

$i$1

The dataset-level score is

$i$2

This is explicitly proposed as a VFRI-style quantity: the fraction of samples for which the answer causally depends on the relevant visual objects and not on irrelevant ones (Reich et al., 2023).

In multimodal medical reasoning, the corresponding emphasis shifts from object subsets to counterfactual image conditions. The framework uses real, blank, and shuffled images, and defines the Visual Reliance Score

$i$3

the Blank Drop

$i$4

and Image Sensitivity

$i$5

It then adds Hallucinated Visual Reasoning Rate

$i$6

which counts cases where the rationale contains a novel visual claim but the answer is image-invariant. A proposed VFRI combines accuracy-based visual benefit, behavioral sensitivity, and groundedness of visual claims (Zafar et al., 3 Mar 2026).

V-FAT defines a different but closely related VRS for text-bias diagnostics. For level $i$7, with mean accuracy $i$8, mean textual dominance score $i$9, and resistance $S_i$0, the metric is

$S_i$1

This harmonic mean penalizes “lucky” linguistic guesses and rewards simultaneous correctness and resistance to textual traps. The three-level structure—internal corpus bias, external instruction bias, and synergistic bias—supports per-level or global VFRI definitions (Wang et al., 8 Jan 2026).

A third counterfactual family is the Tri-Layer Diagnostic Framework, which decomposes behavior into perception, dependency, and alignment. It defines Latent Anomaly Detection

$S_i$2

Visual Necessity Score

$S_i$3

and Competition Score

$S_i$4

These scores define four categories: Perceptual Blindness, Language Shortcut, Visual Sycophancy, and Robust Refusal. For VFRI, this matters because high visual dependency alone is not enough: high VNS combined with high CS indicates a model that sees and is visually sensitive, but still hallucinates to satisfy user expectations (Hong et al., 19 Mar 2026).

4. Extensions beyond end-task QA

In chart captioning, the visual-factual problem is cast as visual entailment. CHARTVE scores a sentence $S_i$5 against chart image $S_i$6 using the decoder logits for “yes” and “no”:

$S_i$7

For a multi-sentence caption $S_i$8, the caption-level score is

$S_i$9

Because a chart caption is factual iff all its sentences are factual, this minimum aggregation functions as a reference-free chart-level factuality index. The paper explicitly proposes that this score can serve as a VFRI-like quantity for chart captions (Huang et al., 2023).

A document-level precursor arises in viziometrics. The underlying paper does not define VFRI, but it provides the ingredients: a five-class figure taxonomy, automatic figure extraction and classification, page-normalized figure densities, proportions by type, field differences, and impact correlations (Lee et al., 2016). The basic primitive is figure density per page,

$W_i$0

where $W_i$1 is the count of figure type $W_i$2 in paper $W_i$3 and $W_i$4 is page count. On top of this, proposed document-level VFRI variants include total visual density $W_i$5, type-weighted density $W_i$6, portfolio-shape indices that combine densities and proportions, and field-normalized $W_i$7-scores. This suggests a VFRI for scientific communication rather than for model inference (Lee et al., 2016).

In image search with external knowledge, a related notion appears as the performance delta attributable to factual knowledge. One proposed index is the relative gain from adding knowledge to a visual-only system,

$W_i$8

or, using an oracle-knowledge ceiling,

$W_i$9

Here the issue is factual reliance on encyclopedic knowledge associated with named visual entities rather than direct arbitration against chart or image evidence (Gatti et al., 2022).

A further extension appears in automated detection of visual attribute reliance. A self-reflective agent generates hypotheses about an attribute, produces positive and negative prompts, scores the resulting images, and computes a predictiveness score

$C_i$0

together with a mean score separation

$C_i$1

A proposed VFRI then combines predictiveness and normalized score separation, optionally maximized over self-reflection rounds (Li et al., 24 Oct 2025).

5. Empirical findings and model typologies

The explicit chart-based VFRI produces a split between visualization-oriented and factual-knowledge-oriented LVLMs. The strongest visualization-oriented model reported is Gemini‑3.1‑Pro with VFRI $C_i$2, while a strongly factual-knowledge-oriented example is Grok‑4.20 with VFRI $C_i$3. A human baseline on the same counterfactual items yields positive VFRI for all 30 participants, with mean participant VFRI $C_i$4, factual-target corrected mean $C_i$5, and the interpretation that people overwhelmingly follow the chart under conflict (Lee et al., 2 Jun 2026).

The same chart study also shows that prompt-based control is direction-asymmetric and model-dependent. Claude‑Opus‑4.7 shifts from baseline VFRI $C_i$6 to $C_i$7 under a factual-priority prompt and to $C_i$8 under a visual-priority prompt, while GPT‑5.5 is V-priority-insensitive and Gemini‑3.1‑Pro is near ceiling under visual priority but strongly shifts under factual priority. High chart-reading capability therefore does not predict prompt controllability (Lee et al., 2 Jun 2026).

In medical VQA, the counterfactual results are sharper. Text-only RLVR on PathVQA achieves negative VRS $C_i$9, meaning performance is better with mismatched images than with the correct image. Across all benchmarks, image-text RLVR improves accuracy while reducing overall image sensitivity to $D_i$0, and models make novel visual claims in $D_i$1 of responses while $D_i$2 of all examples are counted as hallucinated visual reasoning by HVRR (Zafar et al., 3 Mar 2026).

In V-FAT, the three-level bias design shows “visual collapse under high linguistic dominance.” Strong models can still achieve high VRS under conflict, but model scale improves accuracy more than resistance to text bias, and inference-time “Thinking” can worsen visual reliance. This is why the paper treats VRS as a complement to plain accuracy rather than a replacement (Wang et al., 8 Jan 2026).

The tri-layer diagnostic study reports that 69.6% of all samples fall into Visual Sycophancy, 23.3% into Language Shortcut, 7.1% into Perceptual Blindness, and 0.0% into Robust Refusal. In the Qwen2.5‑VL scaling comparison, moving from 7B to 72B raises VNS from $D_i$3 to $D_i$4 and LAD from $D_i$5 to $D_i$6, reduces Language Shortcut from $D_i$7 to $D_i$8, but increases Visual Sycophancy from $D_i$9 to $$\text{Score}_i = \max\left(0, S_i - \frac{W_i}{D_i}\right).$$0. This indicates that larger models can become more visually engaged while also becoming more willing to hallucinate instead of refuse (Hong et al., 19 Mar 2026).

6. Interpretation, controversies, and methodological cautions

A recurring caution is that high accuracy is not sufficient evidence of faithful visual reasoning. The chart study states this directly; the medical VQA study shows that accuracy-only rewards enable shortcut exploitation; the tri-layer taxonomy shows that categories with very different grounding properties can have similar full-condition accuracy (Lee et al., 2 Jun 2026, Zafar et al., 3 Mar 2026, Hong et al., 19 Mar 2026).

A second caution concerns explanation signals. The “Attention-Confidence Assumption” is explicitly challenged: structural-attention metrics such as cluster count $$\text{Score}_i = \max\left(0, S_i - \frac{W_i}{D_i}\right).$$1 and spatial entropy $$\text{Score}_i = \max\left(0, S_i - \frac{W_i}{D_i}\right).$$2 have near-zero correlation with accuracy, with $$\text{Score}_i = \max\left(0, S_i - \frac{W_i}{D_i}\right).$$3 and $$\text{Score}_i = \max\left(0, S_i - \frac{W_i}{D_i}\right).$$4, while Self-Consistency is the dominant predictor of truth with $$\text{Score}_i = \max\left(0, S_i - \frac{W_i}{D_i}\right).$$5. This suggests that a VFRI based on visually plausible heatmaps alone is methodologically weak; generation-time dynamics and hidden-state probes are better reliability signals than spatial attention maps (Mann et al., 16 Jun 2026).

A third caution concerns annotation and preprocessing assumptions. FPVG depends on relevance annotations and detector–annotation alignment; CHARTVE depends on chart-specific entailment data and realistic negative generation; medical HVRR depends on rule-based detection of visual claims; document-level VFRI proposals depend on figure extraction, classification, and field normalization; self-reflective attribute-reliance detection depends on the controllability of text-to-image and editing tools (Reich et al., 2023, Huang et al., 2023, Zafar et al., 3 Mar 2026, Lee et al., 2016, Li et al., 24 Oct 2025).

A fourth caution is that VFRI should not be conflated with out-of-distribution prediction quality. VisFIS shows that Right-for-the-Right-Reason metrics are not predictive of out-of-distribution accuracy when controlling for a model’s in-distribution accuracy. This does not make VFRI uninformative; it locates its value differently. VFRI is a diagnostic of whether a system is right for the right visual reasons, not a guaranteed proxy for robustness or impact (Ying et al., 2022).

Taken together, these results position VFRI as an evaluation family for visual dependence, source arbitration, and truthful use of visual evidence. Its strongest formulations are counterfactual, capability-normalized, and explicit about the distinction between perception, dependency, and decision. Its weakest formulations are those that equate visual reliance with raw accuracy, visually plausible explanations, or tightly focused attention alone.