---
title: 'EVA: Extracting Visual Facts in Decoding'
url: https://www.emergentmind.com/topics/decoding-by-extracting-visual-facts-eva
type: topic
---

# EVA: Extracting Visual Facts in Decoding

Decoding by Extracting Visual Facts (EVA) is a training-free, decoding-time method for reducing object hallucination in multimodal large language models (MLLMs). It is designed for settings in which a model’s deeper layers drift toward strong language priors and suppress image-grounded evidence, even though some intermediate layers still retain useful visual factual knowledge. EVA addresses that failure mode by contrasting the model’s behavior under the original multimodal input with its behavior under a pure-text input, identifying the intermediate layer where visual influence is strongest, extracting a visual-fact signal from that layer, and injecting the result into the final decoding logits before token selection [2507.15652].

## 1. Problem setting and core observation

EVA is motivated by **object hallucination**, a failure mode in which an MLLM generates objects or attributes that are not present in the image, or omits real objects while producing text that remains locally plausible. The paper frames this as a consequence of strong textual priors inherited from the autoregressive language model: deeper layers increasingly favor linguistically likely continuations, while image-grounded evidence is attenuated [2507.15652].

The method is built on an **early-exit view** of the MLLM. For each transformer layer, the hidden state is projected through the LM head to obtain a next-token distribution. This is done in two conditions: the original multimodal input, and a **pure-text input** in which the visual tokens are removed. The pure-text run is treated as an estimate of the model’s prior, because without visual input the model can only rely on textual context and pretrained language statistics. The paper reports a central empirical observation: the **layer-wise trend of Jensen–Shannon divergence (JSD)** between the multimodal and pure-text distributions evolves similarly to the **layer-wise probability trend of non-hallucinated or visually correct tokens**. This is taken as evidence that intermediate layers with large multimodal-versus-prior discrepancy are precisely the layers in which visual information remains most active [2507.15652].

This defines what EVA means by **visual factual knowledge**. It is not symbolic knowledge, not a list of explicit visual propositions, and not a human-interpretable scene graph. Rather, it is the part of a token-level logit vector that remains after subtracting a text-only prior from a multimodal intermediate-layer prediction. A common misconception is that EVA “extracts facts” in an explicit knowledge-representation sense; the paper instead operationalizes visual facts as **image-grounded token preference increments** preserved in selected intermediate layers [2507.15652].

## 2. Decoding formulation

The paper defines visual tokens as
$$
\mathbf{X}^V = \{x_{v1}, x_{v2}, \ldots, x_{vP}\},
$$
text tokens as
$$
\mathbf{X}^C = \{x_{c1}, x_{c2}, \ldots, x_{cQ}\},
$$
and the multimodal input as
$$
\mathbf{X} = \mathrm{concat}(\mathbf{X}^V, \mathbf{X}^C).
$$
If the MLLM has \(N\) transformer layers and hidden states \(\mathbf{h}^i = \{h_0^i, h_1^i, \ldots, h_{T-1}^i\}\), then vanilla next-token decoding is
$$
p(x_T \mid x_{<T}) = \mathrm{softmax}\!\left(\phi(h_{T-1}^N)\right)_{x_T}, \quad x_T \in \mathcal{V}.
$$
Here \(\phi(\cdot)\) is the LM head and \(\mathcal{V}\) is the vocabulary [2507.15652].

EVA alters this procedure at each decoding step. First, it runs the model on the original multimodal prompt and obtains early-exit distributions
$$
p^j(x_T \mid x_{<T}) = \mathrm{softmax}(\phi(h_{T-1}^j))
$$
for candidate intermediate layers \(j \in \mathcal{J}\). Second, it constructs the same prompt **without visual tokens**, runs the model again, and obtains
$$
p_{\text{prior}}^j(x_T \mid x_{<T}) = \mathrm{softmax}(\phi(\hat h_{T-1}^j)).
$$
The paper restricts the comparison to a candidate token set \(\mathcal{V}_{\text{candidate}}\) using the truncation strategy from DeCo, with top-\(p\) truncation as default, while noting that the notation in the paper is awkwardly typeset and that no further implementation detail is provided [2507.15652].

For each candidate layer, EVA computes the original-versus-prior discrepancy
$$
d\!\left(p^j(x_T\mid x_{<T}), p_{\text{prior}}^j(x_T\mid x_{<T})\right)
=
\mathrm{JSD}\!\left(
p^j(x_T\mid x_{<T}) \,\|\, p_{\text{prior}}^j(x_T\mid x_{<T})
\right),
\quad x_T \in \mathcal{V}_{\text{candidate}}.
$$
The selected intermediate layer is
$$
M = \underset{j \in \mathcal{J}}{\arg\max}
\left\{
\mathrm{JSD}\!\left(
p^j(x_T\mid x_{<T}) \,\|\, p_{\text{prior}}^j(x_T\mid x_{<T})
\right)
\right\}.
$$
The method then computes two modulation coefficients:
$$
\text{max\_prob} = \max\!\left(\mathrm{softmax}(\phi(h_{T-1}^M))\right),
$$
$$
\text{max\_JSD} =
\mathrm{JSD}\!\left(
p^M(x_T\mid x_{<T}) \,\|\, p_{\text{prior}}^M(x_T\mid x_{<T})
\right).
$$

The extracted visual factual knowledge is defined in logit space as
$$
\mathrm{logits}_v = \phi(h_{T-1}^M) - \phi(\hat h_{T-1}^M).
$$
This subtraction removes token preferences shared by multimodal and text-only runs and retains the residual support attributable to the image. EVA then corrects the final-layer logits by adding a scaled intermediate-layer term:
$$
\mathrm{logits} =
\phi(h_{T-1}^N)
+
\alpha \times \text{max\_prob}
\times
\left(
\phi(h_{T-1}^M)
+
\text{max\_JSD} \times \mathrm{logits}_v
\right).
$$
Decoding uses
$$
\tilde p(x_T \mid x_{<T}) = \mathrm{softmax}(\mathrm{logits}).
$$
The corrected distribution can then be used with greedy search, beam search, or nucleus sampling [2507.15652].

## 3. Operating regime and implementation assumptions

EVA is **model-agnostic** in the limited sense defined by the paper: it assumes a standard MLLM stack in which visual tokens are fused with text tokens, intermediate hidden states are accessible, and the same LM head can project both intermediate and final hidden states into vocabulary logits. The reported experiments apply EVA to **InstructBLIP**, **MiniGPT-4**, **LLaVA-1.5**, and **Qwen-VL**, all in 7B-scale settings [2507.15652].

The method is explicitly not a black-box API technique. It requires access to intermediate hidden states \(h_{T-1}^j\), the LM head \(\phi(\cdot)\), the final-layer hidden state \(h_{T-1}^N\), and the ability to run the model both on the multimodal input and on the pure-text input. The paper therefore characterizes EVA as broadly applicable across architectures, but only when the inference stack is sufficiently open [2507.15652].

For a **7B, 32-layer decoder-only architecture**, the paper uses candidate layers **20–28**, following DeCo. The principal hyperparameter is the correction strength \(\alpha\), tuned in the range **0.6 to 5** for image captioning and VQA. EVA does not change model weights, does not require fine-tuning, and does not require external supervision. It is therefore training-free. It is not, however, compute-free. Each decoding step adds a second forward pass on the pure-text input, intermediate-layer extraction for candidate layers, and JSD computation across those layers. The paper reports that all inference experiments were run on a single **L20 GPU**, but does not provide explicit latency or FLOP figures [2507.15652].

A plausible implication is that EVA is best suited to white-box or semi-white-box deployment settings in which inference-time overhead is acceptable in exchange for improved grounding. The paper itself does not present a systems study beyond describing the extra pure-text pass and layer-level access requirements.

## 4. Empirical performance and ablations

EVA is evaluated on **POPE**, **MME**, and **CHAIR**. POPE is treated as a VQA-style object hallucination benchmark with average **F1** across random, popular, and adversarial splits. MME is evaluated on a hallucination subset covering object existence, count, position, and color, and also on the full set. CHAIR is used for image-caption hallucination on 500 MSCOCO 2014 validation images with the prompt “Please help me describe the image in detail,” reporting \(\text{CHAIR}_I\) and \(\text{CHAIR}_S\), both lower-is-better [2507.15652].

Selected quantitative results are summarized below.

| Benchmark and setting | Baseline | EVA |
|---|---:|---:|
| POPE, MiniGPT-4, nucleus sampling F1 | 52.8 | 74.7 |
| MME hallucination subset, LLaVA-1.5 total score | 565.33 | 651.67 |
| CHAIR, Qwen-VL, greedy, \(\text{CHAIR}_S\) | 46.0 | 6.8 |

On POPE, the paper highlights several gains over vanilla decoding: **MiniGPT-4, nucleus sampling** improves from **52.8** to **74.7**; **MiniGPT-4, greedy** improves from **58.5** to **78.2**; **LLaVA-1.5, nucleus** improves from **83.1** to **87.4**; and **InstructBLIP, greedy** improves from **80.0** to **85.2**. On the MME hallucination subset with nucleus sampling, EVA improves **LLaVA-1.5** from **565.33** to **651.67**, **MiniGPT-4** from **238.33** to **321.67**, **Qwen-VL** from **587.33** to **610.00**, and **InstructBLIP** from **380.33** to **415.00**, although **VCD** attains a higher total on InstructBLIP at **447.67**. On CHAIR under greedy decoding, the paper reports **InstructBLIP** \(\text{CHAIR}_S\) **58.8 \to 41.2** and \(\text{CHAIR}_I\) **23.7 \to 13.0**, **MiniGPT-4** \(\text{CHAIR}_S\) **31.8 \to 26.0**, **LLaVA-1.5** \(\text{CHAIR}_S\) **45.0 \to 35.6**, and **Qwen-VL** \(\text{CHAIR}_S\) **46.0 \to 6.8**. The paper cautions that Qwen-VL may generate shorter sentences, making its CHAIR values unusually low [2507.15652].

The paper also reports a clear trade-off on the **full MME evaluation**: EVA substantially improves perception-based tasks, but this may be accompanied by a reduction in some original recognition capabilities. The authors attribute this to the subtractive treatment of priors: weakening language priors can improve grounding while slightly harming some general recognition behavior [2507.15652].

Ablation results support the decoding design. In a comparison against DeCo on **500 MSCOCO samples** with **LLaVA-1.5-7B**, the number of examples containing at least one activated ground-truth token and the total number of activated ground-truth tokens are **157 / 259** for DeCo and **170 / 302** for EVA. The appendix also evaluates removal of the two modulation factors. On POPE greedy decoding for **MiniGPT-4**, EVA achieves **78.2**, while removing **max\_JSD** yields **77.8**, removing **max\_prob** yields **76.9**, and removing both yields **76.6**. For **Qwen-VL**, the corresponding values are **86.3**, **86.1**, **84.6**, and **84.5**. This suggests that both modulation terms contribute, with **max\_prob** appearing more important than **max\_JSD** in those ablations [2507.15652].

## 5. Position within hallucination-mitigation research

EVA is presented against several decoding baselines: **Vanilla decoding**, **DoLa**, **VCD**, **OPERA**, and **DeCo**. Its distinctive feature is not merely layer contrast, nor merely visual perturbation, nor a generic decoding penalty. The defining move is the explicit contrast between the **original multimodal input** and a **pure-text prior input** at intermediate layers, followed by **dynamic selection of the layer with maximal JSD** and logit correction using the extracted residual visual signal [2507.15652].

This places EVA in a specific methodological niche. **DoLa** uses layer contrast to improve factuality in language models, but is not framed around multimodal prior-versus-visual separation. **VCD** contrasts original and perturbed visual inputs. **OPERA** penalizes over-trusted tokens and adjusts attention allocation. **DeCo** selects preceding layers and uses them to correct final logits, but the EVA paper argues that DeCo may select layers on the basis of high probabilities that are already contaminated by prior interference. EVA’s intervention point is therefore more targeted: it attempts to isolate the token-level component that is due to visual evidence rather than language continuation bias [2507.15652].

A second misconception concerns interpretability. EVA does not recover explicit visual entities, regions, or symbolic facts. The extracted object
$$
\mathrm{logits}_v = \phi(h_{T-1}^M) - \phi(\hat h_{T-1}^M)
$$
is a vocabulary-scale logit difference. Its semantics are indirect: it encodes which next-token preferences are strengthened by the image relative to a text-only prior. The method is therefore best understood as **prior subtraction plus intermediate-layer visual signal reinjection**, not as an explicit visual fact extractor in the knowledge-graph or scene-graph sense [2507.15652].

## 6. Limitations, trade-offs, and nomenclature

The paper identifies several practical limitations. EVA requires white-box access to hidden states and logits, so it is unsuitable for closed APIs that expose only final token probabilities or text outputs. It adds inference overhead through an additional pure-text forward pass and candidate-layer JSD computations. The sensitivity analysis is limited: beyond candidate layers **20–28** for 32-layer 7B models, \(\alpha\) in the range **0.6 to 5**, and the modulation-term ablation, the paper does not provide a broader sweep. It also does not deeply catalog failure cases. The clearest reported trade-off is that subtracting priors can improve grounding while slightly reducing some general recognition capability on the full MME evaluation [2507.15652].

The acronym **EVA** is also ambiguous in the arXiv literature. It refers to a robust audiovisual ASR framework based on mixture-of-experts in “Robust Audiovisual Speech Recognition Models with Mixture-of-Experts” [2409.12370], to **Entity Visual Alignment** for knowledge-graph entity alignment in “Visual Pivoting for (Unsupervised) Entity Alignment” [2009.13603], to **EVent Asynchronous representation learning** in event-based vision [2505.11165], and to **LatEnt Visual StAtes** for internal latent visual reasoning in “Latent Visual States for Efficient Multimodal Reasoning” [2606.24233]. A conceptually related but distinct use appears in “EVA-Net: Subject-Independent EEG Motor Decoding with Video-Derived Motor Priors,” where action-video semantics are used as a training-time semantic anchor for EEG decoding [2606.01884]. In the hallucination-mitigation paper, however, EVA specifically denotes **Decoding by Extracting Visual Facts** [2507.15652].

Within that specific meaning, EVA’s contribution is narrow but technically precise. It proposes that the most visually informative signal for next-token correction is often neither the final layer nor a fixed earlier layer, but the intermediate layer at which multimodal behavior diverges most strongly from a pure-text prior. By extracting that residual and reinserting it into the final decoding distribution, EVA provides a model-agnostic, training-free mechanism for steering generation toward visual fidelity rather than prior-driven hallucination [2507.15652].

Source: https://www.emergentmind.com/topics/decoding-by-extracting-visual-facts-eva