EVA: Extracting Visual Facts in Decoding

• The paper introduces EVA, a method that mitigates object hallucination by contrasting multimodal and pure-text inputs to extract visual factual signals.
• It dynamically selects the intermediate layer where visual influence is strongest based on Jensen–Shannon divergence to refine final token predictions.
• Empirical results across benchmarks show that EVA improves grounding fidelity in models like MiniGPT-4 and Qwen-VL, despite adding extra inference overhead.

Decoding by Extracting Visual Facts (EVA) is a training-free, decoding-time method for reducing object hallucination in multimodal LLMs (MLLMs). It is designed for settings in which a model’s deeper layers drift toward strong language priors and suppress image-grounded evidence, even though some intermediate layers still retain useful visual factual knowledge. EVA addresses that failure mode by contrasting the model’s behavior under the original multimodal input with its behavior under a pure-text input, identifying the intermediate layer where visual influence is strongest, extracting a visual-fact signal from that layer, and injecting the result into the final decoding logits before token selection (Zhou et al., 21 Jul 2025).

1. Problem setting and core observation

EVA is motivated by object hallucination, a failure mode in which an MLLM generates objects or attributes that are not present in the image, or omits real objects while producing text that remains locally plausible. The paper frames this as a consequence of strong textual priors inherited from the autoregressive LLM: deeper layers increasingly favor linguistically likely continuations, while image-grounded evidence is attenuated (Zhou et al., 21 Jul 2025).

The method is built on an early-exit view of the MLLM. For each transformer layer, the hidden state is projected through the LM head to obtain a next-token distribution. This is done in two conditions: the original multimodal input, and a pure-text input in which the visual tokens are removed. The pure-text run is treated as an estimate of the model’s prior, because without visual input the model can only rely on textual context and pretrained language statistics. The paper reports a central empirical observation: the layer-wise trend of Jensen–Shannon divergence (JSD) between the multimodal and pure-text distributions evolves similarly to the layer-wise probability trend of non-hallucinated or visually correct tokens. This is taken as evidence that intermediate layers with large multimodal-versus-prior discrepancy are precisely the layers in which visual information remains most active (Zhou et al., 21 Jul 2025).

This defines what EVA means by visual factual knowledge. It is not symbolic knowledge, not a list of explicit visual propositions, and not a human-interpretable scene graph. Rather, it is the part of a token-level logit vector that remains after subtracting a text-only prior from a multimodal intermediate-layer prediction. A common misconception is that EVA “extracts facts” in an explicit knowledge-representation sense; the paper instead operationalizes visual facts as image-grounded token preference increments preserved in selected intermediate layers (Zhou et al., 21 Jul 2025).

2. Decoding formulation

The paper defines visual tokens as

$\mathbf{X}^V = \{x_{v1}, x_{v2}, \ldots, x_{vP}\},$

text tokens as

$\mathbf{X}^C = \{x_{c1}, x_{c2}, \ldots, x_{cQ}\},$

and the multimodal input as

$$\mathbf{X} = \mathrm{concat}(\mathbf{X}^V, \mathbf{X}^C).$$

If the MLLM has $N$ transformer layers and hidden states $\mathbf{h}^i = \{h_0^i, h_1^i, \ldots, h_{T-1}^i\}$, then vanilla next-token decoding is

$$p(x_T \mid x_{<T}) = \mathrm{softmax}\!\left(\phi(h_{T-1}^N)\right)_{x_T}, \quad x_T \in \mathcal{V}.$$

Here $\phi(\cdot)$ is the LM head and $\mathcal{V}$ is the vocabulary (Zhou et al., 21 Jul 2025).

EVA alters this procedure at each decoding step. First, it runs the model on the original multimodal prompt and obtains early-exit distributions

$$p^j(x_T \mid x_{<T}) = \mathrm{softmax}(\phi(h_{T-1}^j))$$

for candidate intermediate layers $j \in \mathcal{J}$. Second, it constructs the same prompt without visual tokens, runs the model again, and obtains

$\mathbf{X}^C = \{x_{c1}, x_{c2}, \ldots, x_{cQ}\},$0

The paper restricts the comparison to a candidate token set $\mathbf{X}^C = \{x_{c1}, x_{c2}, \ldots, x_{cQ}\},$1 using the truncation strategy from DeCo, with top-$\mathbf{X}^C = \{x_{c1}, x_{c2}, \ldots, x_{cQ}\},$2 truncation as default, while noting that the notation in the paper is awkwardly typeset and that no further implementation detail is provided (Zhou et al., 21 Jul 2025).

For each candidate layer, EVA computes the original-versus-prior discrepancy

$\mathbf{X}^C = \{x_{c1}, x_{c2}, \ldots, x_{cQ}\},$3

The selected intermediate layer is

$\mathbf{X}^C = \{x_{c1}, x_{c2}, \ldots, x_{cQ}\},$4

The method then computes two modulation coefficients:

$\mathbf{X}^C = \{x_{c1}, x_{c2}, \ldots, x_{cQ}\},$5

$\mathbf{X}^C = \{x_{c1}, x_{c2}, \ldots, x_{cQ}\},$6

The extracted visual factual knowledge is defined in logit space as

$\mathbf{X}^C = \{x_{c1}, x_{c2}, \ldots, x_{cQ}\},$7

This subtraction removes token preferences shared by multimodal and text-only runs and retains the residual support attributable to the image. EVA then corrects the final-layer logits by adding a scaled intermediate-layer term:

$\mathbf{X}^C = \{x_{c1}, x_{c2}, \ldots, x_{cQ}\},$8

Decoding uses

$\mathbf{X}^C = \{x_{c1}, x_{c2}, \ldots, x_{cQ}\},$9

The corrected distribution can then be used with greedy search, beam search, or nucleus sampling (Zhou et al., 21 Jul 2025).

3. Operating regime and implementation assumptions

EVA is model-agnostic in the limited sense defined by the paper: it assumes a standard MLLM stack in which visual tokens are fused with text tokens, intermediate hidden states are accessible, and the same LM head can project both intermediate and final hidden states into vocabulary logits. The reported experiments apply EVA to InstructBLIP, MiniGPT-4, LLaVA-1.5, and Qwen-VL, all in 7B-scale settings (Zhou et al., 21 Jul 2025).

The method is explicitly not a black-box API technique. It requires access to intermediate hidden states $$\mathbf{X} = \mathrm{concat}(\mathbf{X}^V, \mathbf{X}^C).$$0, the LM head $$\mathbf{X} = \mathrm{concat}(\mathbf{X}^V, \mathbf{X}^C).$$1, the final-layer hidden state $$\mathbf{X} = \mathrm{concat}(\mathbf{X}^V, \mathbf{X}^C).$$2, and the ability to run the model both on the multimodal input and on the pure-text input. The paper therefore characterizes EVA as broadly applicable across architectures, but only when the inference stack is sufficiently open (Zhou et al., 21 Jul 2025).

For a 7B, 32-layer decoder-only architecture, the paper uses candidate layers 20–28, following DeCo. The principal hyperparameter is the correction strength $$\mathbf{X} = \mathrm{concat}(\mathbf{X}^V, \mathbf{X}^C).$$3, tuned in the range 0.6 to 5 for image captioning and VQA. EVA does not change model weights, does not require fine-tuning, and does not require external supervision. It is therefore training-free. It is not, however, compute-free. Each decoding step adds a second forward pass on the pure-text input, intermediate-layer extraction for candidate layers, and JSD computation across those layers. The paper reports that all inference experiments were run on a single L20 GPU, but does not provide explicit latency or FLOP figures (Zhou et al., 21 Jul 2025).

A plausible implication is that EVA is best suited to white-box or semi-white-box deployment settings in which inference-time overhead is acceptable in exchange for improved grounding. The paper itself does not present a systems study beyond describing the extra pure-text pass and layer-level access requirements.

4. Empirical performance and ablations

EVA is evaluated on POPE, MME, and CHAIR. POPE is treated as a VQA-style object hallucination benchmark with average F1 across random, popular, and adversarial splits. MME is evaluated on a hallucination subset covering object existence, count, position, and color, and also on the full set. CHAIR is used for image-caption hallucination on 500 MSCOCO 2014 validation images with the prompt “Please help me describe the image in detail,” reporting $$\mathbf{X} = \mathrm{concat}(\mathbf{X}^V, \mathbf{X}^C).$$4 and $$\mathbf{X} = \mathrm{concat}(\mathbf{X}^V, \mathbf{X}^C).$$5, both lower-is-better (Zhou et al., 21 Jul 2025).

Selected quantitative results are summarized below.

Benchmark and setting	Baseline	EVA
POPE, MiniGPT-4, nucleus sampling F1	52.8	74.7
MME hallucination subset, LLaVA-1.5 total score	565.33	651.67
CHAIR, Qwen-VL, greedy, $$\mathbf{X} = \mathrm{concat}(\mathbf{X}^V, \mathbf{X}^C).$$6	46.0	6.8

On POPE, the paper highlights several gains over vanilla decoding: MiniGPT-4, nucleus sampling improves from 52.8 to 74.7; MiniGPT-4, greedy improves from 58.5 to 78.2; LLaVA-1.5, nucleus improves from 83.1 to 87.4; and InstructBLIP, greedy improves from 80.0 to 85.2. On the MME hallucination subset with nucleus sampling, EVA improves LLaVA-1.5 from 565.33 to 651.67, MiniGPT-4 from 238.33 to 321.67, Qwen-VL from 587.33 to 610.00, and InstructBLIP from 380.33 to 415.00, although VCD attains a higher total on InstructBLIP at 447.67. On CHAIR under greedy decoding, the paper reports InstructBLIP $$\mathbf{X} = \mathrm{concat}(\mathbf{X}^V, \mathbf{X}^C).$$7 58.8 \to 41.2 and $$\mathbf{X} = \mathrm{concat}(\mathbf{X}^V, \mathbf{X}^C).$$8 23.7 \to 13.0, MiniGPT-4 $$\mathbf{X} = \mathrm{concat}(\mathbf{X}^V, \mathbf{X}^C).$$9 31.8 \to 26.0, LLaVA-1.5 $N$0 45.0 \to 35.6, and Qwen-VL $N$1 46.0 \to 6.8. The paper cautions that Qwen-VL may generate shorter sentences, making its CHAIR values unusually low (Zhou et al., 21 Jul 2025).

The paper also reports a clear trade-off on the full MME evaluation: EVA substantially improves perception-based tasks, but this may be accompanied by a reduction in some original recognition capabilities. The authors attribute this to the subtractive treatment of priors: weakening language priors can improve grounding while slightly harming some general recognition behavior (Zhou et al., 21 Jul 2025).

Ablation results support the decoding design. In a comparison against DeCo on 500 MSCOCO samples with LLaVA-1.5-7B, the number of examples containing at least one activated ground-truth token and the total number of activated ground-truth tokens are 157 / 259 for DeCo and 170 / 302 for EVA. The appendix also evaluates removal of the two modulation factors. On POPE greedy decoding for MiniGPT-4, EVA achieves 78.2, while removing max_JSD yields 77.8, removing max_prob yields 76.9, and removing both yields 76.6. For Qwen-VL, the corresponding values are 86.3, 86.1, 84.6, and 84.5. This suggests that both modulation terms contribute, with max_prob appearing more important than max_JSD in those ablations (Zhou et al., 21 Jul 2025).

5. Position within hallucination-mitigation research

EVA is presented against several decoding baselines: Vanilla decoding, DoLa, VCD, OPERA, and DeCo. Its distinctive feature is not merely layer contrast, nor merely visual perturbation, nor a generic decoding penalty. The defining move is the explicit contrast between the original multimodal input and a pure-text prior input at intermediate layers, followed by dynamic selection of the layer with maximal JSD and logit correction using the extracted residual visual signal (Zhou et al., 21 Jul 2025).

This places EVA in a specific methodological niche. DoLa uses layer contrast to improve factuality in LLMs, but is not framed around multimodal prior-versus-visual separation. VCD contrasts original and perturbed visual inputs. OPERA penalizes over-trusted tokens and adjusts attention allocation. DeCo selects preceding layers and uses them to correct final logits, but the EVA paper argues that DeCo may select layers on the basis of high probabilities that are already contaminated by prior interference. EVA’s intervention point is therefore more targeted: it attempts to isolate the token-level component that is due to visual evidence rather than language continuation bias (Zhou et al., 21 Jul 2025).

A second misconception concerns interpretability. EVA does not recover explicit visual entities, regions, or symbolic facts. The extracted object

$N$2

is a vocabulary-scale logit difference. Its semantics are indirect: it encodes which next-token preferences are strengthened by the image relative to a text-only prior. The method is therefore best understood as prior subtraction plus intermediate-layer visual signal reinjection, not as an explicit visual fact extractor in the knowledge-graph or scene-graph sense (Zhou et al., 21 Jul 2025).

6. Limitations, trade-offs, and nomenclature

The paper identifies several practical limitations. EVA requires white-box access to hidden states and logits, so it is unsuitable for closed APIs that expose only final token probabilities or text outputs. It adds inference overhead through an additional pure-text forward pass and candidate-layer JSD computations. The sensitivity analysis is limited: beyond candidate layers 20–28 for 32-layer 7B models, $N$3 in the range 0.6 to 5, and the modulation-term ablation, the paper does not provide a broader sweep. It also does not deeply catalog failure cases. The clearest reported trade-off is that subtracting priors can improve grounding while slightly reducing some general recognition capability on the full MME evaluation (Zhou et al., 21 Jul 2025).

The acronym EVA is also ambiguous in the arXiv literature. It refers to a robust audiovisual ASR framework based on mixture-of-experts in “Robust Audiovisual Speech Recognition Models with Mixture-of-Experts” (Wu et al., 2024), to Entity Visual Alignment for knowledge-graph entity alignment in “Visual Pivoting for (Unsupervised) Entity Alignment” (Liu et al., 2020), to EVent Asynchronous representation learning in event-based vision (Hao et al., 16 May 2025), and to LatEnt Visual StAtes for internal latent visual reasoning in “Latent Visual States for Efficient Multimodal Reasoning” (Chen et al., 23 Jun 2026). A conceptually related but distinct use appears in “EVA-Net: Subject-Independent EEG Motor Decoding with Video-Derived Motor Priors,” where action-video semantics are used as a training-time semantic anchor for EEG decoding (Li et al., 1 Jun 2026). In the hallucination-mitigation paper, however, EVA specifically denotes Decoding by Extracting Visual Facts (Zhou et al., 21 Jul 2025).

Within that specific meaning, EVA’s contribution is narrow but technically precise. It proposes that the most visually informative signal for next-token correction is often neither the final layer nor a fixed earlier layer, but the intermediate layer at which multimodal behavior diverges most strongly from a pure-text prior. By extracting that residual and reinserting it into the final decoding distribution, EVA provides a model-agnostic, training-free mechanism for steering generation toward visual fidelity rather than prior-driven hallucination (Zhou et al., 21 Jul 2025).