Look Twice: Training-Free Evidence Highlighting in Multimodal Large Language Models

Abstract: Answering questions about images often requires combining visual understanding with external knowledge. Multimodal LLMs (MLLMs) provide a natural framework for this setting, but they often struggle to identify the most relevant visual and textual evidence when answering knowledge-intensive queries. In such scenarios, models must integrate visual cues with retrieved textual evidence that is often noisy or only partially relevant, while also localizing fine-grained visual information in the image. In this work, we introduce Look Twice (LoT), a training-free inference-time framework that improves how pretrained MLLMs utilize multimodal evidence. Specifically, we exploit the model attention patterns to estimate which visual regions and retrieved textual elements are relevant to a query, and then generate the answer conditioned on this highlighted evidence. The selected cues are highlighted through lightweight prompt-level markers that encourage the model to re-attend to the relevant evidence during generation. Experiments across multiple knowledge-based VQA benchmarks show consistent improvements over zero-shot MLLMs. Additional evaluations on vision-centric and hallucination-oriented benchmarks further demonstrate that visual evidence highlighting alone improves model performance in settings without textual context, all without additional training or architectural modifications. Source code will be publicly released.

• The paper introduces Look Twice, a training-free two-pass method that analyzes attention to select one relevant image region and text sentence before generating an answer.
• Across 11 multimodal models and four KB-VQA benchmarks, Look Twice improves average performance for every backbone by 1.1–5.3 points while adding only about 5.6% generation cost.
• The method reduces retrieval noise and can improve hallucination detection, but remains sensitive to model-specific attention sinks and struggles with multiple or dispersed evidence regions.

Overview

"Look Twice: Training-Free Evidence Highlighting in Multimodal LLMs" (2604.01280) addresses a specific failure mode of Multimodal LLMs (MLLMs) in knowledge-based Visual Question Answering (KB-VQA): the inability to reliably identify which retrieved textual passages and which image regions actually support the answer. The authors, Morini et al. from the University of Modena and Reggio Emilia, propose Look Twice (LoT), an inference-time, training-free framework that treats the model's internal attention as an implicit relevance signal. A first, minimal generation step produces a single token whose attention patterns are analyzed to localize query-relevant evidence; a second generation pass then produces the final answer conditioned on this evidence, which is explicitly highlighted through lightweight prompt-level markers. No parameters, architectures, or retrieval pipelines are modified, and the overhead amounts to generating one additional token.

Method

LoT operates on the concatenated multimodal sequence of visual tokens, question tokens, and retrieved context, and extracts two families of attention signals during the first forward pass.

Visual evidence selection is object-conditioned. Using spaCy dependency parsing to identify the noun phrase denoting the target object in the question, the method extracts the object-to-visual attention submatrix and aggregates it over object tokens, a subset of intermediate decoder layers ($L_{\text{vis}}$), and all heads, yielding a per-visual-token relevance score. Because attention in MLLMs is contaminated by sink behavior—both the BOS token in the language backbone and certain visually salient but uninformative patches attract disproportionate attention mass—the method additionally computes a sink score for each visual token. This score is the maximum absolute activation over a fixed set of hidden dimensions $\mathcal{D}_{\text{sink}}$, identified as those where the BOS token exhibits consistently high normalized activations, averaged over layers in $L_{\text{sink}}$. Tokens whose sink score exceeds a threshold $\tau$ (set to the 25th percentile) are zeroed out of the relevance vector. The filtered map is converted to a bounding box via a weighted centroid and per-axis weighted standard deviations scaled by $\beta = 2$.

Textual evidence selection uses last-to-context attention from the final token, aggregated over the deeper half of the decoder layers and all heads, then pooled over sentences. The single sentence with the maximum score is selected as relevant evidence.

Highlighting wraps the selected sentence in <START_IMPORTANT_TXT>/<END_IMPORTANT_TXT> markers and the cropped bounding box region in <START_IMPORTANT_IMG>/<END_IMPORTANT_IMG> markers, with updated system and user prompts instructing the model that the markers denote crucial evidence. The answer is then generated on this modified input. The authors report that the procedure reduces visual tokens from 291 to 208 on average (a 28.5% reduction, exploiting dynamic resolution) and increases generation cost by only about 5.6% due to the single extra token.

Main results

The evaluation covers four KB-VQA benchmarks—Encyclopedic-VQA (E-VQA), InfoSeek, OVEN, and ViQuAE—using a cross-modal retrieval pipeline with EVA-CLIP embeddings and FAISS search ($n=3$ passages), across eleven off-the-shelf MLLMs ranging from Qwen2-VL-2B to InternVL3.5-38B.

LoT improves the four-benchmark average for every backbone, with gains ranging from +1.1 (Qwen2.5-VL-7B) to +5.3 (Qwen2-VL-7B). Representative results:

Backbone	Baseline Avg	+ LoT Avg	Gain
Qwen2-VL-2B	10.2	11.9	+1.7
Qwen2.5-VL-3B	21.2	25.5	+4.3
Qwen2-VL-7B	22.9	28.2	+5.3
Qwen3-VL-8B	31.5	35.0	+3.5
InternVL3.5-38B	34.1	37.5	+3.1

Gains are most pronounced on InfoSeek and ViQuAE. Notable individual improvements include Qwen2-VL-2B on InfoSeek (5.4 → 10.3, nearly doubling a very weak baseline), InternVL3.5-4B on ViQuAE (36.4 → 45.6), and Qwen3-VL-8B on ViQuAE (43.7 → 51.0). The improvements persist at the largest scale (InternVL3.5-38B: 34.1 → 37.5), indicating the mechanism does not become redundant as model capacity grows. One exception appears in the OVEN column, where Qwen3-VL-4B and InternVL3.5-38B slightly degrade (25.1 → 24.5 and 20.2 → 18.5 respectively); the paper does not discuss these regressions.

Ablations confirm that visual and textual highlighting are complementary: on E-VQA with Qwen2.5-VL-3B, textual highlighting alone reaches 29.4 and visual alone 29.6 (from 27.8), while the combination reaches 30.4. When the number of retrieved passages $n$ increases from 2 to 3, baselines stagnate because added recall is offset by contextual noise, whereas LoT continues to improve, supporting the claim that highlighting mitigates retrieval noise. LoT also improves performance when provided oracle Wikipedia evidence for the ground-truth entity, indicating the benefit is not merely an artifact of noisy retrieval.

Generalization beyond KB-VQA

Applying visual highlighting alone (no retrieved context), LoT yields improvements or comparable results on vision-centric benchmarks (RealWorldQA, V-Star), OCR and chart understanding (TextVQA, ChartQA, OCRBench), and hallucination benchmarks (POPE, AMBER-D). The most striking result is on AMBER-D with Qwen2-VL-2B: 42.1 → 75.0, and with Qwen2.5-VL-3B: 17.2 → 40.5—large absolute gains on a discriminative hallucination metric. Qwen2-VL-7B also improves substantially on both POPE (70.3 → 89.1) and AMBER-D (33.7 → 47.3). However, the results are not uniformly positive: InternVL3.5-4B degrades on POPE (89.0 → 86.4) and AMBER-D (80.1 → 73.0), and Qwen3-VL-8B drops on AMBER-D (75.9 → 62.6, though the table appears to duplicate the Qwen2.5-VL-7B row for this entry, which warrants caution). ChartQA with Qwen2.5-VL-32B improves only from a pathological 35.7 to 48.2, remaining far below other models. The paper characterizes these fluctuations honestly but offers no systematic explanation of when visual highlighting hurts.

Design choices validated in ablations

The supplementary ablations establish several non-obvious design decisions. For visual highlighting, simply drawing the bounding box on the input image hurts (−2 points on E-VQA for Qwen2.5-VL-3B); the effective strategy is to feed only the cropped region wrapped in markers, which also reduces visual token count. Adding markers to both the original image and the crop, or including the crop without markers, underperforms. For textual highlighting, sentence-level granularity outperforms passage-level on E-VQA but is slightly worse on InfoSeek, and highlighting the entire retrieved context is inferior to highlighting only the selected evidence. For bounding box extraction, the weighted centroid method achieves the best IoU/coverage/precision trade-off (IoU 0.487) when validated against GroundingDINO predictions as a proxy ground truth—an evaluation choice that is itself a limitation, since no human bounding box annotations exist for the queried objects.

Limitations and open questions

Several limitations are acknowledged or evident. The sink-dimension set $\mathcal{D}_{\text{sink}}$ is architecture-specific and must be determined per backbone, and the layer subsets $L_{\text{vis}}$ and $L_{\text{txt}}$ rely on prior findings about where visual grounding and textual reliance emerge, rather than being learned or automatically selected. The target object extraction depends on spaCy parsing, which may fail on complex or ambiguous questions. The bounding box assumes a single, spatially compact evidence region, which is inadequate for questions requiring multiple dispersed regions. The evaluation of localization quality rests on GroundingDINO as a proxy rather than human annotation. The inconsistent results on OVEN and on some hallucination benchmarks—particularly the AMBER-D degradation for InternVL3.5-4B and Qwen3-VL-8B—are reported without analysis, leaving open the question of which model or benchmark properties determine whether attention-based highlighting helps or harms. Finally, the method highlights exactly one sentence and one region per query by construction; whether adaptive or multiple evidence selection would further improve performance is not explored.

Conclusion

LoT demonstrates that the internal attention dynamics of pretrained MLLMs, when filtered for sink artifacts and surfaced through prompt-level markers, provide a low-cost mechanism for improving multimodal evidence selection. The consistent gains across ten backbones on KB-VQA, the robustness to increased retrieval noise, and the large improvements on hallucination benchmarks for several models establish attention-based evidence highlighting as an effective training-free complement to retrieval-augmented pipelines. The principal open issues are the method's sensitivity to backbone-specific sink characteristics, its single-region and single-sentence assumptions, and the unexplained regressions on a subset of models and benchmarks.