ClueRecall: Evaluating Multimodal Attention

Updated 9 February 2026

ClueRecall is a formal metric that quantifies how well a model’s internal attention aligns with annotated visual evidence, enhancing interpretability in vision-language tasks.
It computes the overlap between top attended visual tokens and gold-standard bounding boxes at each decoder layer to diagnose reasoning drift and hallucination issues.
Empirical results on models like R1-OneVision demonstrate its effectiveness as a training-free, architecture-agnostic tool for identifying optimal grounding layers.

ClueRecall is a formal metric and analytic framework introduced to evaluate the accuracy with which multimodal reasoning models internally localize the visual “clues” that support their outputs. Rather than assessing only task-level performance, ClueRecall quantifies the model’s perceptual fidelity by measuring, at each decoder layer, the overlap between the model’s attention and gold-standard evidence regions in the input, with principal applications in reasoning-driven hallucination suppression and interpretability for large vision–LLMs (Xi et al., 2 Feb 2026).

1. Conceptual Overview

ClueRecall is designed for analyzing the internal workings of multimodal models, particularly their capacity to attend to task-relevant regions of input images while generating textual outputs in vision–language reasoning tasks. Rather than focusing on final prediction accuracy, ClueRecall interrogates internal layers to determine the extent to which a model’s attention aligns with ground-truth objects or regions necessary for correct reasoning.

A central motivation is to identify and alleviate "reasoning drift," where attention deviates from true visual evidence as inference proceeds, leading to unsupported or hallucinatory outputs (Xi et al., 2 Feb 2026). ClueRecall offers a direct, parameter-free, and training-free measure of model “visual clue retrieval” at every decoder layer.

2. Formal Definition and Computation

ClueRecall is defined for each decoder layer $\ell$ as the average proportion of ground-truth visual tokens (those corresponding to annotated object bounding boxes) that are amongst the top attended tokens when the model emits the queried object category in its output. Formally, for a perception-labeled set $\mathcal{X}_{perc}$ , each consisting of a (question, image, bounding box, object label) tuple, the metric is computed as:

$\mathrm{ClueRecall}(\ell) = \frac{1}{|\mathcal{X}_{perc}|} \sum_{(\mathbf{X}_q,\mathbf{X}_v,\mathbf{X}_{bbox},\texttt{cat})\in\mathcal{X}_{perc}} \frac{ \left| \mathrm{TopK}_{|\mathbf{X}_{bbox}|} \left( \sum_{t\,:\,y_t=\texttt{cat}} A_{t,\ell,\mathcal{V}} \right) \cap \mathbf{X}_{bbox} \right| }{|\mathbf{X}_{bbox}|}$

where:

$A_{t,\ell,\mathcal{V}}$ is the model’s attention over visual tokens at decode step $t$ and layer $\ell$ ,
the sum is over decode steps $t$ where the output token $y_t$ is the queried object,
$\mathrm{TopK}_m(\cdot)$ selects the $m$ visual token indices with highest total attention,
$|\mathbf{X}_{bbox}|$ is the number of tokens covering the ground-truth region.

No tunable thresholds or spatial heuristics are introduced; $|\mathbf{X}_{bbox}|$ directly determines the number of top tokens for overlap computation.

3. Practical Workflow and Integration

To compute ClueRecall:

Data Preparation: Assemble $\mathcal{X}_{perc}$ using datasets such as MSCOCO, pairing images and object bounding-boxes with templated yes/no queries about object presence.
Model Execution: Input tokenized question, image tokens, and process through the decoder, recording attention tensors $A_{t,\ell,i}$ at each step and layer.
Metric Calculation: For each $(𝑞, 𝑣, 𝑏𝑜𝑥, 𝑐𝑎𝑡)$ instance and each layer, sum attention over output steps emitting the object token, select top attended visual tokens equal in cardinality to the box, and compute their intersection with ground-truth. Average this recall across the data set.

The layer $\ell$ with maximal ClueRecall, $L_{\max}$ , is used as the principal "grounding layer" for downstream interventions such as suppressing hallucinations in reasoning models by reinforcing or cropping attention onto the most relevant patches (Xi et al., 2 Feb 2026).

4. Empirical Findings and Interpretation

Applied to several 28-layer, 7B-parameter vision–language decoders (including R1-OneVision, Ocean-R1, MM-Eureka, ORSTA-R1), ClueRecall reveals a rise–fall profile across layers:

Early layers ( $\ell\le6$ ): ClueRecall ≈ 30–34%
Mid-layers ( $\ell\approx12$ –24): ClueRecall peaks at 50–55%
Final layers ( $\ell\approx27$ –28): ClueRecall decreases to 44–49%

For instance, in R1-OneVision:

Layer ( $\ell$ )	ClueRecall (%)
0	30.8
6	33.7
12	43.1
18 ( $L_{max}$ )	50.6
24	47.3
27	44.1

The maximally grounded layer, $L_{max}$ , is the optimal point for extracting the model's underlying object-perception trace, informing interventions that can directly bias attention, select regions to crop or refeed, and thereby robustly suppress reasoning-induced hallucinations.

5. Strengths, Limitations, and Future Directions

Strengths

Training and Parameter Free: ClueRecall operates without modifying model weights or requiring supplementary training.
Architecture Agnostic: Applicable to any Transformer-style multimodal decoder exposing internal attention matrices.
Granular: Provides step-aware localization of model focus, conditioned on when key tokens are emitted.

Limitations

Currently restricted to yes/no object queries using bounding box overlap; does not extend natively to complex spatial, relational, or open-ended language tasks.
Captures recall but not precision; a high ClueRecall does not penalize attention spreading outside ground-truth regions.

Potential Extensions

Defining a “CluePrecision” metric to penalize spurious attention.
Generalization to multi-object, spatial-relation, or segmentation-based labeling tasks.
Augmenting with segmentation or panoptic masks to expand beyond datasets with bounding boxes.

A plausible implication is that integrating ClueRecall-like internal measures with external prediction accuracy can yield more nuanced diagnostics for model evaluation and targeted remediation.

ClueRecall is distinct from general task accuracy metrics (e.g., VQA accuracy) and from the ROUGE-L-based “KeyInfo.” evaluation used in reading comprehension for LLMs (Gu et al., 2023). It also differs from recall probability estimation in adaptive memory frameworks (as in educational systems), which are modeled using logistic regression or power-law forgetting curves (Mooney et al., 2018); ClueRecall operates on model attention over visual input, not over spaced repetition of factual cues.

No direct analog to ClueRecall is present in attribute-specific associative memory models (e.g., Cue Ball–Recall Net) or in language-only key information detection benchmarks, which evaluate recall indirectly via token- or span-level sequence metrics.

7. Impact and Research Significance

ClueRecall establishes a standard for layer-wise, attention-centric evaluation of visual clue localization in multimodal reasoning architectures (Xi et al., 2 Feb 2026). By exposing where and how visual evidence is integrated or neglected during long-chain inference, it facilitates fine-grained ablation, interpretability analyses, and mechanism design for reducing hallucinations. The metric serves both as a core diagnostic for architectural and training interventions, and as a practical tool for robust, training-free hallucination suppression workflows across diverse vision–LLMs.