Utility-Oriented Visual Evidence Selection for Multimodal Retrieval-Augmented Generation

Abstract: Visual evidence selection is a critical component of multimodal retrieval-augmented generation (RAG), yet existing methods typically rely on semantic relevance or surface-level similarity, which are often misaligned with the actual utility of visual evidence for downstream reasoning. We reformulate multimodal evidence selection from an information-theoretic perspective by defining evidence utility as the information gain induced on a model's output distribution. To overcome the intractability of answer-space optimization, we introduce a latent notion of evidence helpfulness and theoretically show that, under mild assumptions, ranking evidence by information gain on this latent variable is equivalent to answer-space utility. We further propose a training-free, surrogate-accelerated framework that efficiently estimates evidence utility using lightweight multimodal models. Experiments on MRAG-Bench and Visual-RAG across multiple model families demonstrate that our method consistently outperforms state-of-the-art RAG baselines while achieving substantial reductions in computational cost.

• The paper reformulates visual evidence selection as a model-centric, information-theoretic problem that maximizes utility over mere semantic similarity.
• It introduces a latent Bernoulli variable to proxy evidence helpfulness, enabling efficient surrogate-based ranking of images.
• Empirical results show up to +16.18 accuracy gains and 20× efficiency improvements, outperforming traditional relevance-based methods.

Utility-Oriented Visual Evidence Selection in Multimodal RAG

Motivation and Problem Formulation

Multimodal retrieval-augmented generation (RAG) systems rely on the ability to retrieve relevant visual evidence (e.g., images) to enhance the factuality and reasoning capabilities of large multimodal models (LMMs). Existing retrieval and evidence selection strategies overwhelmingly prioritize semantic similarity or surface-level relevance, rooted in embedding-based metrics or pointwise/listwise judgments from multimodal transformers. However, high semantic or visual similarity does not necessarily translate to high utility for downstream tasks: retrieved images may share salient but irrelevant attributes, or even be misleading for the intended reasoning objective. This central misalignment between retrieval relevance and evidence utility is a foundational challenge in retrieval-augmented vision-language pipelines.

Figure 1: Illustrative case: the image on the left ranks higher in similarity but does not provide discriminative support for the query; the correct visual evidence is not the most "relevant" in the usual embedding sense.

The work proposes a reformulation of visual evidence selection as a model-centric, information-theoretic problem, focusing explicitly on the information gain induced by a candidate image on an LMM's output distribution. The aim is to systematically select evidence that alters the probability of the correct answer, not just evidence that is visually similar or salient.

Theoretical Framework: From Information Gain to Latent Helpfulness

The naïve approach is to define the utility of a candidate image $c$ as the information gain (KL divergence) between the model's output distributions with and without the image, i.e.,

$$\mathrm{IG}(Y; C = c \mid q) = D_{\mathrm{KL}}( P_{Y | C = c, q} \;||\; P_{Y | q} )$$

where $P_{Y | C = c, q}$ is the posterior over answers with evidence $c$ given query $q$, and $P_{Y | q}$ is the prior (no evidence).

Directly optimizing this metric in the full answer space is intractable due to implicit, high-dimensional generative distributions and instability induced by linguistic variation and model hallucination. The authors therefore introduce a latent Bernoulli variable $Z$ representing whether a candidate is helpful to generation, and analytically demonstrate that, under mild assumptions, the information gain on this latent helpfulness variable is monotonically aligned with answer-space information gain.

This yields an efficient, low-dimensional objective:

$\mathrm{IG}(Z; C = c \mid q)$

which can be further reduced to ranking candidates by the modeled probability that $Z = 1$ (i.e., the likelihood that the visual evidence is helpful):

$\max_{c} P(Z=1|C=c, q)$

Theoretical analysis (including monotonicity proofs and regularity assumptions) supports that this latent variable formulation retains the essential discriminative power of answer-space optimization while avoiding its computational pathologies.

Surrogate-Accelerated and Training-Free Evidence Ranking

To operationalize this criterion, the paper proposes a surrogate-accelerated selection pipeline (see Figure 2), with two principal components:

1. Utility-Centric Metric Design: Evidence helpfulness is directly assessed via an auxiliary query that prompts the model to output a binary helpful/unhelpful signal for a given (query, image) pair.
2. Surrogate Model Acceleration: Instead of querying the heavyweight main model for each candidate, a lightweight surrogate multimodal model efficiently computes helpfulness logits for all retrieved images. The main LMM is invoked only once on the final, top-ranked subset.

   Figure 2: Framework overview—discriminative utility estimation reduces to latent helpfulness modeling, enabling high-throughput surrogate-based ranking for scalable RAG.

This pipelined architecture yields strong computational benefits: the number of high-cost model inferences scales with the number of final selections (not candidates), and lightweight surrogates provide rapid scoring without retraining.

Experimental Validation and Results

The method was rigorously evaluated on two challenging, vision-centric RAG benchmarks:

1. MRAG-Bench: 1,353 image-question pairs with annotated ground-truth images requiring visual reasoning variations (angle, perspective, temporal, biological processes).
2. Visual-RAG: 374 text queries with associated ground-truth images focused on knowledge-intensive visual attributes and hard negatives.

Key empirical findings:

• Utility-oriented selection with lightweight surrogates consistently outperforms all state-of-the-art baselines (CLIP-style, MLLM-based retrievers, supervised rerankers, and answer-level uncertainty quantifiers), achieving up to +16.18 improvement in task accuracy metrics.
• Surrogate models (2B–4B parameters) closely match or exceed the performance of the corresponding main LMMs (8B–12B), indicating strong transferability of helpfulness signal across scales.
• The new criterion matches or slightly surpasses the ground-truth oracle in some settings, i.e., selected evidence sometimes proves more informative for model generation than human-labeled relevant images—a surprising outcome highlighting the limitations of "relevance" as currently annotated in benchmarks.
• Direct answer-space uncertainty criteria (average token probability, MC dropout, or choice entropy) are less stable and are outperformed by the latent helpfulness approach, especially in open-ended generation scenarios.
• The surrogate-accelerated method achieves 20× or greater efficiency gains in decoding FLOPs for utility estimation compared to standard answer-level uncertainty estimation workflows.

Qualitative Analysis and Case Studies

The qualitative study further clarifies failure regimes of relevance-based retrieval—retrieving visually similar but non-informative evidence—and demonstrates that the utility-driven criterion selects images carrying task-discriminative features, especially in settings where subtle attribute differences dictate the correct answer.

Figure 3: (a) An example where CLIP and GME relevance scores prioritize non-discriminative, visually similar images, while utility-based ranking surfaces the evidence required for correct model output. (b) A demonstration that answer-level uncertainty is misguided by generation stability, while utility estimation keys on informative evidence.

Practical Implications, Limitations, and Future Directions

This work compels a methodological shift: multimodal evidence selection must be explicitly utility-aware, not simply relevance-driven. Practical multimodal RAG pipelines should incorporate discriminative, model-centric utility probes for evidence selection, leveraging lightweight surrogates for scalable deployment.

The approach is model-agnostic, training-free, and robust to moderate prompt variations, showing promise for transfer to larger model scales and more complex modalities. However, its current validation is limited to vision-centric QA scenarios; extension to other modalities (video, audio), longer context, and broader generation tasks (captioning, dialogue, complex reasoning) remain open directions.

Another major implication is the necessity for new benchmark annotation paradigms—current human-labeled relevance signals may be insufficient for utility-aware model behavior. As such, deploying utility-oriented selection could improve not just RAG accuracy but the interpretability and safety of downstream multimodal systems.

Conclusion

This work redefines visual evidence selection for multimodal RAG as an information-theoretic, model-centric optimization problem, departing from longstanding relevance-focused approaches. By introducing a latent helpfulness proxy and pairing it with efficient surrogate-based execution, the framework achieves superior task performance and computational scalability. The theoretical and empirical results argue for the adoption of utility-based evidence selection as the standard in retrieval-augmented vision-language generation and set the agenda for further work on generalizing discriminative selection criteria to broader multimodal and cross-modal RAG applications.

Figure 4: Query image illustrative of the vision-centric reasoning benchmarks explored in the study.