Purifying Multimodal Retrieval: Fragment-Level Evidence Selection for RAG

Abstract: Multimodal Retrieval-Augmented Generation (MRAG) is widely adopted for Multimodal LLMs (MLLMs) with external evidence to reduce hallucinations. Despite its success, most existing MRAG frameworks treat retrieved evidence as indivisible documents, implicitly assuming that all content within a document is equally informative. In practice, however, sometimes only a small fraction of a document is relevant to a given query, while the remaining content introduces substantial noise that may lead to performance degradation. We address this fundamental limitation by reframing MRAG as a fine-grained evidence selection problem. We propose Fragment-level Evidence Selection for RAG (FES-RAG), a framework that selects atomic multimodal fragments rather than entire documents as grounding evidence. FES-RAG decomposes retrieved multimodal documents into sentence-level textual fragments and region-level visual fragments, enabling precise identification of evidence that directly supports generation. To guide fragment selection, we introduce Fragment Information Gain (FIG), a principled metric that measures the marginal contribution of each fragment to the MLLM's generation confidence. Based on FIG, we distill fragment-level utility judgments from a high-capacity MLLM into a lightweight selector, achieving accurate evidence selection with low inference overhead. Experiments on the M2RAG benchmark show that FES-RAG consistently outperforms state-of-the-art document-level MRAG methods, achieving up to 27 percent relative improvement in CIDEr. By selecting fewer yet more informative fragments, our approach substantially reduces context length while improving factual accuracy and generation coherence.

• The paper presents a novel FES-RAG framework that decomposes multimodal documents into fine-grained fragments for improved retrieval and evidence selection.
• It leverages the FIG metric and knowledge distillation to isolate high-utility fragments, significantly enhancing MMQA accuracy and reducing noise.
• The approach achieves reduced inference latency and lower context token usage while boosting factual generation reliability.

Fragment-Level Evidence Purification in Multimodal RAG

Motivation and Limitations of Traditional MRAG

Multimodal Retrieval-Augmented Generation (MRAG) architectures have become a core strategy for equipping Multimodal LLMs (MLLMs) with up-to-date, external knowledge sources and for mitigating generation hallucinations. However, the canonical MRAG approach treats retrieved evidence as atomic, document-level units. This assumption is misaligned with downstream reasoning, as often only localized regions or key sentences within retrieved documents contain the factual evidence required to answer a given query, while other content introduces semantically adjacent but ultimately irrelevant or distracting noise. Traditional pipelines thus lead to noise accumulation, exhibit sensitivity to context window position (“lost-in-the-middle”), and degrade both the accuracy and efficiency of multimodal generation.

Figure 1: Comparison of the traditional MRAG pipeline (a) and the FES-RAG framework (b). Traditional methods focus on document-level retrieval and reranking, whereas FES-RAG introduces document segmentation and fragment selection for evidence purification.

FES-RAG: Architecture and Methodology

To address these deficiencies, the paper proposes FES-RAG (Fragment-level Evidence Selection for RAG), a general framework that decomposes multimodal retrieved documents—text and images—into fine-grained segments, isolates high-utility fragments at the sentence and region-of-interest (ROI) level, and filters evidence based on their direct contribution to generation reliability.

The framework proceeds in several key stages:

1. Multimodal Segmentation: Retrieved documents are recursively decomposed. Text documents are split using a score-driven binary segmentation that prioritizes fragments which maximize cross-encoder relevance to the query. Visual documents undergo grounding-based ROI extraction using Grounding DINO, generating atomic visual fragments subject to objectness, semantic alignment, and size constraints.
2. Fragment Information Gain (FIG): The central metric, FIG, quantifies the impact of inserting a fragment into the context on the generation probability of the ground-truth answer, using a high-capacity, frozen MLLM as a teacher. This principled, length-normalized log-likelihood delta acts as the ground for utility supervision.
3. Selector Model with Knowledge Distillation: FIG signals are discretized into hard “evidence/nonevidence” labels and, together with teacher output distribution, distilled into a lightweight student selector trained via binary cross-entropy and KL loss.
4. Hybrid Pool and Context Purification: At inference, a hybrid candidate pool is formed by combining segmented fragments and remaining coarse documents. The selector scores these efficiently, and only the top-K fragments are prepended as evidence for the final MLLM generation stage.

   Figure 2: FES-RAG uses a teacher MLLM to label fragment utility (left) and a lightweight student selector to efficiently filter fragments for generation (right).

Empirical Analysis

Accuracy and Efficiency Gains

On the $M^2$RAG multimodal benchmark, FES-RAG consistently surpasses state-of-the-art document-level MRAG systems across MMQA, image captioning, and fact verification. The most pronounced gains are realized in MMQA, where CIDEr scores improve by up to 27% relative to the strongest baselines, while context token consumption is substantially reduced. Notably, these improvements are observed even when compared under matched retrieval budgets, thus isolating the effect of fragment-level purification.

Selector Efficacy and Distillation

Ablation on selector supervision reveals that knowledge distillation from a large teacher MLLM enables the lightweight student selector to outperform hard-label supervised analogs by a significant margin. Distillation not only preserves fine-grained fragment ordering but also robustly captures local utility differences not expressible through discrete binary targets alone.

Figure 3: Knowledge distillation enables the student selector to significantly outperform hard-label only training, demonstrating consistent gains even with restricted teacher capacity.

Computational Trade-offs

Despite the introduction of additional segmentation and selection phases, end-to-end inference latency is reduced. This counterintuitive result is driven by the drop in context length, which disproportionately decreases the cost of MLLM generation due to the quadratic scaling of attention operations.

Figure 4: FES-RAG reduces both context token utilization and total inference latency relative to coarse-grained baselines.

Qualitative Fragment Analysis

The selector demonstrates the ability to distinguish between fragments containing answer-bearing evidence and those containing only semantically related but uninformative content. Visual segmentations focus on the query-relevant ROI while ignoring visually salient distractors, and textual segmentations can filter out misleading named entities sharing query terms.

Figure 5: FES-RAG’s selector prunes irrelevant yet semantically close fragments and preserves key evidence across both modalities.

Implications and Future Directions

FES-RAG’s paradigm shift—from document-level to atomic fragment-level evidence selection—challenges the default grain of MRAG contexts and establishes fragment utility, not semantic similarity, as the primary retrieval objective. This framework gives rise to several theoretical and practical implications:

• MLLMs exhibit improved robustness to context position and noise when evidence context is purified at the appropriate granularity.
• Selector models distilled from powerful teacher MLLMs can perform fast, fine-grained fragment selection with minimal computational overhead, enabling practical deployments.
• The methodology generalizes across architectures, suggesting underlying properties of factuality and distraction in large multimodal models are independent of backbone design.

Further research should explore dynamic fragment granularity selection, joint optimization of fragmentation and generation, and applicability of fragment-based RAG in domains with long-chain reasoning or dominant visual grounding requirements. Moreover, interleaved multi-hop selection and adversarial segment filtering stand as promising directions to further enhance factual consistency and rationales in multimodal generation.

Conclusion

FES-RAG demonstrates that fine-grained, FIG-driven evidence filtration yields substantial improvements in efficiency, factuality, and robustness for MRAG systems. By exposing the limitations of document-level evidence and enabling atomic, cross-modal fragmentation supervised by generative confidence scores, this framework defines a new standard for evidence utility assessment and retrieval-augmented multimodal reasoning (2604.27600).