RAG-IGBench: Interleaved Multimodal Q&A Benchmark

• The paper introduces RAG-IGBench, integrating retrieval, selection, ordering, and integration of multimodal evidence into coherent markdown responses.
• It overcomes unimodal evaluation limitations by jointly assessing text quality, image selection, ordering, and image–text consistency.
• Empirical results with diverse MLLMs and refined metrics demonstrate improved correlation with human judgments and potential for reinforcement learning optimization.

RAG-IGBench is a benchmark for RAG-based interleaved generation (RAG-IG) in open-domain question answering, a setting in which a multimodal LLM (MLLM) must retrieve relevant multimodal evidence and generate a markdown response that interleaves explanatory text with the correct images in the correct order (Zhang et al., 11 Oct 2025). Its central premise is that such outputs cannot be evaluated adequately by text-only metrics or image-only metrics, because the object of evaluation is the interleaved answer as a whole: textual correctness, image selection, image ordering, and image–text alignment all matter simultaneously.

1. Motivation and conceptual scope

RAG-IGBench was proposed to evaluate a multimodal generation regime that is practically distinct from standard retrieval-augmented generation and from conventional multimodal question answering. In the task formulation described by the benchmark, the system does not merely answer with text after retrieval; it must retrieve, select, arrange, and integrate multimodal evidence into a coherent interleaved response (Zhang et al., 11 Oct 2025).

The benchmark is motivated by two deficiencies in prior work. First, existing generation methods for multimodal answers remain limited. The paper identifies two broad tendencies: systems that use separate pipelines for text and image generation, which can induce semantic mismatch between modalities, and unified transformer-style models, which can remain costly and can still struggle with complex instructions, long contexts, and precise image–text coordination (Zhang et al., 11 Oct 2025). Second, evaluation methodologies for interleaved image–text outputs are underdeveloped. The paper argues that unimodal metrics such as BLEU, ROUGE, perplexity, FID, and IS are structurally insufficient for a task in which correctness depends on both modality content and modality interaction.

A recurring misconception is to treat RAG-IGBench as a generic multimodal RAG benchmark. The benchmark is narrower and more specific. It targets cases in which visual content is genuinely needed, and it explicitly excludes queries that can be answered adequately with text alone (Zhang et al., 11 Oct 2025). Another misconception is to view it as an image generation benchmark. The paper instead formulates image evaluation as a selection-and-ordering problem, because the model selects images from retrieved candidates rather than synthesizing new images from scratch.

2. Formal task definition

The benchmark formalizes RAG-IG as follows. Given a query $q$, the system retrieves a set of documents and associated image sets:

$$D = \{d_1, d_2, \ldots, d_n\}, \qquad I = \{I_1, I_2, \ldots, I_n\}$$

where each $I_i$ is the image set for the $i$-th retrieved document (Zhang et al., 11 Oct 2025). The retrieved documents, images, query, and instruction prompt are then provided to an MLLM.

The model must produce a response in markdown format, with image references expressed as placeholders such as

$$

where $k$ denotes an image index (Zhang et al., 11 Oct 2025). Those placeholders are subsequently replaced with actual image URLs to obtain the final interleaved answer.

Two operational constraints define the scope of the benchmark. First, the model must handle multiple images in context, which the paper treats as materially harder than single-image settings. Second, the benchmark retrieves at most three documents per query, because larger multimodal contexts enlarge the input window and degrade MLLM performance (Zhang et al., 11 Oct 2025). These design choices make the benchmark simultaneously a retrieval problem, a generation problem, and an output-structuring problem.

The benchmark’s required output format is itself part of the task. Correctness is not exhausted by semantic adequacy; models must also preserve markdown validity and refer only to images that are actually present in the retrieved set. This makes instruction following and reference discipline integral to evaluation rather than incidental implementation details.

3. Corpus construction and curation

RAG-IGBench is constructed from Xiaohongshu, a Chinese social media platform (Zhang et al., 11 Oct 2025). The paper states that the dataset was checked for privacy, patents, and intellectual property issues, and that content containing identifiable personal information was removed. Although the original corpus is Chinese, the benchmark includes a high-fidelity English version produced using strong LLMs such as GPT-4o for translation.

The final benchmark contains 6,057 queries. Each query is paired with images and answer labels. The paper reports an average of 12.96 retrieved images per query and 2.26 generated images per answer. Each sample is represented as

$(q, gt, D, I)$

where $q$ is the original user query, $gt$ is the ground-truth answer in markdown format with text and image indices, $D$ is the retrieved contextual document set, and $$D = \{d_1, d_2, \ldots, d_n\}, \qquad I = \{I_1, I_2, \ldots, I_n\}$$0 is the corresponding image set (Zhang et al., 11 Oct 2025).

Construction proceeds in three stages. In Stage 1, MLLMs including GPT-4o, Claude-3.5-Sonnet, Gemini-1.5-Pro, and others generate raw markdown answers from the query, retrieved documents, and images. Queries are first classified into four categories: what-is, how-to, yes-or-no, and head-to-head. The paper motivates this categorization by noting that these query types require different forms of visual support; for example, procedural “how-to” questions favor stepwise visual guidance, while “head-to-head” questions often benefit from comparative visual presentation (Zhang et al., 11 Oct 2025).

In Stage 2, raw outputs are manually annotated and refined by expert multimodal evaluators with high inter-rater reliability of $$D = \{d_1, d_2, \ldots, d_n\}, \qquad I = \{I_1, I_2, \ldots, I_n\}$$1. Annotators score each answer on text quality, image quality, image-text consistency, and overall quality, and they also refine the image selection by adding, removing, or replacing images to construct high-quality ground-truth answers (Zhang et al., 11 Oct 2025).

In Stage 3, question–answer pairs with low scores in any dimension are removed, while cases improved through manual refinement are retained. The raw pool contains 18,634 cases, and the final benchmark is distilled to 6,057 curated samples (Zhang et al., 11 Oct 2025). This curation procedure indicates that the benchmark is not a simple scrape-and-score corpus but a filtered and manually regularized evaluation set.

4. Metric design and validation

RAG-IGBench evaluates three dimensions: text quality, image quality, and image-text consistency (Zhang et al., 11 Oct 2025). The paper emphasizes reference-based, rule-based, and multi-dimensional evaluation in order to reduce the bias, instability, and prompt sensitivity associated with MLLM-as-a-judge protocols.

For text quality, the benchmark uses ROUGE-1. The paper notes that ROUGE-2 and ROUGE-L were considered, but that ROUGE-1 was found to be more discriminative on this benchmark (Zhang et al., 11 Oct 2025).

For image quality, the benchmark treats evaluation as an ordered-list comparison problem. Let

$$D = \{d_1, d_2, \ldots, d_n\}, \qquad I = \{I_1, I_2, \ldots, I_n\}$$2

and

$$D = \{d_1, d_2, \ldots, d_n\}, \qquad I = \{I_1, I_2, \ldots, I_n\}$$3

The first metric is a normalized Edit Distance over image sequences:

$$D = \{d_1, d_2, \ldots, d_n\}, \qquad I = \{I_1, I_2, \ldots, I_n\}$$4

where $$D = \{d_1, d_2, \ldots, d_n\}, \qquad I = \{I_1, I_2, \ldots, I_n\}$$5 is the dynamic programming matrix (Zhang et al., 11 Oct 2025). This measures how many insertion, deletion, and substitution operations are needed to transform the generated image sequence into the reference sequence.

The second image metric is Kendall Score, which evaluates whether the correctly selected images are arranged in the correct order. Let

$$D = \{d_1, d_2, \ldots, d_n\}, \qquad I = \{I_1, I_2, \ldots, I_n\}$$6

be the intersection of generated and ground-truth image lists, ordered by appearance in the generated answer. The score is defined as

$$D = \{d_1, d_2, \ldots, d_n\}, \qquad I = \{I_1, I_2, \ldots, I_n\}$$7

where $$D = \{d_1, d_2, \ldots, d_n\}, \qquad I = \{I_1, I_2, \ldots, I_n\}$$8 is the position of $$D = \{d_1, d_2, \ldots, d_n\}, \qquad I = \{I_1, I_2, \ldots, I_n\}$$9 in the ground-truth list (Zhang et al., 11 Oct 2025).

For image-text consistency, the benchmark uses both CLIP Score and a new Alignment Score. CLIP Score provides a standard embedding-space alignment signal, but the paper explicitly notes its limitations for complex semantics and interleaved context sensitivity. Alignment Score is designed to compare the textual environment of each correctly selected image in the generated answer to the corresponding environment in the reference answer:

$I_i$0

where $I_i$1, $I_i$2 is the context of $I_i$3 in the generated answer, $I_i$4 is the context of $I_i$5 in the ground-truth answer, and $I_i$6 is computed using embedding similarity (Zhang et al., 11 Oct 2025).

The metric suite is validated on 200 randomly sampled cases against human ratings. For image quality, FID obtains Pearson $I_i$7 and Spearman $I_i$8, IS obtains Pearson $I_i$9 and Spearman $i$0, while the RAG-IGBench metrics obtain Pearson 0.256 and Spearman 0.244. For image-text consistency, CLIP-Score obtains Pearson $i$1 and Spearman $i$2, while the RAG-IGBench metrics obtain Pearson 0.172 and Spearman 0.145. For overall assessment, GPT4o-based evaluation obtains Pearson 0.017 and Spearman 0.028, while RAG-IGBench obtains Pearson 0.143 and Spearman 0.152 (Zhang et al., 11 Oct 2025). The paper interprets these results as evidence that its metrics correlate better with human judgments than the conventional alternatives it tests.

5. Empirical performance and failure modes

The benchmark evaluates both proprietary and open-source MLLMs. Proprietary models include GPT-4o, Claude 3.5-Sonnet, Gemini-1.5-Pro, and QwenVL-Max. Open-source models include Qwen2VL-7B / 72B, LLaVA-OneVision 72B, InternVL2.5 8B / 78B, NVLM-D-72B, InternVL2-Llama3-76B, and Qwen2.5VL-7B / 72B (Zhang et al., 11 Oct 2025). All models use greedy decoding, experiments run on a single machine with 8 H800 GPUs, CLIP Score uses CLIP-ViT-Large-Patch14, Alignment Score uses Conan-embedding, and images are resized to a uniform width of 540 pixels.

The overall ranking reported by the paper places GPT-4o first overall, with Gemini-1.5-Pro next among proprietary models. Among open-source systems, Qwen2.5VL-72B and InternVL2.5 78B are the strongest (Zhang et al., 11 Oct 2025). The paper also reports that proprietary models retain a clearer advantage on image-quality-related metrics than on CLIP-based image–text matching, suggesting that image selection and ordering remain more difficult than coarse semantic matching.

Three principal failure modes are identified. The first is hallucination, in which models refer to image indices that do not exist. The second is instruction-following failure, especially malformed markdown output. The third is long-context overload, in which too many images and documents reduce performance and memory efficiency (Zhang et al., 11 Oct 2025). The paper specifically notes that LLaVA-OneVision 72B produces many invalid-format outputs, while other models hallucinate image references.

An ablation on Qwen2VL-7B varies both the number of input images and image resolution. Performance decreases as the number of input images increases, even at low resolution. Higher resolution helps when the number of images is small to moderate, but with many images higher resolution can reduce performance by expanding context length and memory burden (Zhang et al., 11 Oct 2025). This places the benchmark in a regime where retrieval breadth and visual fidelity interact directly with generation quality.

6. Use as a training resource

RAG-IGBench is presented not only as an evaluation benchmark but also as a source of supervision for multimodal interleaved generation. The paper reports that supervised fine-tuning on the benchmark’s training split improves both in-domain and cross-benchmark performance (Zhang et al., 11 Oct 2025).

For Qwen2VL-7B, supervised fine-tuning increases the RAG-IGBench dev score from 26.12 to 36.10, and the average performance across other benchmarks from 59.33 to 63.23. Reported gains include MANTIS-Eval $i$3, Q-Bench2 $i$4, BLINK $i$5, and MMB $i$6, with a slight drop on NLVR2 of $i$7 (Zhang et al., 11 Oct 2025). The paper interprets this as evidence that the dataset provides useful supervision beyond its own test distribution.

The appendix further extends the benchmark’s role into optimization. It includes experiments with reasoning-oriented MLLMs such as VL-Rethinker-7B, Vision-R1-7B, Claude 3.7-Sonnet-thinking, and OpenAI o1. The reported pattern is that Claude 3.7-thinking outperforms Claude 3.5, VL-Rethinker improves over Qwen2.5VL-7B, and Vision-R1 performs worse, which the paper attributes to a specialization toward mathematical reasoning rather than broad multimodal reasoning (Zhang et al., 11 Oct 2025).

The proposed metrics are also used as rule-based rewards for GRPO training. On Qwen2.5VL-3B and 7B, the paper reports that SFT mainly improves text quality, GRPO more strongly improves image quality and image-text consistency, and SFT + GRPO is compatible and yields strong overall results (Zhang et al., 11 Oct 2025). This suggests that the benchmark can function as an optimization scaffold for models whose outputs contain both symbolic structure and multimodal selection behavior.

7. Relation to adjacent benchmarks and limitations

RAG-IGBench occupies a distinct position within the broader RAG benchmark landscape. RAGBench is an explainable benchmark for RAG evaluation centered on span-level labels and the TRACe framework—uTilization, Relevance, Adherence, and Completeness—across text-centric domains such as biomedical research, legal contracts, customer support, finance, and general knowledge (Friel et al., 2024). UniDoc-Bench evaluates document-centric multimodal RAG across text, tables, and figures with unified protocols for text-only, image-only, fusion, and joint retrieval, and it emphasizes retrieval and answer completeness rather than markdown-level interleaving of images and text (Peng et al., 4 Oct 2025). IRB argues that future RAG benchmarks such as RAG-IGBench should avoid static, heavily manual evaluation and should prefer grounded, controllable, updatable benchmark generation (Do et al., 8 Feb 2026). This suggests a complementary division of labor: RAG-IGBench stresses interleaved multimodal answer composition, while adjacent work stresses explainability, document-centric multimodal retrieval, or benchmark freshness and controllability.

The benchmark’s limitations are stated explicitly. Its data comes from a single platform, Xiaohongshu. The original corpus is predominantly Chinese, though translated into English. The benchmark is relatively small compared with pretraining corpora, so it is not intended as a pretraining-scale resource. The ground truth is partly derived from model-generated answers, which introduces possible bias, although the paper attempts to mitigate this through multiple generating models and human refinement (Zhang et al., 11 Oct 2025).

The future directions proposed by the paper follow directly from these constraints: scaling the benchmark while preserving quality, improving evaluation methods for interleaved multimodal generation, reducing hallucination, handling longer multi-image contexts, developing more robust reasoning and retrieval strategies for RAG-IG, and using rule-based metrics to support RL or GRPO training in open-domain multimodal settings (Zhang et al., 11 Oct 2025). In that sense, RAG-IGBench defines not only a benchmark but also a problem class: retrieval-augmented, visually grounded answer synthesis in which the structure of the output is part of the semantic task.