Inserter-R1-3B: RL-based Image Inserter

• Inserter-R1-3B is a reinforcement-learning-based image inserter that selects and positions candidate images to create coherent multimodal responses.
• The module reformulates image insertion as a structured planning task, using a discrete output format to map images to specific answer sentences.
• Trained with Group Relative Policy Optimization on a tailored M2IO-Inserter dataset, it achieves competitive performance with lower latency compared to larger models.

Searching arXiv for the target paper and closely related R1-style multimodal RL work. First, locating "M2IO-R1" and the Inserter-R1-3B description. Now checking related R1-style 3B multimodal reasoning and action models for context. Inserter-R1-3B is a reinforcement-learning-based image inserter within M2IO-R1, a framework for Multimodal Retrieval-Augmented Multimodal Generation (MRAMG) that supports both multimodal inputs and outputs. Its function is narrowly defined but technically central: given a user question, a generated answer split into sentences, and a set of candidate images, it determines which images to include and where to place them in the answer. The model is presented as the core module for the third stage of M2IO-R1—image insertion—and is trained with Group Relative Policy Optimization (GRPO) to optimize image selection and placement as a sequential, outcome-driven decision problem. Empirically, the reported system is a lightweight 3B inserter that achieves strong reasoning capabilities with significantly reduced latency, while outperforming baselines in both quality and efficiency (Xiao et al., 8 Aug 2025).

1. System role within multimodal retrieval-augmented multimodal generation

M2IO-R1 decomposes generation into four stages: Multimodal Retrieval, Text Answer Generation, Image Insertion, and Merging. Inserter-R1-3B is the module responsible for the third stage. Its purpose is not to produce the answer text itself, but to decide which retrieved or candidate images should be inserted and at which sentence positions so that the final output becomes an interleaved, coherent multimodal response (Xiao et al., 8 Aug 2025).

This design places image insertion between textual answer synthesis and final multimodal composition. In the reported formulation, the inserter supplements text with visuals rather than jointly generating unrestricted multimodal content from scratch. A plausible implication is that M2IO-R1 separates text generation from multimodal layout reasoning so that image selection and placement can be optimized with task-specific rewards that are difficult to express in standard supervised fine-tuning.

The task setting is explicitly sentence-indexed. The inserter receives the ground-truth answer split into sentences and a set of candidate images with visual features, captions, or contexts. It then produces an insertion plan indicating which image IDs should appear after which sentences. This makes the image insertion problem operationally close to structured multimodal planning rather than generic captioning or visual question answering (Xiao et al., 8 Aug 2025).

2. Model architecture, input structure, and output specification

Inserter-R1-3B is initialized from Qwen2.5-VL-3B-Instruct, a 3-billion-parameter multimodal LLM chosen for efficiency and competitive performance. The model is trained to emit two structured blocks: a > block containing token-level reasoning for image selection and placement, and an <answer> block specifying a dictionary that maps each chosen image ID to its insertion position, defined as a sentence index (Xiao et al., 8 Aug 2025).

The model’s input structure contains three elements: the user question, the answer split into sentences, and the candidate image set. The action space is sentence-local but globally constrained. For each sentence in the answer, the inserter either selects a single image from the candidate set to insert after that sentence, or chooses to insert none. This formulation explicitly limits the per-sentence action to one image or no image, while still requiring the model to maintain coherence across the full answer sequence (Xiao et al., 8 Aug 2025).

The architecture is therefore best understood as a multimodal policy model with a structured textual interface. The <think> channel is presented as increasing interpretability, while the <answer> dictionary is the executable insertion specification. Because the final decision object is a dictionary over image IDs and sentence positions, the system converts multimodal reasoning into a discrete, verifiable output format amenable to automatic reward computation.

3. Reinforcement-learning formulation and reward design

The training method is GRPO, implemented through the EasyR1 library. The paper motivates RL by emphasizing that image selection and placement is intrinsically sequential and outcome-driven, and that supervised fine-tuning does not capture optimum strategies across all steps. The training regime is explicitly inspired by R1-style RL paradigms in which only the final outcome is rewarded rather than intermediate decisions (Xiao et al., 8 Aug 2025).

The total reward for an inserter output $\mathcal{I}$ is defined as

$$\mathcal{R}_{\text{total}} = \mathcal{R}_{\text{format}} + \mathcal{R}_{\text{answer}}.$$

The format reward enforces structural validity:

$$\mathcal{R}_{\text{format}} = \begin{cases} 0, & \text{if format is incorrect} \ 1, & \text{if format is correct.} \end{cases}$$

The answer reward is computed only when the format is valid and combines image recall with positional accuracy:

$$\mathcal{R}_{\text{rec}} = \frac{|\mathcal{I}\cap \mathcal{I}^*|}{|\mathcal{I}^*|}$$

$$\mathcal{R}_{\text{pos}} = \frac{1}{m}\sum_{k=1}^{m}\mathbb{I}\{i_k = i_k^*\}$$

$$\mathcal{R}_{\text{answer}} = \begin{cases} 0, & \text{if } \mathcal{R}_{\text{format}} = 0 \ \alpha \mathcal{R}_{\text{rec}} + (1-\alpha)\mathcal{R}_{\text{pos}}, & \text{otherwise.} \end{cases}$$

The reported optimal trade-off is $\alpha = 0.8$ (Xiao et al., 8 Aug 2025).

Several training details are specified. The model is initialized from an SFT model based on Qwen2.5-VL-3B-Instruct. The batch size is 32, the learning rate is $1\mathrm{e}{-6}$, and the KL-penalty coefficient is 0.01. Training uses a high-quality, stratified M2IO-Inserter dataset with manual image-text alignments for supervision. Only the final state is rewarded in each episode, matching the desired outcome-driven insertion logic (Xiao et al., 8 Aug 2025).

This reward design is notable for separating structural correctness from semantic and positional correctness. Format validity is a hard gate, while semantic coverage and placement fidelity are softly balanced. That decomposition makes the objective verifiable and non-differentiable in precisely the manner that motivates policy optimization.

4. Image selection and placement as multimodal reasoning

The reported criteria for image selection are based on visual-textual semantic relevance between each image and each answer segment. Placement decisions are made sentence by sentence, but with reference to the utility of each image for augmenting a sentence’s meaning and to the alignment of image position with the flow and referential logic of the text. The model may also omit images when none are relevant. Global coherence is mediated by the explicit reasoning process emitted in <think> tags (Xiao et al., 8 Aug 2025).

This formulation treats insertion as more than retrieval reranking. The model must jointly solve two coupled subproblems: relevance filtering and temporal or discourse placement. An image can be relevant yet misplaced; conversely, a correctly positioned image can still fail if it is semantically weak. The reward terms reflect that distinction by separately evaluating recall and positional agreement.

The model’s “single image per sentence” constraint also shapes the reasoning problem. It prevents unrestricted visual densification and forces a sparse allocation strategy, which is consistent with the reported objective of controllable and semantically aligned multimodal output. A plausible implication is that Inserter-R1-3B is designed for concise, interleaved document-style outputs rather than gallery-style presentation.

5. Reported performance, efficiency, and generalization

The principal empirical claim is that RL-tuned Inserter-R1-3B matches or surpasses state-of-the-art reasoning ability of models over $20\times$ larger, including M2IO-Base-72B. On the Arxiv setting with GPT-4o as generator, the reported scores are Recall 84.2, Rel 97.4, and Ovr 76.3 for M2IO-R1-3B, compared with Recall 83.7, Rel 97.6, and Ovr 76.6 for M2IO-Base-72B. On complex Manual datasets, the RL inserter is reported to significantly outperform Single-Shot and Rule-Based baselines, including on Order and Overall scores (Xiao et al., 8 Aug 2025).

The main reported task-wise results are as follows.

Task	Reported result	Interpretation in the paper
Arxiv	Recall 84.2, F1 68.4, Rel 97.4, Ovr 76.3	Matches 72B-scale performance
Manual	Recall 52.6, F1 46.5, Rel 90.5, Ovr 62.6	Stronger than Single-Shot and Rule-Based
Web	Recall 93.3, F1 93.3, Rel 97.0, Ovr 84.4	Competitive with 72B model
FTII	Recall 52.7	Reported as SOTA on streaming image insertion

Efficiency is another explicit result. The reported inference time is approximately 4.34 seconds per instance, compared with 5.98 seconds for Single-Shot and 22.6 seconds for Rule-Based. The cost is described as comparable to or lower than Rule-Based, while also being significantly cheaper than Single-Shot because the inserter runs locally and avoids expensive inference on large models (Xiao et al., 8 Aug 2025).

The paper also reports robustness and out-of-domain behavior. When trained on just web data, performance on Arxiv, Manual, and Recipe remains competitive, and the model shows less overfitting than SFT. The Arxiv F1 score is reported to improve relative to full-data training, which the paper attributes to RL’s regularization via outcome-based reward. In ablations, RL-based fine-tuning consistently outperforms SFT-only training, especially as answer complexity grows; one explicit comparison on Manual with web-only training reports F1 = 40.0 for M2IO-R1-3B versus F1 = 27.9 for the SFT variant (Xiao et al., 8 Aug 2025).

These results support a narrow but important claim: in the reported benchmarks, a compact RL-tuned inserter can be competitive with much larger multimodal systems when the task is formulated as structured image selection and placement rather than unconstrained multimodal generation.

6. Position within R1-style multimodal RL research

Inserter-R1-3B belongs to a broader 2025 line of compact, Qwen2.5-based, R1-style multimodal systems that use verifiable rewards, structured outputs, and policy optimization to improve reasoning or action selection. LMM-R1 applies a two-stage rule-based RL pipeline to Qwen2.5-VL-Instruct-3B and reports average improvements in multimodal and text-only benchmarks, framing reasoning transfer from text to multimodal domains as a data-efficient paradigm (Peng et al., 10 Mar 2025). GUI-R1 uses unified action space rule modeling and GRPO for GUI agents and reports superior performance with only 3K training samples compared to 13M used by prior state-of-the-art methods (Luo et al., 14 Apr 2025). UI-R1 similarly uses rule-based action rewards and GRPO for GUI action prediction with only 136 curated training samples (Lu et al., 27 Mar 2025). Embodied-R1 uses a pointing-centric intermediate representation and a two-stage reinforced fine-tuning curriculum, again with a 3B Qwen2.5-VL-based backbone and GRPO-style optimization (Yuan et al., 19 Aug 2025).

At the same time, related work exposes relevant design tensions. GUI-G1 argues that blindly transplanting R1-Zero-like training into GUI grounding can introduce reward hacking, length bias, and under-optimization on hard samples; it proposes a Fast Thinking Template, box size constraints in the reward, and difficulty-aware policy updates (Zhou et al., 21 May 2025). This suggests that Inserter-R1-3B’s success depends not merely on using RL, but on defining a task-specific action space and reward that are well aligned with insertion quality and placement fidelity.

Within this family, Inserter-R1-3B is distinctive in targeting multimodal output construction rather than multimodal question answering, GUI control, or embodied pointing. Its policy does not ground a click, emit a robot waypoint, or solve a visual reasoning benchmark directly. Instead, it optimizes interleaving decisions over answer sentences and candidate images. In that sense, it operationalizes a specific form of multimodal reasoning: deciding how retrieved visual evidence should be sequenced into generated discourse.

A plausible implication is that the model illustrates an emerging decomposition strategy for multimodal systems: large or external modules may handle retrieval and answer drafting, while a smaller RL-tuned specialist handles discrete, verifiable multimodal structuring decisions. That interpretation is consistent with the paper’s emphasis on decoupling large-model text generation from local, lightweight multimodal reasoning (Xiao et al., 8 Aug 2025).