M2IO-R1-3B: RL-Enhanced Multimodal Inserter

• M2IO-R1-3B is an RL-enhanced multimodal inserter that integrates chain-of-thought reasoning and sequential decision making for precise text–image interleaving.
• It leverages a compact 3B vision–language transformer within a four-stage pipeline to reduce latency and API cost while achieving state-of-the-art performance.
• The model employs Group Relative Policy Optimization to enhance reward alignment through format validation, recall, and positional accuracy in image insertion.

M2IO-R1-3B is an RL-enhanced multimodal inserter model architected for fine-grained, outcome-driven image placement in Multimodal Retrieval-Augmented Multimodal Generation pipelines. It integrates a chain-of-thought reasoning protocol with sequential decision making, leveraging reinforcement learning (specifically Group Relative Policy Optimization) to deliver efficient, high-quality text–image interleaving within a 3B-parameter vision–language transformer. M2IO-R1-3B combines lightweight model size, controllable multimodal output, and strong empirical performance relative to much larger models, establishing a new state-of-the-art for retrieval-augmented generation with multimodal outputs (Xiao et al., 8 Aug 2025).

1. Architectural Principles

M2IO-R1-3B builds on Qwen2.5-VL-3B-Instruct, a 3B-parameter multimodal decoder-only transformer. Key architectural components include:

• Vision–Language Interface: Each candidate image (retrieved via the upstream retriever, e.g., BGE-M3) is encoded by a fixed vision backbone (BGE-VL-base), with image embeddings prepended, via cross-attention, to the input token stream.
• Prompt Structure: Input consists of a natural-language "question" $q$, a list of $m$ answer sentences $S = \{s_1, ..., s_m\}$, and a set of $k$ candidate image embeddings $I_n = \{i_1, ..., i_k\}$.
• Think–Answer Protocol: The model interleaves chain-of-thought reasoning ($<$think$>$...$<$/think$>$) with structured decision emission ($<$answer$>${"img17":3, ...}$<$/answer$>$), separating semantic evaluation from action selection for transparency and controllability.
• Autoregressive Action Space: The decision process is cast as a sequential selection problem over sentences, with per-step actions $a_t$ corresponding to an image selection or null "no-insert" for sentence $s_t$. The overall output space is $(|I_n| + 1)^m$, but with $m, |I_n| \leq 6$, this remains tractable.

2. RL Formulation and GRPO Optimization

Image insertion within M2IO-R1-3B is formulated as a finite-horizon Markov Decision Process:

• State Definition: At step $t$, the state $s_t$ comprises the prefix of previous actions $(a_1, ..., a_{t-1})$, the full sentence set $S$, and all image embeddings.
• Actions: $a_t \in I_n \cup \{\varnothing\}$, representing either inserting one of the candidate images after sentence $s_t$, or abstaining.
• Reward Design: After the full sequence $a = (a_1, ..., a_m)$, a single scalar reward $R_{total}$ is computed, comprising a format check and a composite answer reward:
  • $R_{format}$: 1 if $<$answer$>$ format is valid, 0 otherwise.
  • $R_{answer} = \alpha R_{rec} + (1-\alpha)R_{pos}$, where $R_{rec}$ is the fraction of correctly selected images and $R_{pos}$ is the average positional accuracy; $\alpha=0.8$ is empirically optimal.
  • $R_{total} = R_{format} + R_{answer}$.

Policy learning proceeds via Group Relative Policy Optimization (GRPO), a PPO-variant tailored to multi-modal decision problems:

• Objective: For each minibatch,

$$L^{CLIP}(\theta) = \mathbb{E}_t[\min (r_t(\theta) A_t,\, \mathrm{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\,A_t)]$$

with $A_t = R_{total} - b$ and $$r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{old}(a_t|s_t)}$$; $\epsilon \approx 0.2$.

• KL Penalty: A regularization term $$L_{KL}(\theta) = \mathbb{E}_t[\mathrm{KL}(\pi_{old}(\cdot|s_t)\|\pi_\theta(\cdot|s_t))]$$ is added with $\lambda = 0.01$ to prevent policy drift.
• Variance Reduction: The baseline $b$ is the batch-mean reward. Clipping $r_t$ further stabilizes training.

3. Training Pipeline and Data Regime

Training M2IO-R1-3B follows a four-stage retrieval–generation–insertion–merger pipeline:

• Stage 1: Retriever: The question $q$ is embedded (BGE-M3), retrieving top-$k$ documents. Text and associated images are extracted.
• Stage 2: Text Generation: Retrieved texts and $q$ are input to a large LLM (e.g., GPT-4o, Qwen2.5-VL-72B), generating a multi-sentence answer $S$.
• Stage 3: Inserter: M2IO-R1-3B is prompted with $(q, S, I_n)$, generating interleaved $<$think$>$/$<$answer$>$ output.
• Stage 4: Merger: Sentences and selected images are interleaved at specified insertion points.

Training uses the M2IO-Inserter dataset, bottlenecked to 2.4K examples for efficiency and diversity (Web, Wiki, Wit, Arxiv, Recipe, Manual domains). Each example balances positive and adversarial distractor images. RL fine-tuning is performed with a global batch size of 32, learning rate $1 \times 10^{-6}$, KL penalty 0.01, on 4× A100 GPUs. Convergence is typically reached within $10^4$ steps (~6 hours).

4. Empirical Evaluation and Ablations

On MRAMG-Bench (using GPT-4o text answers), M2IO-R1-3B delivers:

• Recall: 84.2 (vs 80.1 for single-shot baseline)
• F1: 68.4 (vs 69.1 single-shot, but higher for image-position/relevance metrics)
• Overall (Ovr): 76.3 (vs 74.8 single-shot, 69.8 rule-based)
• Latency: 4.34s/instance (vs 5.98s for M2IO-Base-72B)
• API Cost: $0.24/instance, halved relative to a 72B model

On FTII-Bench, F1 reaches 52.7, significantly outperforming rule-based (29.8) and SFT-3B (46.3) baselines.

An ablation over reward weights $\alpha\in\{0.0,0.2,0.4,0.5,0.6,0.8,1.0\}$identifies$\alpha=0.8$$as the point of maximal balance between recall and positional alignment.</p> <p>In contrast to supervised finetuning, RL-based optimization with outcome-only rewards is credited for both the accuracy and efficiency gains: chain-of-thought ($$<$think$>$$) traces supply transparency and improve action alignment, while the GRPO objective induces stable, high-reward behavior.</p> <h2 class='paper-heading' id='model-trade-offs-and-design-insights'>5. Model Trade-Offs and Design Insights</h2> <p>M2IO-R1-3B demonstrates that compact <a href="https://www.emergentmind.com/topics/multimodal-models-lmms" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">multimodal models</a> can reach and even exceed the performance of much larger systems under RL-enhanced, outcome-centric training. Principal design characteristics and constraints include:</p> <ul> <li><strong>Compactness vs. Quality</strong>: The 3B model rivals 72B models for output <a href="https://www.emergentmind.com/topics/fidelity-alpha-precision" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">fidelity</a> and layout, while incurring roughly half the latency and cost, largely due to RL-based policy stabilization and outcome-focused <a href="https://www.emergentmind.com/topics/reward-engineering" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">reward engineering</a>.</li> <li><strong>Reasoning Decomposition</strong>: Explicit division into$$<$think$>$and$<$answer$>$ promotes interpretability and supports error analysis at the reasoning or decision level.

Modality Control: The design enables tight control over placement and relevance, necessary for fine-grained multimodal synthesis.

Limitation: The current framework allows only one image per sentence. Extensions to multi-image insertions or richer layouts remain an open research problem.

Reward Function: Present reward structure is rule-based, not learned. Moving towards learned, preference-driven rewards may improve alignment with human judgement or downstream performance.

6. Practical Applications and Future Directions

M2IO-R1-3B offers deployment advantages in settings where retrieval-augmented, multimodal outputs are necessary (document summarization, instructional content creation, AI-powered report generation). Key potential avenues for further development include:

• Reward Learning: Integrating human preference signals or learned reward models could replace hand-crafted objectives, yielding even better task alignment.
• Beyond Sequential Insertion: Generalizing the decision process to allow for hierarchical state/action representations will enable more complex output formats, such as tables or highly structured multimedia documents.
• Scaling Data and Model: While the current regime focuses on low-data, small-model efficiency, further data scaling and integration of larger backbones may enable even stronger generalization, particularly for longer or more complex input-output pairs.
• Rich Inference Constraints: Enabling multi-image per sentence insertions, variable layout, and expanded post-processing steps for more intricate user requirements.

M2IO-R1-3B establishes a new benchmark for RL-driven multimodal output generation, pairing efficient inference with transparent, stepwise decision making and demonstrating the viability of outcome-directed RL optimization in compact multimodal models (Xiao et al., 8 Aug 2025).