M2IO-R1-3B: RL-Enhanced Multimodal Inserter

• M2IO-R1-3B is an RL-enhanced multimodal inserter that integrates chain-of-thought reasoning and sequential decision making for precise text–image interleaving.
• It leverages a compact 3B vision–language transformer within a four-stage pipeline to reduce latency and API cost while achieving state-of-the-art performance.
• The model employs Group Relative Policy Optimization to enhance reward alignment through format validation, recall, and positional accuracy in image insertion.

M2IO-R1-3B is an RL-enhanced multimodal inserter model architected for fine-grained, outcome-driven image placement in Multimodal Retrieval-Augmented Multimodal Generation pipelines. It integrates a chain-of-thought reasoning protocol with sequential decision making, leveraging reinforcement learning (specifically Group Relative Policy Optimization) to deliver efficient, high-quality text–image interleaving within a 3B-parameter vision–language transformer. M2IO-R1-3B combines lightweight model size, controllable multimodal output, and strong empirical performance relative to much larger models, establishing a new state-of-the-art for retrieval-augmented generation with multimodal outputs (Xiao et al., 8 Aug 2025).

1. Architectural Principles

M2IO-R1-3B builds on Qwen2.5-VL-3B-Instruct, a 3B-parameter multimodal decoder-only transformer. Key architectural components include:

• Vision–Language Interface: Each candidate image (retrieved via the upstream retriever, e.g., BGE-M3) is encoded by a fixed vision backbone (BGE-VL-base), with image embeddings prepended, via cross-attention, to the input token stream.
• Prompt Structure: Input consists of a natural-language "question" $q$, a list of $m$ answer sentences $S = \{s_1, ..., s_m\}$, and a set of $k$ candidate image embeddings $I_n = \{i_1, ..., i_k\}$.
• Think–Answer Protocol: The model interleaves chain-of-thought reasoning ($<$think$>$...$<$/think$>$) with structured decision emission ($<$answer$m$0{"img17":3, ...}$m$1/answer$m$2), separating semantic evaluation from action selection for transparency and controllability.
• Autoregressive Action Space: The decision process is cast as a sequential selection problem over sentences, with per-step actions $m$3 corresponding to an image selection or null "no-insert" for sentence $m$4. The overall output space is $m$5, but with $m$6, this remains tractable.

2. RL Formulation and GRPO Optimization

Image insertion within M2IO-R1-3B is formulated as a finite-horizon Markov Decision Process:

• State Definition: At step $m$7, the state $m$8 comprises the prefix of previous actions $m$9, the full sentence set $S = \{s_1, ..., s_m\}$0, and all image embeddings.
• Actions: $S = \{s_1, ..., s_m\}$1, representing either inserting one of the candidate images after sentence $S = \{s_1, ..., s_m\}$2, or abstaining.
• Reward Design: After the full sequence $S = \{s_1, ..., s_m\}$3, a single scalar reward $S = \{s_1, ..., s_m\}$4 is computed, comprising a format check and a composite answer reward:
  • $S = \{s_1, ..., s_m\}$5: 1 if $S = \{s_1, ..., s_m\}$6answer$S = \{s_1, ..., s_m\}$7 format is valid, 0 otherwise.
  • $S = \{s_1, ..., s_m\}$8, where $S = \{s_1, ..., s_m\}$9 is the fraction of correctly selected images and $k$0 is the average positional accuracy; $k$1 is empirically optimal.
  • $k$2.

Policy learning proceeds via Group Relative Policy Optimization (GRPO), a PPO-variant tailored to multi-modal decision problems:

• Objective: For each minibatch,

$k$3

with $k$4 and $k$5; $k$6.

• KL Penalty: A regularization term $k$7 is added with $k$8 to prevent policy drift.
• Variance Reduction: The baseline $k$9 is the batch-mean reward. Clipping $I_n = \{i_1, ..., i_k\}$0 further stabilizes training.

3. Training Pipeline and Data Regime

Training M2IO-R1-3B follows a four-stage retrieval–generation–insertion–merger pipeline:

• Stage 1: Retriever: The question $I_n = \{i_1, ..., i_k\}$1 is embedded (BGE-M3), retrieving top-$I_n = \{i_1, ..., i_k\}$2 documents. Text and associated images are extracted.
• Stage 2: Text Generation: Retrieved texts and $I_n = \{i_1, ..., i_k\}$3 are input to a large LLM (e.g., GPT-4o, Qwen2.5-VL-72B), generating a multi-sentence answer $I_n = \{i_1, ..., i_k\}$4.
• Stage 3: Inserter: M2IO-R1-3B is prompted with $I_n = \{i_1, ..., i_k\}$5, generating interleaved $I_n = \{i_1, ..., i_k\}$6think$I_n = \{i_1, ..., i_k\}$7/$I_n = \{i_1, ..., i_k\}$8answer$I_n = \{i_1, ..., i_k\}$9 output.
• Stage 4: Merger: Sentences and selected images are interleaved at specified insertion points.

Training uses the M2IO-Inserter dataset, bottlenecked to 2.4K examples for efficiency and diversity (Web, Wiki, Wit, Arxiv, Recipe, Manual domains). Each example balances positive and adversarial distractor images. RL fine-tuning is performed with a global batch size of 32, learning rate $<$0, KL penalty 0.01, on 4× A100 GPUs. Convergence is typically reached within $<$1 steps (~6 hours).

4. Empirical Evaluation and Ablations

On MRAMG-Bench (using GPT-4o text answers), M2IO-R1-3B delivers:

• Recall: 84.2 (vs 80.1 for single-shot baseline)
• F1: 68.4 (vs 69.1 single-shot, but higher for image-position/relevance metrics)
• Overall (Ovr): 76.3 (vs 74.8 single-shot, 69.8 rule-based)
• Latency: 4.34s/instance (vs 5.98s for M2IO-Base-72B)
• API Cost: $0.24/instance, halved relative to a 72B model

On FTII-Bench, F1 reaches 52.7, significantly outperforming rule-based (29.8) and SFT-3B (46.3) baselines.

An ablation over reward weights $<$2 identifies $<$3 as the point of maximal balance between recall and positional alignment.

In contrast to supervised finetuning, RL-based optimization with outcome-only rewards is credited for both the accuracy and efficiency gains: chain-of-thought ($<$4think$<$5) traces supply transparency and improve action alignment, while the GRPO objective induces stable, high-reward behavior.

5. Model Trade-Offs and Design Insights

M2IO-R1-3B demonstrates that compact multimodal models can reach and even exceed the performance of much larger systems under RL-enhanced, outcome-centric training. Principal design characteristics and constraints include:

• Compactness vs. Quality: The 3B model rivals 72B models for output fidelity and layout, while incurring roughly half the latency and cost, largely due to RL-based policy stabilization and outcome-focused reward engineering.
• Reasoning Decomposition: Explicit division into $<$6think$<$7 and $<$8answer$<$9 promotes interpretability and supports error analysis at the reasoning or decision level.
• Modality Control: The design enables tight control over placement and relevance, necessary for fine-grained multimodal synthesis.
• Limitation: The current framework allows only one image per sentence. Extensions to multi-image insertions or richer layouts remain an open research problem.
• Reward Function: Present reward structure is rule-based, not learned. Moving towards learned, preference-driven rewards may improve alignment with human judgement or downstream performance.

6. Practical Applications and Future Directions

M2IO-R1-3B offers deployment advantages in settings where retrieval-augmented, multimodal outputs are necessary (document summarization, instructional content creation, AI-powered report generation). Key potential avenues for further development include:

• Reward Learning: Integrating human preference signals or learned reward models could replace hand-crafted objectives, yielding even better task alignment.
• Beyond Sequential Insertion: Generalizing the decision process to allow for hierarchical state/action representations will enable more complex output formats, such as tables or highly structured multimedia documents.
• Scaling Data and Model: While the current regime focuses on low-data, small-model efficiency, further data scaling and integration of larger backbones may enable even stronger generalization, particularly for longer or more complex input-output pairs.
• Rich Inference Constraints: Enabling multi-image per sentence insertions, variable layout, and expanded post-processing steps for more intricate user requirements.

M2IO-R1-3B establishes a new benchmark for RL-driven multimodal output generation, pairing efficient inference with transparent, stepwise decision making and demonstrating the viability of outcome-directed RL optimization in compact multimodal models (Xiao et al., 8 Aug 2025).