InterleaveThinker: Reinforcing Agentic Interleaved Generation

Abstract: Recent image generators have demonstrated impressive photorealism and instruction-following capabilities in single-image generation and editing. However, constrained by their architectures, they cannot achieve interleaved generation (text-image sequence), which has crucial applications in visual narratives, guidance, and embodied manipulation. Even the latest open-source Unified Multimodal Models (UMMs) exhibit limited performance in this regard. In this paper, we introduce InterleaveThinker, the first multi-agent pipeline designed to endow any existing image generator with interleaved generation capabilities. Specifically, we employ a planner agent to organize the image-text input sequence, instructing the image generator on the required execution at each step. Subsequently, we introduce a critic agent to evaluate the generator's outputs, identify samples that deviate from the planned instructions, and refine the instructions for regeneration. To implement this pipeline, we construct the Interleave-Planner-SFT-80k and Interleave-Critic-SFT-112k to perform a format cold-start. Then we develop Interleave-Critic-RL-13k to reinforce the step-wise instruction correction capability within a generation trajectory using GRPO. Since a single interleaved generation trajectory may involve over 25 generator calls, optimizing the entire trajectory is computationally impractical. Therefore, we propose accuracy reward and step-wise reward, allowing single-step RL to effectively guide the entire generation trajectory. The results show that InterleaveThinker improves performance across various image generators. On interleaved generation benchmarks, it achieves performance comparable to Nano Banana and GPT-5. Surprisingly, it also significantly enhances the base model on reasoning-based benchmarks; for example, on 4-step FLUX.2-klein, we observe substantial gains on WISE and RISE.

• The paper introduces a multi-agent pipeline that decouples planning, generation, and evaluation to reduce step-wise error accumulation in interleaved text-image tasks.
• It employs a novel single-step RL formulation with dual-reward signals that optimizes prompt refinement and enhances visual fidelity.
• Empirical results demonstrate superior long-horizon consistency and semantic fidelity, outperforming traditional unified multimodal models.

InterleaveThinker: Reinforcing Agentic Interleaved Generation

Motivation and Problem Setting

Conventional image generators deliver photorealistic outputs and follow natural language instructions effectively for single-image generation and editing. However, they lack the architectural capacity to handle interleaved, multi-modal generation tasks—specifically, the sequential alternation of text and image steps. Such interleaved generation is critical for visual storytelling, sequential guidance, and embodied manipulation. Unified Multimodal Models (UMMs) can ostensibly address this but suffer from compounding step-wise errors and visual over-reliance during long-horizon generation. This induces premature task termination and performance degradation as generation length increases.

Figure 1: Typical error modes in image generators and UMMs for interleaved generation, including step-wise error accumulation and visual myopia.

Proposed Multi-Agent Architecture

InterleaveThinker is introduced as a model-agnostic, multi-agent pipeline to equip arbitrary frozen image generators with robust interleaved generation capabilities. The core design decouples planning, generation, and evaluation across independent agents:

• Planner Agent: Constructs a step-wise execution plan upfront, producing detailed instructions and model-tailored prompts for each generation or editing stage. The plan is generated solely from the user’s multi-modal input sequence.
• Generator: Applies the planned prompts sequentially, performing image generation (first step) or image editing (subsequent steps), conditioned on previous outputs as needed.
• Critic Agent: Post-processes each generation step, evaluating semantic alignment and visual fidelity. If discrepancies are detected, the Critic synthesizes refined prompts for the Generator and iterates until successful satisfaction or a refinement cap.

The planner-critic separation is crucial: it disrupts the feedback loop that causes visual over-reliance and allows for explicit, externalized self-correction, mitigating error accumulation.

Figure 2: Overview of the InterleaveThinker multi-agent inference pipeline, illustrating the iterative planner-generator-critic loop and refinement mechanism.

Figure 3: Detailed example of InterleaveThinker’s working flow. Each step includes proposed instructions, image generation, critic-based evaluation, and prompt refinement if needed.

Data Pipeline and Training Paradigm

A pivotal contribution is the scalable data construction pipeline, which produces high-quality, domain-diverse training data comprising interleaved plans, intermediate images, critic judgments, and prompt refinements.

• Comprehensive prompt banks spanning ~8 root domains and >75 detailed subcategories are assembled by programmatic expansion using Gemini 2.5 Pro and Nano Banana Pro as prompters.
• Multi-agent trajectories are harvested via composed planner, generator, and critic actions, creating explicit step-wise records suitable for supervised fine-tuning (SFT) or reinforcement learning (RL).
• Rigorous critic-based filtering and balancing are employed to enforce annotation quality and to avoid collapsed critic behaviors.

  Figure 4: Data construction pipeline for InterleaveThinker, covering prompt synthesis, multi-agent trajectory sampling, and step-wise data curation.

Three core datasets are curated:

• Interleave-Planner-SFT-80k: For planner instruction breakdown and prompt engineering.
• Interleave-Critic-SFT-112k: For critic supervision and iterative refinement.
• Interleave-Critic-RL-13k: For optimizing critic correction capabilities using Gradual Reward Policy Optimization (GRPO).

Dual-Reward RL Innovation

Multi-step RL on full interleaved trajectories is computationally prohibitive given the high number of generator calls (>25 per trajectory). InterleaveThinker’s solution is a novel single-step RL formulation with dual-reward signals:

• Accuracy Reward: Penalizes deviation between the Critic’s step-level judgment and the expert reference, enforcing reliable satisfaction detection.
• Step-wise Reward: Rewards the merit of each refinement based on its effect on third-party quality metrics (e.g., Gemini scoring differentials), directly optimizing prompt rewrite efficacy.

Combined, these reward signals facilitate trajectory-level alignment while maintaining feasible compute demands.

Empirical Results

Comprehensive experiments validate the efficacy and generality of InterleaveThinker:

• On UEval, InterleaveThinker (with FLUX.2-klein-9B generator) achieves average scores of 66.3, surpassing all open-sourced UMMs, and aligning with proprietary leaders such as Nano Banana and GPT-5.
• On the CoMM benchmark, integration with both FLUX.2-klein-9B and Qwen-Image-Edit-2511 achieves state-of-the-art across all evaluation axes, including style/entity consistency and text-image alignment.
• For reasoning-centric generation (WISE, RISE), the uncoupled agentic structure yields dramatic improvements. E.g., WISE jumps from 0.47 (FLUX.2-klein-9B baseline) to 0.73 (with InterleaveThinker) and RISE improves from 13.3 to 28.9.

  Figure 5: Comparative performance with Emu3.5 and Nano Banana Pro in pure-text interleaved generation, demonstrating InterleaveThinker’s improved text-image coherence and step-wise accuracy.

  

  Figure 6: Comparative performance in multi-modal input scenarios, establishing InterleaveThinker’s superior style, entity, and temporal consistency.

Ablation studies confirm the necessity of the multi-agent design (vs. single-agent alternatives), critic data filtering, and the dual-reward RL formulation. Increasing the refinement cap $T_{max}$ consistently boosts performance by affording the Critic more opportunities for correction and improvement.

Limitations and Negative Cases

The framework’s limitations are fundamentally governed by the generative prior of the base image generator: concepts outside the generator’s learned domain remain inaccessible, and failure cases emerge when the generator cannot represent required visual information. These deficiencies are model-intrinsic and cannot be rectified solely by improved planning or critique.

Figure 7: Failure case of InterleaveThinker+FLUX.2-klein, highlighting inability to correct generator-internal limitations and domain gaps.

Implications and Future Prospects

InterleaveThinker substantially advances the modularity, reliability, and task generality of interleaved multi-modal generation. The planner-generator-critic division offers both a practical blueprint for production pipelines and theoretical insight into overcoming horizon limitations in sequential reasoning and generation tasks. The RL formulation provides a scalable methodology for agentic training in environments where trajectory length and reward sparsity impose severe optimization constraints.

In practical terms, this architecture opens new avenues for visual narrative automation, process-centric instructional systems, and agent-robot interfaces where precise, multi-step visual manipulation is essential.

Theoretically, future developments may focus on tighter co-adaptation between Critic and Generator (e.g., generator fine-tuning based on critic signals), extension to richer modalities (video, 3D, action), and further automated co-evolution of agentic policies in richer, more complex multi-agent environments.

Conclusion

InterleaveThinker enables robust, reliable agentic interleaved generation by combining explicit step planning, externalized critique, and reinforcement-based instruction refinement. It is model-agnostic, scales across image generators, and demonstrates superior long-horizon consistency and semantic fidelity in interleaved text-image generation tasks. This approach sets a foundational path for modular, agent-driven multi-modal generation, with broad implications for both practical systems and the future trajectory of multimodal AI research.