Scaling Multi-Reference Image Generation with Dynamic Reward Optimization

Abstract: While personalized image generation has achieved remarkable progress, multi-reference image generation (MRIG) remains a challenging task. Most existing benchmarks fail to adequately evaluate complex MRIG scenarios, hindering further progress in this area. To better assess model performance on complex MRIG tasks, we introduce OmniRef-Bench, a benchmark that covers complex combinations of reference image types and a large number of reference images. Evaluations on OmniRef-Bench show that mainstream open-source models struggle in complex MRIG scenarios, and their performance deteriorates significantly as the number of mixed-type reference images increases. To address this issue, we propose DyRef, a two-stage training framework. In the first stage, supervised fine-tuning equips the model with the basic capability to handle complex MRIG tasks. In the second stage, we introduce Difficulty-aware Advantage Reweighting (DAR) and Discriminative Reward Scaling (DRS). DAR dynamically adjusts the optimization objective to improve performance when handling a large number of mixed-type reference images. DRS enlarges intra-group reward differences for more effective policy optimization. Experiments demonstrate that DyRef significantly improves the performance of open-source models on OmniRef-Bench and single-image editing benchmarks, demonstrating the effectiveness and generalization capability of our approach.

• The paper introduces OmniRef-Bench and DyRef to integrate diverse visual references into a single high-fidelity output.
• It details the DAR and DRS methods that dynamically reweight loss and amplify reward contrast for complex compositional tasks.
• Empirical results show DyRef-enhanced models achieving up to 29% improvements over open-source baselines in quality metrics.

Scaling Multi-Reference Image Generation with Dynamic Reward Optimization: An Expert Overview

Introduction

The paper "Scaling Multi-Reference Image Generation with Dynamic Reward Optimization" (2606.26947) addresses the critical challenge of Multi-Reference Image Generation (MRIG): the integration of heterogeneous visual references (e.g., subject, style, pose, background, lighting) into a single high-fidelity output under complex user instructions. Existing approaches are constrained by limited benchmark complexity and insufficient evaluation granularity, resulting in a substantial performance gap between open-source and closed-source systems when handling multiple, mixed-type reference images. This work introduces (1) OmniRef-Bench, a rigorously constructed and fine-grained MRIG benchmark, and (2) DyRef, a two-stage framework leveraging Difficulty-aware Advantage Reweighting (DAR) and Discriminative Reward Scaling (DRS) to enhance open-source diffusion models for complex compositional generation.

OmniRef-Bench: A Comprehensive Evaluation Platform

OmniRef-Bench is designed to stress-test MRIG models with intricate combinations of reference types, covering five principal controls: subject, style, background, lighting, and pose. Each benchmark instance may include between two and seven references, with up to four types combined, thereby reflecting the semantic complexity inherent to professional visual design and real-world creative workflows.

Figure 2: Overview of OmniRef-Bench, with example multi-reference compositions and a sunburst chart illustrating coverage of combinations and reference count.

Data construction employs both synthetic (via T2I models informed by LLM-generated prompts) and real-world imagery, coupled with robust automatic and expert data filtering for quality assurance. The benchmark's evaluation protocol fuses low-level objective metrics (e.g., CLIP-I, DINOv2, custom lighting/pose scores) with high-level Multimodal LLM (MLLM)-based ratings (Gemini 3 Flash), enabling nuanced, multidimensional performance assessment. This hybrid strategy overcomes the semantic myopia of objective metrics and the spatial imprecision of LLMs, as demonstrated by targeted failure analyses.

Analysis of Limitations in Baseline MRIG Approaches

Empirical analysis on OmniRef-Bench reveals a significant performance drop in open-source MRIG models as the number and heterogeneity of references increases. Mainstream models (Qwen-Image-Edit, BAGEL, DreamOmni2, OmniGen2) exhibit high fidelity with limited references but rapidly accumulate artifacts and semantic loss in complex settings.

Figure 4: Open-source models maintain quality with a few references but degrade severely with increasing type-mix and count; quantitative CLIP-I scores confirm the drop.

This degradation is primarily due to optimization bias toward simple cases and reward signal compression, making it challenging for models to learn compositional reasoning and integration from underrepresented, harder samples.

DyRef: Training Framework and Algorithmic Contributions

To close the gap with advanced closed-source baselines (e.g., Nano Banana Pro), the paper introduces the DyRef two-stage training architecture:

Stage I: Supervised Fine-Tuning (SFT)

Weaponsing established dual-stream diffusion architectures (e.g., Qwen-Image-Edit-2511 with MMDiT backbone), SFT is performed on a diverse multi-reference dataset using LoRA parameter-efficient tuning and flow-matching loss. This establishes baseline capabilities for MRIG under controlled supervision.

Stage II: Difficulty-aware Reinforcement Optimization

DyRef introduces two mechanisms fundamental to overcoming reward distribution collapse and sample imbalance:

• Difficulty-aware Advantage Reweighting (DAR): Based on empirically observed decrease in mean and median reward for high-complexity (many mixed-type reference) samples, DAR dynamically scales group loss contributions inversely with intra-group mean reward. This mechanism ensures that compositional, underperforming samples exert greater influence during RL, thereby directly mitigating underfitting of challenging cases.
• Discriminative Reward Scaling (DRS): Standard reward metrics (e.g., CLIP, SigLIPv2 cosine similarity) are often over-concentrated, failing to reflect qualitative variation. DRS employs reward transformation (e.g., sigmoid-based shaping) to amplify the discrimination between samples within a group, preserving the gradient signal required for effective policy improvement on fine-grained aspects (especially in style, background, and lighting).

  Figure 3: DyRef training framework, with SFT initialization, application of DRS to magnify reward contrast, and DAR to dynamically reweight sampling.

Empirical Results and Numerical Performance

OmniRef-Bench and Single-Image Editing

On OmniRef-Bench, DyRef-enhanced models (Qwen-2511 + DyRef, FLUX.2-klein + DyRef) deliver a substantial absolute increase over open-source baselines in both objective and MLLM scores. Notably:

• Qwen-2511 + DyRef exceeds baseline FLUX.2-klein by at least 12% (Objective) and 29% (MLLM), and achieves performance within margin of Nano Banana Pro, the leading closed-source model.
• On single-image editing benchmarks (ImgEdit, DreamBench++), DyRef further demonstrates generalization, matching or outperforming Seedream 4.5 and GPT Image 1 (closed-source) in overall user-aligned metrics.

  Figure 6: DyRef surpasses all open-source models and matches top closed-source models on diverse multi-reference and single image editing tasks.

Qualitative comparisons (Figure 7) show superior compositional integration and style transfer fidelity, particularly in high-type, high-count reference regimes.

Figure 5: Qualitative results on OmniRef-Bench; DyRef accurately aligns generated content with multiple, complex references, outperforming all baselines.

Ablation and User Studies

Ablation studies confirm that both CSD-weighted style rewards and the novel DAR/DRS mechanisms are essential for maintaining high objective and MLLM scores, especially for style and pose metrics. User studies with experts and fine-grained LLM-based evaluations yield high correlation coefficients (PLCC: 0.9) between automated and human ratings, evidencing the credibility of the evaluation pipeline.

Figure 7: Model performance on complex reference cases degrades significantly without DAR, as shown by reward and consistency score distributions.

Implications and Directions for Future Research

Practical Impact

DyRef demonstrates that open-source MRIG systems, with appropriate reward shaping and dynamic optimization, can achieve parity with the best closed-source solutions on compositionally challenging visual generation tasks. The generalizability across backbone architectures and editing regimes suggests applicability for downstream domains such as digital design, advertising, content creation, and simulation.

Theoretical and Methodological Advances

The integration of adaptive group-wise loss reweighting (DAR) and reward sharpening (DRS) constitutes a robust template for RL-based optimization in high-complexity generative tasks. The dual-strategy approach targets two orthogonal sources of degradation: optimization bias against hard samples, and collapsed reward signal.

Open Challenges and Future Work

• Explicit Feature Disentanglement: The paper notes residual limitations in reference leakage and instruction omission, especially with over- or under-utilization of specific reference types, indicating a need for fine-grained feature disentanglement or alignment mechanisms beyond reward engineering.
• Scaling and New Reference Types: The presented architecture is robust to increasing numbers and types of references, yet further investigation is required for unconstrained compositionality (e.g., additional modalities, fine-grained spatial constraints).
• Improved Evaluation: While hybrid objective–MLLM scoring addresses many limitations, further development of spatially explicit, semantically robust evaluation metrics remains an open problem for MRIG tasks.

Conclusion

This work establishes a new foundation for robust and general Multi-Reference Image Generation at scale. By developing OmniRef-Bench for fine-grained evaluation and introducing the DyRef framework with DAR and DRS, it demonstrates that open-source diffusion models can be reliably trained to handle arbitrary compositions of diverse references. These advances both narrow the open/closed-source performance gap and provide a methodological blueprint for future MRIG research.

Figure 1: DyRef generates high-quality results in complex multi-reference scenarios, consistently adhering to user instructions.