OneReward: Unified Mask-Guided Image Generation

• OneReward is a unified framework for multi-task, mask-guided image generation that replaces task-specific supervised fine-tuning with a single RLHF approach.
• It employs reinforcement learning where a policy network is optimized to outperform a frozen reference model using a unified vision-language reward model across diverse editing tasks.
• Empirical benchmarks show state-of-the-art performance in inpainting, outpainting, object removal, and text rendering, with significant gains over existing commercial and open-source systems.

OneReward is a unified framework for multi-task, mask-guided image generation via reinforcement learning from human feedback (RLHF), utilizing a single vision-language reward model to enable a generative network to excel across heterogeneous image editing objectives. It replaces the prevalent paradigm of task-specific supervised fine-tuning (SFT) with a single multi-task reward signal, enabling robust generalization and high efficiency across inpainting, outpainting, object removal, and text rendering tasks. This approach demonstrates state-of-the-art performance compared to both commercial and open-source alternatives, as established by comprehensive empirical benchmarks (Gong et al., 28 Aug 2025).

1. Unification of Multi-Task Image Generation

OneReward formalizes mask-guided image editing as a reinforcement learning problem in which a single policy network $\pi_\theta$ is optimized to outperform or “beat” a frozen reference model $\pi_\mathrm{ref}$ under a universal human-aligned reward signal. Each generation task—image fill (inpainting), image extend (outpainting), object removal, and mask-constrained text rendering—shares a conditioning triplet $c = (I_\mathrm{src}, M, P)$, where $I_\mathrm{src}$ is the source image, $M$ is a binary mask, and $P$ is an optional textual prompt describing user intent.

Although all subtasks are unified at the conditioning interface, each is characterized by distinct data distributions and evaluation metrics. For example, inpainting and outpainting require metrics such as structure preservation, prompt fidelity, and aesthetics, while object removal fundamentally relies on the removal quality metric. OneReward’s key innovation lies in employing a single, powerful vision-LLM (VLM) as a reward model, $r_\phi$, that can judge across all these axes by conditioning on task-specific queries and evaluation dimensions (Gong et al., 28 Aug 2025).

2. Architecture and RL Formulation

The general learning objective for the policy $\pi_\theta$ is defined as: $$\operatorname{maximize}\; \mathbb{E}_{c\sim\mathcal{D}} \Bigl[ R(c, \pi_\theta(c), \pi_\mathrm{ref}(c)) \Bigr]$$ where $R(\cdot)$ is a reward derived from the VLM. At each RL iteration, $\pi_\mathrm{ref}$0 (frozen) generates a baseline output $\pi_\mathrm{ref}$1 via complete denoising. In contrast, $\pi_\mathrm{ref}$2 generates a candidate $\pi_\mathrm{ref}$3 by partial denoising and direct latent prediction. Both outputs are decoded and jointly presented to the VLM reward model with a query encoding task type and evaluation criterion (e.g., “Structure Coherence for Outpainting”).

The VLM outputs a binary judgment: “Yes” if $\pi_\mathrm{ref}$4 surpasses $\pi_\mathrm{ref}$5 under the criterion $\pi_\mathrm{ref}$6, “No” otherwise. The probability of “Yes” constitutes the RL reward: $\pi_\mathrm{ref}$7 The model is optimized using a margin loss: $\pi_\mathrm{ref}$8 where $\pi_\mathrm{ref}$9 is a cap for reward hacking mitigation.

3. Training Procedure

OneReward’s RL training alternates over tasks (inpainting, outpainting, removal, text rendering) according to a sampling probability vector $c = (I_\mathrm{src}, M, P)$0. Each training step samples conditions, generates reference and candidate outputs (using independent noise seeds for stochasticity), and evaluates across all relevant metrics for the sampled task.

Unlike previous approaches, OneReward forgoes any task-specific SFT. It starts from a pre-trained Seedream 3.0 base and directly applies multi-task RLHF using the VLM reward. The policy and reward model utilize distinct learning rates (1e-5 and 1e-6, respectively). An exponential-moving-average (EMA) copy of $c = (I_\mathrm{src}, M, P)$1 may serve as the dynamic or static reference, with dynamic reference yielding lower memory usage and often matching or improving quality. Batch sizes are chosen as 8 for policy and 16 for reward training (Gong et al., 28 Aug 2025).

4. Empirical Performance and Evaluation

Comprehensive benchmarking demonstrates that OneReward-trained Seedream 3.0 Fill outperforms commercial (Ideogram, Adobe Photoshop) and open-source (FLUX Fill [Pro], Higgsfield) systems across all subtasks and objective metrics. Specific findings:

Subtask	OneReward Usability (%)	Best Prior Usability (%)
Inpainting	69.0	Higgsfield: 52.1
Text-guided Outpainting	64.7	Ideogram: 63.1
Text-free Outpainting	87.5	Ideogram: 73.7
Object Removal	82.2	Photoshop: 73.9

Mean opinion scores on structure, consistency, and text alignment improve by 0.2–0.5 points. Qualitative analyses reveal crisper text, sharper edges, and more coherent textures. Reward curves show stable and monotonic gains across all evaluation axes. Ablation studies indicate that dynamic EMA references often reduce memory usage while achieving comparable or better results (Gong et al., 28 Aug 2025).

5. Mask-Guided Generation Task Specification

The following table details the four primary mask-guided generation tasks with their corresponding metrics and inputs:

Task	Conditioning	Primary Evaluation Metrics
Inpainting	$c = (I_\mathrm{src}, M, P)$2	Structure, Text Align., Consistency, Aesth., Text Render.
Outpainting	$c = (I_\mathrm{src}, M, P)$3	Same as inpainting
Object Removal	$c = (I_\mathrm{src}, M, P)$4	Removal Quality
Text Rendering	$c = (I_\mathrm{src}, M, P)$5	Typographic Fidelity

The VLM-based reward model is able to evaluate any of these objectives via appropriate specification of the “query” argument, rendering the reward model universally reusable across editing types (Gong et al., 28 Aug 2025).

6. Significance, Limitations, and Future Directions

OneReward provides a principled approach for eliminating the need for task-specific SFT or multiple reward models in complex, multi-task conditional image generation. It enables a single generative policy network to generalize fluidly across inpainting, outpainting, text rendering, and object removal via a unified training signal that is both sample- and task-efficient.

The reported advances are robust across diverse evaluation metrics and competitors. However, current evaluation axes remain limited to structure, consistency, text alignment, aesthetics, and removal quality. Future work may introduce richer evaluation criteria (e.g., color harmony, semantic depth), expand to sequential domains (e.g., text-to-video), incorporate learned value functions for convergence acceleration, and develop curriculum or self-play protocols for generalization beyond observed edit distributions (Gong et al., 28 Aug 2025).

A plausible implication is that OneReward’s design foreshadows a broader trend of deploying multi-purpose, instruction-tunable reward models within vision-language RL frameworks, minimizing the need for extensive human curation and per-task engineering without sacrificing generality or performance.