Seedream 3.0 Fill: Unified Mask-Guided Editing

• Seedream 3.0 Fill is a unified mask-guided image generation model that employs multi-task reinforcement learning from human feedback to excel in fill, extend, removal, and text rendering tasks.
• The model uses a diffusion/flow VAE with a U-Net architecture and mask conditioning to support flexible spatial editing and prompt-based cross-attention.
• Reinforcement learning driven by a frozen vision–language reward model yields superior usability and performance, outperforming both commercial and open-source systems.

Seedream 3.0 Fill is a unified mask-guided image generation model trained via multi-task reinforcement learning from human preferences. Based on the OneReward framework, it supports image fill, image extend (outpainting), object removal, and text rendering, and outperforms both commercial and open-source systems across human-evaluated and task-specific metrics. Unlike prior art, which typically uses task-specific supervised fine-tuning (SFT), Seedream 3.0 Fill enables generalization and training efficiency by leveraging a frozen, single vision–language reward model that evaluates candidate outputs according to multidimensional criteria in a reinforcement learning loop (Gong et al., 28 Aug 2025).

1. Model Architecture and Conditioning

Seedream 3.0 Fill operates via a mask-guided, multi-modal pipeline:

• Input: A source image $I_{\text{src}} \in \mathbb{R}^{H\times W\times 3}$, binary edit mask $M \in \{0,1\}^{H\times W}$ specifying the editable region, and (optionally) a text prompt $P$ for tasks requiring prompt alignment.
• Encoding: The image and mask are encoded via a diffusion/flow VAE into a latent representation $z_T$, with the mask concatenated as an extra channel. The prompt $P$ is mapped to CLIP-style embeddings $T$ via a frozen text encoder.
• Network Core: The main network is based on the Seedream 3.0 rectified-flow (flow-matching) model, employing a standard U-Net with 40 ResBlocks (base channel 320, doubling upon downsampling, peaking at 1280 channels).
• Conditioning at Each Block: Each ResBlock receives the noisy latent $x_t$, a sinusoidal positional time embedding $t$, the mask $M$ (downsampled as needed), and—where relevant—cross-attention to $T$.
• Partial Denoising: During RL training, the policy model $M \in \{0,1\}^{H\times W}$0 performs partial denoising starting from a random timestep $M \in \{0,1\}^{H\times W}$1 (default $M \in \{0,1\}^{H\times W}$2, $M \in \{0,1\}^{H\times W}$3, $M \in \{0,1\}^{H\times W}$4); the reference model $M \in \{0,1\}^{H\times W}$5 always performs full denoising.
• Decoding: Both predicted latents ($M \in \{0,1\}^{H\times W}$6 from the policy and $M \in \{0,1\}^{H\times W}$7 from the reference) are mapped back to image space by the shared VAE decoder.

Mask conditioning is handled at all U-Net scales, allowing for arbitrary spatial configurations including border-filling and irregular or free-form holes. Cross-attention to the prompt embedding is included only for tasks requiring prompt alignment.

2. Reinforcement Learning and Reward Formalism

The image-generation task is cast as a sequential decision process. At each episode, the policy $M \in \{0,1\}^{H\times W}$8 generates a candidate edit $M \in \{0,1\}^{H\times W}$9 given state $P$0. The reward is computed by the OneReward vision–LLM (VLM), which compares the candidate $P$1 to the reference output $P$2 under specific task-dimension queries.

• Policy: $P$3, realized by a single-step denoising procedure from a partially noisy latent.
• Reward: $P$4, where $P$5 indexes parameters of the fixed reward VLM, $P$6 is the natural-language query about task $P$7 and evaluation criterion $P$8.
• Objective: Learning maximizes the expected margin-based reward:

$P$9

with expectation taken over tasks, dataset samples, and partial denoising steps.

No KL-constraint or PPO-style clipping is employed; instead, a simple hinge margin loss discourages reward hacking. The reference model $z_T$0 may be kept static or updated as an exponential moving average (EMA) of $z_T$1 (with $z_T$2).

3. OneReward: Vision–Language Reward Model

All training and evaluation leverages a unified VLM-based reward:

• Architecture: The reward model uses Qwen2.5-VL-7B-Instruct, a 7B-parameter VLM.
• Input: Pairs of RGB images $z_T$3 and a query $z_T$4 "In task {fill/extend/remove} under dimension {Structure/Consistency/...}, which image is better? First or second?".
• Output: Binary classification over tokens {Yes, No}; $z_T$5 (for "first" better) is used directly as the reward.
• Training: The VLM is trained on $z_T$6200k human-annotated winner/loser pairs across four tasks and up to four evaluation axes per task. Annotations follow Best-of-N and Worst-of-N among four candidates, per metric. Loss uses multi-label cross-entropy:

$z_T$7

• Task Dimensions: For image-fill, queries include (but are not limited to) Structure, Consistency, Aesthetics, Text Alignment, and Removal Quality.

4. Training Regimen and Data

The RL training incorporates diverse instances drawn from wide datasets and tasks:

• Task Distribution: Each RL step samples one of three tasks: Image Fill (50%; with scribble/random masks, detailed text prompts), Image Extend (25%; border masks, prompts present/absent 50/50), Object Removal (25%; masks correspond to objects, prompt "remove the specified object"). Source data includes LAION, COCO, ADE20K, and open photo collections.
• Batching: Each batch (size 8) reflects expected task proportions (e.g. 4 fill, 2 extend, 2 removal).
• Hyperparameters: Policy learning rate 1e-5, AdamW optimizer; batch size 8. The reward model is frozen during RL (previously trained at LR 1e-6, batch 16, 50k steps).
• Reference Update: The static and EMA-based (dynamic) reference strategies are both supported.

A purely SFT variant (L2 loss in latent space, same data) underperforms by 8–12% on usability metrics for fill and extend tasks, demonstrating the benefit of RL with direct reward maximization over SFT.

5. Evaluation, Quantitative Performance, and Failure Modes

Evaluation is performed both quantitatively and qualitatively with human-in-the-loop assessment. Main metrics include usability rate, MOS (1–5) for key dimensions, text rendering success rate, and removal quality.

Table 1: Selected Human-Evaluation Metrics Across Tasks

Task / Model	Usability ↑	Structure ↑	TextAlign ↑	TextRender ↑	RemovalQuality ↑
Image Fill
– Adobe Photoshop	45.22%	2.89	4.09	26.69%	–
– Ideogram	50.65%	3.80	3.78	24.81%	–
– Flux Fill [pro]	50.97%	3.46	4.12	29.32%	–
– Seedream 3.0 Fill	69.04%	4.02	4.57	70.68%	–
Image Extend (w/prompt)	64.72%	4.05	4.26	–	–
Image Extend (w/o prompt)	87.54%	4.19	–	–	–
Object Removal	82.22%	4.32	–	–	86.33%

Seedream 3.0 Fill consistently exceeds competitors in usability and metric-averaged performance. Usability for image fill and object removal is higher than all other reported models, and text rendering success for fill tasks is $z_T$8, compared to $z_T$9 for Flux Fill [pro], $P$0 for Adobe Photoshop, and $P$1 for Ideogram. This suggests significant superiority in text-driven fill and compositional consistency.

Good–Same–Bad (GSB) analysis finds that the OneReward-trained variant is categorized "Good" more often than the base Seedream 3.0 across all tasks. In qualitative examples, Seedream 3.0 Fill produces sharper structures, better alignment to prompts, and cleaner object removal. However, style consistency remains a relative weakness (MOS $P$23.7), and minor mask-edge artifacts or text misalignments can occur with detailed backgrounds and complex fonts.

6. Ablations and Analytical Insights

Several ablations and analyses illuminate the impact of design decisions:

• Dynamic vs. Static Reference: Applying an EMA (dynamic) update to $P$3 yields smoother reward learning curves and a small ($P$41–2%) boost in usability, without higher memory cost.
• Reward-Dimension Exclusion: Omitting any single evaluation criterion during RL training reduces the corresponding test-time metric by 2–4%, affirming the necessity of full multi-dimensional reward supervision.
• RL vs. SFT: As above, SFT underperforms by 8–12% on usability versus the direct RL approach.
• Failure Cases: Mask edges sometimes show discontinuities on highly textured backgrounds. Text rendering may misalign strokes on complex fonts. Style consistency lags behind other quality axes.

7. Conclusions and Prospective Directions

Seedream 3.0 Fill constitutes the first end-to-end unified mask-guided generator trained via multi-task RL from human feedback, eschewing task-specific SFT. Its core enablers are the OneReward VLM, which generalizes reward assessment across diverse tasks and dimensions with a single head, and the architectural flexibility to support arbitrary mask/prompt conditioning within a single model instance.

Planned research extensions include:

• Incorporating additional editing classes, such as color grading and geometric transformations, through new task IDs and reward queries.
• Enhancing style consistency, potentially by introducing animated benchmarks or integrating pretrained style encoders into OneReward.
• Investigating advanced RL objectives such as KL-regularized policy updates or PPO variants for improved stability.

Seedream 3.0 Fill and the OneReward framework establish new benchmarks for generalizable, mask-conditioned image editing and set a precedent for future unified, RL-driven generative editing systems (Gong et al., 28 Aug 2025).