Stable-Layers: Fine-Tuning Image Layer Decomposition Models with VLM-Scored Reinforcement Learning

Abstract: We present Stable-Layers, a reinforcement learning framework that eliminates the need for paired supervision by fine-tuning a pretrained layer decomposition model using only feedback from a vision-LLM (VLM). Starting from Qwen-Image-Layered, we apply Flow-GRPO with LoRA adaptation, sampling multiple candidate decompositions per image, scoring them with a VLM, and optimising the policy from group-relative advantages. The key challenge lies in designing a reliable reward signal: VLMs scoring samples in isolation tend to compress their judgements into a narrow band, leaving GRPO with little within-group variance to learn from. We address this with a two-stage evaluation pipeline that pairs structured per-sample scoring across five edit-centric criteria with a grid-based calibration step in which the VLM re-scores all candidates side-by-side. Stable-Layers produces decompositions with stronger layer separation, fewer blank or artifact-heavy layers, and lower per-layer reconstruction error on the Crello dataset compared to the base model.

• The paper introduces a reinforcement learning paradigm that uses a vision-language model to score and improve image layer decompositions.
• It fine-tunes a flow-matching transformer using LoRA, resulting in enhanced semantic separation, improved alpha mask quality, and better background inpainting.
• The method outperforms baseline models on quantitative metrics by reducing bad layers and optimizing feature distribution and content validity.

Stable-Layers: Image Layer Decomposition Fine-Tuning via VLM-Scored Reinforcement Learning

Introduction

Image layer decomposition—mapping a raster image into a stack of editable RGBA layers whose recomposition reconstructs the input—is a critical operation for professional image editing, compositing, and content-aware manipulation. Supervised approaches typically rely on synthetic datasets with aligned decompositions or forced regression to a canonical layer arrangement, which fundamentally underconstrains the ill-posed problem for natural images. The absence of paired supervision and the inherently multi-modal structure of valid decompositions diminish the practical relevance of pointwise regression objectives, biasing models toward trivial or artifact-prone solutions.

"Stable-Layers: Fine-Tuning Image Layer Decomposition Models with VLM-Scored Reinforcement Learning" (2605.30257) addresses this with a reinforcement learning (RL) paradigm that eschews dependence on paired targets. Instead, a strong vision-LLM (VLM) acts as the sole supervisory signal, providing structured evaluations on the perceptual quality of decompositions sampled from a backbone model. The method leverages Flow-GRPO—an SDE-injected extension of flow-matching generative models with group-relative policy optimization—and introduces a two-phase evaluation pipeline to overcome VLM reward collapse. Fine-tuning with this protocol produces decompositions exhibiting improved semantic separation, alpha mask quality, feature distribution, and plausible background inpainting, supported by both quantitative and qualitative metrics.

Methodology

Backbone and Adaptation

The base model, Qwen-Image-Layered [yin2025qwen], is a flow-matching transformer architecture capable of producing variable-length RGBA layer stacks from a single image. Images are encoded by a 3D VAE, yielding compressed latents processed by a token-based transformer, with text conditioning for prompts. Fine-tuning is performed exclusively via Low-Rank Adaptation (LoRA) on attention and feed-forward layers, leaving the backbone weights fixed for efficient adaptation.

RL with Flow-GRPO and GRPO-Guard

Policy optimization employs Flow-GRPO [flowgrpo2025], which replaces standard diffusive ODE sampling with a tractable SDE variant, exposing exact per-step log-probabilities necessary for model-based RL. Candidate decompositions are generated by SDE rollouts, grouped, and scored; GRPO computes relative advantages within groups, forming the surrogate loss for LoRA update steps. GRPO-Guard [grpoguard2025] is used to stabilize loss signals by normalizing per-step importance ratios (RatioNorm) and equalizing gradient magnitudes (gradient reweighting).

The novel insight for stable RL here is the handling of high-dimensional packed latents: as the model's latent sequences encode up to 5 RGBA layers per sample, naive spatial averaging of log-likelihoods artificially squashes ratio variance. The proposed modification sums log-probabilities spatially and rescales by $\sqrt{D}$, restoring numerically meaningful gradients.

Two-Phase Structured VLM Reward Protocol

A central technical challenge is the design of a VLM-derived reward signal sufficiently informative for within-group discrimination without paired ground truth. The proposed two-stage protocol is as follows:

Phase 1: Structured Rubric Scoring—The VLM evaluates each sample decomposition on five axes (semantic separation, alpha cleanliness, background inpainting, feature distribution, content validity) using explicit anchor descriptions. Each criterion is scored $0$–$5$ and summed for a normalized total. This anchors categorical perceptual failures.

Phase 2: Grid-Based Relative Calibration—When Phase 1 scores are compressed by similar candidate quality, all group candidates are tiled into a labeled grid and resubmitted to the VLM. The model re-scores each candidate, conditioning on the first-stage scores, enforcing discriminative spread within the group even under minimal absolute differences.

The final reward for policy improvement is the Phase 2 calibrated value, which empirical ablation shows is critical for learning fine-grained perceptual distinctions.

Experimental Results

Reward Signal Dynamics

RL training on Fine-T2I images (unlabeled, $640 \times 640$) quickly eliminates coarse failure modes, with calibrated VLM rewards increasing from $\sim$0.70 to steady-state values around $0.83$–$0.85$ over the first 100 steps. Plateaus in absolute reward are theoretically anticipated under normalized group-relative objectives: within-group variance, not global improvement, is required for learning progress.

Figure 2: The calibrated VLM reward rises during early training then plateaus; residual reward variance is attributable to input difficulty, not policy drift.

Quantitative Metrics

Evaluation on held-out LAION-Aesthetics images with automated metrics reveals pronounced improvement across all axes:

• Bad layer count per decomposition drops from $\sim$1.65 to $\sim$0.4.
• Feature distribution evenness (degree to which content is meaningfully partitioned among layers) rises from $\sim$0.53 to $0$00.73.
• Layer 0 inpainting quality rises from $0$10.38 to $0$20.62.

  Figure 4: Held-out evaluation metrics over training: reduction in blank/artifact layers, improved content distribution, and background quality.

Quantitative comparison using modified per-layer RGB L1 on the Crello dataset demonstrates that the stable-layers model outperforms the backbone on mean errors for all layer counts, reflecting its improved content assignment and background reconstruction.

Qualitative Decomposition

Qualitative analysis illustrates that Stable-Layers shifts the base model from degenerate (blank or trivially inpainted) layer 0s and high inter-slot content duplication to highly plausible background reconstruction, clean semantic foreground separation, and distinct, crisp alpha masks.

Figure 3: Stable-Layers vs. base model on held-out images: superior background inpainting and semantic foreground allocation post-fine-tuning.

Figure 5: Diverse gallery evidencing consistent decomposition across variable subject matter—foreground isolation, credible inpainting, balanced feature separation.

Ablations

Grid Calibration: Training without the grid calibration phase results in flatter learning curves for fine-grained metrics (e.g., Layer 0 quality, sharpness, SSIM). Grid-based relative calibration distinctly closes the gap on perceptual metrics where Phase 1 absolute scoring fails to offer learning signal due to low group-wise variance.

Figure 6: Grid calibration consistently improves qualitative Layer 0 metrics (quality and sharpness) versus ablation.

Text Conditioning: Using detailed prompts for conditioning, even when aligned with the reward rubric, yields inferior empirical results—the model benefits from general prompts and learns decomposition priors from VLM feedback, rather than being directly guided by prompt structure.

Implications and Future Directions

Stable-Layers demonstrates that RL fine-tuning with post-hoc VLM judge signals is a viable, effective approach to edit-oriented generative modeling in the absence of paired decompositions. This protocol both sidesteps the ill-posedness inherent in synthetic regression pipelines and yields decompositions with enhanced downstream usability (editability, semantic isolation). The approach generalizes to any generator whose outputs can be reliably assessed by VLMs (text-to-image, scene manipulation, relighting, etc.).

Key future extensions include:

• Replacement of numeric calibration with logit-based relative probing (pairwise or ranking queries), mitigating residual calibration deficits in text-based VLM interfaces.
• Automated rubric refinement using meta-critique capabilities of VLMs for dynamic reward model adaptation.
• Higher-layer-count decomposition beyond the training regime, facilitated by scalable compute or sequence-efficient architectures.

The reliance on proprietary VLMs as the reward model remains a practical limitation for reproducibility and process cost, as does the absence of human-in-the-loop validation of the subjective metrics. However, the framework sets a new technical direction for leveraging pre-trained, general-purpose VLM critic models to drive self-improving, judge-supervised generative pipelines.

Conclusion

Stable-Layers establishes an RL paradigm wherein VLM-authored structured, multi-criteria feedback is sufficient to drive meaningful improvements in layer decomposition models, closing the gap between synthesized and human-interpretable editability. The deployment of two-phase reward extraction with relative group calibration is critical for effective, fine-grained credit assignment. The methodology signals an important transition toward VLM-centric self-improving editing models, with applicability to a spectrum of vision tasks wherein explicit ground truth is inherently unavailable.