---
title: 'GO-MLVTON: Diffusion-Based Multi-Layer VTON'
url: https://www.emergentmind.com/topics/go-mlvton
type: topic
---

# GO-MLVTON: Diffusion-Based Multi-Layer VTON

GO-MLVTON (Garment Occlusion-Aware Multi-Layer Virtual Try-On with Diffusion Models) is a methodology for multi-layer garment virtual try-on (ML-VTON) that jointly addresses the challenges of garment occlusion modeling, spatial deformation, and layer-to-layer visual coherence. By explicitly learning pixel-level occlusion relationships between inner and outer garments, GO-MLVTON produces artifact-free, realistic multi-layer try-on results, addressing limitations in prior single-layer or multi-garment VTON approaches. The system integrates a dedicated Garment Occlusion Learning module, a StableDiffusion-based morphing and fitting module, and is benchmarked on the newly introduced MLG dataset with a layer- and edge-sensitive evaluation metric, LACD [2601.13524].

## 1. Multi-Layer Virtual Try-On and the Challenge of Occlusion

Image-based virtual try-on (VTON) research has traditionally focused on either single-garment try-on (SG-VTON), synthesizing one target garment onto a person, or multi-garment (MG-VTON), compositing non-overlapping garments without enforcing physical overlap or occlusion. Real-world usage, however, typically involves dressing multiple, sometimes overlapping, layers (e.g., T-shirt under a jacket), where physical realism demands:

- Accurate spatial occlusion—outer garments must mask overlapping inner garment regions.
- Realistic deformation—proper draping and fitting of both inner and outer layers to the target body and pose.
- Artifact-free compositing—elimination of “bleed-through” or ghosting from occluded inner garment pixels.

Previous methods lack explicit pixel-level occlusion reasoning or the ability to deform and layer garments in a physically plausible manner. GO-MLVTON directly addresses these by introducing two interlinked components: the Garment Occlusion Learning (GOL) module and the Garment Morphing & Fitting (GMF) module built atop a diffusion model.

## 2. Garment Occlusion Learning (GOL) Module

The GOL module is designed to infer a spatially-varying attention mask that determines which pixels of an inner garment remain visible after layering with an outer garment. The process involves:

- **Dual-encoder feature extraction:** Separate encoders $E_i$ and $E_o$, each with five layers (composed of downsample convolutions followed by residual blocks), process the inner ($g_i$) and outer ($g_o$) garment images to yield feature maps $f_i$ and $f_o$.
- **Mapping to occlusion attention:** Channel-concatenated features $[f_o\Vert f_i]$ are passed through a mapping network $U$ and a final $1\times 1$ convolution, followed by a sigmoid nonlinearity, producing the occlusion mask $A \in [0,1]^{1 \times H' \times W'}$.
- **Occlusion-weighted inner garment latent:** This attention mask is applied element-wise to the latent representation of the inner garment ($z_i$ from the shared VAE encoder), yielding $z_{iv} = A \odot z_i$.

**Loss function:** A reconstruction loss enforces that $z_{iv}$ matches the visible inner garment as cropped from the real person image $x_{pi}$ via
$$
\mathcal{L}_{\rm OCC} = \left\| \varepsilon(x_{pi}) - z_{iv} \right\|_2^2,
$$
with $\lambda_2 = 0.1$ in the overall objective. This direct supervision ensures the GOL module learns to suppress occluded regions effectively. Backpropagating $\mathcal{L}_{\rm OCC}$ leads to optimal occlusion-aware masking, especially at garment boundaries.

## 3. StableDiffusion-Based Garment Morphing & Fitting (GMF) Module

Following occlusion reasoning, the GMF module synthesizes the final try-on output, adapting diffusion modeling to multi-layer conditioning:

- **Latent conditioning:** The core conditional input is the concatenation $[z_a \Vert z_o \Vert z_{iv}]$ where $z_a$ is the clothing-agnostic person latent (body pose/shape), $z_o$ is the outer garment latent, and $z_{iv}$ is the occlusion-weighted inner garment latent.
- **Diffusion UNet:** A StableDiffusion v1.5 UNet, with cross-attention layers removed for architectural alignment with ML-VTON tasks, jointly models the diffusion denoising process conditioned on these latents as well as a binary inpainting mask for masked person areas.
- **Training objective:** The denoising loss per diffusion step $t$ is
$$
\mathcal{L}_{\rm GMF} = \mathbb{E}_{t, \epsilon} \left\| \epsilon - \epsilon_\theta(z^t, t, z^t_{\rm in}, m_{\rm in}) \right\|_2^2,
$$
and the combined objective is
$$
\mathcal{L} = \mathcal{L}_{\rm GMF} + \lambda_2\mathcal{L}_{\rm OCC}.
$$
- **Optimization and inference:** Model initialization starts from InstructPix2Pix-pretrained StableDiffusion weights; most weights remain frozen except for GOL and self-attention layers. AdamW is used with learning rate $1\times10^{-5}$, batch size $8$, and $10\%$ conditional dropout. Inference incorporates classifier-free guidance (CFG) with scale $s=2.5$ for optimal fidelity/diversity balance.

## 4. Multi-Layer Garment (MLG) Dataset

MLG is a dataset specifically constructed for the ML-VTON task, supporting robust training and evaluation:

- **Composition:** 3,538 samples, split into 2,783 training and 755 test quadruplets $(g_i, g_o, x_p, x_a)$.
  - $g_i$: inner garment product shot
  - $g_o$: outer garment product shot
  - $x_p$: person image wearing both garments
  - $x_a$: clothing-agnostic image ($x_p$ with upper-body region masked using SCHP parsing)
- **Garment and pose diversity:** Categories include T-shirts, vests, cardigans, jackets, dresses, and coats; poses cover standing, walking, and dynamic actions in varied environments.
- **Annotation schema:** Product crops obtained or segmented via SAM; person parsing via SCHP provides upper-body mask $M$; for each layer $i$, pixelwise masks $A_i$, boundary bands $B_i$, and interiors $C_i$ are defined for metric evaluation.

## 5. Evaluation Metric: Layered Appearance Coherence Difference (LACD)

Standard metrics such as FID, SSIM, and LPIPS provide limited sensitivity to inter-layer artifacts and occlusion errors. LACD is introduced to address these limitations by directly measuring per-layer and seam consistency:

- **Definitions:**
  - For each garment layer $i$, $A_i$ is its pixel region, $B_i$ is the seam band adjoining layer $i+1$, and $C_i = A_i\setminus B_i$ is the interior.
- **Layer discrepancy:**
  $$
  \ell_i = \lambda_1 \sum_{j\in B_i} \left\| x_{gt}^{(i,j)} - x_{gen}^{(i,j)} \right\|_2 + \sum_{k\in C_i} \left\| x_{gt}^{(i,k)} - x_{gen}^{(i,k)} \right\|_2,
  $$
  where $\lambda_1 = 3$ upweights seam artifacts.
- **Overall metric:**
  $$
  \mathrm{LACD} = \frac{1}{N} \sum_{i=1}^N \ell_i
  $$
- **Interpretation:** The LACD emphasizes seam fidelity and inter-layer coherence, penalizing unrealistic edge transitions and “leakage” of occluded layers, making it highly discriminative for ML-VTON scenarios.

## 6. Experimental Results and Ablations

Extensive experiments on MLG benchmark the quantitative and qualitative advantages of GO-MLVTON:

**Quantitative Comparison:**
| Method              | FID ↓   | KID ↓   | SSIM ↑  | LPIPS ↓ | LACD ↓  |
|---------------------|---------|---------|---------|---------|---------|
| CAT-DM (CVPR’24)    | 32.36   | 5.97    | 0.845   | 0.138   | 0.719   |
| MV-VTON (AAAI’25)   | 36.31   | 7.60    | 0.830   | 0.175   | 0.973   |
| CATVTON (ICLR’25)   | 30.54   | 4.61    | 0.841   | 0.127   | 0.626   |
| GO-MLVTON           | 22.82   | 0.35    | 0.858   | 0.108   | 0.623   |

GO-MLVTON outperforms all baselines, achieving significant improvements (e.g., a 7.7-point drop in FID, 4.26 lower KID versus the best baseline, and consistent gains in SSIM, LPIPS, and LACD).

**GOL Module Ablation:**
Ablation studies reveal that the GOL module alone (without $\mathcal{L}_{\rm OCC}$) may leave occluded inner-layer details visible. Full supervision with $\mathcal{L}_{\rm OCC}$ is necessary for optimal suppression and minimal seam and LACD errors.

**Classifier-Free Guidance (CFG) Hyperparameter:**
The optimal CFG scale for the StableDiffusion UNet is $s=2.5$, producing the best accuracy/diversity trade-off.

**Qualitative Assessment:**
Visual results confirm the elimination of inner garment bleed-through, preservation of garment boundary sharpness, and physically coherent layer interaction and deformation.

## 7. GO-MLVTON Pipeline Summary

A high-level pseudocode outline of the GO-MLVTON process is as follows:
```
Input:
  x_p     ← person image wearing both garments
  g_i, g_o ← product-shot images of inner & outer garments
  M       ← upper-body mask from SCHP

1. Build agnostic input:
   x_a ← x_p ⊙ (1 - M)

2. VAE encode:
   z_a ← ε(x_a)
   z_i ← ε(g_i)
   z_o ← ε(g_o)

3. Garment Occlusion Learning (GOL):
   f_i ← E_i(g_i)
   f_o ← E_o(g_o)
   A   ← sigmoid( Linear( U( [f_o‖f_i] ) ) )
   z_iv ← A ⊙ z_i

4. Prepare diffusion input:
   z_in ← [ z_a ‖ z_o ‖ z_iv ]
   m_in ← [ M ‖ 0 ‖ 0 ]
   ε_in ← sample Gaussian noise

5. StableDiffusion-based Garment Morphing & Fitting (GMF):
   for t = T…1 do
     predict   ε_θ ← UNet( z^t , t | z_in, m_in )
     compute    L_GMF += || ε_actual - ε_θ ||^2
     z^{t-1}   ← denoise_step(z^t, ε_θ)
   end for

6. Decode output:
   x_g ← D( z^0 )

7. Total loss (train):
   L_total = L_GMF + 0.1 × L_OCC
```

GO-MLVTON thus establishes an effective paradigm for multi-layer virtual try-on, innovating across occlusion modeling, diffusion-based compositing, and dedicated evaluation for fashion-oriented generative vision tasks [2601.13524].

Source: https://www.emergentmind.com/topics/go-mlvton