---
title: 'M2E-Try-On Net: Realistic Virtual Try-On'
url: https://www.emergentmind.com/topics/m2e-try-on-net
type: topic
---

# M2E-Try-On Net: Realistic Virtual Try-On

M2E-Try-On Net is a virtual try-on system designed to synthesize realistic images of a person ($P$) wearing garments from a different source “model” image ($M$), without requiring a clean, catalog-style product photograph of the clothing. Unlike prior paradigms that rely heavily on segmented or well-aligned product shots, M2E-Try-On Net operates directly on in-the-wild model images, addressing key challenges in pose alignment, texture preservation, and seamless compositing between heterogeneous identities. The method is structured as a sequential network pipeline, employs self-supervised learning from multi-pose datasets, and introduces adversarial and perceptual constraints tailored to the unpaired nature of fashion data [1811.08599].

## 1. Architectural Design and Subnetwork Workflow

M2E-Try-On Net decomposes the model-to-everyone try-on problem into three main stages:

1. **Pose Alignment Network (PAN):** Aligns the model garment $M$ to the desired pose of the person $P$ using DensePose-derived per-pixel UV correspondences ($M_D, P_D$). The model image $M$ is warped to match $P$ via barycentric interpolation across a body-part mesh, producing a coarse pose-warped image ($M'_W$). This, together with the original model and DensePose fields, is input to a conditional generator performing further feature encoding, residual transformation, and upsampling to yield an aligned model image $M'_A$. Training is adversarial, guided by a pose-conditional GAN loss:
   $$
   \mathcal{L}_{\mathrm{PGAN}} = \mathbb{E}_{P}[\log D(P,P_D)] + \mathbb{E}_{M,P}[\log(1 - D(M'_A,P_D))]
   $$
   where $D$ is the discriminator receiving both RGB and pose cues.

2. **Texture Refinement Network (TRN):** Directly warping and aligning the model image can blur garment textures. TRN addresses this by fusing the warped garment ($M'_W$) and the synthesized pose-aligned output ($M'_A$) using an automatically generated garment mask $R$. The initial composite $\hat M = M'_W \odot R + M'_A \odot (1 - R)$ is passed to a lightweight U-Net which refines garment boundaries, restores high-frequency details, and outputs the blended image $M'_R$.

3. **Fitting Network (FTN):** To synthesize the final try-on image $P'$ that preserves the person’s face, background, and identity, FTN composites the refined, re-posed garment $M'_R$ into $P$. A region-of-interest mask $P_{\mathrm{RoI}}$ is derived by fusing semantic segmentations, covering the upper body and arms. The masked regions $\hat{M'_R}, \hat{P}$ are concatenated and processed through an encoder–residual–decoder architecture to yield $P'$. The FTN is also trained adversarially and, in paired settings, with additional reconstruction and perceptual/style losses.

## 2. Training Paradigms: Self-supervised and Paired Strategies

Due to the absence of direct supervision $(M, P, P')$—i.e., triplets where a person is imaged both in new and previous garments—the method exploits two key training modalities:

- **Unpaired GAN Training:** Pairs of $(M,P)$ are randomly sampled, and only the pose-conditional GAN loss ($\mathcal{L}_{\mathrm{PGAN}}$) is imposed, fostering pose correctness and realism without access to ground-truth outputs.
- **Self-supervised Paired Training:** Many fashion datasets feature individuals photographed in the same outfit but with different poses. Such pairs $(P_{\mathrm{src}}, P_{\mathrm{tgt}})$ are treated as pseudo ground-truth, enabling supervised losses:
  - $\mathcal{L}_1 = \|P' - P\|_1$ and $\mathcal{L}_2 = \|P' - P\|_2$
  - Perceptual loss, derived from differences in features $\Phi_i$ extracted from a pre-trained VGG-16,
    $$
    \mathcal{L}_{\mathrm{perc}}(P',P) = \sum_{i=1}^n \|\Phi_i(P') - \Phi_i(P)\|_2^2
    $$
  - Style loss, based on Gram matrices $G_i$ computed from each feature layer,
    $$
    \mathcal{L}_{\mathrm{style}}(P',P) = \sum_{i=1}^n \|G_i(P') - G_i(P)\|_2^2
    $$
  - The total joint loss is a weighted combination of all components. The relative weights are set to emphasize adversarial and perceptual contributions, mitigating over-smoothing in the output.

## 3. Implementation Specifications

- **Image preprocessing:** All inputs are resized to $256 \times 256$ resolution.
- **Optimization:** Adam optimizer with $\beta_1=0.5, \beta_2=0.99$, and a learning rate of $2 \times 10^{-4}$ is used. The training schedule involves 80 epochs for PAN and 50 epochs each for TRN and FTN.
- **Subnetwork architectures:**
  - PAN/FTN employ standard encoder (three $3 \times 3$ conv stride-2 layers), 6–8 residual blocks, and decoder (two deconvs, one conv layer, tanh).
  - TRN is a compact U-Net with four downsampling/upsampling stages and skip connections.
- **RoI mask production:** RoI for compositing is computed by fusing LIP_SSF parsing and DensePose, and refined by an additional $\ell_1$-supervised CNN.

## 4. Experimental Evaluation

Experiments are conducted on the following datasets:
- DeepFashion Women Tops (mini): 3,256 unpaired, 7,764 paired, 4,064 test images.
- MVC Women Tops (mini): 2,606 unpaired, 4,000 paired, 1,498 test images.
- MVC Women Pants: 3,715 unpaired, 5,423 paired, 650 test images.

Standard quantitative metrics such as SSIM or Inception Score are not reported. Instead, performance is evaluated via a user study on randomly sampled test images from the mini-DeepFashion dataset, with preference rates compared against several baseline methods:

| Method     | User Preference (%) |
| ---------- | ------------------ |
| GAN-VT     | 0.8                |
| VITON      | 8.5                |
| CP-VTON    | 7.0                |
| M2E-Try-On | 83.7               |

Qualitative results demonstrate superior garment alignment to novel poses, better recovery of logo and texture detail, and more natural blending to the individual’s torso compared to these baselines [1811.08599].

## 5. Analysis of Capabilities, Limitations, and Prospective Enhancements

**Strengths:**  
- Uniquely, M2E-Try-On Net synthesizes try-on images directly from photoshoot-style or in-situ model images, obviating the requirement for isolated product shots.
- The joint self-supervised training technique capitalizes on existing multi-pose captures in fashion datasets, enhancing generalizability and reducing annotation overhead.
- Pose-conditional adversarial objectives enforce both output realism and accurate pose mapping.
- The two-stage strategy for texture refinement coupled with perceptual and style constraints maintains garment textural fidelity.

**Limitations:**  
- The pipeline can fail when complex garment graphics (such as face prints) are misclassified as body regions by parsing or pose estimation modules.
- Empirical validation is concentrated on upper-body clothing; lower garments and accessories are relatively unexplored.
- Reliance on user studies, rather than objective SSIM or LPIPS metrics, limits cross-comparison with other approaches.

**Paths for Improvement:**  
- Incorporation of more robust semantic parsing that can distinguish intricate garment patterns from skin or other body regions would increase reliability.
- Extension to lower-body attire by introducing additional masking and compositing logic for hips, legs, and full-body clothing.
- Enabling video-clip processing with temporal consistency mechanisms such as optical flow.
- Interactive refinement based on sparse user feedback, enabling error correction in extreme or unusual poses.

## 6. Relevance and Positioning in Virtual Try-On Research

M2E-Try-On Net advances the paradigm of virtual try-on by circumventing product image constraints and leveraging self-supervision in a three-stage network involving dense pose alignment, high-frequency texture compensation, and robust fitting. Its architecture represents a convergence of geometric warping, generative adversarial settings, and multi-task learning driven by both paired and unpaired supervision. These features position M2E-Try-On Net as a reference solution for generating photo-realistic personalized try-on imagery from unstructured model data, with implications for e-commerce, visual search, and personalized recommendation systems [1811.08599].

Source: https://www.emergentmind.com/topics/m2e-try-on-net