Image-based Virtual Try-On

• Image-based Virtual Try-On is a technique that synthesizes realistic images of a person wearing a garment using only 2D inputs to preserve identity, pose, and context.
• Recent methods utilize warping, GANs, and diffusion models in coarse-to-fine, warping-then-blend, and end-to-end pipelines to enhance detail and alignment.
• Current research addresses challenges like occlusion, misalignment, and semantic detail loss while advancing mask-free, training-free, and universally adaptable VTON systems.

Image-based Virtual Try-On (VTON) refers to a class of computer vision methods for synthesizing photorealistic images of a target person wearing a desired garment, using as input only 2D images of both the person and the clothing. The core goal is to ensure that the try-on result preserves the person’s identity, body shape, pose, and background, while accurately transferring the shape and appearance of the selected garment—without recourse to physical 3D body or clothing scanning. This field has seen rapid advances in warping-based, generative adversarial, and diffusion-model-driven methods, now supporting use cases from fashion e-commerce and styling to AR/VR applications.

1. Foundational Pipeline Architectures and Person Representations

VTON pipelines exhibit diverse architectural philosophies, all aiming to bridge the signal gap between an input “model” (person) image and a shop-catalog garment image. Common design motifs include:

• Person Representation Construction: Early methods define “clothing-agnostic” person representations by removing or masking the original garment pixels while retaining body shape and pose cues. Typical components are an 18-channel pose keypoint map $P$, a binary body mask $B$, and a 3-channel cropped head $H$. These are concatenated along the channel axis:

$$x_{\text{person}} = [P; B; H] \in \mathbb{R}^{h \times w \times 22}$$

This baseline—introduced and formalized by "VITON: An Image-based Virtual Try-on Network" (Han et al., 2017)—remains widely used, with variants incorporating DensePose, fine-grained parsing, or silhouette channels.

• Pipeline Typology: Pipelines are often classifiable as:
  1. Two-stage (Coarse-to-Fine): A coarse synthesis generator predicts the global structure, followed by a refinement network for photo-realistic garment details and mask-based compositing. This is exemplified by encoder-decoder U-Nets with skip connections and VGG-based perceptual losses (Han et al., 2017).
  2. Warp-then-Blend: Garment warping (e.g., via TPS/flow fields) aligns the clothing to body shape and pose before fusing with the person representation, usually via another generator.
  3. Direct or End-to-End: Fully convolutional models with skip/attention schemes synthesize the dressed person in a single stage, either by latent fusion (Song et al., 2023), spatio-channel cross-attention, or by leveraging strong global guidance from large pretrained diffusion or multimodal models (Zhang et al., 19 Nov 2025, Yang et al., 20 Jul 2025).

Person/garment alignment may be posed as a joint or cascaded problem; explicit mask-based approaches gradually give way to parser-free and training-free pipelines in more recent work.

2. Core Modules: Warping, Compositing, and Loss Formulations

Clothing Warping

A critical step is deforming the input garment to fit the target person's body and pose. Classical pipelines (VITON, CP-VTON) use thin-plate spline (TPS) warping to estimate a smooth transform $T_{\text{TPS}}$ mapping garment landmarks to body silhouettes: $$x_{\text{cloth}}^\prime = T_{\text{TPS}}\left(x_{\text{cloth}}\right)$$ This is driven by keypoint or part-level correspondences, sometimes via part-aware or semantic parsing (Xie et al., 2023, Yang et al., 20 Jul 2025).

Advanced warping modules:

• Local-Flow Global-Parsing (LFGP): GP-VTON (Xie et al., 2023) splits warping into local flows for garment parts, glued by full-image parsing, thus enabling more realistic handling of bent arms, occlusion, and nonrigid materials.
• Piecewise Homography and Structured Garment Morphing (SGM): Training-free frameworks (OmniVTON/OmniVTON++) use part-wise homography fits for anatomical accuracy across garment types (Yang et al., 20 Jul 2025, Yang et al., 16 Feb 2026).

Synthesis and Refinement

• Coarse-to-Fine Decoding: A coarse generator predicts a blurry dressed-person image, with a refinement module learning a mask $M$ to composite detailed warped garment and coarse prediction: $$I_{\text{final}} = M \odot x_{\text{cloth}}^\prime + (1-M) \odot I_{\text{coarse}}$$ Losses here combine pixel L1, mask regularization, and VGG-based perceptual regularization (Han et al., 2017).
• Diffusion and Transformer-based Generators: Recent models apply diffusion U-Nets or DiT backbones, sometimes in temporal (CatVTON (Chong et al., 20 Jan 2025)) or multi-condition settings (Any2AnyTryon (Guo et al., 27 Jan 2025)), leveraging classifier-free guidance, adaptive position embeddings, and multimodal cross-attention.

Loss Formulation

• Reconstruction ($\mathcal{L}_{1}$), perceptual loss ($\mathcal{L}_{\text{perc}}$), and sparsity/regularization on blend masks ($\|\cdot\|_{1}$) are standard (Han et al., 2017).
• Feature matching, adversarial (multi-scale PatchGAN), and VGG-based perceptual terms are prominent in GAN-based pipelines (Adhikari et al., 2023).
• Cycle consistency and identity (face region) loss may be incorporated in pipelines requiring preservation of personal details under strong spatial transformations (Hu et al., 2021).
• Semantic alignment loss (e.g., via CLIP, cosine similarity) and spatial attention focusing ensure semantic correctness in universal/multitask try-on (Zhang et al., 19 Nov 2025, Guo et al., 27 Jan 2025).

3. Algorithmic Advances: Mask-Free, Universal, and Training-Free Designs

Historically, VTON methods have been heavily dependent on human parsing or masking, limiting robustness to occlusion and "in-the-wild" conditions. Several architectural advances address these limitations:

• Mask-Free Pipelines: BooW-VTON (Zhang et al., 2024) achieves mask-free try-on by training on pseudo-paired (unmasked) data generated by a mask-based teacher, and by using cross-attention regularization that implicitly localizes garment changes. This reduces artifacts under occlusion, arbitrary backgrounds, and complex poses.
• Universal and Flexible Control: Any2AnyTryon (Guo et al., 27 Jan 2025) and UniFit (Zhang et al., 19 Nov 2025) enable multi-task VTON (single/multi-garment, model-to-model, garment extraction, layered try-on) via adaptive position embeddings or multimodal LLM-guided conditioning, eliminating rigid dependencies on masks, category, or fixed image size.
• Training-Free and Domain-Generalization: OmniVTON (Yang et al., 20 Jul 2025) and OmniVTON++ (Yang et al., 16 Feb 2026) perform entirely at inference—SGM aligns garments to bodies via local region homographies, pose guidance is injected through spectral frequency or codebook noise mixing, and boundaries are seamlessly repaired via continuous boundary stitching modules, supporting cross-domain, multi-human, and stylized inputs.

These advances collectively remove the reliance on dataset-specific retraining, human parsing, and category constraints, yielding robust “any-person, any-garment, any-scene” VTON.

4. Quantitative and Qualitative Evaluation

VTON models are evaluated using a mixture of distributional and perceptual metrics. Among widely adopted measures:

• Paired Accuracy:
  • SSIM for global structure, LPIPS for learned perceptual similarity, and FID/KID for dataset-level feature distance, are primary (Han et al., 2017, Adhikari et al., 2023, Yang et al., 20 Jul 2025, Song et al., 2023).
  • User studies assess preference for realism, detail, and garment/person consistency (e.g., C-VTON preferred by 76% of raters over CP-VTON (Fele et al., 2022)).
• Specialized Benchmarks:
  • OmniTry-Bench (Feng et al., 19 Aug 2025) extends beyond garments to any wearable object (glasses, jewelry, shoes) using object/personal consistency and localization accuracy.
  • VTON-IQA (Hirakawa et al., 13 Mar 2026) provides reference-free, image-level quality assessment, using a three-branch transformer with interleaved cross-attention, calibrated on a large-scale human-annotated dataset (VTON-QBench, 62,688 try-on images, >400,000 ratings).
• Robustness and Generalization:
  • In-the-wild datasets (StreetVTON, WildVTON) and cross-dataset transfer benchmarks (e.g., DressCode→VITON-HD) validate domain robustness.
  • Training-free and universal pipelines (OmniVTON, OmniVTON++, Any2AnyTryon) deliver top or near-top ranks on FID/LPIPS/SSIM across paired and unpaired, category-specific, and open-domain settings—without retraining (Yang et al., 20 Jul 2025, Yang et al., 16 Feb 2026, Guo et al., 27 Jan 2025).

5. Challenges, Failure Modes, and Emerging Directions

Open Challenges

• Pose and Occlusion Sensitivity: Severe body pose articulation, occlusion (crossed arms, long hair), and multi-layered garments still degrade many models, often due to errors in keypoint/part detection or limitations of warping modules.
• Semantic and Attribute Transfer: Fine details such as logos, embroidery, or lacework, and semantically meaningful garment features (e.g., length, fit, sleeve type) are occasionally lost. The semantic gap between textual instructions and image-level transfer remains nontrivial (Zhang et al., 19 Nov 2025).
• Temporal and Multi-View Consistency: Video-based and multi-view try-on (e.g., MV-VTON (Wang et al., 2024), DreamVTON (Xie et al., 2024)) introduce unique difficulties in maintaining frame-to-frame and viewpoint consistency, requiring specialized conditioning and cross-view alignment.

Notable Failure Modes

• Mask leakage: Blurring and artifacts when the compositing mask "leaks" into the background (Han et al., 2017).
• Geometric misalignment: Warping failures cause distorted or misplaced garments under dramatic pose shifts (Fele et al., 2022, Xie et al., 2023, Han et al., 16 Mar 2025).
• Bleeding and body detail loss: Masking out limbs for clothing-agnostic person representations can cause incorrect skin rendering, notably when switching sleeve lengths (Han et al., 16 Mar 2025).

Research Directions

• Parser- and Mask-Free Universality: Diffusion transformers with adaptive positional embeddings, LLM-guided semantic alignment, and implicit garment/body localization are rapidly reducing the need for explicit pre-segmentation (Yang et al., 20 Jul 2025, Guo et al., 27 Jan 2025, Zhang et al., 19 Nov 2025).
• Physics and 3D Awareness: Integrating implicit or mesh-based 3D priors (e.g., DreamVTON (Xie et al., 2024)), or garment simulation for physically plausible drape and body interaction.
• Interactive & Multimodal Editing: Enabling language-driven garment modification, attribute editing, and style transfer via multimodal LLMs (Zhang et al., 19 Nov 2025, Guo et al., 27 Jan 2025).
• Acceleration and Automation: Reducing the runtime bottleneck of diffusion generation through distilled/student networks (DM-VTON (Nguyen-Ngoc et al., 2023)), fast attention schemes, and real-time mobile deployment.

6. Representative Quantitative Results (Table)

Model/Method	Dataset/Task	SSIM ↑	LPIPS ↓	FID ↓	Notes
VITON (Han et al., 2017)	Zalando	0.86	–	13.4	2-stage U-Net, mask blend
VTON-IT (Adhikari et al., 2023)	FGVC6	0.93	–	50	GAN + U²-Net, high-res
GP-VTON (Xie et al., 2023)	VITON-HD	0.894	0.080	9.2	Local flow/global parsing
BooW-VTON (Zhang et al., 2024)	StreetVTON	–	–	20.6	Mask-free, robust-to-occl.
CatVTON (Chong et al., 20 Jan 2025)	VITON-HD	0.890	0.057	8.10	DiT, temporal concaten.
Any2AnyTryon (Guo et al., 27 Jan 2025)	VITON-HD	0.839	0.088	6.93	Text-ctrl., APE, mask-free
OmniVTON++ (Yang et al., 16 Feb 2026)	DressCode→HD	0.843	0.130	6.99	Training-free, universal
UniFit (Zhang et al., 19 Nov 2025)	VITON-HD	0.883	0.065	8.80	MLLM alignment, multi-task
PL-VTON (Han et al., 16 Mar 2025)	VITON	–	–	12.2	Limb-aware, progressive

This table summarizes major architectural contributors and their quantitative performance, reflecting diversity across supervised, unsupervised, mask-based, and mask-free paradigms.

7. Conclusion

Image-based Virtual Try-On stands at the confluence of dense geometric alignment, high-fidelity generative modeling, and context-aware semantic reasoning. Advances have expanded VTON’s scope to arbitrary garment types, poses, backgrounds, and accessory classes, with pipelines moving toward parser-free, text-controllable, universal, and training-free operation. Persistent challenges—pose/occlusion robustness, semantic detail transfer, and real-time deployment—remain central, driving ongoing innovation in model architecture, evaluation, and practical deployment (Han et al., 2017, Song et al., 2023, Zhang et al., 19 Nov 2025, Yang et al., 20 Jul 2025, Zhang et al., 2024, Yang et al., 16 Feb 2026).