TryOffDiff: High-Fidelity Virtual Try-Off Diffusion

• The paper introduces TryOffDiff, a robust model that reconstructs canonical catalog-style garment images from single in-the-wild photos.
• It leverages a frozen SigLIP image encoder and a transformer-based adapter to inject garment features via cross-attention, ensuring high-fidelity shape and texture recovery.
• Experimental results on the VITON-HD benchmark show significant improvements in SSIM, LPIPS, FID, and DISTS, highlighting its superior perceptual quality.

The TryOffDiff model defines a canonical approach for the Virtual Try-Off (VTOFF) task—reconstructing standardized garment images from single in-the-wild photos of clothed individuals. Unlike traditional Virtual Try-On (VTON), which synthesizes a dressed model given a clean catalog garment and a person image, VTOFF inverts the objective: given (I) an image of a dressed person, generate (G) a clean, unfolded, catalog-style depiction of the worn garment, neutral in pose and isolated on a blank background. TryOffDiff employs a latent diffusion framework based on Stable Diffusion, replacing text-based conditioning with visual features from a frozen SigLIP image encoder and introducing a streamlined architectural pipeline for high-fidelity shape and texture reconstruction. This approach directly optimizes perceptually aligned image fidelity with minimal pre- and post-processing, establishing a robust benchmark for generative garment reconstruction (Velioglu et al., 2024, Velioglu et al., 17 Apr 2025).

1. Task Formulation: VTOFF vs. VTON

Virtual Try-Off (VTOFF) represents a canonical garment reconstruction problem fundamentally distinct from Virtual Try-On (VTON). In VTON, the model synthesizes the appearance of a person wearing a specified garment; the model has direct access to the unoccluded, canonical garment representation, enabling warping and blending for transfer onto the person. VTOFF, conversely, receives only a dressed person image, demanding inference of occluded or distorted garment areas and reconstruction of a uniquely-defined canonical form. This setup is less ambiguous, permitting precise evaluation of reconstruction quality, and requires the generative model to infer both visible and concealed shape/texture cues solely from the input image (Velioglu et al., 2024).

2. Architectural Overview and SigLIP Conditioning

TryOffDiff is architected as a fine-tuned Stable Diffusion v1.4 Latent Diffusion Model (LDM), with critical replacements and restrictions:

• Backbone and Latent Space: Utilizes a frozen VAE encoder/decoder to operate in the latent domain, increasing efficiency and reconstruction fidelity over pixel-space diffusion.
• Conditioning Channel: Substitutes the original text-based CLIP embeddings in the cross-attention layers with visual tokens from a frozen SigLIP-B/16 image encoder. The image input is encoded to a sequence of 1024 tokens of 768 dimensions, which are projected via a Transformer-based Adapter to 77 tokens × 768 dimensions (aligning with the standard conditioning token shape).
• Cross-Attention Injection: During every denoising step, garment features from SigLIP tokens are injected via cross-attention, which grants the U-Net access to spatially resolved texture, pattern, and shape cues sampled from the exhibited garment in the input person image.

The U-Net's trainable blocks focus on cross-attention and adaptation to the SigLIP-based latent features; all other major components (VAE, SigLIP) are fixed, offering a compact and stable fine-tuning scenario (Velioglu et al., 2024, Velioglu et al., 17 Apr 2025).

3. Training Objective and Loss Functions

TryOffDiff employs the standard diffusion denoising loss ("$L_{\mathrm{simple}}$" per [Ho et al.]):

$$L_{\mathrm{denoise}} = \mathbb{E}_{t \sim \{1..T\},\,\mathbf{x}_0,\,\epsilon} \Bigl\lVert \epsilon - \epsilon_\theta(\mathbf{x}_t,\, t,\, \mathrm{cond}=C(\mathbf{I})) \Bigr\rVert^2$$

where $$\mathbf{x}_t = \sqrt{\bar{\alpha}_t} \, \mathbf{x}_0 + \sqrt{1 - \bar{\alpha}_t}\epsilon$$, and $\epsilon_\theta$ denotes the U-Net's predicted noise conditioned on the adapted SigLIP features. No explicit adversarial or perceptual losses are used, relying entirely on the fidelity of the latent diffusion and conditioning mechanism for high-detail garment recovery.

Classifier-free guidance is integrated in inference within a scale range $s \in [1.5, 2.5]$, balancing between sharper detail and suppression of artifacts (Velioglu et al., 2024, Velioglu et al., 17 Apr 2025).

4. Dataset Preparation and Processing

Experiments leverage a cleaned VITON-HD benchmark, with duplicate or overlapping train/test pairs removed—resulting in 11,552 training pairs and 1,990 test pairs. Each pair consists of (I) a dressed person and (G) the clean garment catalog image. Both images are padded and resized to 512×512, normalized to $[-1,1]$. For multi-garment support (as in DressCode), explicit class-conditioning embeddings are used, allowing a single model to extract upper-body, lower-body, or dress garments with comparable fidelity to category-specific models (Velioglu et al., 17 Apr 2025).

5. Evaluation Metrics and Analysis

Standard full-reference metrics (SSIM, MS-SSIM, CW-SSIM, LPIPS), distributional metrics (FID, CLIP-FID, KID), and pairwise perceptual metrics are employed, with a strong focus on the Deep Image Structure and Texture Similarity metric (DISTS). The DISTS metric operates over VGG features, decomposing structure and texture per layer as

$$\mathrm{DISTS}(x,y) = \sum_{\ell=1}^L \alpha_\ell S_\ell(x, y) + \beta_\ell T_\ell(x, y)$$

where low DISTS implies close perceptual and structural similarity. Empirical results indicate that traditional pixel or distributional metrics show weak correlation with human perception of garment fidelity, while DISTS better tracks structural and textural reconstruction quality (Velioglu et al., 2024).

6. Experimental Results and Ablation

TryOffDiff achieves strong improvements over all baselines on VITON-HD, with SSIM 79.5, MS-SSIM 70.4, LPIPS 32.4, FID 25.1, CLIP-FID 9.4, KID 8.9, and DISTS 23.0, yielding a 5.2 point improvement in DISTS over the next-best competitor. Qualitatively, TryOffDiff preserves detailed logos, fine patterns, and is robust to garment occlusions unseen in the input (such as hidden sleeves or bodysuit sections).

Ablation demonstrates:

• Replacement of CLIP with SigLIP image encoder decreases DISTS by 2–3 points.
• Inclusion of a transformer layer in the Adapter module enhances texture alignment over mere linear+LN adapters.
• Latent diffusion is roughly 5× faster and more detailed than pixel-space DDPMs.
• Guidance and denoising steps stabilize for $n \geq 50$ sampling steps.

Multi-garment evaluation on DressCode confirms that class-conditioned joint models match category-specific models within 0.5 points SSIM, supporting Dress, Upper, and Lower categories without notable detail loss (Velioglu et al., 17 Apr 2025).

7. Limitations and Future Directions

TryOffDiff occasionally exhibits failure modes with extremely complex graphics or densely packed multicolor patterns, with some stochastic detail loss across seeds. While DISTS is the preferred metric, further perceptual studies or category-specific perceptual metrics could refine evaluation. Future work entails integrating perceptual or contrastive losses, scaling with larger diffusion backbones (e.g., SD-3), extending to multi-view/multi-garment tasks, and aligning quantitative results with human preference through user studies (Velioglu et al., 2024, Velioglu et al., 17 Apr 2025).