Two-Way Garment Transfer Model

• TWGTM is a unified diffusion framework that bridges virtual try-on and try-off using dual-conditioned guidance and bidirectional feature disentanglement.
• The model employs a latent diffusion backbone augmented by Semantic Abstraction and Spatial Refinement modules, effectively addressing mask dependency asymmetry.
• Phased training and ablation studies demonstrate that joint learning and dual-path fusion lead to superior synthesis performance on benchmarks like VITON-HD and DressCode.

Searching arXiv for TWGTM and closely related garment transfer papers to ground the article in the cited literature. The Two-Way Garment Transfer Model (TWGTM) is a unified diffusion framework for joint clothing-centric image synthesis that simultaneously resolves mask-guided virtual try-on (VTON) and mask-free virtual try-off (VTOFF) through bidirectional feature disentanglement. In this formulation, VTON transfers a garment to a human subject, whereas VTOFF reconstructs a canonical garment template from a dressed human image. TWGTM addresses the complementary symmetry between these tasks by combining dual-conditioned guidance from both latent and pixel spaces of reference images with a phased training paradigm designed to bridge the mask dependency asymmetry between mask-guided VTON and mask-free VTOFF (Zhang et al., 6 Aug 2025).

1. Problem setting and conceptual scope

TWGTM is defined around two dual tasks. Virtual try-on overlays garments onto a person, enabling digital fashion experience, while virtual try-off extracts a canonical, clean garment image from a photo of a dressed person. The underlying asymmetry is central: VTON methods typically rely on mask guidance to localize the swap region, whereas VTOFF cannot use such explicit masks because the boundaries of garments are ambiguous when extracting from real photos (Zhang et al., 6 Aug 2025).

This framing distinguishes TWGTM from earlier garment transfer systems that focused on one direction of synthesis or on transfer without a unified treatment of dressing and undressing. The model is presented as a single framework in which both tasks are modeled as reversible conversions through the same backbone and feature set, with the direction determined by the order and type of conditional inputs. A common misconception is to treat VTOFF as a trivial inverse of VTON; the paper explicitly characterizes VTOFF as critically underexplored and harder because it must infer unseen or ambiguous garment structure without explicit mask supervision (Zhang et al., 6 Aug 2025).

The notion of “two-way” in TWGTM is therefore not merely bidirectional image translation in a generic sense. It denotes a unified treatment of garment dressing and undressing, with task symmetry enforced through conditional design and training, rather than through a separate forward model for VTON and a separate inverse model for VTOFF.

2. Unified diffusion architecture

TWGTM uses a latent diffusion model backbone augmented with dual-path conditional guidance mechanisms. The architecture operates through bidirectional feature disentanglement in latent space and pixel space. In latent space, conditional features from the person or model image $x$ and the garment image $c$ are concatenated spatially; for VTON the ordering is $[x, c]$, and for VTOFF it is $[c, x]$. This ordering encodes the synthesis direction while keeping the backbone shared (Zhang et al., 6 Aug 2025).

For VTON, explicit mask maps localize the try-on region. For VTOFF, the model learns to hallucinate the mask. This design directly addresses the mask dependency asymmetry rather than eliminating it at the architectural level. The framework instead learns to compensate for the asymmetry through training.

At the pixel-space level, TWGTM supplements the diffusion backbone with modules that inject semantic and spatial guidance into the UNet. The following table summarizes the modules described for this purpose.

Module	Components	Role
Semantic Abstraction Module (SAM)	frozen CLIP image encoder and QFormer	extract global and category-specific garment semantics
Spatial Refinement Module (SRM)	Swin Transformer and TaskFormer	trace garment regions, refine spatial details, and predict masks
Extended Attention Block	self-attention, cross-attention, Zero Linear Layer	fuse UNet features with SAM and SRM outputs

The Semantic Abstraction Module combines a frozen CLIP image encoder and QFormer, described as BLIP-2 inspired, to extract global and category-specific garment semantics. Text prompts such as “upper garment” guide QFormer toward relevant regions. The Spatial Refinement Module uses a Swin Transformer to extract multi-scale spatial features and a TaskFormer, described as Mask2Former-inspired, to produce region-specific feature maps and predicted masks. The Extended Attention Block then fuses UNet features with outputs of SRM and SAM through standard self-attention plus cross-attention from the spatial and semantic paths, with a Zero Linear Layer modulating cross-attention outputs to limit noise and enhance discriminative patterns (Zhang et al., 6 Aug 2025).

This architecture suggests that TWGTM treats semantic identity and spatial localization as complementary conditioning sources rather than as a single fused representation from the outset.

3. Dual-conditioned guidance and bidirectional disentanglement

The defining mechanism of TWGTM is dual-conditioned guidance across latent and pixel spaces. In latent space, images and masks are composed and encoded by the VAE, and the arrangement of conditional inputs toggles the synthesis direction. In pixel space, SAM provides global and semantic garment context, while SRM supplies spatial detail and mask inference. These signals are injected into the diffusion UNet through extended attention (Zhang et al., 6 Aug 2025).

The latent-space branch maintains geometric and topological consistency of garments via spatial feature concatenation. The pixel-space branch refines fine-grained and semantic details through the specialized modules. The authors describe this pairing as enabling the model to reason about both the structure of the edit and the appearance of the garment during synthesis and extraction.

A technical implication of the bidirectional disentanglement scheme is that VTON and VTOFF are not implemented as unrelated prompt conditions on a generic image generator. Instead, the same backbone is reused while the semantic role of each condition is swapped. This is a stronger coupling than simple multi-task training because the task reversal is represented explicitly in the conditional layout.

The paper also states that removing SAM, SRM, or their integration degrades both VTON and VTOFF performance, and that alternate fusion strategies likewise reduce performance. Task-specific training from scratch is reported to yield inferior results for the harder VTOFF task. These ablations are presented as evidence that joint learning and dual-path guidance are necessary rather than incidental design choices (Zhang et al., 6 Aug 2025).

4. Phased training paradigm and objective functions

TWGTM introduces a phased training protocol to bridge the modality gap created by mask dependency asymmetry. The first stage focuses on inpainting capability. For VTOFF, a generated pseudo-ground-truth garment mask is used as the inpainting region, and the model is supervised to both inpaint the masked region and predict the mask with auxiliary dice and BCE losses. The emphasis in this phase is accurate mask localization and region-specific inpainting (Zhang et al., 6 Aug 2025).

The stage-1 objective is given as

$$\mathcal{L} = \mathbb{E} \left\| \epsilon - \epsilon_\theta(I_t, t, \tau_{SAM}(\mathbf{x}_{ref}))\right\|_2^2 + \lambda \mathcal{L}_{mask}$$

with

$$\mathcal{L}_{mask} = \lambda' \mathcal{L}_{dice} + \lambda'' \mathcal{L}_{bce}.$$

The second stage emphasizes shape awareness and robustness. True masks are replaced with morphologically perturbed square masks, described as eroded or dilated, to challenge geometric reasoning. In this stage there are no explicit garment masks; the model relies on learned shape priors and is trained to reconstruct under imprecisely localized cues. The stated purpose is to bridge the VTON mask-guided regime and the VTOFF mask-free regime (Zhang et al., 6 Aug 2025).

The stage-2 objective is

$$\mathcal{L} = \mathbb{E}\left\| \epsilon - \epsilon_\theta(I_t, t, \tau_{SAM}(\mathbf{x}_{ref}), \tau_{SRM}(\mathbf{x}_{ref}))\right\|_2^2,$$

where $\tau_{SAM}$ and $\tau_{SRM}$ denote outputs from the Semantic Abstraction and Spatial Refinement Modules.

This two-stage design is significant because it does not assume that a model trained under mask supervision will automatically generalize to a mask-free inverse problem. Instead, it progressively weakens the localization prior. A plausible implication is that TWGTM treats mask prediction as an intermediate competency that must later be regularized away from exact-mask dependence.

5. Empirical evaluation

TWGTM is evaluated on the DressCode and VITON-HD datasets. The summary describes VITON-HD as containing approximately 10,000 upper-body garment/person pairs and DressCode as containing more than 40,000 multi-category garment/person pairs spanning upper, lower, and dresses (Zhang et al., 6 Aug 2025).

The evaluation protocol includes structural metrics such as SSIM, MS-SSIM, and CW-SSIM; texture fidelity metrics such as LPIPS and DISTS; perceptual and semantic realism metrics such as FID, KID, and CLIP-FID; and DINO Similarity for high-level feature matching. On VITON-HD VTON, TWGTM reports FID $6.107$, DINO $c$0, SSIM $c$1, and LPIPS $c$2, with the summary noting that it excels particularly in DINO, SSIM, and LPIPS. On VITON-HD VTOFF, TWGTM reports SSIM $c$3, MS-SSIM $c$4, CW-SSIM $c$5, LPIPS $c$6, FID $c$7, CLIP-FID $c$8, KID $c$9, and DISTS $[x, c]$0, and is described as achieving the lowest DISTS and KID together with strong FID (Zhang et al., 6 Aug 2025).

Qualitative comparisons are reported in both directions. For VTON, the model is described as recovering subtle textures such as text and logos while maintaining consistent color and structure. For VTOFF, it is described as recovering intricate patterns and correct shapes with less color distortion and fewer extraneous artifacts than TryOffDiff or TryOffAnyone (Zhang et al., 6 Aug 2025).

These results support the paper’s claim that joint learning benefits the harder VTOFF problem. They also indicate that the unified formulation is not merely a parameter-sharing convenience; the shared representation appears to improve synthesis quality under the inverse task where explicit spatial cues are unavailable.

6. Position within the garment-transfer literature

TWGTM belongs to a broader line of research on disentangled garment transfer, geometric alignment, and controllable appearance synthesis, but its specific contribution is the unification of mask-guided VTON and mask-free VTOFF in a diffusion framework (Zhang et al., 6 Aug 2025).

Earlier 2D systems emphasized controllable transfer without a unified try-off formulation. The controllable garment transfer model built on DiOR decomposes person images into explicit, editable representations of pose, body, and garments, and introduces “garment tweaking” through separate shape masks $[x, c]$1 and texture encodings $[x, c]$2. That framework is described as modular and sequential and as supporting practical two-way transfer in principle, but it is also explicitly noted to be “not a strict cycle-consistent” approach (Son et al., 2022). This contrast is important: TWGTM does not center on post-transfer attribute editing, but on joint dressing and undressing synthesis.

GAN-based try-on systems such as GarmentGAN separate shape transfer from appearance transfer, use segmentation maps and keypoint information, and address self-occlusions through explicit spatial guidance. GarmentGAN is described as a practical blueprint for robust, identity-preserving transfer, but it remains a try-on system rather than a unified try-on/try-off model (Raffiee et al., 2020). Likewise, the knowledge-distillation-based garment transfer method of 2024 uses transfer parsing, STN alignment, progressive flow estimation, and an arm regrowth strategy to preserve body features, but its supervision is still organized around transfer parsing and teacher-student warping rather than a unified reverse garment reconstruction problem (Fang et al., 2024).

In 3D, Multi-Garment Net separates body and garments as layered meshes over SMPL and is described as inherently two-way because garments can be extracted from one subject and dressed onto another in arbitrary poses (Bhatnagar et al., 2019). This provides a different notion of two-way transfer: geometry retargeting rather than joint 2D dressing and undressing synthesis.

A close contemporary point of comparison is OMFA, another unified diffusion framework for try-on and try-off. OMFA is described as entirely mask-free, based on partial diffusion, and able to support arbitrary poses through SMPL-X-based conditioning (Liu et al., 6 Aug 2025). Relative to OMFA, TWGTM directly addresses mask dependency asymmetry by staged training rather than by adopting a fully mask-free formulation from the outset. This suggests two different research directions within unified garment transfer: one that progressively bridges asymmetric supervision, and one that removes the asymmetry at the input specification level.

Taken together, these systems indicate that TWGTM occupies a specific position in the literature: it combines diffusion-based synthesis, bidirectional feature disentanglement, and phased modality-bridging training to treat dressing and undressing as a single clothing-centric problem, while remaining distinct from controllable attribute-editing pipelines, two-stage GAN try-on frameworks, 3D garment retargeting systems, and fully mask-free partial-diffusion approaches.