DEFT-VTON: Efficient Diffusion VTON

• DEFT-VTON is a diffusion-based virtual try-on method that adapts a frozen unconditional diffusion model using a compact h-transform network for conditional image synthesis.
• It reduces trainable parameters to about 1.42% and requires as few as 15 denoising steps, achieving competitive results with lower inference costs.
• The framework incorporates adaptive consistency and a generalized h-transform formulation to robustly preserve garment details and improve VTON fidelity on standard benchmarks.

Searching arXiv for DEFT-VTON and closely related diffusion-based virtual try-on and DEFT papers. {"query":"DEFT-VTON virtual try-on Doob h-transform diffusion arXiv", "max_results": 10} {"query":"DEFT-VTON", "max_results": 5} DEFT-VTON is a diffusion-based virtual try-on method that adapts a large pre-trained unconditional model to image-conditioned virtual try-on by applying Doob’s h-transform efficient fine-tuning. Its defining characteristics are that the backbone remains frozen, only a small h-transform network is trained, and inference can be performed with as few as 15 denoising steps while maintaining competitive or state-of-the-art virtual try-on performance. In the reported formulation, the trainable component is about 1.42 percent of the frozen backbone, compared to a baseline of 5.52 percent in traditional parameter-efficient fine-tuning, and the method is presented as a low-cost route from unconditional diffusion to conditional VTON (Xu et al., 16 Sep 2025).

1. Position within diffusion-based virtual try-on

DEFT-VTON addresses a practical constraint in contemporary VTON systems: large diffusion backbones can produce high-quality results, but real-world deployment often prioritizes limited training and inference, serving, and deployment budgets. The method is therefore framed not primarily as a new garment warping or parser design, but as an efficient conditional adaptation strategy for existing large unconditional diffusion models. Its goal is to retrofit those backbones for image-conditioned VTON without retraining the backbone itself (Xu et al., 16 Sep 2025).

This design choice separates DEFT-VTON from a substantial part of the VTON literature. Many recent systems fine-tune backbone components, use end-to-end conditional diffusion training, or rely on more expensive denoising schedules. DEFT-VTON instead learns only the conditional correction term required for virtual try-on. A plausible implication is that the method treats conditionality as an add-on to a frozen generative prior rather than as a property that must be baked into the backbone during large-scale retraining.

The method is also explicitly fast at inference. The reported operating point is as few as 15 denoising steps, in contrast to 50 steps for CatVTON, 20 for OOTDiffusion, and 30 for IDM-VTON. The paper further reports up to 40 percent reduction in NFEs while maintaining competitive results, which positions DEFT-VTON in the efficiency-oriented branch of diffusion VTON research rather than in the branch centered on increasingly specialized garment deformation modules alone (Xu et al., 16 Sep 2025).

2. Generalised h-transform formulation

The theoretical foundation of DEFT-VTON is inherited from "DEFT: Efficient Fine-Tuning of Diffusion Models by Learning the Generalised $h$-transform" (Denker et al., 2024). In the underlying SDE formulation, the backbone is a standard forward diffusion process,

$$dX_t = f_t(X_t)\,dt + \sigma_t\,dW_t,\quad X_0 \sim P_0,$$

with reverse-time drift determined by the unconditional score. DEFT-VTON retains the frozen unconditional score model and learns only the conditional term required for VTON (Denker et al., 2024).

For conditioning on a signal $Y=y$, the conditional score is decomposed as

$$\nabla_x \log p_t(x \mid Y = y) = \nabla_x \log p_t(x) + \nabla_x \log p_t(Y = y \mid x).$$

DEFT interprets the second term as a generalized h-transform,

$h_t(x, y) = \nabla_x \log p_t(y \mid x),$

and learns it with a small network while keeping the unconditional model fixed. In DEFT-VTON, this means that the conditional score used at inference is the sum of the frozen unconditional backbone output and the learned h-transform output:

$$s_{\text{cond}}(x_t,t,y) = s_t^{\theta^*}(x_t,t) + h_t(x_t,y).$$

The method therefore performs conditional adaptation by learning guidance rather than by modifying the main score network weights (Xu et al., 16 Sep 2025).

The training objective specializes denoising score matching to this decomposition. With DDPM-style $\epsilon$-prediction and

$$H_t = \sqrt{\bar{\alpha}_t} X_0 + \sqrt{1 - \bar{\alpha}_t}\,\epsilon,$$

the DEFT objective becomes

$$\mathcal{L}_{\text{DSM}} = \mathbb{E}_{(x_0,y), \epsilon, t} \Big[ \big\| h(H_t,y) + s_t^{\theta^*}(H_t,t) - \epsilon \big\|^2 \Big].$$

Factually, this is the core mechanism by which DEFT-VTON converts a frozen unconditional model into a conditional VTON model (Xu et al., 16 Sep 2025).

3. Latent architecture and VTON conditioning

The VTON pipeline is formulated in latent space. The inputs are a person image $I_p \in \mathbb{R}^{3 \times H \times W}$, a garment image $$dX_t = f_t(X_t)\,dt + \sigma_t\,dW_t,\quad X_0 \sim P_0,$$0, and a mask image $$dX_t = f_t(X_t)\,dt + \sigma_t\,dW_t,\quad X_0 \sim P_0,$$1 indicating the garment or inpainting region. A garment-agnostic person image is formed as

$$dX_t = f_t(X_t)\,dt + \sigma_t\,dW_t,\quad X_0 \sim P_0,$$2

where $$dX_t = f_t(X_t)\,dt + \sigma_t\,dW_t,\quad X_0 \sim P_0,$$3 denotes element-wise product. A frozen VAE encoder $$dX_t = f_t(X_t)\,dt + \sigma_t\,dW_t,\quad X_0 \sim P_0,$$4 maps the masked person and garment images to latent tensors

$$dX_t = f_t(X_t)\,dt + \sigma_t\,dW_t,\quad X_0 \sim P_0,$$5

The mask is interpolated to latent resolution to form $$dX_t = f_t(X_t)\,dt + \sigma_t\,dW_t,\quad X_0 \sim P_0,$$6, and the person and garment latents are spatially concatenated:

$$dX_t = f_t(X_t)\,dt + \sigma_t\,dW_t,\quad X_0 \sim P_0,$$7

where $$dX_t = f_t(X_t)\,dt + \sigma_t\,dW_t,\quad X_0 \sim P_0,$$8 denotes spatial concatenation and $$dX_t = f_t(X_t)\,dt + \sigma_t\,dW_t,\quad X_0 \sim P_0,$$9 is an all-zero tensor (Xu et al., 16 Sep 2025).

Sampling begins from Gaussian noise

$Y=y$0

which is combined with the condition tensors to produce the initial noisy latent

$Y=y$1

The frozen backbone UNet $Y=y$2 denoises iteratively:

$Y=y$3

After the reverse process, the virtual try-on latent is split out and decoded by the frozen VAE decoder $Y=y$4 to obtain the final try-on image $Y=y$5 (Xu et al., 16 Sep 2025).

The h-transform network follows the same conditioning interface as the backbone. It takes the same latent state, time, and conditioning inputs, acts as an auxiliary branch, and outputs a tensor of the same shape as the backbone output. Its output is added to the frozen backbone output. The paper states that the h-transform network is small enough that total trainable parameters are about 1.42 percent of the frozen backbone, whereas baseline PEFT trains about 5.52 percent. Because the network is external to the backbone, the unconditional generator remains unchanged, and the same adaptation principle can in principle be attached to new unconditional backbones with only the small h-transform retrained (Xu et al., 16 Sep 2025).

4. Training objectives and adaptive consistency

DEFT-VTON combines the DEFT denoising score matching objective with a consistency objective. The DSM term is the conditional adaptation loss described above. The consistency component is motivated by the observation that consistency training can distill slow but high-performing diffusion models into faster ones by enforcing consistencies along the inference path. Rather than distilling into a separate model, DEFT-VTON applies consistency constraints directly to the combined output of the frozen backbone and the h-transform network (Xu et al., 16 Sep 2025).

The combined prediction is written as

$Y=y$6

and the consistency loss enforces agreement between outputs from different timesteps:

$Y=y$7

where $Y=y$8 is chosen as pseudo-Huber loss. The paper’s interpretation is that consistency regularizes the diffusion trajectory so that predictions at different noise levels agree on the same underlying clean sample (Xu et al., 16 Sep 2025).

A direct linear combination of consistency and DSM losses is reported to risk degrading VTO correctness. DEFT-VTON therefore introduces an adaptive objective,

$Y=y$9

with constants $$\nabla_x \log p_t(x \mid Y = y) = \nabla_x \log p_t(x) + \nabla_x \log p_t(Y = y \mid x).$$0, $$\nabla_x \log p_t(x \mid Y = y) = \nabla_x \log p_t(x) + \nabla_x \log p_t(Y = y \mid x).$$1, $$\nabla_x \log p_t(x \mid Y = y) = \nabla_x \log p_t(x) + \nabla_x \log p_t(Y = y \mid x).$$2, $$\nabla_x \log p_t(x \mid Y = y) = \nabla_x \log p_t(x) + \nabla_x \log p_t(Y = y \mid x).$$3, and $$\nabla_x \log p_t(x \mid Y = y) = \nabla_x \log p_t(x) + \nabla_x \log p_t(Y = y \mid x).$$4. The stated intuition is that when $$\nabla_x \log p_t(x \mid Y = y) = \nabla_x \log p_t(x) + \nabla_x \log p_t(Y = y \mid x).$$5 is large, the objective emphasizes reconstruction and conditional correctness; when $$\nabla_x \log p_t(x \mid Y = y) = \nabla_x \log p_t(x) + \nabla_x \log p_t(Y = y \mid x).$$6 is small, the objective gives more room to consistency and faster sampling. The paper describes this as inspired by constrained optimization (Xu et al., 16 Sep 2025).

This training scheme yields two practically distinct variants. A DSM-only version emphasizes direct conditional adaptation and is reported with 25-step and 15-step configurations. A DSM+CM version adds the adaptive consistency component and is explicitly intended to reduce the number of denoising steps needed for good VTON results (Xu et al., 16 Sep 2025).

5. Datasets, metrics, and reported results

The primary public benchmark is VITON-HD, described as a public VTON benchmark whose test split contains 2,032 pairs, each comprising a reference garment image, a source person image representing the ground truth of a person wearing the garment, and associated mask and pose information. The paper also uses an internal "Virtual Try-All" dataset with broad categories including jackets, pajamas, bags, hats, shoes, jewelry, toys, and other items, as well as reference-image styles such as lay-flat, ghost-mannequin, on-figure, and off-figure (Xu et al., 16 Sep 2025).

Evaluation follows the standard paired and unpaired protocol. In the paired setting, the reported metrics are SSIM, where higher is better, and LPIPS, where lower is better. In the unpaired setting, the reported metrics are FID and KID, both lower-is-better. This aligns DEFT-VTON with the evaluation conventions already established in diffusion VTON benchmarks (Xu et al., 16 Sep 2025).

On VITON-HD, the main comparison table reports the following baseline values: StableVTON reaches SSIM 0.8543, LPIPS 0.0905, FID 11.054, KID 3.914; LaDI-VTON reaches 0.8603, 0.0733, 14.648, 8.754; IDM-VTON reaches 0.8499, 0.0603, 9.842, 1.123; OOTDiffusion reaches 0.8187, 0.0876, 12.408, 4.680; and CatVTON reaches 0.8704, 0.0565, 9.015, 1.091. Against these, DEFT with $$\nabla_x \log p_t(x \mid Y = y) = \nabla_x \log p_t(x) + \nabla_x \log p_t(Y = y \mid x).$$7 at 25 steps reports SSIM 0.9118, LPIPS 0.0533, FID 8.3351, and KID 0.5212; DEFT with $$\nabla_x \log p_t(x \mid Y = y) = \nabla_x \log p_t(x) + \nabla_x \log p_t(Y = y \mid x).$$8 at 15 steps reports SSIM 0.9098, LPIPS 0.0521, FID 8.6339, and KID 0.7916; DEFT with DSM+CM at 25 steps reports SSIM 0.9105, LPIPS 0.0564, FID 8.6310, and KID 0.7243; and DEFT with DSM+CM at 15 steps reports SSIM 0.9130, LPIPS 0.0542, FID 8.8567, and KID 1.016 (Xu et al., 16 Sep 2025).

The paper’s qualitative discussion attributes these results to stronger preservation of garment colors, printed text, and logos, along with robust handling of pose variation and occlusions. It also reports that consistency-finetuned DEFT-VTON at 15 steps is qualitatively superior to pure DEFT at 15 or 25 steps in difficult cases involving complex text or graphics, unclear reference images, and rare scenarios in the training data. The numerical and qualitative results together support the paper’s central claim: efficient conditional adaptation and fast sampling need not require aggressive degradation of VTON fidelity (Xu et al., 16 Sep 2025).

6. Relation to adjacent VTON methods

DEFT-VTON belongs to the same broad diffusion VTON design space as FLDM-VTON and D$$\nabla_x \log p_t(x \mid Y = y) = \nabla_x \log p_t(x) + \nabla_x \log p_t(Y = y \mid x).$$9-VTON, but it addresses a different bottleneck. "FLDM-VTON: Faithful Latent Diffusion Model for Virtual Try-on" focuses on faithfulness to clothing details by incorporating warped clothes as both the starting point and local condition, introducing a clothes flattening network for clothes-consistent supervision, and devising clothes-posterior sampling for faithful inference. "D$h_t(x, y) = \nabla_x \log p_t(y \mid x),$0-VTON: Dynamic Semantics Disentangling for Differential Diffusion based Virtual Try-On" targets semantic inconsistencies in warping and learning ambiguity in simultaneous inpainting and denoising by combining Dynamic Semantics Disentangling Modules with a Differential Information Tracking Path (Wang et al., 2024, Yang et al., 2024).

By contrast, DEFT-VTON does not rely on explicit 2D warping modules, parser redesign, or garment-specific inverse-flow supervision in its main formulation. Its conditioning is handled through latent concatenation of the garment-agnostic person, garment reference, and mask, while the core innovation lies in learning the conditional h-transform on top of a frozen unconditional diffusion backbone. This suggests that DEFT-VTON is architecturally orthogonal to methods whose main contribution is garment deformation fidelity. A plausible implication is that the DEFT mechanism could coexist with more specialized warping or faithfulness modules, although such a combination is not documented in the supplied material.

The comparison is also informative at the level of research priorities. FLDM-VTON and D$h_t(x, y) = \nabla_x \log p_t(y \mid x),$1-VTON emphasize garment faithfulness, semantic consistency, and the coordination of warping with synthesis. DEFT-VTON emphasizes adaptation efficiency, low trainable parameter ratio, and reduction of denoising steps. All three belong to the current diffusion VTON trajectory, but they optimize different resource-quality trade-offs (Wang et al., 2024, Yang et al., 2024).

7. Distinctions, limitations, and future directions

A recurrent source of ambiguity is the acronym "DEFT." In the VTON context, DEFT refers to Doob’s h-transform efficient fine-tuning, derived from the conditional diffusion framework of (Denker et al., 2024) and instantiated for virtual try-on in (Xu et al., 16 Sep 2025). This is distinct from "From PEFT to DEFT: Parameter Efficient Finetuning for Reducing Activation Density in Transformers," where DEFT denotes Density-Efficient Fine-Tuning for inducing activation sparsity in Transformer MLP blocks. That transformer paper explicitly states that it does not define DEFT-VTON (Runwal et al., 2024).

The limitations reported for DEFT-VTON follow directly from its efficiency strategy. Because the h-transform network is only about 1.42 percent of the backbone, the method relies heavily on the pre-trained unconditional model. Any failure mode, artifact, bias, or limitation present in the backbone is therefore expected to propagate to DEFT-VTON. The paper also notes that extreme poses, rare garment types, and very complex deformations may remain challenging, that very low step counts can induce hallucinated garment shapes or textures, and that the method is focused on 2D image-based VTON without explicit 3D or temporal modeling (Xu et al., 16 Sep 2025).

The future directions identified in the supplied sources are correspondingly structural. For DEFT-VTON itself, the paper suggests video VTON with temporal consistency losses, 3D-aware models, more systematic multi-garment composition, adaptation to non-diffusion backbones, and improved h-transform formulations. In the broader diffusion VTON literature, FLDM-VTON and D$h_t(x, y) = \nabla_x \log p_t(y \mid x),$2-VTON point toward richer garment-specific priors, stronger detail preservation, explicit handling of semantic consistency, and more robust reasoning under occlusion. Taken together, these directions indicate that DEFT-VTON should be understood as an efficiency-centered conditionalization framework for VTON rather than as a complete replacement for all garment-specific modeling strategies (Xu et al., 16 Sep 2025)