Papers
Topics
Authors
Recent
Search
2000 character limit reached

OmniVTON: Universal Virtual Try-On Framework

Updated 4 July 2026
  • OmniVTON is a training-free virtual try-on framework that decouples garment and pose conditioning using a diffusion-based formulation.
  • It employs geometrically aligned garment priors, continuous boundary stitching, and spectral pose injection to synthesize realistic person imagery.
  • Extensions like OmniVTON++ and OmniDiT refine pose guidance and integrate transformer architectures for unified virtual try-on and try-off tasks.

Searching arXiv for the OmniVTON paper family and closely related entries to ground the article in the latest available preprints. OmniVTON is a training-free universal virtual try-on framework for image-based garment transfer that was introduced to unify supervised in-shop and unsupervised in-the-wild settings within a single diffusion-based formulation. Its central design principle is to decouple garment conditioning and pose conditioning: garment appearance is injected through a geometrically aligned garment prior and boundary-aware inpainting, whereas pose is injected through DDIM inversion and spectral manipulation of latent noise. Subsequent work extends this line in two directions: OmniVTON++ replaces single-step pose injection with step-wise pose guidance and refines the boundary module for both U-Net and DiT backbones, while OmniDiT moves to a trained, unified Diffusion Transformer that combines model-based VTON, model-free VTON, and VTOFF in one model (Yang et al., 20 Jul 2025, Yang et al., 16 Feb 2026, Zeng et al., 20 Mar 2026).

1. Problem formulation and conceptual basis

OmniVTON addresses image-based Virtual Try-On (VTON), where the task is to synthesize realistic person imagery through garment re-rendering under human pose and body constraints. The original formulation was motivated by a split in the literature between supervised in-shop approaches, which ensure high fidelity but struggle with cross-domain generalization, and unsupervised in-the-wild methods, which improve adaptability but remain constrained by data biases and limited universality (Yang et al., 20 Jul 2025).

The framework is explicitly training-free. It does not retrain the diffusion backbone for each dataset or task condition, and instead composes pretrained components with geometric warping, frequency-domain pose conditioning, and attention-level feature fusion. This design is intended to avoid the bias that arises when diffusion models handle multiple conditions simultaneously. In the original method, garment guidance enters through the input image Ip′I_p' and mask, while pose guidance enters exclusively through the mixed latent noise z^T\hat z_T (Yang et al., 20 Jul 2025).

A later extension, OmniVTON++, retains the training-free premise but reorganizes the pipeline into three named modules: Structured Garment Morphing (SGM), Principal Pose Guidance (PPG), and Continuous Boundary Stitching (CBS), all wrapped around an off-the-shelf diffusion inpainting backbone such as Stable Diffusion or FLUX. No retraining is performed; all modules rely on pretrained tools, including a human parser, pose estimator, and diffusion model (Yang et al., 16 Feb 2026).

This suggests that OmniVTON is best understood not as a single architecture in the narrow sense, but as a design pattern for disentangling garment geometry, human structure, and diffusion synthesis.

2. Original OmniVTON pipeline

The original OmniVTON pipeline begins with two inputs: a source garment image IcI_c, which may be standalone or worn, and a target person image IpI_p. The first stage constructs a garment prior; the second stage performs inpainting with decoupled pose conditioning and continuous boundary stitching (Yang et al., 20 Jul 2025).

When only a standalone garment image is available, OmniVTON first generates a pseudo-person image IoI_o that “wears” the garment. This is done by cross-attention in a pretrained diffusion U-Net. Let ztz_t denote diffusion noise at timestep tt, M~c\tilde M_c the inverted cloth mask on IcI_c, ImI_m a cloth-agnostic version of the real person, and MpM_p the agnostic mask on ImI_m. Two parallel denoising streams are run, and the wearer’s semantics are injected into the garment stream by

fc=Softmax⁡ ⁣(Qc [Kc∥Kp]⊤d) [Vc∥Vp].(1)f_c = \operatorname{Softmax}\!\Bigl(\tfrac{Q_c\,[K_c\mathbin{\Vert}K_p]^\top}{\sqrt{d}}\Bigr)\,[V_c\mathbin{\Vert}V_p]. \tag{1}

Semantic correspondence is then established by running OpenPose to obtain 25 keypoints and grouping them into NN semantic regions. For each region ii, a bounding box BoiB_o^i is defined on IoI_o and BpiB_p^i on IpI_p, and a part-segmentation map PoiP_o^i refines the support: IRegioni(xo,yo)={1,(xo,yo)∈Poi∩Boi,0,otherwise.(2)\mathbb{I}_{\mathrm{Region}_i}(x_o,y_o) = \begin{cases} 1,&(x_o,y_o)\in P_o^i\cap B_o^i,\\ 0,&\text{otherwise.} \end{cases} \tag{2}

Localized piecewise homography is then used to align the garment to the target body. For each region ii, a 3×33\times 3 homography Ho→piH^i_{o\to p} is solved by Levenberg–Marquardt, and the pixels of IoI_o are mapped by

[xo′yo′1]=∑i=1NIRegioni(xo,yo)  Ho→pi[xoyo1].(3)\begin{bmatrix} x_o'\\ y_o'\\ 1 \end{bmatrix} = \sum_{i=1}^{N} \mathbb{I}_{\mathrm{Region}_i}(x_o,y_o)\; H^i_{o\to p} \begin{bmatrix} x_o\\ y_o\\ 1 \end{bmatrix}. \tag{3}

The output IωI_\omega is a coarse aligned garment prior. Because the warping is region-wise, seams can appear at region boundaries; this motivates the next stage (Yang et al., 20 Jul 2025).

In the inpainting stage, a garment-infused image is formed as

Ip′=Ip⊙(1−Mp)+Iω⊙Mp.I_p' = I_p\odot(1-M_p) + I_\omega\odot M_p.

The pretrained U-Net itself is unchanged. The explicitly training-free components are the homography solver, the Fourier-based pose conditioning, and attention calls to key/value buffers (Yang et al., 20 Jul 2025).

3. Garment prior generation and boundary refinement

In OmniVTON terminology, the garment prior is a coarse but texture-faithful deformation of the source garment. Its purpose is to preserve garment details under substantial spatial deformation. The original paper divides this into pseudo-person generation, semantic correspondence, and localized homography, collectively yielding a body-aligned prior IωI_\omega (Yang et al., 20 Jul 2025).

To remove seams introduced by stitching across warped regions, OmniVTON uses Continuous Boundary Stitching (CBS) during the final inpainting stage. CBS modulates self-attention in each transformer block using both the garment-infused person image Ip′I_p' and the source garment IcI_c. The Ip′I_p'-path enhancement is

fp′=Softmax⁡ ⁣(Qp′[Kp′∥Kc]⊤d)  [Vp′∥(Vc ⁣⋅↓Mc)],(4)f_p' = \operatorname{Softmax}\!\Bigl(\tfrac{Q_p'[K_p'\mathbin{\Vert}K_c]^\top}{\sqrt{d}}\Bigr)\; \bigl[V_p'\mathbin{\Vert}(V_c\!\cdot\downarrow M_c)\bigr], \tag{4}

where ↓Mc\downarrow M_c is the low-resolution downsampled mask. The IcI_c-path refinement is

Ac=Softmax⁡ ⁣(Qc[Kc∥Kp′]⊤d),fc=Ac[:, 1:n]  Vc.(5)A_c = \operatorname{Softmax}\!\Bigl(\tfrac{Q_c[K_c\Vert K_p']^\top}{\sqrt{d}}\Bigr), \quad f_c = A_c[:,\,1:n]\;V_c. \tag{5}

The stated role of CBS is to erase multi-region seams and blend fine texture details. By back-and-forth attention mixing, it removes hard seams while preserving cloth texture (Yang et al., 20 Jul 2025).

OmniVTON++ retains the same basic concern but re-specifies the garment module as Structured Garment Morphing. When the garment image is flat-lay and keypoint detection and parsing fail, a pseudo-person IoI_o is synthesized by feeding IcI_c through a pretrained garment-centric dressing module, IMAGDressing-v1, together with a fixed “A-pose” keypoint set and a generic prompt, “white background, no occlusion” (Yang et al., 16 Feb 2026). It then defines effective supports by

IRegioni(x,y)={1,(x,y)∈Poi∩Mo∩Boi,0,otherwise,(1)\mathbb{I}_{\mathrm{Region}_i}(x,y) = \begin{cases} 1,& (x,y)\in P_o^i \cap M_o \cap B_o^i,\\ 0,& \text{otherwise,} \end{cases} \tag{1}

and assembles the warped prior IwI_w through localized homographies

[x′y′1]=∑i=1nIRegioni(x,y)  Ho→pi[xy1].(2)\begin{bmatrix} x'\\ y'\\ 1 \end{bmatrix} = \sum_{i=1}^n \mathbb{I}_{\mathrm{Region}_i}(x,y)\;\mathcal{H}^i_{o\to p} \begin{bmatrix} x\\ y\\ 1 \end{bmatrix}. \tag{2}

In OmniVTON++, CBS is also generalized. For U-Net backbones, dual-stream attention refines boundaries by mixing person and garment streams asymmetrically; for DiT backbones, CBS-DiT uses Positional Index Realignment so that tokens from text, person, and garment receive disjoint rotary-encoding index ranges rather than colliding indices (Yang et al., 16 Feb 2026).

4. Pose conditioning and diffusion dynamics

The most distinctive element of the original OmniVTON is its decoupled pose-conditioning strategy. Rather than encode garment and pose in the same pathway, it obtains target pose from DDIM inversion of the person image IpI_p to a latent noise zTinvz_T^{\mathrm{inv}}, then suppresses texture interference by retaining only the low-frequency components of the inverted noise (Yang et al., 20 Jul 2025).

The method first computes

fTinv=Shift(F(zTinv)),fT=Shift(F(zT)),(6)f_T^{\mathrm{inv}} = \mathrm{Shift}\bigl(\mathcal{F}(z_T^{\mathrm{inv}})\bigr), \quad f_T = \mathrm{Shift}\bigl(\mathcal{F}(z_T)\bigr), \tag{6}

then fuses them with a Gaussian mask

f^T=Gτ⊙fTinv+(1−Gτ)⊙fT,(7)\hat f_T = G_\tau\odot f_T^{\mathrm{inv}} + (1-G_\tau)\odot f_T, \tag{7}

and reconstructs

z^T=F−1 ⁣(Shift−1(f^T)).(8)\hat z_T = \mathcal{F}^{-1}\!\bigl(\mathrm{Shift}^{-1}(\hat f_T)\bigr). \tag{8}

This mechanism is called Spectral Pose Injection (SPI). Its stated effect is to preserve coarse silhouette and pose while re-randomizing high-frequency clothing cues (Yang et al., 20 Jul 2025).

Sampling then proceeds with garment guidance in Ip′I_p' and pose guidance in z^T\hat z_T. The one-step update is written as

xt−1=αt−1 xt−1−αt ϵθ(xt, Ip′, z^T, t)αt+1−αt−1 ϵθ(xt, Ip′, z^T, t).x_{t-1} = \sqrt{\alpha_{t-1}}\, \frac{x_t-\sqrt{1-\alpha_t}\,\epsilon_\theta(x_t,\,I_p',\,\hat z_T,\,t)}{\sqrt{\alpha_t}} + \sqrt{1-\alpha_{t-1}}\,\epsilon_\theta(x_t,\,I_p',\,\hat z_T,\,t).

The paper’s interpretation is that because pose lives in the latent noise and garment in the image hints, neither condition “overpowers” the other (Yang et al., 20 Jul 2025).

OmniVTON++ replaces this single-initialization strategy with Principal Pose Guidance. It operates within the Discrete Diffusion Codebook Model framework, where at timestep tt,

zt−1=μθ(zt,t,c)+σt ϵt,ϵt∼Ct.z_{t-1} = \mu_\theta(z_t,t,c)+\sigma_t\,\epsilon_t, \quad \epsilon_t\sim C_t.

Because the target image z0z_0 is unavailable, the method builds a proxy image IproxyI_{\mathrm{proxy}} containing pose information but not the original garment texture. The proxy is formed by ordered region composition: background recovery with inpainting over the garment mask, body completion using average skin color chc_h, target-garment cues colored with average garment color ctc_t, and preserved regions copied from IpI_p (Yang et al., 16 Feb 2026).

At each diffusion step, OmniVTON++ performs Pose-Guided Noise Selection: kt=arg⁡max⁡1≤k≤K⟨Ct(k), zproxy−zˉ0∣tPCA⟩,(12)k_t =\arg\max_{1\le k\le K} \Big\langle C_t(k),\, z_{\mathrm{proxy}} - \bar z_{0|t}^{\mathrm{PCA}} \Big\rangle, \tag{12} where zˉ0∣tPCA\bar z_{0|t}^{\mathrm{PCA}} retains only the top few principal components of zˉ0∣t\bar z_{0|t}. The update becomes

zt−1=αt−1  μθ(zt,t,c)+1−αt−1−σt2  ϵθ(zt,t,c)+σt Ct(kt).(13)z_{t-1} = \sqrt{\alpha_{t-1}}\;\mu_\theta(z_t,t,c) +\sqrt{1-\alpha_{t-1}-\sigma_t^2}\;\epsilon_\theta(z_t,t,c) +\sigma_t\,C_t(k_t). \tag{13}

The stated purpose is to steer the sampling subspace along pose dimensions while leaving orthogonal components free to model garment texture and lighting (Yang et al., 16 Feb 2026).

5. Empirical performance and demonstrated scope

The original OmniVTON was evaluated on VITON-HD, DressCode, and StreetTryOn under paired and unpaired settings using FID, SSIM, and LPIPS. On VITON-HD, with model checkpoints pre-trained on DressCode, it reports FIDu=9.621\mathrm{FID}_u = 9.621, FIDp=7.758\mathrm{FID}_p = 7.758, SSIMp=0.832\mathrm{SSIM}_p = 0.832, and LPIPSp=0.145\mathrm{LPIPS}_p = 0.145, compared with $23.035$, $20.460$, $0.812$, and $0.147$ for IDM-VTON. On DressCode, with VITON-HD pre-trained models, it reports FIDu=6.450\mathrm{FID}_u = 6.450, FIDp=5.335\mathrm{FID}_p = 5.335, SSIMp=0.865\mathrm{SSIM}_p = 0.865, and LPIPSp=0.119\mathrm{LPIPS}_p = 0.119, compared with $9.685$, $8.377$, $0.842$, and $0.138$ for IDM-VTON. On StreetTryOn, the four cross-scenario FID scores are Shop→Street $33.92$, Model→Model $8.98$, Model→Street $33.45$, and Street→Street $23.47$ (Yang et al., 20 Jul 2025).

A notable capability claim in the original paper is that OmniVTON is the first framework capable of multi-human VTON. The described mechanism is horizontal concatenation of multiple IcI_c, generation of multiple pseudo-persons, and independent warping for each (Yang et al., 20 Jul 2025).

The ablation study on VITON-HD isolates the contribution of the three core modules. The base text-only system reports $18.45$, $16.88$, $0.773$, and $0.222$ for FIDu\mathrm{FID}_u, FIDp\mathrm{FID}_p, SSIMp\mathrm{SSIM}_p, and LPIPSp\mathrm{LPIPS}_p. Adding SGM only yields $13.30$, $11.48$, $0.809$, and $0.177$. Adding SGM+CBS yields $9.80$, $7.99$, $0.824$, and $0.158$. SPI only gives $13.15$, $10.77$, $0.813$, and $0.180$. The full OmniVTON gives $9.62$, $7.76$, $0.832$, and $0.145$ (Yang et al., 20 Jul 2025).

OmniVTON++ evaluates cross-dataset and cross-garment-type generalization using unpaired FID and, where ground truth exists, paired FID, SSIM, and LPIPS. The paper states that in every setting, OmniVTON++ in both SD-2.0 and FLUX variants ranks first or second on FID, SSIM, and LPIPS, often improving over the original OmniVTON by $5$–10%10\%. Its ablation study reports, for example, that adding SGM to a vanilla diffusion base cuts unpaired FID on VITON-HD SD-2.0 from $18.44$ to $12.95$; PPG further reduces FID to $12.32$ and raises SSIM from $0.773$ to $0.827$; adding CBS then yields $9.33$ FID and $0.821$ SSIM; and all three modules together yield $9.19$ FID and $0.843$ SSIM (Yang et al., 16 Feb 2026).

Within the broader omni-VTON/VTOFF problem, OmniDiT reports results on model-based VTON, model-free VTON, and Try-Off. On VITON-HD model-based VTON, it reports FID=6.46\mathrm{FID}=6.46, KID=0.75\mathrm{KID}=0.75, SSIM=0.884\mathrm{SSIM}=0.884, and LPIPS=0.078\mathrm{LPIPS}=0.078; on DressCode, FID=3.45\mathrm{FID}=3.45, KID=0.52\mathrm{KID}=0.52, SSIM=0.898\mathrm{SSIM}=0.898, and LPIPS=0.079\mathrm{LPIPS}=0.079; and on Omni-TryOn, FID=15.96\mathrm{FID}=15.96, KID=1.43\mathrm{KID}=1.43, SSIM=0.712\mathrm{SSIM}=0.712, and LPIPS=0.173\mathrm{LPIPS}=0.173. For model-free VTON on VITON-HD, it reports FID=12.44\mathrm{FID}=12.44, KID=3.25\mathrm{KID}=3.25, SSIM=0.701\mathrm{SSIM}=0.701, and LPIPS=0.284\mathrm{LPIPS}=0.284. For Try-Off on VITON-HD, it reports FID=10.55\mathrm{FID}=10.55, CLIP ⁣− ⁣I=0.939\mathrm{CLIP\!-\!I}=0.939, LPIPS=0.186\mathrm{LPIPS}=0.186, KID=1.56\mathrm{KID}=1.56, and DISTS=0.196\mathrm{DISTS}=0.196 (Zeng et al., 20 Mar 2026).

6. Extensions, unification, and limitations

OmniVTON++ is presented as an extension over the ICCV ’25 OmniVTON. Relative to the original system, SGM replaces attention-modulated outpainting with a garment-centric virtual dressing module for more reliable pseudo-person geometry; PPG replaces single-step Spectral Pose Injection with step-wise pose guidance via codebook noise selection on principal components; CBS is extended to DiT backbones via Positional Index Realignment, yielding CBS-DiT; and the evaluation adds robustness tests across backbones and scenarios as well as multi-garment, multi-human, and anime character virtual try-on (Yang et al., 16 Feb 2026).

A separate line of development is OmniDiT, which is not training-free. It proposes an omni Virtual Try-On framework based on the Diffusion Transformer and combines try-on and try-off tasks into one unified model. The method builds a large VTON dataset, Omni-TryOn, through a self-evolving curation pipeline. The dataset contains 382 223382\,223 samples, approximately 335 146335\,146 unique garments, spans $23$ categories, and holds out 1 8951\,895 stratified samples for testing. Its architecture concatenates text tokens, noisy image latent tokens, and multiple reference-image tokens into a single sequence,

S=[T;X;C1;… ;Cn],S=[T;X;C_1;\dots;C_n],

augmented with adaptive positional encoding using “3D RoPE-style” position indices (i,w,h)(i,w,h) (Zeng et al., 20 Mar 2026).

To control the computational cost of long token sequences, OmniDiT introduces Shifted Window Attention into the diffusion model. The paper states that global self-attention over a sequence of length LL costs O(L2)O(L^2), whereas the shifted-window formulation reduces this to O((WH⋅nref)⋅M2)O((WH\cdot n_{\mathrm{ref}})\cdot M^2) per layer. On an A800 with two 1024×10241024\times1024 inputs and a 1024×10241024\times1024 output, SWA cuts inference time from 55 s55\,\mathrm{s} to 47 s47\,\mathrm{s}, a reduction of 14.5%14.5\% (Zeng et al., 20 Mar 2026). OmniDiT also introduces multiple timestep prediction and an alignment loss

Lalign=1−cos⁡(E(M⊙GT),E(M⊙G)),\mathcal{L}_{\mathrm{align}} = 1 - \cos(\mathcal{E}(M\odot GT), \mathcal{E}(M\odot G)),

combined as

L=LMTP+λLalign,λ=0.1.\mathcal{L} = \mathcal{L}_{\mathrm{MTP}} + \lambda \mathcal{L}_{\mathrm{align}}, \quad \lambda=0.1.

The reported ablation shows that MTP with K=2K=2 lowers FID from $8.51$ to $7.61$ on VITON-HD, and adding alignment further lowers FID to $6.46$, with KID=0.75\mathrm{KID}=0.75 and LPIPS=0.078\mathrm{LPIPS}=0.078 (Zeng et al., 20 Mar 2026).

The limitations explicitly documented for OmniVTON++ remain relevant to the training-free line. The method depends on external analysis tools: misdetected keypoints or failed parsing or DensePose can impair SGM and PPG. Accessories such as necklaces and hats may be mistakenly masked and inpainted over; the paper suggests that one may mitigate this by incorporating a segmentation model such as SAM 3 to exclude such regions. The approach is also currently limited to single-view images, and true 3D consistency or video stability remains open (Yang et al., 16 Feb 2026).

A common interpretation of the sequence from OmniVTON to OmniVTON++ and OmniDiT is that the field has explored two complementary routes to universality. One route preserves the training-free premise and improves conditioning fidelity through better geometry, step-wise pose control, and boundary-aware attention. The other route adopts a unified trained transformer with large-scale data curation and multi-task supervision.

Topic to Video (Beta)

No one has generated a video about this topic yet.

Whiteboard

No one has generated a whiteboard explanation for this topic yet.

Follow Topic

Get notified by email when new papers are published related to OmniVTON.