DreamVVT: Two-Stage Video Virtual Try-On

• The paper presents DreamVVT, a two-stage video virtual try-on framework that decouples high-fidelity keyframe synthesis from temporally consistent video generation.
• It leverages multi-modal inputs—including paired/unpaired human-centric data, VLM priors, and pose-masked images—using Diffusion Transformers with LoRA adaptations to preserve fine garment details.
• Experimental results demonstrate improved physical realism and temporal stability, outperforming previous methods in both quantitative metrics and human evaluations.

DreamVVT is a two-stage framework for video virtual try-on (VVT) designed to deliver realistic garment fitting with high temporal coherence under unconstrained real-world scenarios. Building on Diffusion Transformers (DiTs), DreamVVT leverages both paired and unpaired human-centric data, incorporates Vision–LLM (VLM) priors, and employs a stage-wise decomposition to disentangle appearance synthesis from video generation. The framework is engineered to overcome limitations of previous end-to-end VVT approaches, particularly in preserving fine-grained garment details and maintaining motion consistency when applied to diverse, uncurated data (Zuo et al., 4 Aug 2025).

1. Framework Architecture and Stages

DreamVVT decomposes VVT into two distinct stages:

1. High-Fidelity Keyframe Try-On Synthesis (Stage 1):
   • Representative video frames are sampled and synthesized into high-fidelity "keyframe" try-on images via a multi-frame DiT model.
   • These images serve as detailed appearance references.
2. Video Generation with Long-Term Temporal Consistency (Stage 2):
   • Leveraging extracted pose, motion, and textual conditions, together with keyframe try-on images, the system generates the full try-on video.
   • Temporal coherence is ensured via a pre-trained video DiT enhanced with low-rank adaptation (LoRA).

This stage-wise design decouples the complexities of garment appearance fitting and spatiotemporal synthesis, improving generalization to in-the-wild scenarios and unpaired real-world data sources.

2. Keyframe Sampling and Try-On Synthesis

The Stage 1 pipeline begins by selecting keyframes from the input video based on both pose and visual scale similarity. The algorithm:

• Defines the anchor frame as a canonical A-pose frontal image (f_anchor).
• Computes, for each frame f_i, the RTMPose-generated skeletal direction vector D_v[i].
• Scores each frame using cosine similarity ($S_m[i] = D_{anchor} \cdot D_v[i]$) and subject-to-frame area ratio ($S_r[i] = A_{subject}/A_{frame}$), yielding $S_{final}[i] = S_m[i] + \lambda S_r[i]$ (with $\lambda = 0.3$).
• Sorts frames by $S_{final}$ and selects K=2 frames such that their scores differ by at least $\alpha \cdot \text{mean}(S_{final})$, $\alpha = 0.2$.

The multi-frame try-on model uses a Seedream 3.0 DiT backbone (stacked MMDiT U-Net blocks) with conditional input streams: pose-cropped and mask-occluded keyframes, segmented garment images, and rewritten textual instructions from Seed1.5-VL. The architecture follows a parallel-branch paradigm, concatenating Q, K, and V tokens across main (latent noise) and reference (appearance) streams at every attention layer. LoRA adapters enable efficient fine-tuning by injecting low-rank updates into each attention weight ($r=256$, ~10% trainable parameters per stream).

The forward process follows standard DiT diffusion, and denoising is guided by classifier-free formulation, using both image and text conditions. The rectified flow simple loss is minimized:

$$L_{stage1} = \mathbb{E}_{t, x_0, \epsilon}[ \| \epsilon - \epsilon_\theta(x_t, c, t) \|^2 ].$$

3. Video Generation with Temporal Consistency

Stage 2 synthesizes the output video by integrating multi-modal priors and maintaining long-range temporal alignment. The conditioning streams are:

• Pose Guider: Skeleton maps ($S \in \mathbb{R}^{T \times H \times W \times K}$) are encoded into pose latents ($S_r[i] = A_{subject}/A_{frame}$0) using temporal self-attention.
• Agnostic Inputs: Original clothing is masked, and features are encoded via a Video VAE, yielding $S_r[i] = A_{subject}/A_{frame}$1 and mask $S_r[i] = A_{subject}/A_{frame}$2.
• Keyframe Appearance: Stage 1 try-on images are encoded to $S_r[i] = A_{subject}/A_{frame}$3 using the same VAE.
• Motion and Text: Qwen2.5-VL extracts structured JSON ({ENVIRONMENT, APPEARANCE, MOTION}); the original APPEARANCE is replaced. Tokenized prompts create $S_r[i] = A_{subject}/A_{frame}$4.

The video generator is a DiT-based image-to-video U-Net with cross-frame full self-attention. All weights are frozen except Q, K, V projections of image and video streams, which receive LoRA adapters (~10% of parameters). Tokens from $S_r[i] = A_{subject}/A_{frame}$5 are concatenated, allowing for adaptive spatial-temporal fusion within the MMDiT blocks. The generation follows a standard diffusion process with Euler scheduling (50 steps, guidance=2.5). Final video is reconstructed via VAE decoding and Laplacian pyramid blending for seamless background integration.

Training employs the rectified-flow L2 loss with a multi-task schedule, randomly sampling objectives (e.g., text$S_r[i] = A_{subject}/A_{frame}$6video, pose+keyframes+text$S_r[i] = A_{subject}/A_{frame}$7video). No explicit adversarial objective is used, relying on self-attention and task mixing for temporal stability.

4. Technical and Implementation Details

Training utilizes a mix of 1.01M paired garment–frame images (with multi-view captions) and 187K filtered unpaired video clips, augmenting with 102K raw videos for robustness. The DiT models are pre-trained on large-scale generic image/video datasets; Stage 1 and Stage 2 are trained with AdamW (learning rate $S_r[i] = A_{subject}/A_{frame}$8, weight decay 0.01, grad clip 1.0), on 8×NVIDIA H20 (96 GB) GPUs over 10 days.

Loss functions include:

• Pixel-wise reconstruction: $S_r[i] = A_{subject}/A_{frame}$9
• Perceptual: $S_{final}[i] = S_m[i] + \lambda S_r[i]$0
• Temporal (optional fine-tuning): $S_{final}[i] = S_m[i] + \lambda S_r[i]$1
• Total: $S_{final}[i] = S_m[i] + \lambda S_r[i]$2

Vision–language integration is realized through Seed1.5-VL and Qwen2.5-VL, which provide structured scene captions for consistent multimodal fusion via cross-attention in DiT layers.

5. Experimental Results and Comparative Performance

Evaluation employed the ViViD-S benchmark (combining paired and unpaired videos) and Wild-TryOnBench (human evaluation). Results for DreamVVT and the baseline MagicTryON:

Method	VFID_I ↓	VFID_R ↓	SSIM ↑	LPIPS ↓
DreamVVT	11.02	0.255	0.874	0.062
MagicTryON	12.20	0.235	0.884	0.082

Human evaluations (Wild-TryOnBench):

Metric	DreamVVT (mean/5)
Garment Preservation	3.41
Physical Realism	3.69
Temporal Consistency	3.32

Ablations show that using K=2 keyframes (vs. K=1) increases garment detail by 0.3 points, realism by 0.1, and temporal consistency by 0.03. LoRA adaptation yields a 0.2 improvement in realism with no degradation in other metrics compared to full model fine-tuning.

6. Strengths, Limitations, and Extensions

DreamVVT's decoupled architecture allows for robust garment appearance synthesis (Stage 1) and effective spatiotemporal video generation (Stage 2), leading to improved performance in uncontrolled, in-the-wild settings. The method generalizes by leveraging unpaired real-world data and pre-trained DiTs plus VLM priors. Flexible multi-modal fusion (keyframes, pose, text) underpins both high-fidelity rendering and smooth motion.

Current constraints include the use of agnostic masks, which may occlude extensive regions; mask-free editing is a prospective direction. Performance degrades with extreme garment occlusion or complex interactions, indicating the need for improved captioning and generative priors. Extensions may encompass integration with 3D pose and motion priors, adaptation to dynamic lighting, and enabling real-time interactive try-on.

7. Significance and Broader Impact

By combining a two-stage DiT framework, low-rank adaptation, and structured VLM priors, DreamVVT advances realistic, temporally coherent video virtual try-on in unconstrained scenarios, supporting applications in e-commerce, advertising, and entertainment. Its methodological innovations address data scarcity and adaptation challenges with a modular, extensible design (Zuo et al., 4 Aug 2025).