- The paper proposes a cacheable diffusion framework that decouples reference encoding from denoising, achieving a 3.5× inference speedup.
- It introduces a Cacheable UNet with a Semi-Attention mechanism and Reference KV caching to maintain fidelity in multi-item compositions.
- Empirical results on the DressCode-MR dataset show superior FID (9.311) and KID (1.512) scores, validating efficient multi-reference try-on.
FastFit: Cacheable Diffusion Models for Efficient Multi-Reference Virtual Try-On
Introduction
The FastFit framework addresses two critical limitations in virtual try-on systems: (1) the lack of support for multi-reference outfit composition, including garments and accessories, and (2) significant computational inefficiency due to redundant reference feature computation at each denoising step in diffusion-based architectures. FastFit introduces a cacheable diffusion model architecture that decouples reference feature encoding from the iterative denoising process, enabling high-fidelity, multi-reference virtual try-on with substantial acceleration in inference speed. The framework is supported by the DressCode-MR dataset, which provides large-scale, high-quality multi-reference outfit pairings across five key categories.
Architectural Innovations
Cacheable UNet
FastFit's core innovation is the Cacheable UNet, which enables reference features to be computed once and reused losslessly throughout the denoising process. This is achieved by:
- Reference Class Embedding: Each reference item (e.g., top, shoes) is associated with a static, learnable class embedding, replacing the conventional timestep embedding. This ensures that reference features are independent of the denoising timestep and thus cacheable.
- Semi-Attention Mechanism: The attention mask restricts information flow such that denoising features can attend to all reference features, but reference features only attend to themselves. This prevents contamination of cached reference features by dynamic denoising features, maintaining their static nature.
Reference KV Cache
During inference, reference features are pre-computed and cached as Key/Value pairs for each Semi-Attention layer. In each denoising step, only the dynamic denoising features are processed, and the cached reference features are concatenated for attention computation. This design eliminates redundant computation and scales efficiently with the number of reference items.
Conditioning Pipeline
- Person Conditioning: Combines a cloth-agnostic mask and pose skeleton to preserve identity and pose.
- Reference Conditioning: Encodes each reference item via a VAE encoder, conditioned on its class embedding.
The denoising UNet processes noisy latents, person conditioning, and timestep embedding, integrating static reference information via Semi-Attention.
Empirical Results
Efficiency
FastFit achieves an average 3.5× speedup in inference over comparable methods, with negligible parameter overhead (904.86M parameters, 1.16s inference time, 6944MB peak memory on H100 GPU). Ablation studies confirm that the Reference KV Cache and Semi-Attention are essential for efficiency and quality; disabling these components increases latency and degrades fidelity.
Fidelity
On VITON-HD, DressCode, and DressCode-MR datasets, FastFit consistently matches or surpasses state-of-the-art methods in SSIM, LPIPS, FID, and KID metrics. Notably, in multi-reference scenarios, FastFit achieves the best FID (9.311) and KID (1.512) on DressCode-MR, with competitive SSIM (0.859) and LPIPS (0.079). Qualitative results demonstrate superior preservation of fine-grained garment details and coherent composition of multiple items.
Generalization
Category-wise analysis on DressCode shows robust performance across upper, lower, and dress categories, with FastFit achieving best or second-best scores in most metrics. The model generalizes well to diverse garment types and complex outfit compositions.
Dataset Contribution
DressCode-MR is the first large-scale dataset for multi-reference virtual try-on, comprising 28,179 samples with paired person and canonical item images across five categories. The dataset is constructed via expert restoration models and human feedback, providing a foundation for future research in complex outfit generation.
Implementation Considerations
- Training: FastFit is trained on StableDiffusion v1.5 inpainting, with fine-tuning on DressCode-MR. AdamW optimizer, batch size 32, 1024×768 resolution, and classifier-free guidance are used.
- Hardware: All experiments utilize 8 NVIDIA H100 GPUs.
- Integration: The cacheable architecture can be implemented in PyTorch using custom attention masks and static embeddings. The reference KV cache requires careful management of pre-computed tensors for efficient concatenation during inference.
Example: Semi-Attention Mask Construction
1
2
3
4
5
6
7
8
9
10
11
12
13
|
import torch
def semi_attention_mask(num_denoise_tokens, num_ref_tokens):
# Denoising tokens attend to all; reference tokens attend only to themselves
total_tokens = num_denoise_tokens + num_ref_tokens
mask = torch.zeros(total_tokens, total_tokens)
# Denoising-to-all
mask[:num_denoise_tokens, :] = 1
# Reference-to-self
for i in range(num_ref_tokens):
idx = num_denoise_tokens + i
mask[idx, idx] = 1
return mask |
Example: Reference Feature Caching
1
2
|
ref_features = [unet(ref_latent, class_embed) for ref_latent, class_embed in zip(ref_latents, class_embeds)]
ref_kv_cache = [compute_kv(feature) for feature in ref_features] |
Trade-offs and Limitations
- Scalability: The cacheable design scales efficiently with the number of reference items, but memory usage increases linearly with the number of cached features.
- Physical Interactions: Current modeling does not explicitly handle complex garment layering or physical interactions.
- Real-Time Inference: While FastFit significantly accelerates inference, further optimization (e.g., step distillation, advanced caching) is required for real-time applications.
Implications and Future Directions
FastFit's cacheable diffusion architecture sets a new standard for efficient, high-fidelity multi-reference virtual try-on. The decoupling of reference feature encoding from the denoising process is broadly applicable to other subject-driven generation tasks requiring multi-modal conditioning. The DressCode-MR dataset will catalyze research into more complex outfit synthesis and evaluation.
Future work should address modeling of garment interactions, expand dataset diversity, and pursue real-time inference via further architectural and algorithmic optimizations. The cacheable conditioning paradigm may be extended to other domains, such as multi-object scene synthesis and interactive design systems.
Conclusion
FastFit demonstrates that cacheable diffusion models, enabled by static class embeddings and semi-attention, can fundamentally resolve the efficiency bottleneck in multi-reference virtual try-on. The framework achieves state-of-the-art fidelity and efficiency, supports coherent composition across five garment categories, and is validated on a new large-scale dataset. These advances substantially lower the barriers to practical deployment of virtual try-on technology in e-commerce and intelligent outfit visualization, and provide a blueprint for efficient multi-modal conditioning in generative models.
