CAT-DM: Controllable Accelerated Virtual Try-on with Diffusion Model (2311.18405v2)

Abstract: Generative Adversarial Networks (GANs) dominate the research field in image-based virtual try-on, but have not resolved problems such as unnatural deformation of garments and the blurry generation quality. While the generative quality of diffusion models is impressive, achieving controllability poses a significant challenge when applying it to virtual try-on and multiple denoising iterations limit its potential for real-time applications. In this paper, we propose Controllable Accelerated virtual Try-on with Diffusion Model (CAT-DM). To enhance the controllability, a basic diffusion-based virtual try-on network is designed, which utilizes ControlNet to introduce additional control conditions and improves the feature extraction of garment images. In terms of acceleration, CAT-DM initiates a reverse denoising process with an implicit distribution generated by a pre-trained GAN-based model. Compared with previous try-on methods based on diffusion models, CAT-DM not only retains the pattern and texture details of the inshop garment but also reduces the sampling steps without compromising generation quality. Extensive experiments demonstrate the superiority of CAT-DM against both GANbased and diffusion-based methods in producing more realistic images and accurately reproducing garment patterns.

• The paper introduces a novel hybrid framework combining diffusion models with GANs to provide controllable and 25-fold accelerated virtual try-on performance.
• It integrates a garment-conditioned diffusion model with ControlNet and DINO-V2, ensuring precise replication of complex garment details and textures.
• The approach outperforms state-of-the-art models on benchmarks like DressCode and VITON-HD while reducing training time and resource requirements.

The paper "CAT-DM: Controllable Accelerated Virtual Try-on with Diffusion Model" introduces a novel approach to virtual try-on systems by leveraging the strengths of diffusion models while addressing their inherent limitations in terms of controllability and efficiency. This work combines diffusion models with generative adversarial networks (GANs) to enhance image fidelity and reduce inference time, which is critical for real-time applications.

Main Contributions:

1. Controllable Accelerated Virtual Try-On Model (CAT-DM): The authors propose an architecture that integrates a diffusion-based model with GANs to achieve both high controllability and accelerated image synthesis. This hybrid approach capitalizes on the robust generative capabilities of diffusion models and the stable sampling efficiencies provided by GANs.
2. Garment-Conditioned Diffusion Model (GC-DM): The core component of CAT-DM, GC-DM, incorporates ControlNet to provide additional control conditions. This allows for finer manipulation of garment features and ensures that complex patterns and textures are accurately recreated in virtual try-on images. The use of advanced feature extraction techniques, like DINO-V2, further enhances the detail and realism of generated apparel.
3. Truncation-Based Acceleration Strategy: The model begins the reverse denoising process not from Gaussian noise but from an initial state generated by a pre-trained GAN. This significantly reduces the number of sampling steps required, achieving a 25-fold acceleration compared to typical diffusion models. The authors utilize a method inspired by the Truncated Diffusion Probabilistic Models (TDPM) to robustly integrate this acceleration mechanism.

Experimental Evaluation:

• The proposed method demonstrates superior performance across several benchmarks including DressCode and VITON-HD datasets, beating state-of-the-art models in terms of FID, KID, SSIM, and LPIPS metrics. This indicates an improvement in both the perceptual quality and the realism of generated images.
• Extensive experiments validate the ability of CAT-DM to maintain garment consistency and adapt to varying poses and garment types, outperforming other GAN-based and diffusion-based methods in generating realistic images that faithfully reproduce garment patterns.
• The model's architecture, which freezes the majority of diffusion model parameters, significantly reduces training time and resources, making it suitable for practical applications.

Technical Insights:

• ControlNet Integration for Controllability: By leveraging ControlNet, the authors effectively introduce additional conditional variables that steer the diffusion process, thus improving the pixel-level control over garment representations. This architecture ensures that garment alterations remain semantically and contextually accurate.
• Feature Extraction with DINO-V2: The transition from CLIP to DINO-V2 as the feature extractor marks a significant enhancement, allowing the model to receive more comprehensive inputs by preserving local and global garment details.
• Use of Poisson Blending: This post-processing technique ensures that generated try-on images blend seamlessly with original images, eliminating stitching artifacts common in naive image concatenation.

In conclusion, the paper makes significant strides in advancing virtual try-on technology through innovative use of diffusion models and GANs, setting a new benchmark in terms of both quality and efficiency. The proposed techniques provide a foundation for further exploration and development in real-time fashion retail applications.