A Variational U-Net for Conditional Appearance and Shape Generation (1804.04694v1)

Abstract: Deep generative models have demonstrated great performance in image synthesis. However, results deteriorate in case of spatial deformations, since they generate images of objects directly, rather than modeling the intricate interplay of their inherent shape and appearance. We present a conditional U-Net for shape-guided image generation, conditioned on the output of a variational autoencoder for appearance. The approach is trained end-to-end on images, without requiring samples of the same object with varying pose or appearance. Experiments show that the model enables conditional image generation and transfer. Therefore, either shape or appearance can be retained from a query image, while freely altering the other. Moreover, appearance can be sampled due to its stochastic latent representation, while preserving shape. In quantitative and qualitative experiments on COCO, DeepFashion, shoes, Market-1501 and handbags, the approach demonstrates significant improvements over the state-of-the-art.

• The paper introduces a Variational U-Net that disentangles shape and appearance by modeling them as separate latent variables.
• It combines a conditional U-Net with a variational autoencoder to enable high-fidelity, shape-conditioned image generation without multi-view datasets.
• Experimental results show improved SSIM and Inception Scores over state-of-the-art models, underscoring its potential in graphics and AR applications.

A Variational U-Net for Conditional Appearance and Shape Generation

This paper introduces a novel approach to conditional image generation that emphasizes the nuanced interplay between an object's shape and appearance. The proposed methodology features a Variational U-Net architecture, which is designed to enhance deep generative models’ performance, especially when handling complex spatial deformations. By separately modeling shape and appearance, the paper aims to overcome the limitations of existing generative models that struggle with large spatial variations.

Key Contributions

The core contribution of this work is the integration of a conditional U-Net with a variational autoencoder (VAE), enabling the independent manipulation and generation of object shape and appearance. The model is trained end-to-end on static images, negating the necessity for datasets containing multiple perspectives of identical objects. This capacity allows for a variety of applications, such as transferring an object’s appearance to a different shape or synthesizing a new appearance while maintaining an existing shape.

Methodology

Significantly, the model posits shape and appearance as two separate latent variables, a design choice that facilitates their disentanglement during the generative process. The approach involves:

• Shape Estimation: Used as a conditioning factor, shape information is extracted via automatic methods like edge detection or joint location estimation, reflecting geometric properties such as layout and pose.
• Variational Autoencoders: These are utilized to handle the stochastic nature of appearance representation, allowing for sampling of diverse appearance while maintaining a given shape.
• U-Net Architecture: This ensures the preservation of spatial information inherent in shape, facilitating high-fidelity image synthesis by directly mapping from shape embeddings to the resulting image.

Experimental Results

The empirical evaluation shows robust improvement over state-of-the-art models, such as pix2pix and PG$^2$, across various datasets including COCO, DeepFashion, Market-1501, and others. Quantitative assessments using metrics like Structural Similarity Index (SSIM) and Inception Scores (IS) highlight this model’s ability to preserve appearance and shape, achieving higher fidelity reconstructions with substantial perceptual quality.

• SSIM Scores: The proposed model consistently outperformed comparison models, indicating superior preservation of structural information during reconstruction tasks.
• Inception Scores: Demonstrations of sampling capability showed not only diverse but also perceptually convincing appearance variation from shape-conditioned generation.

Implications and Future Directions

By independently modeling and manipulating shape and appearance, the work opens up intriguing practical implications for fields like computer graphics, augmented reality, and fashion design, offering sophisticated tools for image generation without requiring extensive pose-labeled datasets. Theoretically, the integration of VAEs with conditional U-Nets marks a promising direction for more granular control over generative processes.

Future research could refine the disentanglement strategies further and apply this framework to even more diverse and unstructured datasets. Additionally, extending the approach to handle complex scenes with multiple interacting objects could provide significant advancements in generative modeling of visual data.

In conclusion, the paper contributes a significant advancement in conditional image generation, addressing key limitations in handling spatial deformation by adeptly separating and recombining the visual semantics of shape and appearance.