Semi-Supervised StyleGAN for Disentanglement Learning (2003.03461v3)

Abstract: Disentanglement learning is crucial for obtaining disentangled representations and controllable generation. Current disentanglement methods face several inherent limitations: difficulty with high-resolution images, primarily focusing on learning disentangled representations, and non-identifiability due to the unsupervised setting. To alleviate these limitations, we design new architectures and loss functions based on StyleGAN (Karras et al., 2019), for semi-supervised high-resolution disentanglement learning. We create two complex high-resolution synthetic datasets for systematic testing. We investigate the impact of limited supervision and find that using only 0.25%~2.5% of labeled data is sufficient for good disentanglement on both synthetic and real datasets. We propose new metrics to quantify generator controllability, and observe there may exist a crucial trade-off between disentangled representation learning and controllable generation. We also consider semantic fine-grained image editing to achieve better generalization to unseen images.

• The paper introduces Info-StyleGAN and Semi-StyleGAN architectures that achieve near full-supervision performance using only 0.25% to 2.5% labeled data.
• It proposes novel metrics like MIG-gen and L2-gen to evaluate the trade-off between disentangled representation and generator controllability.
• The approach enables fine-grained, high-resolution image editing and semantic translation, paving the way for practical, controllable generation.

Semi-Supervised StyleGAN for Disentanglement Learning

The paper focuses on semi-supervised disentanglement learning using StyleGAN, exploring ways to overcome limitations inherent to current disentanglement methods. Disentanglement learning in deep generative models is critical for acquiring disentangled representations for controllable generation. Current methods face significant challenges such as problems with high-resolution images, a primary focus on representation rather than generation, and issues due to non-identifiability in unsupervised settings.

Main Contributions

1. Advancements in StyleGAN Architecture: The authors present Info-StyleGAN, an augmentation of StyleGAN with mutual information loss. This approach outperforms existing unsupervised methods, evidencing the structural benefits of StyleGAN for disentanglement learning.
2. Semi-StyleGAN Architecture: The introduction of Semi-StyleGAN achieves near fully-supervised disentanglement with limited supervision (0.25\% to 2.5\%) on synthetic and real datasets, offering a practical solution for improving control in generation.
3. New Disentanglement Metrics: They propose metrics like MIG-gen and L2-gen to evaluate generator controllability, highlighting a trade-off between learning disentangled representations and achieving controllable generation.
4. Image-to-Image Translation: Semi-StyleGAN-fine is an extension to facilitate semantic fine-grained image editing, thus improving generalization to unseen images.
5. High-Resolution Synthetic Datasets: Two new datasets, Falcor3D and Isaac3D, are introduced, with high resolution and rich factors of variation, offering scalable and reliable environments for disentanglement testing.

Numerical Results and Insights

The empirical evaluations underscore the sufficiency of minimal labeled data in achieving effective disentanglement. Using only 0.25\% to 2.5\% of labeled data, Semi-StyleGAN approximates the performance of fully-supervised approaches, underscoring significant improvements over unsupervised techniques. This highlights the role limited supervision plays in overcoming non-identifiability and driving efficient disentanglement learning. Also notable are the results demonstrating a trade-off between encoder disentanglement and generator controllability, crucial for practical applications where controlling generation is paramount.

Theoretical and Practical Implications

This paper emphasizes the necessity of developing models that jointly focus on disentangled representation learning and controllable generation. The introduction of metrics for generator controllability signifies progress towards better evaluation and comparison of generative models. The paper indicates a promising direction for enhancing the generalization capabilities of GAN architectures, particularly via Semi-StyleGAN-fine, potentially expanding into broader real-world applications that demand semantic fine-grained editing capabilities.

Future Directions

The authors propose scaling up Semi-StyleGAN to more extensive high-resolution datasets, addressing biases in existing datasets such as CelebA, and exploring extensions in weakly-supervised scenarios. Additionally, further investigation into the interplay between structural inductive biases and explicit supervision could offer deeper insights into improving both representation and generative fidelity in complex, high-resolution domains.

Overall, this paper makes substantial technical contributions to disentanglement learning, providing methodologies that potentially bridge gaps between high-quality image generation and precise feature attribution control. The frameworks proposed could significantly impact future advancements in generative models applied to real-world scenarios, where clarity in feature manipulation and control is of utmost importance.