GANSpace: Discovering Interpretable GAN Controls (2004.02546v3)

Abstract: This paper describes a simple technique to analyze Generative Adversarial Networks (GANs) and create interpretable controls for image synthesis, such as change of viewpoint, aging, lighting, and time of day. We identify important latent directions based on Principal Components Analysis (PCA) applied either in latent space or feature space. Then, we show that a large number of interpretable controls can be defined by layer-wise perturbation along the principal directions. Moreover, we show that BigGAN can be controlled with layer-wise inputs in a StyleGAN-like manner. We show results on different GANs trained on various datasets, and demonstrate good qualitative matches to edit directions found through earlier supervised approaches.

A Study of Interpretable Controls in GANs via PCA and Layer-wise Perturbation

This paper presents a method for discovering interpretable control mechanisms in Generative Adversarial Networks (GANs). The researchers employ Principal Component Analysis (PCA) to identify significant latent directions, which can then be manipulated to influence image synthesis outcomes. The central insight is that PCA on intermediate feature spaces—whether in latent space for StyleGAN or feature space for BigGAN—reveals principal directions that control important aspects of generated images.

The authors' approach simplifies the otherwise complex process of generating controlled image modifications, offering both practical and theoretical contributions.

Methodology

Principal Component Analysis on Latent Space

For StyleGAN architectures, the authors perform PCA directly in the latent space $\mathcal{W}$. By sampling a large number of latent vectors $z$ and mapping them through the learned network to obtain their corresponding $w$ values, they compute a PCA basis in $\mathcal{W}$ space. This basis captures principal directions along which the variations in the generated images occur.

For BigGAN, PCA is applied in the feature space at an intermediate layer where high-level abstractions begin to form. To transfer these principal directions back to the latent space $z$, they use a regression approach. This involves representing each principal component by a corresponding direction in the latent space. The result is an array of latent directions $U$ that mimic the effects of PCA components derived from feature spaces.

Layer-wise Perturbations

A significant aspect of the paper's contribution is the layer-wise application of these principal directions. For StyleGAN, this involves applying edits to specific subsets of network layers. For instance, modifying only the initial layers $G_1 .. G_i$ captures geometric transformations like head rotation, while changes to later layers $G_{i+1} .. G_L$ affect finer stylistic aspects such as lighting and texture. This separation yields more interpretable and disentangled controls over generated images.

Similarly, for BigGAN, the researchers extend layer-wise controls by varying the input latents across different layers, enabling complex adjustments without needing modifications to the GAN’s training process.

Numerical Results and Evaluation

The models evaluated include BigGAN512-deep, StyleGAN, and StyleGAN2 trained on various datasets like FFHQ, Cars, Cats, and Landscape datasets. The numerical results show that a relatively small number of principal components (the first 100 out of 512) account for significant variations in image generation. The remaining components fine-tune these variations, adding subtle details and texture enhancements.

Comparisons with random directions elucidate the advantage of PCA-derived directions. Randomized tests demonstrate that directions identified via PCA enable more meaningful and targeted edits—such as geometric transformations remaining faithful to object structures—compared to arbitrary navigations through the latent space.

Notably, the layer-wise decomposition isolates effects effectively; for example, head rotation can be achieved without the unintended entanglement with other factors like gender. This level of control is aligned remarkably well with supervised edits effectuated in prior studies, but without requiring labeled data.

Implications and Future Directions

Practical Implications

The ability to perform fine-grained, interpretable edits on GAN-generated images has wide-ranging applications, from enhancing artistic workflows to refining synthetic data generation for training machine learning models. The method established here circumvents the need for extensive supervision, making it broadly applicable without additional training data, thus reducing computational overhead.

Theoretical Implications

The findings underscore the inherent structure in GAN latent spaces, revealing that principal directions are class-agnostic for BigGAN and relate well to functional aspects of images (e.g., geometry in early directions, style in later ones). The demonstrated independence and simplicity of the latent space distributions hint at potential simplifications for GAN architectures and sampling mechanisms.

Future Work

Identifying more nuanced directions and developing methods to automatically label and interpret these directions remain prominent future directions. Investigating unsupervised alternatives to PCA, or other data-efficient methods, could yield further enhancements. Additionally, comparing results with methods in concurrent GAN research may offer comprehensive insights into GAN interpretability and control.

Overall, the paper advances our understanding of GAN mechanics, presenting practical tools and theoretical insights that open the door to more controlled and interpretable generative models. This stride in understanding GANs without supervision marks a feasibly implementable yet impactful breakthrough in deep learning.