Anycost GANs for Interactive Image Synthesis and Editing (2103.03243v1)

Abstract: Generative adversarial networks (GANs) have enabled photorealistic image synthesis and editing. However, due to the high computational cost of large-scale generators (e.g., StyleGAN2), it usually takes seconds to see the results of a single edit on edge devices, prohibiting interactive user experience. In this paper, we take inspirations from modern rendering software and propose Anycost GAN for interactive natural image editing. We train the Anycost GAN to support elastic resolutions and channels for faster image generation at versatile speeds. Running subsets of the full generator produce outputs that are perceptually similar to the full generator, making them a good proxy for preview. By using sampling-based multi-resolution training, adaptive-channel training, and a generator-conditioned discriminator, the anycost generator can be evaluated at various configurations while achieving better image quality compared to separately trained models. Furthermore, we develop new encoder training and latent code optimization techniques to encourage consistency between the different sub-generators during image projection. Anycost GAN can be executed at various cost budgets (up to 10x computation reduction) and adapt to a wide range of hardware and latency requirements. When deployed on desktop CPUs and edge devices, our model can provide perceptually similar previews at 6-12x speedup, enabling interactive image editing. The code and demo are publicly available: https://github.com/mit-han-lab/anycost-gan.

• The paper presents a novel GAN architecture that dynamically adapts to varying computational budgets while maintaining high output quality.
• It uses elastic resolutions, adaptive channels, and a generator-conditioned discriminator to efficiently handle interactive image synthesis.
• The approach achieves up to 10x computation reduction and 6-12x speedup on edge devices, enhancing real-time editing experiences.

Anycost GANs for Interactive Image Synthesis and Editing

The paper under review introduces a novel approach to enhancing the efficiency and flexibility of Generative Adversarial Networks (GANs) for interactive image synthesis. The authors propose the Anycost GAN, which dynamically adjusts to different computational budgets. This innovation is driven by the need for more responsive user experiences in image editing applications, particularly on resource-constrained devices.

Technical Contributions

1. Elastic Resolutions and Channels: The Anycost GAN is designed to handle elastic resolutions and channels, allowing subsets of the generator to produce outputs that remain perceptually similar to those of the full generator. This capability is achieved via sampling-based multi-resolution training, adaptive-channel training, and the utilization of a generator-conditioned discriminator.
2. Efficiency in Image Editing: The technique enables quick previews at substantially reduced computation costs while preserving high-quality output. The implementation demonstrates a 10x reduction in computation and a 6-12x speedup on edge devices, facilitating an interactive editing process.
3. Encoder Training and Latent Code Optimization: The paper details novel approaches to encoder training and latent code optimization, aimed at maintaining consistency across different sub-generator configurations.
4. Generator-Conditioned Discriminator: To handle the various sub-generators that emerge from the flexible configurations, the authors implement a generator-conditioned discriminator. This component ensures that the model remains stable during training, despite handling multiple sub-generator architectures.
5. Evolutionary Search for Optimal Sub-Generators: The use of evolutionary search allows the system to identify optimal configurations of sub-generators tailored to specific computational budgets, enhancing the adaptability of the system.

Experimental Results

The proposed model has been put through rigorous testing. It shows significant improvements over baseline methods, such as knowledge distillation and channel pruning, both in terms of Fréchet Inception Distance (FID) and perceptual path length. The Anycost GAN achieves superior output consistency and fidelity across diverse computational settings.

1. Quality and Consistency: The model maintains high attribute consistency and better visual coherence than separately trained smaller models. The LPIPS difference—a measure of perceptual difference—was notably lower for the Anycost GAN compared to other approaches.
2. Latency Reduction: Demonstrated speedups are substantial, showcasing the model’s effectiveness in reducing latency while maintaining image quality, a crucial factor for deployment on edge devices like mobile GPUs.
3. Quantitative and Qualitative Validation: Extensive experiments on high-resolution datasets such as FFHQ and LSUN Car confirm the model's performance, displaying consistency in visual attributes and efficient editing capabilities.

Implications and Future Directions

The Anycost GAN represents a significant advancement in making GAN-based technologies more practical for everyday use, particularly on devices where computational resources are limited. The model paves the way for further research into:

• Dynamic Network Architectures: Further exploration into adaptive models that can dynamically configure themselves to meet hardware constraints.
• Extended Application Scenarios: Application of this method to other types of neural networks and different multimedia content beyond images.
• User Interface Integration: Development of intuitive interfaces that empower non-expert users to leverage the full potential of such adaptive models without wrangling with technical complexities.

The paper sets a foundation for strengthening the integration of generative models into commercial software and broadening the accessibility of AI-driven creative tools. The flexibility demonstrated by Anycost GAN could become a cornerstone in the field of interactive AI applications.