Transferring GANs: generating images from limited data (1805.01677v2)

Abstract: Transferring the knowledge of pretrained networks to new domains by means of finetuning is a widely used practice for applications based on discriminative models. To the best of our knowledge this practice has not been studied within the context of generative deep networks. Therefore, we study domain adaptation applied to image generation with generative adversarial networks. We evaluate several aspects of domain adaptation, including the impact of target domain size, the relative distance between source and target domain, and the initialization of conditional GANs. Our results show that using knowledge from pretrained networks can shorten the convergence time and can significantly improve the quality of the generated images, especially when the target data is limited. We show that these conclusions can also be drawn for conditional GANs even when the pretrained model was trained without conditioning. Our results also suggest that density may be more important than diversity and a dataset with one or few densely sampled classes may be a better source model than more diverse datasets such as ImageNet or Places.

• The paper demonstrates that leveraging pre-trained GANs in data-scarce scenarios accelerates learning and significantly enhances image quality.
• The paper reveals that the effectiveness of transfer learning in GANs largely depends on selecting an optimal source domain, with narrow, densely-sampled datasets often outperforming more diverse ones.
• The paper shows that transferring the discriminator contributes more to performance improvements than the generator, particularly in conditional GAN settings.

Transferring GANs: Generating Images from Limited Data

The paper explores the domain transferability of generative adversarial networks (GANs), specifically investigating how the knowledge acquired from a source domain can be utilized to improve image generation in a target domain with limited data. Although transfer learning is extensively used with discriminative models, its application in generative models like GANs has not been thoroughly examined. This paper addresses this gap by evaluating domain adaptation strategies for GANs to enhance their performance under constrained data scenarios.

GANs typically contain a large number of parameters, which necessitates substantial training data to effectively generate high-quality images. However, training GANs traditionally requires starting from scratch; the potential of using pre-trained networks in generative settings remains largely untapped. This research aims to determine if pre-training on large datasets can benefit GANs, much like in discriminative tasks, especially when target domain data is sparse.

Key Contributions

1. Evaluation of Transfer Configurations: The paper evaluates several transfer configurations, establishing that leveraging pre-trained networks can expedite learning and improve image quality. It emphasizes that the degree of improvement is significant when the target domain data is scarce.
2. Source-Target Domain Relationship: The paper examines how the relationship between source and target domains affects GAN performance post-transfer. It suggests that selecting a suitable pre-trained model is more challenging compared to discriminative tasks.
3. Transfer to Conditional GANs: Analyzes the potential of transferring from unconditional to conditional GANs using two common conditioning methods, demonstrating that such transitions retain efficacy.

Insights and Numerical Results

The research demonstrates that pre-trained GANs converge faster and produce higher-quality images compared to those initialized from scratch. Moreover, pre-trained networks require fewer iterations to reach a comparable level of performance. Numerical results illustrate that pre-trained GANs can achieve analogous scores with approximately two to five times less data than non-pre-trained models. Crucially, experiments show that transferring the discriminator exerts a more substantial impact than the generator on the quality of image generation.

One intriguing finding is that contrary to typical assumptions in discriminative tasks, a narrow yet densely sampled source domain often outperforms more diverse datasets. As indicated by FID scores, transferring from source domains like LSUN Bedrooms, despite their limited diversity, yielded better results than broader datasets like ImageNet or Places, which are standard for discriminative tasks.

Implications and Future Directions

The implications of this research extend into both practical applications and theoretical explorations in AI. Practically, this work suggests a new avenue for reducing the computational cost and improving the efficiency of training GANs on limited data settings, which could be instrumental for applications like personalized data generation where comprehensive datasets are often unavailable.

Theoretically, it raises questions about the nature and types of features that are more generative when transferred from one domain to another. This also suggests further investigation into the relationship between domain density versus diversity, encouraging future research to refine methodologies for selecting optimal pre-trained models for specific target domains.

Future advancements might involve enhancing GAN architectures to augment transferability further or developing innovative techniques for selecting source domains that maximize performance improvements in target generative tasks. As GAN technology progresses, incorporating transfer learning mechanisms could substantially impact the deployment and scalability of AI-driven image generation in various fields.