Rejection Sampling IMLE: Designing Priors for Better Few-Shot Image Synthesis (2409.17439v1)

Abstract: An emerging area of research aims to learn deep generative models with limited training data. Prior generative models like GANs and diffusion models require a lot of data to perform well, and their performance degrades when they are trained on only a small amount of data. A recent technique called Implicit Maximum Likelihood Estimation (IMLE) has been adapted to the few-shot setting, achieving state-of-the-art performance. However, current IMLE-based approaches encounter challenges due to inadequate correspondence between the latent codes selected for training and those drawn during inference. This results in suboptimal test-time performance. We theoretically show a way to address this issue and propose RS-IMLE, a novel approach that changes the prior distribution used for training. This leads to substantially higher quality image generation compared to existing GAN and IMLE-based methods, as validated by comprehensive experiments conducted on nine few-shot image datasets.

• The paper introduces RS-IMLE, which uses rejection sampling to align latent distributions and enhance few-shot image synthesis.
• The method addresses latent space misalignment by rejecting codes within a specified radius, achieving an average 45.9% FID improvement across nine datasets.
• Empirical results confirm that RS-IMLE mitigates mode collapse while improving image fidelity and diversity in data-scarce scenarios.

Rejection Sampling IMLE: Designing Priors for Better Few-Shot Image Synthesis

This paper introduces a method known as Rejection Sampling IMLE (RS-IMLE) aimed at enhancing few-shot image synthesis performance by addressing a key issue in Implicit Maximum Likelihood Estimation (IMLE). Prior methods like GANs, diffusion models, and standard IMLE approaches face limitations in generalizing well with limited data. IMLE traditionally encounters problems when the latent codes used in training differ from those drawn during inference, leading to suboptimal outcomes.

Core Contributions

The authors propose RS-IMLE to mitigate the misalignment in latent space that affects test-time performance. The main contributions can be summarized as follows:

1. Theoretical Foundation: Theoretical analysis establishes that existing IMLE methods inaccurately distribute latent codes between training and inference, resulting in a mismatch that affects generative quality.
2. Novel Prior Design: RS-IMLE employs rejection sampling to modify the prior distribution used in training. By rejecting latent codes within a specified radius from any training datum, RS-IMLE aligns the training distribution with the inference distribution more effectively.
3. Empirical Validation: The paper conducts extensive experiments across nine few-shot datasets, demonstrating a substantial improvement in Fréchet Inception Distance (FID) scores by an average of 45.9% over the best baseline, thereby validating the method's efficacy.

Numerical Results and Analysis

The proposed RS-IMLE approach achieves marked improvements in FID across diverse datasets, including domains such as facial imagery and abstract patterns. Notably, the precision and recall metrics highlight improved image quality and diversity, indicating that RS-IMLE better approximates the true data distribution. The qualitative analysis shows that generated images maintain high fidelity while offering diverse attributes, underscoring the model's robust latent space representation.

Theoretical and Practical Implications

• Mode Collapse Mitigation: RS-IMLE advances the field by resolving mode collapse issues inherent in GANs, which are particularly problematic in data-scarce environments.
• Latent Space Utilization: Addressing latent space alignment opens avenues for better generative models that can operate effectively in few-shot scenarios, which is critical for applications with limited training data availability.
• Scalability and Flexibility: By leveraging rejection sampling, RS-IMLE maintains efficiency without needing complicated likelihood computations, potentially facilitating its adaptation to broader applications.

Speculation on Future Developments

Future research could explore the adaptation of RS-IMLE to other forms of data, such as sequential or multimodal inputs. Additionally, integrating RS-IMLE with emerging architectures could further unlock performance gains in generative tasks, especially as computational efficiency continues to improve.

In conclusion, RS-IMLE presents a significant step forward in few-shot image synthesis by effectively aligning training and inference distributions. This improvement not only enhances image quality but also demonstrates the potential of optimizing latent space interactions for complex generative tasks.