Memory Replay GANs: learning to generate images from new categories without forgetting

Abstract: Previous works on sequential learning address the problem of forgetting in discriminative models. In this paper we consider the case of generative models. In particular, we investigate generative adversarial networks (GANs) in the task of learning new categories in a sequential fashion. We first show that sequential fine tuning renders the network unable to properly generate images from previous categories (i.e. forgetting). Addressing this problem, we propose Memory Replay GANs (MeRGANs), a conditional GAN framework that integrates a memory replay generator. We study two methods to prevent forgetting by leveraging these replays, namely joint training with replay and replay alignment. Qualitative and quantitative experimental results in MNIST, SVHN and LSUN datasets show that our memory replay approach can generate competitive images while significantly mitigating the forgetting of previous categories.

• The paper introduces Memory Replay GANs (MeRGANs) to address catastrophic forgetting by replaying prior task samples during new category learning.
• It proposes two methods, Joint Training with Replay and Replay Alignment, that blend real and replayed data to preserve image generation quality.
• Experiments on MNIST, SVHN, and LSUN datasets show improved performance and classification accuracy, validating the mechanism's effectiveness in continual learning.

Overview of Memory Replay GANs: Learning to Generate Images from New Categories Without Forgetting

In this paper, the authors tackle the pivotal issue of catastrophic forgetting within generative models, particularly focusing on Generative Adversarial Networks (GANs) when tasked with learning new image categories sequentially. The salient contribution is the introduction of Memory Replay GANs (MeRGANs), a novel framework that effectively mitigates this problem through the incorporation of a memory replay mechanism. This mechanism allows the GAN to systematically sample and integrate "memories" of previously learned tasks into the learning process of new tasks, thus preserving the ability to generate images from previous categories while learning new ones.

Key Contributions

1. Memory Replay in GANs: Unlike existing approaches that mainly target discriminative models, this study extends the memory replay strategy to generative models using GANs. The approach intriguingly aligns with the concept of pseudorehearsal from cognitive neuroscience, where memory consolidation is facilitated through replay mechanisms.
2. Methodological Innovation: The authors propose two specific methods under the MeRGANs framework. The first method, Joint Training with Replay (MeRGAN-JTR), involves creating an augmented dataset that combines real samples from the current task and replayed samples from past tasks to train the model. The second method, Replay Alignment (MeRGAN-RA), relies on synchronizing the current and replay generators to ensure generated samples are accurately aligned, enhancing retention through pixelwise alignment loss.
3. Experimental Validation: Robust experiments across varied datasets, such as MNIST, SVHN, and LSUN, demonstrate the efficacy of the proposed methods. The integration of these replay mechanisms significantly alleviates forgetting in GANs, showcasing an improved ability to maintain performance on previous tasks in sequence alongside learning new tasks. Noteworthy metrics include a marked improvement in classification accuracy used as a proxy for evaluating generated image quality and fidelity.

Implications and Future Directions

The research provides crucial insights into overcoming the intrinsic shortcoming of catastrophic forgetting in neural networks, a problem that is especially challenging within generative contexts where it directly affects the quality and diversity of generated outputs. The effective mitigation of such forgetting holds potential for practical applications where continual learning is essential—such as autonomous driving vehicles that require adaptability over time to learn and revise categories without access to prior data.

Theoretically, this work bridges the gap between cognitive neuroscience-inspired methodologies and machine learning, providing a template for integrating memory-inspired processes in artificial intelligence systems.

Future exploration could extend these findings through the application in more complex, real-world datasets and the adaptation of similar replay mechanisms to other types of generative models beyond GANs. Additionally, examining the interplay between network architectural innovations and these replay mechanisms could yield further enhancements in robustness to forgetting.

In conclusion, this study presents a thorough investigation into and implementation of a memory replay framework for GANs, establishing a foundation for future research and development of generative models that are resilient to forgetting in sequential learning scenarios.