FearNet: Brain-Inspired Model for Incremental Learning (1711.10563v2)

Abstract: Incremental class learning involves sequentially learning classes in bursts of examples from the same class. This violates the assumptions that underlie methods for training standard deep neural networks, and will cause them to suffer from catastrophic forgetting. Arguably, the best method for incremental class learning is iCaRL, but it requires storing training examples for each class, making it challenging to scale. Here, we propose FearNet for incremental class learning. FearNet is a generative model that does not store previous examples, making it memory efficient. FearNet uses a brain-inspired dual-memory system in which new memories are consolidated from a network for recent memories inspired by the mammalian hippocampal complex to a network for long-term storage inspired by medial prefrontal cortex. Memory consolidation is inspired by mechanisms that occur during sleep. FearNet also uses a module inspired by the basolateral amygdala for determining which memory system to use for recall. FearNet achieves state-of-the-art performance at incremental class learning on image (CIFAR-100, CUB-200) and audio classification (AudioSet) benchmarks.

• The paper presents FearNet by proposing a dual-memory architecture that mimics the human hippocampus and cortex for rapid and sustained learning.
• It employs a pseudorehearsal mechanism using a generative autoencoder to prevent catastrophic forgetting and minimize memory requirements.
• Empirical results on datasets like CIFAR-100 demonstrate FearNet’s superior performance and efficiency compared to existing incremental learning models.

FearNet: A Brain-Inspired Model for Incremental Learning

The paper "FearNet: Brain-Inspired Model for Incremental Learning" by Ronald Kemker and Christopher Kanan introduces a novel approach to incremental class learning, addressing the pervasive problem of catastrophic forgetting in deep neural networks (DNNs). This issue arises when a model loses previously acquired knowledge upon learning new data. FearNet circumvents this through a biologically inspired architecture that mimics the dual-memory system found in the mammalian brain, offering a memory-efficient alternative to existing methods.

Key Contributions

1. Architecture Inspired by Neuroscience: FearNet comprises three distinct components:
   • A short-term memory (STM) system analogous to the hippocampus (HC) for rapid learning and recall of new information.
   • A long-term memory (LTM) system akin to the medial prefrontal cortex (mPFC) for sustained storage and retrieval.
   • A decision module inspired by the basolateral amygdala (BLA) that chooses between STM and LTM for recalling specific memories.
2. Memory Consolidation through Pseudorehearsal: Drawing on the idea of memory replay during sleep, FearNet utilizes a generative autoencoder to simulate previous learning experiences without retaining the original data, thereby limiting memory demands while retaining learned knowledge.
3. Empirical Performance: The model delivers state-of-the-art results on benchmark datasets such as CIFAR-100, CUB-200, and AudioSet, outperforming existing methods like iCaRL, especially noteworthy given its limited use of memory resources.

Numerical Results and Implications

The paper reports superior performance metrics: FearNet not only maintains high accuracy on the base-knowledge but also exhibits adeptness at integrating new information without substantial degradation of old knowledge. Importantly, FearNet achieves this with a significantly reduced memory footprint compared to models like iCaRL, which rely heavily on storing numerous exemplars.

The model's ability to manage multi-modal data, incorporating both audio and image inputs, showcases its versatility in practical applications that demand real-time learning on resource-constrained devices.

Potential and Future Directions

FearNet presents substantial implications for the design of AI systems capable of continual learning. The approach aligns with the cognitive processes observed in biological systems, emphasizing the advantages of incorporating brain-inspired mechanisms over traditional data-intensive methods.

Looking ahead, the authors propose avenues for enhancement, including refining the BLA module for more integrated operation and replacing pseudorehearsal mechanisms with models capable of generating realistic pseudoexamples without class statistics. These developments could further optimize FearNet’s efficiency and applicability in diverse real-world scenarios.

In summary, FearNet constitutes a significant advancement in the domain of continuous learning, offering a promising framework for deploying AI in environments where memory and computational efficiency are paramount. The marriage of neuroscience principles with engineering solutions here can inspire further exploration of brain-inspired innovations in artificial intelligence.