Latent Replay for Real-Time Continual Learning (1912.01100v2)

Abstract: Training deep neural networks at the edge on light computational devices, embedded systems and robotic platforms is nowadays very challenging. Continual learning techniques, where complex models are incrementally trained on small batches of new data, can make the learning problem tractable even for CPU-only embedded devices enabling remarkable levels of adaptiveness and autonomy. However, a number of practical problems need to be solved: catastrophic forgetting before anything else. In this paper we introduce an original technique named "Latent Replay" where, instead of storing a portion of past data in the input space, we store activations volumes at some intermediate layer. This can significantly reduce the computation and storage required by native rehearsal. To keep the representation stable and the stored activations valid we propose to slow-down learning at all the layers below the latent replay one, leaving the layers above free to learn at full pace. In our experiments we show that Latent Replay, combined with existing continual learning techniques, achieves state-of-the-art performance on complex video benchmarks such as CORe50 NICv2 (with nearly 400 small and highly non-i.i.d. batches) and OpenLORIS. Finally, we demonstrate the feasibility of nearly real-time continual learning on the edge through the deployment of the proposed technique on a smartphone device.

• The paper presents Latent Replay, which stores intermediate neural activations to mitigate catastrophic forgetting while lowering memory and computational costs.
• It employs selective layer freezing to stabilize lower network representations while rapidly adapting upper layers to new data.
• Empirical results on video benchmarks show that the method narrows the accuracy gap to within 5% of cumulative learning limits on resource-constrained devices.

Latent Replay for Real-Time Continual Learning

The paper "Latent Replay for Real-Time Continual Learning" presents a novel approach aimed at enhancing the efficiency and effectiveness of continual learning on lightweight computational devices. The authors introduce the Latent Replay technique, which proposes an innovative method of handling the challenge of catastrophic forgetting—a critical issue in continual learning where neural networks forget previously acquired knowledge upon exposure to new data.

Overview

Latent Replay operates by storing intermediate activations of a neural network instead of raw input data, as is the case with traditional rehearsal methods. The approach reduces both storage requirements and computational costs, benefiting applications on devices with limited computational resources, such as embedded systems and smartphones. This strategy particularly shows its potential when operating on CPU-only devices, a scenario common in edge computing where privacy and connectivity issues necessitate on-device training.

The paper supports the proposed method through empirical evidence, demonstrating that Latent Replay, when combined with other continual learning strategies, achieves state-of-the-art performance. This is validated using complex video benchmarks such as CORe50 NICv2 and OpenLORIS, where the model must learn from highly non-i.i.d. and small batches incrementally. The practicality of the approach is further illustrated by its application on a smartphone, demonstrating nearly real-time continual learning.

Key Contributions

1. Intermediate Activation Storage: By storing activations rather than raw data, the proposed method significantly reduces storage and computational requirements. This allows the network to be incrementally updated without exhaustive computational resources.
2. Selective Layer Freezing: Implementing a selective freezing method, the authors propose slowing down learning for layers beneath the replay layer while allowing upper layers to learn rapidly. This approach maintains a stable representation, preserving older knowledge while adapting to new information.
3. Benchmarked Performance: The paper asserts the superior performance of Latent Replay, particularly when combined with strategies like AR1* and CWR*. In extensive experiments, it achieves competitive accuracy that closes the gap with the cumulative learning upper bound, a theoretical model trained on the entire dataset at once.

Numerical Results and Implications

The numerical results indicate that the proposed method reduces the accuracy gap with the cumulative upper bound to approximately 5% on certain benchmarks. Compared to approaches like AR1*, AR1* free, and CWR*, the latent replay strategy improves performance, showcasing its utility in resource-constrained environments.

The implications of this work are significant for real-world applications that involve edge devices. By deploying efficient continual learning strategies, systems can adapt in real-time to new data samples directly in the field. This has direct implications for privacy preservation and adaptive intelligence in IoT devices, robotics, and mobile applications.

Future Directions

The paper opens several avenues for future research. Potential directions include optimizing pattern replacement strategies to further combat activation aging, and exploring generative models to create pseudo-activations, potentially eliminating the need for explicit external memory. Furthermore, the investigation into the plasticity-stability trade-off through architectural modifications or advanced hyperparameter scheduling could further refine the balance between resource constraint and learning efficacy.

Conclusion

Latent Replay represents a significant advancement in the continual learning domain, particularly for low-resource environments where real-time adaptability is crucial. By effectively reducing catastrophic forgetting and optimizing both storage and computation through intermediate layer replay, this method sets the stage for broader application in embedded systems and edge AI. The paper convincingly argues its case through both theoretical exposition and empirical validation, establishing a foothold for continued exploration and innovation in efficient continual learning methodologies.