Memory Efficient Experience Replay for Streaming Learning (1809.05922v2)

Abstract: In supervised machine learning, an agent is typically trained once and then deployed. While this works well for static settings, robots often operate in changing environments and must quickly learn new things from data streams. In this paradigm, known as streaming learning, a learner is trained online, in a single pass, from a data stream that cannot be assumed to be independent and identically distributed (iid). Streaming learning will cause conventional deep neural networks (DNNs) to fail for two reasons: 1) they need multiple passes through the entire dataset; and 2) non-iid data will cause catastrophic forgetting. An old fix to both of these issues is rehearsal. To learn a new example, rehearsal mixes it with previous examples, and then this mixture is used to update the DNN. Full rehearsal is slow and memory intensive because it stores all previously observed examples, and its effectiveness for preventing catastrophic forgetting has not been studied in modern DNNs. Here, we describe the ExStream algorithm for memory efficient rehearsal and compare it to alternatives. We find that full rehearsal can eliminate catastrophic forgetting in a variety of streaming learning settings, with ExStream performing well using far less memory and computation.

• The paper introduces ExStream, a memory-efficient rehearsal method that combats catastrophic forgetting in streaming neural networks.
• It leverages prototype clustering to store limited examples, significantly reducing memory and computational demands compared to full rehearsal strategies.
• Evaluated on datasets like iCub World and CORe50, ExStream outperforms methods such as Reservoir Sampling and Online k-means on non-iid data streams.

Memory Efficient Experience Replay for Streaming Learning

The paper "Memory Efficient Experience Replay for Streaming Learning" introduces the ExStream algorithm, an innovative approach to address the challenges associated with streaming learning within Deep Neural Networks (DNNs). Streaming learning requires a model to learn from a continuous flow of data, which potentially presents a non-iid (independent and identically distributed) nature, unlike traditional supervised learning paradigms where data is typically processed in batches and revisited multiple times. This paper examines how ExStream and other memory-efficient methods can mitigate catastrophic forgetting—a significant challenge with streaming data—by leveraging rehearsal strategies.

The authors effectively underscore the limitations of conventional DNNs when applied to environments where data arrives in sequential streams, particularly in scenarios demanding an immediate learning response. In such scenarios, the network's innate inclination toward catastrophic forgetting emerges predominantly due to its inability to reinforce its existing knowledge without revisiting past data. Traditionally, rehearsal, involving revisiting and retraining on prior data, has shown potential in mitigating this issue. However, the paper proposes that full rehearsal, while effective, is memory and computationally intensive, especially when employed with modern large-scale datasets and models.

ExStream offers a memory-efficient rehearsal approach by maintaining buffers of cluster prototypes rather than storing all prior examples. Unlike full rehearsal, which necessitates extensive memory storage to retrain models on all past seen data, ExStream, along with other stream clustering methods, limits the number of prototypes stored, optimizing both memory distribution and computational burdens. Their paper evaluates the efficacy of ExStream against other methods such as Reservoir Sampling, Queue, Online $k$-means, CluStream, and HPStream clustering buffers.

The evaluation protocol was rigorous, employing various data ordering schemes and benchmarks, including highly temporally structured datasets like iCub World 1.0, CORe50, and more class-intense examples like CUB-200-2011. The results indicated that ExStream generally performed best across contexts, demonstrating particular strengths in non-iid data orderings, which more closely reflect real-world robotic applications. The paper shows that methods more attuned to high-dimensional data streams exhibit superior memory-constrained performances.

One of the notable contributions of the research is the provision of metrics specifically designed for streaming classification, with $\Omega_b$ and $\mu_{\text{total}}$ providing an indicative measure of performance relative to an offline baseline model. These metrics facilitate a more granular understanding of how well DNNs adapted with ExStream and other methods perform against a non-streaming upper-bound model, supporting direct comparability across different experimental setups.

The implications of this research extend into the practical deployment of robotic and AI systems in dynamic environments, where computational resources must be preserved without sacrificing adaptive learning capabilities. Future developments in AI, as anticipated from this paper's discourse, may consider integrating memory-efficient buffer management strategies in addressing scalability challenges associated with expanding dataset domains or class numbers, especially in constrained computing environments. This research effectively balances between immediate applicability and theoretical advancement, paving the path toward more resilient learning systems capable of thriving amid the data-intensive challenges of streaming contexts in AI infrastructures.