May the Forgetting Be with You: Alternate Replay for Learning with Noisy Labels (2408.14284v1)

Abstract: Forgetting presents a significant challenge during incremental training, making it particularly demanding for contemporary AI systems to assimilate new knowledge in streaming data environments. To address this issue, most approaches in Continual Learning (CL) rely on the replay of a restricted buffer of past data. However, the presence of noise in real-world scenarios, where human annotation is constrained by time limitations or where data is automatically gathered from the web, frequently renders these strategies vulnerable. In this study, we address the problem of CL under Noisy Labels (CLN) by introducing Alternate Experience Replay (AER), which takes advantage of forgetting to maintain a clear distinction between clean, complex, and noisy samples in the memory buffer. The idea is that complex or mislabeled examples, which hardly fit the previously learned data distribution, are most likely to be forgotten. To grasp the benefits of such a separation, we equip AER with Asymmetric Balanced Sampling (ABS): a new sample selection strategy that prioritizes purity on the current task while retaining relevant samples from the past. Through extensive computational comparisons, we demonstrate the effectiveness of our approach in terms of both accuracy and purity of the obtained buffer, resulting in a remarkable average gain of 4.71% points in accuracy with respect to existing loss-based purification strategies. Code is available at https://github.com/aimagelab/mammoth.

• The paper introduces a novel method called Alternate Experience Replay (AER) that leverages the forgetting phenomenon to distinguish between clean and noisy samples.
• It employs a dual-phase training approach, yielding a 4.71% accuracy boost on Seq. CIFAR-100 with 40% symmetric noise compared to traditional methods.
• The method integrates Asymmetric Balanced Sampling (ABS) to ensure sample purity and diversity, enhancing robustness in continual learning scenarios.

Alternate Replay for Learning with Noisy Labels: A Detailed Overview

The paper "May the Forgetting Be with You: Alternate Replay for Learning with Noisy Labels" presents a novel approach to address the challenges of Continual Learning (CL) under Noisy Labels (CLN). Continual Learning involves the sequential assimilation of new data without forgetting previously acquired knowledge. However, it often encounters difficulties with catastrophic forgetting and noisy labels, which degrade performance over time. This paper introduces a method called Alternate Experience Replay (AER) to tackle these intrinsic issues.

Key Contributions and Methodology

1. Continuous Learning with Noisy Labels

CL methodologies traditionally rely on a memory buffer storing a limited subset of past data to mitigate forgetting. Noisy labels, whether introduced through automatic data collection or time-constrained human annotation, pose significant challenges in this setup. Existing approaches often depend on static replay mechanisms of buffered data, which can be vulnerable to noisy annotations, severely affecting the overall learning process.

2. Alternate Experience Replay (AER)

AER leverages the phenomenon of forgetting to distinguish between clean and noisy samples in the memory buffer. The core idea posits that noisy or mislabeled examples, which do not conform closely with already learned data distributions, are easier to forget. This insight drives the two main phases of AER:

• Buffer Learning Phase: Standard replay-based training on both current and past data.
• Buffer Forgetting Phase: Training exclusively on current task data, causing buffer samples to be forgotten and thereby revealing noisy samples through increased loss values.

3. Asymmetric Balanced Sampling (ABS)

To exploit the clean-noisy separation further, ABS is introduced. ABS is a sample selection strategy designed to:

• Ensure sampling purity: Prioritize clean samples during the current task.
• Retain diversity: Preserve complex and informative past samples, avoiding the potential bias introduced by focusing solely on loss values for sample selection.

ABS achieves this by applying asymmetric scoring, where clean current-task samples are selected based on low loss values, and past-task samples are selected to maximize retention of complex examples, even if they exhibit higher loss values.

Numerical Results and Practical Implications

The experimental results demonstrate the efficacy of the proposed method. AER and ABS together achieve significant improvements across various datasets and noise levels. Notably, for Seq. CIFAR-100 with 40% symmetric noise, the paper reports an average gain of 4.71% in accuracy when compared to existing baseline methods, such as loss-based purification strategies.

This numerical improvement underscores the robustness of the method, ensuring better buffer purity and diversity, critical for credible CL systems. Beyond accuracy improvements, the results also showcase reduced forgetting, illustrating the method's potential to maintain high performance over extended tasks.

Implications and Future Directions

The practical implications of this research are vast, especially in real-world scenarios where data annotation noise is inevitable. The enhanced ability to filter out noisy data while retaining complex yet informative examples ensures a more consistent and robust learning process.

Theoretically, AER provides a compelling approach to using forgetting, traditionally seen as a drawback, to an advantage in mitigating noise. This innovative perspective can inspire further research into adaptive strategies leveraging model dynamics over time for improved performance.

Future developments could explore:

• Hybrid Approaches: Combining regularization and rehearsal methods tailored for specific applications.
• Scalability: Extending the approach to more complex datasets and larger-scale problems.
• Detailed Analysis: Investigating the long-term effects of buffer management strategies under various noise distributions.

Conclusion

"May the Forgetting Be with You: Alternate Replay for Learning with Noisy Labels" sets a new direction in addressing the twin challenges of catastrophic forgetting and noisy labels in CL through AER and ABS. The significant improvements in accuracy and reduced forgetting rates showcase its potential for practical deployment in real-world AI systems where data annotation is prone to errors. By turning the problem of noisy labels on its head, using forgetting as a tool rather than a hindrance, this paper provides a solid foundation for future research in both the theoretical and applied domains of Continual Learning.