CORe50: a New Dataset and Benchmark for Continuous Object Recognition (1705.03550v1)

Abstract: Continuous/Lifelong learning of high-dimensional data streams is a challenging research problem. In fact, fully retraining models each time new data become available is infeasible, due to computational and storage issues, while na\"ive incremental strategies have been shown to suffer from catastrophic forgetting. In the context of real-world object recognition applications (e.g., robotic vision), where continuous learning is crucial, very few datasets and benchmarks are available to evaluate and compare emerging techniques. In this work we propose a new dataset and benchmark CORe50, specifically designed for continuous object recognition, and introduce baseline approaches for different continuous learning scenarios.

• The paper introduces CORe50, a dataset and benchmark challenging continuous learning with real-world object recognition tasks.
• It defines three learning scenarios—new instances, new classes, and both—to simulate online adaptation under varied conditions.
• Baseline evaluations show that methods like CWR reduce catastrophic forgetting, though significant performance gaps persist.

Assessing CORe50 for Continuous Object Recognition

The paper "CORe50: a New Dataset and Benchmark for Continuous Object Recognition" presents an important step in the evaluation and development of continuous learning algorithms, specifically within the domain of real-world object recognition. The authors introduce CORe50, a dataset and a suite of benchmarks designed to support research in lifelong learning where traditional approaches often fall short due to issues of computational overload and catastrophic forgetting.

CORe50, short for Continuous Object Recognition, includes 50 domestic objects across 10 categories captured over 11 sessions with varied backgrounds, lighting, and occlusions. This dataset is designed to challenge current machine learning models with continuous data streams, demanding incremental learning capabilities. Such scenarios are crucial for applications like robotic vision, where real-time adaptation and recognition of known and unknown objects are vital.

Distinctive Characteristics of CORe50

The paper identifies several key aspects of CORe50 that differentiate it from existing datasets. Unlike those static counterparts—such as ImageNet and Pascal VOC—CORe50 offers:

1. Temporal Coherence: Essential for simulating realistic, continuous learning scenarios, CORe50 features temporal sequences where objects exhibit varying poses, lighting conditions, and occlusions.
2. Multiple Scenarios: The dataset explores three main learning scenarios:
   • New Instances (NI): Additional views or instances of known classes.
   • New Classes (NC): Introduction of entirely new object categories.
   • New Instances and Classes (NIC): A combination of both evolving views and new categories.
3. Rich Contextual Settings: Including both indoor and outdoor acquisition sessions, the dataset mimics real-world variability in environmental conditions.

Baseline Methodologies and Findings

The paper proposes baseline approaches as references for testing continuous learning models:

• For NI, simple incremental training can mitigate catastrophic forgetting when correctly tuned, though the challenge remains for NC and NIC.
• The authors introduce the CopyWeights with Re-init (CWR) approach for NC and NIC scenarios. This method outperforms naïve tuning by managing weights in a fashion that minimizes interference across incremental learning phases.

The baseline results indicate that while current methodologies offer some solutions, the gap remains significant compared to cumulative methods that retrain models with all past data. The researchers suggest that techniques like Elastic Weight Consolidation and Learning without Forgetting could be promising for further reducing this gap.

Implications and Future Directions

CORe50 not only provides a challenging benchmark but also pushes the machine learning community to pursue more biologically-plausible methods that align with human learning efficiencies. Practical implications extend to robotics, autonomous systems, and beyond, where the ability to learn continuously without catastrophic forgetting would enhance adaptability and decision-making.

Future developments in this field could focus on enhancing feature extraction techniques, integrating motion and depth data for richer training vectors, and extending datasets with more classes and sessions. Additionally, adapting pre-trained models more effectively to variable input sizes, as discussed in the paper's appendix, remains an area for potential gains in processing efficiency.

By supplying a well-structured environment for testing, the work on CORe50 sets a foundation for iterating on and improving upon current limitations in incremental learning, making it an invaluable resource for advancing continuous object recognition research.