NEVIS'22: A Stream of 100 Tasks Sampled from 30 Years of Computer Vision Research

Abstract: A shared goal of several machine learning communities like continual learning, meta-learning and transfer learning, is to design algorithms and models that efficiently and robustly adapt to unseen tasks. An even more ambitious goal is to build models that never stop adapting, and that become increasingly more efficient through time by suitably transferring the accrued knowledge. Beyond the study of the actual learning algorithm and model architecture, there are several hurdles towards our quest to build such models, such as the choice of learning protocol, metric of success and data needed to validate research hypotheses. In this work, we introduce the Never-Ending VIsual-classification Stream (NEVIS'22), a benchmark consisting of a stream of over 100 visual classification tasks, sorted chronologically and extracted from papers sampled uniformly from computer vision proceedings spanning the last three decades. The resulting stream reflects what the research community thought was meaningful at any point in time, and it serves as an ideal test bed to assess how well models can adapt to new tasks, and do so better and more efficiently as time goes by. Despite being limited to classification, the resulting stream has a rich diversity of tasks from OCR, to texture analysis, scene recognition, and so forth. The diversity is also reflected in the wide range of dataset sizes, spanning over four orders of magnitude. Overall, NEVIS'22 poses an unprecedented challenge for current sequential learning approaches due to the scale and diversity of tasks, yet with a low entry barrier as it is limited to a single modality and well understood supervised learning problems. Moreover, we provide a reference implementation including strong baselines and an evaluation protocol to compare methods in terms of their trade-off between accuracy and compute.

• The paper introduces Nevis'22, a benchmark of 100 tasks sampled over 30 years of computer vision research designed to evaluate continual learning models.
• It employs a chronological sequence of diverse tasks and a dual-phase evaluation protocol to measure adaptation, knowledge transfer, and computational efficiency.
• Empirical results demonstrate that sequential fine-tuning outperforms naive learning approaches, emphasizing the potential of optimizing both accuracy and computational cost.

Analysis of Nevis'22: A Benchmark for Continual Learning in Computer Vision

The paper "Nevis'22: A Stream of 100 Tasks Sampled from 30 Years of Computer Vision Research" presents a sophisticated benchmark designed to evaluate continual learning models in visual classification tasks. This unique benchmark is constructed to address key challenges in continual learning, such as efficiently adapting to a sequence of tasks, transferring knowledge, and minimizing computational costs.

Overview of Nevis'22

Nevis'22 consists of over 100 visual classification tasks sampled chronologically from the past three decades of computer vision literature. The authors aimed to create a benchmark that reflects the evolving interests of the vision research community. The tasks were selected using a methodologically rigorous process to ensure diversity and relevance, covering an array of domains from OCR and texture analysis to scene recognition and object identification. Notably, dataset sizes vary immensely, encompassing four orders of magnitude, which together pose a substantial challenge for sequential learning approaches.

Key Features

• Chronologically Ordered Tasks: Tasks are ordered chronologically, mimicking the historical progression of tasks encountered by the research community. This arrangement allows for evaluating models' capabilities in dealing with non-stationary data streams, a key aspect of lifelong learning.
• Diversity Across Domains: The paper emphasizes the inclusion of diverse tasks in Nevis'22. The benchmark includes datasets that address various domains such as object classification, face recognition, and scene understanding, among others.
• Rigorous Evaluation Protocol: Nevis'22 implements a dual-layer evaluation protocol emphasizing both error minimization and computational efficiency. The authors advocate for a protocol that includes a meta-training phase for hyperparameter tuning using older tasks and a meta-testing phase focused on recent tasks.

Insights and Implications

The empirical results presented highlight that sequential fine-tuning methods significantly outperform naive independent task learning. This finding suggests that there exists rich structural information across tasks that models can exploit to facilitate learning. Fine-tuning, especially when initialized from well-related previous tasks, offers substantial benefits in terms of the trade-off between computational cost and model accuracy.

Moreover, the results from the Nevis'22 benchmark challenge preconceptions regarding the inferiority of finetuning chains. Contrary to traditional views that posited multi-task learning as superior, the performance of fine-tuning with adequately chosen priors demonstrates its practicality and efficiency, even after multiple tasks.

The implications of Nevis'22 are profound for research in continual and lifelong learning. The benchmark not only provides a rigorous testing ground for new algorithms but also sets a precedent for the necessity of multi-objective optimization in machine learning, balancing accuracy with computational expenditure.

Future Directions

Given the preliminary successes and challenges outlined in this research, future work in the field of continual learning could focus on refining task similarity metrics, improving hyperparameter optimization strategies, and exploring the architectural nuances that support variable input resolutions. Expanding Nevis'22 to include tasks from modalities beyond visual classification—such as language or multimodal tasks—could enhance its utility and drive advances in more generalizable and adaptive learning systems.

Conclusion

The Nevis'22 benchmark represents a significant contribution to the field of continual learning by integrating a diverse array of tasks and a robust evaluation methodology that accents both efficacy and efficiency. As the community endeavors to realize models that learn over sequential experiences, benchmarks like Nevis'22 will be critical in steering research that fosters models with true lifelong learning capabilities.