Evaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurements (2210.01970v2)

Abstract: Evaluation is a key part of ML, yet there is a lack of support and tooling to enable its informed and systematic practice. We introduce Evaluate and Evaluation on the Hub --a set of tools to facilitate the evaluation of models and datasets in ML. Evaluate is a library to support best practices for measurements, metrics, and comparisons of data and models. Its goal is to support reproducibility of evaluation, centralize and document the evaluation process, and broaden evaluation to cover more facets of model performance. It includes over 50 efficient canonical implementations for a variety of domains and scenarios, interactive documentation, and the ability to easily share implementations and outcomes. The library is available at https://github.com/huggingface/evaluate. In addition, we introduce Evaluation on the Hub, a platform that enables the large-scale evaluation of over 75,000 models and 11,000 datasets on the Hugging Face Hub, for free, at the click of a button. Evaluation on the Hub is available at https://huggingface.co/autoevaluate.

• The paper presents Evaluate and Evaluation on the Hub as tools that standardize ML evaluation practices, ensuring reproducibility and consistent metric application.
• It details a unified API and comprehensive documentation that simplify model comparisons and promote community contributions.
• The study demonstrates how these tools address evaluation gaps by centralizing diverse metrics and extending assessments to include fairness and robustness.

Overview of "Evaluate and Evaluation on the Hub"

This paper introduces two significant tools in the field of ML evaluation: the open-source library "Evaluate" and the "Evaluation on the Hub" platform. These tools are developed to tackle prevalent challenges in ML model assessments, such as reproducibility, centralization, and comprehensive coverage.

Evaluation Challenges in ML

The paper identifies existing gaps in ML evaluation practices. It highlights the complexities arising from increased datasets and metrics being used per research paper, as documented through an analysis of EMNLP papers. There is a noted deficiency in auxiliary testing such as statistical significance checks, leading to unreliable comparisons. Thus, even as data accessibility has improved via shared repositories, evaluation methodologies remain fragmented with no clear consensus on best practices.

Goals of the Tools

The primary objectives addressed by these tools are:

1. Reproducibility: The "Evaluate" library standardizes ML evaluation processes with consistent metrics and methodologies to improve reliability in results.
2. Centralization: By centralizing evaluation metrics and documentation, the tools enhance understanding and usage efficiency across the community.
3. Coverage: The tools focus on expanding evaluation beyond accuracy to include dimensions such as efficiency, bias, fairness, and robustness.

The Evaluate Library

The "Evaluate" library provides over 50 evaluation modules with a unified API, allowing for seamless model comparisons. It supports various metrics, comparisons, and dataset measurements. Special attention is given to documentation, with detailed cards accompanying each module, thereby educating users on usage, scope, and limitations.

Library Features:

• Structured Interface: A coherent API that simplifies metric implementation and execution.
• Evaluator Component: An advanced API that handles tasks from pre-processing to metric computation.
• Documentation and Community Contributions: Encourages shared practices and improvements through easy community integration.

Evaluation on the Hub Platform

The "Evaluation on the Hub" is a complementary service that provides no-code evaluations of models and datasets, leveraging the centralized resources of the Hugging Face Hub. It aims to allow extensive and reproducible evaluations, promising consistency through standardized implementations.

System Architecture:

• Job Submission: Users configure evaluations via a straightforward interface, which then automatically processes tasks and updates results on the model cards.
• Leaderboard: Aggregated results offer a comparative view of model performances, aiding in informed decision-making.

Use Cases

The tools presented cater to several practical scenarios:

• Model Selection: Facilitates choosing suitable models for specific tasks.
• Result Reproducibility: Ensures consistent and verifiable evaluations across new datasets or metrics.
• Deployment Decisions: Provides comprehensive insights into model performance, including efficiency and latency.
• Extending Evaluation Metrics: Supports easy addition of new metrics, enhancing versatility.

Implications and Future Directions

The paper underscores the importance of these tools in achieving better evaluation practices within the ML community. By addressing historical challenges, these resources provide a robust foundation for more rigorous and comprehensive model assessments. The ongoing development and community engagement promise continuous improvements, catering to emerging needs in multi-modal, multilingual, and diverse domain applications. Such resources pave the way for more informed, systematic advancements in ML deployment and research.