ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended Capabilities

Abstract: Traditional fixed test sets fall short in evaluating open-ended capabilities of foundation models. To address this, we propose ONEBench(OpeN-Ended Benchmarking), a new testing paradigm that consolidates individual evaluation datasets into a unified, ever-expanding sample pool. ONEBench allows users to generate custom, open-ended evaluation benchmarks from this pool, corresponding to specific capabilities of interest. By aggregating samples across test sets, ONEBench enables the assessment of diverse capabilities beyond those covered by the original test sets, while mitigating overfitting and dataset bias. Most importantly, it frames model evaluation as a collective process of selecting and aggregating sample-level tests. The shift from task-specific benchmarks to ONEBench introduces two challenges: (1)heterogeneity and (2)incompleteness. Heterogeneity refers to the aggregation over diverse metrics, while incompleteness describes comparing models evaluated on different data subsets. To address these challenges, we explore algorithms to aggregate sparse measurements into reliable model scores. Our aggregation algorithm ensures identifiability(asymptotically recovering ground-truth scores) and rapid convergence, enabling accurate model ranking with less data. On homogenous datasets, we show our aggregation algorithm provides rankings that highly correlate with those produced by average scores. We also demonstrate robustness to ~95% of measurements missing, reducing evaluation cost by up to 20x with little-to-no change in model rankings. We introduce ONEBench-LLM for LLMs and ONEBench-LMM for vision-LLMs, unifying evaluations across these domains. Overall, we present a technique for open-ended evaluation, which can aggregate over incomplete, heterogeneous sample-level measurements to continually grow a benchmark alongside the rapidly developing foundation models.

• The paper introduces ONEBench, a novel framework that aggregates diverse datasets for open-ended evaluation of foundation models.
• It demonstrates that the aggregation algorithm recovers ground-truth scores with high correlation, even when over 95% of data is missing, significantly reducing evaluation costs.
• The study applies ONEBench to language and vision-language models, promoting a scalable, democratized benchmark for evolving AI capabilities.

An Overview of ONEBench: A Sample-Level Benchmarking Paradigm for Foundation Models

The paper "ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended Capabilities" proposes a novel framework for evaluating the capabilities of foundation models. The authors introduce ONEBench (OpeN-Ended Benchmarking), a system designed to overcome the limitations of traditional fixed test datasets in assessing the diverse and evolving abilities of foundation models. This approach consolidates individual evaluation datasets into a unified, flexible pool, allowing for custom, open-ended benchmarking that extends beyond the constraints of conventional static benchmarks.

Core Concepts and Challenges

ONEBench fundamentally shifts the paradigm of model evaluation from task-specific to open-ended capability testing. Two primary challenges are identified in this shift: heterogeneity and incompleteness. Heterogeneity refers to the variability in evaluation metrics, while incompleteness pertains to comparing models that have been tested on different subsets of data. To address these challenges, the authors propose an aggregation algorithm that ensures identifiability and robustness, allowing for the asymptotic recovery of ground-truth scores even with sparse data.

Strong Numerical Results

The authors present compelling numerical evidence supporting their aggregation algorithm's effectiveness. They demonstrate that on homogeneous datasets, the algorithm achieves rankings that strongly correlate with average scores. Additionally, the system shows robustness to scenarios with over 95% of data measurements missing, achieving significant reductions in evaluation costs without adversely affecting model rankings. This aspect indicates a reduction in evaluation costs by up to 20 times, enabling a more efficient benchmarking process.

Practical and Theoretical Implications

Practically, ONEBench provides a flexible, scalable benchmarking framework that allows for the continual expansion of evaluation datasets alongside the development of new foundation models. Theoretically, it promotes a democratized evaluation approach where a plurality of rankings and dynamic assessments can coexist, allowing for personalized benchmarks that reflect diverse interests and needs.

The paper introduces specific implementations, namely ONEBench-LLM for LLMs and ONEBench-LMM for vision-LLMs, demonstrating the framework's applicability across multiple domains. The authors empirically validate the robustness of their aggregation algorithm against widely adopted metrics like ELO and Bradley-Terry, emphasizing its superior performance in accuracy and robustness to missing information.

Future Directions

While ONEBench presents a robust framework for current foundation model evaluation, the authors highlight areas for future research, such as scaling the benchmark to include all existing LLMs and LMMs, enhancing retrieval mechanisms for more accurate capability probing, and exploring additional aggregation algorithms from social choice theory to potentially improve results.

In conclusion, ONEBench represents a significant advancement in benchmarking methodologies for foundation models, combining practical flexibility with theoretical robustness to better capture the capabilities of modern AI systems. As foundation models continue to evolve, frameworks like ONEBench will be crucial in providing the comprehensive and adaptive evaluations necessary to facilitate effective model development and deployment.