Active Model Selection for Large Language Models

Abstract: We introduce LLM SELECTOR, the first framework for active model selection of LLMs. Unlike prior evaluation and benchmarking approaches that rely on fully annotated datasets, LLM SELECTOR efficiently identifies the best LLM with limited annotations. In particular, for any given task, LLM SELECTOR adaptively selects a small set of queries to annotate that are most informative about the best model for the task. To further reduce annotation cost, we leverage a judge-based oracle annotation model. Through extensive experiments on 6 benchmarks with 151 LLMs, we show that LLM SELECTOR reduces annotation costs by up to 59.62% when selecting the best and near-best LLM for the task.

• The paper introduces LLM Selector, a framework that leverages information gain to efficiently select the best LLM for specific tasks.
• It employs an oracle-based judge model, reducing annotation costs by up to 59.62% while reliably identifying high-performing models.
• Extensive experiments across six benchmarks and 151 LLMs validate its scalability and robustness under budget constraints.

Active Model Selection for LLMs

Introduction

The paper "Active Model Selection for LLMs" (2510.09418) introduces a framework termed LLM Selector, designed to efficiently identify the optimal LLM for specific tasks with minimal annotation requirements. This research addresses the challenge of selecting the best model from an expanding pool of LLMs without the need for exhaustive benchmarks or fully annotated datasets. The framework adaptively selects a limited yet highly informative subset of queries for annotation, which significantly reduces the annotation cost by leveraging an oracle-based judge model.

Figure 1: An overview of LLM Selector. For an arbitrary pool of $n$ queries and a set of candidate LLMs, LLM Selector adaptively annotates the most informative $b \ll n$ queries for identifying the best LLM.

Framework and Methodology

LLM Selector is built on the foundation of information gain principles, where the informativeness of a query is quantified as Shannon’s mutual information between the unknown best model and the potential annotations. This methodological choice allows LLM Selector to prioritize queries that maximize the reduction in uncertainty about the best model without retraining or full benchmarking.

The paper proposes a unique judge-based annotation model where each query is annotated by comparing model responses against a baseline using oracle preference judgments, thereby reducing reliance on human annotations. This approach mitigates the expensive and often imprecise nature of reference-based evaluations.

The algorithmic framework of LLM Selector involves sequential information maximization, employing a two-parameter model to characterize oracle responses. This setup facilitates efficient selection of queries, which in turn lowers the cost of identifying top-performing models by a substantial margin—up to 59.62% in terms of annotation savings.

Experimental Evaluation

The authors conducted extensive experimentation on six benchmarks covering various domains: AlpacaEval, Arena-Hard, MT-Bench, Flickr30k, Bingo, and MediQA. The experiments involved 151 LLMs, highlighting the versatility and general applicability of the LLM Selector framework across different tasks.

Figure 2: Candidate LLM win rate histograms.

The results demonstrate that LLM Selector consistently achieves competitive performance while maintaining significant reductions in annotation requirements. Notably, it can identify models with a near-best performance without fully depleting the annotation budget, showing robustness under budget constraints.

Figure 3: Best model identification probability of LLM Selector and the baselines.

Implications and Future Work

The introduction of LLM Selector notably advances the field of LLM evaluation by providing a scalable, cost-effective solution to the model selection problem. It captures the nuances of model performance with fewer resources, which is critical as the catalogue of available LLMs continues to grow.

Potential future developments include enhancing the oracle model with more sophisticated evaluation criteria and extending the framework to incorporate adaptive learning features that could dynamically adjust to changes in model architectures or data domains.

Conclusion

The paper offers a significant contribution to the efficient evaluation of LLMs by introducing LLM Selector, a framework that significantly cuts down on annotation cost while reliably identifying high-performing models. This work paves the way for practical, scalable evaluation practices in large model ecosystems, providing a foundation for ongoing advancements in AI model selection methodologies.