100 instances is all you need: predicting the success of a new LLM on unseen data by testing on a few instances

Abstract: Predicting the performance of LLMs on individual task instances is essential to ensure their reliability in high-stakes applications. To do so, a possibility is to evaluate the considered LLM on a set of task instances and train an assessor to predict its performance based on features of the instances. However, this approach requires evaluating each new LLM on a sufficiently large set of task instances to train an assessor specific to it. In this work, we leverage the evaluation results of previously tested LLMs to reduce the number of evaluations required to predict the performance of a new LLM. In practice, we propose to test the new LLM on a small set of reference instances and train a generic assessor which predicts the performance of the LLM on an instance based on the performance of the former on the reference set and features of the instance of interest. We conduct empirical studies on HELM-Lite and KindsOfReasoning, a collection of existing reasoning datasets that we introduce, where we evaluate all instruction-fine-tuned OpenAI models until the January 2024 version of GPT4. When predicting performance on instances with the same distribution as those used to train the generic assessor, we find this achieves performance comparable to the LLM-specific assessors trained on the full set of instances. Additionally, we find that randomly selecting the reference instances performs as well as some advanced selection methods we tested. For out of distribution, however, no clear winner emerges and the overall performance is worse, suggesting that the inherent predictability of LLMs is low.

• The paper introduces a framework that predicts new LLM performance using only 100 test instances to significantly reduce evaluation costs.
• The methodology combines clustering and embedding techniques to select representative instances from previous LLM evaluations for accurate predictions.
• Empirical results reveal competitive in-distribution accuracy while uncovering challenges in out-of-distribution predictability.

Analyzing Predictability in LLMs: Methodology and Insights

The paper "100 instances is all you need: predicting the success of a new LLM on unseen data by testing on a few instances" deals with a burgeoning topic in AI research—performance prediction for LLMs. The authors propose a methodology to assess the reliability of LLMs on individual task instances with minimal data requirement, which could have significant practical implications, especially in high-stakes applications.

The central thesis of this work is that it is possible to reliably predict the performance of new LLM models by evaluating them on a very small set of instances, in this case, only 100, drawn from a larger set of evaluation results for previously tested LLMs. The proposed framework leverages previously evaluated LLMs to build a generic assessor. This assessor only requires the model to be evaluated on a small set of reference instances, reduced from the full set, to predict model performance on new, unseen data.

Methodological Summary

The paper introduces an innovative framework that combines information about LLM performance across previously evaluated instances to predict the correctness of new LLMs. Two distinct datasets, HELM-Lite and the authors' own KindsOfReasoning dataset, were utilized in the empirical studies. The HELM-Lite dataset includes subsets from prior HELM benchmarks, while KindsOfReasoning is a comprehensive collection that evaluates various reasoning types including logical, inductive, and abductive reasoning.

The methodology involves a two-step process: first identifying a representative set of reference instances, then training a "generic assessor" that amalgamates instance-specific features with the performance data from the reference set. Various algorithms were tested for the selection of these reference instances, including clustering and factor analysis. The prediction capabilities of their framework were assessed through tests on multiple splits of their datasets, comparing in-distribution and out-of-distribution (OOD) prediction performance.

Numerical Performance and Insights

Empirically, the paper displays competitive performance of their generic assessor against a LLM-specific assessor (which requires evaluation on the entire dataset). In-distribution settings consistently showed the proposed method performing comparably to the specific assessors. Notably, the out-of-distribution scenarios revealed inherent limitations in predictability, indicating that current LLMs illuminate boundaries where predictability may be intrinsically low.

The choice of reference instances plays a vital role but is surprisingly robust across random selections, suggesting that even random sampling could be viably utilized for subset selection. Moreover, different embedding techniques were analyzed, with OpenAI embeddings often providing marginally better prediction capability for the assessors on specific tasks over standard word embedding methods like Word2Vec.

Implications and Future Directions

The framework offers a substantial reduction in computational cost for assessing new LLMs by limiting the number of necessary evaluations. This reduction is not just a practical gain but could also aid in making AI deployment more environmentally sustainable.

Moreover, the authors open a dialogue on the predictability of AI systems beyond mere performance enhancement—they espouse "predictable AI" as an objective that researchers and practitioners should prioritize. This notion aligns with emerging regulatory frameworks that place a premium on reliability and explainability. As general AIs get embedded in more critical applications, ensuring error prediction may become just as vital as improving raw benchmark scores.

The inherent unpredictability observed in OOD settings prompts compelling questions about the fundamental capabilities of LLMs. Future research could investigate which attributes make an instance or an entire dataset unpredictable, proposing novel architectures or training methodologies designed to mitigate these shortcomings.

Overall, the paper provides instrumental insights and methodologies for AI evaluations, aiming to bridge the gap between performance evaluation and practical applicability in model deployment by using concise evaluation methods. As AI continues to mature, predictability research such as this might guide innovations towards more reliable and accountable AI systems.