Stronger Random Baselines for In-Context Learning

Abstract: Evaluating the in-context learning classification performance of LLMs poses challenges due to small dataset sizes, extensive prompt-selection using the validation set, and intentionally difficult tasks that lead to near-random performance. The standard random baseline--the expected accuracy of guessing labels uniformly at random--is stable when the evaluation set is used only once or when the dataset is large. We account for the common practice of validation set reuse and existing small datasets with a stronger random baseline: the expected maximum accuracy across multiple random classifiers. When choosing the best prompt demonstrations across six quantized LLMs applied to 16 BIG-bench Lite tasks, more than 20% of the few-shot results that exceed the standard baseline do not exceed this stronger random baseline. When held-out test sets are available, this stronger baseline is also a better predictor of held-out performance than the standard baseline, avoiding unnecessary test set evaluations. This maximum random baseline provides an easily calculated drop-in replacement for the standard baseline.

• The paper proposes a stronger random baseline metric to more accurately evaluate in-context learning performance, accounting for small datasets and extensive prompt engineering.
• Experiments reveal that many few-shot results previously exceeding traditional random baselines do not surpass the proposed stronger maximum random baseline, indicating potential overestimation.
• This new analytically computed baseline provides a higher threshold for evaluation, helps prevent overfitting, and is practically applicable to various classification tasks beyond in-context learning.

Stronger Random Baselines for In-Context Learning: A Critical Examination

The paper "Stronger Random Baselines for In-Context Learning" presents an insightful exploration of the evaluation metrics commonly used in in-context learning (ICL), particularly the random baselines employed to gauge LLMs' performances. The authors argue for the necessity of a refined baseline that better reflects the complexities and nuances of ICL tasks, which often involve small datasets and meticulous prompt engineering.

Background and Motivation

The domain of LLMs is complemented by ICL, where models are expected to perform tasks with minimal examples, or "few-shots." Traditional evaluation methods utilize a random baseline where the expected accuracy is the probability of guessing correctly by chance. This notion, while robust in contexts with large datasets or singular evaluation setups, proves inadequate when datasets are small or validation sets are re-used for multiple prompt evaluations. The authors identify a gap in how existing random baseline models inadequately address these typical ICL settings characterized by limited data and multiple prompt searches.

The Development of a Stronger Baseline

The authors propose an alternative metric: the expected maximum accuracy across multiple random classifiers. This stronger random baseline accounts for the variation in results due to small dataset sizes and the repeated use of validation datasets—a common practice in prompt engineering.

Experiments were conducted over 16 tasks from the BIG-bench Lite benchmark using six different LLM configurations, both base and instruction-tuned variants. The findings reveal that a significant portion of few-shot results that surpass the traditional random baseline do not exceed the proposed maximum random baseline. This indicates that previous models' performances were often overstated when assessed with standard baselines.

Key Insights and Results

1. Higher Threshold Establishment: The stronger random baseline provides a higher benchmark for evaluating model performance, preventing overestimation of results derived from small sample sets or extensive prompt optimization.
2. Prevention of Overfitting: By better predicting held-out performance, this baseline dismisses unnecessary scrutiny on test sets, thus safeguarding against overfitting.
3. Practicality in Application: The new baseline offers an analytically computed, simple-to-apply alternative, which can seamlessly replace the traditional baseline—as it incorporates the maximum order statistic of the binomial distribution.
4. General Applicability: The paper confirms that the maximum random baseline can extend beyond ICL, benefiting any classification task where prediction accuracy is the primary metric of performance.

Implications and Future Outlook

The implications of this research are manifold, touching both on theoretical modeling and the pragmatic considerations of deploying LLMs. The proposed method enhances the robustness of ICL assessments, with potential extensions into broader machine learning contexts where small datasets are prevalent. In terms of future developments, there is scope for further exploration into adapting this baseline to other metrics such as F1 scores or expanding its use in multi-label classification scenarios.

This research introduces a pivotal paradigm shift in how ICL performance is gauged, emphasizing the significance of contextual and adaptive baselines. By addressing the strengths and limitations of existing evaluation approaches, the paper sets a foundation for developing even more granular and precise measurement techniques in AI research.