ReIFE: Re-evaluating Instruction-Following Evaluation

Abstract: The automatic evaluation of instruction following typically involves using LLMs to assess response quality. However, there is a lack of comprehensive evaluation of these LLM-based evaluators across two dimensions: the base LLMs and the evaluation protocols. Therefore, we present a thorough meta-evaluation of instruction following, including 25 base LLMs and 15 recently proposed evaluation protocols, on 4 human-annotated datasets, assessing the evaluation accuracy of the LLM-evaluators. Our evaluation allows us to identify the best-performing base LLMs and evaluation protocols with a high degree of robustness. Moreover, our large-scale evaluation reveals: (1) Base LLM performance ranking remains largely consistent across evaluation protocols, with less capable LLMs showing greater improvement from protocol enhancements; (2) Robust evaluation of evaluation protocols requires many base LLMs with varying capability levels, as protocol effectiveness can depend on the base LLM used; (3) Evaluation results on different datasets are not always consistent, so a rigorous evaluation requires multiple datasets with distinctive features. We release our meta-evaluation suite ReIFE, which provides the codebase and evaluation result collection for more than 500 LLM-evaluator configurations, to support future research in instruction-following evaluation.

• The paper shows that LLM performance rankings remain consistent across varied evaluation protocols.
• It finds that robust evaluation requires diverse protocols tailored to different base model capabilities.
• The study introduces the ReIFE suite, a meta-evaluation toolkit covering over 500 configurations.

ReIFE: Re-evaluating Instruction-Following Evaluation

The paper "ReIFE: Re-evaluating Instruction-Following Evaluation" presents a comprehensive study on the evaluation of instruction-following capabilities in LLMs. This work addresses the need for a thorough meta-evaluation across two primary dimensions: the base LLMs used and the evaluation protocols applied. The study assesses 25 base LLMs and 15 evaluation protocols, using four human-annotated datasets, aiming to determine the best-performing configurations with robust accuracy.

Key Findings and Contributions

1. Base LLMs Consistency: The researchers found that performance rankings of base LLMs remain largely consistent across different evaluation protocols. This suggests that a single protocol might be sufficient for evaluating LLMs' capabilities consistently.
2. Diverse Protocol Necessity: The evaluation results emphasize that a robust examination of evaluation protocols requires employing multiple base LLMs with varying capability levels. Protocol effectiveness can be base LLM dependent.
3. Dataset Variability: The study highlights that results across different datasets are not always consistent; hence, a comprehensive evaluation requires testing across datasets with distinct features to ensure reliability.
4. Release of ReIFE Suite: The authors introduce the ReIFE suite, a meta-evaluation toolkit containing codebase and evaluation results for over 500 configurations, facilitating future research in this domain.

Evaluated Components

• Open-Source vs. Proprietary Models: Among the 38 models evaluated, llama-3.1-405B stood out as the best open-source model, nearing the performance of proprietary models like GPT-4.
• Protocol Evaluation: The paper critiques existing evaluation protocols used in popular benchmarks like AlpacaEval, ArenaHard, and WildBench, finding them less effective at the instance level compared to the base protocol.
• In-Depth Analysis: The study provides an analysis framework that questions both the base LLMs' and protocols' contributions to performance, helping identify potential improvements.

Implications and Future Directions

The results from this study have significant implications both theoretically and practically. The identification of robust LLM-evaluators and optimal evaluation protocols can inform the development of more effective AI training and validation processes. Moreover, understanding the variability in dataset difficulty and protocol effectiveness guides future research in creating more generalizable and reliable AI models.

As AI continues to evolve, further research could explore integrating fine-tuned LLMs and reward models within this evaluation framework. Additionally, a deeper examination involving multiple prompt variants and qualitative human evaluations could offer further insights into LLMs' evaluation capabilities.

In summary, this paper sheds light on the complexities of instruction-following evaluation in LLMs, providing valuable resources and findings that advance the field of AI model development and assessment. The ReIFE suite offers a robust foundation for continued exploration and enhancement of LLMs' evaluative capabilities.