From Rankings to Insights: Evaluation Should Shift Focus from Leaderboard to Feedback (2505.06698v2)

Abstract: Automatic evaluation benchmarks such as MT-Bench, Arena-Hard, and Auto-Arena are seeing growing adoption for the evaluation of LLMs. Existing research has primarily focused on approximating human-based model rankings using limited data and LLM-as-a-Judge. However, the fundamental premise of these studies, which attempts to replicate human rankings, is flawed. Specifically, these benchmarks typically offer only overall scores, limiting their utility to leaderboard rankings, rather than providing feedback that can guide model optimization and support model profiling. Therefore, we advocate for an evaluation paradigm shift from approximating human-based model rankings to providing feedback with analytical value. To this end, we introduce \textbf{Feedbacker}, an evaluation framework that provides comprehensive and fine-grained results, thereby enabling thorough identification of a model's specific strengths and weaknesses. Such feedback not only supports the targeted optimization of the model but also enhances the understanding of its behavior. Feedbacker comprises three key components: an extensible tree-based query taxonomy builder, an automated query synthesis scheme, and a suite of visualization and analysis tools. Furthermore, we propose a novel LLM-as-a-Judge method: PC$^{2}$ (Pre-Comparison-derived Criteria) pointwise evaluation. This method derives evaluation criteria by pre-comparing the differences between several auxiliary responses, achieving the accuracy of pairwise evaluation while maintaining the time complexity of pointwise evaluation. Finally, leveraging the evaluation results of 17 mainstream LLMs, we demonstrate the usage of Feedbacker and highlight its effectiveness and potential. Our project homepage and dataset are available at https://liudan193.github.io/Feedbacker.

From Rankings to Insights: Evaluation Should Shift Focus from Leaderboard to Feedback

The paper authored by Wang et al. evaluates the current methodologies employed in assessing LLMs and outlines the intrinsic limitations associated with commonly used automatic evaluation benchmarks. These benchmarks, which primarily focus on replicating human-based model rankings, are often limited by their lack of comprehensive feedback, instead offering only leaderboard rankings. This inadequacy fails to provide actionable insights into the specific strengths and weaknesses of an evaluated model. The paper posits an evaluation paradigm shift from leaderboard ranking to feedback provision with analytical value, achieved through their proposed framework named Feedbacker.

Evaluation Paradigm Shift

Feedbacker aims to provide detailed feedback that serves as a guide for model optimization and profiling. The framework comprises a tree-based query taxonomy builder, an automatic query synthesis scheme, and analytical tools designed for visualization. The evaluation method focuses on delivering fine-grained results, offering insights into the characteristic behavior of models rather than merely assigning them a rank.

Feedbacker's taxonomy builder is an extensible system that constructs a query-type tree taxonomy essential for thorough evaluations. The RealMix component generates high-quality queries simulating various real-world scenarios without the risk of data contamination. Together, these establish a comprehensive dataset that includes underrepresented aspects of model capabilities, which traditional leaderboard evaluations may overlook.

PC\textsuperscript{2} Pointwise Evaluation

Significantly, the authors introduce a novel LLM-as-a-Judge method: Pre-Comparison-derived Criteria (PC\textsuperscript{2}) pointwise evaluation. This method derives evaluation criteria by pre-comparing differences among auxiliary responses generated by a diverse set of LLMs. This innovative approach aims to achieve the accuracy of pairwise evaluations while maintaining the efficiency of pointwise evaluations. The PC\textsuperscript{2} method allows for direct scoring via fixed query-specific criteria, effectively balancing comprehensive analysis with computational efficiency.

Numerical results indicate PC\textsuperscript{2} evaluation's effectiveness as it surpasses traditional pointwise methods in accuracy and offers competitive results to pairwise methods with significantly reduced computational overhead.

Implications and Future Developments

This framework not only facilitates targeted model improvements by pinpointing specific weaknesses but also has implications for aligning LLM development more closely with nuanced performance metrics. By focusing evaluation on feedback rather than ranking, the research improves understanding of model behaviors, ultimately contributing to refinement strategies that extend beyond superficial metric improvements.

Feedbacker's approach also lays groundwork for the future development of more sophisticated and comprehensive evaluation measures in AI. This includes enriching LLM benchmarks with more diverse and representative datasets that reflect real-world applications and reduce bias in model training and assessment processes. The authors note the potential expansions of their framework to incorporate evaluations in multimodal models and other domains, which remains a future research direction.

Conclusion

The research presented by Wang et al. provides a detailed analysis and solution to the prevalent issue in LLM evaluation systems that emphasize rankings over insights. By introducing Feedbacker and the PC\textsuperscript{2} evaluation method, the authors offer a transformative approach to assessing and understanding LLMs, paving the way for advanced model optimization and alignment with real-world demands.