Prompt-to-Leaderboard (2502.14855v2)

Abstract: LLM evaluations typically rely on aggregated metrics like accuracy or human preference, averaging across users and prompts. This averaging obscures user- and prompt-specific variations in model performance. To address this, we propose Prompt-to-Leaderboard (P2L), a method that produces leaderboards specific to a prompt. The core idea is to train an LLM taking natural language prompts as input to output a vector of Bradley-Terry coefficients which are then used to predict the human preference vote. The resulting prompt-dependent leaderboards allow for unsupervised task-specific evaluation, optimal routing of queries to models, personalization, and automated evaluation of model strengths and weaknesses. Data from Chatbot Arena suggest that P2L better captures the nuanced landscape of LLM performance than the averaged leaderboard. Furthermore, our findings suggest that P2L's ability to produce prompt-specific evaluations follows a power law scaling similar to that observed in LLMs themselves. In January 2025, the router we trained based on this methodology achieved the #1 spot on the Chatbot Arena leaderboard. Our code is available on GitHub at https://github.com/lmarena/p2l.

Prompt-to-Leaderboard: Addressing Variability in LLM Evaluations

The paper "Prompt-to-Leaderboard" introduces a methodological innovation aimed at refining the evaluation of LLMs. Traditional evaluation of LLMs often utilizes aggregated metrics such as accuracy or human preference scores averaged across diverse prompts and users. However, this averaging process can obscure specific variations tied to individual prompts or user interactions. The authors propose Prompt-to-Leaderboard (P2L) as a novel framework to overcome these limitations by generating leaderboards tailored to specific prompts, providing a nuanced insight into model performance.

Overview of Prompt-to-Leaderboard (P2L)

At its core, the P2L method employs an LLM trained to parse natural language prompts and output a vector of Bradley-Terry coefficients. These coefficients are used to predict human preference votes, essentially determining which model is favored given a particular prompt. This prompt-specific ranking allows for more granular evaluations of LLMs, capturing variations in performance across different contexts and use-cases.

The paper presents several significant implications of this approach:

• Unsupervised Task-Specific Evaluation: P2L can assess model performance in a task-specific manner without explicit supervision.
• Optimal Model Routing: By determining which model is best suited for a given prompt, resources and computational efforts can be allocated more efficiently.
• Personalization: The method enables personalization by building user-specific leaderboards based on individual prompt histories.
• Automated Evaluation: P2L facilitates automated evaluations, providing insights into both the strengths and weaknesses of various models.

Empirical Validation and Insights

The authors conducted experiments using data from Chatbot Arena, a platform for live model evaluation based on human preferences. P2L demonstrated an enhanced ability to capture the nuanced landscape of LLM performance compared to traditional averaged leaderboards. Notably, a router trained on this method achieved the top ranking in a benchmark test at Chatbot Arena, underscoring its efficacy.

Moreover, the findings reveal a power law scaling in P2L's ability to deliver prompt-specific evaluations, analogous to trends observed in LLM development. This scaling underscores the potential of P2L to maintain or even improve performance as more data becomes available or as models grow in complexity.

Theoretical and Practical Implications

The adoption of P2L could lead to significant advancements in both the theoretical understanding and practical application of AI models. Theoretically, it provides a new lens through which to measure model performance, moving beyond aggregate scores to more context-rich evaluations. Practically, it could drive more efficient use of computational resources and improve user satisfaction by tailoring interactions based on specific needs and preferences.

Looking forward, P2L's framework can be extended into the broader field of AI, influencing how other machine learning models are evaluated and optimized. By addressing prompt variability, the method sets a new standard for precision in model assessment and highlights the importance of context in performance evaluation.

Overall, the "Prompt-to-Leaderboard" paper presents a significant methodological contribution that enhances the granularity and applicability of LLM evaluations, offering valuable insights for future research and development in artificial intelligence.