LLM-as-a-Judge & Reward Model: What They Can and Cannot Do

Abstract: LLM-as-a-Judge and reward models are widely used alternatives of multiple-choice questions or human annotators for LLM evaluation. Their efficacy shines in evaluating long-form responses, serving a critical role as evaluators of leaderboards and as proxies to align LLMs via reinforcement learning. However, despite their popularity, their effectiveness in diverse contexts, such as non-English prompts, factual verification, or challenging questions, remains unexplored. In this paper, we conduct a comprehensive analysis of automated evaluators, reporting several key findings on their behavior. First, we discover that English evaluation capabilities significantly influence language-specific evaluation capabilities, often more than the language proficiency itself, enabling evaluators trained in English to easily transfer their skills to other languages. Second, we identify critical shortcomings, where LLMs fail to detect and penalize errors, such as factual inaccuracies, cultural misrepresentations, and the presence of unwanted language. Finally, we find that state-of-the-art evaluators struggle with challenging prompts, in either English or Korean, underscoring their limitations in assessing or generating complex reasoning questions. We release the dataset and codes used.

• The paper demonstrates that English-trained evaluation capabilities transfer effectively to non-English contexts, particularly in Korean.
• The paper identifies critical weaknesses, including failure to flag factual inaccuracies, cultural misrepresentations, and unwanted language.
• The paper introduces 'Kudge,' a novel meta-evaluation dataset with 5,012 Korean annotations for benchmarking multilingual evaluation models.

The paper "LLM-as-a-Judge & Reward Model: What They Can and Cannot Do" explores the effectiveness and limitations of using LLMs as automated evaluators—referred to as "LLM-as-a-Judge"—and reward models. These models are often employed as alternatives to traditional evaluation methods like multiple-choice questions or human annotators, particularly for assessing long-form responses. They also play a crucial role in leaderboard evaluations and serve as proxies to align other LLMs via reinforcement learning techniques.

The authors undertake a comprehensive analysis to investigate how well these automated evaluators function in non-English contexts, with a particular focus on Korean. Their key findings are as follows:

1. Transfer of English Evaluation Capabilities: One of the most striking discoveries is that the evaluation capabilities developed in English significantly impact the model's performance in other languages. This transfer is often more influential than the evaluator's proficiency in the target language itself. The implication is that evaluators trained in English can often perform reasonably well in other languages without additional fine-tuning.
2. Identification of Critical Shortcomings: Despite their potential, the paper identifies several critical weaknesses of LLM-as-a-Judge in a non-English environment. Specifically, the models struggle to detect and penalize various types of errors, including:
   • Factual Inaccuracies: Incorrect information not being flagged or penalized appropriately.
   • Cultural Misrepresentations: Failures in recognizing context-specific cultural nuances or misrepresentations.
   • Unwanted Language: Difficulty in identifying and addressing inappropriate or unwanted language within responses.
3. Release of Kudge Dataset: The authors introduce "Kudge," the first meta-evaluation dataset designed for non-English languages, containing 5,012 human annotations in Korean. This dataset aims to provide a robust benchmark for future research on automated evaluators in non-English contexts and facilitate further exploration into the identified shortcomings.

The findings underscore the need for more specialized training and evaluation methodologies to enhance the effectiveness of LLM-based evaluators in a multilingual setting. This research provides a foundation for addressing the existing gaps and improving the robustness of LLM-as-a-Judge and reward models across different languages.