Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments (2504.17087v1)

Abstract: LLMs are being widely applied across various fields, but as tasks become more complex, evaluating their responses is increasingly challenging. Compared to human evaluators, the use of LLMs to support performance evaluation offers a more efficient alternative. However, most studies focus mainly on aligning LLMs' judgments with human preferences, overlooking the existence of biases and mistakes in human judgment. Furthermore, how to select suitable LLM judgments given multiple potential LLM responses remains underexplored. To address these two aforementioned issues, we propose a three-stage meta-judge selection pipeline: 1) developing a comprehensive rubric with GPT-4 and human experts, 2) using three advanced LLM agents to score judgments, and 3) applying a threshold to filter out low-scoring judgments. Compared to methods using a single LLM as both judge and meta-judge, our pipeline introduces multi-agent collaboration and a more comprehensive rubric. Experimental results on the JudgeBench dataset show about 15.55\% improvement compared to raw judgments and about 8.37\% improvement over the single-agent baseline. Our work demonstrates the potential of LLMs as meta-judges and lays the foundation for future research on constructing preference datasets for LLM-as-a-judge reinforcement learning.

This paper introduces a practical framework for evaluating the quality of judgments made by LLMs when they are used to assess the output of other LLMs (a process often called LLM-as-a-judge). The core problem is that evaluating these judgments is challenging, and relying on human alignment is often insufficient due to human biases and limitations, especially for complex tasks. The paper proposes using LLMs themselves as meta-judges to evaluate the initial LLM judgments.

The core contribution is a three-stage pipeline designed to automatically select high-quality, trustworthy judgments from a pool generated by an LLM judge:

1. Prompt Design & Rubric Generation:
   • Starts with a basic rubric defined by human experts outlining evaluation criteria.
   • Uses a powerful LLM (GPT-4 in the paper) to refine this basic rubric into a detailed version. This includes comprehensive descriptions for each criterion, a scoring scale (e.g., 1-5), explanations for each score level, and assigning weights ($w_c$) to reflect the relative importance of each criterion for the specific task.
   • The criteria used include: Accuracy of Judgment, Logical Soundness, Completeness of Evaluation, Fairness, Relevance to Context, Clarity of Explanation, and Impactfulness.
   • The final prompt for the meta-judge LLMs includes the original instruction given to the judge LLM, the actual judgment text (conclusion and explanation) being evaluated, and the detailed, weighted rubric.
2. Multi-Agent Meta-Judge Score Calculation:
   • Instead of relying on a single LLM, the framework employs multiple (N) advanced LLMs as meta-judge agents to score a given judgment based on the rubric.
   • Three collaboration/aggregation strategies are explored:
     • Weighted Averaging: Each agent i independently scores the judgment across all criteria j. The final score is calculated as a weighted average: $$final\_score = \sum_{i=1}^{N} w_{agent\_i} \sum_{j=1}^{M} w_{c\_j} S_{ij}$$. Agent weights ($w_{agent}$) can be uniform (1/N) or adjusted based on perceived agent reliability. Criterion weights ($w_c$) come from the rubric.
     • Majority Voting: Each agent calculates its overall weighted score ($S_i$). If more than half the agents' scores exceed a predefined threshold $T$, the judgment is effectively given a high score (e.g., 5); otherwise, it gets a low score (e.g., 1). This focuses on consensus.
     • Panel Discussion: Agents engage in a collaborative discussion, potentially influencing each other's assessments before a final score is determined. This can involve agents playing different roles (e.g., expert, critic) or sequential refinement. The paper found this adds significant computational cost and didn't consistently outperform simpler methods for this specific meta-judging task, potentially due to opinion convergence hindering evaluation on difficult problems.
3. Score-Based Selection:
   • A final meta-judge score is obtained from the aggregation strategy in Stage 2.
   • A threshold T (e.g., 4.5 on a 1-5 scale in the paper) is applied to this score.
   • Judgments scoring above the threshold are selected as trustworthy and reliable; those below are filtered out.

Implementation Details and Experimental Findings:

• Dataset: Experiments used JudgeBench (Tan et al., 16 Oct 2024 ), which provides challenging response pairs with objective ground truth labels, allowing for precise evaluation of judgment correctness without human annotation reliance. Raw judgments were generated by models like GPT-4o-mini.
• Meta-Judge Agents: GPT-4o, GPT-4o-mini, Claude-3.5-Sonnet, LLaMa-3.1-405B-Instruct were used.
• Evaluation Metric: Precision was prioritized to ensure the selected judgments are indeed correct.
• Key Results:
  • The meta-judging pipeline significantly improved the precision of selected judgments compared to the raw, unfiltered judgments (approx. 15.5% improvement) and a baseline single-agent meta-judging approach (approx. 8.4% improvement).
  • Multi-agent strategies consistently outperformed single-agent meta-judging. Majority Voting showed the best overall precision improvement (~77% vs ~69% baseline), especially for knowledge and coding tasks. Weighted Averaging was effective for reasoning tasks.
  • Using a more powerful LLM as a meta-judge (e.g., GPT-4o) substantially improved precision when evaluating judgments from a less capable judge LLM (e.g., Llama-3.1-8B).
  • Rubric design matters: More detailed rubrics helped for complex reasoning/math tasks, while simpler rubrics were better for coding.
  • Panel discussion ablation studies suggested two agents were better than three, and assigning specific roles or adding a summarizer did not improve (and sometimes hurt) performance for this task.

Practical Applications and Considerations:

• Automated Quality Control: This framework provides a method to automatically filter LLM judgments, increasing the reliability of LLM-as-a-judge systems.
• Dataset Creation for RLAIF: The primary application is generating high-precision datasets of LLM preferences (selected judgments) that can be used to train better judge LLMs via Reinforcement Learning from AI Feedback, reducing reliance on expensive and potentially biased human feedback.
• Implementation Steps:

1. Define core evaluation criteria for judgments. 2. Use a powerful LLM (e.g., GPT-4) to expand these into a detailed, weighted rubric (see Appendix A for examples). 3. Select 2-3 diverse, capable LLMs as meta-judge agents. 4. Choose an aggregation strategy (Majority Voting or Weighted Averaging are recommended starting points based on the paper's findings due to simplicity and strong performance). 5. Implement the scoring process using the chosen agents and rubric. Aggregate scores. 6. Tune the selection threshold T based on the required precision for the downstream task.

• Cost: Multi-agent approaches increase computational cost (API calls/inference time) compared to single agents. Weighted Averaging/Majority Voting are less costly than Panel Discussion.
• Limitations: The findings are based on the JudgeBench dataset and specific models; generalizability requires further testing.

This research provides a practical blueprint for enhancing the trustworthiness of LLM evaluation systems by using multiple LLMs in a structured meta-evaluation process, paving the way for more robust automated assessment and alignment techniques.