ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate (2308.07201v1)

Abstract: Text evaluation has historically posed significant challenges, often demanding substantial labor and time cost. With the emergence of LLMs, researchers have explored LLMs' potential as alternatives for human evaluation. While these single-agent-based approaches show promise, experimental results suggest that further advancements are needed to bridge the gap between their current effectiveness and human-level evaluation quality. Recognizing that best practices of human evaluation processes often involve multiple human annotators collaborating in the evaluation, we resort to a multi-agent debate framework, moving beyond single-agent prompting strategies. The multi-agent-based approach enables a group of LLMs to synergize with an array of intelligent counterparts, harnessing their distinct capabilities and expertise to enhance efficiency and effectiveness in handling intricate tasks. In this paper, we construct a multi-agent referee team called ChatEval to autonomously discuss and evaluate the quality of generated responses from different models on open-ended questions and traditional natural language generation (NLG) tasks. Our analysis shows that ChatEval transcends mere textual scoring, offering a human-mimicking evaluation process for reliable assessments. Our code is available at https://github.com/chanchimin/ChatEval.

ChatEval: Multi-Agent Collaboration for Enhanced LLM-Based Text Evaluation

The paper "ChatEval: Towards better LLM-based evaluators through multi-agent debate" presents a novel approach to improving text evaluation methodologies by harnessing the potential of multiple LLMs working in concert. Historically, text evaluation has been an intricate task, often reliant on human annotators, which is time-intensive and costly. The advent of LLMs has opened up possibilities for automating this process; however, current single-agent LLM systems have not yet reached the level of human evaluators in terms of effectiveness and accuracy.

Key Components and Methodology

The researchers introduce ChatEval, a multi-agent debate framework designed to enhance the evaluation of LLM-generated responses by employing a team of LLMs as agents. This framework draws inspiration from human group evaluation practices, which often involve multiple perspectives to improve reliability and mitigate bias. ChatEval consists of several core components:

1. Debater Agents: Each LLM functions as an individual agent, taking on a unique role, or "persona," within the evaluation process. These roles ensure that agents approach evaluations from diverse perspectives, leading to more nuanced assessments.
2. Communication Strategies: The paper outlines various communication strategies (e.g., One-By-One, Simultaneous-Talk, and Simultaneous-Talk-with-Summarizer) that dictate how agents interact during debates. These strategies help maintain dynamic interactions and efficient information sharing among agents.
3. Role Specification and Diversity: Diverse role prompts are essential for the multi-agent framework, as they ensure that each LLM agent contributes uniquely, preventing performance degradation that can occur when identical roles are assigned.

Experimental Evaluation

ChatEval's performance was rigorously tested on two benchmark tasks: FairEval (open-ended question-answer evaluation) and Topical-Chat (dialogue response generation). The experimental results demonstrate that ChatEval, with its multi-agent setup, surpasses the accuracy and alignment with human judgments compared to single-agent and other extant LLM-based evaluators. Notably, ChatEval improved accuracy by 6.2% for ChatGPT and 2.5% for GPT-4 over single-agent baselines in the FairEval task.

Implications and Future Directions

The implications of ChatEval are significant for both theoretical and practical aspects of AI and NLP. Theoretically, the framework highlights the efficacy of collaborative debate mechanisms in refining LLM-based evaluations, suggesting that diversity in role perspectives is crucial for achieving human-equivalent performance levels. Practically, ChatEval provides a scalable, less labor-intensive alternative to traditional human annotation methods, fostering more reliable automated text evaluation processes.

The paper paves the way for future research exploring heterogeneous agent groups, possibly combining multiple types of models within a single system, which could further enhance the robustness and depth of evaluations. Additionally, refining communication strategies and roles could lead to even more effective collaborative evaluation frameworks.

In summary, ChatEval represents a significant step forward in the field of LLM-based text evaluation, demonstrating the potential of multi-agent systems to both emulate and surpass human evaluators in accuracy and efficiency. This work opens new avenues for developing AI systems that not only process information but also critically assess and refine each other's outputs through structured debate and collaboration.