Survey on Evaluation Methods for Dialogue Systems (1905.04071v2)

Abstract: In this paper we survey the methods and concepts developed for the evaluation of dialogue systems. Evaluation is a crucial part during the development process. Often, dialogue systems are evaluated by means of human evaluations and questionnaires. However, this tends to be very cost and time intensive. Thus, much work has been put into finding methods, which allow to reduce the involvement of human labour. In this survey, we present the main concepts and methods. For this, we differentiate between the various classes of dialogue systems (task-oriented dialogue systems, conversational dialogue systems, and question-answering dialogue systems). We cover each class by introducing the main technologies developed for the dialogue systems and then by presenting the evaluation methods regarding this class.

• The paper surveys diverse evaluation methods for different dialogue system types, including task-oriented, conversational, and question-answering systems, highlighting the need for automation beyond costly human evaluation.
• Evaluation criteria and methods vary significantly by system type, focusing on task success for task-oriented systems, response appropriateness for conversational systems (critiquing metrics like BLEU), and answer correctness for QA systems.
• Key challenges include the lack of standardized datasets and the need for innovative metrics that capture the multidimensional nature of effective dialogue, with ongoing benchmarking efforts aiming to advance the field.

Evaluation Methods for Dialogue Systems: A Survey

This survey paper, authored by Deriu et al., offers a comprehensive examination of the methods utilized in evaluating dialogue systems. It encapsulates the core challenge of evaluating varying dialogue systems, including task-oriented, conversational, and question-answering systems. The motivation behind this exploration is the substantial cost and time investment required for human evaluations, motivating research into automation of evaluation processes.

The paper delineates dialogue systems into three categories: task-oriented, conversational, and question-answering systems, each necessitating distinct evaluation approaches due to their inherent characteristics:

1. Task-Oriented Systems: These are intended to accomplish specific tasks efficiently within structured dialogue, such as finding restaurant information or booking tickets. The evaluation here focuses on task success and dialogue efficiency, often using user satisfaction modeling or user simulation. User satisfaction modeling aims to correlate objective measures such as task success and number of dialogue turns with user-rated satisfaction. User simulations, conversely, are crafted to simulate user interactions as a means to evaluate dialogue strategies efficiently, a method that is relatively robust due to its allowance for systematic comparisons across different strategies.
2. Conversational Systems: Unlike the task-oriented systems, conversational systems engage in open-domain exchanges largely for entertainment or emulation of human-like interactions. The evaluation here hinges on the appropriateness of responses and the system’s ability to emulate human-like characteristics. Evaluations are typically coarse-grained due to the broad definitions of what constitutes a "good" conversation. The paper critiques classical approaches like BLEU for their poor correlation with human judgments and introduces models like ADEM, which utilize neural networks to predict human ratings of appropriateness.
3. Question-Answering (QA) Systems: These systems focus on providing concise answers to questions, whether single-turn or interactive dialogues with contextual dependencies. While the primary metric remains the correctness of the answers, increasingly, there is acknowledgment of the need for dialogue evaluation to incorporate the conversational aspect, such as flow and coherence.

The survey proceeds to outline the datasets available for each type of dialogue system, drawing attention to challenges like lack of standardization and representativeness, which complicate comparisons across different systems. It also highlights recent and ongoing benchmarking efforts, such as the Dialog State Tracking Challenge (DSTC) and the Conversational Intelligence Challenge (ConvAI), which continue to catalyze advances by providing a common platform and data for evaluation.

This comprehensive synthesis illustrates the nuance in evaluating dialogue systems, recognizing that current approaches, while varied, require substantial improvement to fully automate evaluations and ensure they are both repeatable and reflective of user perceptions. The paper advocates for the development of innovative metrics that inherently capture the multidimensional aspect of what makes dialogue systems effective, extending beyond simple models of task completion or dialogue act prediction.

In conclusion, this paper not only surveys the state-of-the-art but sets a context for the ongoing challenges and future directions in dialogue system evaluation. The drive towards more automated, high-fidelity evaluation methods is evident, and the paper posits lifelong learning and dynamic environment adaptation as emerging frontiers in dialogue systems’ development and assessment. The implications for artificial intelligence, particularly in human-computer interaction, are manifold, providing both a rich field for academic inquiry and practical application.