Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities (2503.04721v2)

Abstract: Spoken dialogue modeling poses challenges beyond text-based LLMing, requiring real-time interaction, turn-taking, and backchanneling. While most Spoken Dialogue Models (SDMs) operate in half-duplex mode-processing one turn at a time - emerging full-duplex SDMs can listen and speak simultaneously, enabling more natural conversations. However, current evaluations remain limited, focusing mainly on turn-based metrics or coarse corpus-level analyses. To address this, we introduce Full-Duplex-Bench, a benchmark that systematically evaluates key interactive behaviors: pause handling, backchanneling, turn-taking, and interruption management. Our framework uses automatic metrics for consistent, reproducible assessment and provides a fair, fast evaluation setup. By releasing our benchmark and code, we aim to advance spoken dialogue modeling and foster the development of more natural and engaging SDMs.

• The paper introduces Full-Duplex-Bench, a novel benchmark assessing turn-taking, pause management, backchanneling, and interruption handling in dialogue models.
• It employs automatic metrics such as Takeover Rate and Jensen-Shannon Divergence to quantify conversational performance.
• Empirical evaluations reveal distinct strengths across models, driving improvements in natural and real-time conversational AI.

Evaluation of Full-Duplex Spoken Dialogue Models: The Introduction of Full-Duplex-Bench

The paper presented in the paper entitled "Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities" addresses a significant challenge in spoken dialogue modeling: the evaluation of full-duplex spoken dialogue systems. Full-duplex systems differ fundamentally from half-duplex models as they allow for simultaneous listening and speaking, mimicking natural human conversation dynamics more closely. However, a systematic framework for evaluating these systems has been lacking.

Spoken dialogue models, particularly with the advent of numerous voice assistants, have highlighted the critical importance of functionalities such as turn-taking, backchanneling, and real-time interaction capabilities. While half-duplex SDMs, predominant in the current landscape, process speech sequentially, full-duplex models enable more seamless conversational flow by allowing synchronous processing. This paper posits that the evaluation of such systems has predominantly focused on rudimentary turn-based metrics or analyzed corpus statistics related to conversational gaps and pauses, thus inadequately capturing the intricacies of real-time interaction enabled by full-duplex systems.

To fill this evaluative gap, the authors propose "Full-Duplex-Bench," a nuanced benchmark designed to assess full-duplex SDMs comprehensively. The benchmark evaluates models based on four critical conversational attributes: pause management, backchanneling, proficient turn-taking, and handling user interruptions. The framework utilizes automatic metrics, ensuring consistent and reproducible evaluations of a model's interactive performance.

Key Methodological Framework

The Full-Duplex-Bench framework operates on a multi-dimensional evaluation strategy:

1. Pause Handling: Evaluates if the model can recognize natural conversational pauses without assuming control, which is measured by the Takeover Rate (TOR).
2. Backchanneling: Assesses the model's capacity to provide timely and meaningful non-intrusive feedback during conversations, using metrics such as TOR, backchannel frequency, and Jensen-Shannon Divergence to quantify alignment with human behavior.
3. Smooth Turn-taking: Determines the model's ability to transition conversational turns efficiently, focusing on reducing latency in response time post-user input.
4. User Interruption Management: This examines the model's adaptability to interruptions, assessing how quickly and coherently a model responds after a user-initiated interruption.

Empirical Evaluation

This research evaluated three models: dGSLM, Moshi, and Freeze-Omni, using the proposed benchmark. Results revealed nuanced insights into model performance:

• Pause Handling: Freeze-Omni demonstrated lower interruption rates during pauses, suggesting it effectively utilizes a state-control component to wait for turn completion more accurately than the end-to-end models which frequently interrupted during pauses.
• Backchanneling: dGSLM displayed superior backchannel behaviors, offering timely feedback akin to human interaction. In contrast, Freeze-Omni generally remained silent, and Moshi's persistent responses indicated low effectiveness in backchanneling.
• Turn-taking and Interruption Handling: While Freeze-Omni demonstrated optimal interruption handling with more coherent and contextually relevant response capabilities, end-to-end models, particularly Moshi, showed limited semantic coherence when managed interruptions.

Implications and Future Directions

The introduction of Full-Duplex-Bench represents a pivotal step toward advancing the development of more interactive full-duplex SDMs. The benchmark's focus on real-time, conversationally relevant metrics addresses the previously unmet need for robust evaluation mechanisms in this domain. Practically, better benchmarks drive the improvement of conversational AI, impacting applications ranging from customer service to virtual reality.

Future developments should consider expanding beyond English-only corpora to capture cross-linguistic conversational dynamics. Additionally, exploring backchannel detection with more sophisticated models that integrate lexical and prosodic data can yield greater accuracy. Addressing user interruptions in closed-source commercial systems and incorporating non-verbal conversational elements present further research opportunities to refine the naturalness and functionality of SDMs.

By making the benchmark and related resources publicly available, the authors encourage wider adoption and enhancement of these evaluation standards, driving significant improvements in the realization of natural spoken dialogue systems.