Full-Duplex-Bench: Real-Time Dialogue Benchmark

• Full-Duplex-Bench is an open benchmarking framework that evaluates full-duplex spoken dialogue models by capturing overlapping speech, barge-ins, and backchanneling.
• The framework employs key metrics such as Takeover Rate, response latency, and Backchannel Frequency to quantify nuanced real-time dialogue behaviors.
• It enables reproducible evaluation for both academic and commercial systems, driving improvements in natural conversation while highlighting challenges like language limitations.

Full-Duplex-Bench is an open, systematic benchmarking framework for quantifying the interactive capabilities of full-duplex spoken dialogue models (SDMs) that both listen and speak simultaneously under realistic, real-time conditions. Unlike traditional half-duplex evaluation, which assesses models in rigid, sequential turn-taking, Full-Duplex-Bench specifically targets the nuanced, overlapping, and interruptive phenomena characteristic of natural human conversation. Across its versions and extensions, Full-Duplex-Bench has established itself as the primary yardstick for evaluating fundamental dialogue behaviors, robustness to conversational disfluency, tool-use performance, and advanced scenario management in both open-source and commercial full-duplex SDMs (Lin et al., 6 Mar 2025, Lin et al., 30 Jul 2025, Lin et al., 9 Oct 2025, Lin et al., 6 Apr 2026).

1. Background and Motivation

Historically, nearly all spoken dialogue models operated in half-duplex: the user speaks, the agent waits, listeners alternate in strictly serialized turns, and models are evaluated on content accuracy or coarse temporal statistics. This structure inherently misses critical real-time behaviors found in human-human dialogue, such as:

• Overlapping speech: Natural conversations feature frequent overlaps, pauses, and backchannel feedback.
• Barge-in and repair: Effective agents must correctly handle user interruptions, revisions, and rapid turn-exchanges.
• Backchanneling and engagement: Listeners signal engagement non-intrusively, an essential dynamic for conversational naturalness.

Full-duplex SDMs, exemplified by systems like GPT-4o voice mode, Moshi, and Freeze-Omni, allow simultaneous listening and speaking but introduce new failure modes: inappropriate interruptions, ignored user cues, and incorrect response timing. There was no reproducible, open-standard method for evaluating these capabilities until the introduction of Full-Duplex-Bench (Lin et al., 6 Mar 2025). Subsequent iterations have expanded the evaluation axis to multi-role, multi-task, and real-world tool-use settings (Lin et al., 30 Jul 2025, Lin et al., 9 Oct 2025, Lin et al., 6 Apr 2026).

2. Core Behavioral Dimensions and Metric Formalizations

The original Full-Duplex-Bench formalizes full-duplex interaction along four primary behavioral axes (Lin et al., 6 Mar 2025):

• Pause Handling: Determines if a model misclassifies intra-turn user silences (e.g., hesitations) as turn boundaries. The metric is the Takeover Rate (TOR), where TOR$\to$0 is ideal.

$$\mathrm{TOR} = \frac{1}{N}\sum_{i=1}^N \mathrm{TO}_i, \quad \mathrm{TO}_i = \begin{cases} 1 & \text{model takes over} \ 0 & \text{otherwise} \end{cases}$$

• Backchanneling: Quantifies the model’s ability to offer brief, well-timed listener feedback. Metrics: TOR (for deferring turns), Backchannel Frequency (per second of user speech), and Jensen–Shannon Divergence (JSD) for alignment with human backchannel timing distributions.
• Smooth Turn-Taking: Measures response latency and temporal coordination after user turn-ends:

$$L = \frac{1}{K} \sum_{i:\mathrm{TO}_i=1} (t^{(m)}_{\mathrm{start},i} - t^{(u)}_{\mathrm{end},i})$$

• User Interruption Management: Assesses model reaction to mid-utterance user barge-ins—detecting, yielding, and responding to new intent. Metrics include TOR (should be 1 in interruption condition), GPT-4o semantic relevance score (0–5), and response latency post-interruption.

Later versions, such as Full-Duplex-Bench v1.5, introduce categorical dialogue behaviors (Respond, Resume, Uncertain, Unknown), stop/response latency, prosodic adaptation (pitch, intensity, speaking rate), and MOS prediction for segmental speech quality (Lin et al., 30 Jul 2025).

3. Data, Pipelines, and Experimental Protocols

Full-Duplex-Bench assembles carefully partitioned corpora and automated measurement scripts for reproducible experimentation:

• Pause/Turn-Taking Data: Extracted from large-scale, spontaneous two-channel English dialog datasets (e.g., CANDOR), filtered by intra-turn pause or inter-turn gap duration for targeting the correct phenomena (Lin et al., 6 Mar 2025).
• Backchanneling: Uses annotated corpora with human backchannel timing, segmented and normalized to provide reference distributions (Lin et al., 6 Mar 2025).
• Synthetic Interruptions: Scenarios are generated via prompt-based synthesis (e.g., GPT-4o, ChatTTS), spanning a range of interruption types and timing conditions (Lin et al., 6 Mar 2025, Lin et al., 30 Jul 2025).
• Streaming Evaluation: All model outputs and user streams are resampled, transcribed (e.g., with CrispyWhisper or Parakeet-TDT), then segmented via VAD for precise timestamp annotation. Metric computation is fully automated for batch evaluation with user-configurable thresholds and scenario selection.

The benchmark includes baseline results on leading open-source systems (dGSLM, Moshi, Freeze-Omni), systematically reported across all metrics (Lin et al., 6 Mar 2025).

4. Key Results and Model Comparison

Experimental evaluations reveal significant differences in how modern SDMs negotiate full-duplex settings. Comparative findings include (Lin et al., 6 Mar 2025, Lin et al., 30 Jul 2025, Lin et al., 6 Apr 2026):

Dimension	dGSLM (TOR)	Moshi (TOR)	Freeze-Omni (TOR)
Pause Handling (synthetic)	0.949	1.000	0.672
Pause Handling (Candor)	0.953	0.989	0.287
Backchannel (TOR / Freq / JSD)	0.782/0.013/0.950	1.000/0.005/0.977	0.782/0.002/0.984
Smooth Turn-Taking (TOR/Latency)	0.989/0.572	0.996/0.112	0.369/1.168
User Interruption (TOR/Score/Lat)	0.895/0.201/3.972	1.000/0.765/0.037	0.795/3.371/1.200

• MOSHI replies with minimal latency but frequently interrupts or under-backchannels.
• Freeze-Omni’s explicit state-control yields the lowest TOR in pause handling but at the expense of increased response latency.
• dGSLM produces the most human-like backchannel timing (lowest JSD) but exhibits over-eager turn responses.

Benchmarks on advanced scenario categories—overlap, background distractors, corrections, and tool chaining—consistently show that interruption management and self-correction remain open research challenges for all architectures (Lin et al., 30 Jul 2025, Lin et al., 9 Oct 2025, Lin et al., 6 Apr 2026).

5. Extensibility, Language Coverage, and Successors

Full-Duplex-Bench is designed for plug-and-play extensibility:

• New audio assets: Support for real or synthetic speech; audio-asset interchangeability with explicit onset/offset annotation.
• Scenario injection: Arbitrary overlap types, environmental conditions (side-conversations, ambient noise), and complex barge-in profiles.
• Language extension: Localization through scenario/script translation and normalization of the segmentation/VAD pipeline.

All code, scenario definitions, and metric scripts are open-sourced, facilitating easy adaptation and extension to novel research questions (Lin et al., 6 Mar 2025, Lin et al., 30 Jul 2025).

• Full-Duplex-Bench v1.5: Systematic analysis of overlap handling introduces repair-first versus continuity-first agent strategies, scenario-specific performance trends, and prosodic adaptation statistics (Lin et al., 30 Jul 2025).
• FDB-v2 and v3: Integrate automated examiners (streaming speech-LLMs), multi-turn stateful tasks (entity tracking, corrections, safety), disfluency-annotated real human audio, and multi-step tool-use chains (Lin et al., 9 Oct 2025, Lin et al., 6 Apr 2026).

6. Applications and Limitations

Applications:

• Rigorous, reproducible benchmark for emerging speech-driven LMs in academia and industry.
• Diagnostic tool for pinpointing conversational breakdowns in full-duplex agent architectures—enabling ablations and architectural comparisons.
• Essential for measuring role-conditional behavior, naturalness (DMOS), timing fluidity, memory in multi-turn, and robustness to disfluent, noisy, or adversarial input.

Limitations:

• Initial releases limited to English; cross-cultural turn-taking and backchannel cues are not covered (Lin et al., 6 Mar 2025).
• Heuristic detection for backchannels; paralinguistic (laughter, sighs) and non-lexical phenomena not comprehensively measured.
• Commercial, closed-source systems must be evaluated via API wrappers; not locally reproducible.
• Subjective scoring uses automated LLMs (e.g., GPT-4o) in lieu of crowd-sourced human ratings; future work could improve external validity (Lin et al., 6 Mar 2025, Lin et al., 9 Oct 2025, Lin et al., 6 Apr 2026).

7. Impact and Outlook

Full-Duplex-Bench has defined core quantitative standards for turn-taking, interruption, backchanneling, and tool-use in real-time voice agents. Extensions support advanced conversational setting: role-specific interactions, multi-turn flows, and naturalistic disfluency. The benchmark’s automation, open-source release, and modular architecture have enabled rapid progress and transparent comparison across leading full-duplex agent platforms (Lin et al., 6 Apr 2026).

Further advances are directed at expanding language and cultural coverage, integrating richer paralinguistic markers, combining LLM and human evaluation for subjective metrics, and supporting a broader array of deployment scenarios (e.g., multi-party, multi-modal, cross-lingual). Full-Duplex-Bench remains the reference test suite for spoken dialogue models seeking human-parity in conversational dynamics.