Full-Duplex-Bench v1.5: Overlap Evaluation
- Full-Duplex-Bench v1.5 is a modular, fully automated benchmark designed to evaluate overlap handling in full-duplex speech agents using controlled scenarios.
- It distinguishes four overlap scenarios—user interruption, listener backchannel, side conversation, and background speech—by measuring dialogue behavior, stop latency, response latency, and prosodic adaptation.
- The benchmark provides actionable insights through detailed metrics on addressee discrimination and speech quality, facilitating diagnostic and extensible assessment for conversational systems.
Full-Duplex-Bench v1.5 is a modular, fully automated benchmark for evaluating overlap handling in full-duplex speech agents, understood here as systems that can speak and listen concurrently rather than following a strictly alternating turn-taking regime. It was introduced to measure how such agents behave when overlapping speech occurs during ongoing system output, with controlled scenarios for user interruption, listener backchannel, side conversation, and ambient speech, and with metrics spanning categorical dialogue behavior, stop and response latency, prosodic adaptation, and perceived speech quality (Lin et al., 30 Jul 2025).
1. Scope and conceptual basis
Full-Duplex-Bench v1.5 is motivated by the claim that overlap management is central to natural, low-latency spoken dialogue and remains under-evaluated in conventional speech benchmarks. In the benchmark’s framing, natural conversation routinely includes interruptions, acknowledgments, speech addressed to others, and background talk; the paper cites prior conversational analysis indicating that over 40% of turns involve some degree of overlap. The benchmark therefore targets a capability set broader than ordinary turn-taking: a model should detect overlap quickly, infer whether the overlapping speech is directed at the system, decide whether to stop, continue, respond, or ignore, resume coherently, and adapt timing and speaking style appropriately (Lin et al., 30 Jul 2025).
The benchmark is positioned against several limitations in prior evaluation practice. Most earlier benchmarks are described as half-duplex or single-turn, which leaves simultaneous speaking and listening largely untested. Human evaluation is acknowledged as nuanced but costly and difficult to reproduce at scale. Corpus-level turn-taking measures based on VAD, inter-pausal units, or floor-transfer offsets are treated as too coarse to isolate scenario-specific overlap behavior. Classifier-based approaches are described as less generalizable because they depend on corpus-specific training and annotation protocols. Within the Full-Duplex-Bench family itself, version 1.5 is presented as extending an earlier benchmark that focused mainly on interruptions toward a broader, explicitly overlap-centered design (Lin et al., 30 Jul 2025).
A central conceptual distinction in v1.5 is between speech addressed to the agent and speech that merely co-occurs acoustically with the agent’s output. This distinction is tested under both same-speaker and different-speaker conditions. The benchmark therefore evaluates not only overlap detection, but also a form of addressee discrimination under controlled acoustic variation. This suggests a shift from generic barge-in sensitivity toward a more interactionally differentiated notion of full-duplex competence.
2. Overlap taxonomy and scenario construction
The benchmark simulates four overlap scenarios: user interruption, listener backchannel, user talking to others, and background speech. Each trial is parameterized by context speech in the user channel, overlap timing relative to the model’s ongoing speech, overlap content, and speaker identity and acoustic channel. For analysis, the model output is segmented into pre-overlap and post-overlap regions using the end time of the overlap event, and most semantic metrics are computed on the post-overlap portion (Lin et al., 30 Jul 2025).
To ensure that the injected overlap occurs while the model is still speaking, the authors first measure each model’s clean-context response latency and output duration. Across the five evaluated systems, mean response latency lies between 1.18 and 2.25 seconds, and mean output duration lies between 6.79 and 10.28 seconds. On that basis, the benchmark inserts an approximately 4-second silence gap between the initial context and the overlap onset so that the overlap reliably falls during ongoing system output (Lin et al., 30 Jul 2025).
| Scenario | Construction | Intended post-overlap behavior |
|---|---|---|
| User interruption | Same speaker; follow-up statement during model speech; 200 samples | Respond |
| Listener backchannel | Same speaker; short acknowledgments such as “uh-huh” or “yeah”; 99 samples | Resume |
| User talking to others | Same speaker; non-system-related name; volume reduced by about 8 dB; high-shelf attenuation above 4 kHz by 5 dB; reflections at 45 ms/6 dB and 120 ms/12 dB; 100 samples | Resume or silence |
| Background speech | Different speaker; unrelated content; volume reduced by 15 dB; low-pass at 3 kHz; echo at 100 ms/10 dB; 100 samples | Resume |
The user interruption condition represents a genuine barge-in: the user cuts in with a follow-up statement while the model is speaking, using the same speaker voice and without acoustic alteration. The benchmark treats rapid stopping and meaningful redirection to the new utterance as the desired outcome. The listener backchannel condition instead uses short acknowledgment tokens and is designed to test whether the model can continue smoothly rather than overreacting to non-floor-taking feedback. The “user talking to others” condition preserves speaker identity but changes semantic addressee and far-field characteristics, making it a same-speaker but non-addressed case. The background speech condition changes both speaker identity and acoustics to simulate incidental third-party speech (Lin et al., 30 Jul 2025).
This scenario design makes v1.5 unusual among speech benchmarks in that it separates overlap by conversational function rather than treating all simultaneous speech as a single phenomenon. Later survey work formalized this by arguing that v1.5’s four-scenario decomposition aligns nearly one-to-one with the overlap row of a broader interaction ontology for full-duplex dialogue (Lu et al., 17 Jun 2026).
3. Evaluation pipeline and metric suite
The benchmark is architecture-agnostic and operates at the level of streamed audio input and streamed audio output, which allows it to evaluate both open-source local systems and closed commercial APIs. Pre-recorded user speech is fed to the model in a streaming manner while the model’s streaming output is collected and saved. Post-hoc evaluation uses Nvidia parakeet-tdt-0.6b-v2 for transcription and word-level timestamps, silero_vad for voice activity timing, GPT-4o for behavior classification, acoustic-prosodic feature extraction for pitch, intensity, and speaking rate, and UTMOSv2 for predicted MOS on non-silent regions (Lin et al., 30 Jul 2025).
The principal semantic metric is categorical dialogue behavior. The benchmark classifies post-overlap output into four reported categories: Respond, Resume, Uncertain, and Unknown. The paper’s broader framing also mentions silence, but the reported behavior tables group results under those four labels. The meaning of a “good” label is scenario dependent. In interruption trials, Respond is generally desirable. In backchannel and non-addressed overlap scenarios, Resume is generally desirable. Uncertain corresponds to clarification-style behavior, while Unknown denotes semantically unrelated or low-information output (Lin et al., 30 Jul 2025).
The core timing metrics are stop latency and response latency. Operationally, these are defined as
and
Stop latency measures how quickly the model yields after overlapping speech begins. Response latency measures how quickly the model re-enters after the overlap ends. Their interpretation is scenario sensitive: low stop latency is desirable under true interruption, but indiscriminate rapid stopping can be counterproductive under backchannel or ambient speech (Lin et al., 30 Jul 2025).
Prosodic analysis compares three conditions: non-overlap, pre-overlap, and post-overlap. The benchmark computes mean and standard deviation of pitch over voiced frames, mean and standard deviation of intensity over speech-active regions, and speaking rate in words per minute excluding silences. A descriptive form of the speaking-rate calculation is
These measures are not treated as scalar “higher is better” metrics; rather, they quantify adaptation after overlap. Statistical testing uses paired t-tests with a significance threshold of , and the published prosody analysis is restricted to respond-labeled output segments in the user interruption condition (Lin et al., 30 Jul 2025).
Perceived quality is estimated using UTMOSv2. This provides a predicted MOS-style measure of whether overlap handling degrades speech naturalness. The benchmark does not define a composite score across behavior, timing, prosody, and quality, which reinforces its role as a diagnostic suite rather than a single leaderboard scalar.
4. Evaluated systems and empirical patterns
The reported study evaluates five full-duplex speech agents: Freeze-Omni and Moshi as open-source systems, and Gemini Live, Nova Sonic, and GPT-4o Realtime as commercial APIs. The specific commercial endpoints are gemini-2.0-flash-live-001 with voice id Puck, amazon.nova-sonic-v1:0, and gpt-4o-realtime-preview-2024-12-17 with voice id alloy (Lin et al., 30 Jul 2025).
The benchmark highlights two principal strategies. One is repair-first rapid yielding, associated most strongly with GPT-4o Realtime and, to a lesser extent, Freeze-Omni. These systems stop quickly and often treat overlap as something that demands explicit repair or redirection. The other is continuity-first sustained flow, associated with Gemini and Nova Sonic. These systems preserve output continuity through overlap, which helps under backchannel and non-addressed speech, but can yield multi-second double-talk under genuine interruption. Moshi is reported as less stable than either of these poles and frequently falls into Unknown behavior (Lin et al., 30 Jul 2025).
The behavior tables show these differences sharply. In user interruption, Respond rates are 0.78 for GPT-4o, 0.72 for Freeze-Omni, 0.50 for Moshi, 0.33 for Gemini, and 0.24 for Sonic. In listener backchannel, Resume rates are 0.98 for Sonic, 0.93 for Gemini, 0.80 for Freeze-Omni, 0.70 for GPT-4o, and 0.06 for Moshi. In the side-conversation condition, GPT-4o responds to non-addressed speech at rate 0.91, whereas Gemini resumes at 0.99 and Sonic at 0.90. In background speech, Sonic resumes at 0.98, while GPT-4o responds at 0.93 and Gemini at 0.70, indicating substantial false engagement with incidental speech in those systems (Lin et al., 30 Jul 2025).
Latency results reinforce the same split. GPT-4o is the fastest stopper in all four scenarios, with stop latency between 0.18 and 0.23 seconds. Freeze-Omni is slower but still comparatively reactive. Gemini and Sonic are much slower to yield under genuine interruption, with interruption stop latencies of 2.20 and 2.25 seconds respectively. The paper compares these numbers with a typical human response latency of around 200 ms and notes that even the best benchmarked system’s re-entry latency remains far above human timing; the best reported re-entry is 1.16 seconds, implying current systems are still roughly five to six times slower than typical human responsiveness (Lin et al., 30 Jul 2025).
Prosodic analysis further differentiates system behavior under interruption. GPT-4o increases speaking rate and raises pitch while preserving relatively stable loudness, which suggests an active repair style. Gemini also speeds up after interruption. Freeze-Omni and Sonic are more prosodically stable. Moshi accelerates strongly but becomes flatter and lower in pitch. Predicted speech quality generally declines after interruption, with the largest degradations reported for Sonic and Moshi; GPT-4o is comparatively stable within-turn, though slightly worse than its clean non-overlap baseline (Lin et al., 30 Jul 2025).
One especially notable empirical result is the near absence of clarification-style strategies. Uncertain remains under 5% for all models across scenarios. The paper interprets this as evidence that current systems rarely choose to acknowledge ambiguity explicitly, even when social appropriateness might favor clarification over either blunt continuation or aggressive response.
5. Place within the Full-Duplex-Bench lineage
Within the benchmark family, the original Full-Duplex-Bench centered on pause handling, backchanneling, smooth turn-taking, and user interruption, using automatic metrics such as takeover rate, backchannel frequency, Jensen–Shannon divergence, and response latency (Lin et al., 6 Mar 2025). Full-Duplex-Bench v1.5 preserves the family’s emphasis on streaming, automatic evaluation but shifts the focal point from generic turn-taking dynamics toward controlled overlap handling, with four canonical overlap scenarios and a metric suite that explicitly includes behavior labels, stop and response latency, prosodic adaptation, and perceived quality (Lin et al., 9 Oct 2025).
Later work treats v1.5 as a distinct intermediate stage in the family’s evolution. Full-Duplex-Bench-v2 describes v1.5 as the version that standardized four canonical overlap events—user interruption, user backchannel, user talking to others, and background speech—and expanded outputs to behavior labels such as RESPOND, RESUME, and UNCERTAIN plus overlap-sensitive timing and speech-quality measures (Lin et al., 9 Oct 2025). Full-Duplex-Bench-v3, in turn, characterizes v1.5 as the stage that expanded the benchmark line to overlapping scenarios before later versions incorporated multi-turn evaluation, real human audio, disfluency annotation, and tool use (Lin et al., 6 Apr 2026).
A later survey gives the most explicit theoretical reinterpretation of v1.5. It argues that v1.5 is best understood not as a generic full-duplex scorecard, but as a strong benchmark for the overlap slice of full-duplex dialogue. In that survey’s terminology, v1.5’s scenario decomposition aligns nearly one-to-one with the overlap row of a broader interaction ontology, and its stop-latency metric is described as the closest existing public metric to a substantive full-duplex test (Lu et al., 17 Jun 2026). This suggests that the scholarly significance of v1.5 lies not only in its measured results, but also in having turned overlap into a scenario-controlled evaluation object.
6. Limitations, extensibility, and research use
The benchmark explicitly acknowledges several limitations. All overlap events are generated with TTS rather than recorded human speech, which improves timing control and reproducibility but leaves a domain gap with respect to spontaneous emotion, disfluency, microphone variation, and natural overlap timing. The release is English only. Scenarios focus on single short overlaps injected into otherwise clean dialogues rather than on repeated barge-ins, long overlap cascades, overlapping laughter, or dense multi-party thread switching. The design is therefore best understood as controlled and diagnostic rather than fully ecological (Lin et al., 30 Jul 2025).
At the same time, extensibility is a core design claim. The benchmark is described as open-source and modular, with code, metrics, and annotation schemas made available, and with support for replacing synthetic overlaps with recorded human speech, adding non-English and code-switching scenarios, integrating crowd ratings for naturalness and social appropriateness, and extending toward multi-party, multi-device settings (Lin et al., 30 Jul 2025). This suggests that v1.5 functions both as a benchmark instance and as a reusable evaluation template.
For research use, v1.5 is most informative when treated as a diagnostic stress test rather than a single-score competition. Safety-critical or hands-free systems may prioritize high interruption Respond rates and low stop latency. Shared-environment assistants may instead prioritize Resume under side conversation and ambient speech, with low false activation. Conversational agents aimed at human-likeness may additionally care about prosodic adaptation and speech-quality stability. The benchmark’s principal contribution is that it makes these trade-offs measurable under controlled overlap conditions rather than subsuming them into coarse turn-taking averages.