---
title: 'FDB-v3: Full-Duplex Spoken Interaction Benchmark'
url: https://www.emergentmind.com/topics/full-duplex-bench-v3-fdb-v3
type: topic
---

# FDB-v3: Full-Duplex Spoken Interaction Benchmark

Full-Duplex-Bench-v3 (FDB-v3) is a benchmark for evaluating spoken language models under naturalistic speech conditions and multi-step tool use. It is designed for real-time, full-duplex voice agents that must listen continuously, speak with appropriate timing, and execute chained API calls despite fillers, pauses, hesitations, false starts, and mid-utterance self-corrections. Its dataset consists entirely of real human audio, and its evaluation combines task completion, tool-use correctness, latency, and turn-taking behavior in a single framework [2604.04847].

## 1. Historical placement and benchmark scope

FDB-v3 belongs to a line of full-duplex spoken interaction benchmarks, but it targets a broader problem setting than earlier turn-taking evaluations. Earlier Full-Duplex-Bench work focused on pause handling, backchanneling, smooth turn taking, and user interruption management, with automatic metrics derived from time-aligned audio and transcripts [2503.04721]. FDB-v3 extends that agenda to a setting in which the agent must not only manage conversational timing, but also perform multi-step external actions under disfluent speech [2604.04847].

Its defining premise is that three capabilities must be evaluated together: full-duplex interaction, tool use with chained API calls, and robustness to real-world speech disfluency. FDB-v3 positions itself against several earlier benchmark families. Prior Full-Duplex-Bench versions are described as evaluating turn-taking with largely synthetic TTS audio and without multi-step tool use; Audio MultiChallenge and WildSpeech-Bench use real human speech but do not evaluate tool use; and τ-Voice, AudioCRAG, and VoiceAgentBench evaluate tool use with synthetic audio and often single-step tasks [2604.04847]. A plausible implication is that FDB-v3 should be read not merely as a turn-taking benchmark, but as an integrated benchmark for streaming speech interaction, online reasoning, and external action execution.

The benchmark therefore evaluates failure modes that do not appear in text-only or turn-bounded settings. A representative case is a request such as “Book me a flight um… to New York — actually, wait… make that Boston,” where a robust system must avoid prematurely calling tools with the obsolete destination and must revise its internal state when the correction arrives [2604.04847].

## 2. Dataset composition, disfluency taxonomy, and task domains

FDB-v3 contains 100 recordings, one per scenario, from 12 speakers that include native and non-native English speakers, accents from Korean and Russian backgrounds, and varying accent strength [2604.04847]. Recording conditions are intentionally ordinary rather than laboratory-grade: 11 of 12 speakers use built-in laptop or phone microphones, and environments range from quiet rooms to mild background noise [2604.04847]. Speakers are asked to perform prompts organically rather than read scripts.

The corpus design also incorporates non-synthetic silence. Each speaker contributes 30 seconds of real ambient noise from the recording environment, and this is used instead of digital silence at the tail of utterances [2604.04847]. Each speaker is assigned 10 scenarios spanning all four task domains, and 21 of the 100 scenarios are explicitly designed to include self-correction events requiring state rollback [2604.04847].

FDB-v3 annotates every recording for the presence of five disfluency categories. **Fillers** are canonical items such as “um,” “uh,” “like,” and “you know” that do not change semantics. **Pauses** are mid-utterance silences that stress end-of-turn detection. **Hesitations** are filler-repetition combinations such as “I need a fl– uh, a flight to…”. **False starts** occur when a speaker begins one intent and abandons it for another. **Self-corrections** alter parameters mid-sentence, as in changing a destination or date after an initial specification [2604.04847].

The benchmark’s task layer is built around locally executed mock APIs with deterministic, zero-latency responses, fixed signatures, and fixed behavior. This isolates model reasoning and interaction behavior from backend network variance [2604.04847]. Each scenario maps to a specific sequence of tool calls, with arguments that may depend on earlier outputs through placeholders such as `"$RESULT_0.flights[0].flight_id"` [2604.04847].

| Domain | Mock APIs |
|---|---|
| Travel / Identity | `search_flights(destination, date)`, `book_ticket(passenger_name, flight_id)`, `update_travel_profile(document_type, document_number)` |
| Finance / Billing | `query_card_benefits(card_last_4, category)`, `calculate_currency_exchange(amount, from_currency, to_currency)`, `modify_autopay_source(new_account_id)` |
| Housing / Location | `search_apartments(max_budget, amenities)`, `update_search_filter(condition, new_value)` |
| E-Commerce Support | `check_order_status(order_id)`, `cancel_pending_action(action_type)`, `process_exchange(order_id, new_shipping_address)` |

Scenarios are grouped into three difficulty tiers: **Easy** for single-step tool use, **Medium** for two-step chains with moderate ambiguity, and **Hard** for multi-step chains with conflicting constraints, self-corrections, or complex parameter dependencies [2604.04847]. This tiering is not decorative; it is reflected in systematic performance degradation across all evaluated systems.

## 3. Full-duplex interaction model and evaluation protocol

In FDB-v3, full-duplex means that the agent listens to the user continuously while simultaneously producing speech and calling tools. This is distinct from half-duplex pipelines that wait for user completion before beginning recognition, reasoning, and synthesis [2604.04847]. The benchmark therefore evaluates not only whether a system eventually answers correctly, but also when it starts speaking, when it invokes tools, and whether it interrupts.

All systems are evaluated in the same streaming environment: audio is streamed through LiveKit Realtime Voice Agent, tool calls are executed against the same local mock APIs, and every model receives the same human recordings as input [2604.04847]. Semantic scoring is partly automated through GPT-4o-based judges, including an **Argument Accuracy Judge**, a **Response Quality Judge**, and a **Key Information Identifier** for latency segmentation [2604.04847].

The tool-use metrics are structured hierarchically. **Tool Selection F1** compares the set of expected tool calls with the set of actual tool calls. **Argument Accuracy** measures semantic correctness of tool arguments, while allowing flexible formatting, dynamic references beginning with `"$"`, minor normalization differences, and numeric tolerance of \(\pm 5\%\) [2604.04847]. **Pass@1** is a strict binary task-completion metric: a scenario counts as correct only if the system invokes exactly the expected set of tools, with no missing or extra calls, and every call achieves perfect Argument Accuracy [2604.04847]. **Response Quality** evaluates whether the spoken response fulfills the requested task, with partial completion of multi-step tasks scored as 0 [2604.04847].

The interaction metrics formalize timing behavior. **Base latency** is defined as
\[
\Delta t = t_{\text{agent\_start}} - t_{\text{user\_end}}.
\]
If \(\Delta t < 0\), the system starts speaking before the user has finished, and this is counted as an interruption [2604.04847]. The benchmark further decomposes latency into **First Response Latency**, **Tool Call Latency**, and **Task Completion Latency**, where the last quantity measures when the agent begins the sentence containing the key information the user cares about [2604.04847]. A related distinction is operationally important: pre-emptive tool calls with negative latency are not necessarily interruptions, because a system may call tools silently before speaking [2604.04847]. This separates silent pre-processing from eager speaking.

**Turn-take rate** measures the proportion of scenarios in which the agent produces a natural-timing response rather than remaining silent. **Interruption rate** measures the fraction of scenarios with \(\Delta t < 0\). **Filler rate** measures how often the agent emits a content-free filler sentence such as “Sure, let me look that up” before delivering the key information [2604.04847].

## 4. Evaluated systems and aggregate empirical profile

FDB-v3 evaluates six model configurations: GPT-Realtime, Gemini Live 2.5, Gemini Live 3.1, Grok, Ultravox v0.7, and a cascaded baseline built from Whisper \(\rightarrow\) GPT-4o \(\rightarrow\) TTS [2604.04847]. The first five are treated as end-to-end or near-end-to-end speech systems with streaming tool use, while the cascaded system represents the classic ASR-LLM-TTS decomposition.

GPT-Realtime is the strongest overall system in task accuracy. It achieves Pass@1 \(= 0.600\), Tool Selection F1 \(= 0.876\), Argument Accuracy \(= 0.680\), and Response Quality \(= 0.792\), all the best reported overall values [2604.04847]. It also has the lowest interruption rate, 13.5%, making it the strongest observed balance between tool-use accuracy and interaction smoothness [2604.04847].

Gemini Live 3.1 defines a different operating point. It achieves the fastest Task Completion Latency, \(4.25\) s, and a First-word Latency of \(3.95\) s, but it also records the lowest turn-take rate, \(78.0\%\), meaning that 22 of 100 scenarios have no agent speech [2604.04847]. The benchmark characterizes this as a “silent worker” failure mode: a system may call tools or process input, yet fail to produce an audible response.

The cascaded baseline provides the clearest contrast with end-to-end systems. It attains a perfect turn-take rate of \(100.0\%\), so it always responds, but it incurs the highest Task Completion Latency, \(10.12\) s, and the slowest First-word Latency, \(8.78\) s [2604.04847]. This latency profile is attributed to the serialized ASR \(\rightarrow\) LLM \(\rightarrow\) TTS structure rather than to turn-taking indecision.

Ultravox v0.7 exhibits a different compromise. It is relatively fast to first word, at \(3.88\) s, but it is also the most interruption-prone system, with a \(47.9\%\) interruption rate, and the most filler-heavy, with an \(88.0\%\) filler rate [2604.04847]. Grok occupies an intermediate position, with high pre-emptive tool-call behavior and a Task Completion Latency of \(6.65\) s [2604.04847].

These aggregate numbers instantiate the benchmark’s central empirical claim: accuracy, latency, and turn-taking do not collapse into a single ranking. FDB-v3 therefore measures a multi-dimensional operating surface rather than a one-dimensional leaderboard [2604.04847].

## 5. Difficulty structure, disfluency sensitivity, and characteristic failures

The difficulty tiers are reflected directly in Pass@1. GPT-Realtime drops from \(0.750\) on Easy scenarios to \(0.588\) on Medium and \(0.433\) on Hard [2604.04847]. The same monotone pattern appears elsewhere: the cascaded baseline declines from \(0.639\) to \(0.441\) to \(0.233\), and Grok from \(0.583\) to \(0.471\) to \(0.200\) [2604.04847]. This indicates that multi-step reasoning under streaming speech remains unstable even when surface recognition is adequate.

Domain-specific results are similarly uneven. Finance is the easiest domain, with GPT-Realtime at \(0.960\), Gemini Live 3.1 at \(0.920\), and the cascaded baseline at \(0.800\) Pass@1 [2604.04847]. Housing is the hardest, with GPT-Realtime at \(0.308\), Grok at \(0.115\), and the cascaded and Ultravox systems both at \(0.192\) [2604.04847]. A plausible implication is that domain difficulty is driven not only by vocabulary, but by the structure of chained calls and the degree of parameter revision required.

The most persistent benchmark failure is self-correction handling. On self-correction scenarios, Pass@1 is \(0.588\) for GPT-Realtime, \(0.471\) for Gemini Live 2.5, \(0.353\) for Gemini Live 3.1, \(0.294\) for Grok, \(0.353\) for Ultravox, and \(0.176\) for the cascaded baseline [2604.04847]. Across systems, the common pattern is premature commitment: a model locks onto initial arguments, invokes a tool, and then fails to revise its state when the user supplies a correction.

The case study `travel_19` makes this behavior explicit. The user first says Rome and June 1, then corrects both to Milan and June 3. GPT-Realtime achieves Tool Selection \(= 1.00\), Argument Accuracy \(= 1.00\), and Response Quality \(= 1.00\), with tool-call latency \(2.55\) s and completion at \(4.40\) s [2604.04847]. Gemini Live 3.1 also selects the correct tool set, but its Argument Accuracy is \(0.00\), and its tool call begins at \(-2.27\) s, before the user has finished speaking, so it commits to the obsolete Rome/June 1 parameters [2604.04847]. The cascaded baseline records Tool Selection \(= 0.67\) and Argument Accuracy \(= 0.00\), consistent with a segmentation error in which the correction is not integrated into the final plan [2604.04847]. Ultravox recovers the corrected parameters but begins speaking at \(-1.36\) s, creating an interruption [2604.04847].

A complementary case is `finance_18`, a hard three-tool scenario without disfluency. GPT-Realtime, Gemini Live 2.5, Gemini Live 3.1, and Grok all achieve perfect tool selection, argument accuracy, and response quality, while Gemini Live 3.1 completes in \(3.92\) s versus GPT-Realtime’s \(9.20\) s [2604.04847]. By contrast, Ultravox passes an incorrect currency pair and the cascaded system misroutes the conversion while also producing a first response at \(-2.40\) s [2604.04847]. This contrast shows that not all hard cases are disfluency-driven; some remain failures of chained reasoning and parameter binding.

The disfluency-conditioned results further sharpen the picture. Strong models are relatively robust to fillers and hesitations, while pauses are notably harder for Grok and Ultravox, each at \(0.333\) Pass@1 on Pause scenarios [2604.04847]. False starts produce mixed behavior, but self-corrections remain the most difficult category overall. This suggests that surface disfluency is less consequential than disfluency that changes semantic commitments.

## 6. Methodological significance, limitations, and benchmark lineage

FDB-v3 demonstrates that end-to-end speech models are not uniformly superior along every axis. GPT-Realtime is the strongest accuracy–interaction compromise; Gemini Live 3.1 is the extreme speed point but sacrifices reliability in spoken response; Ultravox uses fillers aggressively and interrupts often; and the cascaded baseline is reliable in the narrow sense of always replying, but slow and brittle on self-correction [2604.04847]. The benchmark therefore makes visible a three-way trade-off among accuracy, speed, and turn-taking behavior.

Methodologically, several design choices are central. The use of real human audio rather than synthetic TTS makes disfluency, accent variation, microphone quality, and ambient noise part of the test distribution. The use of deterministic local APIs removes backend variance. The use of GPT-4o judges enables semantic scoring for arguments, responses, and latency segmentation when rigid string matching would be inadequate [2604.04847]. This combination gives the benchmark both realism and reproducibility, though it does not eliminate all sources of evaluation ambiguity.

The limitations are explicit. FDB-v3 contains 100 scenarios and 12 speakers, and it is English-only [2604.04847]. Its APIs are mock services and therefore do not capture real-world failures such as timeouts, malformed responses, or backend nondeterminism [2604.04847]. Even though all cloud systems are invoked from a single server region with high-bandwidth connections, proprietary APIs still reflect unobservable latency and load variability [2604.04847]. Evaluation also depends on GPT-4o as an automatic judge, which introduces model-based judgment biases [2604.04847].

Within the benchmark lineage, FDB-v3 extends the scope of the earlier Full-Duplex-Bench. That earlier benchmark systematized pause handling, backchanneling, smooth turn taking, and user interruption management through time-aligned transcripts and automatic metrics [2503.04721]. FDB-v3 preserves the concern with real-time, overlapping interaction, but relocates it into a more demanding task setting: natural disfluency, tool orchestration, and multi-step action under streaming speech [2604.04847]. This suggests a shift in what “full-duplex” evaluation means for spoken agents: from measuring conversational timing in isolation to measuring timing jointly with online reasoning and external action execution.

Future directions are correspondingly clear. The benchmark points toward more domains and tools, more languages and accents, explicit network-level realism such as API timeouts, and better model-side support for provisional state, delayed commitment, and rollback under self-correction [2604.04847]. In that sense, FDB-v3 does not present full-duplex voice agents as a solved problem. It formalizes the fact that the central challenge is not simply hearing or speaking quickly, but deciding when to listen, when to act, and when to revise an action already underway.

Source: https://www.emergentmind.com/topics/full-duplex-bench-v3-fdb-v3