FDB-v3: Full-Duplex Spoken Interaction Benchmark
- FDB-v3 is a benchmark for evaluating full-duplex spoken interaction that integrates real human audio, natural disfluencies, and chained API calls.
- It quantifies performance using metrics like task completion, tool call latency, turn-taking, and interruption rates under real-world speech conditions.
- The design emphasizes simultaneous speech and tool execution, highlighting trade-offs between accuracy, speed, and interaction smoothness.
Full-Duplex-Bench-v3 (FDB-v3) is a benchmark for evaluating spoken LLMs under naturalistic speech conditions and multi-step tool use. It is designed for real-time, full-duplex voice agents that must listen continuously, speak with appropriate timing, and execute chained API calls despite fillers, pauses, hesitations, false starts, and mid-utterance self-corrections. Its dataset consists entirely of real human audio, and its evaluation combines task completion, tool-use correctness, latency, and turn-taking behavior in a single framework (Lin et al., 6 Apr 2026).
1. Historical placement and benchmark scope
FDB-v3 belongs to a line of full-duplex spoken interaction benchmarks, but it targets a broader problem setting than earlier turn-taking evaluations. Earlier Full-Duplex-Bench work focused on pause handling, backchanneling, smooth turn taking, and user interruption management, with automatic metrics derived from time-aligned audio and transcripts (Lin et al., 6 Mar 2025). FDB-v3 extends that agenda to a setting in which the agent must not only manage conversational timing, but also perform multi-step external actions under disfluent speech (Lin et al., 6 Apr 2026).
Its defining premise is that three capabilities must be evaluated together: full-duplex interaction, tool use with chained API calls, and robustness to real-world speech disfluency. FDB-v3 positions itself against several earlier benchmark families. Prior Full-Duplex-Bench versions are described as evaluating turn-taking with largely synthetic TTS audio and without multi-step tool use; Audio MultiChallenge and WildSpeech-Bench use real human speech but do not evaluate tool use; and τ-Voice, AudioCRAG, and VoiceAgentBench evaluate tool use with synthetic audio and often single-step tasks (Lin et al., 6 Apr 2026). A plausible implication is that FDB-v3 should be read not merely as a turn-taking benchmark, but as an integrated benchmark for streaming speech interaction, online reasoning, and external action execution.
The benchmark therefore evaluates failure modes that do not appear in text-only or turn-bounded settings. A representative case is a request such as “Book me a flight um… to New York — actually, wait… make that Boston,” where a robust system must avoid prematurely calling tools with the obsolete destination and must revise its internal state when the correction arrives (Lin et al., 6 Apr 2026).
2. Dataset composition, disfluency taxonomy, and task domains
FDB-v3 contains 100 recordings, one per scenario, from 12 speakers that include native and non-native English speakers, accents from Korean and Russian backgrounds, and varying accent strength (Lin et al., 6 Apr 2026). Recording conditions are intentionally ordinary rather than laboratory-grade: 11 of 12 speakers use built-in laptop or phone microphones, and environments range from quiet rooms to mild background noise (Lin et al., 6 Apr 2026). Speakers are asked to perform prompts organically rather than read scripts.
The corpus design also incorporates non-synthetic silence. Each speaker contributes 30 seconds of real ambient noise from the recording environment, and this is used instead of digital silence at the tail of utterances (Lin et al., 6 Apr 2026). Each speaker is assigned 10 scenarios spanning all four task domains, and 21 of the 100 scenarios are explicitly designed to include self-correction events requiring state rollback (Lin et al., 6 Apr 2026).
FDB-v3 annotates every recording for the presence of five disfluency categories. Fillers are canonical items such as “um,” “uh,” “like,” and “you know” that do not change semantics. Pauses are mid-utterance silences that stress end-of-turn detection. Hesitations are filler-repetition combinations such as “I need a fl– uh, a flight to…”. False starts occur when a speaker begins one intent and abandons it for another. Self-corrections alter parameters mid-sentence, as in changing a destination or date after an initial specification (Lin et al., 6 Apr 2026).
The benchmark’s task layer is built around locally executed mock APIs with deterministic, zero-latency responses, fixed signatures, and fixed behavior. This isolates model reasoning and interaction behavior from backend network variance (Lin et al., 6 Apr 2026). Each scenario maps to a specific sequence of tool calls, with arguments that may depend on earlier outputs through placeholders such as "$RESULT_0.flights[0].flight_id" (Lin et al., 6 Apr 2026).
| Domain | Mock APIs |
|---|---|
| Travel / Identity | search_flights(destination, date), book_ticket(passenger_name, flight_id), update_travel_profile(document_type, document_number) |
| Finance / Billing | query_card_benefits(card_last_4, category), calculate_currency_exchange(amount, from_currency, to_currency), modify_autopay_source(new_account_id) |
| Housing / Location | search_apartments(max_budget, amenities), update_search_filter(condition, new_value) |
| E-Commerce Support | check_order_status(order_id), cancel_pending_action(action_type), process_exchange(order_id, new_shipping_address) |
Scenarios are grouped into three difficulty tiers: Easy for single-step tool use, Medium for two-step chains with moderate ambiguity, and Hard for multi-step chains with conflicting constraints, self-corrections, or complex parameter dependencies (Lin et al., 6 Apr 2026). This tiering is not decorative; it is reflected in systematic performance degradation across all evaluated systems.
3. Full-duplex interaction model and evaluation protocol
In FDB-v3, full-duplex means that the agent listens to the user continuously while simultaneously producing speech and calling tools. This is distinct from half-duplex pipelines that wait for user completion before beginning recognition, reasoning, and synthesis (Lin et al., 6 Apr 2026). The benchmark therefore evaluates not only whether a system eventually answers correctly, but also when it starts speaking, when it invokes tools, and whether it interrupts.
All systems are evaluated in the same streaming environment: audio is streamed through LiveKit Realtime Voice Agent, tool calls are executed against the same local mock APIs, and every model receives the same human recordings as input (Lin et al., 6 Apr 2026). Semantic scoring is partly automated through GPT-4o-based judges, including an Argument Accuracy Judge, a Response Quality Judge, and a Key Information Identifier for latency segmentation (Lin et al., 6 Apr 2026).
The tool-use metrics are structured hierarchically. Tool Selection F1 compares the set of expected tool calls with the set of actual tool calls. Argument Accuracy measures semantic correctness of tool arguments, while allowing flexible formatting, dynamic references beginning with "$"</code>, minor normalization differences, and numeric tolerance of $\pm 5\%\Delta t = t_{\text{agent\_start}} - t_{\text{user\_end}}.\Delta t < 0\Delta t < 0\rightarrow\rightarrow= 0.600= 0.876= 0.680= 0.792\Delta t = t_{\text{agent\_start}} - t_{\text{user\_end}}.$0 s, and a First-word Latency of $\Delta t = t_{\text{agent\_start}} - t_{\text{user\_end}}.$1 s, but it also records the lowest turn-take rate, $\Delta t = t_{\text{agent\_start}} - t_{\text{user\_end}}.$2, meaning that 22 of 100 scenarios have no agent speech (Lin et al., 6 Apr 2026). The benchmark characterizes this as a “silent worker” failure mode: a system may call tools or process input, yet fail to produce an audible response.
The cascaded baseline provides the clearest contrast with end-to-end systems. It attains a perfect turn-take rate of $\Delta t = t_{\text{agent\_start}} - t_{\text{user\_end}}.$3, so it always responds, but it incurs the highest Task Completion Latency, $\Delta t = t_{\text{agent\_start}} - t_{\text{user\_end}}.$4 s, and the slowest First-word Latency, $\Delta t = t_{\text{agent\_start}} - t_{\text{user\_end}}.$5 s (Lin et al., 6 Apr 2026). This latency profile is attributed to the serialized ASR $\Delta t = t_{\text{agent\_start}} - t_{\text{user\_end}}.$6 LLM $\Delta t = t_{\text{agent\_start}} - t_{\text{user\_end}}.$7 TTS structure rather than to turn-taking indecision.
Ultravox v0.7 exhibits a different compromise. It is relatively fast to first word, at $\Delta t = t_{\text{agent\_start}} - t_{\text{user\_end}}.$8 s, but it is also the most interruption-prone system, with a $\Delta t = t_{\text{agent\_start}} - t_{\text{user\_end}}.$9 interruption rate, and the most filler-heavy, with an $\Delta t < 0$0 filler rate (Lin et al., 6 Apr 2026). Grok occupies an intermediate position, with high pre-emptive tool-call behavior and a Task Completion Latency of $\Delta t < 0$1 s (Lin et al., 6 Apr 2026).
These aggregate numbers instantiate the benchmark’s central empirical claim: accuracy, latency, and turn-taking do not collapse into a single ranking. FDB-v3 therefore measures a multi-dimensional operating surface rather than a one-dimensional leaderboard (Lin et al., 6 Apr 2026).
5. Difficulty structure, disfluency sensitivity, and characteristic failures
The difficulty tiers are reflected directly in Pass@1. GPT-Realtime drops from $\Delta t < 0$2 on Easy scenarios to $\Delta t < 0$3 on Medium and $\Delta t < 0$4 on Hard (Lin et al., 6 Apr 2026). The same monotone pattern appears elsewhere: the cascaded baseline declines from $\Delta t < 0$5 to $\Delta t < 0$6 to $\Delta t < 0$7, and Grok from $\Delta t < 0$8 to $\Delta t < 0$9 to $\Delta t < 0$0 (Lin et al., 6 Apr 2026). This indicates that multi-step reasoning under streaming speech remains unstable even when surface recognition is adequate.
Domain-specific results are similarly uneven. Finance is the easiest domain, with GPT-Realtime at $\Delta t < 0$1, Gemini Live 3.1 at $\Delta t < 0$2, and the cascaded baseline at $\Delta t < 0$3 Pass@1 (Lin et al., 6 Apr 2026). Housing is the hardest, with GPT-Realtime at $\Delta t < 0$4, Grok at $\Delta t < 0$5, and the cascaded and Ultravox systems both at $\Delta t < 0$6 (Lin et al., 6 Apr 2026). A plausible implication is that domain difficulty is driven not only by vocabulary, but by the structure of chained calls and the degree of parameter revision required.
The most persistent benchmark failure is self-correction handling. On self-correction scenarios, Pass@1 is $\Delta t < 0$7 for GPT-Realtime, $\Delta t < 0$8 for Gemini Live 2.5, $\Delta t < 0$9 for Gemini Live 3.1, $\rightarrow$0 for Grok, $\rightarrow$1 for Ultravox, and $\rightarrow\rightarrow$3, Argument Accuracy $\rightarrow$4, and Response Quality $\rightarrow$5, with tool-call latency $\rightarrow$6 s and completion at $\rightarrow$7 s (Lin et al., 6 Apr 2026). Gemini Live 3.1 also selects the correct tool set, but its Argument Accuracy is $\rightarrow$8, and its tool call begins at $\rightarrow$9 s, before the user has finished speaking, so it commits to the obsolete Rome/June 1 parameters (Lin et al., 6 Apr 2026). The cascaded baseline records Tool Selection $\rightarrow$0 and Argument Accuracy $\rightarrow$1, consistent with a segmentation error in which the correction is not integrated into the final plan (Lin et al., 6 Apr 2026). Ultravox recovers the corrected parameters but begins speaking at $\rightarrow\rightarrow$3 s versus GPT-Realtime’s $\rightarrow$4 s (Lin et al., 6 Apr 2026). By contrast, Ultravox passes an incorrect currency pair and the cascaded system misroutes the conversion while also producing a first response at $\rightarrow$5 s (Lin et al., 6 Apr 2026). This contrast shows that not all hard cases are disfluency-driven; some remain failures of chained reasoning and parameter binding.
The disfluency-conditioned results further sharpen the picture. Strong models are relatively robust to fillers and hesitations, while pauses are notably harder for Grok and Ultravox, each at $\rightarrow$6 Pass@1 on Pause scenarios (Lin et al., 6 Apr 2026). False starts produce mixed behavior, but self-corrections remain the most difficult category overall. This suggests that surface disfluency is less consequential than disfluency that changes semantic commitments.
6. Methodological significance, limitations, and benchmark lineage
FDB-v3 demonstrates that end-to-end speech models are not uniformly superior along every axis. GPT-Realtime is the strongest accuracy–interaction compromise; Gemini Live 3.1 is the extreme speed point but sacrifices reliability in spoken response; Ultravox uses fillers aggressively and interrupts often; and the cascaded baseline is reliable in the narrow sense of always replying, but slow and brittle on self-correction (Lin et al., 6 Apr 2026). The benchmark therefore makes visible a three-way trade-off among accuracy, speed, and turn-taking behavior.
Methodologically, several design choices are central. The use of real human audio rather than synthetic TTS makes disfluency, accent variation, microphone quality, and ambient noise part of the test distribution. The use of deterministic local APIs removes backend variance. The use of GPT-4o judges enables semantic scoring for arguments, responses, and latency segmentation when rigid string matching would be inadequate (Lin et al., 6 Apr 2026). This combination gives the benchmark both realism and reproducibility, though it does not eliminate all sources of evaluation ambiguity.
The limitations are explicit. FDB-v3 contains 100 scenarios and 12 speakers, and it is English-only (Lin et al., 6 Apr 2026). Its APIs are mock services and therefore do not capture real-world failures such as timeouts, malformed responses, or backend nondeterminism (Lin et al., 6 Apr 2026). Even though all cloud systems are invoked from a single server region with high-bandwidth connections, proprietary APIs still reflect unobservable latency and load variability (Lin et al., 6 Apr 2026). Evaluation also depends on GPT-4o as an automatic judge, which introduces model-based judgment biases (Lin et al., 6 Apr 2026).
Within the benchmark lineage, FDB-v3 extends the scope of the earlier Full-Duplex-Bench. That earlier benchmark systematized pause handling, backchanneling, smooth turn taking, and user interruption management through time-aligned transcripts and automatic metrics (Lin et al., 6 Mar 2025). FDB-v3 preserves the concern with real-time, overlapping interaction, but relocates it into a more demanding task setting: natural disfluency, tool orchestration, and multi-step action under streaming speech (Lin et al., 6 Apr 2026). This suggests a shift in what “full-duplex” evaluation means for spoken agents: from measuring conversational timing in isolation to measuring timing jointly with online reasoning and external action execution.
Future directions are correspondingly clear. The benchmark points toward more domains and tools, more languages and accents, explicit network-level realism such as API timeouts, and better model-side support for provisional state, delayed commitment, and rollback under self-correction (Lin et al., 6 Apr 2026). In that sense, FDB-v3 does not present full-duplex voice agents as a solved problem. It formalizes the fact that the central challenge is not simply hearing or speaking quickly, but deciding when to listen, when to act, and when to revise an action already underway.