DuplexSLA-Bench for Real-time Spoken Agents

• DuplexSLA-Bench is a specialized benchmark that measures real-time synchronization of speech, language, and action using a shared 160 ms timeline.
• It comprises 2,100 streaming cases split into turn-taking and tool-call subsets, evaluating semantic turn control and precise timing of in-conversation actions.
• Empirical results show competitive accuracy with sub-second delays compared to ASR+LLM cascades, while highlighting challenges in multi-action tool calling.

DuplexSLA-Bench is a purpose-built benchmark for evaluating a narrow but consequential capability gap in spoken-agent research: whether a system can synchronize listening, speaking, and action emission on a shared real-time timeline rather than treating tool use as a turn-final, text-side add-on. Introduced alongside the DuplexSLA model, it is a 2,100-case streaming benchmark for real-time spoken agents that jointly evaluates semantic turn-taking and in-conversation tool use on a shared $160$ ms timeline, using timing-aware case-level accuracy and delay measures rather than transcript-only or final-turn correctness alone (Zhang et al., 20 May 2026).

1. Design target and benchmark rationale

DuplexSLA-Bench is explicitly designed to test the conjunction of two capabilities: semantic-driven turn-taking control and in-conversation tool calling. The benchmark exists because prior duplex benchmarks mainly evaluate pause, interruption, and backchannel behavior, whereas common tool-use benchmarks are typically turn-based, text-oriented, or indifferent to when an action is emitted relative to ongoing speech. DuplexSLA-Bench therefore targets a more specific question than generic spoken-dialogue evaluation: whether a system can realize speech and action events at the right time during continuous duplex interaction (Zhang et al., 20 May 2026).

This design is tightly coupled to the model family it accompanies. DuplexSLA itself is formulated as a “dual-stream three-channel” system with a continuous user audio channel, a discrete assistant audio channel, and a rate-limited textual action channel, all aligned to a shared $160$ ms chunk clock. DuplexSLA-Bench mirrors that temporal formalism. In the model, each chunk contains two causal user-audio features at 80 ms stride, one assistant TA4 unit consisting of one text anchor plus four 40 ms audio tokens, and up to 10 action-text tokens. The action channel may carry turn-taking labels such as response / interrupt / backchannel, planning text, and tool-call JSON. The benchmark is therefore not merely checking whether a tool call eventually occurs; it tests whether synchronized speaking, listening, and action emission are achieved on one clock (Zhang et al., 20 May 2026).

A recurring misconception is to treat DuplexSLA-Bench as a generic assistant benchmark. The benchmark is narrower than that. Its central concern is not broad product-style assistant quality, but timing-sensitive interaction under streaming constraints. A plausible implication is that it is best understood as an evaluation counterpart to a synchronized speech-language-action architecture rather than as a universal benchmark for all spoken-assistant properties.

2. Benchmark composition and task taxonomy

DuplexSLA-Bench contains 2,100 cases split into two major subsets: a turn-taking subset with 1,200 cases and a tool-call subset with 900 cases. The composition is explicit in the appendix table on benchmark structure (Zhang et al., 20 May 2026).

Subset	Scenario or style	Cases
Turn-taking	normal end-of-turn response	300
Turn-taking	pause	300
Turn-taking	interrupt	300
Turn-taking	backchannel	300
Tool-call	single-action requests	300
Tool-call	multi-action requests	300
Tool-call	backchannel-action requests	300

The turn-taking subset covers four scenarios: normal, pause, interrupt, and backchannel. The tool-call subset covers three styles of in-conversation action use. Single-action cases require one function call. Multi-action cases require multiple ordered function calls from a single user turn. Backchannel-action cases require a short, topically unrelated user utterance to trigger a tool call without interrupting the assistant’s ongoing speech. The benchmark scope therefore directly mirrors the paper’s two flagship claims: native semantic turn-taking and online action emission (Zhang et al., 20 May 2026).

The benchmark’s tool space is also explicitly bounded. It covers 50 function schemas spanning cabin and hardware control, system settings and apps, navigation, media playback, and search/query functions. The appendix lists the full set and includes examples such as open_car_setting, set_car_setting, navigate, play_media, search_music, query_weather, and make_call. This places DuplexSLA-Bench closer to an action-conditioned spoken-agent benchmark than to a pure dialogue-quality corpus (Zhang et al., 20 May 2026).

Each test case is a duplex audio session with semantic anchor times annotated. For tool-calling cases, the session contains a context plus a user request that requires one or more functions and includes ground-truth action timing offsets. Some cases include dialogue history $\mathcal{H}$, and the evaluation supports both a context-prefill mode and a no-prefill mode. The benchmark is therefore structured as continuous streaming sessions rather than isolated single-turn prompts (Zhang et al., 20 May 2026).

3. Temporal representation and online protocol

The evaluation protocol is online and chunk-synchronous. A turn-taking test case is formalized as

$(\mathbf{u}, \mathcal{H}, s, \mathcal{A}),$

with user audio $\mathbf{u}$, dialogue history $\mathcal{H}$, scenario $s \in \{normal, pause, interrupt, backchannel\}$, and anchor set $\mathcal{A}$. A prefill flag $p \in \{0,1\}$ determines whether history is loaded before streaming begins. Initialization is

$\mathcal{S} \leftarrow reset.$

If prefilling is enabled,

$160$0

The user audio is then split into $160$1 chunks of $160$2. For $160$3, the system consumes $160$4 and produces $160$5, with chunk timestamp $160$6 (Zhang et al., 20 May 2026).

The evaluator appends timestamped events to an event log $160$7. When assistant speech is emitted, it appends $160$8; otherwise it appends $160$9. For each action label $\mathcal{H}$0, it appends $\mathcal{H}$1. In DuplexSLA itself, the action stream is directly readable as chunk-aligned labels or tool calls. For generic systems, assistant speech activity is post-processed with an external VAD to recover speak/stop transitions. The representation is thus explicitly tri-channel and time-aligned at $\mathcal{H}$2 ms resolution (Zhang et al., 20 May 2026).

Semantic anchor times are part of the benchmark definition. For turn-taking, examples include $\mathcal{H}$3 for user end, $\mathcal{H}$4 for semantic interruption, and $\mathcal{H}$5 for backchannel start and end. Scenario-specific legal windows are given in Table 5. Normal and pause cases score assistant speech onset relative to $\mathcal{H}$6; interrupt cases score assistant stop time within $\mathcal{H}$7; backchannel cases score events within $\mathcal{H}$8. The paper notes that the table as printed has brace-formatting issues, but the intended semantics in the prose are timing relative to explicit anchors with asymmetric tolerances (Zhang et al., 20 May 2026).

This protocol distinguishes DuplexSLA-Bench from ordinary utterance-level spoken-dialogue evaluation. The system is stepped chunk by chunk, events are timestamped on a common clock, and correctness is coupled to temporal legality. A plausible implication is that a model can fail even with semantically correct outputs if those outputs are emitted at the wrong point in the dialogue stream.

4. Metrics and scoring rules

The benchmark’s metric inventory is compact but formal. For turn-taking, let $\mathcal{H}$9 be the scenario-specific legal window and $(\mathbf{u}, \mathcal{H}, s, \mathcal{A}),$0 the scenario anchor time. Let $(\mathbf{u}, \mathcal{H}, s, \mathcal{A}),$1 for normal and pause, $(\mathbf{u}, \mathcal{H}, s, \mathcal{A}),$2 for interrupt, and $(\mathbf{u}, \mathcal{H}, s, \mathcal{A}),$3 for backchannel. The first relevant event is

$(\mathbf{u}, \mathcal{H}, s, \mathcal{A}),$4

Turn-taking accuracy is

$(\mathbf{u}, \mathcal{H}, s, \mathcal{A}),$5

and delay is

$(\mathbf{u}, \mathcal{H}, s, \mathcal{A}),$6

whenever $(\mathbf{u}, \mathcal{H}, s, \mathcal{A}),$7 (Zhang et al., 20 May 2026).

Backchannel scoring includes an explicit asymmetry. Because most baselines do not expose a backchannel label, the benchmark relaxes backchannel accuracy to any $(\mathbf{u}, \mathcal{H}, s, \mathcal{A}),$8 event inside $(\mathbf{u}, \mathcal{H}, s, \mathcal{A}),$9, and reports backchannel delay only when an explicit label is present. This explains why baseline backchannel delay is shown as N/A in the reported comparisons. The point is methodological rather than incidental: interface observability constrains comparability (Zhang et al., 20 May 2026).

Tool-call scoring is case-level rather than event-hit based. A predicted tool call is correct only if all three conditions hold. First, every ground-truth action has a predicted action with the same function name. Second, arguments either match exactly, are both empty, or are judged by an LLM to be semantically consistent with no “core information conflict.” Third, the trigger time is legal: it must not be earlier than the ground-truth offset by more than $\mathbf{u}$0 second and not later than the end of the audio by more than $\mathbf{u}$1 seconds. Tool-call accuracy is the fraction of cases in which all ground-truth actions are matched, and tool-call delay is the average temporal gap on matched actions (Zhang et al., 20 May 2026).

The benchmark therefore reports four primary quantities: turn-taking accuracy, turn-taking delay, tool-call accuracy, and tool-call delay. It does not define precision, recall, or $\mathbf{u}$2 for tool calls, and it does not introduce a composite score over timing plus semantic correctness. No train/dev/test split is specified in the paper, and no benchmark licensing terms, annotation workforce details, or inter-annotator agreement are provided in the supplied description. What is explicit is benchmark size, scenario counts, timing formalism, and the public release of the evaluation suite (Zhang et al., 20 May 2026).

5. Empirical results and characteristic failure modes

On the 900-case tool-call subset, the principal comparison is between DuplexSLA and an ASR + LLM cascade. The cascade achieves 91.33% average accuracy with 2.77 s average delay, whereas DuplexSLA achieves 85.56% average accuracy with 0.64 s average delay. By subtype, DuplexSLA records 85.67% / 0.67 s on single-action, 75.00% / 0.68 s on multi-action, and 96.00% / 0.57 s on backchannel-action; the cascade records 89.33% / 2.33 s, 89.33% / 4.71 s, and 95.33% / 1.27 s, respectively. The reported interpretation is that DuplexSLA is slightly worse in aggregate tool accuracy but around $\mathbf{u}$3 faster in delay on average, with the strongest relative advantage in backchannel-triggered action use (Zhang et al., 20 May 2026).

On the context-prefill turn-taking setting, DuplexSLA reports 96.00% accuracy and 0.27 s delay for normal, 93.33% and 0.27 s for pause, 99.33% and 0.40 s for interrupt, and 98.33% and 0.32 s for backchannel. The baselines reported are gemini-3.1-flash-live, gpt-realtime-1.5 (semantic-vad-high), and gpt-realtime-1.5 (server-vad-40ms). Numerically, DuplexSLA is the only system in that comparison that the paper states “handles backchannel correctly,” and it achieves the lowest delay in every scenario. The backchannel difference is particularly large: 98.33% for DuplexSLA versus 40.00%, 0.33%, and 13.00% for the three baselines under the benchmark’s relaxed backchannel criterion (Zhang et al., 20 May 2026).

In the no-context-prefill setting, only normal and pause are evaluated. DuplexSLA obtains 94.34% average accuracy with 0.30 s delay, from 95.67% / 0.29 s on normal and 93.00% / 0.31 s on pause. Other systems include Freeze-Omni, PersonaPlex, MiniCPM-o, gemini-3.1-flash-live, and the two gpt-realtime-1.5 VAD configurations. The key empirical takeaway reported in the paper is that DuplexSLA is the only system in this comparison that combines sub-second delay with competitive accuracy. Commercial APIs can reach high accuracy but incur delays above 1 second, while open-source duplex backbones without the paper’s targeted post-training perform very poorly, especially on pause cases; the paper explicitly states that these open-source models “collapse on the pause subset” (Zhang et al., 20 May 2026).

The clearest failure mode surfaced by DuplexSLA-Bench is multi-action tool calling. DuplexSLA falls from 85.67% on single-action cases to 75.00% on multi-action cases. The paper does not provide a dedicated qualitative error-analysis section, but it explicitly notes the architecture’s action channel budget of up to 10 text tokens per chunk and that overflow spills into later chunks via a FIFO queue. This suggests, though does not prove, that coordinating multiple ordered actions on a constrained streaming action channel is harder than emitting a single action. Another explicit difficulty is that backchannel delay is unobservable for closed-source systems that do not expose an explicit backchannel label, which limits direct comparability (Zhang et al., 20 May 2026).

6. Position within duplex-benchmarking research

DuplexSLA-Bench occupies a specific position among contemporaneous duplex benchmarks. FD-Bench focuses on interruption handling, response quality, noise robustness, and event-level timing metrics such as interruption-response delay and early-interrupt behavior for full-duplex spoken dialogue systems, but it does not jointly evaluate in-conversation tool timing on a shared action channel (Peng et al., 25 Jul 2025). Full-Duplex-Bench-v2 extends duplex evaluation to multi-turn spoken interaction with an automated examiner, staged goals, correction handling, entity tracking, and safety under Fast and Slow pacing, but its emphasis is longer-horizon spoken task competence under overlap and interruption rather than synchronized speech-language-action emission (Lin et al., 9 Oct 2025). MTR-DuplexBench targets multi-round evaluation of full-duplex speech LLMs, with turn segmentation, turn-by-turn evaluation, dialogue quality, conversational features, instruction following, and safety, yet its framing remains that of multi-round conversational assessment rather than tool-calling on a shared chunk timeline (Zhang et al., 13 Nov 2025).

Against that background, DuplexSLA-Bench differs along three axes that are explicit in the paper. First, it is streaming rather than turn-final: systems are stepped chunk by chunk with timestamps. Second, it is timing-aware rather than transcript-only or semantic-only: the central quantities are legal temporal windows and delay relative to semantic anchors. Third, it integrates action/tool timing with turn-taking: prior duplex benchmarks are mostly speech-only, and prior tool-use benchmarks are usually text-turn-based or indifferent to whether an action is triggered while the assistant is still speaking (Zhang et al., 20 May 2026).

Its limitations are correspondingly specific. The public evaluation suite, demos, and repository are stated to exist, but the paper text does not specify data splits, benchmark license, or detailed repository contents. Benchmark comparability is partly constrained by system interfaces, since backchannel delay is unavailable for closed-source systems lacking an exposed label. More broadly, DuplexSLA-Bench should not be conflated with a complete spoken-agent benchmark: its strength is the evaluation of synchronized speech, language, and action on a shared $\mathbf{u}$4 ms clock. A plausible implication is that it is best used where online timing of action emission is itself the object of study, or as a specialized component within a broader spoken-agent evaluation stack.