SPEARBench: Benchmark for Speech Naturalness

• SPEARBench is an open, fully automated benchmark that evaluates conversational naturalness in streaming speech-to-speech models using metrics such as timing, prosody, and emotional fit.
• It constructs controlled dialogue prompts from the Seamless Interaction corpus to ensure consistent evaluation against human conversational behavior.
• Results indicate that while models perform well in signal quality and intelligibility, they lag in latency, turn-taking, and emotional adaptation compared to human references.

SPEARBench is an open, fully automated benchmark for evaluating naturalness in end-to-end streaming speech-to-speech LLMs from question-answer interactions. It was introduced to measure aspects of conversational behavior that standard speech and text benchmarks do not capture, including timing, turn-taking, prosody, interpersonal stance, emotional fit, dialect consistency, and relationship-aware appropriateness. The benchmark constructs controlled dialogue prompts from the Seamless Interaction corpus, evaluates generated spoken answers with a multidimensional protocol, includes original human answers as a reference condition, and reports results for several contemporary models (Thebaud et al., 6 Jul 2026).

1. Definition, scope, and naming

In the 2026 speech-to-speech literature, SPEARBench denotes a benchmark for naturalness evaluation in streaming speech-to-speech LLMs. Its stated objective is to provide an open, fully automated protocol for evaluating end-to-end streaming speech-to-speech models on a unified set of naturalness metrics drawn from conversational turn-taking, prosody, emotion, stance, and dialect. It also establishes distributional, explainable baselines, including original human answers, so that model progress can be tracked across key facets of human-like spoken interaction.

The benchmark is motivated by a specific measurement gap. Standard speech-and-text benchmarks, including ASR WER, UTMOS, and task-correctness, do not capture core elements of real conversation such as timing, latency, interruptions, prosodic expressiveness, interpersonal stance, emotional fit, dialect consistency, or relationship-aware appropriateness. Existing spoken-dialogue evaluations are described as typically focusing on one or two dimensions, such as latency, interruptions, or intelligibility, or as requiring costly human annotation.

The name has also been used in a distinct line of work on sparse tensor computing. In that usage, SPEARBench is a community-driven benchmark suite for arbitrary-order sparse tensor kernels on CPUs and GPUs, supporting COO and HiCOO formats, multiple tensor kernels, real-world and synthetic tensors, and Roofline performance models (Li et al., 2020). The two usages concern different domains: conversational speech evaluation in one case, and sparse tensor performance characterization in the other.

2. Corpus grounding and benchmark construction

SPEARBench is built from the dev and test sets of the Seamless Interaction dataset, described as totaling approximately 113.8 hours of dyadic English conversation spanning improvised and naturalistic speech. From this source, the benchmark extracts valid dialogues defined as any sequence of at least three consecutive turns between exactly two speakers where turn $K-1$ ends in a question mark. Under this construction, the context consists of turns $1 \ldots K-2$, the question is turn $K-1$, and the answer is turn $K$, which is the original human answer (Thebaud et al., 6 Jul 2026).

The extraction process yielded 5,419 dialogues, amounting to approximately 37.3 hours in total, and these were balanced across improvised and naturalistic subsets. Several controlled variables are fixed for consistency across model comparisons. Audio contexts and questions are identical for every model, the setting remains single-language and two-speaker throughout, train/dev/test splits are inherited uniformly from Seamless Interaction, and the same VAD, ASR evaluators, emotion, stance, and dialect models are applied to all outputs.

This construction makes the evaluation deliberately controlled rather than open-ended. A plausible implication is that observed differences are intended to reflect model behavior under matched prompting and matched automatic evaluators, rather than prompt variation or assessor variation.

3. Multidimensional evaluation protocol

SPEARBench computes nine complementary dimensions, all automatically via pretrained evaluators. The protocol spans signal-level quality, timing, interactional dynamics, sociolinguistic consistency, affective fit, stance, and explainable acoustic-distributional features (Thebaud et al., 6 Jul 2026).

Dimension group	Operationalization	Evaluator or statistic
Intelligibility and speech quality	WER, CER, predicted MOS	whisper-large-v3, qwen3-ASR-0.6B, UTMOS
Interruptions and latency	Latency, interruption count, total overlap duration	Silero VAD
Turn-taking naturalness	Surprisals of voice-activity patterns	TurnNat (DualTurn)
Language and dialect consistency	Non-English rate, dialect profile, entrainment, variance	MMS, Voxlect
Emotional naturalness and entrainment	Emotional naturalness score, Pearson $\rho$ on AVD	TRACE, VoxProfile, Sentence-BERT
Interpersonal stance	Ten stance dimensions with same/more positive/more negative comparisons	GPT-audio judge
Temporal and spectral features	Duration, voiced ratio, normalized pitch deciles	Distributional baselines

For intelligibility and speech quality, WER and CER are computed using two ASR systems, whisper-large-v3 and qwen3-ASR-0.6B, against each model’s own transcript. UTMOS provides a predicted MOS on a 1–5 scale, with higher values indicating better quality.

For interruptions and latency, latency is defined as the time between the end of the question and the start of the answer, measured with Silero VAD. Negative latencies count as interruptions. The benchmark also reports interruption count and total overlap duration.

Turn-taking naturalness is measured with the TurnNat (DualTurn) evaluator, defined as surprisals of voice-activity patterns, with higher values interpreted as more human-like turn dynamics.

Language and dialect consistency are separated into categorical language identity and distributional dialect behavior. MMS language identification is used to report the percentage of non-English answers, with non-English output treated as a failure in this English-only setup. Voxlect produces an 11-dimensional logit vector over dialect groups. From these profiles, dialectal entrainment $\beta$ is defined as the least-squares regression coefficient relating question and answer profiles, and dialectal variance is defined as the trace of the covariance of profiles, with higher values implying more model drift across dialogues.

Emotional naturalness and entrainment are evaluated with TRACE. Its inputs are question and answer speech-emotion embeddings from VoxProfile, arousal/valence/dominance values, context embeddings from Sentence-BERT, and relationship one-hots; its output is an emotional naturalness score. Emotional entrainment is summarized by Pearson correlation between question and answer arousal, valence, and dominance values.

Interpersonal stance is represented by ten dimensions: politeness, empathy, warmth, calmness, deference, sociability, organization, focus, honesty, and disinhibition. These are scored by a GPT-audio judge, and for each dimension the benchmark reports the percentage of cases in which the model’s stance is the same as, more positive than, or more negative than, the original human answer.

Temporal and spectral features serve as explainable baselines. These include answer duration, voiced ratio defined as voiced time divided by total time, and normalized pitch decile statistics computed by per-speaker normalization followed by per-decile means and standard deviations.

4. Reference condition and model set

A defining feature of SPEARBench is the use of original human answers as a reference condition. Every metric is first computed on these original human answers to serve as a gold conversational behavior baseline. Reported examples include human latency of approximately 742 ms, human turn-taking naturalness of 4.99, and human UTMOS of 2.22 (Thebaud et al., 6 Jul 2026).

The benchmark reports results for several contemporary speech-to-speech systems spanning non-streaming, full-duplex streaming, and half-duplex streaming settings.

Model	Descriptor	Interaction mode
GPT-audio-1.5	OpenAI, non-streaming	Non-streaming
GPT-realtime-2	OpenAI	Full-duplex streaming
Gemini-2.5-flash-native-audio	Google	Full-duplex
Gemini-3.1-flash-live	Google	Full-duplex
Qwen3-Omni-30B-Instruct	30B, open	Full-duplex
Qwen2.5-Omni-7B	7B, open	Half-duplex
Mini-Omni	0.5B, open	Half-duplex

This design places model outputs and human answers inside the same measurement framework. A plausible implication is that the benchmark is intended less as a pure leaderboard of task success than as a comparative profile of conversational behavior against human reference distributions.

5. Reported results and empirical regularities

The reported results show a clear separation between signal-level quality and conversational naturalness. On intelligibility and quality, the evaluated models achieve much lower WER and CER than human answers, with WER ranging from 5.8% to 26.7% versus human 27.1%, and they obtain higher UTMOS values, ranging from 3.7 to 4.4 versus human 2.2 (Thebaud et al., 6 Jul 2026). A common misconception in this area is that low ASR error and high MOS are sufficient proxies for natural conversation; the benchmark is explicitly designed to show that they are not.

On latency and interruptions, all models respond more slowly than humans, with latencies from 959 ms to 2724 ms compared with 742 ms for the human reference. Interruptions are reported as rare except for GPT-realtime-2, with interruptions in 23.6% of dialogues and 431 ms overlap, and Mini-Omni, with interruptions in 58.1% of dialogues and 1243 ms overlap.

Turn-taking naturalness shows a different pattern. Full-duplex models, specifically Gemini-2.5, Gemini-3.1, GPT-realtime, and Qwen3, match or slightly exceed the human TurnNat score, at approximately 5.3 versus 4.99. Mini-Omni scores much lower, at 3.10, consistent with its high interruption rates.

Language identification remains nearly perfect in the benchmark’s English setting, with nearly 100% English output across all models. Dialectal behavior is less human-like. Dialectal entrainment is reported as far below human, at 0.28–0.54 versus 0.53, and dialectal variance is lower in smaller and half-duplex models; Qwen2.5, for example, is reported at 84.5 versus human 250.6, indicating constrained dialect expression.

Emotional naturalness and emotional entrainment also remain below the human reference. All models score significantly below human emotional naturalness, at 10.2–10.9 versus 11.10, with a Friedman test reported at $p < 1e\!-\!13$. Valence correlations are partly preserved, reaching $\rho$ up to 0.29 versus human 0.36, but arousal and dominance entrainment are near zero for most models, compared with human correlations of 0.53 and 0.51.

Interpersonal stance is comparatively stable. Models preserve human answer stances in more than 95% of utterances, although Mini-Omni shows noticeably lower focus. Temporal and spectral baselines indicate that model answers are on average longer, reported as 6.6–19.3 s versus human 8.8 s, and have similar or higher voiced ratios, reported as 0.384–0.592 versus 0.50. Pitch variation is substantially lower in all models, with standard deviations from 0.118 to 0.299 versus 0.321 for humans, indicating flattened prosody.

Taken together, these results support a specific interpretation already built into the benchmark’s design: current models can achieve high signal-level quality and low ASR error while still differing from human conversational behavior in latency, overlap, dialect preservation, emotional adaptation, and interpersonal stance dynamics.

6. Strengths, limitations, and projected extensions

The benchmark highlights several strengths of current systems. These include high signal-level quality, modern full-duplex models that can emulate human-like turn-taking surprisal patterns, and strong interpersonal stance preservation (Thebaud et al., 6 Jul 2026). At the same time, it identifies recurring weaknesses: slower response times, occasional interruptions in streaming setups, limited dialectal adaptation and entrainment, reduced pitch dynamics and emotional intensity entrainment, and overlong or uneven answer durations.

The benchmark’s own future directions are tightly coupled to these observations. Proposed extensions include integrating turn-timing losses or utilities into speech-to-speech training to reduce latency and unwanted overlap, improving prosody modeling and emotional intensity control in neural TTS decoders, adding dialect adaptation or style-transfer modules to preserve speaker regional traits, expanding the benchmark to multilingual, multi-party, and backchannel-rich scenarios, and corroborating automatic metrics with human perceptual studies to calibrate scores.

The benchmark is also domain-bounded by construction. It uses fixed audio contexts and questions, a single language, and a two-speaker setting, while applying the same automatic evaluators across all outputs. This suggests a deliberate trade-off: stronger control and reproducibility in exchange for narrower conversational coverage. The authors accordingly position the public release of data, code, and a web leaderboard as a shared platform for tracking progress toward more human-like spoken dialogue agents.