SPEARBench: A Benchmark for Naturalness Evaluation in Streaming Speech-to-Speech Language Models

Abstract: Streaming speech-to-speech LLMs aim to answer spoken queries directly with synthetic speech. However, standard speech and text benchmarks do not capture whether these systems behave naturally in conversations, where timing, turn-taking, prosody, interpersonal stance, language and dialect consistency, and relationship-aware appropriateness jointly shape perceived quality. We introduce SPEARBench, a benchmark for evaluating naturalness in speech-to-speech LLMs from question-answer interactions. SPEARBench constructs controlled dialogue prompts from the Seamless Interaction corpus, runs inference across multiple models, and evaluates generated answers using a multidimensional protocol that covers response latency, interruptions, speech quality, ASR robustness, language and dialect consistency, emotional naturalness, interpersonal stance, and explainable distributional baselines. The benchmark includes original human answers as a reference condition and reports results for several contemporary models. Results show that current models can achieve high signal-level quality and low ASR error while still differing from human conversational behavior in latency, overlap, dialect preservation, emotional adaptation, and interpersonal stance dynamics.

• The paper introduces SPEARBench, a benchmark that quantifies naturalness in speech-to-speech models using metrics for timing, turn-taking, and expressive prosody.
• It shows that despite high signal quality, current S2S models lag in emotional adaptation and dialect fidelity compared to human dialogues.
• The evaluation uses controlled dyadic interactions and robust automatic metrics to deliver actionable insights for advancing conversational AI.

SPEARBench: A Benchmark for Naturalness Evaluation in Streaming Speech-to-Speech LLMs

Introduction

SPEARBench presents a comprehensive framework for evaluating the naturalness of streaming speech-to-speech LLMs (S2S LMs), addressing an underexplored but pivotal dimension in conversational AI evaluation. Whereas prior benchmarks emphasize intelligibility and task accuracy, SPEARBench explicitly targets interactional attributes critical for natural human-computer exchange: response timing, turn-taking, prosody, interpersonal stance, language and dialect fidelity, and appropriateness contingent on social relationships. The authors utilize controlled, dyadic question-answer dialogues from the Seamless Interaction corpus to quantify these high-level interactional phenomena, combining robustly validated automatic metrics with multi-model system outputs in an extensible platform.

Figure 1: The SPEARBench pipeline covers data extraction, model inference, and multi-faceted evaluation of dialogue naturalness.

Data and Benchmark Structure

SPEARBench’s evaluation protocol consists of three main procedures: (1) extraction of representative two-speaker, question-answer contexts from the Seamless Interaction dataset, (2) model generation of synthetic answers conditioned on real spoken contexts, and (3) application of a unified suite of automatic metrics. Dialogue selection ensures realistic conversational diversity: the dataset spans multiple demographics, spontaneous and scenario-driven exchanges, and is fully transcribed and diarized. For each prompt, both human and model answers are available for detailed comparative analysis.

Evaluation Dimensions and Metrics

SPEARBench integrates a spectrum of metrics tailored to the complexities of spoken dialogue:

• Signal-Level Quality: Intelligibility is assessed via WER and CER using multiple ASR models; UTMOS provides a proxy for human opinion on MOS.
• Temporal Coordination: Latency and interruption metrics characterize the timing of responses and prevalence of overlap.
• Turn-Taking Naturalness: Employing state-of-the-art models, SPEARBench scores the predictive typicality of a model’s response in natural conversation flow.
• Language and Dialect Consistency: MMS provides robust language ID, while Voxlect-English encodes 11-category dialect profiles, enabling analysis of entrainment and stability.
• Emotional Naturalness and Entrainment: TRACE and VoxProfile estimate appropriateness and adaptation along arousal, valence, and dominance axes, contextualized by dialogue history and relationship.
• Interpersonal Stance: Building on StanceBench, SPEARBench applies an LLM-based judge to align answers on ten interpretable stance dimensions.
• Temporal and Spectral Features: Distributional metrics assess pitch variability, answer duration, and speech/voiced ratios, benchmarking against human behavior.

This framework delivers diagnostic granularity far exceeding standard metrics, surfacing nuanced discrepancies and modeling deficits that would otherwise be obscured by high-level signal quality scores.

Analysis of Model Performance

SPEARBench benchmarks seven major S2S models, including both open-weight and proprietary systems operating in full- and half-duplex streaming regimes.

Signal Quality and ASR Robustness

All major S2S LMs surpass human reference on wav-based intelligibility and speech quality, with recent models achieving WERs below 8% on automatic transcripts. While MOS metrics are also strong, these high-level metrics do not correlate with natural turn-taking and expressive behavior.

Response Timing and Turn-Taking

Response latency in S2S models remains globally higher than in natural human conversation, with half- and full-duplex models’ latencies ranging from 950ms to 2700ms (human mean: 740ms). Some models, notably Mini-Omni and GPT-realtime-2, display substantial overlap/interruption rates, which are infrequent in careful human turn transitions.

Figure 2: Distributions of turn-taking and emotional naturalness scores indicate that model answers are often less well-timed and emotionally adaptive compared to humans.

Turn-taking evaluators reveal nontrivial gaps: full-duplex models approach human-like predictive plausibility, while models with significant overlap or delayed timing, such as Mini-Omni, diverge sharply. This dimension exposes limitations not evidenced by gross timing statistics alone.

Language and Dialect Fidelity

Virtually all systems maintain target language identity (>98% English), but dialect profiling uncovers strong model biases. With the exception of Qwen2.5 and Mini-Omni, which produce Asian-influenced dialect profiles, most systems display a strong North American bias with muted adaptation to interlocutor dialect—even when the question accent diverges.

Figure 3: Model dialect profiles show systematic differences, with some systems showing significant Asian, Slavic, or Semitic dialectal cues.

Emotional Naturalness and Entrainment

Despite high intelligibility, models fail to match human emotional adaptation. Emotional naturalness scores are significantly lower for all model outputs (paired Friedman test $p < 10^{-13}$ relative to human answers); TRACE-based analysis shows model answers can often preserve gross valence but fail to track intensity (arousal) and control dimension (dominance) of the preceding question.

Figure 4: Correlations between question and answer arousal/valence/dominance show diminished emotional entrainment for model outputs relative to human performance.

Interpersonal Stance Preservation

Models largely match the interpersonal stance of human answers, as measured by high stance-alignment rates (>95%). The principal outlier, Mini-Omni, is systematically rated as less “Focused,” indicating a dimension of pragmatic appropriateness not captured by other models.

Figure 5: Most contemporary S2S models maintain consistent positive stance in interaction, though certain pragmatic dimensions (e.g., Focus) are not always matched.

Temporal and Spectral Characteristics

Model answers typically display lower pitch variance and greater duration than human answers, indicating less expressive prosody and longer, potentially unnatural, formulations.

Figure 6: Pitch distributional analysis reveals that synthesized answers cluster more narrowly and are less expressive than human conversational speech.

Practical and Theoretical Implications

SPEARBench demonstrates conclusively that advances in S2S LMs’ ASR and text-content fidelity do not automatically yield human-like conversational competence. The inability of current models to realize naturalistic timing, dialectal alignment, emotional adaptation, and expressive prosody indicates specific deficits in their generative architectures, training protocols, or serving routines. These gaps imply that current paradigms—mono-modal, text-first, or signal-level decoders alone—are insufficient for genuine conversational parity and motivate continued research into training objectives, data diversity, and real-time adaptation.

On the practical side, SPEARBench enables rapid, reproducible cross-system comparisons without human raters, accelerating progress for both academic and commercial actors and informing system deployment in social, assistive, or collaborative robotic environments. Theoretically, the modular approach points towards the decomposition of “naturalness” into measurable, scientifically grounded subcomponents, facilitating more mechanistic understanding of conversational AI.

Future Directions

SPEARBench’s extensible framework invites augmentation with new tasks (multi-party dialogue, code-switching, spontaneous emotion), more languages, and human-perceptual validation. The modularization of metrics encourages the fusion of signal, content, and interactional evaluation, providing a template for holistic conversational AI benchmarking. Furthermore, the performance shortfall in emotional and dialectal alignment highlights critical research directions for both model pretraining and prompt grounding strategies.

Conclusion

SPEARBench establishes a new baseline for the evaluation of conversational naturalness in S2S LMs, leveraging systematically validated metrics, extensible implementation, and publicly available data and tools. Results define the state of the art and illuminate open problems at the intersection of speech modeling, affective computing, and social conversational AI. While signal-level performance continues to improve, interactional competence remains a differentiator; SPEARBench operationalizes this gap and provides a foundation for quantifiable progress in the next generation of dialogue models.