Reference-Based Prosody and Rhythm Evaluation for Spoken Dialogue Systems

Abstract: Speech-to-speech (S2S) AI agents are advancing rapidly, yet evaluation lacks interpretable speech-native measures for conversational prosody and rhythm. Because $F_0$, speaking rate, articulation rate, and pausing shift with model-predicted speaker traits and interaction state, pooled human statistics can be poorly calibrated for evaluating a particular output. Using 4000+ hours of dyadic English conversation from the Seamless Interaction dataset, we construct matched reference regimes for $F_0$ mean, $F_0$ expressivity, speech rate, articulation rate, pause ratio, and mean pause duration. We then define a percentile-based evaluation protocol: extract the same metrics from an S2S output waveform, compare them to the closest matched human reference stratum, and report percentile deviations or 5th-95th percentile out-of-regime flags. On held-out human rows, pooled references over-flag state-conditioned $F_0$ expressivity and rhythm, while matched references return flag rates closer to the nominal 10% and make deviation direction interpretable. These outputs serve as behavioral plausibility checks that complement, rather than replace, perceptual and user-centered evaluation.

• The paper demonstrates that stratifying acoustic metrics by speaker traits and interactional states avoids miscalibration in prosody and rhythm evaluation.
• It employs robust analysis of a 4000+ hour natural dialogue dataset using metrics like F₀ variability, speech rate, and pause patterns.
• The reference-based protocol offers interpretable deviation metrics, guiding iterative improvements in spoken dialogue systems.

Reference-Based Prosody and Rhythm Evaluation for Spoken Dialogue Systems

Introduction

This paper addresses a critical deficiency in the evaluation methods for speech-to-speech (S2S) AI agents, namely the absence of interpretable, speech-native benchmarks for assessing conversational prosody and rhythm. Traditional evaluation methods—principally text-based metrics, task success, or human subjective ratings—fail to meaningfully capture prosodic and temporal patterns essential to natural human dialogue. The authors propose a reference-based, percentile-driven evaluation protocol grounded in large-scale natural conversational data, enabling system outputs to be assessed for behavioral plausibility in the same acoustic dimensions that underlie human-human interactions.

Dataset and Metrics

The core empirical resource is the Seamless Interaction dataset, a 4000+ hour corpus of face-to-face English dyadic conversations, incorporating both Naturalistic and Improvised segments across more than 4,200 participants. Analyses are performed at the speaker-channel level using robust statistics to summarize prosody (mean, standard deviation, and range of fundamental frequency $F_0$) and temporal organization (speech rate in words per minute, articulation rate, pause ratio, and mean pause duration).

Speaker traits and interactional dynamics are inferred via Vox-Profile, which provides automatic predictions of speaker sex, age bin, and continuous emotion variables (arousal and dominance) for each utterance. These annotations enable stratification of reference distributions along operationally salient axes.

Empirical Characterization of Prosodic and Temporal Regimes

Prosodic Operating Regimes and Trait Conditioning

Mean $F_0$ is robustly separated by model-predicted sex label, reflecting physiological and anatomical determinants (Figure 1).

Figure 1: Prosodic operating regimes for F₀ mean, SD, and range, distinctly stratified by model-predicted sex label.

Pooling references across both sex groups produces biased evaluation: almost all below-threshold cases are male, nearly all above-threshold cases are female, rendering pooled targets operationally faulty as evaluation baselines.

Beyond mean $F_0$, prosodic expressivity—defined via trimmed standard deviation and range—demonstrates systematic upscaling with increasing arousal or dominance (Figures 2 and 3), consistent with the literature on emotional and interactive modulation of prosody.

Figure 2: $F_0$ expressivity (SD, range) scales monotonically with arousal sextile.

Figure 3: $F_0$ expressivity (SD, range) shows positive scaling across dominance sextiles.

Linear monotonic relationships are evidenced by Spearman correlations exceeding 0.4 between arousal/dominance and $F_0$ variability metrics. Pooled reference-based evaluation substantially over-flags both low- and high-state scenarios, whereas matched stratification by state brings flag rates close to the calibrated 10% expectation.

Temporal Organization and Rhythm

Conversational rhythm, captured by speech rate and pause organization, also varies meaningfully with interactional state (Figure 4).

Figure 4: Speech rate increases and pause ratio decreases with rising arousal and dominance; capturing dynamic conversational rhythm as a function of state variables.

Sex differences in tempo are negligible, but arousal and dominance yield effect sizes ($\rho$ ≈ 0.19–0.20) considerable in magnitude. System outputs with tempo characteristics outlying relative to state-matched distributions are more likely to be perceived as atypical or unnatural.

Age, as predicted by Vox-Profile, shifts the distributions of both prosodic and temporal metrics (Figure 5). This further emphasizes the futility of universal targets for timing and pausing, reinforcing the necessity of stratified references.

Figure 5: Prosodic and tempo metrics shift across model-predicted age bins, diverging by sex for mean $F_0$ and by age for speech rate and pause ratio.

Reference-Based Evaluation Protocol

The proposed protocol requires practitioners to extract the same prosodic and temporal metrics from S2S system outputs and compare these to the closest-matched human reference distributions, stratified by available speaker and interactional state annotations. Each metric is mapped to a reference percentile, with deviations outside the central 90% flagged for plausibility review. This granular, vector-based reporting supplies interpretable deviation information—direction, magnitude, and operational regime—rather than a collapsed score.

The protocol functions as an automated, speech-native behavioral plausibility filter, identifying outputs with atypical acoustic profiles along dimensions that matter for interactional credibility. Practically, this allows system developers to pinpoint where outputs diverge from empirical conversational norms and adjust prosodic or temporal modeling accordingly.

Empirical Strengths and Contradictory Claims

A core numerical result is the demonstration that pooled references often double or more the expected out-of-regime flag rate in state-conditioned scenarios: e.g., pooled references flag >16% of high-arousal cases for $F_0$ expressivity, whereas matched references return rates within 1% of calibrated expectation. Another strong claim is the necessity of operational stratification for all major acoustic metrics, including sex, arousal, dominance, and age bin, due to their quantifiable, interpretable effects.

Implications and Future Directions

The protocol sets a new calibration standard for S2S evaluation: behavioral plausibility rooted in large-scale, context-sensitive human distributions. Practically, this approach can inform the iterative refinement of speech generation modules and the tuning of TTS and S2S systems to avoid prosodic or rhythmic implausibility, particularly when deployed in diverse user-facing contexts. The paradigm also establishes clear diagnostic axes for future research on user adaptation, conversational alignment, and system personalization.

Theoretically, the empirical findings reinforce models of dialogue as adaptive, state-dependent coordination and establish normative reference regimes for multi-dimensional acoustic behavior. Beyond English, similar protocols could generalize to other languages and interactional domains contingent on the availability of equivalent datasets and robust speaker-state annotations.

Future research should validate the perceptual correlates of percentile-based deviations—whether out-of-regime flags indeed correspond to judgments of unnaturalness or interactional breakdown—and extend to multilingual, multiparty, and cross-modal (e.g., audiovisual) regimes. Integration with real-time, model-in-the-loop S2S evaluation pipelines is a critical avenue for deployment.

Conclusion

This work introduces a comprehensive, reference-based evaluation framework for conversational S2S systems, replacing static or pooled acoustic targets with context-aware percentile regimes. Empirical analysis establishes the necessity of stratification by speaker and state traits to avoid miscalibration and ensure plausibility in behavioral metrics of prosody and rhythm. The protocol augments, without supplanting, subjective and user-centered evaluation, providing an interpretable, auditable foundation for the next generation of S2S dialogue systems.