SwanBench-Speech: Long-Form Speech Benchmark

• SwanBench-Speech is a benchmark for long-form speech generation that assesses both single-speaker and dialogue outputs using seven automated metrics.
• It decomposes quality into acoustics, semantics, and expressiveness, addressing issues like speaker consistency, acoustic environment stability, and prosodic coherence.
• The benchmark applies scenario-specific stress tests across 17 diverse contexts to highlight the challenges of maintaining long-range expressive structure.

SwanBench-Speech is a benchmark for long-form speech generation that evaluates both single-speaker long-form speech generation and dialogue generation. It is motivated by the claim that sentence-level fidelity no longer captures the dominant failure modes of modern speech systems: a model may remain locally clear while drifting in speaker identity, changing acoustic environment, flattening prosody, or failing to build paragraph-level expressive structure. The benchmark therefore decomposes long-form quality into acoustics, semantics, and expressiveness, and operationalizes these axes with seven automated metrics over 1,101 samples spanning 17 downstream scenarios (Pan et al., 27 May 2026).

1. Scope and defining problem

SwanBench-Speech is explicitly framed against short-utterance TTS evaluation. The benchmark’s premise is that long-context synthesis introduces qualitatively different errors from sentence-level synthesis, especially in consistency, coherence, and hierarchy. In this framing, conventional metrics such as WER and CER are useful but insufficient, because long-form systems can preserve local pronunciation while degrading in speaker stability, acoustic-scene continuity, paragraph-level prosody, or expressive development over time (Pan et al., 27 May 2026).

The benchmark covers two task settings: single-speaker long-form speech generation and dialogue generation. Its dataset contains 1,101 samples with an average length of 228.6 words in the main comparison table. The appendix additionally reports average text lengths of 271.8 Chinese characters and 174.6 English words, with most samples in the range [80, 500]. Language balance is reported as 49.3% Chinese and 50.7% English. The collection also includes 101 multi-speaker samples involving 3 or 4 speakers, so the benchmark is not restricted to monologue or two-speaker interaction (Pan et al., 27 May 2026).

A central design claim is that long-form evaluation should distinguish local waveform quality from long-range structure. This is why SwanBench-Speech separates Timbre Consistency and Reverb Consistency from Content Accuracy, Prosodic Coherence, Expressive Richness, and Expressive Hierarchy. The benchmark thereby treats identity stability, acoustic-scene stability, semantic faithfulness, local expressive quality, and paragraph-level expressive development as distinct but related properties (Pan et al., 27 May 2026).

2. Scenario taxonomy and corpus composition

The benchmark organizes its 17 scenarios into three challenge groups. This taxonomy is one of its defining features because the scenarios are meant to instantiate different long-form stresses rather than merely topical variation (Pan et al., 27 May 2026).

Challenge axis	Scenarios
Acoustics Challenge	customer service, podcast, chat, debate, audiobook, interview
Semantics Challenge	lesson, popular science, presentation, seminar, news
Expressiveness Challenge	drama, talk show, hosting/host, speech, live streaming, sportscast

The Acoustics Challenge includes customer service, podcast, chat, debate, audiobook, and interview. These settings emphasize sustained speaker stability and shared acoustic-scene consistency, especially in dialogue-like or narratively extended material. The Semantics Challenge includes lesson, popular science, presentation, seminar, and news, and is intended to be information-dense and syntactically complex, stressing semantic faithfulness and paragraph-level prosodic organization. The Expressiveness Challenge includes drama, talk show, hosting/host, speech, live streaming, and sportscast, each chosen because it requires stronger emotion, energy, and dynamic expressive variation over extended context (Pan et al., 27 May 2026).

The corpus is built from online text corpora, online audio media, and LLM generation. For text-rich scenarios such as audiobook, news, drama, and host, the creators crawl or OCR long-form text from web sources including classic literature, web novels, and TouTiao, then clean it with the clean-text library and manually proofread it. For audio-rich scenarios, materials are crawled from YouTube, Bilibili, Spotify, RedNote, and Apple Podcasts, denoised with ZipEnhancer, filtered with DNS-MOS at a threshold of 3.5, diarized with 3D-Speaker, and transcribed with SenseVoice-Small, after which annotators manually verify transcripts and metadata. For scenarios such as chat, presentations, and customer service, the benchmark also uses GPT-5 to generate structured test cases from prompts containing scenario, topic, and task information (Pan et al., 27 May 2026).

This construction suggests a deliberate attempt to cover both naturally occurring long-form speech situations and structured synthetic prompts. A plausible implication is that SwanBench-Speech is intended as a downstream-oriented evaluation suite rather than a purely corpus-preserving test collection.

3. Curation, filtering, and annotation procedure

The curation pipeline is multi-stage and mixes automatic filtering with manual review. The benchmark first performs semantic de-duplication: GPT-5 extracts topics, keywords, and summaries, then Sentence-BERT embeddings are used for deduplication with cosine similarity above 0.8. It then performs quality evaluation, where GPT-5 scores textual expressiveness and content consistency on a 1–5 scale, retaining only samples with an overall recommendation score above 2. Finally, privacy and ethical filtering is performed with DeepSeek V3.2 plus chain-of-thought prompting, selectively anonymizing private individuals while preserving public figures, and discarding toxic content (Pan et al., 27 May 2026).

A later manual review stage replaces placeholders with harmless fictitious entities, removes residual errors, and replenishes discarded samples. Human annotation is reported in two stages. One stage used three undergraduate students for data annotation and verification at \$0.20 per instance**, with all data double-checked, for **\$220 total. A later manual-review stage involved five undergraduate students at \$0.30 per instance**, for **\$330 total (Pan et al., 27 May 2026).

The benchmark also standardizes reference voices for model evaluation. For open-source systems, the appendix lists 25 reference audio prompts drawn from Emilia, AISHELL-3, NCSSD, LibriSpeech, MSPPodcast, and ChildMandarin, spanning genders, ages, and both benchmark languages. However, the reported scores use the best-performing voice per model. Closed-source systems are evaluated with fixed built-in voices such as Alloy/Onyx/Nova for OpenAI, Puck/Aoede for Gemini, and named ElevenLabs voices (Pan et al., 27 May 2026).

This evaluation choice is consequential. It does not make the benchmark prompt-invariant; rather, it tries to reduce model-specific voice-prompt mismatch. That design differs from a strictly fixed-prompt protocol and should be interpreted accordingly.

4. Evaluation dimensions and metric formalization

SwanBench-Speech defines seven automated metrics over three principal axes: Acoustics, Semantics, and Expressiveness (Pan et al., 27 May 2026).

Acoustics

Timbre Consistency measures whether speaker identity remains stable over time. For single-speaker audio $w$, the waveform is segmented with a 3 s window and 2 s stride, speaker embeddings $\mathbf{e}_i$ are extracted, and pairwise cosine similarity is averaged: $$\mathrm{sim}_{i,j} = \cos\!\left( \frac{\mathbf{e}_i}{\lVert \mathbf{e}_i \rVert}, \frac{\mathbf{e}_j}{\lVert \mathbf{e}_j \rVert} \right), \quad \forall i \neq j.$$ For multi-speaker audio, the benchmark verifies speaker turns, uses forced alignment to obtain sentence timestamps, concatenates segments for each speaker $k$ into $\tilde{w}_k$, computes per-speaker consistency $a_k$, and averages: $\mathrm{Score}_{\text{multi} = \frac{1}{K} \sum_{k=1}^{K} a_k.$ The implementation uses WavLM TDCNN speaker embeddings (Pan et al., 27 May 2026).

Reverb Consistency measures stability of the acoustic environment over time. The benchmark computes SRMR on sliding windows with the same 3 s / 2 s settings, discards windows with more than 60% non-speech frames using VAD, and takes the standard deviation of the valid reverberation sequence $\{r_i\}_{i=1}^n$. Lower values are better (Pan et al., 27 May 2026).

Sound Fidelity is measured with SQUIM-PESQ, used as a non-intrusive approximation to PESQ in the absence of reference audio. The appendix notes an approximate score range of $0.30 per instance**, for **\$0, typically above 1.0 for speech (Pan et al., 27 May 2026).

Semantics

Content Accuracy is computed by transcribing generated speech with FunASR-Nano and comparing it to ground-truth text using CER for Chinese and WER for English. The paper reports normalization with punctuation removal, whitespace standardization, Traditional-to-Simplified conversion via zhconv, and ASCII filtering for English through clean-text, with final scoring by JiWER (Pan et al., 27 May 2026).

Prosodic Coherence evaluates pauses, speaking rate, intonational flow, rhythmic structure, and naturalness over extended speech. SwanBench-Speech uses SpeechJudge, described as fine-tuned from Qwen2.5-Omni-7B, and prompts it over three subdimensions: Prosodic Coherence & Flow, Rhythmic Hierarchy & Layering, and Overall Naturalness. The score range is 1.0–5.0. The appendix states that the benchmark runs 10 independent evaluations per sample and averages the result (Pan et al., 27 May 2026).

Expressiveness

Expressive Richness is a chunk-level measure of emotional and performative quality. The audio is partitioned into non-overlapping 10-second chunks $0.30 per instance**, for **\$1, each chunk receives a score $0.30 per instance**, for **\$2, and the final score is

$0.30 per instance**, for **\$3

The prompt asks the evaluator to judge emotional resonance, character portrayal, storytelling, and immersion while ignoring sudden stop, audio quality, timbre consistency, and pronunciation (Pan et al., 27 May 2026).

Expressive Hierarchy evaluates the entire audio sequence rather than chunks. It targets Emotional Variation & Arc, Vocal Dynamics, and Scene Appropriateness & Structural Fit, again on a 1.0–5.0 scale. This is the benchmark’s paragraph-level expressive metric and is designed to capture whether the performance develops over time rather than remaining flat (Pan et al., 27 May 2026).

For expressiveness evaluation, the paper reports Gemini 3 Pro as the strongest evaluator among tested LALMs. For forced alignment, it uses Paraformer for Chinese and WhisperX for English. Audio is generally resampled to 24 kHz for evaluation (Pan et al., 27 May 2026).

5. Empirical results and benchmark-level findings

The benchmark’s most important empirical claim is that many current systems are already strong on local properties but remain substantially below real recordings on long-range properties, especially Reverb Consistency, Prosodic Coherence, Expressive Richness, and Expressive Hierarchy (Pan et al., 27 May 2026).

For single-speaker long-form speech, the Real Speech reference baseline reports: 0.96 Timbre Consistency, 1.91 Reverb Consistency, 3.62 Sound Fidelity, 0.070 / 0.074 CER/WER, 4.04 Prosody, 4.35 Expressive Richness, and 3.94 Expressive Hierarchy. Many synthetic systems approach the real baseline on timbre, fidelity, and lexical accuracy. For example, Minimax-Speech-02-hd reports 0.93, 1.38, 3.82, 0.032 / 0.119, 3.95, 3.80, and 3.26 on those respective metrics, while Gemini-2.5-pro-preview-tts reaches 4.14 Expressive Richness and 3.51 Expressive Hierarchy. Among open-source models, VibeVoice is the strongest on expressiveness with 3.71 Richness, 3.34 Hierarchy, and 3.90 Prosody (Pan et al., 27 May 2026).

For dialogue generation, the Real Dialogue baseline reports: 0.95 Timbre Consistency, 2.73 Reverb Consistency, 2.94 Sound Fidelity, 0.050 / 0.137 CER/WER, 3.95 Prosody, 4.42 Richness, and 4.17 Hierarchy. The paper emphasizes that dialogue remains particularly challenging for acoustic-scene stability: open-source dialogue models average 3.45 on Reverb Consistency and closed-source systems average 3.36, both notably worse than 2.73 for real dialogue. Among proprietary systems, Gemini-2.5-pro-preview-tts is the strongest overall dialogue model with 4.06 Prosody, 4.06 Richness, and 4.02 Hierarchy; SeedTTS-Podcast is also strong at 3.93, 3.84, and 3.84 on those three metrics. Among open-source dialogue systems, SoulX-Podcast is the strongest overall (Pan et al., 27 May 2026).

The benchmark also reports scenario- and length-dependent degradation. Acoustic Challenge scenarios primarily stress reverb consistency and shared sound field. Semantic Challenge scenarios often preserve pronunciation but degrade Prosodic Coherence, suggesting that information-dense and syntactically complex texts remain difficult to deliver naturally. The most severe degradation occurs in Expressiveness Challenge scenarios, where models often deteriorate not only in expressive scores but across multiple metrics. The authors interpret this as evidence that expressive long-form data remains underrepresented in current training mixtures (Pan et al., 27 May 2026).

Length sensitivity is similarly pronounced. On samples longer than about 100 words, many systems degrade in Reverb Consistency, Prosodic Coherence, and Expressive Hierarchy, while timbre-related metrics are comparatively stable. This supports the benchmark’s core distinction between identity retention and long-range structural control (Pan et al., 27 May 2026).

The paper also draws an architectural contrast between autoregressive and non-autoregressive systems. It reports that non-autoregressive systems are generally more robust and efficient for long text but tend to over-smooth rhythm and expression, whereas autoregressive systems often achieve better expressiveness and prosody but can show stronger content-accuracy degradation due to error propagation. F5TTS is identified as the strongest NAR system in the experiments but still weaker than many AR models on expressive hierarchy; SparkTTS is highlighted for poor long-form content accuracy and strong length sensitivity (Pan et al., 27 May 2026).

6. Validation, adoption, and position among speech benchmarks

A distinctive feature of SwanBench-Speech is that it validates several automated metrics against human judgments. For Prosodic Coherence, the paper samples 50 pairs of audios generated from the same text by different models, collects judgments from 10 human evaluators, and reports SRCC = 0.82 between the automated metric difference and human preference. For Expressiveness, it samples 200 audio clips across models and tasks, compares evaluator outputs to human MOS from 10 human evaluators, and finds that traditional MOS models such as UTMOS, UTMOSv2, SQUIM-MOS, and DNS-MOS correlate poorly or negatively with expressiveness judgments, whereas Gemini3-Pro achieves SRCC = 0.71 for Expressive Richness and SRCC = 0.62 for Expressive Hierarchy. For Timbre Consistency, the reported correlations with human MOS are SRCC = 0.75, PLCC = 0.77, and KRCC = 0.59; for Sound Fidelity, SQUIM-PESQ obtains SRCC = 0.72, PLCC = 0.47, and KRCC = 0.53 (Pan et al., 27 May 2026).

Later work already uses SwanBench-Speech as an external evaluation framework. SwanVoice evaluates zero-shot monologue TTS on the “Expressive Challenge subset” of SwanBench-Speech and dialogue generation on the benchmark’s dialogue setting, following the three axes of acoustics, semantics, and expressiveness. In that evaluation, SwanVoice reports higher richness and hierarchy than the evaluated open-source baselines in both monologue and dialogue, while content accuracy remains the main limitation (Li et al., 29 May 2026). This later usage indicates that SwanBench-Speech functions not only as a leaderboard but as a stress test for long-form expressive coherence.

Within the broader benchmark landscape, SwanBench-Speech is narrower than LongSpeech, which evaluates long-form speech transcription, translation, summarization, speaker counting, content separation, and temporal localization over roughly 10-minute recordings (Yang et al., 20 Jan 2026); narrower than VocalBench, which targets speech interaction models over semantic quality, acoustic performance, conversational abilities, and robustness with 9,400 instances (Liu et al., 21 May 2025); and narrower than StyleBench, which specializes in multi-turn style-intensity control across emotion, speed, volume, and pitch (Zhao et al., 8 Mar 2026). Its distinguishing feature is not general speech interaction breadth but a concentrated evaluation of long-form generation quality under monologue and dialogue settings (Pan et al., 27 May 2026).

The benchmark also states several limitations. It currently covers only Chinese and English, not low-resource languages or broader accent variation. The authors describe semantic evaluation as still limited, particularly for emotional and stylistic transitions grounded in deeper semantic understanding. They also note dependence on closed-source evaluators, especially Gemini 3 Pro, for expressiveness scoring, and acknowledge that prompt-speech diversity is limited relative to the full space of possible reference voices (Pan et al., 27 May 2026).

Taken together, SwanBench-Speech defines long-form speech generation as a problem of sustained identity, scene, semantic fidelity, prosodic coherence, and expressive development rather than short-utterance naturalness alone. The benchmark’s reported results indicate that contemporary systems are already strong on local fidelity and often competitive on content accuracy, but still exhibit a substantial gap to real recordings in the long-range dimensions that dominate extended monologue and dialogue synthesis (Pan et al., 27 May 2026).