---
title: 'SwanBench-Speech: Long-Form Speech Benchmark'
url: https://www.emergentmind.com/topics/swanbench-speech
type: topic
---

# SwanBench-Speech: Long-Form Speech Benchmark

SwanBench-Speech is a benchmark for **long-form speech generation** that evaluates both **single-speaker long-form speech generation** and **dialogue generation**. It is motivated by the claim that sentence-level fidelity no longer captures the dominant failure modes of modern speech systems: a model may remain locally clear while drifting in speaker identity, changing acoustic environment, flattening prosody, or failing to build paragraph-level expressive structure. The benchmark therefore decomposes long-form quality into **acoustics**, **semantics**, and **expressiveness**, and operationalizes these axes with **seven automated metrics** over **1,101 samples** spanning **17 downstream scenarios** [2605.28618].

## 1. Scope and defining problem

SwanBench-Speech is explicitly framed against short-utterance TTS evaluation. The benchmark’s premise is that long-context synthesis introduces qualitatively different errors from sentence-level synthesis, especially in **consistency**, **coherence**, and **hierarchy**. In this framing, conventional metrics such as WER and CER are useful but insufficient, because long-form systems can preserve local pronunciation while degrading in speaker stability, acoustic-scene continuity, paragraph-level prosody, or expressive development over time [2605.28618].

The benchmark covers **two task settings**: **single-speaker long-form speech generation** and **dialogue generation**. Its dataset contains **1,101 samples** with an average length of **228.6 words** in the main comparison table. The appendix additionally reports average text lengths of **271.8 Chinese characters** and **174.6 English words**, with most samples in the range **[80, 500]**. Language balance is reported as **49.3% Chinese** and **50.7% English**. The collection also includes **101 multi-speaker samples** involving **3 or 4 speakers**, so the benchmark is not restricted to monologue or two-speaker interaction [2605.28618].

A central design claim is that long-form evaluation should distinguish local waveform quality from long-range structure. This is why SwanBench-Speech separates **Timbre Consistency** and **Reverb Consistency** from **Content Accuracy**, **Prosodic Coherence**, **Expressive Richness**, and **Expressive Hierarchy**. The benchmark thereby treats identity stability, acoustic-scene stability, semantic faithfulness, local expressive quality, and paragraph-level expressive development as distinct but related properties [2605.28618].

## 2. Scenario taxonomy and corpus composition

The benchmark organizes its **17 scenarios** into three challenge groups. This taxonomy is one of its defining features because the scenarios are meant to instantiate different long-form stresses rather than merely topical variation [2605.28618].

| Challenge axis | Scenarios |
|---|---|
| Acoustics Challenge | customer service, podcast, chat, debate, audiobook, interview |
| Semantics Challenge | lesson, popular science, presentation, seminar, news |
| Expressiveness Challenge | drama, talk show, hosting/host, speech, live streaming, sportscast |

The **Acoustics Challenge** includes customer service, podcast, chat, debate, audiobook, and interview. These settings emphasize sustained speaker stability and shared acoustic-scene consistency, especially in dialogue-like or narratively extended material. The **Semantics Challenge** includes lesson, popular science, presentation, seminar, and news, and is intended to be information-dense and syntactically complex, stressing semantic faithfulness and paragraph-level prosodic organization. The **Expressiveness Challenge** includes drama, talk show, hosting/host, speech, live streaming, and sportscast, each chosen because it requires stronger emotion, energy, and dynamic expressive variation over extended context [2605.28618].

The corpus is built from **online text corpora**, **online audio media**, and **LLM generation**. For text-rich scenarios such as **audiobook, news, drama, and host**, the creators crawl or OCR long-form text from web sources including classic literature, web novels, and TouTiao, then clean it with the `clean-text` library and manually proofread it. For audio-rich scenarios, materials are crawled from **YouTube, Bilibili, Spotify, RedNote, and Apple Podcasts**, denoised with **ZipEnhancer**, filtered with **DNS-MOS** at a threshold of **3.5**, diarized with **3D-Speaker**, and transcribed with **SenseVoice-Small**, after which annotators manually verify transcripts and metadata. For scenarios such as **chat, presentations, and customer service**, the benchmark also uses **GPT-5** to generate structured test cases from prompts containing scenario, topic, and task information [2605.28618].

This construction suggests a deliberate attempt to cover both naturally occurring long-form speech situations and structured synthetic prompts. A plausible implication is that SwanBench-Speech is intended as a downstream-oriented evaluation suite rather than a purely corpus-preserving test collection.

## 3. Curation, filtering, and annotation procedure

The curation pipeline is multi-stage and mixes automatic filtering with manual review. The benchmark first performs **semantic de-duplication**: **GPT-5** extracts topics, keywords, and summaries, then **Sentence-BERT** embeddings are used for deduplication with cosine similarity above **0.8**. It then performs **quality evaluation**, where **GPT-5** scores textual expressiveness and content consistency on a 1–5 scale, retaining only samples with an overall recommendation score above **2**. Finally, **privacy and ethical filtering** is performed with **DeepSeek V3.2** plus chain-of-thought prompting, selectively anonymizing private individuals while preserving public figures, and discarding toxic content [2605.28618].

A later manual review stage replaces placeholders with harmless fictitious entities, removes residual errors, and replenishes discarded samples. Human annotation is reported in two stages. One stage used **three undergraduate students** for data annotation and verification at **\$0.20 per instance**, with all data double-checked, for **\$220** total. A later manual-review stage involved **five undergraduate students** at **\$0.30 per instance**, for **\$330** total [2605.28618].

The benchmark also standardizes reference voices for model evaluation. For open-source systems, the appendix lists **25 reference audio prompts** drawn from **Emilia, AISHELL-3, NCSSD, LibriSpeech, MSPPodcast, and ChildMandarin**, spanning genders, ages, and both benchmark languages. However, the reported scores use the **best-performing voice per model**. Closed-source systems are evaluated with fixed built-in voices such as **Alloy/Onyx/Nova** for OpenAI, **Puck/Aoede** for Gemini, and named ElevenLabs voices [2605.28618].

This evaluation choice is consequential. It does not make the benchmark prompt-invariant; rather, it tries to reduce model-specific voice-prompt mismatch. That design differs from a strictly fixed-prompt protocol and should be interpreted accordingly.

## 4. Evaluation dimensions and metric formalization

SwanBench-Speech defines **seven automated metrics** over three principal axes: **Acoustics**, **Semantics**, and **Expressiveness** [2605.28618].

### Acoustics

**Timbre Consistency** measures whether speaker identity remains stable over time. For single-speaker audio \(w\), the waveform is segmented with a **3 s window** and **2 s stride**, speaker embeddings \(\mathbf{e}_i\) are extracted, and pairwise cosine similarity is averaged:
\[
\mathrm{sim}_{i,j} = \cos\!\left( \frac{\mathbf{e}_i}{\lVert \mathbf{e}_i \rVert}, \frac{\mathbf{e}_j}{\lVert \mathbf{e}_j \rVert} \right), \quad \forall i \neq j.
\]
For multi-speaker audio, the benchmark verifies speaker turns, uses forced alignment to obtain sentence timestamps, concatenates segments for each speaker \(k\) into \(\tilde{w}_k\), computes per-speaker consistency \(a_k\), and averages:
\[
\mathrm{Score}_{\text{multi} = \frac{1}{K} \sum_{k=1}^{K} a_k.
\]
The implementation uses **WavLM TDCNN** speaker embeddings [2605.28618].

**Reverb Consistency** measures stability of the acoustic environment over time. The benchmark computes **SRMR** on sliding windows with the same **3 s / 2 s** settings, discards windows with more than **60% non-speech frames** using VAD, and takes the **standard deviation** of the valid reverberation sequence \(\{r_i\}_{i=1}^n\). Lower values are better [2605.28618].

**Sound Fidelity** is measured with **SQUIM-PESQ**, used as a non-intrusive approximation to PESQ in the absence of reference audio. The appendix notes an approximate score range of **\([-0.5, 4.5]\)**, typically above 1.0 for speech [2605.28618].

### Semantics

**Content Accuracy** is computed by transcribing generated speech with **FunASR-Nano** and comparing it to ground-truth text using **CER for Chinese** and **WER for English**. The paper reports normalization with punctuation removal, whitespace standardization, Traditional-to-Simplified conversion via `zhconv`, and ASCII filtering for English through `clean-text`, with final scoring by **JiWER** [2605.28618].

**Prosodic Coherence** evaluates pauses, speaking rate, intonational flow, rhythmic structure, and naturalness over extended speech. SwanBench-Speech uses **SpeechJudge**, described as fine-tuned from **Qwen2.5-Omni-7B**, and prompts it over three subdimensions: **Prosodic Coherence & Flow**, **Rhythmic Hierarchy & Layering**, and **Overall Naturalness**. The score range is **1.0–5.0**. The appendix states that the benchmark runs **10 independent evaluations per sample** and averages the result [2605.28618].

### Expressiveness

**Expressive Richness** is a chunk-level measure of emotional and performative quality. The audio is partitioned into non-overlapping **10-second chunks** \(\{c_i\}_{i=1}^{M}\), each chunk receives a score \(s_i\), and the final score is
\[
\text{Score}_{\text{rich} = \frac{1}{M} \sum_{i=1}^{M} s_i.
\]
The prompt asks the evaluator to judge emotional resonance, character portrayal, storytelling, and immersion while ignoring sudden stop, audio quality, timbre consistency, and pronunciation [2605.28618].

**Expressive Hierarchy** evaluates the **entire audio sequence** rather than chunks. It targets **Emotional Variation & Arc**, **Vocal Dynamics**, and **Scene Appropriateness & Structural Fit**, again on a **1.0–5.0** scale. This is the benchmark’s paragraph-level expressive metric and is designed to capture whether the performance develops over time rather than remaining flat [2605.28618].

For expressiveness evaluation, the paper reports **Gemini 3 Pro** as the strongest evaluator among tested LALMs. For forced alignment, it uses **Paraformer** for Chinese and **WhisperX** for English. Audio is generally resampled to **24 kHz** for evaluation [2605.28618].

## 5. Empirical results and benchmark-level findings

The benchmark’s most important empirical claim is that many current systems are already strong on local properties but remain substantially below real recordings on long-range properties, especially **Reverb Consistency**, **Prosodic Coherence**, **Expressive Richness**, and **Expressive Hierarchy** [2605.28618].

For **single-speaker long-form speech**, the **Real Speech** reference baseline reports: **0.96** Timbre Consistency, **1.91** Reverb Consistency, **3.62** Sound Fidelity, **0.070 / 0.074** CER/WER, **4.04** Prosody, **4.35** Expressive Richness, and **3.94** Expressive Hierarchy. Many synthetic systems approach the real baseline on timbre, fidelity, and lexical accuracy. For example, **Minimax-Speech-02-hd** reports **0.93**, **1.38**, **3.82**, **0.032 / 0.119**, **3.95**, **3.80**, and **3.26** on those respective metrics, while **Gemini-2.5-pro-preview-tts** reaches **4.14** Expressive Richness and **3.51** Expressive Hierarchy. Among open-source models, **VibeVoice** is the strongest on expressiveness with **3.71** Richness, **3.34** Hierarchy, and **3.90** Prosody [2605.28618].

For **dialogue generation**, the **Real Dialogue** baseline reports: **0.95** Timbre Consistency, **2.73** Reverb Consistency, **2.94** Sound Fidelity, **0.050 / 0.137** CER/WER, **3.95** Prosody, **4.42** Richness, and **4.17** Hierarchy. The paper emphasizes that dialogue remains particularly challenging for acoustic-scene stability: open-source dialogue models average **3.45** on Reverb Consistency and closed-source systems average **3.36**, both notably worse than **2.73** for real dialogue. Among proprietary systems, **Gemini-2.5-pro-preview-tts** is the strongest overall dialogue model with **4.06** Prosody, **4.06** Richness, and **4.02** Hierarchy; **SeedTTS-Podcast** is also strong at **3.93**, **3.84**, and **3.84** on those three metrics. Among open-source dialogue systems, **SoulX-Podcast** is the strongest overall [2605.28618].

The benchmark also reports scenario- and length-dependent degradation. **Acoustic Challenge** scenarios primarily stress reverb consistency and shared sound field. **Semantic Challenge** scenarios often preserve pronunciation but degrade **Prosodic Coherence**, suggesting that information-dense and syntactically complex texts remain difficult to deliver naturally. The most severe degradation occurs in **Expressiveness Challenge** scenarios, where models often deteriorate not only in expressive scores but across multiple metrics. The authors interpret this as evidence that expressive long-form data remains underrepresented in current training mixtures [2605.28618].

Length sensitivity is similarly pronounced. On samples longer than about **100 words**, many systems degrade in **Reverb Consistency**, **Prosodic Coherence**, and **Expressive Hierarchy**, while timbre-related metrics are comparatively stable. This supports the benchmark’s core distinction between identity retention and long-range structural control [2605.28618].

The paper also draws an architectural contrast between **autoregressive** and **non-autoregressive** systems. It reports that non-autoregressive systems are generally more robust and efficient for long text but tend to over-smooth rhythm and expression, whereas autoregressive systems often achieve better expressiveness and prosody but can show stronger content-accuracy degradation due to error propagation. **F5TTS** is identified as the strongest NAR system in the experiments but still weaker than many AR models on expressive hierarchy; **SparkTTS** is highlighted for poor long-form content accuracy and strong length sensitivity [2605.28618].

## 6. Validation, adoption, and position among speech benchmarks

A distinctive feature of SwanBench-Speech is that it validates several automated metrics against human judgments. For **Prosodic Coherence**, the paper samples **50 pairs** of audios generated from the same text by different models, collects judgments from **10 human evaluators**, and reports **SRCC = 0.82** between the automated metric difference and human preference. For **Expressiveness**, it samples **200 audio clips** across models and tasks, compares evaluator outputs to human MOS from **10 human evaluators**, and finds that traditional MOS models such as **UTMOS**, **UTMOSv2**, **SQUIM-MOS**, and **DNS-MOS** correlate poorly or negatively with expressiveness judgments, whereas **Gemini3-Pro** achieves **SRCC = 0.71** for **Expressive Richness** and **SRCC = 0.62** for **Expressive Hierarchy**. For **Timbre Consistency**, the reported correlations with human MOS are **SRCC = 0.75**, **PLCC = 0.77**, and **KRCC = 0.59**; for **Sound Fidelity**, **SQUIM-PESQ** obtains **SRCC = 0.72**, **PLCC = 0.47**, and **KRCC = 0.53** [2605.28618].

Later work already uses SwanBench-Speech as an external evaluation framework. **SwanVoice** evaluates zero-shot monologue TTS on the **“Expressive Challenge subset”** of SwanBench-Speech and dialogue generation on the benchmark’s dialogue setting, following the three axes of **acoustics**, **semantics**, and **expressiveness**. In that evaluation, SwanVoice reports higher **richness** and **hierarchy** than the evaluated open-source baselines in both monologue and dialogue, while **content accuracy remains the main limitation** [2605.30993]. This later usage indicates that SwanBench-Speech functions not only as a leaderboard but as a stress test for long-form expressive coherence.

Within the broader benchmark landscape, SwanBench-Speech is narrower than **LongSpeech**, which evaluates long-form speech transcription, translation, summarization, speaker counting, content separation, and temporal localization over roughly **10-minute** recordings [2601.13539]; narrower than **VocalBench**, which targets speech interaction models over semantic quality, acoustic performance, conversational abilities, and robustness with **9,400 instances** [2505.15727]; and narrower than **StyleBench**, which specializes in multi-turn style-intensity control across **emotion, speed, volume, and pitch** [2603.07599]. Its distinguishing feature is not general speech interaction breadth but a concentrated evaluation of **long-form generation quality** under monologue and dialogue settings [2605.28618].

The benchmark also states several limitations. It currently covers only **Chinese and English**, not low-resource languages or broader accent variation. The authors describe semantic evaluation as still limited, particularly for emotional and stylistic transitions grounded in deeper semantic understanding. They also note dependence on **closed-source evaluators**, especially **Gemini 3 Pro**, for expressiveness scoring, and acknowledge that prompt-speech diversity is limited relative to the full space of possible reference voices [2605.28618].

Taken together, SwanBench-Speech defines long-form speech generation as a problem of sustained **identity**, **scene**, **semantic fidelity**, **prosodic coherence**, and **expressive development** rather than short-utterance naturalness alone. The benchmark’s reported results indicate that contemporary systems are already strong on local fidelity and often competitive on content accuracy, but still exhibit a substantial gap to real recordings in the long-range dimensions that dominate extended monologue and dialogue synthesis [2605.28618].

Source: https://www.emergentmind.com/topics/swanbench-speech