---
title: 'SwanData-Speech: TTS Benchmark & Corpus'
url: https://www.emergentmind.com/topics/swandata-speech
type: topic
---

# SwanData-Speech: TTS Benchmark & Corpus

SwanData-Speech is the name used in recent long-form text-to-speech literature for two closely related resources: a benchmark and test set called SwanBench-Speech, and a large in-the-wild speech mining pipeline and corpus used to train SwanVoice. In the benchmark formulation, it targets long-form single-speaker synthesis and multi-speaker dialogue generation, contains 1,101 test samples spanning 17 common speech scenarios, is split between 49.3 % Mandarin Chinese and 50.7 % English, and includes 1–4 speakers per sample, with 101 cases involving 3–4 speakers. In the corpus formulation, it is built from approx. 2.59 million hours of audio and supplies pause-aware word-level alignments, speaker-turn annotations, synthetic pronunciation-hard cases, quality and emotion filtering, and 24 kHz training data for zero-shot TTS [2605.28618] [2605.30993].

## 1. Nomenclature and research scope

The literature uses the label in two senses. In "Comprehensive Benchmarking of Long-Form Speech Generation in Diverse Scenarios" [2605.28618], the benchmark is explicitly named **SwanBench-Speech (aka SwanData-Speech)** and is presented as a comprehensive evaluation benchmark for long-form speech. In "SwanVoice: Expressive Long-Form Zero-Shot Speech Synthesis for Both Monologue and Dialogue" [2605.30993], **SwanData-Speech** denotes a large in-the-wild speech mining pipeline that produces two clean TTS training subsets: a monologue subset of single-speaker utterances and a dialogue subset of 2–4-speaker conversations, with speaker-turn labels and expressive variation.

The benchmark version focuses on diagnostic evaluation. It covers acoustics, semantics, and expressiveness challenges, and is organized around long-form speech generation and dialog generation. The corpus version focuses on data acquisition and supervision for model training. Its primary purpose is to feed SwanVoice, an end-to-end zero-shot text-to-speech model that must preserve both monologue quality and long-form, multi-speaker conversational coherence and expressiveness.

The distinction is operationally important. The benchmark is described as a released test set with code and prompts, whereas the corpus is drawn from internal resources plus mixed open-source subsets, and the paper does not announce a public download or license for SwanData-Speech itself. This suggests that the name identifies an ecosystem rather than a single uniformly distributed dataset [2605.30993].

## 2. Corpus construction, segmentation, and annotation

The corpus pipeline begins from in-the-wild sources: internal ByteDance resources plus selected open-source Chinese and English corpora, including podcasts, radio dramas, films, TV, livestreams, and audiobooks. The raw collection is **approx. 2.59 million hours of audio**, with **~2.24 million hours Chinese** and **~0.35 million hours English**. The monologue pool used for pretraining contains **~2 million hours of single-speaker speech**. The dialogue pool is formed from speaker-diarized multi-speaker segments merged to contain **2–4 speakers**, up to **120 s** long, with **no silence gap > 2 s**. Speaker demographics such as gender, age, and accents are **not specified in the paper**, and the number of unique speakers is **not reported** [2605.30993].

Filtering and preprocessing are speaker-aware. The pipeline applies ASR language identification to retain only Chinese and English, isolates the vocal component via a vocal-separation tool, and performs speaker diarization using the **3D-Speaker toolkit + CAM++ + spectral clustering**. Single-speaker segments are constrained to **≥ 0.1 s and ≤ 60 s**. Adjacent same-speaker segments are merged if the gap is **≤ 2 s**. Quality filtering uses **DNSMOS (non-intrusive), non-intrusive PESQ, non-intrusive STOI**, and emotion filtering uses **emotion2vec+ confidence** to extract a high-expressiveness subset.

Transcription and alignment are pause-aware. ASR uses **SenseVoice-Small (no ITN)**, followed by a punctuation restoration model. In dialogue, each speaker turn is wrapped in **`<S{id}> … </S{id}>`** tokens. Word-level timestamps and explicit blank states are then produced by **Swan Forced Aligner**. Acoustic pauses are converted into symbols by gap duration between word-anchor timestamps: **gap < 0.08 s → ignored**, **0.08 s ≤ gap < 0.18 s → insert `<|sp|>`**, **0.18 s ≤ gap < 0.45 s → insert comma `,`**, and **gap ≥ 0.45 s → insert period/exclamation/question (default `.`)**. If transcript punctuation and acoustic pause disagree, punctuation is removed or inserted to match the audio. The reported path-score formula is
$$
\mathrm{Score}(z) = \sum_t u^*_{t,s} + \sum_t \tau^*(z_t, r_t).
$$

The pipeline also adds explicit pronunciation supervision. A synthetic “hard” subset is generated with **RobustMegaTTS3** to cover rare dictionary words, polyphonic characters, tone sandhi, erhua, homographs, stress patterns, and code-switching. The specification includes **1 entry per word from GCIDE 0.54 + Table of General Standard Chinese Characters → 5 sentences each → ~full dictionary coverage**, plus **20 K Chinese “hard” pronunciations + 20 K English hard cases + 100 K Chinese–English code-switch examples**. The tokenizer combines **BPE (CosyVoice) + dedicated `<|sp|>` token + 1,549 pinyin syllables in vocabulary**; some Chinese characters are randomly replaced with pinyin during training, and pinyin hints can be provided at inference to disambiguate polyphonic characters [2605.30993].

## 3. Scenario taxonomy and benchmark design

The benchmark decomposes long-form evaluation into **17 common speech scenarios** distributed across three challenge families. It contains **~380 samples** for acoustics (**~34.5 % of data**), **~310** for semantics (**~28.1 %**), and **~411** for expressiveness (**~37.4 %**) [2605.28618].

| Axis | Scenario count | Scenarios |
|---|---:|---|
| Acoustics | 6 | Customer Service; Audiobook; Podcast; Chat; Debate; Interview |
| Semantics | 5 | News; Popular Science; Lesson; Seminar; Presentation |
| Expressiveness | 6 | Sportcast; Live Streaming; Speech (motivational); Host; Talk Show; Drama |

The scenario descriptions are deliberately tied to long-context failure modes. **Customer Service** involves long product or policy explanations by a single speaker and requires high fidelity and zero artifacts. **Audiobook** uses single-speaker narration of **10 min+** and demands timbre and environmental consistency for immersion. **Podcast** requires strict speaker-switch fidelity and stable reverberation. **Chat / Debate / Interview** covers real-world multi-speaker settings with frequent turn-taking and varied acoustic backgrounds.

The semantics scenarios are constructed to stress textual faithfulness and paragraph-scale prosody. **News & Popular Science** are fact-dense and terminology-heavy, and test content accuracy under long context. **Lesson, Seminar, Presentation** are monologues with prosodic variety, including pauses and emphasis, and test paragraph-level prosody coherence. The expressiveness scenarios probe sentence-level and paragraph-level affective control. **Sportcast & Live Streaming** require sustained expressive richness, while **Speech (motivational), Host, Talk Show, Drama** require both sentence-level richness and paragraph-level dynamic hierarchy. This organization makes the benchmark explicitly scenario-wise rather than domain-agnostic [2605.28618].

## 4. Three-axis evaluation protocol

SwanBench-Speech defines **long-form speech quality** along three orthogonal axes: **Acoustics**, **Semantics**, and **Expressiveness**. Acoustics comprises **Timbre Consistency**, **Reverb Consistency**, and **Sound Fidelity**. Semantics comprises **Content Accuracy** and **Prosodic Coherence**. Expressiveness comprises **Expressive Richness** and **Expressive Hierarchy** [2605.28618].

For **Timbre Consistency**, speaker embeddings are extracted by sliding **3 s** windows with **2 s** stride. For a single-speaker sample $w$, with embeddings $e_1 \ldots e_n$, pairwise cosine similarities are computed as
$$
\mathrm{sim}_{ij} = \cos\!\left(\frac{e_i}{\lVert e_i \rVert}, \frac{e_j}{\lVert e_j \rVert}\right), \quad i \ne j,
$$
and
$$
\mathrm{Score}_{\mathrm{timbre}}(w) = \frac{1}{n(n-1)} \sum_{i \ne j} \mathrm{sim}_{ij}.
$$
For multi-speaker samples, the benchmark averages single-speaker scores per diarized stream. The range is **[–1, 1]**, though in practice **[0.8, 1.0]**, and higher indicates better identity stability. For **Reverb Consistency**, the metric uses **SRMR(windows; 3 s/2 s)** after VAD-based non-speech filtering:
$$
\mathrm{Score}_{\mathrm{reverb}}(w) = \sqrt{\frac{1}{n}\sum_{i=1}^{n}(r_i-\mu)^2},
$$
where $\mu=\mathrm{mean}(r_i)$. Lower is better, with ideal value **0** and typical long-form targets **< 2**. **Sound Fidelity** is measured by **non-intrusive PESQ via SQUIM-PESQ**, with range **–0.5…4.5**, and higher denotes more artifact-free clarity.

For **Content Accuracy**, the benchmark uses ASR-based transcript error rate. **WER** is used for English and **CER** for Chinese:
$$
\mathrm{WER} = \frac{S + I + D}{N_{\mathrm{text}}},
$$
with the same form for CER at character level. Lower values indicate better accuracy, and real speech is reported as often **~ 0.07**. **Prosodic Coherence** is evaluated by **LALM-based scoring (SpeechJudge)** on a **1…5** scale, with prompts tuned for long-form flow, rhythmic hierarchy, and native naturalness.

For expressiveness, **Score\_{richness}** segments audio into non-overlapping **10 s** chunks $\{c_i\}$, obtains per-chunk expressiveness scores $s_i$ via LALM, and averages them:
$$
\mathrm{Score}_{\mathrm{richness}} = \frac{1}{M}\sum_{i=1}^{M} s_i.
$$
**Score\_{hierarchy}** feeds the full utterance to the LALM and prompts for paragraph-level emotional arc, vocal dynamics, and scene fit. Both metrics range from **1…5**, with higher values indicating stronger sentence-level emotional or storytelling quality, and better dynamic variation over the whole narrative [2605.28618].

## 5. Empirical findings and diagnostic significance

Benchmarking results indicate that current long-form TTS systems are substantially stronger in acoustics than in semantics and expressiveness. For **Timbre Consistency**, open-source and closed-source model averages are reported as **≈ 0.93** versus **real ≈ 0.96**; the models are therefore close to real recordings, but still show slight drift, especially in dialogues. For **Reverb Consistency**, single-speaker open and closed systems are **≈ 1.95** versus **real = 1.91**, which is comparable, but dialogue settings are markedly worse: **≈ 3.45/3.36** versus **real = 2.73**. **Fidelity** is comparatively strong, and models often match or exceed web-crawled real data, with **fidelity ~ 3.6** [2605.28618].

The semantics results are less favorable. **Content Accuracy** shows **average WER/CER ~ 0.16** versus **real ~ 0.07**, and the authors note that modern ASR saturates quickly; autoregressive end-to-end models, with **SparkTTS** given as an example, degrade with length. **Prosodic Coherence** reaches **open ≈ 3.43**, **closed ≈ 3.79**, versus **real = 4.04**, indicating a systematic gap in paragraph-level prosody. A central implication of these numbers is that perceptual cleanliness and textual faithfulness diverge under long-context generation.

The expressiveness gap is larger still. **Expressive Richness** is reported as **open ≈ 3.03**, **closed ≈ 3.42**, versus **real = 4.35**. **Expressive Hierarchy** is **open ≈ 2.67**, **closed ≈ 3.01**, versus **real = 3.94**. The largest performance drops occur in the highest expressiveness settings—**sports, drama, talk-show**—which the paper interprets as evidence that current models lack richly dynamic training data. Architectural comparisons sharpen this diagnosis: **NAR models run faster (RTF ~ 0.2–0.3) but tend to be over-smoothed in expressiveness and hierarchy**, whereas **AR models capture prosody & expressiveness better but suffer error propagation in content fidelity at long lengths**. The reported trade-off suggests future **Coarse-to-Fine architectures** [2605.28618].

## 6. SwanVoice integration, metadata, and availability

Within the SwanVoice framework, SwanData-Speech functions as the data substrate for expressive long-form zero-shot synthesis in both monologue and dialogue. SwanVoice is described as a zero-shot TTS model for **1–4 speakers**, combining a **25 Hz VAE**, **raw-text conditioning with pause-aware symbols and pinyin substitution**, and a **flow-matching DiT with speaker-turn conditioning**. Training starts from monologue speech, moves through mixed and real dialogue data, and then applies **DiffusionNFT post-training with phone-level and speaker-similarity rewards**. The corpus therefore supplies the pause-aware and speaker-aware supervision needed for conversational continuity and expressive control [2605.30993].

Per-utterance metadata includes **text transcript (with refined punctuation, `<|sp|>` symbols, speaker-turn tags)**, **word-level timestamp alignment (start/end)**, **speaker ID / speaker-turn tag sequence**, **pause markers (`<|sp|>`, commas, periods)**, **ASR confidence scores and language ID**, **objective quality scores (DNSMOS, PESQ, STOI)**, and **emotion label + confidence (from emotion2vec+)**. Alignment accuracy is reported with **Accumulated Average Shift (AAS)**: **45.19 ms** on **GTSinger-Speech (ZH)**, **27.67 ms** on **LibriSpeech-Clean (EN)**, and **29.92 ms** on **LibriSpeech-Others**. Corpus-level filtering is described as ensuring that only high-fidelity speech passes, typically **≥ ∘MOS 3.0 in DNSMOS** and **PESQ > 2.5 in non-intrusive PESQ**, while **emotion2vec+** selects non-neutral expressive utterances with **≥ 0.8 confidence**.

Downstream results on SwanBench-Speech show that SwanVoice trained on SwanData-Speech achieves the **highest Richness (3.81) and Hierarchy (3.62)** among evaluated open-source baselines on the **Expressive Challenge subset (long-form monologues)**. On **multi-speaker dialogue**, it scores **3.62 (Richness) / 3.71 (Hierarchy)**, approximately **0.5 points above the best baselines**. At the same time, the abstract states that **content accuracy remains the main limitation**. Availability is asymmetric: the benchmark is released on **Hugging Face under CC BY-NC-SA 4.0**, with **full code + prompts on GitHub**, whereas the corpus itself is derived from internal resources plus mixed open-source subsets and is not announced as publicly downloadable. A common misunderstanding is therefore to treat SwanData-Speech as a single public dataset; the published record instead distinguishes between a public benchmark and a separately described, non-public training corpus [2605.28618] [2605.30993].

Source: https://www.emergentmind.com/topics/swandata-speech