SwanData-Speech: TTS Benchmark & Corpus

• SwanData-Speech is a comprehensive ecosystem for long-form TTS, uniting a public benchmark (SwanBench-Speech) and a large-scale in-the-wild speech corpus used for training SwanVoice.
• The benchmark offers 1,101 test samples across 17 diverse scenarios, balancing Mandarin and English with settings for both single-speaker and multi-speaker dialogues.
• The corpus comprises approximately 2.59 million hours of audio with advanced annotations like pause-aware alignment and speaker-turn labels, supporting expressive and zero-shot TTS synthesis.

SwanData-Speech is the name used in recent long-form text-to-speech literature for two closely related resources: a benchmark and test set called SwanBench-Speech, and a large in-the-wild speech mining pipeline and corpus used to train SwanVoice. In the benchmark formulation, it targets long-form single-speaker synthesis and multi-speaker dialogue generation, contains 1,101 test samples spanning 17 common speech scenarios, is split between 49.3 % Mandarin Chinese and 50.7 % English, and includes 1–4 speakers per sample, with 101 cases involving 3–4 speakers. In the corpus formulation, it is built from approx. 2.59 million hours of audio and supplies pause-aware word-level alignments, speaker-turn annotations, synthetic pronunciation-hard cases, quality and emotion filtering, and 24 kHz training data for zero-shot TTS (Pan et al., 27 May 2026, Li et al., 29 May 2026).

1. Nomenclature and research scope

The literature uses the label in two senses. In "Comprehensive Benchmarking of Long-Form Speech Generation in Diverse Scenarios" (Pan et al., 27 May 2026), the benchmark is explicitly named SwanBench-Speech (aka SwanData-Speech) and is presented as a comprehensive evaluation benchmark for long-form speech. In "SwanVoice: Expressive Long-Form Zero-Shot Speech Synthesis for Both Monologue and Dialogue" (Li et al., 29 May 2026), SwanData-Speech denotes a large in-the-wild speech mining pipeline that produces two clean TTS training subsets: a monologue subset of single-speaker utterances and a dialogue subset of 2–4-speaker conversations, with speaker-turn labels and expressive variation.

The benchmark version focuses on diagnostic evaluation. It covers acoustics, semantics, and expressiveness challenges, and is organized around long-form speech generation and dialog generation. The corpus version focuses on data acquisition and supervision for model training. Its primary purpose is to feed SwanVoice, an end-to-end zero-shot text-to-speech model that must preserve both monologue quality and long-form, multi-speaker conversational coherence and expressiveness.

The distinction is operationally important. The benchmark is described as a released test set with code and prompts, whereas the corpus is drawn from internal resources plus mixed open-source subsets, and the paper does not announce a public download or license for SwanData-Speech itself. This suggests that the name identifies an ecosystem rather than a single uniformly distributed dataset (Li et al., 29 May 2026).

2. Corpus construction, segmentation, and annotation

The corpus pipeline begins from in-the-wild sources: internal ByteDance resources plus selected open-source Chinese and English corpora, including podcasts, radio dramas, films, TV, livestreams, and audiobooks. The raw collection is approx. 2.59 million hours of audio, with ~2.24 million hours Chinese and ~0.35 million hours English. The monologue pool used for pretraining contains ~2 million hours of single-speaker speech. The dialogue pool is formed from speaker-diarized multi-speaker segments merged to contain 2–4 speakers, up to 120 s long, with no silence gap > 2 s. Speaker demographics such as gender, age, and accents are not specified in the paper, and the number of unique speakers is not reported (Li et al., 29 May 2026).

Filtering and preprocessing are speaker-aware. The pipeline applies ASR language identification to retain only Chinese and English, isolates the vocal component via a vocal-separation tool, and performs speaker diarization using the 3D-Speaker toolkit + CAM++ + spectral clustering. Single-speaker segments are constrained to ≥ 0.1 s and ≤ 60 s. Adjacent same-speaker segments are merged if the gap is ≤ 2 s. Quality filtering uses DNSMOS (non-intrusive), non-intrusive PESQ, non-intrusive STOI, and emotion filtering uses emotion2vec+ confidence to extract a high-expressiveness subset.

Transcription and alignment are pause-aware. ASR uses SenseVoice-Small (no ITN), followed by a punctuation restoration model. In dialogue, each speaker turn is wrapped in <S{id}> … </S{id}> tokens. Word-level timestamps and explicit blank states are then produced by Swan Forced Aligner. Acoustic pauses are converted into symbols by gap duration between word-anchor timestamps: gap < 0.08 s → ignored, 0.08 s ≤ gap < 0.18 s → insert <|sp|>, 0.18 s ≤ gap < 0.45 s → insert comma ,, and gap ≥ 0.45 s → insert period/exclamation/question (default .). If transcript punctuation and acoustic pause disagree, punctuation is removed or inserted to match the audio. The reported path-score formula is

$$\mathrm{Score}(z) = \sum_t u^*_{t,s} + \sum_t \tau^*(z_t, r_t).$$

The pipeline also adds explicit pronunciation supervision. A synthetic “hard” subset is generated with RobustMegaTTS3 to cover rare dictionary words, polyphonic characters, tone sandhi, erhua, homographs, stress patterns, and code-switching. The specification includes 1 entry per word from GCIDE 0.54 + Table of General Standard Chinese Characters → 5 sentences each → ~full dictionary coverage, plus 20 K Chinese “hard” pronunciations + 20 K English hard cases + 100 K Chinese–English code-switch examples. The tokenizer combines BPE (CosyVoice) + dedicated <|sp|> token + 1,549 pinyin syllables in vocabulary; some Chinese characters are randomly replaced with pinyin during training, and pinyin hints can be provided at inference to disambiguate polyphonic characters (Li et al., 29 May 2026).

3. Scenario taxonomy and benchmark design

The benchmark decomposes long-form evaluation into 17 common speech scenarios distributed across three challenge families. It contains ~380 samples for acoustics (~34.5 % of data), ~310 for semantics (~28.1 %), and ~411 for expressiveness (~37.4 %) (Pan et al., 27 May 2026).

Axis	Scenario count	Scenarios
Acoustics	6	Customer Service; Audiobook; Podcast; Chat; Debate; Interview
Semantics	5	News; Popular Science; Lesson; Seminar; Presentation
Expressiveness	6	Sportcast; Live Streaming; Speech (motivational); Host; Talk Show; Drama

The scenario descriptions are deliberately tied to long-context failure modes. Customer Service involves long product or policy explanations by a single speaker and requires high fidelity and zero artifacts. Audiobook uses single-speaker narration of 10 min+ and demands timbre and environmental consistency for immersion. Podcast requires strict speaker-switch fidelity and stable reverberation. Chat / Debate / Interview covers real-world multi-speaker settings with frequent turn-taking and varied acoustic backgrounds.

The semantics scenarios are constructed to stress textual faithfulness and paragraph-scale prosody. News & Popular Science are fact-dense and terminology-heavy, and test content accuracy under long context. Lesson, Seminar, Presentation are monologues with prosodic variety, including pauses and emphasis, and test paragraph-level prosody coherence. The expressiveness scenarios probe sentence-level and paragraph-level affective control. Sportcast & Live Streaming require sustained expressive richness, while Speech (motivational), Host, Talk Show, Drama require both sentence-level richness and paragraph-level dynamic hierarchy. This organization makes the benchmark explicitly scenario-wise rather than domain-agnostic (Pan et al., 27 May 2026).

4. Three-axis evaluation protocol

SwanBench-Speech defines long-form speech quality along three orthogonal axes: Acoustics, Semantics, and Expressiveness. Acoustics comprises Timbre Consistency, Reverb Consistency, and Sound Fidelity. Semantics comprises Content Accuracy and Prosodic Coherence. Expressiveness comprises Expressive Richness and Expressive Hierarchy (Pan et al., 27 May 2026).

For Timbre Consistency, speaker embeddings are extracted by sliding 3 s windows with 2 s stride. For a single-speaker sample $w$, with embeddings $e_1 \ldots e_n$, pairwise cosine similarities are computed as

$$\mathrm{sim}_{ij} = \cos\!\left(\frac{e_i}{\lVert e_i \rVert}, \frac{e_j}{\lVert e_j \rVert}\right), \quad i \ne j,$$

and

$$\mathrm{Score}_{\mathrm{timbre}}(w) = \frac{1}{n(n-1)} \sum_{i \ne j} \mathrm{sim}_{ij}.$$

For multi-speaker samples, the benchmark averages single-speaker scores per diarized stream. The range is [–1, 1], though in practice [0.8, 1.0], and higher indicates better identity stability. For Reverb Consistency, the metric uses SRMR(windows; 3 s/2 s) after VAD-based non-speech filtering:

$$\mathrm{Score}_{\mathrm{reverb}}(w) = \sqrt{\frac{1}{n}\sum_{i=1}^{n}(r_i-\mu)^2},$$

where $\mu=\mathrm{mean}(r_i)$. Lower is better, with ideal value 0 and typical long-form targets < 2. Sound Fidelity is measured by non-intrusive PESQ via SQUIM-PESQ, with range –0.5…4.5, and higher denotes more artifact-free clarity.

For Content Accuracy, the benchmark uses ASR-based transcript error rate. WER is used for English and CER for Chinese:

$$\mathrm{WER} = \frac{S + I + D}{N_{\mathrm{text}}},$$

with the same form for CER at character level. Lower values indicate better accuracy, and real speech is reported as often ~ 0.07. Prosodic Coherence is evaluated by LALM-based scoring (SpeechJudge) on a 1…5 scale, with prompts tuned for long-form flow, rhythmic hierarchy, and native naturalness.

For expressiveness, Score_{richness} segments audio into non-overlapping 10 s chunks $\{c_i\}$, obtains per-chunk expressiveness scores $s_i$ via LALM, and averages them:

$w$0

Score_{hierarchy} feeds the full utterance to the LALM and prompts for paragraph-level emotional arc, vocal dynamics, and scene fit. Both metrics range from 1…5, with higher values indicating stronger sentence-level emotional or storytelling quality, and better dynamic variation over the whole narrative (Pan et al., 27 May 2026).

5. Empirical findings and diagnostic significance

Benchmarking results indicate that current long-form TTS systems are substantially stronger in acoustics than in semantics and expressiveness. For Timbre Consistency, open-source and closed-source model averages are reported as ≈ 0.93 versus real ≈ 0.96; the models are therefore close to real recordings, but still show slight drift, especially in dialogues. For Reverb Consistency, single-speaker open and closed systems are ≈ 1.95 versus real = 1.91, which is comparable, but dialogue settings are markedly worse: ≈ 3.45/3.36 versus real = 2.73. Fidelity is comparatively strong, and models often match or exceed web-crawled real data, with fidelity ~ 3.6 (Pan et al., 27 May 2026).

The semantics results are less favorable. Content Accuracy shows average WER/CER ~ 0.16 versus real ~ 0.07, and the authors note that modern ASR saturates quickly; autoregressive end-to-end models, with SparkTTS given as an example, degrade with length. Prosodic Coherence reaches open ≈ 3.43, closed ≈ 3.79, versus real = 4.04, indicating a systematic gap in paragraph-level prosody. A central implication of these numbers is that perceptual cleanliness and textual faithfulness diverge under long-context generation.

The expressiveness gap is larger still. Expressive Richness is reported as open ≈ 3.03, closed ≈ 3.42, versus real = 4.35. Expressive Hierarchy is open ≈ 2.67, closed ≈ 3.01, versus real = 3.94. The largest performance drops occur in the highest expressiveness settings—sports, drama, talk-show—which the paper interprets as evidence that current models lack richly dynamic training data. Architectural comparisons sharpen this diagnosis: NAR models run faster (RTF ~ 0.2–0.3) but tend to be over-smoothed in expressiveness and hierarchy, whereas AR models capture prosody & expressiveness better but suffer error propagation in content fidelity at long lengths. The reported trade-off suggests future Coarse-to-Fine architectures (Pan et al., 27 May 2026).

6. SwanVoice integration, metadata, and availability

Within the SwanVoice framework, SwanData-Speech functions as the data substrate for expressive long-form zero-shot synthesis in both monologue and dialogue. SwanVoice is described as a zero-shot TTS model for 1–4 speakers, combining a 25 Hz VAE, raw-text conditioning with pause-aware symbols and pinyin substitution, and a flow-matching DiT with speaker-turn conditioning. Training starts from monologue speech, moves through mixed and real dialogue data, and then applies DiffusionNFT post-training with phone-level and speaker-similarity rewards. The corpus therefore supplies the pause-aware and speaker-aware supervision needed for conversational continuity and expressive control (Li et al., 29 May 2026).

Per-utterance metadata includes text transcript (with refined punctuation, <|sp|> symbols, speaker-turn tags), word-level timestamp alignment (start/end), speaker ID / speaker-turn tag sequence, pause markers (<|sp|>, commas, periods), ASR confidence scores and language ID, objective quality scores (DNSMOS, PESQ, STOI), and emotion label + confidence (from emotion2vec+). Alignment accuracy is reported with Accumulated Average Shift (AAS): 45.19 ms on GTSinger-Speech (ZH), 27.67 ms on LibriSpeech-Clean (EN), and 29.92 ms on LibriSpeech-Others. Corpus-level filtering is described as ensuring that only high-fidelity speech passes, typically ≥ ∘MOS 3.0 in DNSMOS and PESQ > 2.5 in non-intrusive PESQ, while emotion2vec+ selects non-neutral expressive utterances with ≥ 0.8 confidence.

Downstream results on SwanBench-Speech show that SwanVoice trained on SwanData-Speech achieves the highest Richness (3.81) and Hierarchy (3.62) among evaluated open-source baselines on the Expressive Challenge subset (long-form monologues). On multi-speaker dialogue, it scores 3.62 (Richness) / 3.71 (Hierarchy), approximately 0.5 points above the best baselines. At the same time, the abstract states that content accuracy remains the main limitation. Availability is asymmetric: the benchmark is released on Hugging Face under CC BY-NC-SA 4.0, with full code + prompts on GitHub, whereas the corpus itself is derived from internal resources plus mixed open-source subsets and is not announced as publicly downloadable. A common misunderstanding is therefore to treat SwanData-Speech as a single public dataset; the published record instead distinguishes between a public benchmark and a separately described, non-public training corpus (Pan et al., 27 May 2026, Li et al., 29 May 2026).