Comprehensive Benchmarking of Long-Form Speech Generation in Diverse Scenarios

Abstract: Recent advances in speech generation have enabled high-fidelity synthesis, yet systematic evaluation of models under long-context conditions remains largely underexplored. A comprehensive evaluation benchmark for long-form speech is indispensable for two reasons: 1) existing test scenarios are often confined to limited domains, creating a significant gap with the diverse downstream applications; 2) existing metrics overlook critical long-text factors such as consistency and coherence, failing to generalize reliably. To this end, we propose Swanbench-Speech, a comprehensive benchmark that decomposes long-form speech quality into specific, disentangled dimensions. SwanBench-Speech has three key properties. 1) Rich speech scenarios: Focusing on long-form speech generation and dialog generation, SwanBench-Speech covers acoustics, semantics, and expressiveness challenges, and consists of 1,101 samples spanning 17 common speech scenarios; 2) Comprehensive evaluation dimensions: Along the acoustics, semantics, and expressiveness axes, SwanBench-Speech defines an automated evaluation protocol with seven metrics to provide a comprehensive, accurate, and standardized assessment; 3) Valuable Insights: Through extensive experiments, we reveal that current models still struggle in highly expressive scenarios and exhibit a notable gap in consistency and hierarchy compared to real recordings.

• The paper introduces SwanBench-Speech, a benchmark addressing long-context consistency, prosody, and expressive structure in diverse TTS scenarios.
• It employs a seven-metric hierarchy—including timbre, reverb, sound fidelity, and expressive hierarchy—to evaluate over 20 models across multiple scenarios.
• Empirical results reveal that while some open-source models match commercial systems in fidelity, they lag in prosody and expressiveness, especially with longer sequences.

Comprehensive Benchmarking of Long-Form Speech Generation: The SwanBench-Speech Framework

Motivation and Context

Recent progress in TTS and spoken dialogue modeling has dramatically improved short-form speech synthesis, but systematic, multi-dimensional evaluation of long-form and dialog generation remains underdeveloped. Existing benchmarks are typically restricted to sentence-level and single-speaker scenarios, while automated evaluation metrics fail to comprehensively address long-context consistency, prosody, and expressive structure. To bridge this evaluation gap, the paper introduces SwanBench-Speech, a rigorously curated benchmark and evaluation protocol targeting the high-fidelity, semantically faithful, and expressively rich synthesis of long-form speech and dialogues across diverse, real-world scenarios.

Benchmark Construction and Coverage

The SwanBench-Speech framework is built upon three high-level challenges in long-form TTS: Acoustics, Semantics, and Expressiveness, operationalized via coverage of 17 distinct downstream scenarios—including audiobooks, podcasts, customer service, live streams, drama, presentations, and news broadcasting. Dataset design involves four tightly integrated phases: scenario selection, hybrid data collection (combining text corpora, web audio, and LLM-generated scripts), algorithmic data refinement (semantic deduplication, LLM-aided quality checks, privacy/ethical filtering), and expert manual review.

Figure 1: Overview of dataset construction and refinement for SwanBench-Speech: scenario selection, hybrid data sourcing, refinement, and manual validation.

The dataset comprises 1,101 test cases, dynamically balanced across language (Chinese and English), speaker configuration (single to multi-speaker), challenge dimension, topic, and scenario distribution.

Figure 2: Categorical statistics of SwanBench-Speech, reflecting balance in language, speaker number, core challenge category, thematic topic, and scenario.

Text length is tightly controlled to match minute-level content prevalent in downstream applications, with length distributions paralleling those found in real-world Chinese and English materials.

Figure 3: Text length distribution within SwanBench-Speech, with means denoted for each language.

Evaluation Protocol: Dimensions and Metrics

SwanBench-Speech introduces a seven-metric hierarchy to fully disentangle the composite aspects of long-form speech quality:

• Timbre Consistency: Mean cosine similarity between segmental speaker embeddings.
• Reverb Consistency: Variance of sliding-window SRMR scores, penalizing acoustic field drift.
• Sound Fidelity: Non-intrusive SQUIM-PESQ for broad-scope perceptual clarity.
• Content Accuracy: ASR-based WER/CER relative to the transcript.
• Prosodic Coherence: Scalar output from a fine-tuned LALM (SpeechJudge), focused on intra/inter-sentential rhythm and flow.
• Expressive Richness: Local (10s windowed) LALM evaluation for emotional/layered performance.
• Expressive Hierarchy: Global LALM-driven score for long-term expressive structure, dynamic range, and engagement trajectory.

All LALM-based metrics, including those for prosody and expressiveness, are validated for human alignment with SRCC values up to 0.82, demonstrating high perceptual correlation across expert subjects.

Empirical Insights: Multi-Dimensional Comparative Evaluation

Comprehensive benchmarking is performed on over 20 models, spanning leading open-source systems (e.g., CosyVoice, VibeVoice, ZipVoice, GLM-TTS, MegaTTS) and closed-source APIs (OpenAI TTS, Gemini, ElevenLabs, SeedTTS, Minimax), covering both single- and multi-speaker as well as dialogue generation. Evaluations include analyses of inference speed (RTF), robustness to input reference diversity, and scaling behavior over increasing sequence length.

Key findings:

• Certain SOTA open-source models achieve parity with commercial systems in fidelity and content accuracy, but a systematic gap remains in prosody, expressiveness, and acoustic field consistency.
• Closed-source models, especially Gemini-2.5-pro and Minimax-Speech-02-hd, outperform open-source systems in global expressiveness, prosodic coherence, and multi-speaker dialogue quality.
• As sequence length increases, most models exhibit notable degradation in reverb consistency, expressive hierarchy, and prosody, highlighting limitations in long-term dependency capture.

  Figure 4: LFS-Bench (SwanBench-Speech) results across Acoustics, Semantics, and Expressiveness, with normalization for visible cross-challenge comparison.

  

  Figure 5: Performance trend as a function of sequence length for key models: content accuracy, prosody, and reverb consistency all degrade as sequence length increases.

  

  Figure 6: Direct E2E vs. chunked synthesis comparison for sequence length effects; end-to-end models suffer more pronounced prosody and accuracy decay with longer sequences.

Scenario and Model Architecture Analyses

Radial and per-challenge decompositions further reveal that models degrade most severely under high-expressiveness scenarios (drama, sportscast, talkshow). In dialog settings, maintaining global acoustic consistency and correct speaker transitions remains difficult, with reverb consistency and expressive hierarchy particularly challenging.

Figure 7: Closed-source single-speaker model performance visualized via radar chart for all challenge-aligned scenarios; normalized values for direct metric comparability.

The dichotomy between autoregressive (AR) and non-autoregressive (NAR) architectures is also pronounced:

• NAR models are robust and fast for long input but tend toward oversmoothed prosody with limited expressive control.
• AR models excel in localized expressiveness and prosody but exhibit error propagation and rapid performance decay over longer contexts.

The paper advocates for a coarse-to-fine architecture paradigm, moving beyond the strict AR/NAR divide to achieve both global context modeling and robust local structure.

Implications, Limitations, and Future Directions

Practically, SwanBench-Speech establishes a rigorous, reproducible testbed for long-form TTS and spoken dialogue research, directly enabling advances in high-immersion applications (audiobooks, customer support, live streaming). The clear multidimensional gap between current synthesis and real data underscores pressing research needs: improved long-term sequence modeling, better training data curation with greater temporal continuity and high expressiveness, and broader coverage of language, demographic, and paralinguistic variation.

Theoretical implications center on the limits of contemporary generative TTS under global structural constraints and the inadequacy of naive scaling alone, pointing to new directions in curriculum learning, hierarchical modeling, and human-aligned evaluation.

Notable limitations include: dependence on closed-source LALMs for perceptual metric alignment, language coverage restricted to Chinese/English, and a non-exhaustive range of voice prompts and demographics. The authors call for community contributions, open-source evaluator distillation, and expansion to low-resource and culturally diverse settings.

Conclusion

SwanBench-Speech delivers the first widely accessible, multi-scenario, challenge-disentangled benchmark for long-form TTS and dialogue evaluation, grounded in metric-human alignment and explicit scenario/task diversity. Results expose both the progress of modern systems in fidelity and accuracy and the persistent deficiencies in prosody, expressiveness, and long-range structure. The benchmark is poised to accelerate robust, expressive, and contextually coherent speech generation, with broad implications for both industry deployments and foundational research in generative spoken AI.

Reference: "Comprehensive Benchmarking of Long-Form Speech Generation in Diverse Scenarios" (2605.28618)