SwanVoice: Expressive Long-Form Zero-Shot Speech Synthesis for Both Monologue and Dialogue

Abstract: Zero-shot text-to-speech (TTS) has improved substantially for single-speaker synthesis, yet expressive long-form multi-speaker dialogue remains difficult. A common workaround is to synthesize each turn with a monologue TTS model and stitch the outputs together. This adds inference cost and often breaks acoustic consistency, conversational coherence, and affective continuity across turns. Recent dialogue TTS systems have begun to address this setting, but they still struggle to keep expressive coherence, controllable speaker switching, and monologue quality at the same time. We present SwanData-Speech and SwanVoice. SwanData-Speech builds monologue and dialogue corpora from in-the-wild audio, using Swan Forced Aligner for pause-aware word-level alignment and RobustMegaTTS3 for pronunciation-hard cases. Built on these data, SwanVoice is a zero-shot TTS model for 1--4 speakers, combining a 25 Hz VAE, raw-text conditioning with pause-aware symbols and pinyin substitution, and a flow-matching DiT with speaker-turn conditioning. Training starts from monologue speech, moves through mixed and real dialogue data, and then uses DiffusionNFT post-training with phone-level and speaker-similarity rewards. On SwanBench-Speech, SwanVoice obtains higher richness and hierarchy scores than all evaluated open-source baselines in both monologue and dialogue settings, while content accuracy remains the main limitation. Audio demos are available at https://swanaigc.github.io//#swanvoice.

• The paper presents a unified zero-shot TTS model that synthesizes both monologue and dialogue while addressing acoustic discontinuities and turn control challenges.
• The paper leverages a state-of-the-art data pipeline including SwanData-Speech and pause-aware forced alignment to ensure expressive prosody and robust speaker segmentation.
• The paper introduces a non-autoregressive architecture with hierarchical prosody control that outperforms baselines in expressiveness, despite some limitations in content accuracy.

SwanVoice: Expressive Long-Form Zero-Shot Speech Synthesis for Both Monologue and Dialogue

Introduction and Motivation

"SwanVoice: Expressive Long-Form Zero-Shot Speech Synthesis for Both Monologue and Dialogue" (2605.30993) presents a unified model for zero-shot text-to-speech (TTS) generation in both monologue and dialogue contexts, addressing intrinsic limitations in current systems. Existing approaches synthesize multi-party conversational speech by stitching monologue segments, introducing discontinuities in acoustic consistency, conversational coherence, and affective continuity. Even recent dialogue TTS architectures demonstrate weaknesses in turn control, monologue performance erosion, and expressive coherence. SwanVoice directly models full-context conversational audio with robust speaker switching, expressive modeling, and hierarchical prosody control.

Hierarchical Data Processing and Alignment

The creation of SwanVoice hinges upon SwanData-Speech, a pipeline that transforms in-the-wild recordings (podcasts, radio dramas, film/TV) into high-quality monologue and dialogue speech corpora. The pipeline employs advanced speech enhancement, speaker diarization, ASR transcription, and quality/emotion filtering, resulting in rigorous segmentation and expressive data selection.

Figure 1: The hierarchical data processing pipeline, including enhancement, segmentation, diarization, and selection of monologue versus dialogue pools.

To address potential prosody failures arising from inaccurate pause annotation, Swan Forced Aligner is employed for pause-aware, word-level temporal alignment. The aligner operates over an explicit interleaved topology, incorporating blank, word, and transition states with global and local calibration, and it achieves competitive timestamp accuracy in both Chinese and English contexts.

Figure 2: Overview of Swan Forced Aligner, highlighting structured topology for robust pause-aware alignment.

Additionally, a synthetic subset, RobustMegaTTS3, is constructed by leveraging dictionary-level, polyphonic, and code-switching hard cases, which are rendered by the MegaTTS3 phoneme-based synthesizer. This ensures pronunciation robustness and coverage of rare linguistic phenomena, critical for long-form modeling.

Model Architecture and Training Curriculum

SwanVoice integrates a multicomponent architecture: a 25 Hz VAE-based codec for length reduction, a BPE-based tokenizer with pinyin and pause augmentation for robust Chinese and expressive modeling, and a flow-matching DiT with speaker-turn conditioning. The model avoids the generative limitations of autoregressive (AR) architectures, instead leveraging non-autoregressive (NAR) full-context conditioning to mitigate exposure bias and decoding latency.

Figure 3: Overall training and inference procedure for SwanVoice, illustrating reference-driven synthesis with speaker and textual guidance.

Speaker-turn annotation is handled through paired label sequences and explicit turn tokens, enabling fine-grained speaker discrimination even across acoustically similar voices. The flow-based DiT pads text and speaker embeddings to the latent waveform resolution, employing a Transformer stack for pre-feature interaction. RMSNorm and AdaLN-based adapters stabilize style and timbre preservation.

Training proceeds through a curriculum: initial monologue pretraining (approx. 2M hours), injection of hard cases, followed by mixed conversational training with concatenated turns, and finally SFT with real conversational data. This curriculum prevents monologue performance erosion prevalent in models directly fine-tuned on dialogue.

Post-training utilizes DiffusionNFT, an online RL strategy targeting phone-level WER and speaker similarity rewards, enforcing pronunciation and timbre consistency without reference-path drift. This is critical for robust deployment in uncontrolled conversational environments.

Technical Evaluation and Results

Evaluations are conducted on SwanBench-Speech, encompassing acoustics, semantics, and expressiveness. SwanVoice demonstrates high sound fidelity (3.60), prosodic coherence (3.56), and timbre consistency (0.93) in monologue generation—each at or above open-source averages. However, content error rate remains suboptimal compared to select baselines.

Expressive scores stand out: SwanVoice achieves 3.81 (richness) and 3.62 (hierarchy) in monologue, outperforming VibeVoice—the strongest baseline—by 0.39 and 0.56 points. In dialogue synthesis, SwanVoice attains 3.62/3.71 on richness/hierarchy, surpassing all evaluated systems. Gains are most pronounced in multi-speaker settings, where the model’s scene and affect continuity remain stable across turn boundaries.

Implications, Limitations, and Future Directions

SwanVoice’s architectural and data innovations advance zero-shot long-form speech synthesis by improving expressive and hierarchical modeling, speaker consistency, and robustness to real-world linguistic edge cases. Practically, this broadens applicability for automated drama, podcast, and conversational agents. Theoretically, the results motivate further exploration of NAR latent diffusion models constrained by structured alignment and reward-driven adaptation.

The model retains several limitations, including lower content accuracy than select baselines and imperfect speaker-turn discrimination under challenging prompts. These deficiencies imply opportunities in enhanced pronunciation control, refined pause modeling, and more robust speaker-turn inference. Further directions include tighter integration of multimodal cues, increased linguistic diversity, and reinforcement of content faithfulness.

Conclusion

SwanVoice models conversational speech as a holistic generation task, attaining superior expressive scores in hierarchical settings for both monologue and dialogue. Its structured data pipeline, alignment methods, and post-training rewards contribute to superior affective continuity, timbre stability, and expressiveness. Despite current limitations in content accuracy and turn separation, the framework sets a new baseline for zero-shot TTS modeling in complex, multi-speaker scenarios and directs future research toward improved reliability, multimodality, and fine-grained control.