SoulX-Podcast: Towards Realistic Long-form Podcasts with Dialectal and Paralinguistic Diversity (2510.23541v2)

Abstract: Recent advances in text-to-speech (TTS) synthesis have significantly improved speech expressiveness and naturalness. However, most existing systems are tailored for single-speaker synthesis and fall short in generating coherent multi-speaker conversational speech. This technical report presents SoulX-Podcast, a system designed for podcast-style multi-turn, multi-speaker dialogic speech generation, while also achieving state-of-the-art performance in conventional TTS tasks. To meet the higher naturalness demands of multi-turn spoken dialogue, SoulX-Podcast integrates a range of paralinguistic controls and supports both Mandarin and English, as well as several Chinese dialects, including Sichuanese, Henanese, and Cantonese, enabling more personalized podcast-style speech generation. Experimental results demonstrate that SoulX-Podcast can continuously produce over 90 minutes of conversation with stable speaker timbre and smooth speaker transitions. Moreover, speakers exhibit contextually adaptive prosody, reflecting natural rhythm and intonation changes as dialogues progress. Across multiple evaluation metrics, SoulX-Podcast achieves state-of-the-art performance in both monologue TTS and multi-turn conversational speech synthesis.

• The paper introduces a unified LLM-driven framework that synthesizes long-form, multi-speaker podcasts with diverse dialects and expressive paralinguistic cues.
• It leverages advanced data processing pipelines, dual-ASR transcription, and curriculum learning to ensure speaker timbre stability and coherent dialogue generation.
• Empirical results demonstrate state-of-the-art performance in both monologue and multi-turn dialogue synthesis, achieving high speaker similarity and intelligibility.

SoulX-Podcast: A Unified Framework for Realistic Long-form Podcast Synthesis with Dialectal and Paralinguistic Diversity

Introduction

SoulX-Podcast presents a comprehensive framework for long-form, multi-speaker, and multi-dialect conversational speech synthesis, addressing the limitations of prior TTS systems that predominantly focus on single-speaker monologue generation. The system leverages a LLM backbone to enable seamless multi-turn dialogue generation, robust paralinguistic control, and support for Mandarin, English, and several Chinese dialects. The architecture is designed to maintain speaker timbre stability, smooth transitions, and contextually adaptive prosody over extended durations, with empirical results demonstrating over 90 minutes of continuous, coherent conversation synthesis.

Figure 1: Overview of SoulX-Podcast, illustrating the interleaved text-speech modeling paradigm and multi-speaker, multi-dialect support.

Data Processing and Annotation Pipeline

SoulX-Podcast's data pipeline is engineered for in-the-wild dialogue speech, incorporating advanced pre-processing, segmentation, diarization, and annotation strategies. The pipeline applies UVR-MDX-based vocal separation for denoising, VAD for utterance segmentation, and Sortformer-based diarization for reliable speaker-turn annotation. Dual-ASR transcription (Paraformer and Whisper for Chinese; Parakeet and Whisper for English) ensures transcript reliability, with CER/WER-based filtering to maximize data quality. Speaker purity is refined via WavLM-large embeddings and clustering, mitigating label inconsistencies.

Paralinguistic annotation employs a two-stage process: high-throughput event detection using fine-tuned ASR models (Beats for Mandarin, Whisperd for English), followed by Gemini-2.5-Pro model-assisted verification for precise, time-aligned labeling. Dialectal annotation combines public data collection and dialect identification models, with Seed-ASR for robust transcription. The resulting corpus comprises approximately 0.3M hours of conversational speech and 1.0M hours of monologue data, totaling 1.3M hours for training.

Figure 2: Processing pipeline for in-the-wild dialogue speech data, detailing enhancement, segmentation, diarization, transcription, and annotation steps.

Model Architecture and Training

SoulX-Podcast adopts a two-stage generative framework, extending the CosyVoice paradigm. The LLM backbone (Qwen3-1.7B) predicts semantic tokens, which are converted to acoustic features via flow matching and synthesized into waveforms by a vocoder. The text codebook is augmented to encode speech tokens, paralinguistic, and dialectal attributes. Token organization follows an interleaved sequence: each speaker's text tokens are followed by corresponding speech tokens, with explicit speaker and dialect tokens for control. Paralinguistic cues are embedded as textual tokens at precise positions.

Training utilizes curriculum learning: initial pre-training on mixed monologue and dialogue data, followed by fine-tuning on multi-speaker, multi-dialect dialogue data. Dialectal capability is further enhanced by targeted fine-tuning on dialectal subsets. Context regularization is introduced to improve long-form synthesis stability, progressively dropping historical speech tokens while retaining textual context, thus promoting semantic continuity over acoustic memory.

Inference and Dialect-Guided Prompting

Inference mirrors the interleaved token organization, enabling autoregressive generation of multi-speaker, multi-dialect conversational speech. Cross-dialectal voice cloning is achieved via the Dialect-Guided Prompting (DGP) strategy: a dialect-typical sentence is prepended to the input, amplifying the dialectal control signal and enabling a Mandarin prompt to generate speech in any supported dialect. This approach addresses the challenge posed by shared orthography among Chinese dialects, ensuring robust dialectal style transfer.

Figure 3: Inference procedure of SoulX-Podcast, demonstrating cross-dialectal prompting and Dialect-Guided Prompting (DGP) for dialectal style control.

Empirical Evaluation

Monologue TTS Performance

SoulX-Podcast achieves state-of-the-art results in zero-shot monologue TTS, outperforming leading models on Seed-TTS-eval. It attains the lowest CER for Chinese and competitive WER for English, with speaker similarity metrics (SIM) closely trailing the best-performing systems. These results confirm the model's generalization capability beyond dialogue synthesis.

Multi-turn Podcast Generation

On the ZipVoice-Dia benchmark, SoulX-Podcast demonstrates superior performance in multi-turn, multi-speaker dialogue synthesis, achieving the lowest CER/WER and highest cross-speaker similarity (cpSIM) for both Chinese and English. UTMOS scores indicate competitive perceptual quality, with the model maintaining speaker coherence and naturalness over extended conversations.

Figure 4: Performance of SoulX-Podcast, highlighting its superiority in intelligibility, speaker similarity, and cross-speaker consistency across TTS and podcast benchmarks.

Paralinguistic Control

SoulX-Podcast exhibits strong controllability over paralinguistic events, with an average recognition accuracy of 0.82 across five event types. Laughter is detected with perfect accuracy, while sighs and throat clearing achieve high fidelity. Lower accuracy for breathing and coughing reflects the inherent acoustic ambiguity of these events and potential limitations of the automated evaluator.

Dialectal Synthesis

The model supports Sichuanese, Henanese, and Cantonese, achieving consistent speaker similarity across dialects. CER values are higher than for Mandarin, likely due to ASR limitations, but cpSIM remains robust, indicating effective dialectal style transfer and speaker identity preservation in both monologue and dialogue settings.

Implications and Future Directions

SoulX-Podcast establishes a unified framework for expressive, long-form podcast synthesis, integrating multi-speaker, multi-dialect, and paralinguistic control within a scalable LLM-driven architecture. The interleaved text-speech modeling paradigm and curriculum learning strategy enable robust generalization across diverse speech generation scenarios. The DGP method for cross-dialectal prompting addresses a critical challenge in dialectal synthesis, facilitating zero-shot style transfer even with shared orthography.

Practically, SoulX-Podcast can be deployed for automated podcast production, conversational agents, and personalized voice interfaces, with fine-grained control over speaker, dialect, and expressive cues. The large-scale, annotated corpus and modular pipeline provide a foundation for further research in dialogue speech synthesis, paralinguistic modeling, and dialectal TTS.

Theoretically, the work highlights the importance of token organization, context regularization, and curriculum learning in scaling LLM-based speech synthesis to long-form, multi-speaker scenarios. Future research may explore multilingual extension, more granular paralinguistic control, and improved dialectal ASR for evaluation. Ethical considerations, including watermarking and misuse detection, remain paramount for responsible deployment.

Conclusion

SoulX-Podcast advances the state of conversational speech synthesis by integrating multi-speaker, multi-dialect, and paralinguistic diversity within a unified LLM-driven framework. Its empirical performance across monologue and dialogue tasks, robust dialectal and expressive control, and scalable architecture position it as a versatile solution for realistic podcast generation and beyond. The methodological innovations and annotated corpus set a new standard for future research in expressive, context-aware speech synthesis.