SoulX-Podcast: Neural Podcast Tech Stack

• SoulX-Podcast is a neural podcast technology stack that integrates advanced synthesis, transcription, recommendation, and previewing modules in a modular pipeline.
• The system employs a two-stage generative backbone with explicit paralinguistic and multi-dialect control to achieve natural, long-form dialogue generation.
• Integrated with SoulX-Transcriber and music-driven recommenders, it demonstrates significant empirical improvements in accuracy, engagement, and scalability over legacy systems.

SoulX-Podcast is a comprehensive neural podcast technology stack designed for state-of-the-art podcast-style speech generation, robust multi-speaker transcription, cross-domain recommendation, and LLM-powered content previewing. Developed to address the demands of high naturalness, multi-dialect diversity, scalability, and integration with music-driven recommender systems, SoulX-Podcast and its associated modules represent significant advancements over prior single-domain and single-speaker solutions in both algorithmic architecture and empirical outcomes (Xie et al., 27 Oct 2025).

1. System Architecture and Workflow

SoulX-Podcast employs a CosyVoice-style, two-stage generative backbone for long-form, multi-speaker, and multi-dialect podcast synthesis (Xie et al., 27 Oct 2025). The modular pipeline consists of the following stages:

• Data Processing and Curation:
  • Speech enhancement via UVR-MDX and amplitude normalization.
  • VAD segmentation followed by diarization and speaker assignment (SortFormer-based).
  • Quality filtering: SNR, DNSMOS, lexical/phonetic thresholds.
  • Dual-ASR transcription, with joint filtering on CER/WER thresholds.
  • Speaker-purity refinement via WavLM-large embeddings and clustering.
  • Paralinguistic and dialect annotation.
• Generative Modeling:
  • Stage 1: Autoregressive LLM (Qwen3-1.7B) predicts an interleaved sequence of [<SPEAKER><DIALECT><TextTokens><AudioTokens>], with paralinguistic tokens (e.g., 〈|laughter|〉, 〈|sigh|〉) inserted into the text stream.
  • Stage 2: Flow-matching decoder converts discrete semantic/acoustic tokens into continuous acoustic features (mel-spectrograms).
  • Stage 3: Neural vocoder (HiFiGAN-type) reconstructs the final waveform.
• Loss Functions and Training:
  • Cross-entropy over the extended token vocabulary:

  $$\mathcal{L}_{\text{CE}} = - \sum_{t=1}^T \log P_\theta(w_t|w_{<t})$$ - Flow-matching loss for continuous acoustic targets:

  $$\mathcal{L}_{\text{flow}} = \mathbb{E}_{m,\,\varepsilon\sim\mathcal{N}(0,I)}\|m - \mathcal{F}_\phi(\varepsilon; \text{semtokens})\|^2$$ - Aggregate loss:

  $$\mathcal{L} = \mathcal{L}_{\text{CE}} + \lambda_{\text{flow}}\mathcal{L}_{\text{flow}} + \lambda_{\text{voc}}\mathcal{L}_{\text{voc}}$$

This architecture enables SoulX-Podcast to synthesize 90+ minutes of coherent, speaker-consistent, dialect-rich dialogue with contextual prosody and explicit paralinguistic cues.

2. Paralinguistic, Prosody, and Dialectal Control

SoulX-Podcast incorporates explicit paralinguistic events and dialectal variation at the token level (Xie et al., 27 Oct 2025):

• Paralinguistic Events: Mapped to discrete tokens (e.g., 〈|laughter|〉, 〈|sigh|〉, 〈|breathing|〉), these are inserted into the input sequence during both training and inference. The model learns to synthesize corresponding non-verbal audio signatures aligned with the local speech context, achieving .82 overall accuracy in paralinguistic recognition (perfect for laughter; 0.7–0.85 for subtler cues).

• Prosody: Fine-grained pitch, energy, and duration features are embedded via context-aware semantic/acoustic tokens and learned through LLM-autoregressive prediction and flow-matching.

• Dialect Support: Dialectal variation is encoded as special tokens (<Dialect_j>). SoulX-Podcast supports Mandarin, English, Sichuanese, Henanese, and Cantonese. Dialect-guided prompting (DGP) enables domain-consistent rendition even from limited training data.

Such controls allow for deliberate, contextually-invoked shifts in expressiveness and style, which is critical for the authenticity of long-form dialogic podcast content.

3. Long-Form Dialogue Generation and Speaker Management

To sustain multi-turn, long-horizon generations, SoulX-Podcast uses the following strategies (Xie et al., 27 Oct 2025):

• Context Regularization: Low-level audio tokens from earlier history are progressively attenuated ($\gamma(t) \rightarrow 0$), forcing the model to rely on high-level semantic and speaker tokens as context. This enables stable timbre and dialogic continuity across 90+ minutes.

• Speaker Embeddings and Shifts: Each utterance prepends a learned <SPEAKER> token. During inference, explicit token insertion enforces speaker turns, yielding robust and rapid timbre transitions across speakers with minimal drift.

• Multi-Dialect and Code-Switching: Cross-dialect adaptation is implemented by inserting both speaker and dialect tokens and, if needed, a short dialect-typical sentence before the primary utterance.

Observed outputs maintain high-quality, context-responsive rhythm and intonation, consistent speaker identity, and seamless multi-speaker handovers.

4. Multi-Speaker Transcription with SoulX-Transcriber

SoulX-Podcast is tightly integrated with SoulX-Transcriber, a unified diarization and ASR system built on a single-pass LALM (Qwen3-Omni) backbone (Dai et al., 1 Jun 2026).

• Architecture: Audio Encoder → Shared Transformer Encoder (32 layers) → Multi-Task Heads (speaker verification, turn prediction, ASR CTC) → Autoregressive Decoder generating interleaved [<time> <spk> words] flat token sequences.

• Losses:

  • Speaker contrastive loss (InfoNCE), boundary cross-entropy, ASR CTC, and supervised joint diarization-ASR CE.
• Benchmarking:
  • DER (Diarization Error Rate), WER/CER (Word/Character Error Rate), cpWER (cross-speaker WER in dialogue).
  • Results: DER as low as 2.89% (AISHELL-4), cpWER deltas tightly controlled (internal podcast test Δcp=11.87). System demonstrates substantial gains over VibeVoice-ASR and Gemini-3.1-Pro.
• Adaptation for Podcasts: Sliding-window chunking, context carry-over, simulated ad/music insertions, domain fine-tuning on podcast data, and augmented training with reverberation and speed/volume perturbation.

SoulX-Transcriber thus provides the backbone for high-accuracy, multi-speaker, and multi-dialect transcript generation and episode structuring.

5. Recommendation and Podcast Discovery

SoulX-Podcast deploys a cold-start recommender leveraging users’ music listening history to deliver relevant podcast content (Nazari et al., 2020). The approach relies on:

• Feature Construction:
  • Demographic, artist, genre, and dense playlist embedding (via skip-gram).
• Architecture:
  • Multi-layer Perceptron (2 x 512 ReLU layers) with user vector $$u = [\mathrm{demo}; \sum \mathrm{artist_i}; \sum\mathrm{genre_j}; \ldots; u_\mathrm{playlist}]$$ and negative-sampled softmax over the podcast catalog.
• Empirical Results:
  • Precision@1 improvement from 0.1045 to 0.1662,
  • nDCG@10 improvement from 0.1531 to 0.2201,
  • 50% increase in both online engagement and new podcast follows.
• Bias & Fairness:
  • Reduces popularity bias, ensures categorical diversity, and implements regularization toward organic distributions.

This framework facilitates effective cold-start podcast recommendation, increasing new podcast consumption while avoiding over-concentration on highly popular shows.

6. LLM-Based Podcast Preview Generation

SoulX-Podcast incorporates an LLM-powered pipeline for episode preview generation, adapted from large-scale deployment strategies (Zhu et al., 29 May 2025):

• Pipeline:
  • Audio ingestion, metadata retrieval, ASR-based sentence segmentation,
  • Context-windowed prompt construction for LLM (including chain-of-thought instructions, negative constraints, strict format),
  • LLM returns preview metadata (start/end, text, reason, tags) as JSON.
• Empirical Outcomes:
  • LLM-generated previews “win or tie” in 81.09% of cases; “win only” rate is 54.2%.
  • Per-metric significance in understandability (Z = –4.05, $p=5.09\times10^{-5}$), contextual clarity (Z = –3.40, $p=6.7\times10^{-4}$), and interest level (Z = –4.32, $p=1.59\times10^{-5}$).
  • Processing efficiency: 5× over legacy ML (under 20s/episode).
  • User engagement: 4.6% increase in evaluation time per user (online, $p<0.05$).
• Deployment:
  • Modular API for integration into UIs, support for tag taxonomy, and creator-override flows.

This system obviates the need for multiple expert ML models and manual feature engineering, providing scalable, explainable, and performant podcast previewing.

7. Comparative Assessment and Impact

SoulX-Podcast advances the state-of-the-art in long-form, multi-speaker, and multi-dialect podcast synthesis, surpassing prior systems such as CosyVoice2, VibeVoice-ASR, and traditional expert-model based recommenders and previewers (Xie et al., 27 Oct 2025, Dai et al., 1 Jun 2026, Zhu et al., 29 May 2025). Distinctive strengths include:

• Explicit control over paralinguistic and dialect cues at generation time.
• Robust handling of extended conversational context and speaker identity (>90 minutes continuous synthesis).
• Integrated, domain-adapted multi-speaker transcription tailored for variable podcast conditions.
• Scalable, accurate cold-start recommendation system leveraging cross-domain preference transfer.
• Modular, LLM-powered preview generation with significant user engagement improvements.

Empirical results on multiple objective and subjective metrics show sustained improvements in naturalness, coherence, personalization, and downstream discoverability of podcast content.

---

Key references: (Xie et al., 27 Oct 2025, Dai et al., 1 Jun 2026, Nazari et al., 2020, Zhu et al., 29 May 2025)