SoulX-Transcriber: A Robust End-to-End Framework for Multi-Speaker Speech Transcription

Abstract: Recent advances in Automatic Speech Recognition (ASR) and LLMs have significantly improved speech understanding capabilities. However, multi-speaker speech transcription remains challenging task, constrained by highly similar speaker voices, rapid turn-taking transitions, overlapping utterances and inaccurate speaker boundary segmentation. These challenges become particularly pronounced in real-world conversational audio, where speaker dynamics and acoustic conditions are highly variable. This technical report presents SoulX-Transcriber, a unified multi-speaker transcription system that jointly models speaker diarization (SD) and ASR within an LLM-based framework. SoulX-Transcriber adopts a two-stage training strategy to improve both speaker discrimination and transcription robustness. In the first stage, speaker-aware multi-task continuous pre-training enhances speaker representation learning and boundary perception. In the second stage, supervised fine-tuning further optimizes the model for accurate end-to-end speaker-attributed transcription under complex multi-speaker conditions. SoulX-Transcriber delivers strong performance and robustness across multiple public benchmarks, including AliMeeting, AISHELL-4, and AMI, while maintaining high adaptability to multi-domain scenarios.

• The paper presents a unified system that integrates speaker-aware multi-task pre-training with a scalable simulation pipeline to achieve state-of-the-art transcription performance.
• It employs a two-stage training regime—continuous pre-training on 100K hours of data followed by supervised fine-tuning—yielding significant improvements in DER and cpWER across Mandarin and English benchmarks.
• The framework eliminates cascaded errors by jointly addressing Speaker Diarization and ASR, thereby reducing error propagation and ensuring robust transcription in challenging multi-speaker scenarios.

SoulX-Transcriber: End-to-End Robust Multi-Speaker Speech Transcription

Introduction

The paper "SoulX-Transcriber: A Robust End-to-End Framework for Multi-Speaker Speech Transcription" (2606.02400) addresses persistent challenges in multi-speaker speech transcription, particularly the combined tasks of Speaker Diarization (SD) and Automatic Speech Recognition (ASR) under adverse conditions such as acoustically similar speakers, rapid turn transitions, heavy overlap, and degraded audio. Conventional cascaded SDR systems suffer from error propagation and complexity, while recent Large Audio-LLMs (LALMs) have demonstrated promise for direct, joint modeling of both tasks. However, extant approaches have not optimally addressed robust speaker representation learning during the training stage, limiting generalization and accuracy in complex scenarios.

Methodology

SoulX-Transcriber operates as a unified, end-to-end system without architectural modifications to its backbone LALM (Qwen3-Omni). The framework comprises two central pillars: a structured conversational data pipeline and a two-stage speaker-aware training regime.

Data Engineering Pipeline

The data pipeline encompasses both large-scale pseudo-labeled real conversational data and high-diversity simulated multi-speaker dialogues. Pseudo-labeled data is generated via cascaded pipelines involving advanced Voice Activity Detection (VAD), multi-hypothesis ASR fusion, and speaker embedding clustering (with HDBSCAN), enabling robust yet scalable annotation of unlabeled corpora. Simulated data augments speaker diversity and turn-taking dynamics using a four-stage synthesis pipeline: conversational text extraction, reference audio selection and attribute-based annotation, fine-grained speaker-reference matching with attribute similarity weighting, and full-dialogue waveform generation.

Figure 1: The pipeline for simulation of naturalistic and acoustically diverse multi-speaker dialogue data.

The reference matching process leverages nine-dimensional structured speaker embeddings derived with the bge-m3 encoder, incorporating high-level attributes such as timbre, emotion, and prosody. This approach enables fine-grained control over dialog construction and serves as an engine for difficult sample generation, critical for learning from acoustically confounding scenarios.

Model Architecture and Training

SoulX-Transcriber receives up to 10-minute conversational audio and in a single forward pass produces a sequence of timestamped, speaker-attributed text segments.

Figure 2: The model architecture processes multi-minute conversational audio in one pass, producing structured outputs (timestamps, speaker labels, transcribed text).

The training strategy is bifurcated:

Stage 1: Speaker-Aware Multi-Task Continuous Pre-Training. The backbone model is exposed to a suite of jointly optimized tasks:

• Speaker Turn Prediction (STP): Explicit modeling of sharp turn boundaries through token-level supervision
• Target Speaker Extraction/Recognition (TSER): Leveraging reference utterances for segment-level extraction and attribution
• Speaker Verification (SV): Segment pair discrimination for inter-speaker separability
• Speaker Diarization and Recognition (SDR): Direct end-to-end generation of time-aligned, speaker-attributed transcriptions
• ASR (auxiliary): Preserving general recognition performance and robustness across lexical and acoustic variability

The relative weight of these tasks is carefully managed (2:2:1:5:1), with 100,000 hours of training data (including 3,000 hours simulated), chunked uniformly.

Figure 3: Example LLM prompt configuration for continuous multi-task pre-training used to encourage joint speaker-aware behavior.

Stage 2: Supervised Fine-Tuning (SFT). Model weights are further refined on 1,000 hours of manually verified high-precision SDR data and narrowly filtered simulated dialogues, maximizing robustness and speaker attribution fidelity.

Empirical Results

Evaluation spans both Mandarin (AliMeeting, AISHELL-4) and English (AMI), with metrics including DER, WER, concatenated permutation WER (cpWER), and the speaker attribution gap $\Delta cp$ (cpWER minus WER).

SoulX-Transcriber establishes strong numerical superiority over both academic and commercial benchmarks:

• Short-form (utterance-level) evaluation: Achieves the best DER (2.89/5.39) and cpWER (13.90/13.61) on AISHELL-4 and AliMeeting, with minimal $\Delta cp$ (−0.26/0.54), indicating precise end-to-end attribution.
• Long-form (5-minute) evaluation: Outperforms strong closed and open baselines on extended contexts, with DER reduced to 5.72 and 7.73 vs. the nearest open method's 18 and 9.17, and cpCER (for Mandarin) approaching 16.99 and 17.82.
• General-domain evaluation: Maintains robust transcription and speaker attribution on challenging non-conversational genres (movies, podcasts, daily conversations), where DER drops to 1.32 (daily conversation) and cpWER remains below 8 across all internal sets.

  Figure 4: SoulX-Transcriber achieves substantial DER and cpWER reductions versus baselines on AliMeeting and AISHELL-4 benchmarks.

Theoretical and Practical Implications

The results substantiate several claims:

• The speaker-aware multi-task pre-training paradigm substantially improves intra-model speaker representation separability without architecture augmentation, directly boosting attribution stability in real conditions featuring confounding speaker acoustic similarity.
• The scalable, attribute-driven simulation pipeline expands the coverage and difficulty of training data efficiently, counteracting the expense and sparsity of manual multi-speaker annotation, and driving significant generalization and robustness improvements.
• The strictly end-to-end approach (i.e., not requiring cascade decoding or post-hoc clustering) ultimately reduces error propagation and system latency, creating a directly deployable framework suitable for industrial and research use in long-context SDR.

Future Outlook

Given the flexibility of the simulation pipeline and the task modularity of the multi-task pre-training protocol, further improvements are likely via:

• Incorporation of richer multi-modal cues (such as visual or metadata context) to further disambiguate speakers in multimedia scenarios.
• Adaptive curriculum learning on simulated data to push discriminative learning against adversarial and rare speaker scenario edge cases.
• Extending the backbone to universal speech domains (code-switching, low-resource, pathological speech) via targeted pre-training augmentation.

Conclusion

SoulX-Transcriber demonstrates that careful integration of speaker-aware, multi-task pre-training and diverse, attribute-driven data simulation markedly advances state-of-the-art performance in multi-speaker speech transcription, both in terms of accuracy and speaker attribution. The architecture achieves strong results on Mandarin and English benchmarks, robustly generalizes to challenging and long-form scenarios, and provides an extensible foundation for subsequent advances in unified SDR systems.