SoulX-Transcriber: End-to-End Multi-Speaker Transcription

• The paper introduces a unified SDR framework that combines speaker diarization and ASR into a single autoregressive generation process.
• It employs a two-stage training strategy with speaker-aware pre-training and supervised fine-tuning using diverse real and synthetic conversational data.
• Evaluations on benchmarks like AISHELL-4 and AliMeeting demonstrate competitive performance in challenging conditions such as overlapping speech and rapid turn-taking.

SoulX-Transcriber is an end-to-end system for multi-speaker speech transcription: the joint task of determining who spoke what and when directly from conversational audio. In the underlying technical report, this joint task is called SDR, because it unifies speaker diarization (SD) and automatic speech recognition (ASR) into a single model and output space. The system is built on Qwen3-Omni and is presented less as a new backbone architecture than as a training-centric framework that combines speaker-aware pre-training, supervised refinement, and large-scale data construction to improve speaker-attributed transcription under overlapping speech, rapid turn-taking, acoustically similar speakers, and domain variability (Dai et al., 1 Jun 2026).

1. Task definition and operating assumptions

SoulX-Transcriber addresses conversational audio conditions that are substantially more difficult than ordinary single-speaker ASR. The motivating setting includes highly similar speaker voices, rapid turn-taking transitions, overlapping utterances, and inaccurate speaker boundary segmentation. The target domains include meetings, daily chats, movies, and podcasts, and the paper treats these as acoustically variable, real-world conditions rather than as narrowly curated benchmarks (Dai et al., 1 Jun 2026).

A central premise is that modular diarization-plus-ASR pipelines suffer from boundary errors and error propagation. SoulX-Transcriber therefore frames SDR as a unified generation problem rather than a cascade of independently optimized subsystems. This design is intended to reduce pipeline complexity, avoid the propagation of upstream diarization errors into downstream recognition, and permit interactions among lexical content, timing, and speaker identity to be learned within one autoregressive process (Dai et al., 1 Jun 2026).

The system accepts conversational audio clips up to 10 minutes, processes them in a single forward pass, and directly generates structured text outputs containing speaker labels, timestamp boundaries, and transcribed text. The paper does not specify a streaming mode, chunk-stitching procedure for recordings beyond 10 minutes, or detailed decoding hyperparameters. This suggests an offline or non-streaming long-context transcription setting rather than a low-latency streaming SDR formulation (Dai et al., 1 Jun 2026).

2. Architecture and output representation

At the architectural level, SoulX-Transcriber is built on Qwen3-Omni, described as a large omni-modal or large audio-language backbone with strong long-context audio understanding and autoregressive generation ability. The authors explicitly do not present a radically new backbone with layer-by-layer internals; instead, they retain the backbone largely intact and improve it through a speaker-aware two-stage training strategy and a large-scale data pipeline (Dai et al., 1 Jun 2026).

The output space is serialized and sequence-based. For the core SDR task, the model “directly generates structured outputs containing speaker labels, timestamp boundaries, and transcribed text.” In one auxiliary task, special boundary tokens are inserted into the target text to model speaker turns. The exact serialization syntax is not specified, but the conceptual design is clear: speaker attribution and temporal structure are encoded in the generated text stream rather than produced by a separate diarization head followed by a separate recognizer (Dai et al., 1 Jun 2026).

The training framework emphasizes five tasks:

• Speaker Turn Prediction (STP): special boundary tokens are inserted into the target sequence to improve temporal perception of turn changes.
• Target Speaker Extraction and Recognition (TSER): given a reference audio for a target speaker plus a multi-speaker conversation, the model identifies the target speaker’s regions and transcribes them.
• Speaker Verification (SV): given two speech segments, the model predicts whether they come from the same speaker.
• Speaker Diarization and Recognition (SDR): the main end-to-end task, jointly generating speaker labels, timestamps, and text.
• Automatic Speech Recognition (ASR): multilingual and out-of-domain ASR data are included to preserve general ASR competence and acoustic robustness (Dai et al., 1 Jun 2026).

The paper’s most important architectural claim is therefore not a new neural block, but a unified autoregressive interface that binds speaker discrimination, turn perception, and transcription into one output channel.

3. Speaker-aware two-stage training strategy

The methodological core of SoulX-Transcriber is a two-stage training pipeline. Stage 1 is speaker-aware multi-task continuous pre-training; Stage 2 is supervised fine-tuning (SFT). The first stage is large-scale and mixed-supervision, intended to improve speaker representation learning, speaker turn perception, and multi-speaker conversational understanding before higher-precision supervised refinement (Dai et al., 1 Jun 2026).

Stage 1 combines pseudo-labeled real conversational data, simulated multi-speaker dialogue data, public corpora, and proprietary data. The total pre-training audio is about 100,000 hours, including about 3,000 hours of synthetic multi-speaker conversational data. Samples are chunked into 5-minute segments, with maximum segment length 10 minutes. The task mixture ratio is approximately

$$\text{STP} : \text{TSER} : \text{SV} : \text{SDR} : \text{ASR} = 2 : 2 : 1 : 5 : 1.$$

This stage is presented as the main mechanism for strengthening speaker discrimination, boundary perception, and robustness to acoustically similar speakers and rapid turn-taking (Dai et al., 1 Jun 2026).

Stage 2 uses about 1,000 hours of manually annotated conversational SDR data and carefully filtered simulated dialogue data. Its stated purpose is to correct the label noise and speaker-attribution uncertainty that remain after large-scale pseudo-labeled pre-training, thereby improving speaker attribution accuracy, instruction consistency, generalization, and final end-to-end speaker-attributed transcription (Dai et al., 1 Jun 2026).

The paper is explicit that it does not provide full loss formulas for STP, TSER, SV, SDR, or timestamp alignment. A conceptual multi-task objective is possible, but it is not printed in the report. This suggests that the distinctive contribution lies less in formal objective design than in task composition, data curation, and large-scale speaker-aware supervision (Dai et al., 1 Jun 2026).

4. Data construction and simulation pipeline

The Stage 1 data pipeline has two major branches: pseudo-labeled dialogue data and simulated multi-speaker dialogue data. The pseudo-labeled branch has three steps. First, speech segmentation uses silero VAD and pyannote-vad, whose outputs are aligned and merged, after which the pyannote speaker diarization pipeline is applied to refine turn boundaries and split long audio into speaker-aware utterance segments. Second, each segment is transcribed by multiple heterogeneous ASR models, fused by consensus fusion, and low-confidence segments are filtered out. Third, speaker embeddings are extracted and clustered with HDBSCAN within each session, with clusters treated as pseudo speaker IDs; neighboring segments with the same speaker label and a short temporal gap are merged (Dai et al., 1 Jun 2026).

The simulated-data branch is more explicitly formalized. Conversational texts are collected from podcasts, novels, and dialogue-centric corpora in Chinese and English, and an LLM constructs structured dialogue scripts with 3–8 speakers. Candidate reference clips of 3–10 seconds are extracted from long-form conversational recordings and multimedia content, and each is annotated with nine speaker-related attributes: gender, age range, emotion, speaking rate, pitch characteristics, timbre style, expression style, vocal characteristics, and speaking style. Quality control uses UTMOS and SNR (Dai et al., 1 Jun 2026).

These nine textual attributes are encoded with bge-m3 into

$$\mathbf{E_i} \in \mathbb{R}^{9 \times 1024}, \quad i=1,\ldots,N,$$

and stacked as

$$\mathbf{E} \in \mathbb{R}^{N \times 9 \times 1024}.$$

For a target dialogue role, the corresponding representation is

$$\mathbf{Q_j} \in \mathbb{R}^{9 \times 1024}, \quad j=1,\ldots,M.$$

Attribute-wise similarity is computed as

$$\mathbf{V} = \mathbf{E} \cdot \mathbf{Q_j}^{\top} \in \mathbb{R}^{N \times 9 \times 9},$$

after which only diagonal elements are retained, producing a 9-dimensional vector $\mathbf{d}_i$ for each candidate. With a predefined weight vector

$$\mathbf{w} = [w_1, w_2, \dots, w_9]^{\top} \in \mathbb{R}^{9},$$

the final matching score is

$$s_i = \mathbf{d}_i \cdot \mathbf{w} = \sum_{j=1}^{9} w_j \cdot \mathbf{d}_i^{(j)}.$$

Top-$k$ references are then filtered so that different dialogue roles do not use audio from the same original speaker, and UTMOS differences remain within a threshold. The paper treats this simulation pipeline as a way to generate more natural and contextually coherent multi-speaker training data while controlling difficult speaker conditions (Dai et al., 1 Jun 2026).

5. Evaluation protocol and benchmark results

SoulX-Transcriber is evaluated on public benchmarks and internal test sets. The public benchmarks are AliMeeting, AISHELL-4, and AMI-SDM. The evaluation metrics are DER for diarization error rate, WER for word error rate ignoring speaker labels, cpWER for concatenated minimum-permutation WER, and $\Delta cp$, defined as cpWER minus WER, to isolate speaker-attribution penalty. For long-form Mandarin evaluation, the paper uses CER/cpCER instead of WER/cpWER (Dai et al., 1 Jun 2026).

On AISHELL-4, SoulX-Transcriber reports DER 2.89, WER 14.16, cpWER 13.90, and $$\mathbf{E_i} \in \mathbb{R}^{9 \times 1024}, \quad i=1,\ldots,N,$$0. On AliMeeting, it reports DER 5.39, WER 13.07, cpWER 13.61, and $$\mathbf{E_i} \in \mathbb{R}^{9 \times 1024}, \quad i=1,\ldots,N,$$1. On both datasets it outperforms the compared systems Vibevoice-ASR, Gemini-2.5-Pro, and Gemini-3.1-pro-preview across the principal metrics listed in the paper (Dai et al., 1 Jun 2026).

On AMI-SDM, the picture is more mixed. SoulX-Transcriber achieves DER 11.67, WER 25.55, cpWER 32.78, and $$\mathbf{E_i} \in \mathbb{R}^{9 \times 1024}, \quad i=1,\ldots,N,$$2. The report characterizes this as competitive rather than dominant: SoulX has the best DER, but Gemini-3.1-pro-preview shows slightly better cpWER, and Vibevoice-ASR has much better $$\mathbf{E_i} \in \mathbb{R}^{9 \times 1024}, \quad i=1,\ldots,N,$$3. The paper explicitly notes that despite primarily Mandarin-centric training, the system still maintains reasonable English performance. This suggests that SoulX-Transcriber is strongest in Mandarin and in Mandarin cross-domain robustness, with room for improvement in English speaker-attributed transcription (Dai et al., 1 Jun 2026).

Long-form results reinforce the system’s stated long-context competence. On 5-minute AliMeeting, SoulX-Transcriber reports DER 5.72, CER 16.22, cpCER 16.99, and $$\mathbf{E_i} \in \mathbb{R}^{9 \times 1024}, \quad i=1,\ldots,N,$$4. On 5-minute AISHELL-4, it reports DER 7.73, CER 14.49, cpCER 17.82, and $$\mathbf{E_i} \in \mathbb{R}^{9 \times 1024}, \quad i=1,\ldots,N,$$5. The relatively small $$\mathbf{E_i} \in \mathbb{R}^{9 \times 1024}, \quad i=1,\ldots,N,$$6 on AliMeeting is presented as evidence of strong speaker tracking consistency over several minutes (Dai et al., 1 Jun 2026).

On internal datasets, the system is especially strong on Daily Conversation, with DER 1.32, WER 6.73, cpWER 7.31, and $$\mathbf{E_i} \in \mathbb{R}^{9 \times 1024}, \quad i=1,\ldots,N,$$7. On Movies, it obtains DER 23.56, WER 5.17, cpWER 20.58, and $$\mathbf{E_i} \in \mathbb{R}^{9 \times 1024}, \quad i=1,\ldots,N,$$8. On Podcast, it reports DER 21.15, WER 7.5, cpWER 19.37, and $$\mathbf{E_i} \in \mathbb{R}^{9 \times 1024}, \quad i=1,\ldots,N,$$9. These results indicate that transcript quality can remain strong even when diarization generalization weakens, particularly in podcast-like scenarios (Dai et al., 1 Jun 2026).

6. Interpretation, limitations, and research context

The report presents SoulX-Transcriber as a robust end-to-end LLM-based SDR framework, but it also leaves several issues open. It does not provide explicit ablation tables for removing Stage 1, Stage 2, STP, TSER, SV, or simulated data. It also does not publish detailed formulas for the core SDR loss, timestamp alignment loss, or any overlap-specific objective. This suggests that the claimed roles of STP, TSER, and SV are plausible design rationales supported by the overall empirical package, rather than individually isolated causal findings (Dai et al., 1 Jun 2026).

A second clarification concerns overlap and boundaries. The paper repeatedly motivates the system with overlapping utterances and noisy speaker boundaries, but it does not introduce an explicit overlap-specific module such as permutation-invariant masking, frame-level overlap labels, or separate overlapping-speaker streams. Overlap handling therefore appears to be implicit, emerging from unified generation, pseudo-labeled and simulated conversational data, timestamp supervision, and speaker-aware multi-task pre-training rather than from a dedicated overlap architecture (Dai et al., 1 Jun 2026).

A third clarification is architectural. SoulX-Transcriber should not be read as a newly specified backbone on the scale of a custom encoder-decoder stack; it is primarily a framework layered on Qwen3-Omni. Its novelty lies in the combination of unified SDR generation, speaker-aware multi-task pre-training, explicit turn-token supervision, speaker-conditioned extraction and verification tasks, and a conversation-oriented simulation pipeline (Dai et al., 1 Jun 2026).

Within the broader transcription literature, SoulX-Transcriber occupies a distinct position. Unlike rapid low-resource ASR bootstrapping systems that rely on modular acoustic models, pronunciation lexica, and weighted finite-state decoding for newly encountered languages (Hasegawa-Johnson et al., 2019), SoulX-Transcriber is concerned with long-context, multi-speaker conversational audio and end-to-end speaker-attributed generation. A plausible implication is that it belongs less to the classical modular ASR tradition and more to the recent line of large audio-LLMs that collapse recognition, attribution, and structural formatting into one sequence-generation interface (Dai et al., 1 Jun 2026).

The main encyclopedic significance of SoulX-Transcriber is therefore methodological. It exemplifies a shift from pipeline SDR toward unified autoregressive transcription, while also showing that robust performance in multi-speaker settings depends heavily on speaker-aware training signals and data engineering. At the same time, the mixed AMI and podcast results, the absence of detailed ablations, and the lack of deployment-oriented latency or streaming analysis indicate that the framework is strongest as a long-context offline research system rather than as a fully specified production stack (Dai et al., 1 Jun 2026).