SoulX-Duplug: Duplex State Predictor

• SoulX-Duplug is a streaming state prediction module that integrates voice activity detection, ASR, and semantic turn prediction to support full-duplex dialogues.
• It decouples state control from dialogue generation, enhancing controllability, interpretability, and scalability compared to end-to-end models.
• The system employs an interleaved streaming architecture with a lightweight LLM backbone, demonstrating low latency and robust bilingual performance.

to=arxiv_search 大发pk്രമാണം 大发扑克:{"query":"(Yan et al., 16 Mar 2026) SoulX-Duplug Plug-and-Play Streaming State Prediction Module for Realtime Full-Duplex Speech Conversation","max_results":5} to=arxiv 在天天中彩票 重庆时时彩的:{"search_query":"(Yan et al., 16 Mar 2026)","start":0,"max_results":5} SoulX-Duplug is a plug-and-play, streaming state prediction module for full-duplex spoken dialogue systems that equips conventional half-duplex spoken dialogue stacks with low-latency, full-duplex voice interaction. It unifies voice activity detection, automatic speech recognition, and semantic turn/state prediction in a single streaming architecture, and explicitly leverages textual information to determine user intent, thereby functioning as a semantic VAD. The system is designed to address turn-taking, interruption handling, pause handling, and backchannel management under strong latency constraints, while avoiding the limited controllability, interpretability, and scalability associated with end-to-end full-duplex models (Yan et al., 16 Mar 2026).

1. Conceptual role and problem setting

SoulX-Duplug is positioned as a modular duplex controller rather than a content generator. In the formulation described in "SoulX-Duplug: Plug-and-Play Streaming State Prediction Module for Realtime Full-Duplex Speech Conversation" (Yan et al., 16 Mar 2026), the central motivation is that end-to-end full-duplex models entangle turn control with content generation, which limits controllability, interpretability, and scalability, and also suffers from scarce supervised duplex data and catastrophic forgetting during continual optimization. SoulX-Duplug addresses these issues by decoupling state control from content generation and by reusing abundant half-duplex data for dialogue generation while specializing a lightweight state predictor using bilingual speech corpora.

The module predicts a discrete streaming dialogue state at each chunk and is intended to be inserted into an existing spoken dialogue pipeline. This suggests a division of labor in which semantic turn control is localized in a small state-prediction component, while the dialogue manager and text-to-speech subsystems remain replaceable. A plausible implication is that the architecture is meant to preserve engineering modularity while still supporting overlap, barge-in, and low-latency turn transition.

The output state space is fixed to five tokens:

State token	Function
`<	user_idle
`<	user_nonidle
`<	user_backchannel
`<	user_complete
`<	user_incomplete

A common misconception is to treat SoulX-Duplug as a conventional acoustic VAD. The paper instead defines it as a semantic VAD because it jointly performs streaming ASR and uses textual information to identify user intent, so the state decision is conditioned on the freshest partial transcript rather than on acoustics alone (Yan et al., 16 Mar 2026).

2. Streaming architecture and formalism

The architecture consists of a pretrained, block-causal speech tokenizer, a small LLM backbone, and a state head. The speech encoder is the GLM-4-Voice tokenizer, which produces discrete speech tokens $A_d$ that are projected into the LLM embedding space through a linear/MLP adapter, written as $A = \mathrm{MLP}(A_d)$. The backbone is Qwen3-0.6B, which processes interleaved audio, text, and state tokens and predicts text then state at each step. A state head performs classification over the five state tokens (Yan et al., 16 Mar 2026).

The interleaved design is the defining mechanism. The incremental sequence is

$$\{A_1, T_1, S_1, A_2, T_2, S_2, \dots, A_T, T_T, S_T\}.$$

For chunk $t$, the audio tokens are

$A_t = [a_{t,1}, a_{t,2}].$

ASR decoding is defined as

$T_t \sim P(T_t \mid A_{\le t}, T_{< t}, S_{< t}),$

and state decoding as

$$S_t \sim P(S_t \mid A_{\le t}, T_{\le t}, S_{< t}).$$

The design therefore conditions the state on both audio and the current partial transcript, allowing real-time state estimation from semantically informed context (Yan et al., 16 Mar 2026).

The streaming interface operates on a user audio stream chunked at 160 ms. The reported audio token rate is 12.5 Hz, with GLM-4-Voice block size 12, left context 960 ms, right context 40 ms, and receptive field approximately 1160 ms. For each chunk, the outputs are streaming ASR tokens $T_t$ and one state token $S_t$. During deployment, the text tokens are supplied by teacher forcing from an external ASR system for stability and accuracy; the state token is still predicted by SoulX-Duplug.

The semantic state decision can also be written through an intent posterior over the partial transcript. Let $t_{1:k}$ denote the partial transcript up to chunk $A = \mathrm{MLP}(A_d)$0. Then

$A = \mathrm{MLP}(A_d)$1

where $A = \mathrm{MLP}(A_d)$2 aggregates token embeddings. The paper further gives an optional incremental Bayesian update,

$A = \mathrm{MLP}(A_d)$3

normalized over $A = \mathrm{MLP}(A_d)$4. The state decision is described as a thresholded function of the intent posterior, with additional logic to distinguish complete versus incomplete utterances and backchannels (Yan et al., 16 Mar 2026).

3. Training regime, inference pipeline, and system integration

SoulX-Duplug uses a 3-stage training procedure. Stage 1 is non-streaming ASR pretraining. Stage 2 is streaming ASR adaptation. Stage 3 is state prediction supervised fine-tuning, with LoRA rank $A = \mathrm{MLP}(A_d)$5 for stage 3 bilingual fine-tuning. The speech tokenizer is frozen; the adapter and state head are trained. The training objective is a weighted token-level cross-entropy over the interleaved target stream,

$A = \mathrm{MLP}(A_d)$6

where $A = \mathrm{MLP}(A_d)$7 is the target interleaved token sequence and $A = \mathrm{MLP}(A_d)$8 is the token type. For completeness, the paper also states standard streaming ASR objectives such as CTC and RNN-T, but presents them as general objectives that can be integrated rather than as the reported training loss for SoulX-Duplug itself (Yan et al., 16 Mar 2026).

The ASR pretraining and adaptation data are large-scale bilingual corpora. The paper reports approximately 47k hours of Mandarin data from AISHELL-1, AISHELL-3, WenetSpeech, CommonVoice-CN, Emilia-CN, and MAGICDATA in VoxBox, and approximately 31k hours of English data from LibriSpeech, GigaSpeech, CommonVoice-EN, and Emilia-EN in VoxBox. State prediction training uses English Fisher, about 1000 hours, and a Mandarin in-house Fisher-formatted multi-thousand-hour conversational corpus. Labels are produced via Qwen2.5-72B-Instruct, and noise augmentation uses MUSAN and ESC-50.

At inference time, SoulX-Duplug is not used as a standalone dialogue system. Paraformer for Chinese or SenseVoice Small for English supplies partial transcripts in real time. The dialogue manager can be any half-duplex LLM; the reported system uses Qwen2.5-7B-Instruct. TTS is exemplified by IndexTTS-1.5. The dialogue manager consumes final user utterances when <|user_complete|> is predicted, handles interruptions and backchannels based on the state token, and can stop or resume speech under overlap conditions (Yan et al., 16 Mar 2026).

The real-time loop consists of buffering 160 ms of audio, tokenizing it through GLM-4-Voice, obtaining teacher-forced ASR tokens, predicting the state token, updating the dialogue manager, and repeating. The module exposes a streaming API that consumes audio chunks or audio tokens and optionally teacher-forced text tokens, and returns the partial transcript $A = \mathrm{MLP}(A_d)$9, the state token $$\{A_1, T_1, S_1, A_2, T_2, S_2, \dots, A_T, T_T, S_T\}.$$0 with confidence $$\{A_1, T_1, S_1, A_2, T_2, S_2, \dots, A_T, T_T, S_T\}.$$1, and timestamps and per-chunk latency estimates.

The latency model is explicit. Because the system confirms an endpoint only upon processing the next chunk, and because the actual speech endpoint is assumed uniformly distributed within the chunk, the expected offset to the chunk end is 80 ms. The expected detection latency is therefore

$$\{A_1, T_1, S_1, A_2, T_2, S_2, \dots, A_T, T_T, S_T\}.$$2

The paper also gives an optional smoothing rule,

$$\{A_1, T_1, S_1, A_2, T_2, S_2, \dots, A_T, T_T, S_T\}.$$3

to reduce jitter under rapid chunking (Yan et al., 16 Mar 2026).

4. Evaluation protocol and SoulX-Duplug-Eval

To support more uniform comparison, the work introduces SoulX-Duplug-Eval, which extends widely used benchmarks with improved bilingual coverage (Yan et al., 16 Mar 2026). The benchmark combines duplex state prediction tasks and system-level full-duplex dialogue tasks.

For duplex state prediction, the Easy Turn benchmark is used together with an Easy Turn-En extension. The English extension contains 318 Complete and 299 Incomplete utterances, generated from ChatGPT text and ChatTTS synthesis. For system-level full-duplex tasks, the paper describes Full-Duplex-Bench-Zh as Chinese counterparts for system-level tasks derived from FDB v1/v1.5. The subsets include Turn-Taking with 155 samples, Pause Handling with 239 samples via ChatTTS uv_break, User Backchannel with 199 samples via SoulX-Podcast, and User Interruption with 161 samples via ChatTTS.

The metrics follow the official FDB definitions. Turn Taking uses Takeover Rate, defined as the proportion of successful turn transitions, and Response Latency, defined as the delay from the user's end to system start:

$$\{A_1, T_1, S_1, A_2, T_2, S_2, \dots, A_T, T_T, S_T\}.$$4

Pause Handling uses TOR as the proportion of mistakenly taking the floor during pauses, where lower is better. User Interruption v1 uses TOR and RL. User Backchannel uses Resume Rate, the proportion of RESUME behaviors after overlap. User Interruption v1.5 uses Respond Rate, Stop Latency, and RL. Duplex state prediction uses Complete-versus-Incomplete accuracy and per-sample inference latency. Overall turn management aggregates $$\{A_1, T_1, S_1, A_2, T_2, S_2, \dots, A_T, T_T, S_T\}.$$5 with TOR, RsR, and RpR across tasks, while overall latency aggregates RL and SL across tasks (Yan et al., 16 Mar 2026).

This evaluation design is notable because it separates local state-prediction quality from end-to-end conversational behavior. A plausible implication is that it permits analysis of whether a semantically informed controller improves full-duplex interaction even when the dialogue generator itself is unchanged.

5. Reported empirical performance

The reported theoretical average detection latency is 240 ms. Measured module latency is 205 ms for English and 295 ms for Chinese. The paper states that both outperform FlexDuo at 343 ms and VAD baselines at approximately 500 ms (Yan et al., 16 Mar 2026).

On bilingual Full-Duplex-Bench system-level evaluation, the English SoulX-Duplug-based system reports the following values: Pause Handling TOR = 0.352; Turn Taking TOR = 0.933 and RL = 0.511 s; User Backchannel RsR = 0.740; User Interruption v1 TOR = 0.970 and RL = 0.773 s; User Interruption v1.5 RpR = 0.770, SL = 0.450 s, and RL = 1.030 s; Overall ACC = 0.812 and average latency = 0.691 s. The paper characterizes these results as strong turn-taking with lower pause mishandling than dGSLM, PersonaPlex, and Moshi, and as substantially better than Freeze-Omni and Gemini on turn-taking TOR while keeping latency low.

For Chinese, the SoulX-Duplug-based system reports Pause Handling TOR = 0.038; Turn Taking TOR = 0.994 and RL = 0.767 s; User Backchannel RsR = 0.800; User Interruption v1 TOR = 0.994 and RL = 1.089 s; User Interruption v1.5 RpR = 0.830, SL = 0.380 s, and RL = 1.150 s; Overall ACC = 0.916 and average latency = 0.847 s. The paper specifically compares these results with Freeze-Omni, including TT TOR 0.994 versus 0.652 and SL 0.380 s versus 1.300 s (Yan et al., 16 Mar 2026).

On the Easy Turn duplex state prediction task, English performance is reported as 77.67% accuracy for Complete, 88.96% for Incomplete, and 83.32% average, with theoretical latency 240 ms. Chinese performance is 89.33% for Complete, 79.33% for Incomplete, and 84.33% average, again with theoretical latency 240 ms. The paper compares these results to non-streaming baselines: SenseVoice+TEN reaches 86.10% average in English but incurs extra VAD latency; Paraformer+TEN reaches 87.99% average in Chinese; Easy Turn’s own non-streaming pipeline reports 97.00% average with +263 ms VAD and longer end-to-end delays. The interpretation offered in the paper is that SoulX-Duplug trades some non-streaming classification accuracy for substantially lower and more predictable latency (Yan et al., 16 Mar 2026).

The ablation study on Zh Easy Turn isolates two components. Without ASR pretraining, average accuracy drops from 84.33% to 80.50%. Without teacher-forced inference, average accuracy drops to 73.50%. These results are presented as evidence for the importance of text-guided streaming via ASR capability and teacher-forced partial transcripts.

6. Interpretation, limitations, and deployment profile

The system’s principal architectural claim is that modularization reduces catastrophic forgetting risk by avoiding repeated fine-tuning of an end-to-end duplex generator. The paper states that staged training and LoRA supervised fine-tuning keep changes localized to the state predictor. At the same time, it explicitly notes that catastrophic forgetting experiments are not reported; the mitigation is architectural and procedural rather than empirically quantified (Yan et al., 16 Mar 2026). This is an important qualification, because it distinguishes a design rationale from a benchmarked continual-learning result.

The paper also emphasizes scalability. The state predictor runs on a 0.6B-parameter LLM backbone, while recognition is offloaded to smaller optimized models such as Paraformer and SenseVoice Small. The modular design allows independent upgrades of ASR, dialogue manager, and TTS. Reported deployment hardware is a single NVIDIA L20 for evaluation and NVIDIA H20 for training, which the paper presents as compatible with real-time inference.

Several limitations are identified. Small-chunk streaming ASR is inherently difficult, particularly in English where words are often split across chunks, producing transient token instability. Noisy environments and far-field conditions may degrade performance. Code-switching and dialectal variation remain challenging. Edge deployment on CPU or mobile devices requires further slimming and quantization. The paper suggests that more robust chunk-level ASR, richer prosodic and acoustic features such as energy and pitch, expanded bilingual or multilingual training, intent-head calibration, tiny transformer or RNN-T backbones, and fused kernels are plausible directions for improvement (Yan et al., 16 Mar 2026).

An additional misconception concerns scope. SoulX-Duplug is not presented as a universal full-duplex dialogue model and does not replace the dialogue manager or TTS. It is instead a pluggable streaming state predictor with a bilingual evaluation suite, open-sourced repository, training code for the 3-stage pipeline, a streaming inference engine, GLM-4-Voice tokenizer integration, a state token head, evaluation scripts, configuration files for teacher-forced ASR and thresholds, and pretrained checkpoints (Yan et al., 16 Mar 2026). In that sense, its significance lies less in end-to-end generation than in providing a semantically aware control layer for full-duplex turn management.