SoulX-Duplug-Eval: Bilingual Full-Duplex Evaluation

• SoulX-Duplug-Eval is a bilingual benchmark suite that evaluates full-duplex spoken dialogue systems via state prediction and integrated system-level testing.
• It extends existing benchmarks by adding English and Chinese counterparts to reduce language asymmetry and assess overlap-sensitive turn-taking scenarios.
• The evaluation protocol emphasizes real-time performance by measuring both accuracy and latency under realistic streaming conditions.

SoulX-Duplug-Eval is a bilingual benchmark suite for evaluating full-duplex spoken dialogue systems at both the duplex state prediction level and the end-to-end dialogue-system level. Introduced together with SoulX-Duplug, it is framed as a “complementary benchmark” rather than a wholly new standalone resource: it extends Easy Turn for state prediction evaluation and extends Full-Duplex-Bench for system-level evaluation, with the specific goal of improving comparability across papers and improving bilingual coverage in English and Chinese (Yan et al., 16 Mar 2026). Its design is tightly coupled to a streaming, overlap-sensitive conception of full-duplex interaction in which a system must decide not only when to speak, but also when to wait, resume, or stop under pauses, backchannels, and interruptions.

1. Position within full-duplex spoken dialogue research

SoulX-Duplug-Eval was proposed in response to a specific evaluation problem in full-duplex spoken dialogue research: prior work had been assessed largely on self-constructed test sets, without a broadly shared benchmark and with very limited cross-lingual coverage. The resources that were already public were described as fragmented. Easy Turn covered duplex state prediction but only in Chinese, while the Full-Duplex-Bench series covered system-level full-duplex interaction scenarios but only in English. SoulX-Duplug-Eval is designed to fill that gap by adding the missing language counterparts rather than replacing those resources outright (Yan et al., 16 Mar 2026).

This positioning matters because the benchmark’s central notion of fairness is methodological rather than purely statistical. First, it extends widely used public benchmarks instead of introducing a private evaluation set, thereby creating a more common basis for comparison. Second, it reduces language asymmetry: “improved bilingual coverage” specifically means adding English samples to Easy Turn, which had been Chinese-only, and adding Chinese test sets to Full-Duplex-Bench, which had been English-only. The suite is therefore bilingual, not multilingual beyond English and Mandarin/Chinese.

The benchmark is closely related to, but distinct from, SoulX-Duplug. SoulX-Duplug is the plug-and-play streaming state prediction module. SoulX-Duplug-Eval is the evaluation resource used to test that module and competing methods. The paper’s underlying argument is that a modular streaming state controller should be evaluated both in isolation and through downstream dialogue behavior under realistic online constraints. This suggests a two-level evaluation philosophy: state-level correctness alone is insufficient for characterizing full-duplex interaction quality.

2. Benchmark composition and bilingual extensions

SoulX-Duplug-Eval has two components: the Bilingual Easy Turn Testset and the Bilingual Full-Duplex-Bench component. Both are extensions layered onto pre-existing benchmarks rather than independently constructed tasks (Yan et al., 16 Mar 2026).

Component	Extension role	Reported contents
Bilingual Easy Turn Testset	Extends Chinese Easy Turn with English Easy Turn testset-En	English Complete: 318; English Incomplete: 299
Bilingual Full-Duplex-Bench	Extends English FDB v1 and v1.5 with Chinese Full-Duplex-Bench-Zh	Turn-Taking: 155; Pause Handling: 239; User Backchannel: 199; User Interruption: 161

The Bilingual Easy Turn Testset extends the original Easy Turn state-prediction benchmark with an English counterpart called Easy Turn testset-En. It targets duplex state prediction rather than dialogue generation. The English extension has two categories: Complete and Incomplete. The Complete category contains 318 semantically complete utterances, and the Incomplete category contains 299 semantically incomplete utterances. Its text was generated by ChatGPT and synthesized with ChatTTS. The original Chinese Easy Turn testset is reused from prior work. No train/validation/test splits are reported for the English extension beyond its presentation as a testset, and no speaker diversity statistics, acoustic-condition summaries, recording hours, or environment descriptions are reported beyond the fact that samples are synthesized.

The Bilingual Full-Duplex-Bench component creates Full-Duplex-Bench-Zh as a Chinese counterpart to the English Full-Duplex-Bench resources, specifically drawing on Full-Duplex-Bench v1 and v1.5. It covers four representative interaction scenarios. Turn-Taking contains 155 samples in which a user speaks for several seconds and then there are 15 seconds of silence; these are synthesized using ChatTTS. Pause Handling contains 239 samples in which a single utterance contains inserted pauses created using the [uv_break] control token in ChatTTS. User Backchannel contains 199 samples structured as user speech, then 3 seconds of silence, then a short backchannel utterance, then 15 seconds of silence; these are synthesized with SoulX-Podcast. User Interruption contains 161 samples in which an initial user utterance is followed by 3 seconds of silence and then a semantically related second utterance, followed by 15 seconds of silence; these are synthesized using ChatTTS.

These choices make the suite explicitly overlap-sensitive. The emphasis on pauses, backchannels, and interruptions reflects the edge cases that distinguish full-duplex from half-duplex interaction. A plausible implication is that the benchmark is designed less around linguistic generation quality than around online dialogue control.

3. Underlying interaction model and evaluated state space

Although SoulX-Duplug-Eval is a benchmark rather than a model, its design is grounded in the state formulation used by SoulX-Duplug. The paper formulates full-duplex interaction control as streaming state prediction over incrementally arriving audio. Instead of a cascaded “VAD–ASR–Turn Detection” stack, the associated system unifies VAD-like activity detection, streaming ASR, and turn-state prediction in one streaming architecture. The key concept is “semantic VAD”: speech activity and endpoint detection are informed by text derived from streaming ASR rather than by acoustics alone (Yan et al., 16 Mar 2026).

The benchmark is built around five custom control states:

• <|user_idle|>: no semantic content, such as silence or noise
• <|user_nonidle|>: semantically meaningful speech is present
• <|user_backchannel|>: user backchannel behavior
• <|user_complete|>: the user utterance is semantically complete and the assistant may speak
• <|user_incomplete|>: the user has paused, but semantically has not finished, so the assistant should wait

This state design is central to what SoulX-Duplug-Eval measures. It is not limited to endpoint detection in the narrow acoustic sense. Instead, it evaluates whether a system can distinguish semantically complete turn endings from temporary pauses, recognize backchannels, and manage overlap-sensitive control decisions. In the paper’s terminology, “streaming state prediction” means chunk-by-chunk prediction under incremental observability rather than after complete utterance segmentation, while “full-duplex” means the system can listen and speak simultaneously rather than alternating rigidly between the two.

Several formal modeling details explain what the benchmark is testing. SoulX-Duplug uses interleaved audio, text, and state token prediction,

$$\{A_1, T_1, S_1, A_2, T_2, S_2, \dots, A_T, T_T, S_T\},$$

with streaming ASR text tokens predicted first,

$T_t \sim P(T_t \mid A_{\le t}, T_{<t}, S_{<t}),$

and dialogue state tokens then predicted conditioned on audio, text, and prior states,

$S_t \sim P(S_t \mid A_{\le t}, T_{\le t}, S_{<t}).$

The benchmark therefore evaluates behavior that is explicitly text-guided. This suggests that SoulX-Duplug-Eval is not simply measuring acoustic turn-taking, but the effectiveness of semantically conditioned online control.

4. Evaluation protocol, metrics, and latency model

SoulX-Duplug-Eval operates at two levels. At the duplex state prediction level, it evaluates classification of utterances as complete or incomplete on the bilingual Easy Turn testset. The reported metrics are ACC_Complete, ACC_Incomplete, average accuracy, and latency. The comparison in this setting is against non-streaming state prediction modules. The paper does not give formal equations for the accuracy metrics, but it emphasizes latency because the benchmark is intended to reward real-time deployability rather than offline correctness alone (Yan et al., 16 Mar 2026).

For SoulX-Duplug, the theoretical average latency is derived from 160 ms chunks. If speech ends inside chunk $t_i$, the system can only confirm the absence of continued speech after chunk $t_{i+1}$ arrives. Assuming the endpoint is uniformly distributed within the chunk, the expected residual time within $t_i$ is 80 ms, and waiting for the next chunk adds 160 ms, giving

$$\text{Latency}_{\text{avg.}} = 80\,\text{ms} + 160\,\text{ms} = 240\,\text{ms}.$$

At the system level, SoulX-Duplug-Eval uses the official Full-Duplex-Bench metrics on bilingual Full-Duplex-Bench tasks. For Pause Handling, the metric is TOR, Takeover Rate, where lower is better; it measures how often the system takes the floor even though the user is only pausing and has not finished. For Turn Taking, the metrics are TOR, where higher is better, and RL, Response Latency. For User Interruption v1, the metrics are TOR and RL. For User Backchannel, the metric is RsR, Resume Rate. For User Interruption v1.5, the metrics are RpR, Respond Rate, SL, Stop Latency, and RL.

The paper also defines aggregate indicators. Overall turn management is computed by averaging $1-\text{TOR}$ for Pause Handling, TOR for Turn Taking, RsR for User Backchannel, and RpR for User Interruption v1.5. Overall latency is defined as the mean of all RL and SL values across tasks. No additional formal equation is typeset for these aggregates.

The benchmark’s fairness adjustment is especially explicit in the state-level comparison. Because Easy Turn uses pre-segmented clips, non-streaming baselines can appear unrealistically strong unless VAD latency is included. Accordingly, the latency of non-streaming methods is written as latency_vad + ... ms, and the paper stresses that practical deployment requires external VAD segmentation for such methods. This places deployment realism at the center of the evaluation protocol rather than treating segmentation as an externality.

5. Streaming implementation context and benchmarked operating point

The benchmark is designed for online streaming inference. The paper reports a simulated online streaming evaluation setup on a single NVIDIA L20 GPU. Several implementation details are provided because they define the low-latency operating point that SoulX-Duplug-Eval is intended to assess (Yan et al., 16 Mar 2026).

The speech tokenizer is GLM-4-Voice. Audio tokens are extracted at 12.5 Hz. The system uses a block size of 12 for audio token generation. At each step, it encodes a 160 ms target window with 960 ms left context and 40 ms right context, for a total receptive field of 1160 ms and 15 extracted tokens. The training objective is a weighted token-level loss over mixed token types,

$$\mathcal{L} = \sum_{j=1}^{|\mathcal{Y}|} \lambda_{\tau(y_j)} \cdot \mathcal{L}_{\text{CE}(y_j)},$$

where $y_j$ is the $T_t \sim P(T_t \mid A_{\le t}, T_{<t}, S_{<t}),$0-th token in the target sequence $T_t \sim P(T_t \mid A_{\le t}, T_{<t}, S_{<t}),$1, $T_t \sim P(T_t \mid A_{\le t}, T_{<t}, S_{<t}),$2 maps each token to its token type, and $T_t \sim P(T_t \mid A_{\le t}, T_{<t}, S_{<t}),$3 is a token-type-specific weight. The accompanying explanation states that token-type weighting balances ASR tokens, special ASR end tokens, and state tokens during end-to-end training.

During actual inference, the model uses teacher-forced streaming ASR from external lightweight ASR systems: SenseVoice Small for English and Paraformer for Mandarin. The paper states that this preserves the semantic benefits of joint training while improving deployment accuracy and efficiency. This is particularly relevant to SoulX-Duplug-Eval because the benchmark is intended to measure online semantic control, and the reported system relies on explicit textual guidance at inference time.

The benchmark comparisons include both state modules and full systems. On the state-prediction side, SoulX-Duplug is compared against SenseVoice En + TEN Turn Detection, Smart Turn V2, Easy Turn, and Paraformer + TEN Turn Detection, depending on language. At the system level, a modular FD-SDS built from SoulX-Duplug, Qwen2.5-7B-Instruct, and IndexTTS-1.5 is compared against dGSLM, PersonaPlex, Moshi, Freeze-Omni, Gemini Live, Sonic, and GPT-4o, although some entries are missing and Chinese system-level comparison is reported only for Freeze-Omni and SoulX-Duplug.

6. Reported empirical results

On the Bilingual Easy Turn state-level benchmark, SoulX-Duplug is reported as slightly below the best non-streaming baselines in overall accuracy but competitive given its streaming constraint (Yan et al., 16 Mar 2026). In English, SenseVoice En + TEN Turn Detection reaches Complete 95.60%, Incomplete 76.59%, Avg 86.10%, with latency latency_vad + 57 ms; Smart Turn V2 reaches Complete 83.65%, Incomplete 54.85%, Avg 69.25%, with latency latency_vad + 15 ms; and SoulX-Duplug reaches Complete 77.67%, Incomplete 88.96%, Avg 83.32%, with 240 ms theoretical latency. In Chinese, Easy Turn reaches Complete 96.33%, Incomplete 97.67%, Avg 97.00%, with latency latency_vad + 263 ms; Paraformer + TEN Turn Detection reaches Complete 86.67%, Incomplete 89.30%, Avg 87.99%, with latency latency_vad + 204 ms; Smart Turn V2 reaches Complete 78.67%, Incomplete 62.00%, Avg 70.34%, with latency latency_vad + 27 ms; and SoulX-Duplug reaches Complete 89.33%, Incomplete 79.33%, Avg 84.33%, with 240 ms theoretical latency.

A notable pattern in these reported numbers is that SoulX-Duplug is especially strong on English incomplete-utterance detection, with 88.96%, while being weaker on complete utterances than the strongest non-streaming baseline. The paper interprets this as evidence that the text-guided streaming formulation preserves most turn-detection accuracy while offering low and stable latency under realistic streaming deployment.

At the system level on Bilingual Full-Duplex-Bench, SoulX-Duplug is reported as obtaining the best overall score in turn management and highly competitive latency. In English, the reported results are: Pause Handling TOR 0.352; Turn Taking TOR 0.933 and RL 0.511 s; User Backchannel RsR 0.740; User Interruption v1 TOR 0.970 and RL 0.773 s; User Interruption v1.5 RpR 0.770, Stop Latency 0.450 s, and RL 1.030 s; Overall Score ACC 0.812; and Overall latency 0.691 s. In Chinese, the reported results are: Pause Handling TOR 0.038; Turn Taking TOR 0.994 and RL 0.767 s; User Backchannel RsR 0.800; User Interruption v1 TOR 0.994 and RL 1.089 s; User Interruption v1.5 RpR 0.830, Stop Latency 0.380 s, and RL 1.150 s; Overall score 0.916; and Overall latency 0.847 s.

The paper interprets these results as indicating a balanced operating point. It contrasts SoulX-Duplug with aggressive continuous-output models such as dGSLM, PersonaPlex, and Moshi, which score very well on turn-taking but perform poorly on Pause Handling TOR, indicating frequent interruption during pauses. Relative to Gemini Live, SoulX-Duplug is described as much better on turn-taking TOR and much lower in latency, while Gemini is more conservative on Pause Handling. On interruption behavior, the paper notes that User Interruption v1 and v1.5 use the same data but different criteria, and that systems such as Freeze-Omni, Moshi, and Gemini can appear stronger under one protocol than another. This supports the benchmark’s use of multiple interruption metrics rather than a single score.

Practical latency measurements are also reported for deployment-style interpretation. On the Turn Taking task, the latency attributable to SoulX-Duplug itself is estimated by subtracting LLM and TTS first-packet latency from total speak latency. The reported numbers are 205 ms for SoulX-Duplug EN, 295 ms for SoulX-Duplug ZH, 343 ms for FlexDuo, and 500 ms for VAD. The text summarizes SoulX-Duplug’s practical average latency as about 250 ms across English and Chinese, close to the theoretical 240 ms.

7. Reproducibility, caveats, and benchmark significance

The paper states that SoulX-Duplug and SoulX-Duplug-Eval are open-sourced, with a GitHub link provided in the abstract: https://github.com/Soul-AILab/SoulX-Duplug. The text provided does not specify licensing terms, benchmark packaging, file formats, evaluator APIs, split files, hidden labels, or submission-server infrastructure. From the description, a new model would be evaluated in two ways: as a state predictor on the bilingual Easy Turn testset using accuracy and latency, and as a full-duplex system on the bilingual Full-Duplex-Bench tasks using official metrics such as TOR, RL, RsR, RpR, and SL (Yan et al., 16 Mar 2026).

Several caveats are explicit. Some benchmark subsets are synthetic, using ChatGPT-generated text and TTS-generated speech instead of naturally recorded bilingual conversations. This makes the benchmark standardized and scalable, but may make it less representative of messy conversational acoustics. Chinese state-prediction training also relies partly on a large in-house corpus rather than a public resource, which affects strict reproducibility of the model, though not necessarily of the benchmark itself. Cross-system comparison is incomplete in Chinese because only Freeze-Omni and SoulX-Duplug are reported at the system level. Some latency comparisons, such as FlexDuo latency, are taken from the FlexDuo paper rather than from identical re-evaluation.

The ablation results help clarify what the benchmark is sensitive to. On Chinese Easy Turn, full SoulX-Duplug reaches 89.33% complete, 79.33% incomplete, and 84.33% average accuracy. Without ASR pretraining, it reaches 83.00%, 78.00%, and 80.50%. Without teacher-forced inference, it reaches 78.33%, 68.67%, and 73.50%. These results support the benchmark-relevant claim that semantic representations and explicit text guidance materially affect state prediction.

Taken together, SoulX-Duplug-Eval is best characterized as a bilingual extension protocol over Easy Turn and Full-Duplex-Bench rather than an entirely new benchmark family. Its main contribution lies in standardizing evaluation across state-level and system-level full-duplex tasks, aligning measurement with realistic streaming deployment, and reducing language bias by pairing English and Chinese test coverage. The benchmark’s reported findings suggest that it rewards balanced turn management rather than merely aggressive turn capture, making it a targeted instrument for evaluating semantic, low-latency, full-duplex dialogue control.