BayLing-Duplex: Full-Duplex Speech Dialogue Model

• BayLing-Duplex is a full-duplex speech dialogue model that leverages a single autoregressive LLM and four specialized dialogue-state tokens to manage simultaneous listening and speaking.
• It employs a multi-channel interleaved block design combining user speech, assistant text (inner monologue), and assistant speech for balanced latency and stability.
• A two-stage training process, integrating supervised fine-tuning and Direct Preference Optimization, ensures robust turn-taking, interruption handling, and high-quality spoken responses.

BayLing-Duplex is a native full-duplex speech dialogue model built on a single autoregressive LLM and introduced as an alternative to turn-based SpeechLMs that depend on an external Voice Activity Detection module to determine user turn completion (Fang et al., 12 Jun 2026). In the formulation reported for BayLing-Duplex, the model decides internally, through ordinary next-token prediction, when to listen, when to speak, and when to stop, while retaining the standard GLM-4-Voice speech stack and adding only four special dialogue-state tokens. The reported objective is real-time speech interaction that can accommodate overlap, hesitation, and barge-in without auxiliary turn-taking machinery, and the empirical claim is that this simultaneous listen-and-speak formulation reaches strong turn-taking and interruption performance without sacrificing spoken question answering or response quality (Fang et al., 12 Jun 2026).

1. Problem formulation and conversational setting

BayLing-Duplex is positioned against speech LLMs that operate in a turn-based manner. The paper identifies GLM-4-Voice and LLaMA-Omni as representative systems in this category and states that such models rely on an external VAD module to mark the end of the user’s turn (Fang et al., 12 Jun 2026). Within that design, interaction quality is constrained by acoustic end-of-turn detection: false positives can cause premature assistant speech, false negatives can delay responses, and purely acoustic VAD does not model dialogue semantics.

The model is therefore framed around “native full-duplex” dialogue. In the paper’s terminology, full-duplex speech interaction means that the assistant can listen and speak at the same time, rather than alternating between complete user utterances and complete model replies. This formulation is intended to capture conversational phenomena that are difficult to express in rigid turn boundaries, including mid-utterance pauses, interruption or barge-in, brief overlap, and response stopping conditioned on incoming speech (Fang et al., 12 Jun 2026).

A common misconception in this area is that full-duplex behavior necessarily requires a dedicated turn-taking controller, a separate scheduler, or VAD at inference time. BayLing-Duplex explicitly argues the opposite: turn-taking, interruption, and stopping are represented directly in the token stream of a single autoregressive LLM, with no auxiliary classification head, no attention-mask trick, and no state machine (Fang et al., 12 Jun 2026). This suggests that dialogue control can be cast as an in-vocabulary prediction problem rather than a distinct control subsystem.

2. Backbone architecture and multi-channel sequence design

BayLing-Duplex starts from the public GLM-4-Voice checkpoint and retains its basic speech architecture. The backbone includes a speech tokenizer, a decoder-only LLM, and a speech decoder. The speech tokenizer is described as a modified Whisper-large-v3 encoder plus vector quantizer; input waveforms are at 16 kHz; speech tokens are produced at 12.5 Hz, corresponding to one token every 80 ms; the LLM is a 9B decoder-only Transformer initialized from GLM-4-9B; and the speech decoder uses a flow-matching model with a HiFi-GAN vocoder adapted from CosyVoice (Fang et al., 12 Jun 2026). The paper emphasizes that BayLing-Duplex adds no new modules or heads.

The principal modeling device is a multi-channel interleaved sequence with three aligned channels: a user speech channel, an assistant text channel, and an assistant speech channel. These channels are interleaved in fixed-size blocks. The paper defines block sizes with $N=10$ speech tokens for each speech channel and $M=5$ text tokens for the text channel. Because speech tokens arrive at 12.5 Hz, one block spans $\Delta t = 0.8 \text{ s}$ (Fang et al., 12 Jun 2026).

This block structure induces the model’s latency-stability trade-off. The paper states that if $N$ is too small, there are too few text slots per block and turn-taking becomes unstable or jittery; if $N$ is too large, response latency increases. The selected 0.8 s block duration is therefore presented as a compromise between responsiveness and stability (Fang et al., 12 Jun 2026).

An important architectural distinction is that the assistant text channel functions as an internal planning stream. It is not rendered to the end user. Instead, it conditions assistant speech generation within the same block. The paper characterizes this channel as an “inner monologue,” initially filled with a silence marker and overwritten when the model decides that speech should begin (Fang et al., 12 Jun 2026). This design couples semantic planning and speech emission inside a single autoregressive trajectory rather than separating them into pipeline stages.

3. Dialogue-state vocabulary and autoregressive control semantics

BayLing-Duplex adds four special dialogue-state tokens to the existing vocabulary: [SILENCE], [ASSISTANT], [PAD], and [EPAD] (Fang et al., 12 Jun 2026). These tokens encode dialogue control directly in the LLM’s output space.

Their operational semantics are specified as follows. [SILENCE] means the assistant stays silent. [ASSISTANT] marks the onset of an assistant reply. [PAD] indicates that textual content has already been written while speech is still being emitted. [EPAD] marks the condition in which both text and speech for the current reply are complete (Fang et al., 12 Jun 2026). The text channel therefore carries not only internal content planning but also explicit state transitions for dialogue timing.

The paper’s central design claim is that these tokens are sufficient for a single autoregressive LLM to jointly model user speech understanding, internal text planning, assistant speech generation, turn-taking, and interruption timing (Fang et al., 12 Jun 2026). In this view, dialogue management is not an external policy layered on top of generation; it is part of the same token prediction problem that produces linguistic and acoustic outputs.

Because the assistant cannot react before consuming current user audio, the training setup introduces a one-block causal offset. Assistant text and speech associated with block $b$ correspond to the wall-clock interval $[(b+1)\Delta t,\; (b+2)\Delta t)$, allowing the model to condition on the current user block while producing the next assistant block (Fang et al., 12 Jun 2026). This offset is a structural component of the duplex formulation: it preserves causal consistency while maintaining simultaneous listen-and-speak behavior at the interaction level.

4. Data construction, fine-tuning stages, and timing optimization

The reported training corpus begins from a 200K multi-turn speech-to-speech corpus associated with LLaMA-Omni 2 / Fang et al. (2025), derived from Alpaca and UltraChat, rewritten with Llama-3.3-70B-Instruct, and synthesized into speech with CosyVoice zero-shot voice cloning (Fang et al., 12 Jun 2026). BayLing-Duplex converts each dialogue into two full-duplex scenario types.

The first type is turn-taking data. Here, a 0.8 s gap is inserted from the end of user speech to the start of assistant response, and the gap between assistant completion and the next user utterance is sampled from $\mathrm{Uniform}(0.5, 3.0)\text{ s}$ (Fang et al., 12 Jun 2026). The second type is interruption data, in which the user re-enters while assistant speech is still ongoing and the assistant stops after a reaction delay sampled from $$\delta_{\text{react}} \sim \mathrm{Uniform}(0.8, 2.0)\text{ s}$$ (Fang et al., 12 Jun 2026). The paper reports 200K turn-taking samples and 200K interruption samples, for a total of 400K full-duplex fine-tuning samples mixed 1:1.

Training is divided into two stages. Stage I is supervised fine-tuning. The user speech channel is conditioning-only and does not contribute to the loss; supervision is applied only on assistant text and assistant speech positions (Fang et al., 12 Jun 2026). The paper states that token weighting is necessary because silence tokens dominate the sequence. In particular, it tunes a silence-token weight for [SILENCE] and a role-token weight for [ASSISTANT] and [EPAD] to prevent collapse into silence and to ensure that rare role tokens meaningfully influence optimization (Fang et al., 12 Jun 2026).

Stage II is a lightweight Direct Preference Optimization stage. The paper states that Stage I teaches the format but does not optimize timing strongly enough, so DPO is introduced to prefer earlier speaking and earlier stopping without altering semantic content (Fang et al., 12 Jun 2026). Negative samples are constructed by changing timing only: for turn-taking, the 0.8 s response gap is replaced by a longer gap sampled from $\mathrm{Uniform}(2,5)\text{ s}$; for interruption, the reaction delay is replaced by $M=5$0 (Fang et al., 12 Jun 2026). An auxiliary supervised term on positive samples is retained to preserve generation quality and reduce catastrophic forgetting.

Optimization details reported in the paper are specific. The model fully fine-tunes the LLM while freezing the speech tokenizer and speech decoder. Stage I uses the 400K full-duplex dialogues for 1 epoch with batch size 32, peak learning rate $M=5$1, cosine schedule, and 10% warm-up. Stage II runs for 200 DPO steps with peak learning rate $M=5$2, $M=5$3, $M=5$4, cosine schedule, and 5% warm-up. Implementation uses LLaMA-Factory (Fang et al., 12 Jun 2026).

5. Inference procedure, evaluation protocol, and empirical performance

Inference proceeds block by block. The paper specifies the following sequence: receive $M=5$5 user speech tokens; append them to history; generate $M=5$6 text tokens autoregressively while masking outputs to text and state tokens only; generate $M=5$7 assistant speech tokens autoregressively while masking outputs to speech tokens only; decode and play the generated assistant waveform at offset time $M=5$8; and repeat (Fang et al., 12 Jun 2026). The masking is used only at inference to prevent cross-channel token corruption; during training, the loss itself suppresses invalid token types. The user input is never artificially zero-padded, and if the user barges in mid-block, the current assistant speech finishes for that block and the next block conditions on the new user input (Fang et al., 12 Jun 2026).

Evaluation covers spoken question answering, full-duplex turn-taking, and full-duplex interruption. Spoken QA uses Llama Questions with 300 items and Web Questions with 2032 items. Turn-taking and interruption are evaluated on InstructS2S-Eval, with 199 spoken instructions for turn-taking and 199 two-utterance audios for interruption, where the second utterance begins while the first response is still ongoing (Fang et al., 12 Jun 2026). At inference, temperature is 0.8; assistant audio is transcribed by Whisper-large-v3; and Silero VAD is used only for evaluation, not for model inference (Fang et al., 12 Jun 2026).

The paper reports exact-match accuracy for spoken QA via case-insensitive substring matching between the reference answer and the Whisper transcription. On Llama Questions and Web Questions, Moshi attains 21.0% and 9.2%, BayLing-Duplex (SFT) attains 44.3% and 18.0%, and BayLing-Duplex (+DPO) attains 46.0% and 18.1% (Fang et al., 12 Jun 2026). These numbers are used in the paper to support the claim that the duplex layout does not degrade content understanding.

For full-duplex timing and response quality on InstructS2S-Eval, the paper reports the following results: Moshi obtains TT SR@3s of 71.9, S2S of 2.17, Overlap of 2.07, ISR@2s of 81.9, and Q2 S2S of 2.45; BayLing-Duplex (SFT) obtains 88.9, 3.23, 1.51, 91.4, and 2.95; and BayLing-Duplex (+DPO) obtains 92.0, 3.39, 1.10, 100.0, and 3.27 (Fang et al., 12 Jun 2026). The abstract highlights the same result family in condensed form, stating 92% turn-taking success and 100% interruption success on InstructS2S-Eval, with speech-response score improving from 2.17 to 3.39 over Moshi (Fang et al., 12 Jun 2026).

The paper also addresses the question of whether simultaneous listen-and-speak hurts response quality by comparing BayLing-Duplex with a turn-based supervised baseline trained on the same data and backbone. The reported values are 44.3 versus 45.3 on Llama Questions, 18.0 versus 15.9 on Web Questions, and 3.23 versus 3.16 on Alpaca-Eval S2S (Fang et al., 12 Jun 2026). The authors interpret these results as evidence that full-duplex modeling is on par with, or slightly stronger than, the turn-based counterpart.

Ablation results identify two sensitive design elements. First, uniform token weighting causes collapse toward silence, reducing TT SR@3s to 60.3. The best reported weighting is $M=5$9 and $\Delta t = 0.8 \text{ s}$0, which yields TT SR@3s of 88.9 and S2S of 3.23 (Fang et al., 12 Jun 2026). Second, the best DPO hyperparameter setting is $\Delta t = 0.8 \text{ s}$1 and $\Delta t = 0.8 \text{ s}$2, producing the final best results TT SR@3s = 92.0, S2S = 3.39, and ISR@2s = 100% (Fang et al., 12 Jun 2026).

6. Relation to other duplex research and technical significance

Within speech modeling, BayLing-Duplex occupies a distinct point in the recent duplex literature. DuplexMamba also targets duplex and streaming speech interaction, but it does so with a Mamba-based speech encoder, a speech adapter, a Mamba-based LLM, and a duplex decoding strategy built around the state tokens <response>, <incomplete>, and <ignore> (Lu et al., 16 Feb 2025). DuplexMamba processes 2–3 second intervals and duplicates the fixed-size Mamba state to create an auxiliary decoding branch when new input arrives during generation, enabling interruption handling and non-awakening behavior (Lu et al., 16 Feb 2025). By contrast, BayLing-Duplex retains a Transformer-based GLM-4-Voice backbone, uses four dialogue-state tokens, and emphasizes that no architectural adaptation is required beyond vocabulary extension and fine-tuning (Fang et al., 12 Jun 2026). This suggests two different technical routes to duplex speech dialogue: architecture-level redesign around state-space models, and vocabulary-level retrofitting of an existing autoregressive SpeechLM.

The term “duplex” also has a separate technical lineage in wireless communications. In that literature, duplex refers to simultaneous uplink and downlink operation, interference coupling, and resource allocation rather than conversational speech timing. Inter-user interference coordination for full-duplex cellular systems has been studied through geographical context information and radio-map-based scheduling (Duarte et al., 2016). Flexible duplex in multi-cell networks has been formulated as joint uplink/downlink resource allocation under interference coupling, with algorithms such as SAFP and RMDI (Liao, 2017). In cell-free mMIMO-ISAC, duplexing and deployment have been compared through a unified KLD framework over half-duplex and full-duplex modes under shared and separated access-point deployments (Kloob et al., 24 May 2026). These works are conceptually unrelated to BayLing-Duplex’s speech-dialogue architecture, but they clarify that “duplex” in BayLing-Duplex refers specifically to simultaneous listen-and-speak interaction rather than radio-layer full-duplex operation.

The main technical significance claimed for BayLing-Duplex is practical rather than architectural. The paper argues that full-duplex speech dialogue does not require massive full-duplex pretraining, a separate turn-taking module, external VAD at inference, or a complicated multi-branch architecture (Fang et al., 12 Jun 2026). Instead, a strong existing SpeechLM backbone, a coarse multi-channel block layout, four special tokens, 400K synthetic duplex samples, and a lightweight DPO stage are reported to suffice. A plausible implication is that BayLing-Duplex reframes conversational timing as a supervision and representation problem inside the standard autoregressive stack, making retrofit of turn-based speech models a central design avenue for future duplex systems.