BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM

Abstract: Real-time, full-duplex speech interaction is a key feature of next-generation spoken chatbots, allowing the model to listen and speak at the same time and to handle natural phenomena such as overlap, hesitation, and barge-in. Existing speech LLMs (SpeechLMs) such as LLaMA-Omni and GLM-4-Voice are still turn-based and rely on an external Voice Activity Detection (VAD) module to mark the end of the user's turn, which fundamentally limits their interactive ability. In this paper, we introduce BayLing-Duplex, a native full-duplex SpeechLM where a single autoregressive LLM decides when to listen, when to speak, and when to stop, with no auxiliary turn-taking module. The design adds only a few special tokens to the standard vocabulary, so it transfers across LLMs and reuses existing training and serving stacks with no architectural adaptation. Starting from the public GLM-4-Voice checkpoint and using only 400K full-duplex samples for fine-tuning followed by a lightweight DPO stage, BayLing-Duplex reaches 92% turn-taking success and 100% interruption success on InstructS2S-Eval, while improving the speech-response score from 2.17 to 3.39 over Moshi. BayLing-Duplex also matches or surpasses its turn-based counterpart on Llama Questions, Web Questions, and Alpaca-Eval, showing that simultaneous listen-and-speak modeling does not sacrifice response quality.

• The paper presents a novel approach that minimally converts turn-based SpeechLMs into native full-duplex dialogue systems using a multi-channel interleaved sequence design.
• It leverages supervised fine-tuning and direct preference optimization to enhance turn-taking and interruption handling while maintaining high response quality.
• Empirical results show significant improvements in turn-taking success, interruption performance, and QA accuracy compared to existing models.

BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM

Motivation and Problem Statement

Full-duplex, real-time speech interaction is essential for the next generation of conversational agents, enabling seamless dialogue that matches nuanced human conversational phenomena such as overlap, hesitations, and barge-in. Current Speech LLMs (SpeechLMs), despite advances like GLM-4-Voice and Moshi, are fundamentally constrained by a turn-based paradigm due to reliance on external Voice Activity Detection (VAD) or explicit turn-taking modules, resulting in suboptimal interaction flow, high latency, and loss of paralinguistic features. Native full-duplex models circumvent these issues but typically require extensive pretraining on large speech corpora, imposing prohibitive computational costs. BayLing-Duplex proposes a minimalistic conversion of turn-based SpeechLMs to full-duplex, leveraging structured fine-tuning with minimal architectural alteration, thus preserving model generality and scalability.

Model Architecture and Multi-Channel Sequence Design

BayLing-Duplex adopts the GLM-4-Voice backbone, comprising a Whisper-based speech tokenizer, a 9B-parameter decoder-only transformer LLM, and a flow-matching speech decoder. The key innovation is a multi-channel interleaved sequence representation, in which dialogue is structured as blocks of user speech, assistant text monologue, and assistant speech.

Figure 1: Multi-channel interleaved sequence in BayLing-Duplex, showing block-wise alignment of user speech, assistant text monologue, and assistant speech streams; this layout enables seamless turn-taking and interruption modeling as token prediction.

Each block consists of $N$ user-speech tokens (conditioning only), $M$ assistant text tokens (planning, state control), and $N$ assistant-speech tokens (generative); blocks are temporally aligned and interleaved. The text channel embeds four dialogue-state tokens: [SILENCE], [ASSISTANT], [PAD], [EPAD], encoding silence/wait, reply onset, textual completion, and speech completion, respectively. This enables all turn-taking, interruption, and transition decisions to be managed as ordinary next-token prediction within the standard LLM vocabulary, eliminating the need for auxiliary classifiers, heuristic turn classifiers, or state machines. The assistant's speech can be interrupted or resumed dynamically, supporting barge-in and fluid overlap.

Training and Fine-Tuning

BayLing-Duplex initialization begins with the GLM-4-Voice checkpoint, followed by two stages:

Supervised Fine-Tuning (SFT): The interleaved multi-channel format is supervised using 400K synthesized full-duplex dialogue samples. Loss is computed only for assistant text and speech positions, applying token-type weighting to counter the dominance of [SILENCE], ensuring robust learning of rare role tokens (turn-state markers).

Direct Preference Optimization (DPO): A lightweight preference optimization is performed using positive samples (correct timing) and negative samples (wrong timing with altered gaps or delayed interruption), maintaining a Kullback–Leibler regularization against the SFT policy to preserve response quality. This stage explicitly optimizes temporal behavior relevant to turn-taking and interruption without degradation of underlying generative ability.

Inference and Real-Time Decoding

Inference proceeds block-wise, synchronizing reception of user speech tokens, emission of assistant text tokens (masked to text/state vocabulary), and emission of assistant speech tokens (masked to speech vocabulary). Strict autoregressive decoding ensures that barge-in interrupts are handled smoothly: when a user intervenes mid-block, the assistant completes in-flight speech generation before re-conditioning on updated user input. No masking is applied during training, relying on natural suppression by cross-entropy; masking during inference is required to avoid error propagation between channels.

Data Construction and Evaluation

Training data leverages a multi-turn dialogue corpus, synthesized from Alpaca and UltraChat datasets, and converted into full-duplex samples with controlled gaps for turn-taking and deterministic interruptions. Data augmentation produces realistic barge-in and varied temporal structures.

Evaluation employs tasks including spoken question answering (QA), turn-taking, and interruption on Llama Questions, Web Questions, and InstructS2S-Eval. Metrics include turn-taking success rate (SR@3s), speech-response score (S2S), interruption overlap (Ovl), interruption success rate (ISR@2s), and QA exact-match accuracy. Speech-response scoring utilizes GPT-4o judgments, and segmentation analysis is performed via Whisper and Silero VAD.

Experimental Results

BayLing-Duplex demonstrates strong empirical performance on both content and interaction metrics. Turn-taking reaches 92% success within 3s; interruption success achieves 100% with overlap reduced to 1.10s (from Moshi's 2.07s); speech-response scores improve from 2.17 to 3.39. QA accuracy on Llama Questions/Web Questions improves markedly—46.0%/18.1% versus Moshi's 21.0%/9.2%. Critically, full-duplex modeling does not compromise response quality: the duplex-trained model matches or slightly outperforms turn-based baselines in answer accuracy and Alpaca-Eval S2S score, confirming that fine-tuned temporal modeling does not dilute semantic or fluency attributes.

Ablation and Hyperparameter Analysis

Ablation studies validate the necessity of appropriately weighting [SILENCE] and role tokens during SFT; uniform weighting leads to pathological silence or degenerate interruption. DPO hyperparameter sweeps reveal maximum interaction quality with $\beta=0.5$ and $\lambda_{ftx}=0.5$, confirming stability of interruption handling (ISR@2s invariant at 100%) and peak turn-taking success at these settings.

Implications and Future Directions

BayLing-Duplex illustrates that full-duplex conversational ability can be achieved with minimal architectural change and modest data requirements, leveraging structured sequence design and direct preference optimization. The methodology is highly transferable, supporting adaptation to other autoregressive LLMs, and enabling efficient leveraging of existing training stacks.

Practically, this advances deployment of intelligent voice assistants capable of handling natural conversational dynamics without reliance on dedicated VAD, turn classifiers, or complex input segmenters. Theoretically, it highlights the power of token-level dialogue-state modeling, encoding all interaction logic within learned vocabulary.

Limitations include reliance on synthesized, noise-free audio for training and evaluation, constrained latency due to block granularity, and inherited biases from the backbone model. Extensions should investigate robustification against real-world conditions (noise, multi-speaker, backchannels), fine-grained latency trade-offs via block size exploration, and adaptation to rare languages and heterogeneous speech environments.

Conclusion

BayLing-Duplex establishes a streamlined approach to native full-duplex speech dialogue, unifying all turn-taking and interruption phenomena as next-token prediction in a single autoregressive LLM augmented with minimal token additions. Experimental results substantiate high interaction fidelity and strong response quality with only moderate full-duplex data and lightweight optimization. This framework sets a practical and scalable precedent for future speech dialogue models, supporting real-time, flexible conversational AI.