---
title: Full-Duplex Spoken Language Models
url: https://www.emergentmind.com/topics/full-duplex-spoken-language-models-fd-slms
type: topic
---

# Full-Duplex Spoken Language Models

Full-Duplex Spoken Language Models (FD-SLMs) are speech-language models designed for synchronous spoken interaction: they listen and speak simultaneously, rather than alternating between a completed listening phase and a subsequent speaking phase. In the recent literature, they are distinguished from half-duplex systems by their ability to support overlapping speech, backchanneling, rapid turn-taking, and interruption handling under real-time constraints, and are increasingly framed as a shift from turn-based spoken interfaces toward synchronous dialogue [2509.14515]. A recurrent theme across the field is that FD-SLMs must model not only *what* to say but also *when* to say it; temporal coordination, semantic coherence, and acoustic robustness are therefore treated as coupled but non-identical design problems [2509.26388].

## 1. Field definition and taxonomy

The survey literature defines FD-SLMs as systems that can “listen and speak simultaneously” while supporting natural turn-taking, overlap, backchannels, and interruption handling [2509.14515]. This is often contrasted with a sequential “listen-think-speak” loop, which remains the dominant operating mode of half-duplex systems. In that formulation, full-duplex behavior is not reducible to low latency alone; it requires concurrency, floor control, and observable speaker-specific timing.

A common taxonomy separates **Engineered Synchronization** from **Learned Synchronization**. Engineered approaches remain modular, adding explicit synchronization logic to a spoken dialogue pipeline. Learned approaches instead train synchronization as an intrinsic property of the model, typically through end-to-end or native full-duplex training [2509.14515]. The literature also distinguishes **dual-channel SLMs**, which maintain separate listening and speaking channels, from **time-multiplexing SLMs**, which predict when to speak versus remain silent, often through a state-prediction mechanism [2509.26388].

| Family | Defining mechanism | Illustrative instances |
|---|---|---|
| Engineered Synchronization | External or internal control logic for listen/speak arbitration | FlexDuo, Semantic VAD, VITA-1.5, Freeze-Omni, MinMo, Neural-FSM, Mini-Omni2 |
| Learned Synchronization | End-to-end learning of duplex behavior from synchronous data | dGSLM, Moshi, SyncLLM, NTPP, SALMONN-omni, SALM-Duplex |
| Semi-cascaded duplex systems | Modular speech stack with a multimodal controller for unit-level decisions | Unit-Based Agent |

The boundary between these categories is not always sharp. The “Unit-Based Agent” is explicitly described as **not a full end-to-end FD-SLM in the usual sense**, but as a semi-cascaded, plug-and-play full-duplex dialogue framework built around a multimodal LLM plus VAD, ASR, and TTS [2601.20230]. This suggests that the field encompasses both native duplex models and pragmatic duplex control systems, provided that simultaneous listening and speaking remain first-class behavioral targets.

## 2. Formal objectives and architectural patterns

At the formal level, the survey presents two high-level views of FD-SLMs. A joint formulation models aligned environment and agent sequences as
$$
P(S^\mathcal{E}, S^\mathcal{A})=\prod_{t=1}^{T} P\!\left(e_t,a_t \mid S^\mathcal{E}_{<t}, S^\mathcal{A}_{<t}\right),
$$
which motivates **Next-Token-Pair Prediction (NTPP)**. A conditional formulation instead models agent output under concurrency and self-conditioning,
$$
a_t \sim P\!\left(a_t \mid S^\mathcal{E}_{\le t},\, S^\mathcal{A}_{<t};\,\theta\right),
$$
with the explicit real-time requirement that compute time remain below \(200\,\mathrm{ms}\) [2509.14515]. Both formulations treat duplex interaction as aligned sequence modeling rather than post hoc turn segmentation.

Several architectural lineages implement this idea differently. One line serializes dialogue events into a causal token stream. “A Full-duplex Speech Dialogue Scheme Based On Large Language Models” reduces dialogue control to next-token prediction over a live “virtual tape,” while a two-state neural FSM interprets control tokens such as `[S.SPEAK]`, `[S.LISTEN]`, `[C.SPEAK]`, and `[C.LISTEN]` [2405.19487]. This is an engineered but tightly integrated approach: the LLM generates both semantic tokens and floor-control commands.

A second line makes the model synchronous with wall-clock time. “Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents” modifies Llama3-8B to generate HuBERT discrete tokens in fixed-duration time chunks, with periodic speaker tags `[S0]` and `[S1]` serving as synchronization markers [2409.15594]. The model predicts interleaved chunks from both speakers and remains robust to simulated Internet-scale latencies of **160 ms**, **200 ms**, and **240 ms**.

A third line uses explicit listening and speaking channels inside a native speech model. “Language Model Can Listen While Speaking” proposes **LSLM**, an end-to-end architecture with a token-based decoder-only TTS for speech generation, a streaming SSL encoder for incoming audio, and three fusion strategies: early, middle, and late fusion. The paper reports that **middle fusion** achieves the best balance between speech generation and real-time interaction [2408.02622].

Native dual-channel architectures also appear in multilingual transfer. “Towards a Japanese Full-duplex Spoken Dialogue System” adapts **Moshi** into **J-Moshi**, retaining **Mimi** as a frozen neural audio codec operating at **12.5 Hz** and training only the **RQ-Transformer** over parallel model and user audio-token layers plus a text layer [2506.02979]. By contrast, “Hierarchical Acoustic-Semantic Modeling” argues that forcing acoustic generation and semantic reasoning to share a deep parameter space causes gradient conflict; its **Lychee-FD** separates deeper layers into specialized text, speech, and control heads while preserving shared shallow processing and a dedicated semantic alignment channel [2607.06540].

## 3. Data regimes and corpus construction

Data scarcity is a persistent structural constraint. The survey identifies **synchronous data scarcity** as the field’s most critical bottleneck, noting that FD-SLMs require synchronized multi-channel dialogue with natural overlap, interruptions, and speaker-specific timing [2509.14515]. Much of the recent infrastructure work therefore focuses on converting imperfect public or monaural data into duplex-ready supervision.

“Sommelier” addresses preprocessing rather than model design. It standardizes audio to **16kHz, 16-bit, Mono**, normalizes loudness to **-20dBFS**, segments long recordings with **Silero VAD**, uses **Sortformer** diarization, applies overlap separation with speaker embedding matching, and performs ASR ensembling with **Whisper**, **Canary**, and **Parakeet** under a **ROVER** scheme [2603.25750]. The pipeline is explicitly designed not to discard overlap. It reports **RTF 0.1746** for a 120-second sample on one A100 (80GB), dropping to **0.133** without FlowSE denoising, and to **0.0443 per GPU** under three concurrent processes. A LoRA fine-tune of Moshi on approximately **83 hours** of Sommelier-processed data improves Backchanneling, Smooth Turn-Taking, and User Interruption on Full-Duplex-Bench 1.0 [2603.25750].

“DuplexChat” scales this logic to web data. **DuplexChat-Pipe** collects podcast feeds from **PodcastIndex**, filters by language and metadata, segments two-speaker dialogue with **pyannote/speaker-diarization-community-1**, and uses **DialogueSidon** for speech separation and restoration [2607.04941]. The resulting corpus contains **282,634 hours of English** and **132,723 hours of Japanese**, organized as speaker-separated stereo tracks. Turn-taking analysis on sampled dialogues reports **turn exchanges per minute** of **10.5** for Japanese and **6.2** for English, **backchannels per minute** of **5.6** and **3.1**, and **simultaneous speech** of **21%** and **10%**, respectively, indicating that the corpus preserves duplex conversational structure [2607.04941].

Synthetic data remain central when real duplex corpora are insufficient. **SyncLLM** uses **212k hours of synthetic spoken dialogue data** plus **2k hours of real-world spoken dialogue data** [2409.15594]. **J-Moshi** pre-trains on about **60,000 hours** from **J-CHAT** and later augments fine-tuning with **602 hours** of multi-stream-TTS synthetic stereo dialogue, producing **J-Moshi-ext** [2506.02979]. **Lychee-FD** reports a synthetic training set of roughly **140K full-duplex dialogue instances**, while **FLAIR** uses **530K hours** of speech continuation data, **70K hours** of instruction-following QA data, and **20K hours** of ASR-QA data [2607.06540][2603.17837]. This suggests that FD-SLM progress remains tightly coupled to large-scale data synthesis and preprocessing pipelines, not only to architectural novelty.

## 4. Benchmarking and evaluation methodology

Evaluation has become increasingly specialized because token-level likelihood alone does not measure duplex competence. The survey organizes FD-SLM evaluation into four pillars: **Temporal Dynamics**, **Behavioral Arbitration**, **Semantic Coherence**, and **Acoustic Performance** [2509.14515]. Recent benchmarks instantiate different subsets of that framework.

| Benchmark | Primary focus | Key measurements |
|---|---|---|
| Game-Time | Temporal instruction following | Basic tasks, temporal constraints, LLM judge, word-level timing |
| Zero-Buffer Benchmark | Abrupt interruption comprehension | Correctness, IWOR |
| MTR-DuplexBench | Multi-round duplex evaluation | Dialogue quality, conversational features, instruction following, safety |
| Full-Duplex-Bench / 1.5 | Conversational interaction behaviors | Turn-taking, backchanneling, pause handling, interruption, latency |

**Game-Time** is specifically designed for temporal dynamics. It contains **1,475 test instances**, divided into **700 Basic task samples** and **775 Advanced task samples**, and formalizes each instruction-following instance as a base task \(t\) plus a set of constraints \(\mathcal{C}\) [2509.26388]. Basic families include Sequence, Repeat, Compose, Recall, Open-Ended, and Role-Play; advanced families add constraints such as Time-Fast, Time-Slow, Time-Silence, Tempo-Interval, Tempo-Adhere, Simul.-Shadow, and Simul.-Cue. Its evaluation uses a dual-channel setup with word-level timing and an **LLM judge**; human validation on advanced tasks reports **Human–LLM Spearman’s \(\rho = 0.677\)** and **Human–LLM Pearson’s \(r = 0.675\)** [2509.26388].

The **Zero-Buffer Benchmark (ZBB)** isolates a different failure mode: whether a model can understand the very first word of an abrupt interruption. It evaluates **Correctness** and **Initial Word Occurrence Rate (IWOR)** on **100 total ZBB queries**, with the crucial subject word placed at utterance onset [2606.11386]. By contrast, **MTR-DuplexBench** addresses multi-round interaction. It segments continuous duplex conversations into discrete turns using **Whisper-timestamped**, **Silero VAD**, GPT-4o segmentation repeated **six times**, clustering with at least **30% time overlap**, and majority-vote filtering [2511.10262]. It then evaluates **Dialogue Quality**, **Conversational Features**, **Instruction Following**, and **Safety** over multiple rounds rather than single-shot events.

A broader methodological shift is also visible in judge design. Several papers prefer LLM-based evaluation over rigid rule-based scoring because spoken interaction frequently contains preambles, self-repairs, or contextually appropriate variation that simple scripts mis-score [2509.26388]. This has made transcript quality, word-level timing extraction, and segmentation stability central components of benchmark validity.

## 5. Recurrent failure modes

A first recurrent failure is **temporal myopia**. Game-Time reports that, although state-of-the-art models handle many basic tasks well, “nearly all models degrade substantially under temporal constraints,” with especially poor performance on **Time-Silence**, **Tempo** tasks, and synchronized overlap tasks [2509.26388]. The paper’s central conclusion is that current systems can sometimes modulate overall speaking rate but still lack robust **time-awareness** and **full-duplex interaction capability** when exact timing matters.

A second failure is **state inertia**. “Overcoming State Inertia in Full-Duplex Spoken Language Models via Activation Steering” argues that FD-SLM hidden states alternate between a **generative state** and a **perceptive state**, but may lag when the user interrupts [2606.11386]. On PersonaPlex, perception affinity \(\mathcal{S}_{\text{perc}}(t)\) may take **7–8 timesteps**—roughly **0.6 seconds**—to recover under interruption. This delay causes models to miss the beginning of the incoming utterance, especially its first semantic word.

A third failure is **modality interference**. Lychee-FD reports that, in native end-to-end full-duplex designs, acoustic and semantic objectives exhibit positive gradient cosine similarity in early layers but negative similarity in deeper layers, indicating destructive optimization when the modalities are forced to share a deep parameter space [2607.06540]. The same paper argues that temporally aligned text supervision with extensive padding weakens semantic gradients and allows acoustic learning to dominate, producing “knowledge degradation” and reduced semantic integrity.

A fourth failure is **conditioning corruption under acoustic interference**. IRAF targets a dual-channel setting where interfering speakers leak into the user microphone, are encoded as part of the user query, and corrupt LLM conditioning [2606.06559]. The reported symptoms are **unstable turn-taking**, **false barge-in / mistimed responses**, and degraded response quality. The paper’s results show that clean-training-only baselines can collapse under interfering-speaker conditions, with BLEU near zero and very low success rate.

A fifth failure is **multi-round instability**. MTR-DuplexBench evaluates Moshi and reports a dialogue-quality **GPT-score of 1.94/5.0**, declining success rates and increasing latency across rounds, plus a drop in instruction-following from **68.0** at round 1 to **41.9** averaged over rounds 1–10 under smooth turn-taking [2511.10262]. This indicates that single-round fluency does not imply multi-round robustness.

These failure modes also clarify several misconceptions. One misconception is that FD-SLMs are merely low-latency variants of ordinary spoken chat systems; the temporal benchmarks show that precise silence insertion, tempo adherence, synchronized overlap, and abrupt interruption comprehension are separable competencies [2509.26388][2606.11386]. Another is that semantic weakness is explained only by data scarcity; Lychee-FD argues that optimization geometry itself is a root cause [2607.06540].

## 6. Remedial strategies and emerging directions

A major response has been to separate semantics from timing without abandoning full duplex. **TurnGuide** introduces turn-level text guidance: assistant speech is segmented into dialogue turns, and turn-level text is generated before speech output, rather than injected token by token [2508.07375]. The paper reports **more than 30% performance gain** in GPT-score over baselines while keeping Pearson correlations for turn-taking events comparable to SCI and Moshi TS. **FLAIR** takes a different route, replacing silence-token prediction during user speech with latent reasoning. Its latent update
$$
z_{t-1} = \mathrm{Softmax}(y^{txt}_{t-1})E
$$
allows “think-while-listening” under strict causality, and the reported gains include **Llama Questions: 73.0 → 78.0** and **MMSU: 50.2 → 56.2** for the latent-reasoning variant [2603.17837].

Another response is targeted post-training alignment. **ASPIRin** projects the text vocabulary into a binary action space—**active speech** versus **inactive silence**—and applies GRPO to that projected policy rather than to raw lexical tokens [2604.10065]. This improves interactivity while reducing duplicate 2-gram repetition from **0.117** to **0.054** and duplicate 3-gram repetition from **0.072** to **0.029** relative to standard GRPO. **Multi-Faceted Interactivity Alignment** instead uses axis-specific rewards for **pause handling**, **turn-taking**, **backchanneling**, and **user interruption**, plus an LLM-based quality reward to prevent semantic degradation [2606.11167]. **DuplexPO** formulates **Factorized Conversational Dynamics Reward (FCDR)** over high-impact windows and argues that the apparent intelligence–dynamics trade-off is not fundamental: on Fisher, it improves **Onset MAE** from **0.98** to **0.69**, **turn-taking yield rate** from **92.1** to **98.7**, and **backchannel yield rate** from **57.1** to **100.0**, while preserving or slightly improving reasoning and instruction following [2607.07148].

Robustness interventions are becoming more localized and causal. **IRAF** uses a target-speaker embedding from **ECAPA-TDNN** plus a causal transformer gate
$$
g_t = 2\times\text{Sigmoid}(f(s, X_{\leq t}|\psi))\in [0,2]
$$
to rescale user frames before fusion with agent embeddings [2606.06559]. On **InstructS2S-200K** with interfering speakers only, it improves BLEU from **9.64** to **13.76**, sBERT from **0.47** to **0.58**, and response success from **69.2%** to **91.0%**, while reducing response latency from **0.97 s** to **0.82 s**. **Activation steering** provides a complementary training-free remedy for state inertia: on PersonaPlex, interruption-time correctness rises from **0.28** to **0.45**, and IWOR from **0.40** to **0.72** [2606.11386].

Architectural remediation is also shifting toward selective decoupling rather than full separation. Lychee-FD reports **+7.4% S→S accuracy**, **+8.8% S→T accuracy**, and a **28.5% average improvement** on FullDuplexBench 1.5 over system-level baselines, while preserving inference efficiency through shallow sharing and deeper specialization [2607.06540]. This suggests that the intelligence–dynamics trade-off can sometimes be reduced by better optimization structure rather than by sacrificing concurrency.

Two frontier directions recur across the literature. The first is **multilingual and culturally specific duplex modeling**. J-Moshi shows that Japanese FD-SLMs can learn a turn-taking profile closer to Japanese ground truth, with **Overlap 5.0s** for J-Moshi versus **1.2s** for English Moshi and **8.1s** for Japanese ground truth [2506.02979]. The second is **open-mic robustness**. Lychee-FD explicitly notes continued weakness in real-world multi-speaker settings and proposes richer open-mic datasets, intent detection, prosodic cues, and more selective interruption mechanisms as future work [2607.06540].

Taken together, the contemporary literature presents FD-SLMs as a field moving from proof-of-concept overlap handling toward a more granular science of conversational timing, semantic planning, robustness, and evaluation. A plausible implication is that future progress will depend less on any single architectural template than on the co-development of duplex-ready corpora, temporally explicit benchmarks, and mechanisms that disentangle floor control from semantic generation without breaking strict causality.

Source: https://www.emergentmind.com/topics/full-duplex-spoken-language-models-fd-slms