Full-Duplex Spoken Language Models
- Full-Duplex Spoken Language Models (FD-SLMs) are systems that listen and speak simultaneously, enabling natural turn-taking, overlapping speech, and interruption handling in real time.
- They employ both engineered and learned synchronization approaches, such as dual-channel architectures and time-multiplexing strategies, to balance semantic and acoustic objectives.
- Recent research addresses key failure modes like temporal myopia and modality interference while advancing evaluation benchmarks and corpus construction for improved duplex interaction.
Full-Duplex Spoken LLMs (FD-SLMs) are speech-LLMs designed for synchronous spoken interaction: they listen and speak simultaneously, rather than alternating between a completed listening phase and a subsequent speaking phase. In the recent literature, they are distinguished from half-duplex systems by their ability to support overlapping speech, backchanneling, rapid turn-taking, and interruption handling under real-time constraints, and are increasingly framed as a shift from turn-based spoken interfaces toward synchronous dialogue (Chen et al., 18 Sep 2025). A recurrent theme across the field is that FD-SLMs must model not only what to say but also when to say it; temporal coordination, semantic coherence, and acoustic robustness are therefore treated as coupled but non-identical design problems (Chang et al., 30 Sep 2025).
1. Field definition and taxonomy
The survey literature defines FD-SLMs as systems that can “listen and speak simultaneously” while supporting natural turn-taking, overlap, backchannels, and interruption handling (Chen et al., 18 Sep 2025). This is often contrasted with a sequential “listen-think-speak” loop, which remains the dominant operating mode of half-duplex systems. In that formulation, full-duplex behavior is not reducible to low latency alone; it requires concurrency, floor control, and observable speaker-specific timing.
A common taxonomy separates Engineered Synchronization from Learned Synchronization. Engineered approaches remain modular, adding explicit synchronization logic to a spoken dialogue pipeline. Learned approaches instead train synchronization as an intrinsic property of the model, typically through end-to-end or native full-duplex training (Chen et al., 18 Sep 2025). The literature also distinguishes dual-channel SLMs, which maintain separate listening and speaking channels, from time-multiplexing SLMs, which predict when to speak versus remain silent, often through a state-prediction mechanism (Chang et al., 30 Sep 2025).
| Family | Defining mechanism | Illustrative instances |
|---|---|---|
| Engineered Synchronization | External or internal control logic for listen/speak arbitration | FlexDuo, Semantic VAD, VITA-1.5, Freeze-Omni, MinMo, Neural-FSM, Mini-Omni2 |
| Learned Synchronization | End-to-end learning of duplex behavior from synchronous data | dGSLM, Moshi, SyncLLM, NTPP, SALMONN-omni, SALM-Duplex |
| Semi-cascaded duplex systems | Modular speech stack with a multimodal controller for unit-level decisions | Unit-Based Agent |
The boundary between these categories is not always sharp. The “Unit-Based Agent” is explicitly described as not a full end-to-end FD-SLM in the usual sense, but as a semi-cascaded, plug-and-play full-duplex dialogue framework built around a multimodal LLM plus VAD, ASR, and TTS (Yu et al., 28 Jan 2026). This suggests that the field encompasses both native duplex models and pragmatic duplex control systems, provided that simultaneous listening and speaking remain first-class behavioral targets.
2. Formal objectives and architectural patterns
At the formal level, the survey presents two high-level views of FD-SLMs. A joint formulation models aligned environment and agent sequences as
which motivates Next-Token-Pair Prediction (NTPP). A conditional formulation instead models agent output under concurrency and self-conditioning,
with the explicit real-time requirement that compute time remain below (Chen et al., 18 Sep 2025). Both formulations treat duplex interaction as aligned sequence modeling rather than post hoc turn segmentation.
Several architectural lineages implement this idea differently. One line serializes dialogue events into a causal token stream. “A Full-duplex Speech Dialogue Scheme Based On LLMs” reduces dialogue control to next-token prediction over a live “virtual tape,” while a two-state neural FSM interprets control tokens such as [S.SPEAK], [S.LISTEN], [C.SPEAK], and [C.LISTEN] (Wang et al., 2024). This is an engineered but tightly integrated approach: the LLM generates both semantic tokens and floor-control commands.
A second line makes the model synchronous with wall-clock time. “Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents” modifies Llama3-8B to generate HuBERT discrete tokens in fixed-duration time chunks, with periodic speaker tags [S0] and [S1] serving as synchronization markers (Veluri et al., 2024). The model predicts interleaved chunks from both speakers and remains robust to simulated Internet-scale latencies of 160 ms, 200 ms, and 240 ms.
A third line uses explicit listening and speaking channels inside a native speech model. “LLM Can Listen While Speaking” proposes LSLM, an end-to-end architecture with a token-based decoder-only TTS for speech generation, a streaming SSL encoder for incoming audio, and three fusion strategies: early, middle, and late fusion. The paper reports that middle fusion achieves the best balance between speech generation and real-time interaction (Ma et al., 2024).
Native dual-channel architectures also appear in multilingual transfer. “Towards a Japanese Full-duplex Spoken Dialogue System” adapts Moshi into J-Moshi, retaining Mimi as a frozen neural audio codec operating at 12.5 Hz and training only the RQ-Transformer over parallel model and user audio-token layers plus a text layer (Ohashi et al., 3 Jun 2025). By contrast, “Hierarchical Acoustic-Semantic Modeling” argues that forcing acoustic generation and semantic reasoning to share a deep parameter space causes gradient conflict; its Lychee-FD separates deeper layers into specialized text, speech, and control heads while preserving shared shallow processing and a dedicated semantic alignment channel (Liu et al., 7 Jul 2026).
3. Data regimes and corpus construction
Data scarcity is a persistent structural constraint. The survey identifies synchronous data scarcity as the field’s most critical bottleneck, noting that FD-SLMs require synchronized multi-channel dialogue with natural overlap, interruptions, and speaker-specific timing (Chen et al., 18 Sep 2025). Much of the recent infrastructure work therefore focuses on converting imperfect public or monaural data into duplex-ready supervision.
“Sommelier” addresses preprocessing rather than model design. It standardizes audio to 16kHz, 16-bit, Mono, normalizes loudness to -20dBFS, segments long recordings with Silero VAD, uses Sortformer diarization, applies overlap separation with speaker embedding matching, and performs ASR ensembling with Whisper, Canary, and Parakeet under a ROVER scheme (Jung et al., 20 Mar 2026). The pipeline is explicitly designed not to discard overlap. It reports RTF 0.1746 for a 120-second sample on one A100 (80GB), dropping to 0.133 without FlowSE denoising, and to 0.0443 per GPU under three concurrent processes. A LoRA fine-tune of Moshi on approximately 83 hours of Sommelier-processed data improves Backchanneling, Smooth Turn-Taking, and User Interruption on Full-Duplex-Bench 1.0 (Jung et al., 20 Mar 2026).
“DuplexChat” scales this logic to web data. DuplexChat-Pipe collects podcast feeds from PodcastIndex, filters by language and metadata, segments two-speaker dialogue with pyannote/speaker-diarization-community-1, and uses DialogueSidon for speech separation and restoration (Nakata et al., 6 Jul 2026). The resulting corpus contains 282,634 hours of English and 132,723 hours of Japanese, organized as speaker-separated stereo tracks. Turn-taking analysis on sampled dialogues reports turn exchanges per minute of 10.5 for Japanese and 6.2 for English, backchannels per minute of 5.6 and 3.1, and simultaneous speech of 21% and 10%, respectively, indicating that the corpus preserves duplex conversational structure (Nakata et al., 6 Jul 2026).
Synthetic data remain central when real duplex corpora are insufficient. SyncLLM uses 212k hours of synthetic spoken dialogue data plus 2k hours of real-world spoken dialogue data (Veluri et al., 2024). J-Moshi pre-trains on about 60,000 hours from J-CHAT and later augments fine-tuning with 602 hours of multi-stream-TTS synthetic stereo dialogue, producing J-Moshi-ext (Ohashi et al., 3 Jun 2025). Lychee-FD reports a synthetic training set of roughly 140K full-duplex dialogue instances, while FLAIR uses 530K hours of speech continuation data, 70K hours of instruction-following QA data, and 20K hours of ASR-QA data (Liu et al., 7 Jul 2026, Wu et al., 18 Mar 2026). This suggests that FD-SLM progress remains tightly coupled to large-scale data synthesis and preprocessing pipelines, not only to architectural novelty.
4. Benchmarking and evaluation methodology
Evaluation has become increasingly specialized because token-level likelihood alone does not measure duplex competence. The survey organizes FD-SLM evaluation into four pillars: Temporal Dynamics, Behavioral Arbitration, Semantic Coherence, and Acoustic Performance (Chen et al., 18 Sep 2025). Recent benchmarks instantiate different subsets of that framework.
| Benchmark | Primary focus | Key measurements |
|---|---|---|
| Game-Time | Temporal instruction following | Basic tasks, temporal constraints, LLM judge, word-level timing |
| Zero-Buffer Benchmark | Abrupt interruption comprehension | Correctness, IWOR |
| MTR-DuplexBench | Multi-round duplex evaluation | Dialogue quality, conversational features, instruction following, safety |
| Full-Duplex-Bench / 1.5 | Conversational interaction behaviors | Turn-taking, backchanneling, pause handling, interruption, latency |
Game-Time is specifically designed for temporal dynamics. It contains 1,475 test instances, divided into 700 Basic task samples and 775 Advanced task samples, and formalizes each instruction-following instance as a base task plus a set of constraints (Chang et al., 30 Sep 2025). Basic families include Sequence, Repeat, Compose, Recall, Open-Ended, and Role-Play; advanced families add constraints such as Time-Fast, Time-Slow, Time-Silence, Tempo-Interval, Tempo-Adhere, Simul.-Shadow, and Simul.-Cue. Its evaluation uses a dual-channel setup with word-level timing and an LLM judge; human validation on advanced tasks reports Human–LLM Spearman’s and Human–LLM Pearson’s (Chang et al., 30 Sep 2025).
The Zero-Buffer Benchmark (ZBB) isolates a different failure mode: whether a model can understand the very first word of an abrupt interruption. It evaluates Correctness and Initial Word Occurrence Rate (IWOR) on 100 total ZBB queries, with the crucial subject word placed at utterance onset (Chang et al., 9 Jun 2026). By contrast, MTR-DuplexBench addresses multi-round interaction. It segments continuous duplex conversations into discrete turns using Whisper-timestamped, Silero VAD, GPT-4o segmentation repeated six times, clustering with at least 30% time overlap, and majority-vote filtering (Zhang et al., 13 Nov 2025). It then evaluates Dialogue Quality, Conversational Features, Instruction Following, and Safety over multiple rounds rather than single-shot events.
A broader methodological shift is also visible in judge design. Several papers prefer LLM-based evaluation over rigid rule-based scoring because spoken interaction frequently contains preambles, self-repairs, or contextually appropriate variation that simple scripts mis-score (Chang et al., 30 Sep 2025). This has made transcript quality, word-level timing extraction, and segmentation stability central components of benchmark validity.
5. Recurrent failure modes
A first recurrent failure is temporal myopia. Game-Time reports that, although state-of-the-art models handle many basic tasks well, “nearly all models degrade substantially under temporal constraints,” with especially poor performance on Time-Silence, Tempo tasks, and synchronized overlap tasks (Chang et al., 30 Sep 2025). The paper’s central conclusion is that current systems can sometimes modulate overall speaking rate but still lack robust time-awareness and full-duplex interaction capability when exact timing matters.
A second failure is state inertia. “Overcoming State Inertia in Full-Duplex Spoken LLMs via Activation Steering” argues that FD-SLM hidden states alternate between a generative state and a perceptive state, but may lag when the user interrupts (Chang et al., 9 Jun 2026). On PersonaPlex, perception affinity may take 7–8 timesteps—roughly 0.6 seconds—to recover under interruption. This delay causes models to miss the beginning of the incoming utterance, especially its first semantic word.
A third failure is modality interference. Lychee-FD reports that, in native end-to-end full-duplex designs, acoustic and semantic objectives exhibit positive gradient cosine similarity in early layers but negative similarity in deeper layers, indicating destructive optimization when the modalities are forced to share a deep parameter space (Liu et al., 7 Jul 2026). The same paper argues that temporally aligned text supervision with extensive padding weakens semantic gradients and allows acoustic learning to dominate, producing “knowledge degradation” and reduced semantic integrity.
A fourth failure is conditioning corruption under acoustic interference. IRAF targets a dual-channel setting where interfering speakers leak into the user microphone, are encoded as part of the user query, and corrupt LLM conditioning (Zhong et al., 4 Jun 2026). The reported symptoms are unstable turn-taking, false barge-in / mistimed responses, and degraded response quality. The paper’s results show that clean-training-only baselines can collapse under interfering-speaker conditions, with BLEU near zero and very low success rate.
A fifth failure is multi-round instability. MTR-DuplexBench evaluates Moshi and reports a dialogue-quality GPT-score of 1.94/5.0, declining success rates and increasing latency across rounds, plus a drop in instruction-following from 68.0 at round 1 to 41.9 averaged over rounds 1–10 under smooth turn-taking (Zhang et al., 13 Nov 2025). This indicates that single-round fluency does not imply multi-round robustness.
These failure modes also clarify several misconceptions. One misconception is that FD-SLMs are merely low-latency variants of ordinary spoken chat systems; the temporal benchmarks show that precise silence insertion, tempo adherence, synchronized overlap, and abrupt interruption comprehension are separable competencies (Chang et al., 30 Sep 2025, Chang et al., 9 Jun 2026). Another is that semantic weakness is explained only by data scarcity; Lychee-FD argues that optimization geometry itself is a root cause (Liu et al., 7 Jul 2026).
6. Remedial strategies and emerging directions
A major response has been to separate semantics from timing without abandoning full duplex. TurnGuide introduces turn-level text guidance: assistant speech is segmented into dialogue turns, and turn-level text is generated before speech output, rather than injected token by token (Cui et al., 10 Aug 2025). The paper reports more than 30% performance gain in GPT-score over baselines while keeping Pearson correlations for turn-taking events comparable to SCI and Moshi TS. FLAIR takes a different route, replacing silence-token prediction during user speech with latent reasoning. Its latent update
allows “think-while-listening” under strict causality, and the reported gains include Llama Questions: 73.0 → 78.0 and MMSU: 50.2 → 56.2 for the latent-reasoning variant (Wu et al., 18 Mar 2026).
Another response is targeted post-training alignment. ASPIRin projects the text vocabulary into a binary action space—active speech versus inactive silence—and applies GRPO to that projected policy rather than to raw lexical tokens (Hsiao et al., 11 Apr 2026). This improves interactivity while reducing duplicate 2-gram repetition from 0.117 to 0.054 and duplicate 3-gram repetition from 0.072 to 0.029 relative to standard GRPO. Multi-Faceted Interactivity Alignment instead uses axis-specific rewards for pause handling, turn-taking, backchanneling, and user interruption, plus an LLM-based quality reward to prevent semantic degradation (Ohashi et al., 9 Jun 2026). DuplexPO formulates Factorized Conversational Dynamics Reward (FCDR) over high-impact windows and argues that the apparent intelligence–dynamics trade-off is not fundamental: on Fisher, it improves Onset MAE from 0.98 to 0.69, turn-taking yield rate from 92.1 to 98.7, and backchannel yield rate from 57.1 to 100.0, while preserving or slightly improving reasoning and instruction following (Li et al., 8 Jul 2026).
Robustness interventions are becoming more localized and causal. IRAF uses a target-speaker embedding from ECAPA-TDNN plus a causal transformer gate
to rescale user frames before fusion with agent embeddings (Zhong et al., 4 Jun 2026). On InstructS2S-200K with interfering speakers only, it improves BLEU from 9.64 to 13.76, sBERT from 0.47 to 0.58, and response success from 69.2% to 91.0%, while reducing response latency from 0.97 s to 0.82 s. Activation steering provides a complementary training-free remedy for state inertia: on PersonaPlex, interruption-time correctness rises from 0.28 to 0.45, and IWOR from 0.40 to 0.72 (Chang et al., 9 Jun 2026).
Architectural remediation is also shifting toward selective decoupling rather than full separation. Lychee-FD reports +7.4% S→S accuracy, +8.8% S→T accuracy, and a 28.5% average improvement on FullDuplexBench 1.5 over system-level baselines, while preserving inference efficiency through shallow sharing and deeper specialization (Liu et al., 7 Jul 2026). This suggests that the intelligence–dynamics trade-off can sometimes be reduced by better optimization structure rather than by sacrificing concurrency.
Two frontier directions recur across the literature. The first is multilingual and culturally specific duplex modeling. J-Moshi shows that Japanese FD-SLMs can learn a turn-taking profile closer to Japanese ground truth, with Overlap 5.0s for J-Moshi versus 1.2s for English Moshi and 8.1s for Japanese ground truth (Ohashi et al., 3 Jun 2025). The second is open-mic robustness. Lychee-FD explicitly notes continued weakness in real-world multi-speaker settings and proposes richer open-mic datasets, intent detection, prosodic cues, and more selective interruption mechanisms as future work (Liu et al., 7 Jul 2026).
Taken together, the contemporary literature presents FD-SLMs as a field moving from proof-of-concept overlap handling toward a more granular science of conversational timing, semantic planning, robustness, and evaluation. A plausible implication is that future progress will depend less on any single architectural template than on the co-development of duplex-ready corpora, temporally explicit benchmarks, and mechanisms that disentangle floor control from semantic generation without breaking strict causality.