---
title: Full-Duplex Speech Dialogue Systems
url: https://www.emergentmind.com/topics/full-duplex-speech-dialogue-systems-full-duplex-sds
type: topic
---

# Full-Duplex Speech Dialogue Systems

Full-duplex Speech Dialogue Systems (Full-Duplex SDS) are conversational agents capable of simultaneous real-time bidirectional communication—listening and speaking at once. Unlike half-duplex systems that alternate strictly between user and system turns, full-duplex SDS eliminate pipeline stalls by interleaving continuous Automatic Speech Recognition (ASR), large language model (LLM) token prediction, Text-to-Speech (TTS) synthesis, and hierarchical control logic. This architecture enables conversational fluidity with low response latency and human-like behaviors such as interruption management, backchannels, and overlapping speech. Recent developments utilize streaming ASR, control-token-driven neural finite state machines (FSMs), end-to-end joint next-token prediction, and advanced benchmarking protocols to achieve robust interaction dynamics and scalable evaluation [2405.19487][2503.04721][2507.23159][2502.14145][2507.19040].

## 1. Architectural Principles and Core Modules

Modern full-duplex SDS implementations comprise three tightly coupled streaming components governed by a central LLM:

1. **Perception Module**: Streaming ASR segments incoming audio into fixed-length frames (e.g., 640 ms) and produces user token chunks. Chunks are immediately appended to the LLM input if the neural FSM is in the LISTEN state; otherwise, silence frames are dropped in SPEAK.
2. **Neural FSM for Turn-State Control**: A compact FSM governs the dialogue flow, with two principal states—LISTEN and SPEAK. State transitions are triggered by LLM-emitted control tokens such as S.SPEAK (start/interrupt), C.LISTEN (continue listening), C.SPEAK (continue speaking), and S.LISTEN (yield/listen) [2405.19487]. Extensions include multi-party modeling (per-speaker states) and additional states (e.g., PAUSE, THINK).
3. **Motor Function Module**: Streaming TTS converts system tokens to audio. The module signals to the LLM when playback of each token completes, coordinating immediate emission upon becoming active.

All modules operate within a unified next-token prediction loop, updating at each event (new ASR chunk, TTS completion, control-token emission), enabling the agent to anticipate, yield, or override user speech autonomously.

## 2. Next-Token Prediction, Control Flow, and FSM Formalization

The synchronous operation is realized through real-time next-token prediction over a serialized dialogue tape:

- The LLM conditions on dialogue history ($h_t$), recent ASR tokens ($p_t$), and motor function state ($m_t$), sampling either content or control tokens:
  $$
  P(w_{t+1}, c_{t+1} \mid h_t, p_t, m_t) = \mathrm{LLM}([h_t \parallel p_t \parallel m_t])
  $$
- The FSM performs formal state transitions:
  $$
  s_{t+1} = f_\mathrm{FSM}(s_t, c_t)
  $$
  with transition rules such as LISTEN$\stackrel{\text{S.SPEAK}}{\longrightarrow}$SPEAK, SPEAK$\stackrel{\text{S.LISTEN}}{\longrightarrow}$LISTEN.
- Pseudocode for the main interactive loop:
  ```python
  tape ← system prompt
  s ← LISTEN
  while True:
      event = wait_for_event()
      tape.append(new_tokens)
      next_token = LLM(tape)
      if next_token in control_tokens:
          s = FSM_transition(s, next_token)
      elif s == SPEAK:
          TTS.play(next_token)
      tape.append(next_token)
  ```
- LLM-internal instruction tuning (typically on 1.5K+ synthetic transcripts with marked controls) yields robust handling of pauses, interruptions, and hand-offs [2405.19487].

## 3. Real-Time Metrics and Empirical Performance

Full-duplex SDS systems are quantitatively assessed using metrics standardized in recent benchmarks:

- **First-Token Emission Delay (FTED)**: Response latency from user turn-end (or mid-sentence interruption) to system first token. LLM-based full-duplex SDS achieve:
  - Baseline half-duplex: $\sim$2.28 s
  - Streaming ASR + LLM-fd + streaming TTS: 0.68 s
  - $>50\%$ of responses within 500 ms; 90th percentile $<$1.6 s
- **Interruption Precision Rate (IPR)**: Ratio of system interruptions at semantically appropriate mid-sentence points; Llama-3-8B-fd reaches 79.1% (8% higher than the best commercial LLM).
- Benchmarks such as Full-Duplex-Bench and FD-Bench define scenario-specific metrics including Takeover Rate (TOR), Backchannel Frequency, Jensen–Shannon Divergence (JSD) on backchannel timing, Success-Reply Rate (SRR), and robust handling under varying noise and interruption conditions [2503.04721][2507.19040].

Quantitative results demonstrate over 3x reduction in response latency (2.28 s$\rightarrow$ 0.68 s), high success rates in interruption handling and turn-taking, and substantial improvements in reply quality and conditional perplexity.

## 4. Comparison with Alternative and Modular Architectures

Compositional and plug-and-play full-duplex control modules, such as FlexDuo, decouple FSM logic from the core LLM pipeline. FlexDuo introduces a third Idle state and semantic integrity–based buffering for noise filtering and mutual interruption reduction [2502.13472]. It operates outside standard ASR$\rightarrow$LLM$\rightarrow$TTS cascades, emitting control signals and filtered audio for turn-taking without retraining the speech or dialogue models.

Table: Quantitative impact of FlexDuo on Fisher corpus (English baseline)

| System        | Combined Turn-taking | False Interruption | Conditional PPL |
|---------------|---------------------|-------------------|----------------|
| VAD baseline  | 0.81                | 0.53              | 64.32          |
| FlexDuo       | 0.79                | 0.30              | 28.94          |

Removing the Idle state increases false interruptions ($+13.6\%$) and degrades turn-taking ($-15.7\%$), affirming explicit filtering’s importance.

## 5. Extensions to Multilingual, Multi-Party, and Specialized Domains

Full-duplex SDS have been successfully adapted to Japanese conversational modeling by transferring architectures such as Moshi and applying stagewise pre-training, stereo fine-tuning, and synthetic dialogue augmentation [2506.02979]. J-Moshi achieves improved perplexity and naturalness over the dGSLM baseline and more realistic overlap, mirroring Japanese conversational patterns.

FSM transitions and control-token spaces are generalizable for N-party interactions (LISTEN$_i$/SPEAK$_i$, per-speaker control tokens), with added states for THINK, PAUSE, and visual/gestural signals. Multi-modal extensions envisage gaze, facial cues, or context derived from third-party ambient speech [2405.19487][2502.13472].

## 6. Benchmarking, Scenario Coverage, and Future Challenges

A rich ecosystem of benchmarks (Full-Duplex-Bench, FD-Bench, Full-Duplex-Bench v1.5/v2, FLEXI, MTR-DuplexBench, FDB-v2) systematizes evaluation via scenario-driven tests, multi-turn dynamics, interruption robustness, and modular protocol design [2503.04721][2507.19040][2507.23159][2509.22243][2511.10262][2510.07838]. Metrics include:

- Fluency, instruction following, task-specific competence (1–5 LLM-assigned scores)
- Success rates, latency, backchannel and overlap handling
- Multi-round degradation in feature success and instruction following
- Safety/refusal rates across adversarial prompts

Empirical findings highlight outstanding challenges: blurred turn boundaries, context drift, latency spikes as conversations progress, and systematic trade-offs between latency, conversational intelligence, and robustness to noise and interruption.

## 7. Future Directions and Research Recommendations

Research priorities identified include:

- End-to-end architectures employing next-token-pair prediction for joint listening and speaking
- Streaming semantic endpoint detection (e.g., Phoenix-VAD), modular and independently optimizable
- Hierarchical memory, explicit planning guidance (TurnGuide), and token-level safety/instruction filters
- Streaming and chunked decoding with roll-back capabilities
- Scalable adaptation to multi-party, cross-lingual, and multimodal conversational domains

The consensus across contemporary full-duplex SDS research is that embedding listening, turn-taking, interruption detection, and backchanneling within a unified end-to-end joint token prediction paradigm yields the lowest latency and highest naturalness [2509.22243][2405.19487]. The field is rapidly transitioning toward open, extensible benchmarks and modular designs to advance robust, context-aware human-machine interaction.

Source: https://www.emergentmind.com/topics/full-duplex-speech-dialogue-systems-full-duplex-sds