---
title: Full-Duplex Spoken Dialogue Model
url: https://www.emergentmind.com/topics/full-duplex-spoken-dialogue-model
type: topic
---

# Full-Duplex Spoken Dialogue Model

A full-duplex spoken dialogue model is a computational system designed to engage in human-machine conversation by simultaneously listening and speaking, thereby more closely mirroring the natural characteristics of human interaction. Unlike traditional half-duplex (turn-based) systems, which alternate between receiving and generating speech, full-duplex models are engineered for real-time bidirectional exchange, accommodating overlapping speech, interruptions, and rapid backchannels.

## 1. Definition and Core Principles

Full-duplex spoken dialogue refers to the system's capacity to process incoming user speech and generate output speech concurrently. This design extends the telecommunication concept of full-duplex—bidirectional, simultaneous communication—to conversational AI, eliminating artificial turn-taking constraints and supporting natural dialogue constructs such as overlaps, interjections, and interruptions.

Key properties of a full-duplex dialogue model include:
- **Simultaneous listening and speaking:** The model processes input and produces output in real time on parallel channels.
- **Responsive turn-taking:** The system can handle user barge-ins, issue backchannel feedback, and dynamically cede or acquire the conversational floor.
- **Low interaction latency:** System response times must approach natural human conversational gaps (typically <500 ms), minimizing the temporal distinction between turns.

## 2. Representative Architectural Approaches

Architectural advances in full-duplex spoken dialogue modeling have transitioned from cascaded pipelines to end-to-end and streaming neural approaches.

**Cascaded pipelines** traditionally sequence modules for ASR, dialogue management, and TTS, coordinated through voice activity detection (VAD) and explicit turn-state FSMs. Limitations in these systems include high cumulative latency and diminished ability to model overlapping or ambiguous dialogue segments [WavChat 2411.13577].

**End-to-end models**—exemplified by frameworks such as Moshi [2410.00037] and SALMONN-omni [2505.17060][2411.18138]—integrate perception, reasoning, and generation within a single neural model. Common design patterns include:
- **Multi-stream processing:** Parallel channels for user and system audio streams are modeled jointly, capturing natural overlaps and interjections.
- **Streaming encoders/decoders:** Embedding extractors (e.g., HuBERT, Mamba, CosyVoice) process audio in fixed windows (e.g., 80–200 ms) for input and output, supporting causal, low-latency inference.
- **Unified token or embedding spaces:** Systems avoid intermediate text stages (codec-free or token-level modeling) to preserve paralinguistic information and lower response delay.
- **Internal state control:** Explicit or implicit state tokens (e.g., `<think>`, `<shift>`, start/stop-speak) are autoregressively predicted to regulate transitions between listening and speaking [2505.17060].

## 3. Dialogue State Modeling and Turn Management

Realistic full-duplex dialogue requires fine-grained state management, surpassing basic listen/speak alternation. Approaches include:
- **Finite State Machines (FSMs):** FSMs coordinate system states such as SPEAK, LISTEN, and often an explicit IDLE state to handle noise, silence, or backchannels [FlexDuo 2502.13472].
- **Control Tokens and Semantic VAD:** LLM-based dialogue managers can use control tokens (e.g., <|S-S|>, <|C-S|>, <|C-L|>, <|S-L|>) to semantically distinguish complete queries, real vs. spurious barge-ins, and to robustly manage turn switching [2502.14145].
- **Dynamic thinking mechanisms:** Codec-free models (e.g., SALMONN-omni) incorporate "thinking" tokens, allowing the LLM to signal internal deliberation without outputting speech, thus accurately reflecting hesitation and reasoned response timing [2411.18138][2505.17060].

**State transition logic** is typically handled as a next-token or action prediction task, with supervised fine-tuning on annotated full-duplex conversations or, increasingly, with RL-based policy optimization to fine-tune turn-taking and interruption management.

## 4. Data, Training Paradigms, and Modeling Strategies

### Data Requirements and Preparation
- **Dual-channel and stereo datasets** (e.g., Fisher corpus, custom synthetic corpora) are fundamental, as they provide precise alignment and labeling of user/system turns, overlaps, and interjections [2506.00975].
- **Synthetic augmentation:** Multi-stream TTS, as well as LLM-driven rewriting, is employed to expand coverage, simulate interruptions, and mimic spoken linguistic styles in low-resource languages (Japanese, Taiwanese Mandarin) [2506.02979][2411.07111].
- **Time alignment:** Models often align speech and text tokens at fixed steps (e.g., 12.5 Hz for Moshi, 80–200 ms for block-based models), managing variable length and asynchronous speech via padding or explicit marker tokens.

### Training Objectives
- **Multi-task cross-entropy losses** over speech units, text tokens, and state markers are standard. Weighting is often used to prioritize semantic or conversational structure over low-level acoustics.
- **Joint token/channel modeling:** Novel paradigms such as next-token-pair prediction (NTPP) jointly predict next tokens for both user and agent channels at each timestep, establishing speaker independence and natural conversational dynamics [2506.00975].
- **Semi-supervised and data augmentation strategies** are used to exploit unlabeled data, increase generalizability, and improve robustness to domain or environmental variation [2205.15060].

## 5. Evaluation Metrics, Benchmarks, and Comparative Performance

Evaluation encompasses both objective and subjective criteria, addressing the unique requirements of real-time, interactive spoken dialogue:
- **Turn-taking performance:** Metrics include First Token Emission Delay (FTED), turn-switching F1, and takeover rate (TOR) during natural and pause-containing turns [Full-Duplex-Bench 2503.04721].
- **Backchannel and interruption handling:** Frequency, latency, and appropriateness of backchannels; barge-in success rate and latency; and model’s ability to distinguish intentional vs. unintentional interruptions.
- **Dialogue quality:** Perplexity of transcribed outputs (relative to text baselines), conditionally on ground-truth or reference human performance [2409.15594][2506.02979].
- **Naturalness and human-likeness:** Human Mean Opinion Scores (MOS), dialogue meaningfulness, and human evaluations of fluency and overlap dynamics.
- **Automatic benchmarking frameworks:** Full-Duplex-Bench is an example of a dedicated evaluation suite for pause, backchannel, turn-taking, and interruption measurement using automatic and LLM-based grading [2503.04721].
- **Latency:** Measured both as average response time and as the proportion of responses under fixed thresholds (e.g., 500 ms), with top-performing models achieving human-comparable responsiveness (<250 ms in Moshi [2410.00037], <1 second in efficient S2S models [2505.15670]).

## 6. Deployment Considerations and Real-World Impact

Full-duplex spoken dialogue systems have been adopted in real-world applications, including:
- **Commercial deployments:** Alibaba’s customer service integrated a full-duplex multimodal SDS, reporting substantial latency reductions and operational robustness [2205.15060].
- **Open-source toolkits and demos:** Frameworks such as ESPnet-SDS [2503.08533] provide comparative evaluations of cascaded and end-to-end models, supporting community benchmarking and further research.
- **Hardware and system integration:** Preferred architectures are streaming and lightweight, supporting deployment in resource-constrained or real-time environments, with minimal reliance on complex middleware or multi-stage inference [2205.15060][2410.00037][2505.15670].

## 7. Limitations, Challenges, and Future Directions

Despite rapid advances, several technical challenges persist:
- **Data scarcity:** High-quality, dual-channel, fully annotated open conversational datasets remain limited, especially for low-resource languages [2411.07111][2506.02979].
- **Tokenization and alignment:** Language-specific writing systems can lead to extreme sparsity in semantic token space (e.g., Japanese), complicating modeling at the token level.
- **Latency-accuracy trade-offs:** Smaller block sizes improve responsiveness but may degrade turn or context modeling [2410.00037][FlexDuo 2502.13472].
- **Robustness:** Handling of overlapping speech, ambiguous interjections, and noisy background audio is not fully solved in all models, particularly in open-domain settings [2410.17799][2506.00975].
- **Modality scaling:** Native support for more than three modalities (audio, vision, text, action) remains a frontier, with architectural and training innovations required for omnimodal, embodied interactive agents [2506.01934].
- **Unified benchmarking:** Lack of standardized, fine-grained metrics and evaluation suites slowed comparative assessment; recent benchmark releases aim to resolve this [2503.04721].

Future research will likely prioritize more robust multimodal integration, reinforcement learning for state control, low-resource and multilingual adaptation, and real-world deployment studies bridging fine-grained evaluation with interactive system performance.

---

**Table: Selected Open-Source Full-Duplex Models and Core Features**

| Model/Framework         | Core Methodology           | Key Feature                           |
|------------------------|---------------------------|---------------------------------------|
| Moshi [2410.00037]     | Dual-stream, codec tokens | Joint modeling, real-time S2S         |
| SALMONN-omni [2505.17060][2411.18138] | Codec-free, internal "thinking" | Single LLM, explicit state control    |
| Freeze-Omni [2411.00774]| Frozen LLM, chunked I/O   | Low-latency, avoids catastrophic forgetting |
| FlexDuo [2502.13472]   | FSM w/Idle, plug-in design| Decouples duplexity, noise filtering  |
| RoboEgo [2506.01934]   | Omnimodal, parallel streams| Vision/audio/text/actions, 80ms latency |
| NTPP [2506.00975]      | Next-token-pair, 2-channel| Speaker-agnostic joint modeling       |

---

Full-duplex spoken dialogue models represent the emerging standard for natural, robust, and responsive conversational AI, spanning foundational principles in time-synchronous neural modeling, advanced state and signal control, and multimodal integration. Their ongoing advancement is tightly coupled to data availability, systemic evaluation, and practical deployment in dynamic, real-world settings.

Source: https://www.emergentmind.com/topics/full-duplex-spoken-dialogue-model