---
title: 'WavChat: Survey of Spoken Dialogue Models'
url: https://www.emergentmind.com/topics/wavchat-a-survey-of-spoken-dialogue-models
type: topic
---

# WavChat: Survey of Spoken Dialogue Models

Spoken dialogue models, also referred to as spoken dialogue systems (SDS), are artificial agents that engage in real-time, multi-turn interactions with humans using speech as the primary modality. These systems map incoming speech to an internal representation, perform response planning, and synthesize speech in reply, often under stringent requirements for latency, turn-taking, and robustness to uncertainty. The field has shifted from deterministic, modular cascades to neural end-to-end architectures capable of simultaneous listening and speaking, supporting increasingly natural, synchronous, and expressive conversational behaviors [2411.13577], [2509.14515], [2111.01414].

## 1. Historical Evolution and System Paradigms

Early SDS adopted rigid, rule-based architectures (e.g., ELIZA, PARRY) that matched patterns and produced deterministic replies, favoring interpretability at the expense of scalability and learning. The introduction of statistical models, notably Partially Observable Markov Decision Processes (POMDPs), enabled explicit uncertainty modeling across the ASR-to-dialogue-to-TTS pipeline, optimizing response selection with principled reward functions but at the cost of high dimensionality and data requirements [2111.01414].

In the 2010s, deep learning models supplanted modular pipelines. Encoder–decoder architectures, transformers, and hierarchical sequencers facilitated both task-oriented and open-domain dialogues, often trained end-to-end with large-scale conversational corpora [2111.01414], [2411.13577]. Recent milestones include:

- **Cascaded systems** (ASR → LLM → TTS): Modular, with each component trained independently; recent examples integrate paralinguistic features and LLM augmentation.
- **End-to-end models**: Direct mapping from speech input to speech output, often bypassing text entirely. Examples include dGSLM, SpeechGPT, Moshi, and full-duplex models with joint or parallel token streams [2411.13577], [2509.14515].

These trends reveal a clear trajectory from isolated, component-based design to unified architectures emphasizing multi-modality, joint learning, and adaptive synchronization.

## 2. Core Technologies in Spoken Dialogue Modeling

### 2.1 Speech Representations

Modern systems rely on learned or quantized representations, including:

- **Short-Time Fourier Transform (STFT) and Mel-spectrograms**: Conventional feature extractors.
- **Self-supervised representations**: Wav2Vec 2.0 and HuBERT employ contrastive or masked prediction to encode phonetic, prosodic, and speaker information.
- **Discrete acoustic tokens**: Vector quantization approaches (EnCodec, SoundStream, SNAC) compress waveforms into codebook indices for LLM-compatible modeling [2411.13577].

### 2.2 Training Paradigms

- **Cascaded**: Separate ASR models (CTC loss on paired speech-text), dialogue LLMs (supervised cross-entropy or RL-based fine-tuning), and TTS systems, often trained on disjoint data sources.
- **End-to-end**: Modality-aligned post-training on paired speech-text, joint or streaming optimization, shared contextual encodings across modalities, with optional preference optimization (e.g., DPO).

Recent work also investigates joint modeling of dialogue responses and prosodic, phonological, or linguistic features via unified outputs (e.g., text plus JSON- or token-based annotation for TTS pre-conditioning) [2309.11000].

### 2.3 Duplex and Streaming Capabilities

- **Half-duplex**: Strict alternation between input and output.
- **Full-duplex**: Simultaneous, overlapping input and output streams with real-time arbitration; enables natural backchannels, interruptions, and conversational overlaps [2509.14515].
- **Causal architectures**: Causal convolutions and causal self-attention ensure outputs depend only on past (and sometimes anticipated future) context, supporting low latency and synchronous interaction [2411.13577], [2509.14515].

## 3. Interaction Strategies and Synchronization

A central challenge for SDS is synchronizing inbound and outbound speech streams to enable natural conversational flow. Taxonomies distinguish:

- **Engineered synchronization**: Modular, often external, controllers (finite-state machines, binary activity classifiers) manage turn-taking decisions. Systems like FlexDuo and Freeze-Omni typify this approach [2509.14515].
- **Learned synchronization**: End-to-end architectures natively support dual-stream processing, with emergent arbitration learned from synchronous data. Examples include dual-tower transformers (dGSLM), joint autoregressive models (Moshi), and interleaved token streams (SyncLLM) [2509.14515].

Turn-taking, backchanneling, and overlap generation can be modeled with explicit loss formulations (e.g., Floor-Taking Offset, Interruption Success Rate), duration-prediction heads, and specialized input/output representations [2310.01088], [2208.03812].

## 4. Datasets and Benchmarks

The progress of SDS research is tightly coupled to the availability of high-fidelity, well-annotated datasets:

| Dataset          | Modality            | Scale      | Highlights                           |
|------------------|---------------------|------------|--------------------------------------|
| Switchboard      | Speech, Text        | 2,500 conv.| Real telephone speech, ASR noise     |
| MultiWOZ         | Text                | 10k+ dialogs | Task-oriented, multi-domain         |
| SpokenWOZ        | Speech, Text        | 5.7k dialogs, 249 h | Cross-turn, reasoning slots       |
| Fisher           | Speech, Dual-Stream | 964 h      | Synchronous dialogues                |
| MultiDialog      | Audio-Visual, Text  | 9k dialogs, 340 h | Face-to-face, emotion-annotated   |

SpokenWOZ establishes benchmarks for dialogue state tracking (DST), end-to-end completion, and slot reasoning under realistic ASR noise and spontaneous conversational phenomena [2305.13040]. MultiDialog enables training of face-to-face, audio-visual dialogue models with fine-grained emotion annotation [2406.07867].

Benchmarks and metrics include ASR WER, subjective MOS, BLEU/ROUGE/METEOR for textual alignment, Fréchet Audio Distance (FAD), and interaction-centric quantities such as Floor-Taking Offset (FTO), speaker latency, and backchannel duration ratios [2411.13577], [2509.14515].

## 5. Evaluation, Metrics, and Limitations

Evaluation in spoken dialogue is multi-faceted, covering text, audio, and interactive dynamics:

- **Textual metrics**: BLEU, ROUGE, METEOR for semantic similarity; perplexity for model uncertainty; BERTScore and GPT-4 judge scores for open-ended responses.
- **Speech/audio metrics**: WER, CER, MOS (subjective and neural), MCD.
- **Duplex/interaction metrics**: FTO, SL, IRD, ISR, overlap/gap duration matching, and MAE on lead-time prediction [2208.03812], [2509.14515].
- **Task completion**: Joint goal accuracy (DST) and dialog-level completion [2305.13040].

Current limitations span data scarcity for synchronous, spontaneous interaction; semantic vs. acoustic representation trade-offs; high autoregression latency for speech tokens; lack of unified benchmarks for full-duplex evaluation; and brittle adaptation under domain shift or speech variation [2411.13577].

## 6. Emerging Directions and Open Challenges

Key frontiers for spoken dialogue modeling include:

- **Full-duplex synchronization**: Real-time systems require architectures capable of cognitive parallelism—encoding and generating audio simultaneously with minimal latency—with emergent behavioral arbitration and robust interruption management [2509.14515].
- **Unified modeling**: Moving toward end-to-end, text-free approaches that jointly plan response content and prosody, leveraging LLM-scale models for both semantic and phonological generation [2309.11000], [2406.07867].
- **Multimodal and multimodal learning**: Incorporation of visual cues, gestures, and face-to-face synthesis substantially improves robustness in noise and user engagement [2406.07867].
- **Advanced tokenization**: Transitioning to hybrid tokenizers (combining semantic and acoustic features) and low-frame-rate representations to balance compression and expressivity [2411.13577].
- **Improved reward learning and RLHF**: Human feedback and preference optimization for tuning speech quality and safety criteria are underexplored but required for practical deployment.
- **Evaluation standardization**: There is a critical need for shared, multi-domain, open-source benchmarks and measurement suites that reflect both interactive naturalness and downstream task success [2305.13040], [2411.13577].

## 7. Synthesis and Outlook

The WavChat research landscape encompasses deterministic modular cascades, advanced neural architectures for streaming, full-duplex interaction, and multi-modal generation. There is clear evidence that self-supervised and cross-modal learning unlocks robust representations for speech perception and production. Token-based and hybrid models bridge symbolic reasoning with prosodic and paralinguistic control, while benchmarks such as SpokenWOZ and MultiDialog drive research on ASR error tolerance, slot reasoning, real-time turn-taking, and expressive interaction [2411.13577], [2305.13040], [2406.07867].

Despite these advances, pressing challenges persist: seamless full-duplex integration, interpretability, robust few-shot/generalization, and reproducible evaluation. Addressing these will require harmonized efforts in architectural design, data collection, open evaluation, and human-in-the-loop preference optimization to realize conversational agents that truly “hear, think, and speak” in natural, synchronous dialogue [2411.13577], [2509.14515].

Source: https://www.emergentmind.com/topics/wavchat-a-survey-of-spoken-dialogue-models