Full-Duplex Spoken Dialogue Systems

• Full-Duplex Spoken Dialogue Systems are systems that process listening and speaking concurrently, enabling dynamic turn negotiation, interruptions, overlapping speech, and backchannels.
• The architectures range from cascaded pipelines to end-to-end models, with performance evaluated using benchmarks that measure latency, accuracy, and behavioral correctness.
• Emerging research focuses on resolving causal influence during overlaps, internal state synchronization, and adapting dialogue policies to persona and multilingual challenges.

Full-Duplex Spoken Dialogue Systems (FDSDS) are spoken dialogue systems that can listen and speak simultaneously, thereby supporting interruptions, overlapping speech, backchannels, and dynamic turn negotiation rather than waiting for explicit end-of-utterance cues (Wang et al., 23 Apr 2026). In the recent literature, the term has been applied to substantially different capabilities, and survey work argues that substantive full-duplex behavior requires the system’s per-moment behavior while the user is still speaking to be causally sensitive to the user’s audio, including yielding on a floor-claim, continuing through a listener backchannel, and ignoring third-party speech (Lu et al., 17 Jun 2026). FDSDS research therefore spans interaction theory, streaming architectures, turn-taking control, benchmarking, and deployment trade-offs between latency, semantic fidelity, robustness, and computational cost.

1. Conceptual scope and interaction phenomena

The central distinction between half-duplex and full-duplex dialogue lies in concurrency. Half-duplex systems serialize interaction by enforcing alternating listen-then-speak phases, whereas full-duplex systems permit simultaneous listening and speaking and must resolve overlaps causally as they occur (Wang et al., 23 Apr 2026). The phenomena repeatedly treated as canonical in the literature are interruptions, overlapping speech, backchannels, hesitation and pause handling, third-party speech rejection, and dynamic floor negotiation (Lin et al., 6 Mar 2025).

Recent survey work formalizes these phenomena through a $T\times I\times R$ interaction ontology, in which each interaction moment is specified by a temporal relation $T$, a user intent $I$, and a required system response $R$ (Lu et al., 17 Jun 2026). The ontology includes temporal classes such as Sequential, Latched, Overlap, Concurrent, and Silence; intent classes such as Information, Backchannel, Repair, Floor-claim, Floor-yield, Disfluency, and Third-party; and response classes such as Continue, Stop, Wait, Backchannel, Ignore, and Initiate. The same survey also proposes a five-state decision state machine, $\mathcal{S}=\{Idle, Listen, Speak, Wait, Dual\}$, with transitions written as $s \xrightarrow{(T_i, I_j)/R_k} s'$, making overlap resolution an explicit state-transition problem rather than a by-product of endpointing (Lu et al., 17 Jun 2026).

A recurring misconception in this literature is that “barge-in support” alone implies full-duplex capability. The survey literature explicitly rejects that reduction: keyword-triggered or chunk-alternating designs may stop on user speech yet still fail the causal-influence criterion within a chunk (Lu et al., 17 Jun 2026). This distinction is historically important because early production systems already decomposed smooth turn-taking into user state detection, backchannel selection, and barge-in detection. “Duplex Conversation,” deployed to Alibaba intelligent customer service, reported a 50% reduction in response latency in online A/B experiments, from 1400 ms to 700 ms, while framing full-duplex as the coordinated management of speaking, listening, and “thinking” (Lin et al., 2022).

2. Architectural organization of full-duplex systems

Current systems span several architectural families. The ICASSP 2026 HumDial Challenge describes three broad paradigms among submitted systems: cascaded pipelines with VAD, streaming ASR, turn-decision components, and TTS; semi-cascaded systems with independent perception and a centralized controller; and end-to-end multimodal LLMs that unify perception and generation but face “auditory blindness” during generation and output-control difficulties (Wang et al., 23 Apr 2026). This challenge-level taxonomy is complemented by a finer architectural hierarchy that locates where duplex decisions are made.

The L0-L3 hierarchy proposed in the survey literature is summarized below (Lu et al., 17 Jun 2026).

Level	Decision locus	Representative examples
L0	External scheduler or controller outside the LLM	FireRedChat, FlexDuo, Easy Turn, FastTurn
L1	Sidecar predictor over LLM hidden states	MinMo, Freeze-Omni
L2	Duplex encoded directly in the token stream	Moshi, OmniFlatten, SyncLLM, Covo-Audio
L3	Shared continuous latent where Dual is native	No published system yet

At L0, duplex behavior is imposed by explicit control modules. FlexDuo is exemplary: it inserts a plug-and-play controller in front of a half-duplex system and adds an explicit Idle state to Speak and Listen, yielding a three-state, seven-action FSM (Liao et al., 19 Feb 2025). At L1, a hidden-state predictor decides hold, shift, or interruption behavior without moving the entire model into a native duplex tokenization regime. At L2, models such as Moshi encode both streams directly into the autoregressive process; parallel, flattened, and chunk-alternating tokenizations all appear in the literature, but the survey argues that only some of them realize substantive full-duplex behavior rather than apparent full-duplex at chunk boundaries (Lu et al., 17 Jun 2026).

Two additional architectural responses illustrate the breadth of the design space. The “Unit-Based Agent” formulates dialogue as a sequence of minimal conversational units, each with listen and speak states and binary continue/switch actions, implemented in a train-free semi-cascaded system around Qwen3-Omni, Silero VAD, Paraformer ASR, and IndexTTS 1.5 (Yu et al., 28 Jan 2026). By contrast, DuplexCascade retains a cascaded ASR–LLM–TTS pipeline but removes VAD from runtime turn control, replacing utterance-wise turns with chunk-wise micro-turns flushed every $\Delta t=0.6$ s and steered by explicit conversational control tokens such as <user is speaking>, <user finish speaking>, <user is interrupting>, and <system backchannel> (Yang et al., 10 Mar 2026).

3. Turn-taking control, state spaces, and decision modules

Turn-taking control is the technical core of FDSDS. One influential formulation is the “semantic VAD” dialogue manager, implemented as a fine-tuned 0.5B LLM that continuously analyzes streaming ASR partials and recent system context to emit four control tokens: <|Continue-Listening|> (<|C-L|>), <|Start-Speaking|> (<|S-S|>), <|Start-Listening|> (<|S-L|>), and <|Continue-Speaking|> (<|C-S|>) (Zhang et al., 19 Feb 2025). Its decision rule is explicitly framed as four-way interval classification,

$p(a_t \mid X_t)=\operatorname{softmax}(W h_t+b),$

with $$a_t \in \{\langle|C\!-\!L|\rangle,\langle|S\!-\!S|\rangle,\langle|S\!-\!L|\rangle,\langle|C\!-\!S|\rangle\}$$ and sequence-level cross-entropy training. This design separates low-cost, frequent turn decisions from the larger core dialogue engine, which is only invoked when <|S-S|> is predicted, thereby reducing average compute while allowing semantic discrimination between intentional and unintentional barge-ins (Zhang et al., 19 Feb 2025).

Alternative controllers differ mainly in their state spaces and the granularity of supervision. FlexDuo uses Speak, Listen, and Idle, where Idle filters environmental noise, backchannels, and third-party speech from downstream context, and its State Manager predicts one of seven actions every 120 ms while delaying Listen onset by 500 ms from user-turn start to improve semantic discrimination in overlap (Liao et al., 19 Feb 2025). SoulX-Duplug instead unifies streaming ASR and state prediction with a state vocabulary $$\{ \langle|user\_idle|\rangle,\langle|user\_nonidle|\rangle,\langle|user\_backchannel|\rangle,\langle|user\_complete|\rangle,\langle|user\_incomplete|\rangle \}$$, interleaving audio tokens, ASR text, and state tokens as

$T$0

and deriving a theoretical average latency of 240 ms from 160 ms chunks plus an 80 ms expected offset within the final chunk (Yan et al., 16 Mar 2026).

A separate line of work seeks lightweight but semantically informed turn-state predictors. JAL-Turn shares a frozen SenseVoice-Small encoder with ASR and fuses its linguistic features with frozen CPC acoustic features through two stacked cross-attention layers, followed by causal self-attention with ALiBi and a sigmoid classifier for hold versus shift, using a fixed threshold $T$1 (Yang et al., 27 Mar 2026). Easy Turn generalizes the label space to four states—complete, incomplete, backchannel, and wait—and casts turn-taking as an ASR+Turn-Detection generation problem in which Whisper-Medium, a 3-conv/4-Transformer adaptor, and Qwen2.5-0.5B jointly generate a transcript followed by one terminal state token (Li et al., 28 Sep 2025). These designs share an assumption that silence alone is not a reliable proxy for turn completion, and that prosody, lexical content, and dialogue context must be combined.

4. Corpora, benchmarks, and evaluation methodology

As FDSDS diversified, evaluation moved from anecdotal timing analyses to task-specific benchmarks. Full-Duplex-Bench was introduced to evaluate pause handling, backchanneling, smooth turn-taking, and user interruption using automatic metrics over spontaneous and synthetic English data (Lin et al., 6 Mar 2025). Its core metrics include Takeover Rate,

$T$2

backchannel frequency, backchannel timing Jensen–Shannon divergence, smooth turn-taking latency, and interruption latency. In this framework, pause handling and backchanneling prefer low TOR, whereas smooth turn-taking and user interruption prefer high TOR and low latency (Lin et al., 6 Mar 2025).

HumDial-FDBench extends evaluation toward more complex, human-recorded dual-channel interactions (Wang et al., 23 Apr 2026). It provides more than 100 hours of Chinese and English conversations recorded by professional actors, organized into interruption and rejection scenarios such as Follow-up Question, Negation or Dissatisfaction, Repetition Request, Topic Switch, Silence or Stop, User Real-time Backchannels, Pause Handling, Third-party Speech, and Speech Directed to Others. Evaluation uses Paraformer or Parakeet-TDT for time-aligned ASR, Silero-VAD for speech boundaries, and DeepSeek-V3 prompts to classify system behavior into Respond, Resume, Uncertain, or Unknown, with a final score

$T$3

This benchmark emphasizes behavioral correctness under overlap rather than ASR-only quality (Wang et al., 23 Apr 2026).

FD-Bench broadens the metric space further by explicitly targeting interruption robustness, delay control, and noisy conditions through server-client simulation, synthesized user speech, and metrics such as SRR, SIR, SRIR, EIR, NIR, IRD, FSED, ERT, and EIT (Peng et al., 25 Jul 2025). In parallel, task-specific resources have appeared for training controllers rather than end-to-end dialogue models. Easy Turn releases a 1,145-hour Mandarin trainset for four-state turn detection and a manually verified 800-utterance testset (Li et al., 28 Sep 2025). Japanese FDSDS work has likewise required new corpora: J-Moshi uses about 60k hours of J-CHAT for pre-training, 344 hours of genuine stereo dialogues for fine-tuning, and an additional 602 hours of multi-stream TTS-synthesized stereo dialogues for augmentation (Ohashi et al., 3 Jun 2025). Across these efforts, the survey literature identifies a persistent shortage of public Type-C two-stream time-synchronous dialogue data, especially for third-party speech and sustained concurrent speech (Lu et al., 17 Jun 2026).

5. Empirical performance and system trade-offs

Empirical results consistently show that semantically informed controllers improve turn management relative to silence-threshold baselines. The semantic-VAD dialogue manager reports 0.9785 overall accuracy on its four-token synthetic test set, with recall 0.926 for <|C-L|>, 0.989 for <|S-S|>, 0.999 for <|S-L|>, and 1.000 for <|C-S|>, while on real recordings with an acoustic-VAD baseline it achieves 0.962 overall accuracy at a 500 ms silence threshold and at least 93.5% accuracy across thresholds (Zhang et al., 19 Feb 2025). FlexDuo reports a 24.9% reduction in overall false interruption rate relative to integrated full-duplex baselines and a 7.6% improvement in overall turn-taking performance, while also reducing conditional perplexity on Fisher English from 64.32 to 28.94 and on Fisher Chinese from 63.51 to 44.45 when paired with GLM4-Voice (Liao et al., 19 Feb 2025).

Lightweight bimodal turn detectors have also become competitive. On an in-house Japanese customer-service corpus, JAL-Turn reaches accuracy 92.03, F1 0.925, and latency 38 ms, surpassing LLM pipelines such as Gemini-2.5-Flash at 76.91/0.817/595 ms and GPT-5.1 at 85.52/0.874/1205 ms (Yang et al., 27 Mar 2026). Easy Turn reports 96.33% accuracy for complete, 97.67% for incomplete, 91% for backchannel, and 98% for wait on its test set, with 263 ms average latency and a footprint of 850 MB parameters and 2,559 MB GPU memory (Li et al., 28 Sep 2025). These results support the claim, stated explicitly in several papers, that acoustic-only or silence-only endpointing is insufficient once pause handling, backchannel rejection, and stop/wait requests must be recognized under low latency.

Challenge and benchmark leaderboards reveal a sharper systems-level trade-off. On HumDial-FDBench, cookie_asr ranked first with Int. 79.3, Rej. 72.2, Delay 1.260 s, D-Sco. 79.9, and Final 76.6, while Badcat ranked second with stronger interruption handling at 89.7 but weaker rejection at 57.8 and slower delay at 1.632 s; HelloWorld, an end-to-end entry, achieved the smallest delay at 0.624 s and D-Sco. 100.0 but lower interruption and rejection scores at 51.3 and 36.3 (Wang et al., 23 Apr 2026). This is consistent with benchmark analyses stating that end-to-end systems often retain a latency advantage, whereas cascaded or semi-cascaded systems remain stronger on controllability and behavioral correctness.

Within end-to-end FDSDS, the literature increasingly treats user-stream routing and interference handling as first-class design axes. The matched comparison of CF-Duplex and XA-Duplex shows that channel fusion yields stronger spoken QA and stronger interruption handling—on Full-Duplex Bench v1.0, CF-Duplex reaches User Interruption TOR 1.000, GPT-4o response score 3.96, and latency 0.374 s—whereas cross-attention routing preserves generation context more robustly under missed interruptions and achieves stronger smooth-turn-taking behavior, with TOR 0.983 and latency 0.161 s in the same benchmark (Lu et al., 11 May 2026). Under interfering-speaker conditions, IRAF improves an end-to-end dual-channel model by adding a frame-wise scalar reliability gate $T$4 before fusion; on InstructS2S-200K with MUSAN speech interference it raises BLEU from 9.64 to 13.76, sBERT from 0.47 to 0.58, reduces response latency from 0.97 s to 0.82 s, increases response success rate from 69.2% to 91.0%, and maintains stop success rate at 99.8% (Zhong et al., 4 Jun 2026).

6. Open problems and research directions

The most explicit synthesis of outstanding problems is the survey’s “realization gap”: architectural reachability is not equivalent to observed full-duplex behavior (Lu et al., 17 Jun 2026). The survey argues that many architectures can in principle enter Dual and support overlap cells, yet their realized behavior is constrained by the interaction patterns actually represented in training and evaluation. Public Type-C corpora remain limited relative to industrial datasets, third-party speech and sustained concurrent speech are weakly covered, and no published system yet occupies the survey’s L3 representation-level category, where Dual would be native to a shared continuous latent rather than negotiated through external control or discrete tokens (Lu et al., 17 Jun 2026).

Several recent works push beyond explicit turn-state control toward internal mechanisms. FLAIR replaces <SIL> during listening with a continuously updated latent “thought” embedding,

$T$5

and trains this think-while-listening process with an ELBO-based objective; it reports competitive full-duplex behavior and improves barge-in TOR from 89.0% to 92.0% on Full-Duplex-Bench while maintaining latency around 0.36 s (Wu et al., 18 Mar 2026). A different line studies internal synchronization rather than control tokens: simulations with two Moshi instances find linear CKA peaks near zero lag, with mean peak CKA around 0.5 and some conversations reaching about 0.8, while causal LSTM probes decode anticipatory turn-taking cues above chance up to 1920 ms into the future (Riera et al., 19 May 2026). This suggests that future FDSDS research may increasingly combine observable overlap behavior with internal-state diagnostics.

Another active direction is post-training alignment of interactivity itself. Multi-Faceted Interactivity Alignment uses GRPO-based reinforcement learning with axis-specific rewards for pause handling, turn-taking, backchanneling, and interruption, plus an LLM-based quality reward; on Moshi, RL on Seamless Interaction improves interruption TOR from 0.920 to 1.000 and interruption latency from 1.377 s to 0.409 s, while also improving real-time multi-turn evaluation scores (Ohashi et al., 9 Jun 2026). At the interaction-policy level, PersonaKit treats overlap behavior as persona-dependent and configurable through intent-conditioned probabilities over yield, resume, bridge, and override. In a pilot with 8 personas and 5 participants, always-yield was preferred for low-agency, high-communion roles, whereas non-yielding behavior improved naturalness for high-agency, low-communion roles, with Q1 naturalness rising from 0.20 under Yield to 0.60 under Probabilistic control (Jeon et al., 7 May 2026). These studies indicate that future FDSDS will likely optimize not only whether a system yields, but how and for which social role.

Multilingual expansion remains uneven. J-Moshi demonstrates that Moshi-style multi-stream modeling can be adapted to Japanese and that Japanese generations exhibit more overlaps than English, aligning with the paper’s reported ground-truth Japanese statistics (Ohashi et al., 3 Jun 2025). Yet many modular turn managers remain Chinese-only or language-specific, and several papers explicitly identify multilingual code-switching, domain shift, noisy multi-speaker conditions, and robust third-party rejection as unresolved problems (Zhang et al., 19 Feb 2025). The combination of richer public dual-channel corpora, per-cell reporting under the $T$6 ontology, and architectures that remain low-latency under overlap, interference, and persona variation defines the current frontier of full-duplex spoken dialogue research.