L0-L3 Dialogue Architecture Taxonomy

• L0–L3 Architectural Hierarchy is a taxonomy that classifies full-duplex spoken dialogue systems based on where the duplex decision (turn-taking control) is made.
• It distinguishes architectures at module-level (L0), hidden-state (L1), token-level (L2), and representation-level (L3), each with unique latency, complexity, and robustness characteristics.
• The hierarchy facilitates comparative analysis by highlighting a realization gap between architectural capacity and actual overlapping behavior in dialogue systems.

Searching arXiv for the cited survey and closely related full-duplex dialogue work. The L0–L3 Architectural Hierarchy is a taxonomy for full-duplex spoken dialogue systems that classifies architectures by where duplex decisions are made. Introduced in "A Survey of Full-Duplex Spoken Dialogue Systems: Architectural Hierarchy, Interaction Ontology, and Decision State Machine" (Lu et al., 17 Jun 2026), it distinguishes four layers: L0 module-level, L1 hidden-state-level, L2 token-level, and L3 representation-level. The hierarchy was proposed to resolve ambiguity around the term full-duplex, which had been used for systems with substantially different operational capabilities. Within the survey, it is paired with two other frameworks—the $T\times I\times R$ Interaction Ontology and a Decision State Machine over IDLE/LISTEN/SPEAK/WAIT/DUAL—so that architectural placement, interaction type, and moment-to-moment behavior can be analyzed separately rather than collapsed onto a single axis (Lu et al., 17 Jun 2026).

1. Taxonomic role and problem setting

The hierarchy addresses a specific taxonomic problem: many recent spoken dialogue systems claim to be full-duplex, yet the designation does not, by itself, identify whether overlap handling is implemented by external scheduling logic, by a side classifier over model representations, by the token sequence itself, or by a hypothetical shared latent dynamic. The survey argues that previous descriptions such as cascaded versus end-to-end or engineered versus learned are insufficient because they do not specify the decision locus of duplex control (Lu et al., 17 Jun 2026).

In this framework, the critical question is not merely whether a system can process streaming speech, but at what architectural level it decides to continue listening, begin speaking, wait, or sustain dual activity. This reorients analysis from a coarse implementation style toward the operational substrate that governs turn-taking, barge-in, hesitation handling, and overlap.

A second role of the hierarchy is comparative. It enables published systems to be grouped according to a common criterion even when they differ in tokenizer design, decoder topology, scheduler logic, or training regime. The survey uses this structure to compare representative systems and benchmarks, and to identify a realization gap: architectures that can in principle occupy full-duplex states often do not exhibit corresponding behavior in practice because training and evaluation data do not cover the relevant interaction patterns (Lu et al., 17 Jun 2026).

2. Layer definitions and decision loci

The four layers differ primarily by where the control signal for duplex behavior resides.

Layer	Decision locus	Representative systems
L0	External signal modules	FireRedChat, FlexDuo, X-Talk, SoulX-Duplug, EasyTurn, FastTurn
L1	LLM hidden state $h_t$	MinMo, Freeze-Omni
L2	Autoregressive token stream	LSLM, Moshi, OmniFlatten, SyncLLM, Mini-Omni/2, Fun-Audio-Chat, Covo-Audio, DuplexMamba, SALMONN-omni
L3	Shared latent $z_t$	none published

L0, module-level ("external scheduler") places duplex decisions entirely outside any unified model. Streaming modules such as VAD, an end-of-turn classifier, and a dialogue manager observe audio or partial transcripts and decide when to switch between LISTEN, SPEAK, WAIT, or IDLE. The survey characterizes this as a classical cascade: raw audio passes through VAD or ASR-partial processing, then through an end-of-turn decision, then through rule-based or small-classifier scheduling that selects SPEAK versus LISTEN (Lu et al., 17 Jun 2026).

L1, hidden-state-level ("sidecar predictor") moves the decision into a small classifier that reads the hidden state $h_t$ of a frozen or fine-tuned LLM. The purpose is to leverage the LLM’s semantic representation for turn-end prediction, barge-in detection, and discrimination between backchannel and real interruption. The duplex action is still explicit, but it is now predicted from an internal representation rather than from external signal heuristics.

L2, token-level ("duplex in the token stream") encodes duplex control implicitly in the autoregressive sequence itself. The next token may correspond to a user frame, an assistant frame, or silence, and the learned token dynamics determine when the conversational floor shifts. There is no separate duplex classifier; instead, switching behavior is embedded in the vocabulary design and training distribution.

L3, representation-level ("shared latent") is a hypothetical layer in which user and assistant audio occupy the same continuous latent space $z_t$. Here duplex control is not represented by an explicit scheduler or token switch. Instead, overlap would be modeled natively by the autoregressive latent dynamics. No published system is placed in this layer (Lu et al., 17 Jun 2026).

3. Formalization, components, and data flow

At L0, the survey gives signal-level decision rules rather than a unified probabilistic formulation. Typical examples are

$\mathrm{VAD}(t)=1 \text{ if energy}>\theta$

and

$$\mathrm{EoT}(t)=1 \text{ if ASR partial so far triggered turn-end.}$$

In Decision State Machine terms, L0 implements transitions $\tau_1$, $\tau_3$, and $\tau_7$ purely by external detectors. Its components are a streaming personalized VAD ("pVAD") or classic VAD, an end-of-turn classifier, and a dialogue manager that invokes TTS or LLM backends. The canonical data flow is

$h_t$0

At L1, the key object is the hidden vector $h_t$1. The duplex predictor is a sidecar Transformer or MLP with policy head

$h_t$2

and the emitted action may be binary or 4-way at every 80–100 ms. The survey further specifies

$h_t$3

where $h_t$4 indexes LISTEN $h_t$5, SPEAK $h_t$6, WAIT $h_t$7, and DUAL $h_t$8. In FSM terms, L1 implements $h_t$9, $z_t$0, and $z_t$1–$z_t$2 via $z_t$3. The architecture consists of an audio encoder plus projector into the LLM, the LLM hidden state itself, a speech decoder head, and a full-duplex predictor head.

At L2, duplex control is governed by a generative model over a mixed token vocabulary:

$z_t$4

where the vocabulary contains markers for user versus assistant tokens. The next token $z_t$5 may denote a user-frame token, assistant-frame token, or silence. The survey’s point is that the model does not consult a separate decision module; instead, the causal token sequence itself realizes overlap transitions such as $z_t$6–$z_t$7. Architectural components include an audio tokenizer, a shared autoregressive Transformer LLM, and a detokenizer or vocoder. The survey lists four L2 sub-designs: multi-stream RQ-Transformer, flattened four-stream sequence, chunk-alternation with sync tokens, and continuous embeddings (codec-free) (Lu et al., 17 Jun 2026).

At L3, no public formula is yet established. The survey lists candidate ideas rather than a settled implementation: continuous-latent AR with flow matching, dual-stream JEPA, and a world-model latent $z_t$8 with latent policies for SPEAK/LISTEN/DUAL. This suggests an intended endpoint in which discrete codes disappear and overlap is treated as ordinary trajectory structure in a continuous latent space rather than as an explicit control event.

4. Comparative properties of the four layers

The hierarchy is not merely descriptive; it also organizes trade-offs in latency, engineering complexity, robustness, and deployment practicality (Lu et al., 17 Jun 2026).

Layer	Latency	Core characterization
L0	$z_t$9 ms	simplest, interpretable, robust to noise, but post-silence/EoT reaction
L1	$h_t$0–$h_t$1 ms	moderate complexity, semantic anticipation, extra predictor and dual heads
L2	$h_t$2–$h_t$3 ms	single-model integration, low latency, brittle training, data-hungry
L3	aspirational $h_t$4 ms	unknown complexity, new paradigms, high research risk

Latency. The survey reports $h_t$5 ms for L0, reflecting module firing, LLM forward pass, and TTS chunking. L1 is reported at $h_t$6–$h_t$7 ms, combining the LLM pass with the predictor and speech heads. L2 is reported at $h_t$8–$h_t$9 ms through per-frame autoregressive decoding, with Moshi specifically reporting $z_t$0 ms. L3 is described as an aspirational $z_t$1 ms target.

Complexity and engineering cost. L0 is the simplest, easiest to debug, and highly interpretable, but its scheduling is rigid because it can only react after silence or an end-of-turn signal. L1 increases complexity moderately by reusing LLM semantics while retaining separate predictor and speech pathways. L2 has the highest architectural integration because duplex behavior and generation are unified in a single sequence model, but it is brittle with respect to tokenizer idiosyncrasies and requires substantial data coverage. L3 remains undefined enough that its complexity cannot yet be characterized operationally.

Robustness. L0 is described as best in adverse or third-party noise via pVAD modules, though it remains rigid. L1 improves semantic turn prediction and handles hesitation or backchannel better, but retains modular failure modes. L2 can in principle handle all overlap patterns, yet its actual capability depends critically on training-data coverage. L3 would natively model overlap, but its engineering maturity is unknown.

Practicality. L0 remains dominant in legacy industrial pipelines, with steady appearance of pluggable modules such as FastTurn. L1 is described as a structural attractor across both full-duplex and non-full-duplex streaming systems. L2 has received the most attention in the academic literature since mid-2024 and is associated with many open-source demos. L3 is treated as a longer-horizon R&D goal rather than an immediately practical deployment layer.

5. Representative systems and benchmark alignment

The survey associates each layer with representative systems and with benchmarks that probe distinct interactional capabilities (Lu et al., 17 Jun 2026).

At L0, representative systems are FireRedChat, FlexDuo, X-Talk, SoulX-Duplug, EasyTurn, and FastTurn. The benchmark set includes FDB v1/v1.5 stop-latency, described as measuring $z_t$2, TalkingTurns, which contains static turn patterns, and FD-Bench pipeline scenarios. These benchmarks align naturally with L0 because they emphasize interruption timing, turn boundaries, and pipeline responsiveness.

At L1, the survey places MinMo and Freeze-Omni, and it cites Qwen2.5-Omni "Thinker–Talker" and Step-Audio R1.1 as taxonomy references that share the same structural shape without being full-duplex. The benchmark emphasis here is FDB v1.5, especially barge-in versus backchannel, and FDB v2 multi-turn examiner. This pairing reflects L1’s intended role as a semantic decision layer for distinguishing interaction types that are difficult to disambiguate through signal features alone.

At L2, the list is broader: LSLM, Moshi, OmniFlatten, SyncLLM, Mini-Omni/2, Fun-Audio-Chat, Covo-Audio, DuplexMamba, and SALMONN-omni, with dGSLM identified as part of the prehistory. The benchmark suite includes Full-Duplex-Bench v1/v1.5/v2/v3, MTR-Duplex, and TalkingTurns. The concentration of evaluation around L2 reflects the layer’s prominence in recent academic work and its claim to end-to-end overlap modeling.

At L3, there are no published systems and no applicable benchmarks. This absence is itself informative: the hierarchy reserves a layer for a modeling regime that is conceptually articulated but not yet instantiated in the public literature.

6. Realization gap, data bottleneck, and open frontier

A central conclusion of the survey is the realization gap. Architecture alone does not determine observed full-duplex behavior. In particular, token-level designs can reach DUAL by construction, but they often fail on backchannel-during-speech or sustained concurrent speech unless their training data includes those interaction cells. The hierarchy therefore separates architectural capacity from behavioral realization (Lu et al., 17 Jun 2026).

The survey connects this gap to a data bottleneck. Public two-channel corpora are reported as approximately $z_t$3 k h, specifically citing Fisher, Switchboard, CANDOR, MagicData, EasyTurn, and HumDial, and these corpora under-serve $z_t$4 (third-party) and $z_t$5 (sustained concurrent) interactions. Industrial track-B data is described as larger but closed. The proposed research agendas are third-party injection pipelines, ontology-driven synthetic corpora, real concurrent-speech collection, and RL with conversation rewards.

The survey also identifies evaluation gaps. There is no explicit causal-influence probe that cleanly distinguishes substantive mid-turn reaction from merely apparent full-duplex behavior such as chunk alternation. Public benchmarks are mostly English, telephone/Zoom-style, and either static or driven by a scripted examiner. The required extensions are stated explicitly: multilingual, mobile/noisy, and adaptive multi-turn evaluations.

The L3 frontier is presented as the longer-term architectural objective. Three candidate directions are listed: continuous-latent AR streaming using flow matching or diffusion, JEPA-style dual-stream prediction, and world-model-conditioned dialogue based on a latent dialogue-state manifold. The named obstacles are discrete-codec ecosystem inertia, streaming flow-matching latency, streaming-AR throughput, and the lack of a formal L3 definition. A plausible implication is that the hierarchy is intended not only as a descriptive taxonomy for existing systems, but also as a research map: L0 through L2 describe current implementation strata, while L3 names a still-unrealized representation-level endpoint toward which full-duplex spoken dialogue modeling may evolve.