--- title: L0-L3 Dialogue Architecture Taxonomy url: https://www.emergentmind.com/topics/l0-l3-architectural-hierarchy type: topic --- # L0-L3 Dialogue Architecture Taxonomy Searching arXiv for the cited survey and closely related full-duplex dialogue work. The **L0–L3 Architectural Hierarchy** is a taxonomy for full-duplex spoken dialogue systems that classifies architectures by **where duplex decisions are made**. Introduced in "A Survey of Full-Duplex Spoken Dialogue Systems: Architectural Hierarchy, Interaction Ontology, and Decision State Machine" [2606.19453], it distinguishes four layers: **L0 module-level**, **L1 hidden-state-level**, **L2 token-level**, and **L3 representation-level**. The hierarchy was proposed to resolve ambiguity around the term *full-duplex*, which had been used for systems with substantially different operational capabilities. Within the survey, it is paired with two other frameworks—the $T\times I\times R$ Interaction Ontology and a Decision State Machine over **IDLE/LISTEN/SPEAK/WAIT/DUAL**—so that architectural placement, interaction type, and moment-to-moment behavior can be analyzed separately rather than collapsed onto a single axis [2606.19453]. ## 1. Taxonomic role and problem setting The hierarchy addresses a specific taxonomic problem: many recent spoken dialogue systems claim to be *full-duplex*, yet the designation does not, by itself, identify whether overlap handling is implemented by external scheduling logic, by a side classifier over model representations, by the token sequence itself, or by a hypothetical shared latent dynamic. The survey argues that previous descriptions such as *cascaded versus end-to-end* or *engineered versus learned* are insufficient because they do not specify the **decision locus** of duplex control [2606.19453]. In this framework, the critical question is not merely whether a system can process streaming speech, but **at what architectural level it decides to continue listening, begin speaking, wait, or sustain dual activity**. This reorients analysis from a coarse implementation style toward the operational substrate that governs turn-taking, barge-in, hesitation handling, and overlap. A second role of the hierarchy is comparative. It enables published systems to be grouped according to a common criterion even when they differ in tokenizer design, decoder topology, scheduler logic, or training regime. The survey uses this structure to compare representative systems and benchmarks, and to identify a **realization gap**: architectures that can in principle occupy full-duplex states often do not exhibit corresponding behavior in practice because training and evaluation data do not cover the relevant interaction patterns [2606.19453]. ## 2. Layer definitions and decision loci The four layers differ primarily by where the control signal for duplex behavior resides. | Layer | Decision locus | Representative systems | |---|---|---| | L0 | External signal modules | FireRedChat, FlexDuo, X-Talk, SoulX-Duplug, EasyTurn, FastTurn | | L1 | LLM hidden state $h_t$ | MinMo, Freeze-Omni | | L2 | Autoregressive token stream | LSLM, Moshi, OmniFlatten, SyncLLM, Mini-Omni/2, Fun-Audio-Chat, Covo-Audio, DuplexMamba, SALMONN-omni | | L3 | Shared latent $z_t$ | none published | **L0, module-level ("external scheduler")** places duplex decisions entirely outside any unified model. Streaming modules such as VAD, an end-of-turn classifier, and a dialogue manager observe audio or partial transcripts and decide when to switch between **LISTEN**, **SPEAK**, **WAIT**, or **IDLE**. The survey characterizes this as a classical cascade: raw audio passes through VAD or ASR-partial processing, then through an end-of-turn decision, then through rule-based or small-classifier scheduling that selects **SPEAK versus LISTEN** [2606.19453]. **L1, hidden-state-level ("sidecar predictor")** moves the decision into a small classifier that reads the hidden state $h_t$ of a frozen or fine-tuned LLM. The purpose is to leverage the LLM’s semantic representation for turn-end prediction, barge-in detection, and discrimination between backchannel and real interruption. The duplex action is still explicit, but it is now predicted from an internal representation rather than from external signal heuristics. **L2, token-level ("duplex in the token stream")** encodes duplex control implicitly in the autoregressive sequence itself. The next token may correspond to a user frame, an assistant frame, or silence, and the learned token dynamics determine when the conversational floor shifts. There is no separate duplex classifier; instead, switching behavior is embedded in the vocabulary design and training distribution. **L3, representation-level ("shared latent")** is a hypothetical layer in which user and assistant audio occupy the same continuous latent space $z_t$. Here duplex control is not represented by an explicit scheduler or token switch. Instead, overlap would be modeled natively by the autoregressive latent dynamics. No published system is placed in this layer [2606.19453]. ## 3. Formalization, components, and data flow At **L0**, the survey gives signal-level decision rules rather than a unified probabilistic formulation. Typical examples are $$ \mathrm{VAD}(t)=1 \text{ if energy}>\theta $$ and $$ \mathrm{EoT}(t)=1 \text{ if ASR partial so far triggered turn-end.} $$ In Decision State Machine terms, L0 implements transitions $\tau_1$, $\tau_3$, and $\tau_7$ purely by external detectors. Its components are a streaming personalized VAD ("pVAD") or classic VAD, an end-of-turn classifier, and a dialogue manager that invokes TTS or LLM backends. The canonical data flow is $$ \text{audio}\rightarrow\text{pVAD}\rightarrow(\text{if user})\rightarrow\text{EoT}\rightarrow(\text{if end})\rightarrow\text{DM}\rightarrow\text{generator}\rightarrow\text{TTS}. $$ At **L1**, the key object is the hidden vector $h_t\in\mathbb{R}^d$. The duplex predictor is a sidecar Transformer or MLP with policy head $$ \pi(h_t)=\mathrm{softmax}(W\cdot h_t+b), $$ and the emitted action may be binary or 4-way at every 80–100 ms. The survey further specifies $$ y_t=\arg\max_{r\in\{I,S,W,D\}} \mathrm{Softmax}(W_r\cdot h_t+b_r), $$ where $r$ indexes **LISTEN** $(I)$, **SPEAK** $(S)$, **WAIT** $(W)$, and **DUAL** $(D)$. In FSM terms, L1 implements $\tau_4$, $\tau_5$, and $\tau_8$–$\tau_{11}$ via $\pi(h_t)$. The architecture consists of an audio encoder plus projector into the LLM, the LLM hidden state itself, a speech decoder head, and a full-duplex predictor head. At **L2**, duplex control is governed by a generative model over a mixed token vocabulary: $$ P(X)=\prod_t P(x_t\mid x_{