--- title: Two-Stage Decoder-Only Architecture url: https://www.emergentmind.com/topics/two-stage-decoder-only-tdo-architecture type: topic --- # Two-Stage Decoder-Only Architecture A Two-Stage Decoder-Only (TDO) architecture denotes a design paradigm across multiple machine learning fields in which model computation is explicitly organized into two successive phases, both leveraging decoder or decoder-style sub-networks. In contrast to canonical encoder-decoder or fully monolithic decoder architectures, a TDO system decomposes its computational pipeline—either to facilitate modular learning, boost efficiency, enable data-type bridging, or expose internal feature specialization. Key instantiations arise in neural language modeling, sequence transduction, view synthesis, speech recognition, and channel decoding [2510.11958][2405.05254][2309.08876][2605.29891][2412.02101][1705.03242]. ## 1. Foundational Principles and Motivations The canonical decoder-only Transformer (e.g., GPT) applies a stacked series of masked self-attention layers over input and context tokens, producing outputs one token at a time. Recent analyses show that these stacks can be naturally interpreted as comprising at least two functionally distinct depthwise regions: (i) early/middle layers form deep contextual representations or latent alignments, and (ii) late layers specialize in decoding those representations into the final output sequence [2510.11958]. The TDO philosophy formalizes this implicit specialization, organizing the computational graph or workflow into two explicit phases—each operating as a decoder—potentially with different data visibility, data types, masking, or learned parameters. This staging offers multiple theoretical and practical benefits: - Efficient resource utilization, including reduced redundant computation during autoregressive decoding or large-context handling [2510.11958][2405.05254]. - Bridging heterogeneous modalities, e.g., using a first stage to compress audio, images, or source-language text into an intermediate embedding, then a second stage for target token prediction [2309.08876][2605.29891][2412.02101]. - The potential for enhanced transfer, as TDO can force intermediate alignment (linguistic, semantic, or visual) required for better zero-shot generalization or representation disentanglement [2412.02101][2605.29891]. ## 2. Architectural Formulations in Key Domains ### 2.1 Sequence Modeling and Language Models Modern decoder-only models (LLMs) exhibit depthwise specialization. In TDO-based designs, inference is partitioned: - **Stage I:** Early and middle layers (\(L_{\rm mid}\)), generating a rich contextual representation \(\mathbf{H}_{\rm mid}\) from previous tokens. - **Stage II:** Late layers (\(L_{\rm late} = L-L_{\rm mid}\)), decoding \(\mathbf{H}_{\rm mid}\) to probability distributions over output tokens [2510.11958]. Such partitioning underlies Direct Multi-Token Decoding (DMTD), where a full forward pass through early/middle layers is conducted only periodically, and the late decoder layers can operate multiple times on cached representations to accelerate generation [2510.11958]. The YOCO (You Only Cache Once) architecture further realizes this paradigm by: - Using a **self-decoder** with constant-memory attention (sliding window or gated-retention) for the first half of layers. - Generating a global KV cache after this phase, which is reused by a **cross-decoder**—the latter half of the layers—accessing only the fixed memory rather than growing contextual caches [2405.05254]. ### 2.2 Speech Recognition In end-to-end ASR, TDO enables bridging audio and text in a pure Transformer stack: - **Stage 1:** A Conformer encoder with CTC collapse generates a sparse sequence of non-blank prompt vectors from audio, removing redundant frames [2309.08876]. - **Stage 2:** The prompt is prepended to the inputs of a decoder-only Transformer, which performs autoregressive refinement to produce the final transcription. The decoder can be further LM-trained on text-only data without retraining the prompt or encoder mechanisms. This approach delivers both improved sample efficiency (easily leveraging text data) and competitive error rates versus encoder-decoder baselines [2309.08876]. ### 2.3 Multilingual Neural Machine Translation For MNMT, TDO architectures impose explicit separation: - **Stage 1:** Runs on source tokens (plus target-language instruction) with no target token visibility. It acts as a "pseudo-encoder" that aligns source representations into target-language feature space via prefix-masked self-attention [2412.02101]. - **Stage 2:** Consumes both the output of Stage 1 and previously generated target tokens, predicting next target tokens autoregressively. Small FFN adapter modules are added to route representations between stages, and an instruction-level contrastive loss further enhances cross-lingual transfer. Experiments show that TDO outperforms or matches encoder-decoder models in zero-shot translation, highlighting the impact of the explicit two-stage split [2412.02101]. ### 2.4 View Synthesis In high-fidelity view synthesis, a TDO approach decomposes the workflow as: - **Stage 1:** A decoder-only module processes observed views and geometric context, building a scene representation as a multi-layer KV cache [2605.29891]. - **Stage 2:** Another decoder-only pass, operating with identical weights, uses only camera geometry of the query to render a novel view by cross-attending to the cached scene representation. Full parameter sharing and staged patch sizing provide flexibility and efficiency, outperforming encoder-decoder variants in both PSNR and speed [2605.29891]. ### 2.5 Channel Decoding In two-stage soft/hard decoders for coded modulation: - **Stage 1:** A soft-decoding module (turbo/SCCC or ADBP) infers a subset of most vulnerable bits. - **Stage 2:** Hard-decoding on remaining bits, typically via high-rate algebraic codes. Per-symbol computation is reduced by applying ADBP in the first stage, with only minimal capacity loss for appropriate bit partitioning [1705.03242]. ## 3. Mathematical Formalizations The core mathematical principle underpinning TDO is that layered representations \(\mathbf{H}_{\rm mid}\) capture all necessary context for subsequent output decoding. Key equations (representative, domain-specific): - **Language Models (DMTD):** - \( \mathbf{H}_{\rm mid}(\mathbf{x}_{