---
title: Decoder-Only Transformer
url: https://www.emergentmind.com/topics/decoder-only-transformer
type: topic
---

# Decoder-Only Transformer

A decoder-only transformer is a neural architecture comprising a stack of unidirectional (causally masked) self-attention and feed-forward layers, without any separate encoder or cross-attention mechanisms. At each timestep, the model processes a growing prefix of tokens, updating internal “key” and “value” caches for self-attention and generating outputs autoregressively. This paradigm underlies models such as GPT-x, DecoderTrack, and variants used extensively in language, vision, speech, recommendation, and sequential decision-making domains.

## 1. Formal Definition and Theoretical Foundations

A decoder-only transformer consists of $N$ identical blocks, each comprising masked multi-head self-attention and position-wise feed-forward sublayers, linked by residual connections and layer normalization. The masking ensures that each position $i$ can attend only to positions $j \leq i$. For token embeddings $X \in \mathbb{R}^{L \times d}$, the attention mechanism is defined:

\[
\text{Attention}(Q, K, V) = \mathrm{softmax}\left(\frac{Q K^T}{\sqrt{d}} + M\right) V
\]

where $Q = X W^Q$, $K = X W^K$, $V = X W^V$, and $M$ is the causal mask.

Roberts proved Turing-completeness of the decoder-only transformer under mild rational-weight and position-encoding assumptions, casting it as a causal “B-machine” capable of universal computation with an appropriate embedding and hard-attention mechanism [2305.17026]. For a single block, the self-attention calculation can be reformulated as a two-layer RNN, with one recurrence for the log-sum-exp accumulator and another for the weighted running sum [2405.17361];

\[
g_i = \log(e^{g_{i-1}} + e^{q_n \cdot k_i}), \quad f_i = v_i \sigma(q_n \cdot k_i - g_{i-1}) + f_{i-1} \sigma(g_{i-1} - q_n \cdot k_i)
\]

where $\sigma$ is the logistic sigmoid and $h_n$ (the attention output) is given by $f_n$. More generally, the architecture is an instance of an unbounded Multi-State RNN, with the key-value (KV) cache serving as the growing hidden state [2401.06104].

## 2. Architectural Variants and Masking Schemes

Several architectural innovations and refinements have been proposed:

- **Standard Decoder-Only Stack**: Repeats masked multi-head self-attention $+$ FFN for $N$ blocks, using learned or relative positional encodings (RoPE, ALiBi, T5).

- **Parallel and Compressed Variants**: ParallelGPT splits the stack into parallel branches; LinearGPT and ConvGPT compress the width of the layers linearly or via convolution, yielding smaller, faster models with comparable generation quality [2404.14462].

- **StableMask**: The standard causal mask requires non-zero attention to each prior token and row sum $=1$, even when self-attention is undesired. StableMask relaxes this constraint using pseudo-attention slots, allowing attention mass to be dumped outside the actual prefix, breaking the right-stochasticity and permitting universal absolute position encoding. This modification alleviates excessive attention bias and supports efficient extrapolation [2402.04779].

\[
\tilde{A} = \mathrm{Softmax}\left((QK^T / \sqrt{d}) \odot C + P\right) \odot C
\]

where $C$ is the causal indicator and $P$ defines the pseudo slots.

- **Memory-Efficient Architectures**: YOCO decouples the stack into a memory-efficient self-decoder (constant-size KV cache) and a cross-decoder (global cross-attention on a single cached memory), achieving linear ($O(Nd)$) memory while recovering full expressivity and global context [2405.05254].

## 3. Computational and Memory Characteristics

In standard form, the main computational and memory bottleneck is quadratic: $O(N L^2 d)$ compute and $O(N L d)$ cache memory for sequence length $L$, layers $N$, dimension $d$.

Methods for mitigation:

- **KV Cache Compression**: TOVA dynamically evicts the key-value pair in the cache with lowest attention, permitting a bounded cache size $K$ and transforming the model into a finite Multi-State RNN. TOVA closely matches full-context model performance across language modeling, summarization, QA, and generation benchmarks at $1/8$ the memory footprint, yielding $4$–$8\times$ throughput improvement [2401.06104].

- **Vector-Quantized Attention**: Transformer-VQ factorizes dense self-attention using a vector-quantized key codebook, permitting blockwise linear-time ($O(T)$) computation while maintaining high generation quality [2309.16354].

## 4. Applications Across Domains

Decoder-only transformers have become the backbone in a wide array of domains:

- **Language Modeling and Generation**: The GPT-x series, as well as model trends in LLMs, use decoder-only stacks for unmatched performance in domain-general generation, code synthesis, long-context reasoning, and retrieval-augmented generation.

- **Vision and Multimodal**: DTrOCR and GraDeT-HTR demonstrate decoder-only architectures applied to vision (OCR, handwritten text recognition) by mapping images to patch tokens, using the transformer to autoregressively produce character or grapheme sequences; these approaches outperform previous encoder-decoder paradigms in English, Chinese, and Bengali [2308.15996, 2509.18081].

- **Speech**: Unified spoken language models discretize acoustic features into tokens and train decoder-only stacks for speech recognition, translation, and captioning, with losses tailored to the noise/discreteness of speech units (e.g., Smoothed Label Distillation, SLD) [2311.04534].

- **Recommendation Systems**: CADET applies a decoder-only transformer to sequential CTR prediction with major engineering advances—context-conditioned heads, gated attention, timestamp-based RoPE, and custom masking—to serve high throughput online recommendation at LinkedIn, outperforming hybrid DLRM+encoder benchmarks [2602.11410].

- **Computer Vision Detection and Tracking**: D²ETR removes the encoder from DETR, using decoder-only cross-scale attention for efficient object detection competitive with or exceeding encoder-decoder variants at a fraction of the cost [2203.00860]. DecoderTrack uses a decoder-only transformer for multi-object tracking with further architectural and memory optimizations [2310.17170*; see note].

- **Simultaneous Translation**: The Decoder-only Streaming Transformer manipulates source/target prefix concatenation and customized positional encoding to achieve competitive BLEU/latency trade-offs in translation streaming via its Streaming Self-Attention (SSA) mechanism [2406.03878].

| Domain         | Application           | Key Techniques / Benefits                                     |
|----------------|----------------------|---------------------------------------------------------------|
| Language       | LLMs, code gen       | GPT-2/3/4, autoregressive prediction                          |
| Vision         | OCR, detection       | Patch tokenization, decoder-only object queries, D²ETR        |
| Speech         | ASR, multitask       | Discrete tokenization, task-adaptive loss (SLD)               |
| Recommendation | CTR prediction       | Context-tower heads, timestamp RoPE, high-throughput kernels  |
| Multimodal     | OCR, HTR, translation| End-to-end image/patch to text, graph-aware tokenization      |

*2310.17170: Summary info based on reference to DecoderTrack, abstract, and context notes.

## 5. Advanced Inference and Training Strategies

The decoder-only paradigm supports a range of inference improvements:

- **Direct Multi-Token Decoding (DMTD)**: Exploits specialization of late transformer layers (“decoding layers”) to produce multiple tokens per forward pass, amortizing the early/middle layers across several tokens. Achieves up to $2\times$ speedup with minor quality loss, parameterized by cycle length $\tau$ and decoding layer count $L_d$ [2510.11958].

\[
\mathrm{PLT} = \frac{L + (\tau-1) L_d}{\tau L}
\]

- **Streaming and Policy-Learning**: DST for simultaneous translation continually appends new source tokens to the prefix buffer, using separate source/target positional encodings to avoid recomputation, and leverages the SSA mechanism to dynamically decide “READ” vs “WRITE” at each step (see BLEU vs average lagging, [2406.03878]).

- **Robustness Certification**: ARC-Tran leverages the equivalence between a one-layer decoder-only transformer and a two-layer RNN to enable interval-bound propagation and abstract interpretation for certifying robustness to arbitrary input perturbations [2405.17361]. This approach subsumes prior LSTM-centric verification methods and handles non-length-preserving string transformations.

## 6. Limitations, Open Challenges, and Future Directions

While decoder-only transformers are theoretically universal and empirically dominant, limitations remain:

- **Memory and Context Bottlenecks**: Unbounded KV caches cannot be realized in hardware, so practical deployments require well-designed cache compression, windowing, or architectural variants (see TOVA, YOCO).

- **Attention and Masking Biases**: Causal softmax mandates nonzero attention to earlier tokens, causing spurious allocation especially at short prefixes; masking or pseudo-mass (e.g., StableMask) can relieve this bias.

- **Representation Universality**: Standard RoPE and ALiBi encode only relative position, limiting universality on position-critical tasks; architectural remedies such as pseudo-slots (StableMask) or timestamp-based RoPE address this.

- **Specialization Effects**: Layer specialization means late layers dominate next-token prediction; inference strategies such as DMTD leverage this for faster decoding at a controllable quality-speed trade-off.

- **Parameter Efficiency vs. Turing Universality**: While formal models are Turing-complete, practical architectures are vastly overparameterized relative to the theoretical minimum; closing this gap is an ongoing research direction [2305.17026].

Potential avenues include learned or dynamic memory agents, further compressive or content-adaptive attention mechanisms, improved absolute position modeling, and rigorous understanding of the interface between architecture, optimization, and in-situ algorithmic generalization.

## References

- Roberts, D. (2023). "How Powerful are Decoder-Only Transformer Neural Models?" [2305.17026]
- Schwartz et al. (2024). "Transformers are Multi-State RNNs" [2401.06104]
- Zhang et al. (2024). "A One-Layer Decoder-Only Transformer is a Two-Layer RNN: With an Application to Certified Robustness" [2405.17361]
- Wang & Arai (2024). "StableMask: Refining Causal Masking in Decoder-only Transformer" [2402.04779]
- Geng et al. (2024). "You Only Cache Once: Decoder-Decoder Architectures for Language Models" [2405.05254]
- Roy et al. (2024). "Towards smaller, faster decoder-only transformers: Architectural variants and their implications" [2404.14462]
- Suzuki et al. (2023). "Transformer-VQ: Linear-Time Transformers via Vector Quantization" [2309.16354]
- Chen et al. (2024). "Decoder-only Streaming Transformer for Simultaneous Translation" [2406.03878]
- LinkedIn AI (2026). "CADET: Context-Conditioned Ads CTR Prediction With a Decoder-Only Transformer" [2602.11410]
- Fujitake et al. (2023). "DTrOCR: Decoder-only Transformer for Optical Character Recognition" [2308.15996]
- Haque et al. (2025). "GraDeT-HTR: A Resource-Efficient Bengali Handwritten Text Recognition System utilizing Grapheme-based Tokenizer and Decoder-only Transformer" [2509.18081]
- Yu et al. (2023). "Loss Masking Is Not Needed in Decoder-only Transformer for Discrete-token-based ASR" [2311.04534]

Source: https://www.emergentmind.com/topics/decoder-only-transformer