Fast Transformer Decoding

Updated 19 March 2026

Fast Transformer Decoding is a set of algorithmic, architectural, and systems-level innovations that reduce latency, FLOPs, and memory usage in autoregressive and hybrid inference.
Techniques like segment-based attention, key compression, and multi-token prediction accelerate decoding while maintaining near-baseline output quality.
Hybrid approaches, including speculative, non-autoregressive, and bounded-context methods, achieve up to 13× speedups with minimal compromises on model performance.

Fast Transformer Decoding refers to algorithmic, architectural, and systems-level advancements targeting the reduction of wall-clock latency, FLOPs, and/or memory usage in Transformer decoding—particularly under the autoregressive paradigm, but also encompassing non-autoregressive, hybrid, and parallel inference techniques. The dominant goal is optimizing for throughput and scalability, especially for long-context or large-scale sequence generation, without substantial compromise to output quality.

1. The Sequential Bottleneck in Standard Transformer Decoding

Classic Transformer decoding is intrinsically sequential: at step $i$ , the model must process all previous history $(y_1,…,y_{i-1})$ to produce $y_i$ through stacked masked self-attention and feed-forward layers. In particular, causal dot-product self-attention at every decoder layer incurs $O(i)$ computational and memory cost per token, yielding $O(t^2)$ complexity over $t$ output tokens. For long contexts ( $t\gtrsim10^4$ ), both latency and throughput degrade sharply due to this quadratic scaling, compounded by bandwidth-intensive key/value cache growth and per-token dependency chains. Accelerating decoding therefore demands both algorithmic innovations to reduce time complexity and system-level refinements to exploit modern parallel hardware.

2. Linear- and Subquadratic-Time Attention Mechanisms

Approaches to mitigate quadratic cost focus on directly modifying the attention pattern or aggregation, using structural approximations or data-driven token selectivity.

2.1 Randomized and Segment-Based Token Selection: Radar

Radar introduces a training-free, amortized $O(\sqrt{t})$ per-token approach for long-context decoding by organizing past keys into dynamic segments and estimating their aggregate importance through random-feature projections. Specifically, at decode step $t$ , the context is partitioned into $c\approx\sqrt{t}$ segments. Each segment mean $\bar\phi$ (random feature embedding) is scored against the current query, and only the top $k$ segments, plus a buffer of recent tokens, are selected for exact attention computation. The process yields overall $O(t^{1.5})$ decoding cost—significantly below $O(t^2)$ —and maintains near-vanilla quality on long-context perplexity and downstream tasks. Theoretical results guarantee high-probability retrieval of the most salient memory segments, subject to segment attention mass separation, with no risk of irrevocable token eviction (Hao et al., 13 Mar 2025).

2.2 Key Compression and Quantization: Transformer-VQ

Transformer-VQ achieves true linear-time ( $O(t)$ ) attention by quantizing the key vectors into a learned codebook of size $S\ll t$ and maintaining per-code aggregates for past values. At each step, exact softmax attention is computed not over all keys/values but over codebook representatives and cached block-recurrent statistics. This substantially reduces both tokens/sec-per-step and memory footprint, enabling efficient processing at context lengths into the hundred-thousands with negligible loss in perplexity or bits-per-byte on language and vision benchmarks (Lingle, 2023).

2.3 Bounded Context Pruning: N-gram and Sliding-Window Variants

Limiting attention to a fixed window of $N$ previous tokens, as in N-gram masked self-attention, drops computational cost to $O(Nt)$ per layer. Empirical results with $N=4$ –$8$ show <0.4 BLEU loss in machine translation tasks, while yielding 2–4 $\times$ reductions in memory bandwidth and complexity (Chelba et al., 2020). Related streaming or landmark-based methods further trade off context for throughput but risk hard performance cliffs if relevant tokens are evicted.

3. Inference Parallelization and Multi-Token Prediction

A robust avenue for acceleration exploits the partial independence or compressibility of the decoding process to generate blocks/tokens in parallel.

3.1 Parallel Fixed-Point Solvers

Re-interpreting greedy decoding as a fixed-point solution, Santilli et al. introduce Jacobi, Gauss-Seidel, and hybrid block-wise parallel solvers. These schemes decode all or blocks of tokens in parallel, iteratively refining their values until stabilization. On CPUs and with sufficient cores, these algorithms enable up to 2 $\times$ speedups with bit-for-bit output equivalence to sequential decoding (Santilli et al., 2023).

3.2 Direct Multi-Token Decoding (DMTD)

DMTD leverages the semantic stratification of Transformer layers: early/middle layers (encoding/thinking) are run once per cycle, after which late ("decoding") layers are reused to predict multiple tokens per pass. By amortizing the expensive stack over batches of $k$ tokens, DMTD reduces layers-per-token in memory-bound regimes, yielding up to 2 $\times$ speedups for moderate $k$ with negligible accuracy degradation. Fine-tuning with cyclical masking allows the model to adapt to this regime, and scaling benefits increase with model/dataset size (Luo et al., 13 Oct 2025).

3.3 Non-/Semi-Autoregressive and Latent Variable Decoding

Latent variable models auto-encode the target into a shorter ( $m\ll n$ ) sequence of discrete latents; these are decoded autoregressively, after which the full output is parallelized via a non-autoregressive decoder. This pipeline, as in Latent Transformer, achieves order-of-magnitude latency reductions but shows a modest (1–3 BLEU) quality gap versus fully autoregressive models (Kaiser et al., 2018). Directed Acyclic Transformer (DA-T), combined with Viterbi decoding, enables globally optimal non-autoregressive inference as a single pass over the DAG, achieving speedups $\geq13\times$ with BLEU loss recouped by joint path-token optimization (Shao et al., 2022).

3.4 Speculative and Hybrid Decoding

Speculative decoding proposes tokens using a fast, approximate model, then verifies them (and rejects/corrects as needed) with the full model in parallel. This yields 2–3 $\times$ wall-clock speedups for large models like T5-XXL without loss in output distribution fidelity (Leviathan et al., 2022). Hybrid decoders in speech recognition use a fast RNN-based draft pass, then selectively patch errors with the frozen autoregressive Transformer, managing 2–3 $\times$ gains with negligible changes in word error rate (Lim et al., 27 Aug 2025).

4. Decoder Structural Simplification

Numerous approaches improve decoding efficiency by algebraically merging or pruning sub-layers, or by exploiting redundancy across layers.

4.1 Compressed Attention and Shared Weights

Sharing attention weight matrices vertically across blocks of adjacent decoder layers enables substantial reuse of intermediate activations, halving or better the per-token softmax and dot-product cost in practice. When combined with sub-layer consolidation (merging self-attn, cross-attn, and FFN into a single compressed sub-layer), this leads to 1.3–1.4 $\times$ speedups atop strong key/value-cached baselines, with BLEU differences typically below 0.5 (Li et al., 2021, Xiao et al., 2019).

4.2 Average and Multi-Query Attention

Replacing Transformer self-attention in the decoder with a cumulative average augmented by a gating mechanism (AAN) eliminates length-growing attention matrices and permits $O(1)$ per-token updates, achieving 4 $\times$ speed-ups with only $\sim$ 0.1 BLEU change (Zhang et al., 2018). Multi-query attention shares keys and values across all attention heads, dramatically reducing memory bandwidth bottlenecks (from $O(h\,n\,d)$ to $O(n\,d)$ ) and thus boosting throughput by $6$– $12\times$ in incremental and beam search decoding, while model accuracy remains within 0.2 BLEU of baselines (Shazeer, 2019).

5. Architectures for Layer and Depth Parallelization

5.1 Staggered Execution: StagFormer

By partitioning layers into disjoint stacks and introducing a one-step temporal "lag" between stacks, StagFormer enables partial depth-parallel decoding. Lower stack(s) process token $i$ as usual; upper stack(s) process token $i$ conditioned only on output from prior tokens ( $\leq i-1$ ) of earlier stacks. This schedule—implemented with cross-attention to the lagged states—supports up to $\sim$ 33% reduction in per-token latency in ideal hardware settings, with maintained end-task accuracy under balanced weight-sharing or separate weights (Cutler et al., 26 Jan 2025).

5.2 Hybrid Encoders/Decoders

Replacing the deep self-attentive decoder stack with a single-layer GRU (or similar RNN), while keeping a full self-attention encoder, yields 3–4 $\times$ speedups for machine translation, with small BLEU gap recoverable via knowledge distillation from a Transformer teacher (Wang et al., 2019).

6. Empirical Benchmarks and Implementation Details

The following table summarizes empirical throughput and quality trade-offs for several major fast decoding methods:

Approach	Speedup (tokens/sec or wall-clock)	Quality Loss	Model/Task Context
Radar	$2\times+$ at $t=16K$	$\leq0.2$ perplexity diff	Long-context LLMs (Hao et al., 13 Mar 2025)
Transformer-VQ	$3\times$ ($8K$); $12\times$ ($32K$)	$<0.1$ bpb or ppl	Enwik8, PG-19, ImageNet64 (Lingle, 2023)
DMTD	$1.8\times$ ( $\tau=3$ ), up to $2\times$	$<4\%$ (task acc)	Qwen3-4B, LLMs (Luo et al., 13 Oct 2025)
N-gram Masked Attention	$2-4\times$ theoretical	$<0.4$ BLEU	WMT EnDe/EnFr (Chelba et al., 2020)
AAN	$4\times$	$<0.1$ BLEU	WMT17 (Zhang et al., 2018)
Multi-Query Attention	$6-12\times$ (decoder)	$<0.2$ BLEU	WMT14 EnDe, LM1B (Shazeer, 2019)
Speculative Decoding	$2-3\times$ (T5-XXL)	None	T5-XXL, WMT, CNN/DailyMail (Leviathan et al., 2022)
Parallel Fixed-Point	$1.3-2\times$ (CPU/parallel)	None	MT, MBart50 (Santilli et al., 2023)
Hybrid Decoding (RNN+Tr)	$2-4\times$	$<0.1$ BLEU/WER	MT, speech (Wang et al., 2019, Lim et al., 27 Aug 2025)

Implementations benefit from storing projected segment means, random-feature embeddings, or block aggregates in fast memory, matched with customized batched GEMV kernels and efficient attention fusion via FlashAttention or equivalent. Adaptive windowing, beam-size selection, and cyclical reacquisition of intermediate representations are system considerations for maintaining both speed and quality across varied sequence lengths and batch sizes.

7. Trade-Offs, Limitations, and Future Directions

All fast decoding paradigms exhibit intrinsic trade-offs along the axes of concurrency, context utilization, memory footprint, and output quality. Methods that prune history or restrict attention risk missing rare but essential long-range dependencies. Approaches relying on approximate selection or lossy compression have theoretical fidelity bounds, but in practice, large-scale LLMs with "spiky" attention permit aggressive pruning and quantization with minimal degradation. Parallelization schemes hinge on hardware availability and may suffer diminishing returns due to memory access or weight-sharing saturation.

Future extensions aim to combine continual pretraining under hybrid or cyclical masking objectives to further close speed-quality gaps, integrate learnable attention window/segment policies, and unify draft-verification pipelines for large-scale models to support broader speculative and multi-token pathways.

The fast transformer decoding landscape is marked by a rich taxonomy of techniques, each targeting specific computational bottlenecks and model behaviors. Recent consensus, as established across these works, is that hybrid algorithmic, architectural, and systems-level innovations collectively enable order-of-magnitude acceleration relative to naive autoregressive decoding, making real-world long-context and low-latency applications practical at scale.