Residual Stream Decoders (RSDs)

• Residual Stream Decoders (RSDs) are transformer decoders that store only each token’s residual vector, enabling exact reconstruction of key/value attention states without caching.
• They eliminate explicit per-layer KV caching by recomputing projections deterministically, achieving up to a 27× reduction in per-token memory and improved computation throughput.
• Their deterministic, lossless reconstruction guarantees token-identical generation, making RSDs ideal for resource-constrained, long-context, and on-device inference in pre-norm, RoPE-based architectures.

Residual-Stream Decoders (RSDs) are a class of transformer decoders that maintain only a single residual vector per token as their autoregressive state, recomputing all key and value attention states deterministically from this residual vector when required. This approach eliminates the need to cache explicit per-layer key/value (KV) tensors during inference, exploiting the fact that in standard pre-norm transformer architectures, all KV pairs are exact linear projections of the normalized residual stream vector. RSDs thereby provide exact functional equivalence to conventional KV-caching inference schemes while offering substantial reductions in memory usage and, under certain conditions, improved computational throughput (Qasim et al., 20 Mar 2026).

1. Theoretical Foundations and Exact Reconstruction

The RSD formulation builds on the deterministic relationship between the residual stream and attention KV pairs in pre-norm transformer networks. Specifically, at each decoder layer ℓ and position p, the residual just before the attention sublayer is

$$r_{\ell-1,p} = \operatorname{RMSNorm}(h^{(\ell-1)}_p) \in \mathbb{R}^{d_\text{hidden}}$$

where $h^{(\ell-1)}_p$ is the main activation, and $\operatorname{RMSNorm}$ is root-mean-square normalization.

Key and value projections for each head $h$ use frozen matrices $W_k^{(\ell,h)}, W_v^{(\ell,h)}$ and a fixed positional rotation $R_p$ (typically RoPE):

• Key: $$K^{(\ell,h)}_p = R_p \cdot (r_{\ell-1,p} W_k^{(\ell,h)})$$
• Value: $V^{(\ell,h)}_p = r_{\ell-1,p} W_v^{(\ell,h)}$

Concatenating across heads yields:

$$K_{\ell,p} = P^K_\ell r_{\ell-1,p}, \quad P^K_\ell = \operatorname{block-diag}(R_p) W_k^{(\ell)}$$

$$V_{\ell,p} = P^V_\ell r_{\ell-1,p}, \quad P^V_\ell = W_v^{(\ell)}$$

The theoretical result is that, since the cached $h^{(\ell-1)}_p$0 and $h^{(\ell-1)}_p$1 are computed by fixed deterministic projections and rotations applied to the same input residual $h^{(\ell-1)}_p$2, recomputing them from $h^{(\ell-1)}_p$3 yields bit-identical values:

$h^{(\ell-1)}_p$4

$h^{(\ell-1)}_p$5

This property holds under any deterministic floating-point execution. The residual is thus a sufficient statistic for obtaining all past KV tensors per token in any standard pre-norm transformer.

2. Markov Sufficiency of the Residual Stream

RSDs leverage a key Markov property of the residual stream: for any pre-norm decoder layer ℓ, the distribution over the next layer's residual depends solely on the immediately previous layer's residual, not on any older activations:

$h^{(\ell-1)}_p$6

All sublayer computations (attention, MLPs) are deterministic functions of $h^{(\ell-1)}_p$7 and the frozen model weights, with no additional hidden state required. Empirical verification through cross-task residual patching demonstrates that substituting $h^{(\ell-1)}_p$8 from a donor at any layer ℓ yields zero KL divergence between patched and donor output distributions at every layer:

$h^{(\ell-1)}_p$9

Thus, the residual stream is both theoretically and empirically the sole information-carrying state required for autoregressive transformer decoding (Qasim et al., 20 Mar 2026).

3. KV-Direct: Residual-Based Bounded-Memory Inference

The KV-Direct inference scheme implements the RSD principle by checkpointing only the layerwise residual vectors per token. When KV eviction is required under bounded memory, corresponding evicted K/V pairs are deterministically rematerialized from the stored residuals when needed for attention lookup. The essential algorithm is:

$\operatorname{RMSNorm}$6

Consequently, the memory per token for residual checkpointing is $\operatorname{RMSNorm}$0 bytes (e.g., 2560 × 2 = 5 KB on Gemma3-4B), compared to $\operatorname{RMSNorm}$1 bytes (136 KB) for standard KV-caching.

For multi-turn inference (e.g., 20 conversation turns), the standard cache grows linearly (to ~103 MB on Gemma3-4B), while KV-Direct stabilizes at a lower plateau (42 MB). Latency analysis indicates that, due to hardware memory bandwidth constraints, recomputation of K/Vs can be 2–5× faster than reading cached tensors for batch sizes where N (number of evicted tokens) ≥ 50–100.

4. Empirical Evaluation and Model Coverage

RSDs and the KV-Direct algorithm have been empirically evaluated on six pre-norm+RoPE decoders spanning model sizes from 135M to 4B parameters, including SmolLM2-135M, Qwen2.5-0.5B, Qwen3-0.6B, DeepSeek-R1-1.5B, Qwen2.5-1.5B, and Gemma3-4B-IT. Results include:

• For full-attention layers, K/V reconstruction error is zero (bit-identical). On sliding-window layers, K reconstruction may fail (due to positional offset issues), but V remains exact.
• Generation under greedy decoding yields token-identical results across all models in comparison of full KV cache and full residual recompute (30/30 token match); speedup of 1.7–3.8× in favor of caching for short generations.
• Cross-task, layer-by-layer residual patching yields $\operatorname{RMSNorm}$2 at every layer, and exact perplexity match on downstream tasks (HellaSwag, WikiText-2) between KV-Direct and full cache.
• Bounded-memory studies show standard cache growing to 103 MB (Gemma3-4B) across 20 dialogue turns; KV-Direct with zero KV retention caps at 42 MB without any loss of output fidelity.
• Comparative evaluation on cache budgets (32–384 tokens) against H2O, StreamingLLM, SnapKV, TOVA, and window-only baselines shows KV-Direct and full cache retain 100% token match and $\operatorname{RMSNorm}$3 at all budgets, while baselines degrade to 5–28% token match and $\operatorname{RMSNorm}$4 at moderate budgets (Qasim et al., 20 Mar 2026).

5. Definition and Properties of Residual-Stream Decoders

A Residual-Stream Decoder (RSD) is defined as any transformer-style decoder that stores only the residual vector $\operatorname{RMSNorm}$5 per token as its autoregressive state, with all keys/values and attention computations restored exactly on demand by applying fixed, frozen projections to this vector. RSDs stand in contrast to standard decoders, which cache KV pairs per layer and token.

Properties:

• The residual vector acts as a sufficient statistic (Markov), ensuring lossless generation of all downstream computations.
• The residual-to-KV map is an exact linear transformation; no approximation is involved, guaranteeing exact generative behavior in empirical tests.
• For full-attention pre-norm + RoPE decoders, RSDs support lossless and token-identical generation under both greedy and sampling modes.

6. Efficiency and Practical Implications

Empirical results demonstrate up to a 27× reduction in per-token state (e.g., 136 KB→5 KB for Gemma3-4B), a 2.5× peak memory reduction over 20 conversation turns, and up to 5× speedup in effective KV access time under frequent evictions. RSDs make long-context or memory-constrained device inference feasible by eliminating the need for large KV caches. The KV cache becomes a true cache—entries may be evicted and losslessly rematerialized at need, with no degradation in output quality. RSDs are fully compatible with other model modifications such as attention-sparsity or quantization.

7. Limitations and Outlook

Residual-only state enables lossless decoding for full-attention, pre-norm, RoPE-based decoders with fixed geometry. For decoders with sliding-window attention, reconstructing key states from residuals is not always exact due to positional offset mismatch, although value projections remain exact. This suggests some care is required in applying the RSD paradigm to architectures diverging from these constraints. A plausible implication is the need for hybrid approaches or additional metadata in such settings.

RSDs fundamentally reassign the cost of sequence-state storage from memory bandwidth and footprint to matrix multiplication and compute flops, without introducing any approximation error in standard cases. This enables flexible, memory-adaptive inference and supports the viability of on-device and resource-limited LLM deployment (Qasim et al., 20 Mar 2026).