---
title: 'Block AttnRes: Scalable Residual Routing'
url: https://www.emergentmind.com/topics/block-attention-residuals-block-attnres
type: topic
---

# Block AttnRes: Scalable Residual Routing

Block Attention Residuals, commonly abbreviated **Block AttnRes**, are a family of residual-routing mechanisms for deep Transformer architectures in which the fixed additive residual stream of PreNorm models is replaced by a learned, content-dependent mixture over earlier depth sources that have been compressed at the level of residual blocks rather than individual layers. In the canonical formulation, Block AttnRes inherits the depth-wise selection principle of Attention Residuals while reducing its memory and communication cost by routing over block summaries and the current partial block. Later extensions, notably WAV, augment each block summary with directional detail bases so that routing can exploit not only a block’s total residual displacement but also coarse intra-block structure such as attention-versus-MLP imbalance and early-versus-late dynamics [2603.15031] [2606.06564].

## 1. Historical motivation and problem setting

The immediate background to Block AttnRes is the standard **PreNorm** Transformer residual update. For a decoder-only Transformer with attention sublayer output $a_l$ and MLP output $m_l$, the conventional update is
\[
a_l = \mathrm{Attn}\big(\mathrm{LN}(x_l)\big), \quad x'_l = x_l + a_l,
\]
\[
m_l = \mathrm{MLP}\big(\mathrm{LN}(x'_l)\big), \quad x_{l+1} = x'_l + m_l.
\]
Under this scheme, every sublayer contributes to the residual stream with fixed coefficient $1$. The core critique advanced by the AttnRes literature is that such uniform aggregation causes hidden-state magnitude growth with depth, dilutes the relative contribution of each layer, and prevents later layers from selectively retrieving specific earlier representations rather than only the most recent recurrent state [2603.15031].

**Attention Residuals** address this by replacing fixed accumulation with learned, content-dependent routing across depth. In the full formulation, a layer does not merely add its predecessor; instead it forms a convex recombination over earlier depth sources. The practical difficulty is that routing over all preceding layer outputs scales poorly in large models, particularly under activation recomputation and pipeline parallelism. **Block AttnRes** is the scalable answer: layers are partitioned into blocks, each block is summarized, and later sublayers attend over those summaries plus the current partial block rather than all prior layer states [2603.15031].

This design goal is therefore dual. It is intended to preserve the optimization and representational benefits of content-dependent depth-wise selection, while making the mechanism deployable at LLM scale.

## 2. Formal definition and routing mechanics

In the conceptual AttnRes formulation, the residual update at layer $l$ is replaced by softmax attention over earlier sources:
\[
h_l = \sum_{i=0}^{l-1} \alpha_{i\to l} \cdot v_i,
\qquad
\alpha_{i\to l} = \frac{\phi(q_l,k_i)}{\sum_{j=0}^{l-1}\phi(q_l,k_j)},
\]
with
\[
\phi(q,k)=\exp(q^\top \mathrm{RMSNorm}(k)).
\]
In the version reported for large-scale language modeling, the query is a learned pseudo-query $q_l=w_l$, while keys and values are earlier residual sources, with $b_0=h_1$ for the token embedding and subsequent sources derived from layer outputs [2603.15031].

Block AttnRes compresses this source axis. A **block** is a group of consecutive sublayer updates. In the WAV formulation, a block $b$ contains updates $\{u_{b,i}\}_{i=1}^{m}$ and is summarized by
\[
C_b = \sum_{i=1}^{m} u_{b,i}.
\]
At a later sublayer inside block $b$, the routing source set becomes
\[
\mathcal{S}_{\text{Block}} = \{e, C_0, C_1, \dots, C_{b-1}, P_C\},
\]
where $e$ is the token embedding source and $P_C$ is the current partial block sum. The mixer scores each source $s_j$ using
\[
\ell_j = q^\top \operatorname{RMSNorm}(s_j) + \beta_j,
\qquad
\alpha_j = \frac{\exp(\ell_j)}{\sum_{k=1}^{S}\exp(\ell_k)},
\qquad
h = \sum_{j=1}^{S}\alpha_j s_j.
\]
Here $h$ is the routed readout supplied as context to the current attention or MLP sublayer, $q$ is a learned function of the current normalized hidden state, and $\beta_j$ are learnable per-source biases [2606.06564].

Two points define the mechanism precisely. First, routing is **depth-wise** rather than token-neighborhood attention in the usual sequence dimension. Second, Block AttnRes is not a mere scalar gate on the latest residual branch; it is a learned mixture over a structured pool of earlier block-level sources.

## 3. Computational structure and systems considerations

The principal systems advantage of Block AttnRes is that it reduces the source set from layer-wise scale to block-wise scale. Relative to layer-wise Attention Residuals, the per-sublayer routing cost drops from $O(L)$ prior states to $O(N)$ block-level summaries, where $L$ is depth and $N \ll L$ is the number of residual blocks. In the large-scale AttnRes analysis, Full AttnRes has memory footprint $O(Ld)$, whereas Block AttnRes reduces this to $O(Nd)$ by storing block representatives rather than all prior layer outputs [2603.15031].

A substantial part of the method’s practicality comes from the associated execution strategy. The reported implementation uses a **two-phase computation strategy**. Phase 1 batches all queries in a block against cached inter-block sources. Phase 2 then proceeds sequentially through the current block, computes intra-block attention over the evolving partial sum, and merges inter-block and intra-block contributions with an online softmax update. The merge is written as
\[
m_l = \max(m_l^{(1)}, m_l^{(2)}),
\]
\[
h_l = \frac{e^{m_l^{(1)}-m_l} o_l^{(1)} + e^{m_l^{(2)}-m_l} o_l^{(2)}}{e^{m_l^{(1)}-m_l} \ell_l^{(1)} + e^{m_l^{(2)}-m_l} \ell_l^{(2)}}.
\]
This permits efficient reuse of inter-block computations while preserving exact normalization [2603.15031].

The same paper also introduces **cache-based pipeline communication** for pipeline-parallel training. In the stated formulation, naïve communication cost is
\[
\mathrm{Comm}_{\mathrm{naive}} = \frac{C(C-1)}{2} N_p d,
\]
whereas cross-stage caching reduces it to
\[
\mathrm{Comm}_{\mathrm{cached}} =
\frac{P(P-1)}{2}N_p d + (V-1)P^2 N_p d.
\]
This reduces the peak per-transition cost from $O(C)$ to $O(P)$, described as a $V\times$ improvement. Reported inference overhead is less than $2\%$, and training overhead with pipeline parallelism is less than $4\%$ [2603.15031].

The architecture is therefore best understood as a joint algorithmic-and-systems proposal: the routing rule alone is insufficient without block compression, online merging, and communication caching.

## 4. Multi-resolution routing and the WAV extension

A major limitation of basic Block AttnRes is that a single block summary $C_b$ records only the block’s total residual displacement. The WAV paper characterizes this as retaining only the low-frequency or **DC** component of the block trajectory, while discarding directional structure internal to the block. Two specific omissions are emphasized: **attention-versus-MLP imbalance** and **early-versus-late block dynamics** [2606.06564].

WAV v1 addresses this by augmenting each block with two zero-sum detail bases. The **phase basis** contrasts attention and MLP updates:
\[
D^{\mathrm{phase}}_b = \sum_{i=1}^{m} a_i u_{b,i},
\]
where $a_i=+1$ for attention updates and $a_i=-1$ for MLP updates. The **split basis** contrasts the first and second halves of the block:
\[
D^{\mathrm{split}}_b = \sum_{i=1}^{m} r_i u_{b,i},
\]
where $r_i=+1$ for $i \le m/2$ and $r_i=-1$ for $i>m/2$.

The routed source pool is then expanded to include completed and partial versions of both detail types together with the standard block summaries. Crucially, the same depth-wise softmax mixer is reused; WAV does not alter the attention or MLP modules themselves. Training stability is supported by two mechanisms. First, detail sources are initialized with a negative bias,
\[
\beta_D=-2.0,
\]
while embedding and $C$ sources use zero bias, making the initial model close to ordinary Block AttnRes. Second, each detail tensor is RMS-matched to its associated block summary using a detached scaling factor,
\[
\tilde{D} = D \cdot \operatorname{stopgrad}\!\left(\operatorname{clip}\!\left(\frac{\operatorname{RMS}(C)}{\operatorname{RMS}(D)+\epsilon}, \frac{1}{\rho}, \rho\right)\right).
\]
The final output head uses only the embedding and $C$ sources by default, not detail sources [2606.06564].

This extension increases the number of routed sources by roughly a factor of three but adds only four scalar biases per Transformer layer, since the attention, MLP, embedding, and output head functions remain unchanged.

## 5. Empirical behavior and scaling

The most direct small-scale evaluation of Block AttnRes and its WAV extension is reported for character-level GPT language modeling on TinyStories and Text8 with PreNorm RMSNorm, causal self-attention, SwiGLU MLPs, vocabulary size $256$, $d_{\text{model}}=128$, $8$ attention heads, MLP dimension $1024$, sequence length $512$, depths $12/24/48$, and $4$ fixed residual blocks. Under this setup, WAV is not consistently beneficial at $12$ layers, becomes competitive at $24$ layers, and is best at $48$ layers [2606.06564].

| Depth | Text8 validation loss | TinyStories validation loss |
|---|---:|---:|
| 12 | Block AttnRes 0.9801; WAV 1.0085 | Block AttnRes 0.5261; WAV 0.5325 |
| 24 | Block AttnRes 0.9592; WAV 0.9610 | Block AttnRes 0.5036; WAV 0.5012 |
| 48 | Block AttnRes 0.9363; WAV 0.9305 | Block AttnRes 0.4960; WAV 0.4738 |

At $48$ layers, the reported reduction relative to Block AttnRes is from $0.4960$ to $0.4738$ on TinyStories and from $0.9363$ to $0.9305$ on Text8, with negligible additional parameters. The draft notes that training curves at $48$ layers show consistent separation for TinyStories and smaller but consistent improvement by end of training for Text8, while also stating that standard deviations across seeds and error bars are not yet included [2606.06564].

At larger scale, the original AttnRes paper reports scaling-law fits
\[
\mathcal{L}_{\mathrm{Baseline}} = 1.891 \times C^{-0.057},
\qquad
\mathcal{L}_{\mathrm{Full\ AttnRes}} = 1.865 \times C^{-0.057},
\qquad
\mathcal{L}_{\mathrm{Block\ AttnRes}} = 1.870 \times C^{-0.058}.
\]
At $5.6$ PFLOP/s-days, Block AttnRes reaches $1.692$ validation loss versus $1.714$ for the baseline, described as approximately a $1.25\times$ compute advantage. In the same study, Block AttnRes is integrated into the Kimi Linear architecture with $48$B total and $3$B activated parameters and pretrained on $1.4$T tokens, where it improves downstream performance across all evaluated tasks and yields more uniform output magnitudes and gradient distribution across depth [2603.15031].

The consistent empirical pattern is therefore depth sensitivity. Block-level routing by itself is beneficial; directional detail routing becomes increasingly useful as the network deepens and each block accumulates richer internal dynamics.

## 6. Interpretability, scope, and common confusions

Because Block AttnRes exposes routing weights explicitly in the forward pass, it presents an obvious interpretability target. The strongest cautionary result so far is that **architectural exposure is necessary but not sufficient for mechanistic interpretation**. In a causal-probing study on two same-scale $0.6$B Qwen3 checkpoints, a wrapped vanilla baseline with zero routing projection and recency bias $\beta=3.0$ produced routing weights that were content-independent and exactly matched the analytic schedule. The trained Block AttnRes checkpoint instead showed three localized routing motifs: an embedding-source pathway through early-layer MLP, a current-state pathway through early-layer attention and MLP, and an older-history pathway through late-layer attention. Yet the paper also found a sharp dissociation between average routing mass and causal importance, including a source family with appreciable mass but no detectable causal role under intervention [2606.13168].

This interpretability result helps resolve several common confusions.

First, **Block AttnRes is not synonymous with block-sparse attention**. MoBA, NABLA, and “Block-attention for Efficient Prefilling” all operate on block-structured attention masks, routing, or KV reuse in the sequence dimension, but each explicitly retains standard Transformer residual connections rather than replacing the residual stream with depth-wise source mixing [2502.13189] [2507.13546] [2409.15355].

Second, **the term has broadened in domain-specific adaptations**. DeRes introduces a dual-path CTR architecture with an Identity residual path plus a Block Attention Residual path over earlier block summaries, while BARFI-Q uses adaptive block-attention residual aggregation inside a time-series forecasting backbone. These systems inherit the broad idea of adaptive cross-depth reuse, but their exact semantics differ from the canonical Transformer depth-routing formulation of AttnRes and Block AttnRes [2606.07980] [2605.05394].

Third, **higher routing mass should not be read as proof of functional importance**. The causal-probe evidence suggests that routing visualizations are best treated as hypotheses to be tested by intervention rather than as direct evidence of mechanism [2606.13168].

Taken together, the literature defines Block AttnRes most precisely as a scalable, block-compressed form of Attention Residuals for deep PreNorm Transformers, aimed at mitigating residual dilution through learned depth-wise routing. Its later extensions widen the representational bandwidth of block summaries, and its interpretability value appears real but conditional: the routing tensor is informative only when it has been learned as part of optimization and only when descriptive analyses are checked against causal tests.

Source: https://www.emergentmind.com/topics/block-attention-residuals-block-attnres