---
title: Multi-Stream Attention Residuals (MSAR)
url: https://www.emergentmind.com/topics/multi-stream-attention-residuals-msar
type: topic
---

# Multi-Stream Attention Residuals (MSAR)

Multi-Stream Attention Residuals (MSAR) is a residual architecture that combines content-dependent retrieval across network depth with multiple full-dimensional contextual views of each historical representation. Unlike standard residual connections, which accumulate branch outputs with fixed unit coefficients, MSAR expands each single-stream source through multi-scale causal depthwise convolutions, performs independent softmax retrieval over depth for each resulting stream, and adaptively fuses the retrieved streams into a single input for the next Transformer sublayer. It is designed to preserve hidden-representation integrity while avoiding the persistent multi-stream storage and communication required by architectures such as Hyper-Connections and mHC [2610.13630].

## 1. Conceptual foundations and architectural position

A standard pre-norm residual stack maintains one persistent hidden state and applies the recurrence

$$
\mathbf h_{l+1}=\mathbf h_l+f_l(\mathbf h_l),
$$

where $f_l$ includes the branch normalization, attention or MLP, and output projection. Defining the embedding source as $\mathbf v_0=\mathbf h_1$ and subsequent branch outputs as $\mathbf v_i=f_i(\mathbf h_i)$ gives

$$
\mathbf h_l=\sum_{i=0}^{l-1}\mathbf v_i.
$$

This representation provides an identity path for information and gradients, but every historical output receives a fixed coefficient of one. Earlier representations are subsequently accessible only through the accumulated state, so later layers cannot directly select an individual historical source.

Attention Residuals (AttnRes) replace fixed accumulation with normalized depth-wise retrieval:

$$
\mathbf h_l=\sum_{i=0}^{l-1}\alpha_{i\to l}\mathbf v_i,
\qquad
\sum_{i=0}^{l-1}\alpha_{i\to l}=1,
$$

where

$$
\alpha_{i\to l}
=
\frac{\exp\!\left(\mathbf q_l^\top\operatorname{RMSNorm}(\mathbf v_i)\right)}
{\displaystyle\sum_{j=0}^{l-1}\exp\!\left(\mathbf q_l^\top\operatorname{RMSNorm}(\mathbf v_j)\right)}.
$$

AttnRes therefore performs single-stream softmax attention along the depth axis. MSAR generalizes this mechanism by constructing several full-dimensional, context-dependent views of every source and assigning each view an independent depth-attention distribution.

MSAR differs from mHC and other persistent multi-stream residual systems. mHC maintains multiple residual trajectories continuously and uses stream-mixing transformations to read, update, and redistribute them. MSAR instead retains a single-stream source history, expands sources on demand, retrieves independently through multiple contextual streams, and reduces the result back to one layer input. Its defining computational pattern is:

$$
\text{single source}
\rightarrow
\text{multi-scale contextual views}
\rightarrow
\text{independent depth attention}
\rightarrow
\text{adaptive fusion}
\rightarrow
\text{single-stream layer input}.
$$

The term “stream” in MSAR consequently refers to full-dimensional contextual retrieval views rather than permanently stored residual states.

## 2. Source expansion through contextual streams

For a source $\mathbf v_i\in\mathbb R^d$, MSAR constructs $m$ streams:

$$
\mathbf v_i^{(r)}=\mathcal E_{i,r}[\mathbf v_i],
\qquad r=0,\ldots,m-1,
$$

where the identity stream is

$$
\mathcal E_{i,0}=\operatorname{Id},
$$

and the remaining operators $\mathcal E_{i,r}$ are causal depthwise one-dimensional convolutions.

Restoring the token index $t$, the identity stream is

$$
\mathbf v_{i,t}^{(0)}=\mathbf v_{i,t}.
$$

For $r\geq1$, a contextual stream is computed as

$$
\mathbf v_{i,t}^{(r)}
=
\sum_{\delta=0}^{\kappa_r-1}
\mathbf c_{i,r,\delta}\odot\mathbf v_{i,t-\delta},
$$

where $\mathbf c_{i,r,\delta}\in\mathbb R^d$ contains channel-wise convolution weights, $\odot$ denotes elementwise multiplication, and positions preceding the beginning of the sequence are zero-padded. The convolutions are causal: each stream uses the current token and preceding tokens but not future tokens.

Different kernel sizes provide distinct contextual views. The default configuration uses kernel sizes

$$
\{\kappa_r\}_{r=1}^{m-1}=\{4,8,12\},
$$

together with the identity stream, giving $m=4$. Every stream remains full-dimensional in $\mathbb R^d$. This distinguishes MSAR from channel-wise multi-head constructions, which divide the hidden dimension into smaller subspaces. MSAR instead gives each stream access to the complete representation, with contextual diversity introduced along the sequence axis.

The identity stream preserves the original source representation. The convolutional streams incorporate progressively broader preceding context. The resulting diversity is therefore not produced solely by independent learned queries: the source values themselves differ across streams. In the ablation study, removing the convolutions while retaining four independent pseudo-queries raises the loss from $1.847$ to $1.862, indicating that contextual source expansion is central to the method.

## 3. Independent depth retrieval

For each target layer $l$ and stream $r$, MSAR uses a separate learned pseudo-query

$$
\mathbf q_l^{(r)}=\mathbf w_l^{(r)}.
$$

The stream-specific key is the contextualized source itself:

$$
\mathbf k_i^{(r)}=\mathbf v_i^{(r)}.
$$

Using the score kernel

$$
\phi(\mathbf q,\mathbf k)
=
\exp\!\left(
\mathbf q^\top\operatorname{RMSNorm}(\mathbf k)
\right),
$$

MSAR computes independent depth-attention weights:

$$
\alpha_{i\to l}^{(r)}
=
\frac{
\phi\!\left(\mathbf q_l^{(r)},\mathbf k_i^{(r)}\right)
}{
\displaystyle
\sum_{j=0}^{l-1}
\phi\!\left(\mathbf q_l^{(r)},\mathbf k_j^{(r)}\right)
}.
$$

Each stream has its own normalized distribution:

$$
\sum_{i=0}^{l-1}\alpha_{i\to l}^{(r)}=1.
$$

The retrieved state is

$$
\mathbf z_l^{(r)}
=
\sum_{i=0}^{l-1}
\alpha_{i\to l}^{(r)}\mathbf v_i^{(r)}.
$$

Equivalently, separating the embedding source from branch outputs,

$$
\mathbf z_l^{(r)}
=
\alpha_{0\to l}^{(r)}\mathcal E_{0,r}[\mathbf h_1]
+
\sum_{i=1}^{l-1}
\alpha_{i\to l}^{(r)}
\mathcal E_{i,r}[f_i(\mathbf h_i)].
$$

Collecting the retrieved representations gives

$$
\mathbf Z_l
=
\left[
\mathbf z_l^{(0)},\ldots,\mathbf z_l^{(m-1)}
\right]^\top
\in\mathbb R^{m\times d}.
$$

The identity and contextual streams can therefore retrieve different historical depths. A short-context stream may emphasize local or recent representations, whereas a broader-context stream may assign greater weight to sources whose surrounding sequence context matches the learned query. The precise routing is input-dependent because the attention scores depend on the source representations, while the depth preferences are parameterized by layer- and stream-specific pseudo-queries.

MSAR pseudo-queries and convolution weights are initialized to zero. The pre-mixer scale is initialized to $0.01$. Zero initialization makes the initial routing distributions uniform and prevents arbitrary early source selection, while the small pre-mixer scale limits the initial effect of adaptive stream fusion.

## 4. Adaptive fusion and residual computation

The retrieved streams must be converted into one full-dimensional layer input. A conventional output projection from $\mathbb R^{md}$ to $\mathbb R^d$ would require a comparatively large matrix. MSAR instead uses a lightweight adaptive pre-mixer.

After flattening,

$$
\operatorname{Flatten}(\mathbf Z_l)\in\mathbb R^{md},
$$

the input-dependent mixing weights are

$$
\mathcal H_l^{\mathrm{pre}}
=
\sigma\!\left(
\alpha_l^{\mathrm{pre}}
\operatorname{RMSNorm}
\left(
\operatorname{Flatten}(\mathbf Z_l)
\right)
\mathbf W_l^{\mathrm{pre}}
+
\mathbf b_l^{\mathrm{pre}}
\right)
\in\mathbb R^{1\times m},
$$

where

$$
\mathbf W_l^{\mathrm{pre}}\in\mathbb R^{md\times m},
\qquad
\mathbf b_l^{\mathrm{pre}}\in\mathbb R^{1\times m},
$$

$\sigma$ is the sigmoid function, and $\alpha_l^{\mathrm{pre}}$ is a learned scalar initialized to $0.01$.

The fused layer input is

$$
\mathbf h_l
=
\left(
\mathcal H_l^{\mathrm{pre}}\mathbf Z_l
\right)^\top
=
\sum_{r=0}^{m-1}
\left[\mathcal H_l^{\mathrm{pre}}\right]_r
\mathbf z_l^{(r)}
\in\mathbb R^d.
$$

The Transformer branch then computes

$$
\mathbf v_l=f_l(\mathbf h_l),
$$

and the resulting single-stream source is retained for subsequent depth retrieval. The output of the residual mechanism is therefore again a single $d$-dimensional representation, despite the intermediate multi-stream computation.

Adaptive mixing is not the primary source of MSAR’s gain but provides an additional improvement and improves stability. In the reported ablation, replacing the adaptive mixer with mean aggregation gives a loss of $1.851$, compared with $1.847$ for the default configuration.

MSAR should not be confused with an architecture that maintains multiple residual trajectories throughout the network. Its intermediate streams are generated only when depth retrieval is performed, and its output is collapsed back into a single stream before the Transformer branch. This preserves compatibility with ordinary single-stream Transformer blocks.

## 5. Block MSAR and systems implementation

Layer-level MSAR would require retaining every branch output as a depth source. Block MSAR reduces this requirement by aggregating contiguous residual sublayers into block summaries. Let the residual sublayers be partitioned into blocks

$$
\mathcal B_1,\ldots,\mathcal B_N.
$$

The embedding is treated as

$$
\mathbf b_0=\mathbf h_1.
$$

The completed summary of block $n$ is

$$
\mathbf b_n
=
\sum_{j\in\mathcal B_n}
f_j(\mathbf h_j)
\in\mathbb R^d.
$$

For a target layer $l\in\mathcal B_n$, the current partial block summary is

$$
\mathbf b_n^{<l}
=
\sum_{\substack{j\in\mathcal B_n\\j<l}}
f_j(\mathbf h_j),
$$

and is updated after computing branch $l$ by

$$
\mathbf b_n^{<l+1}
=
\mathbf b_n^{<l}+f_l(\mathbf h_l).
$$

Each stream retrieves from the embedding, completed previous blocks, and the current partial block:

$$
\mathbf z_l^{(r)}
=
\sum_{j=0}^{n-1}
\alpha_{j\to l}^{(r)}
\mathcal E_r[\mathbf b_j]
+
\alpha_{n\to l}^{(r)}
\mathcal E_r[\mathbf b_n^{<l}].
$$

At the first layer of a block, the partial source is empty and excluded from the softmax denominator. Since each Transformer layer contains attention and MLP residual sublayers, grouping four Transformer layers corresponds to $S=8$ residual sublayers per block. The main 18B model uses approximately $N=7$ blocks, and the 28B model approximately $N=8$, with the embedding treated as an additional source.

The key systems property is on-demand expansion. MSAR stores and communicates only the single-stream source $\mathbf v_i$ or block summary $\mathbf b_i$. The contextual streams $\mathbf v_i^{(r)}$ are deterministically generated when required by depth attention. Persistent storage is therefore $O(Nd)$ rather than $O(mNd)$ for $N$ stored sources.

Block MSAR is compatible with two-phase execution. Completed block summaries are processed in parallel against the queries for target layers in the current block. The current partial block is then incorporated using online softmax. Fusing causal depthwise convolution with depth attention can avoid materializing expanded source streams.

The leading-order activation-I/O estimates per token and residual sublayer are

$$
\text{Vanilla}: \quad 3d,
$$

$$
\text{Block AttnRes}: \quad \left(\frac NS+5\right)d,
$$

and

$$
\text{Block MSAR}: \quad \left(\frac NS+4m+5\right)d.
$$

For $N=8$, $S=16$, and $m=4$, the reported values are approximately $3d$ for vanilla residuals, $5.5d$ for Block AttnRes, $21.5d$ for Block MSAR, and $34d$ for mHC. These are activation-I/O estimates rather than total training costs.

For $m=4$ and kernel sizes $4$, $8$, and $12$, the additional MSAR parameter count is

$$
\Delta P_{\mathrm{MSAR}}=L(44d+5),
$$

where $L$ is the number of residual sublayers and $d$ is the hidden dimension. The reported additional parameter counts are $5.204$ million for the 18B model and $7.209$ million for the 28B model. Relative to the vanilla models, MSAR adds approximately $0.31\%$ activated parameters and $0.51\%$ training FLOPs at 18B, and $0.27\%$ activated parameters and $0.48\%$ training FLOPs at 28B.

## 6. Empirical evaluation, comparisons, and limitations

The experiments use DeepSeekMoE-style Transformers with grouped-query attention, SwiGLU feed-forward networks, RMSNorm, RoPE, 144 routed experts, top-8 routing, and one shared expert. The principal models contain 18B total parameters with 1.7B activated parameters and 28 Transformer layers, or 28B total parameters with 2.7B activated parameters and 32 Transformer layers. The comparison systems are vanilla residuals, mHC with four persistent streams, AttnRes with approximately eight blocks, and MSAR with four streams and approximately eight blocks.

MSAR outperforms vanilla residuals, mHC, and AttnRes on all 12 reported downstream benchmarks at both model scales.

| Model scale | Vanilla average | mHC average | AttnRes average | MSAR average |
|---|---:|---:|---:|---:|
| 18B | 37.5 | 41.7 | 41.8 | **45.7** |
| 28B | 44.6 | 47.4 | 49.4 | **52.4** |

At 18B, MSAR scores 56.6 on MMLU, 74.9 on ARC-C, 38.0 on BBH, 49.9 on GSM8K, 24.4 on MATH, and 52.3 on C-Eval. At 28B, it scores 62.5 on MMLU, 78.0 on ARC-C, 46.1 on BBH, 65.7 on GSM8K, 28.8 on MATH, and 59.9 on C-Eval. The final training losses are $1.750$ for MSAR at 18B and $1.750$ for MSAR at 28B in the reported main results.

The scaling-law experiments evaluate approximately 2.21B, 3.75B, 9.47B, and 17.40B total-parameter models using a shifted power law

$$
\mathcal L(C)=AC^{-\beta}+E,
$$

with fixed estimated irreducible loss $E=0.72$. The reported average iso-loss compute speedups over vanilla residuals are $1.58\times$ for MSAR, $1.35\times$ for AttnRes, and $1.30\times$ for mHC. These are fitted loss–compute comparisons, not claims that a single MSAR forward pass is literally $1.58$ times faster.

Ablations identify the principal components of the method. Removing contextual convolution raises the loss from $1.847$ to $1.862, demonstrating that multiple independent queries without distinct source views are insufficient. Mean stream aggregation gives $1.851$, showing that adaptive fusion contributes but is not dominant. Increasing the number of streams from $m=2$ to $m=4$ improves the loss from $1.850$ to $1.847, while $m=8$ produces a marginally lower loss of $1.846. Finer block granularity improves performance: full layer-level MSAR obtains $1.843$, while block sizes corresponding to approximately 15, 8, and 4 blocks obtain $1.845$, $1.847$, and $1.852$, respectively. Combining AttnRes and mHC directly reaches $1.861$, whereas MSAR reaches $1.847$, indicating that MSAR is not explained simply by adding independent historical retrieval and persistent multi-stream recurrence.

MSAR has several limitations. Its efficiency depends on block-level source accumulation, on-demand stream expansion, online softmax, and fused kernels. A naive implementation that materializes every contextual stream for every source could lose the storage and bandwidth advantages. Compared with AttnRes, MSAR requires additional convolution, multiple depth-attention computations, storage of retrieved vectors, adaptive fusion, and greater activation traffic. The reported scaling speedups depend on the chosen loss model, fixed irreducible loss, and evaluated model scales. The experiments also provide limited evidence regarding other tasks, larger or smaller stream counts, alternative contextual operators, and extensive hardware-specific throughput.

MSAR is consequently best characterized as a multi-stream extension of depth-wise attention residuals rather than as a persistent multi-stream recurrence. Its central contribution is to use full-dimensional, multi-scale contextual views to create independently routed depth memories while retaining a single-stream source representation for storage, communication, and subsequent Transformer computation.

Source: https://www.emergentmind.com/topics/multi-stream-attention-residuals-msar