Papers
Topics
Authors
Recent
Search
2000 character limit reached

Multi-Stream Attention Residuals (MSAR)

Updated 14 October 2026
  • Multi-Stream Attention Residuals are a single-stream residual architecture that expands historical representations into full-dimensional, multi-scale causal views, retrieves each view independently across depth, and adaptively fuses them before each Transformer sublayer.
  • MSAR improves historical information access without persistent multi-stream states, using identity and causal depthwise-convolution streams to support applications such as large language model training with lower storage and communication demands than persistent multi-stream designs.
  • Reported experiments show MSAR outperforming vanilla residuals, mHC, and AttnRes, achieving average benchmark scores of 45.7 versus 37.5 for vanilla residuals at 18B scale and a fitted 1.58× iso-loss compute speedup over vanilla residuals.

Multi-Stream Attention Residuals (MSAR) is a residual architecture that combines content-dependent retrieval across network depth with multiple full-dimensional contextual views of each historical representation. Unlike standard residual connections, which accumulate branch outputs with fixed unit coefficients, MSAR expands each single-stream source through multi-scale causal depthwise convolutions, performs independent softmax retrieval over depth for each resulting stream, and adaptively fuses the retrieved streams into a single input for the next Transformer sublayer. It is designed to preserve hidden-representation integrity while avoiding the persistent multi-stream storage and communication required by architectures such as Hyper-Connections and mHC (Qin et al., 9 Oct 2026).

1. Conceptual foundations and architectural position

A standard pre-norm residual stack maintains one persistent hidden state and applies the recurrence

hl+1=hl+fl(hl),\mathbf h_{l+1}=\mathbf h_l+f_l(\mathbf h_l),

where flf_l includes the branch normalization, attention or MLP, and output projection. Defining the embedding source as v0=h1\mathbf v_0=\mathbf h_1 and subsequent branch outputs as vi=fi(hi)\mathbf v_i=f_i(\mathbf h_i) gives

hl=∑i=0l−1vi.\mathbf h_l=\sum_{i=0}^{l-1}\mathbf v_i.

This representation provides an identity path for information and gradients, but every historical output receives a fixed coefficient of one. Earlier representations are subsequently accessible only through the accumulated state, so later layers cannot directly select an individual historical source.

Attention Residuals (AttnRes) replace fixed accumulation with normalized depth-wise retrieval:

hl=∑i=0l−1αi→lvi,∑i=0l−1αi→l=1,\mathbf h_l=\sum_{i=0}^{l-1}\alpha_{i\to l}\mathbf v_i, \qquad \sum_{i=0}^{l-1}\alpha_{i\to l}=1,

where

αi→l=exp⁡ ⁣(ql⊤RMSNorm⁡(vi))∑j=0l−1exp⁡ ⁣(ql⊤RMSNorm⁡(vj)).\alpha_{i\to l} = \frac{\exp\!\left(\mathbf q_l^\top\operatorname{RMSNorm}(\mathbf v_i)\right)} {\displaystyle\sum_{j=0}^{l-1}\exp\!\left(\mathbf q_l^\top\operatorname{RMSNorm}(\mathbf v_j)\right)}.

AttnRes therefore performs single-stream softmax attention along the depth axis. MSAR generalizes this mechanism by constructing several full-dimensional, context-dependent views of every source and assigning each view an independent depth-attention distribution.

MSAR differs from mHC and other persistent multi-stream residual systems. mHC maintains multiple residual trajectories continuously and uses stream-mixing transformations to read, update, and redistribute them. MSAR instead retains a single-stream source history, expands sources on demand, retrieves independently through multiple contextual streams, and reduces the result back to one layer input. Its defining computational pattern is:

single source→multi-scale contextual views→independent depth attention→adaptive fusion→single-stream layer input.\text{single source} \rightarrow \text{multi-scale contextual views} \rightarrow \text{independent depth attention} \rightarrow \text{adaptive fusion} \rightarrow \text{single-stream layer input}.

The term “stream” in MSAR consequently refers to full-dimensional contextual retrieval views rather than permanently stored residual states.

2. Source expansion through contextual streams

For a source vi∈Rd\mathbf v_i\in\mathbb R^d, MSAR constructs mm streams:

flf_l0

where the identity stream is

flf_l1

and the remaining operators flf_l2 are causal depthwise one-dimensional convolutions.

Restoring the token index flf_l3, the identity stream is

flf_l4

For flf_l5, a contextual stream is computed as

flf_l6

where flf_l7 contains channel-wise convolution weights, flf_l8 denotes elementwise multiplication, and positions preceding the beginning of the sequence are zero-padded. The convolutions are causal: each stream uses the current token and preceding tokens but not future tokens.

Different kernel sizes provide distinct contextual views. The default configuration uses kernel sizes

flf_l9

together with the identity stream, giving v0=h1\mathbf v_0=\mathbf h_10. Every stream remains full-dimensional in v0=h1\mathbf v_0=\mathbf h_11. This distinguishes MSAR from channel-wise multi-head constructions, which divide the hidden dimension into smaller subspaces. MSAR instead gives each stream access to the complete representation, with contextual diversity introduced along the sequence axis.

The identity stream preserves the original source representation. The convolutional streams incorporate progressively broader preceding context. The resulting diversity is therefore not produced solely by independent learned queries: the source values themselves differ across streams. In the ablation study, removing the convolutions while retaining four independent pseudo-queries raises the loss from v0=h1\mathbf v_0=\mathbf h_12 to $1.862, indicating that contextual source expansion is central to the method.

3. Independent depth retrieval

For each target layer $\mathbf v_0=\mathbf h_1$3 and stream $\mathbf v_0=\mathbf h_1$4, MSAR uses a separate learned pseudo-query

$\mathbf v_0=\mathbf h_1$5

The stream-specific key is the contextualized source itself:

$\mathbf v_0=\mathbf h_1$6

Using the score kernel

$\mathbf v_0=\mathbf h_1$7

MSAR computes independent depth-attention weights:

$\mathbf v_0=\mathbf h_1$8

Each stream has its own normalized distribution:

$\mathbf v_0=\mathbf h_1$9

The retrieved state is

$\mathbf v_i=f_i(\mathbf h_i)$0

Equivalently, separating the embedding source from branch outputs,

$\mathbf v_i=f_i(\mathbf h_i)$1

Collecting the retrieved representations gives

$\mathbf v_i=f_i(\mathbf h_i)$2

The identity and contextual streams can therefore retrieve different historical depths. A short-context stream may emphasize local or recent representations, whereas a broader-context stream may assign greater weight to sources whose surrounding sequence context matches the learned query. The precise routing is input-dependent because the attention scores depend on the source representations, while the depth preferences are parameterized by layer- and stream-specific pseudo-queries.

MSAR pseudo-queries and convolution weights are initialized to zero. The pre-mixer scale is initialized to $\mathbf v_i=f_i(\mathbf h_i)$3. Zero initialization makes the initial routing distributions uniform and prevents arbitrary early source selection, while the small pre-mixer scale limits the initial effect of adaptive stream fusion.

4. Adaptive fusion and residual computation

The retrieved streams must be converted into one full-dimensional layer input. A conventional output projection from $\mathbf v_i=f_i(\mathbf h_i)$4 to $\mathbf v_i=f_i(\mathbf h_i)$5 would require a comparatively large matrix. MSAR instead uses a lightweight adaptive pre-mixer.

After flattening,

$\mathbf v_i=f_i(\mathbf h_i)$6

the input-dependent mixing weights are

$\mathbf v_i=f_i(\mathbf h_i)$7

where

$\mathbf v_i=f_i(\mathbf h_i)$8

$\mathbf v_i=f_i(\mathbf h_i)$9 is the sigmoid function, and $\mathbf h_l=\sum_{i=0}^{l-1}\mathbf v_i.$0 is a learned scalar initialized to $\mathbf h_l=\sum_{i=0}^{l-1}\mathbf v_i.$1.

The fused layer input is

$\mathbf h_l=\sum_{i=0}^{l-1}\mathbf v_i.$2

The Transformer branch then computes

$\mathbf h_l=\sum_{i=0}^{l-1}\mathbf v_i.$3

and the resulting single-stream source is retained for subsequent depth retrieval. The output of the residual mechanism is therefore again a single $\mathbf h_l=\sum_{i=0}^{l-1}\mathbf v_i.$4-dimensional representation, despite the intermediate multi-stream computation.

Adaptive mixing is not the primary source of MSAR’s gain but provides an additional improvement and improves stability. In the reported ablation, replacing the adaptive mixer with mean aggregation gives a loss of $\mathbf h_l=\sum_{i=0}^{l-1}\mathbf v_i.$5, compared with $\mathbf h_l=\sum_{i=0}^{l-1}\mathbf v_i.$6 for the default configuration.

MSAR should not be confused with an architecture that maintains multiple residual trajectories throughout the network. Its intermediate streams are generated only when depth retrieval is performed, and its output is collapsed back into a single stream before the Transformer branch. This preserves compatibility with ordinary single-stream Transformer blocks.

5. Block MSAR and systems implementation

Layer-level MSAR would require retaining every branch output as a depth source. Block MSAR reduces this requirement by aggregating contiguous residual sublayers into block summaries. Let the residual sublayers be partitioned into blocks

$\mathbf h_l=\sum_{i=0}^{l-1}\mathbf v_i.$7

The embedding is treated as

$\mathbf h_l=\sum_{i=0}^{l-1}\mathbf v_i.$8

The completed summary of block $\mathbf h_l=\sum_{i=0}^{l-1}\mathbf v_i.$9 is

$\mathbf h_l=\sum_{i=0}^{l-1}\alpha_{i\to l}\mathbf v_i, \qquad \sum_{i=0}^{l-1}\alpha_{i\to l}=1,$0

For a target layer $\mathbf h_l=\sum_{i=0}^{l-1}\alpha_{i\to l}\mathbf v_i, \qquad \sum_{i=0}^{l-1}\alpha_{i\to l}=1,$1, the current partial block summary is

$\mathbf h_l=\sum_{i=0}^{l-1}\alpha_{i\to l}\mathbf v_i, \qquad \sum_{i=0}^{l-1}\alpha_{i\to l}=1,$2

and is updated after computing branch $\mathbf h_l=\sum_{i=0}^{l-1}\alpha_{i\to l}\mathbf v_i, \qquad \sum_{i=0}^{l-1}\alpha_{i\to l}=1,$3 by

$\mathbf h_l=\sum_{i=0}^{l-1}\alpha_{i\to l}\mathbf v_i, \qquad \sum_{i=0}^{l-1}\alpha_{i\to l}=1,$4

Each stream retrieves from the embedding, completed previous blocks, and the current partial block:

$\mathbf h_l=\sum_{i=0}^{l-1}\alpha_{i\to l}\mathbf v_i, \qquad \sum_{i=0}^{l-1}\alpha_{i\to l}=1,$5

At the first layer of a block, the partial source is empty and excluded from the softmax denominator. Since each Transformer layer contains attention and MLP residual sublayers, grouping four Transformer layers corresponds to $\mathbf h_l=\sum_{i=0}^{l-1}\alpha_{i\to l}\mathbf v_i, \qquad \sum_{i=0}^{l-1}\alpha_{i\to l}=1,$6 residual sublayers per block. The main 18B model uses approximately $\mathbf h_l=\sum_{i=0}^{l-1}\alpha_{i\to l}\mathbf v_i, \qquad \sum_{i=0}^{l-1}\alpha_{i\to l}=1,$7 blocks, and the 28B model approximately $\mathbf h_l=\sum_{i=0}^{l-1}\alpha_{i\to l}\mathbf v_i, \qquad \sum_{i=0}^{l-1}\alpha_{i\to l}=1,$8, with the embedding treated as an additional source.

The key systems property is on-demand expansion. MSAR stores and communicates only the single-stream source $\mathbf h_l=\sum_{i=0}^{l-1}\alpha_{i\to l}\mathbf v_i, \qquad \sum_{i=0}^{l-1}\alpha_{i\to l}=1,$9 or block summary $\alpha_{i\to l} = \frac{\exp\!\left(\mathbf q_l^\top\operatorname{RMSNorm}(\mathbf v_i)\right)} {\displaystyle\sum_{j=0}^{l-1}\exp\!\left(\mathbf q_l^\top\operatorname{RMSNorm}(\mathbf v_j)\right)}.$0. The contextual streams $\alpha_{i\to l} = \frac{\exp\!\left(\mathbf q_l^\top\operatorname{RMSNorm}(\mathbf v_i)\right)} {\displaystyle\sum_{j=0}^{l-1}\exp\!\left(\mathbf q_l^\top\operatorname{RMSNorm}(\mathbf v_j)\right)}.$1 are deterministically generated when required by depth attention. Persistent storage is therefore $\alpha_{i\to l} = \frac{\exp\!\left(\mathbf q_l^\top\operatorname{RMSNorm}(\mathbf v_i)\right)} {\displaystyle\sum_{j=0}^{l-1}\exp\!\left(\mathbf q_l^\top\operatorname{RMSNorm}(\mathbf v_j)\right)}.$2 rather than $\alpha_{i\to l} = \frac{\exp\!\left(\mathbf q_l^\top\operatorname{RMSNorm}(\mathbf v_i)\right)} {\displaystyle\sum_{j=0}^{l-1}\exp\!\left(\mathbf q_l^\top\operatorname{RMSNorm}(\mathbf v_j)\right)}.$3 for $\alpha_{i\to l} = \frac{\exp\!\left(\mathbf q_l^\top\operatorname{RMSNorm}(\mathbf v_i)\right)} {\displaystyle\sum_{j=0}^{l-1}\exp\!\left(\mathbf q_l^\top\operatorname{RMSNorm}(\mathbf v_j)\right)}.$4 stored sources.

Block MSAR is compatible with two-phase execution. Completed block summaries are processed in parallel against the queries for target layers in the current block. The current partial block is then incorporated using online softmax. Fusing causal depthwise convolution with depth attention can avoid materializing expanded source streams.

The leading-order activation-I/O estimates per token and residual sublayer are

$\alpha_{i\to l} = \frac{\exp\!\left(\mathbf q_l^\top\operatorname{RMSNorm}(\mathbf v_i)\right)} {\displaystyle\sum_{j=0}^{l-1}\exp\!\left(\mathbf q_l^\top\operatorname{RMSNorm}(\mathbf v_j)\right)}.$5

$\alpha_{i\to l} = \frac{\exp\!\left(\mathbf q_l^\top\operatorname{RMSNorm}(\mathbf v_i)\right)} {\displaystyle\sum_{j=0}^{l-1}\exp\!\left(\mathbf q_l^\top\operatorname{RMSNorm}(\mathbf v_j)\right)}.$6

and

$\alpha_{i\to l} = \frac{\exp\!\left(\mathbf q_l^\top\operatorname{RMSNorm}(\mathbf v_i)\right)} {\displaystyle\sum_{j=0}^{l-1}\exp\!\left(\mathbf q_l^\top\operatorname{RMSNorm}(\mathbf v_j)\right)}.$7

For $\alpha_{i\to l} = \frac{\exp\!\left(\mathbf q_l^\top\operatorname{RMSNorm}(\mathbf v_i)\right)} {\displaystyle\sum_{j=0}^{l-1}\exp\!\left(\mathbf q_l^\top\operatorname{RMSNorm}(\mathbf v_j)\right)}.$8, $\alpha_{i\to l} = \frac{\exp\!\left(\mathbf q_l^\top\operatorname{RMSNorm}(\mathbf v_i)\right)} {\displaystyle\sum_{j=0}^{l-1}\exp\!\left(\mathbf q_l^\top\operatorname{RMSNorm}(\mathbf v_j)\right)}.$9, and $\text{single source} \rightarrow \text{multi-scale contextual views} \rightarrow \text{independent depth attention} \rightarrow \text{adaptive fusion} \rightarrow \text{single-stream layer input}.$0, the reported values are approximately $\text{single source} \rightarrow \text{multi-scale contextual views} \rightarrow \text{independent depth attention} \rightarrow \text{adaptive fusion} \rightarrow \text{single-stream layer input}.$1 for vanilla residuals, $\text{single source} \rightarrow \text{multi-scale contextual views} \rightarrow \text{independent depth attention} \rightarrow \text{adaptive fusion} \rightarrow \text{single-stream layer input}.$2 for Block AttnRes, $\text{single source} \rightarrow \text{multi-scale contextual views} \rightarrow \text{independent depth attention} \rightarrow \text{adaptive fusion} \rightarrow \text{single-stream layer input}.$3 for Block MSAR, and $\text{single source} \rightarrow \text{multi-scale contextual views} \rightarrow \text{independent depth attention} \rightarrow \text{adaptive fusion} \rightarrow \text{single-stream layer input}.$4 for mHC. These are activation-I/O estimates rather than total training costs.

For $\text{single source} \rightarrow \text{multi-scale contextual views} \rightarrow \text{independent depth attention} \rightarrow \text{adaptive fusion} \rightarrow \text{single-stream layer input}.$5 and kernel sizes $\text{single source} \rightarrow \text{multi-scale contextual views} \rightarrow \text{independent depth attention} \rightarrow \text{adaptive fusion} \rightarrow \text{single-stream layer input}.$6, $\text{single source} \rightarrow \text{multi-scale contextual views} \rightarrow \text{independent depth attention} \rightarrow \text{adaptive fusion} \rightarrow \text{single-stream layer input}.$7, and $\text{single source} \rightarrow \text{multi-scale contextual views} \rightarrow \text{independent depth attention} \rightarrow \text{adaptive fusion} \rightarrow \text{single-stream layer input}.$8, the additional MSAR parameter count is

$\text{single source} \rightarrow \text{multi-scale contextual views} \rightarrow \text{independent depth attention} \rightarrow \text{adaptive fusion} \rightarrow \text{single-stream layer input}.$9

where $\mathbf v_i\in\mathbb R^d$0 is the number of residual sublayers and $\mathbf v_i\in\mathbb R^d$1 is the hidden dimension. The reported additional parameter counts are $\mathbf v_i\in\mathbb R^d$2 million for the 18B model and $\mathbf v_i\in\mathbb R^d$3 million for the 28B model. Relative to the vanilla models, MSAR adds approximately $\mathbf v_i\in\mathbb R^d$4 activated parameters and $\mathbf v_i\in\mathbb R^d$5 training FLOPs at 18B, and $\mathbf v_i\in\mathbb R^d$6 activated parameters and $\mathbf v_i\in\mathbb R^d$7 training FLOPs at 28B.

6. Empirical evaluation, comparisons, and limitations

The experiments use DeepSeekMoE-style Transformers with grouped-query attention, SwiGLU feed-forward networks, RMSNorm, RoPE, 144 routed experts, top-8 routing, and one shared expert. The principal models contain 18B total parameters with 1.7B activated parameters and 28 Transformer layers, or 28B total parameters with 2.7B activated parameters and 32 Transformer layers. The comparison systems are vanilla residuals, mHC with four persistent streams, AttnRes with approximately eight blocks, and MSAR with four streams and approximately eight blocks.

MSAR outperforms vanilla residuals, mHC, and AttnRes on all 12 reported downstream benchmarks at both model scales.

Model scale Vanilla average mHC average AttnRes average MSAR average
18B 37.5 41.7 41.8 45.7
28B 44.6 47.4 49.4 52.4

At 18B, MSAR scores 56.6 on MMLU, 74.9 on ARC-C, 38.0 on BBH, 49.9 on GSM8K, 24.4 on MATH, and 52.3 on C-Eval. At 28B, it scores 62.5 on MMLU, 78.0 on ARC-C, 46.1 on BBH, 65.7 on GSM8K, 28.8 on MATH, and 59.9 on C-Eval. The final training losses are vi∈Rd\mathbf v_i\in\mathbb R^d8 for MSAR at 18B and vi∈Rd\mathbf v_i\in\mathbb R^d9 for MSAR at 28B in the reported main results.

The scaling-law experiments evaluate approximately 2.21B, 3.75B, 9.47B, and 17.40B total-parameter models using a shifted power law

mm0

with fixed estimated irreducible loss mm1. The reported average iso-loss compute speedups over vanilla residuals are mm2 for MSAR, mm3 for AttnRes, and mm4 for mHC. These are fitted loss–compute comparisons, not claims that a single MSAR forward pass is literally mm5 times faster.

Ablations identify the principal components of the method. Removing contextual convolution raises the loss from mm6 to mm71.851mm8m=2mm9m=4flf_l001.850flf_l011.847, while flf_l02 produces a marginally lower loss of flf_l031.843flf_l041.845flf_l051.847flf_l061.852flf_l071.861flf_l081.847$, indicating that MSAR is not explained simply by adding independent historical retrieval and persistent multi-stream recurrence.

MSAR has several limitations. Its efficiency depends on block-level source accumulation, on-demand stream expansion, online softmax, and fused kernels. A naive implementation that materializes every contextual stream for every source could lose the storage and bandwidth advantages. Compared with AttnRes, MSAR requires additional convolution, multiple depth-attention computations, storage of retrieved vectors, adaptive fusion, and greater activation traffic. The reported scaling speedups depend on the chosen loss model, fixed irreducible loss, and evaluated model scales. The experiments also provide limited evidence regarding other tasks, larger or smaller stream counts, alternative contextual operators, and extensive hardware-specific throughput.

MSAR is consequently best characterized as a multi-stream extension of depth-wise attention residuals rather than as a persistent multi-stream recurrence. Its central contribution is to use full-dimensional, multi-scale contextual views to create independently routed depth memories while retaining a single-stream source representation for storage, communication, and subsequent Transformer computation.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (1)

Topic to Video (Beta)

No one has generated a video about this topic yet.

Whiteboard

No one has generated a whiteboard explanation for this topic yet.

Follow Topic

Get notified by email when new papers are published related to Multi-Stream Attention Residuals (MSAR).