Multi-Stream Attention Residuals
Abstract: Attention Residuals (AttnRes) enables content-dependent retrieval over network depth, which can be viewed as single-stream softmax attention along the depth axis. Inspired by the success of multi-head attention along the sequence axis, we introduce Multi-Stream Attention Residuals (MSAR), an extension of AttnRes that generalizes this mechanism to multi-head depth attention, yielding substantial performance gains with minimal additional overhead. MSAR draws stream diversity from sequence context through multi-scale causal convolutions, allowing different heads to retrieve historical representations through distinct contextual views while preserving the integrity of the hidden representation. The retrieved streams are combined through lightweight adaptive mixing to generate the single-stream layer input. By retaining single-stream sources and expanding them on demand, MSAR supports multi-stream computation without requiring persistent multi-stream storage or communication. Across all 12 downstream benchmarks at the 18B and 28B MoE scales, MSAR consistently outperforms prior residual variants, including standard residual connections, manifold-constrained Hyper-Connections (mHC), and AttnRes. Scaling-law experiments yield an iso-loss compute speedup of over the standard residual baseline, compared with for AttnRes and for mHC. These results make MSAR a promising residual architecture for large-scale pre-training.
Paper Prompts
Sign up for free to create and run prompts on this paper.
Explain it Like I'm 14
1. What is this paper about?
This paper introduces a new design for the “shortcut connections” inside LLMs called Multi-Stream Attention Residuals, or MSAR.
LLMs are built from many layers. Each layer improves the model’s understanding of the text. Usually, information from earlier layers is passed forward through a single running pathway, called a residual stream.
The authors argue that this single pathway can cause older information to become mixed together or weakened as the model gets deeper. MSAR is designed to help later layers find and reuse useful information from earlier layers more effectively.
A simple way to think about it is this:
Instead of keeping all information in one long notebook, the model creates several organized views of its past notes and chooses the most useful parts when needed.
2. What questions does the research ask?
The paper mainly investigates four questions:
- Can a LLM use several different “views” of earlier information instead of only one?
- Can these views help later layers retrieve useful information from the model’s past?
- Can this be done without greatly increasing the model’s memory use or computing cost?
- Does this new design improve language understanding, reasoning, mathematics, and coding?
The researchers compare MSAR with three other approaches:
| Method | Basic idea |
|---|---|
| Standard residual connection | Adds each layer’s new information to one shared stream |
| mHC | Keeps several persistent streams of information |
| AttnRes | Lets later layers use attention to retrieve earlier layer outputs |
| MSAR | Creates several context-based streams and lets each one retrieve information separately |
3. How does MSAR work?
The problem with ordinary residual connections
In a standard Transformer, each layer adds its output to the current hidden representation:
1 |
new information = current information + layer update |
This is useful because information can travel through many layers. However, after many additions, it may become difficult for a later layer to tell exactly where a particular piece of information came from.
Imagine adding notes from 30 lessons onto one giant piece of paper. The paper contains everything, but finding one important fact may become difficult.
How AttnRes improves this
Attention Residuals, or AttnRes, keep earlier layer outputs as separate sources. A later layer can then decide which earlier sources are important.
This is similar to searching through a filing cabinet instead of reading one giant page. The model gives higher weights to the files it thinks are most useful.
The technical term attention means that the model calculates scores for different pieces of information and gives more importance to the pieces with higher scores.
What MSAR adds
MSAR extends this idea by creating several different versions, or streams, of each earlier representation.
In the main experiments, the model uses four streams:
- The original representation
- A version looking at a short range of nearby tokens
- A version looking at a medium range
- A version looking at a longer range
To create these versions, MSAR uses causal depthwise convolutions.
This sounds complicated, but the basic idea is simple:
- A convolution is a small sliding window that combines nearby pieces of information.
- Causal means the model only looks at the current token and earlier tokens, never future tokens. This is important because a LLM should not see the answer before predicting it.
- Depthwise means the operation is kept lightweight and is applied separately across parts of the representation.
For example, one stream may focus on the previous four token positions, another on the previous eight, and another on the previous twelve. These different windows give the model different views of the same information.
Retrieving information from earlier layers
Each stream then independently searches through earlier layer outputs using attention.
This means that one stream might focus on information from an early layer, while another stream might focus on a later layer. They do not all have to make the same choice.
Afterward, MSAR combines the streams into one representation for the next layer. It uses learned weights, so the model can decide how much to trust each stream.
An analogy would be asking four students to search a library using different strategies:
- One searches recent books.
- One searches books about a specific topic.
- One searches older reference books.
- One searches broadly.
The teacher then combines their answers into one final response.
Keeping the method efficient
A possible problem is that storing four streams for every layer could require a lot of memory. MSAR avoids this by storing only the main single stream.
When the extra streams are needed, the model recreates them from the stored information. The authors call this expanding the streams on demand.
This is like storing one original photograph and applying different filters only when someone asks to see different versions. The model does not need to permanently save every filtered copy.
4. How did the researchers test it?
The researchers trained large mixture-of-experts, or MoE, LLMs with:
- 18 billion total parameters
- 28 billion total parameters
A parameter is a number learned during training. These numbers help the model recognize patterns in language.
They compared MSAR with standard residual connections, mHC, and AttnRes.
The models were tested on 12 benchmarks covering:
- General knowledge and reasoning
- Mathematics
- Computer programming
- Chinese language understanding
They also performed:
- Scaling-law experiments, which study how performance changes as models receive more computing power
- Ablation studies, which remove or change parts of MSAR to discover which components are most useful
An ablation study is similar to taking a machine apart piece by piece to see which parts are necessary.
5. What were the main findings?
MSAR performed best on all tested benchmarks
In the 18-billion-parameter models, the average scores were:
| Method | Average score |
|---|---|
| Standard residual | 37.5 |
| mHC | 41.7 |
| AttnRes | 41.8 |
| MSAR | 45.7 |
In the 28-billion-parameter models:
| Method | Average score |
|---|---|
| Standard residual | 44.6 |
| mHC | 47.4 |
| AttnRes | 49.4 |
| MSAR | 52.4 |
According to the paper, MSAR achieved the best result on all 12 benchmarks at both model sizes.
It improved performance in areas such as:
- General knowledge
- Difficult reasoning
- Mathematical problem-solving
- Code generation
- Chinese reading and understanding
For example, on the 28-billion-parameter model, the score on GSM8K, a mathematics word-problem test, was:
- Standard residual: 50.3
- mHC: 56.3
- AttnRes: 57.2
- MSAR: 65.7
MSAR also reduced training loss
Training loss measures how often the model’s predictions are wrong during training. Lower loss usually means the model is learning the training patterns more effectively.
For the 18-billion-parameter model, the final losses were:
| Method | Final training loss |
|---|---|
| Standard residual | 1.799 |
| mHC | 1.776 |
| AttnRes | 1.769 |
| MSAR | 1.750 |
The paper also reports that MSAR reached the same loss using about 1.58 times less computing power than the standard residual design. This is called an iso-loss compute speedup: it compares how much computation different methods need to reach the same quality level.
For comparison:
- AttnRes achieved a speedup of 1.35 times.
- mHC achieved a speedup of 1.30 times.
- MSAR achieved a speedup of 1.58 times.
The extra cost was small
In the 28-billion-parameter experiment, MSAR added approximately:
- 0.3% more activated parameters
- 0.5% more training computation
This is important because a method that improves accuracy but requires many times more computing power may not be practical. The authors claim MSAR provides noticeable benefits with only a small additional cost.
The different streams really mattered
The ablation experiments showed that simply copying the same representation four times did not work as well.
When the convolution step was removed, the loss became worse. This suggests that the streams need to contain genuinely different information, rather than being identical copies.
The experiments also found:
- Four streams worked well.
- Eight streams produced only a very small additional improvement.
- Adaptive mixing of streams helped, although most of the improvement came from creating useful, different views.
- Using around eight blocks gave most of the benefit while reducing memory and computation compared with using every layer separately.
6. Why are these results important?
The results suggest that the way information moves between layers can be just as important as the number of layers in a LLM.
Simply adding more layers does not always lead to better performance. If later layers cannot easily access useful information from earlier layers, some of the model’s capacity may be wasted.
MSAR attempts to solve this by giving later layers:
- Direct access to earlier information
- Several different context-based views
- The ability to choose different earlier sources for different streams
- A way to combine the retrieved information flexibly
The method may therefore help LLMs make better use of their depth.
7. Possible impact and limitations
If the results continue to hold in future studies, MSAR could help researchers build LLMs that are:
- More accurate at similar sizes
- Better at reasoning, mathematics, and coding
- More efficient to train
- Better at using information from earlier layers
- Easier to scale to very large models
The paper focuses on large MoE LLMs and reports strong results in its own experiments. However, further research would be useful to test MSAR on other model types, datasets, languages, and real-world applications.
Simple conclusion
MSAR is a new way for a LLM to remember and reuse information as it passes through many layers. Instead of forcing every layer to use one mixed stream of information, it creates several different views of the past. Each view can search for useful information in its own way, and the model combines the results.
The experiments show that this design improved accuracy across all 12 tested tasks while adding very little computing cost. In simple terms, MSAR helps a LLM make better use of its past thoughts without making the model much more expensive to run.
Practical Applications
Immediate Applications
- More compute-efficient pretraining of LLMs — Industry / Software
- Integrate MSAR or Block MSAR into Transformer-based pretraining stacks as a replacement for standard residual connections, particularly for large MoE models.
- The reported results show improved loss–compute efficiency, including a iso-loss compute speedup over standard residuals and higher performance across knowledge, reasoning, mathematics, code, and Chinese-language benchmarks.
- A practical implementation could expose MSAR as a configurable residual module in frameworks such as PyTorch, DeepSpeed, Megatron-style training systems, or internal LLM platforms.
- Dependencies: Reliable reproduction outside the reported 10B–28B MoE settings; implementation of causal depthwise-convolution kernels, depth-attention caching, and efficient stream fusion.
- Reducing distributed-training communication and activation memory — Cloud infrastructure / HPC
- Use Block MSAR in pipeline-parallel training to retain and communicate only single-stream block summaries rather than persistent multi-stream activations.
- According to the paper, this changes persistent source storage and pipeline communication from to , where is the number of streams and the number of stored blocks.
- This could reduce inter-GPU or inter-node communication pressure, especially in deep models distributed across many pipeline stages.
- Dependencies: The claimed savings assume deterministic on-demand stream expansion, effective kernel fusion, and a pipeline schedule compatible with the two-phase computation strategy.
- Improving quality at a fixed training budget — Commercial model development
- Apply MSAR when training budgets are constrained by GPU hours, energy, or cloud cost. The architecture could be used to obtain better model quality without proportionally increasing model size or training FLOPs.
- The paper reports only approximately 0.5% additional training FLOPs and 0.3% additional activated parameters relative to the baseline in its 28B setting.
- Potential products include lower-cost enterprise LLMs, faster domain adaptation, and improved coding or multilingual assistants.
- Dependencies: The observed gains may depend on model depth, block granularity, MoE routing, data quality, and sufficiently large-scale pretraining.
- Engineering and coding assistants — Software development
- Deploy an MSAR-based foundation model for code completion, code review, mathematical programming assistance, test generation, and repository-level question answering.
- The reported improvements on HumanEval, LCBench, GSM8K, and MATH suggest potential value for code and quantitative reasoning workloads, although the benchmarks do not establish production reliability.
- Dependencies: Additional testing for long-context repository reasoning, factuality, secure code generation, hallucination rates, and performance after instruction tuning or reinforcement learning.
- Educational tutoring and assessment systems — Education
- Use MSAR-trained models for multilingual tutoring, automated explanation generation, formative feedback, practice-question generation, and educational content search.
- Gains on MMLU-family benchmarks, ARC-C, BBH, CMMLU, C-Eval, and C3 indicate possible benefits for broad subject knowledge and Chinese-language understanding.
- A practical workflow would combine the model with retrieval-augmented generation, curriculum constraints, citation mechanisms, and teacher review.
- Dependencies: Benchmark gains do not guarantee pedagogical quality, age appropriateness, fairness, or factual accuracy; human oversight remains necessary.
- Chinese-language and multilingual information services — Public-sector and consumer applications
- Incorporate MSAR into Chinese-language search, translation assistance, customer service, document analysis, and public-information chatbots.
- The consistent improvements on CMMLU, C-Eval, and C3 suggest a possible advantage for Chinese-language reasoning and comprehension.
- Dependencies: Validation across dialects, regional terminology, sensitive topics, legal requirements, and real user distributions is required before public deployment.
- Model-training research and ablation infrastructure — Academia
- Use the open implementation and architecture as a testbed for studying depth-wise information retrieval, residual-stream design, and the relationship between sequence context and depth memory.
- Researchers can compare standard residuals, AttnRes, mHC, hybrid designs, and MSAR under matched parameter, token, and compute budgets.
- Useful measurements include layer-utilization profiles, depth-attention distributions, activation norms, gradient flow, memory traffic, and scaling-law parameters.
- Dependencies: Reproducibility requires access to comparable training data, tokenizer settings, optimization schedules, model sizes, and hardware-aware implementations.
- Training-cost and energy accounting — Sustainability / Policy analysis
- Use the reported compute-efficiency measurements as an input to estimating the potential energy and infrastructure savings of large-model training.
- An organization could compare the energy per target loss or benchmark score for standard residuals, AttnRes, mHC, and MSAR.
- Dependencies: The reported speedup is a scaling-law result rather than a universal reduction in wall-clock energy. Actual savings depend on kernel efficiency, memory bandwidth, utilization, hardware generation, and total training duration.
- Daily-life AI assistants — Consumer software
- MSAR-based models could support writing assistants, personal knowledge tools, multilingual chatbots, planning applications, and document summarization.
- The architecture is especially relevant when better reasoning quality is desired without increasing inference model size substantially.
- Dependencies: The paper evaluates pretraining performance, not complete deployed assistants. Instruction tuning, alignment, latency testing, safety evaluation, privacy protection, and serving optimization are still required.
Long-Term Applications
- Deep, high-capacity foundation models with improved layer utilization — Industry / Research
- MSAR could enable deeper Transformer models that make more effective use of early-layer representations instead of allowing information to be diluted through a single additive residual trajectory.
- Potential future products include more capable general-purpose models, reasoning models, scientific assistants, and agentic systems.
- Dependencies: The experiments cover 18B and 28B MoE models, so the architecture must be validated at substantially larger scales, with dense models, long-context models, multimodal models, and alternative normalization or attention designs.
- Multimodal models — Vision, speech, and robotics
- The residual-retrieval principle could be adapted to vision-LLMs, speech models, video transformers, and robotic policies, allowing later layers to retrieve representations formed at different processing depths and contextual scales.
- In robotics, different depth streams might preserve motion, spatial, language, or task-level information for later action selection.
- Dependencies: The current MSAR expansion uses causal one-dimensional convolutions over token sequences. Multimodal adaptation requires suitable causal structures for images, video, continuous sensor streams, and cross-modal token layouts.
- Long-context language modeling — Search, enterprise knowledge, and agents
- Multi-scale source expansion could be extended to retrieve representations associated with different local-context windows, potentially helping models distinguish short-range syntax from broader contextual information.
- Possible tools include long-document analysis, legal or scientific retrieval assistants, and agents operating over large workspaces.
- Dependencies: The paper does not directly evaluate long-context extrapolation. The method may introduce additional memory traffic or retrieval complexity as sequence length grows, and it must be tested for positional robustness and retrieval accuracy.
- Adaptive depth and conditional computation — Efficient inference
- MSAR’s depth-attention weights could become a basis for selectively emphasizing or skipping blocks, creating models that allocate computation according to the input.
- This may support low-latency inference for simple requests and deeper processing for difficult reasoning tasks.
- Dependencies: The paper retrieves from historical representations but does not demonstrate layer skipping. Safe conditional execution requires new training objectives, routing mechanisms, calibration, and guarantees that skipped computation will not degrade reliability.
- Hardware-aware accelerator kernels and compiler support — Systems / Semiconductor
- The architecture motivates fused kernels that combine causal depthwise convolution, depth-wise attention, online softmax, and stream fusion without materializing intermediate streams.
- Future products could include specialized GPU kernels, compiler passes, or accelerator primitives optimized for depth attention and residual retrieval.
- Dependencies: The reported efficiency assumes such fusion is practical. Benefits may disappear on hardware with unfavorable memory-access patterns, limited kernel fusion support, or insufficient bandwidth.
- Energy-efficient AI infrastructure — Data centers and climate policy
- If the compute-efficiency gains persist at larger scales, MSAR could reduce the energy and carbon intensity of foundation-model training and possibly lower the cost of repeated retraining.
- This could inform procurement standards, model-efficiency targets, and reporting frameworks for AI infrastructure.
- Dependencies: Reduced training loss per FLOP does not automatically imply proportional reductions in total emissions. Organizations may use the savings to train larger models, increase experiment volume, or serve more users, producing rebound effects.
- Domain-specialized models in healthcare, finance, and law — High-stakes applications
- An MSAR-based backbone could support medical literature assistants, financial analysis systems, legal document review, and regulated enterprise search.
- Better retrieval over depth may be useful for models that must combine terminology, procedural knowledge, and long documents.
- Dependencies: The paper provides no evidence of clinical, financial, or legal safety. Deployment would require domain-specific validation, privacy controls, auditability, explainability, bias testing, human authorization, and compliance with applicable regulation.
- Policy and standards for efficient model training — Government and research governance
- The paper’s comparison of loss, downstream accuracy, FLOPs, activated parameters, memory, and communication can contribute to standards for reporting the efficiency of model architectures.
- A policy workflow could require reporting not only parameter count but also compute per target quality, communication volume, activation memory, and energy use.
- Dependencies: Standardized hardware and accounting protocols are needed. Results from one family of MoE models should not be treated as universal evidence that MSAR is more efficient for every architecture or workload.
- General residual architectures beyond LLMs — Academic theory
- MSAR suggests a broader design pattern: preserve compact single-stream states, generate multiple contextual views on demand, perform independent retrieval, and adaptively fuse the results.
- This could inspire new architectures for recurrent networks, graph neural networks, diffusion models, neural simulators, and sequential decision systems.
- Dependencies: The benefits may rely on autoregressive token structure and causal convolution. Each new domain would require an appropriate notion of context, source integrity, and efficient historical-state access.
- Automated architecture search for residual mechanisms — Research tooling
- The space of stream counts, convolutional receptive fields, block sizes, fusion functions, and source-selection policies could be optimized automatically for a target hardware and data regime.
- Search systems might learn whether two, four, or more streams are appropriate and whether full-layer or block-level retrieval provides the best quality–cost trade-off.
- Dependencies: Architecture search can be expensive and may overfit benchmark suites. Reliable evaluation must include training cost, inference latency, memory traffic, robustness, and performance across multiple datasets and scales.
Glossary
- Adaptive scalar mixing: Combining retrieved representations using input-dependent scalar weights. “aggregates the retrieved states with adaptive scalar mixing.”
- Autoregressive modeling: A modeling approach in which predictions depend only on preceding sequence positions. “Under autoregressive modeling with a causal mask, earlier hidden states in the same layer encode the context available at their respective positions”
- Birkhoff polytope: The set of doubly stochastic matrices, whose entries are nonnegative and whose rows and columns each sum to one. “projects $\mathcal{H}^{\mathrm{res}_l$ onto the Birkhoff polytope of doubly stochastic matrices”
- Causal convolution: A convolution restricted to current and earlier positions so that future information cannot be used. “we use lightweight causal depthwise 1D convolutions to expand each residual source into streams”
- Causal depthwise convolution: A position-causal convolution applied independently to each feature channel. “a layer-specific causal depthwise convolution with kernel size ”
- Causal mask: An attention mask that prevents a position from accessing later positions. “Under autoregressive modeling with a causal mask”
- Channel-wise convolution weights: Convolution parameters applied independently to individual feature channels. “where contains the channel-wise convolution weights”
- Cross-layer access: Direct access by later layers to representations produced by earlier layers. “Another line of work gives later layers explicit access to earlier representations”
- Cross-stage caching: Reusing stored representations across different stages of a distributed pipeline. “It remains fully compatible with AttnRes's cross-stage caching”
- Depth attention: Attention performed across network layers or residual depth rather than across sequence positions. “MSAR turns residual retrieval into a multi-head depth-attention primitive”
- Depth mixing: Combining representations originating from different network depths. “Thus, standard residual connections perform depth mixing with fixed unit coefficients”
- Depth scaling: Increasing the number of layers in a neural network to improve its representational capacity. “Scaling depth has been a primary lever for improving Transformer-based LLMs”
- Depth-wise information dilution: The gradual weakening or loss of early-layer features as more residual updates are accumulated. “This depth-wise information dilution limits the effective use of additional layers”
- Doubly stochastic matrix: A nonnegative matrix whose rows and columns each sum to one. “mHC further projects $\mathcal{H}^{\mathrm{res}_l$ onto the Birkhoff polytope of doubly stochastic matrices”
- FLOPs: Floating-point operations, used as a measure of computational cost. “In our 28B MoE setting, MSAR increases training FLOPs by only ”
- Fused kernel: A combined implementation of multiple operations that reduces intermediate memory transfers. “this allows stream expansion and consumption to be handled by a fused kernel”
- Gated residual: A residual mechanism that controls information flow using learned gates. “Gated Residual~\citep{qiu2026design} uses elementwise gated reads and input-dependent scalar writes over multiple residual streams”
- Hidden representation: The internal vector representation maintained by a neural network. “while preserving the integrity of the hidden representation”
- Inductive bias: A modeling preference or structural assumption that guides what patterns a model learns. “showing that the gains come from the proposed inductive bias rather than from a model-size-specific artifact”
- Irreducible loss: The asymptotic minimum loss that cannot be eliminated by adding more training computation. “ is the estimated irreducible loss, which represents the asymptotic loss floor as training compute increases”
- Iso-loss compute speedup: The factor by which less computation is needed to reach the same loss. “MSAR achieves an iso-loss compute speedup of ”
- Kernel size: The number of neighboring positions included by a convolution. “with kernel size ”
- Manifold-constrained Hyper-Connections (mHC): A multi-stream residual architecture that constrains its stream-mixing matrix to the manifold of doubly stochastic matrices. “Its manifold-constrained variant, mHC, stabilizes multi-stream recurrence through doubly stochastic residual mixing”
- Multi-head depth attention: Multiple independent attention mechanisms operating over network depth. “we introduce Multi-Stream Attention Residuals (MSAR), an extension of AttnRes that generalizes this mechanism to multi-head depth attention”
- Multi-scale convolution: Convolution using different receptive-field sizes to capture context at multiple scales. “it applies a small set of multi-scale causal depthwise convolutions to the layer output”
- Mixture-of-Experts (MoE): A neural architecture that routes inputs to selected expert subnetworks rather than activating all parameters. “Our main results cover two MoE scales with 18B and 28B total parameters”
- Online softmax: A streaming method for computing softmax statistics while incrementally incorporating additional values. “This recovers the attention in Eq.~\ref{eq:mars-block-retrieval}”
- Persistent residual stream: A residual representation maintained across layers rather than generated temporarily. “A direct way to relax this single-state constraint is to expand the residual state into multiple persistent streams”
- Pipeline communication: Transfer of intermediate representations between stages in a distributed model pipeline. “MSAR reduces pipeline communication and prefill activation storage from to ”
- PreNorm architecture: A Transformer design that applies normalization before the main sublayer computation. “In modern PreNorm architectures”
- Pseudo-query: A learned vector used as a query for retrieving relevant representations, without being derived from a separate input sequence. “AttnRes instantiates this attention with a layer-specific learned pseudo-query”
- Receptive field: The range of input positions that can influence a unit’s output. “producing full-dimensional streams whose receptive fields encode different local-context scales”
- Residual accumulation: The repeated addition of layer outputs to a running hidden representation. “A standard residual stack propagates representations by adding each branch output to the running hidden state”
- Residual branch: A Transformer sublayer, such as an attention or MLP sublayer, whose output participates in a residual update. “each self-attention or MLP branch in a Transformer stack is counted as one layer”
- Residual retrieval: Selecting and combining earlier residual representations for use as a later layer input. “MSAR generalizes AttnRes from single-stream residual retrieval to multi-head depth attention”
- Residual stream: The sequence of hidden states passed through a network’s residual connections. “The residual stream aggregates preceding computations through a single hidden-state trajectory”
- RMSNorm: Root Mean Square Layer Normalization, which rescales a vector using its root-mean-square magnitude. “”
- Scaling law: An empirical relationship describing how model performance changes with scale or computational resources. “Scaling-law experiments further confirm that MSAR reaches the same loss with substantially less compute”
- Sequence-context view: A representation of a source that incorporates contextual information from neighboring sequence positions. “different heads can select different depth histories through distinct sequence-context views”
- Shifted power law: A power-law scaling relationship with an additive offset representing a limiting value. “We fit a shifted power law to the final training loss”
- Sinkhorn--Knopp algorithm: An iterative procedure that normalizes a nonnegative matrix toward a doubly stochastic matrix. “via the Sinkhorn--Knopp algorithm”
- Softmax retrieval: Selecting and weighting representations using softmax-normalized similarity scores. “AttnRes replaces fixed residual accumulation with learned depth-wise softmax retrieval over earlier representations”
- Stream expansion: Transforming one residual representation into multiple contextual streams. “Multi-Stream Source Expansion.”
- Stream fusion: Combining multiple retrieved streams into one representation. “The retrieved streams must be fused into a single input for the target layer”
- Stream diversity: Variation among parallel residual representations that enables them to capture different information. “MSAR draws stream diversity from sequence context through multi-scale causal convolutions”
- Two-phase computation: A computational strategy that separates source-based retrieval from processing of the current partial representation. “This enables a two-phase computation strategy for MSAR”
- Virtual pipeline stage: A logical subdivision of a distributed model pipeline used to organize computation and caching. “allowing previously received summaries to be reused across virtual pipeline stages without retransmission”





