Multi-Scale Linear Attention

• Multi-Scale Linear Attention (MSLA) is a class of attention mechanisms that integrates global context with multi-scale feature aggregation to overcome the quadratic cost of softmax attention.
• It employs diverse implementations such as ReLU-based aggregation, Householder reflections, and latent bottleneck strategies to balance accuracy, positional fidelity, and computational efficiency.
• Empirical evaluations demonstrate that MSLA achieves significant speedups and improved performance in tasks like dense prediction, stereo matching, and hyperspectral image analysis.

Searching arXiv for papers on Multi-Scale Linear Attention and related formulations. Multi-Scale Linear Attention (MSLA) denotes a class of attention mechanisms that combine global context aggregation with explicit multi-scale structure while avoiding the quadratic token-token interaction cost of standard softmax attention. In the most direct usage, EfficientViT presents MSLA for high-resolution dense prediction, where ReLU linear attention is combined with lightweight multi-scale token aggregation to obtain a global receptive field and multi-scale learning using hardware-efficient operations (Cai et al., 2022). Subsequent work instantiates the same design objective in different ways: MirrorLA realizes a multi-scale linear-attention pipeline through local block-wise isometries, long-context variance-aware modulation, and global cross-head reflections (Meng et al., 4 Feb 2026), while LinStereo realizes MSLA through Position-Aware Linear Attention (PALA) together with Hierarchical Semantic Cost Volumes and Depth Prior Initialization (Wang et al., 24 Jun 2026). A related but terminologically distinct usage appears in CLAReSNet, where MSLA denotes “Multi-Scale Spectral Latent Attention,” a latent-bottleneck exact-softmax mechanism for hyperspectral spectral sequences rather than a kernelized linear-attention operator (Bandyopadhyay et al., 15 Nov 2025). This suggests that MSLA is best understood as an architectural pattern centered on multi-scale, subquadratic context modeling rather than as a single canonical formula.

1. Definition and scope

MSLA addresses a recurrent tension in attention-based vision models: dense prediction and long-context reasoning require both global receptive field and multi-scale learning, but standard softmax attention requires forming all pairwise similarities and therefore incurs quadratic time and memory in the number of tokens. EfficientViT formulates this problem for high-resolution dense prediction and proposes a multi-scale linear-attention block that replaces softmax attention, large-kernel convolution, and complicated topology structure with lightweight operations such as ReLU, depthwise and pointwise convolutions, and grouped convolutions (Cai et al., 2022).

The term subsequently appears in distinct but structurally related settings. MirrorLA identifies a specific failure mode in conventional linear attention—information loss induced by non-negativity constraints and “passive truncation”—and builds a three-level multi-scale design around geometric reorientation before ReLU (Meng et al., 4 Feb 2026). LinStereo uses a position-aware linear attention module inside a multi-scale iterative stereo-matching pipeline, where the relevant scales are feature resolutions and disparity pyramids rather than branches in a backbone block (Wang et al., 24 Jun 2026). CLAReSNet uses the same acronym for a latent-attention module over spectral sequences with scales $s \in \{1,2,4\}$, but explicitly states that it preserves exact softmax attention and reduces complexity through learnable latent tokens instead of kernel feature maps (Bandyopadhyay et al., 15 Nov 2025).

A common denominator across these formulations is that “multi-scale” does not have a single architectural meaning. It may refer to local neighborhood aggregation with small kernels, hierarchical semantic cost volumes, block-wise geometric control, or temporal/spectral downsampling with latent fusion. The unifying objective is to preserve global or long-range interaction while embedding scale structure directly into the attention pathway.

2. Computational principle

The baseline reference is standard softmax attention. For queries, keys, and values $Q, K, V \in \mathbb{R}^{N \times d}$, softmax attention computes

$$\mathbf{o}_t = \sum_{i=1}^N \frac{\exp(\mathbf{q}_t \mathbf{k}_i^\top/\sqrt{d})}{\sum_{j=1}^N \exp(\mathbf{q}_t \mathbf{k}_j^\top/\sqrt{d})} \,\mathbf{v}_i,$$

which requires the $N \times N$ similarity matrix and therefore costs $O(N^2 d)$ time and $O(N^2)$ memory (Meng et al., 4 Feb 2026).

Linear attention replaces the softmax kernel with a separable feature map $\phi(\cdot)$ such that

$$\operatorname{sim}(\mathbf{q},\mathbf{k}) = \phi(\mathbf{q})\,\phi(\mathbf{k})^\top.$$

Using associativity, one can compute

$$\mathbf{O} = \frac{\phi(\mathbf{Q})\big(\phi(\mathbf{K})^\top \mathbf{V}\big)}{\phi(\mathbf{Q})\big(\phi(\mathbf{K})^\top \mathbf{1}\big)},$$

or, in EfficientViT notation, precompute a global key-value accumulator $S = \sum_j \phi(K_j)V_j^\top$ and a key summary $Q, K, V \in \mathbb{R}^{N \times d}$0, then evaluate each output as

$Q, K, V \in \mathbb{R}^{N \times d}$1

This yields time complexity $Q, K, V \in \mathbb{R}^{N \times d}$2 and memory $Q, K, V \in \mathbb{R}^{N \times d}$3 in EfficientViT’s formulation, and $Q, K, V \in \mathbb{R}^{N \times d}$4 when the value dimension scales with $Q, K, V \in \mathbb{R}^{N \times d}$5 in MirrorLA’s formulation (Cai et al., 2022).

The practical issue is that many linear-attention kernels require non-negativity so that denominators stay away from zero. Common choices enforce $Q, K, V \in \mathbb{R}^{N \times d}$6 via ReLU, ELU+1, or exponentials. MirrorLA argues that these are axis-aligned “passive truncation” operators, because all negative coordinates are clipped regardless of semantics, which can introduce dead dimensions and reduce representational fidelity (Meng et al., 4 Feb 2026). Its response is not to discard the non-negativity condition, but to learn a reflection before ReLU:

$Q, K, V \in \mathbb{R}^{N \times d}$7

By contrast, LinStereo uses a positive kernel feature map together with an asymmetric positional modification: RoPE is applied to the numerator but not to the denominator, so that

$Q, K, V \in \mathbb{R}^{N \times d}$8

preserving linear complexity while restoring relative positional structure in the numerator (Wang et al., 24 Jun 2026).

CLAReSNet provides an important boundary case. It explicitly contrasts its module with “conventional linear attention” and states that its MSLA preserves exact softmax attention by replacing full $Q, K, V \in \mathbb{R}^{N \times d}$9 self-attention with two cross-attentions between $$\mathbf{o}_t = \sum_{i=1}^N \frac{\exp(\mathbf{q}_t \mathbf{k}_i^\top/\sqrt{d})}{\sum_{j=1}^N \exp(\mathbf{q}_t \mathbf{k}_j^\top/\sqrt{d})} \,\mathbf{v}_i,$$0 tokens and $$\mathbf{o}_t = \sum_{i=1}^N \frac{\exp(\mathbf{q}_t \mathbf{k}_i^\top/\sqrt{d})}{\sum_{j=1}^N \exp(\mathbf{q}_t \mathbf{k}_j^\top/\sqrt{d})} \,\mathbf{v}_i,$$1 learnable latents plus a latent-only self-attention. With $$\mathbf{o}_t = \sum_{i=1}^N \frac{\exp(\mathbf{q}_t \mathbf{k}_i^\top/\sqrt{d})}{\sum_{j=1}^N \exp(\mathbf{q}_t \mathbf{k}_j^\top/\sqrt{d})} \,\mathbf{v}_i,$$2, the dominant term becomes $$\mathbf{o}_t = \sum_{i=1}^N \frac{\exp(\mathbf{q}_t \mathbf{k}_i^\top/\sqrt{d})}{\sum_{j=1}^N \exp(\mathbf{q}_t \mathbf{k}_j^\top/\sqrt{d})} \,\mathbf{v}_i,$$3 rather than $$\mathbf{o}_t = \sum_{i=1}^N \frac{\exp(\mathbf{q}_t \mathbf{k}_i^\top/\sqrt{d})}{\sum_{j=1}^N \exp(\mathbf{q}_t \mathbf{k}_j^\top/\sqrt{d})} \,\mathbf{v}_i,$$4 (Bandyopadhyay et al., 15 Nov 2025). The acronym therefore spans both kernelized linear attention and latent-bottleneck exact-softmax attention in current usage.

3. Multi-scale mechanisms

The “multi-scale” component is the principal source of diversity across MSLA formulations. The following summary captures the mechanisms explicitly described in the cited papers.

Work	Multi-scale axis	Core mechanism
EfficientViT	Spatial neighborhood scale	Two-branch design; one branch aggregates $$\mathbf{o}_t = \sum_{i=1}^N \frac{\exp(\mathbf{q}_t \mathbf{k}_i^\top/\sqrt{d})}{\sum_{j=1}^N \exp(\mathbf{q}_t \mathbf{k}_j^\top/\sqrt{d})} \,\mathbf{v}_i,$$5 nearby tokens before ReLU linear attention
MirrorLA	Local, long-context, global	Block-wise 2D reflections, variance-aware angle modulation, and cross-head reflections
LinStereo	Feature-resolution and disparity scale	HSCV at $$\mathbf{o}_t = \sum_{i=1}^N \frac{\exp(\mathbf{q}_t \mathbf{k}_i^\top/\sqrt{d})}{\sum_{j=1}^N \exp(\mathbf{q}_t \mathbf{k}_j^\top/\sqrt{d})} \,\mathbf{v}_i,$$6 with a 4-level disparity pyramid per scale, refined by PALA
CLAReSNet	Spectral/temporal scale	Downsampled sequences at $$\mathbf{o}_t = \sum_{i=1}^N \frac{\exp(\mathbf{q}_t \mathbf{k}_i^\top/\sqrt{d})}{\sum_{j=1}^N \exp(\mathbf{q}_t \mathbf{k}_j^\top/\sqrt{d})} \,\mathbf{v}_i,$$7 with latent encode-process-decode and fusion

EfficientViT builds multi-scale tokens by aggregating nearby spatial neighbors with small-kernel depthwise-separable convolutions before linear attention. In practice it uses a two-branch design, where one branch is identity or a small kernel and the other aggregates $$\mathbf{o}_t = \sum_{i=1}^N \frac{\exp(\mathbf{q}_t \mathbf{k}_i^\top/\sqrt{d})}{\sum_{j=1}^N \exp(\mathbf{q}_t \mathbf{k}_j^\top/\sqrt{d})} \,\mathbf{v}_i,$$8 neighborhoods. Attention is still global over all tokens at each scale; the scale variation comes from the pre-attention token aggregation and the final concatenation-and-projection fusion (Cai et al., 2022).

MirrorLA makes “multi-scale” a geometric notion. The local scale partitions each per-head feature vector into $$\mathbf{o}_t = \sum_{i=1}^N \frac{\exp(\mathbf{q}_t \mathbf{k}_i^\top/\sqrt{d})}{\sum_{j=1}^N \exp(\mathbf{q}_t \mathbf{k}_j^\top/\sqrt{d})} \,\mathbf{v}_i,$$9 disjoint 2D blocks and assigns each block its own Householder reflection. The long-context scale modulates the reflection angle according to block variance across the sequence length, using

$N \times N$0

to diversify activation patterns in low-variance regimes. The global scale applies a single reflection over the concatenated head space before head-wise decomposition so that covariance is mixed across heads while orthogonality preserves the spectrum (Meng et al., 4 Feb 2026).

LinStereo makes multi-scale structure explicit in the stereo pipeline. It extracts features at $N \times N$1 from Depth Anything V3, projects them to a uniform width $N \times N$2, constructs a correlation volume at each scale, and organizes each volume into a 4-level pyramid by downsampling along the disparity axis. One PALA updater is run per scale, and updates are scheduled in coarse-to-fine order so that coarse estimates guide fine refinements while fine structures inform coarse consistency (Wang et al., 24 Jun 2026).

CLAReSNet uses multi-scale temporal downsampling over hyperspectral sequences. For each scale $N \times N$3, the downsampled sequence $N \times N$4 is encoded into $N \times N$5 latent tokens by cross-attention, processed by latent self-attention and an FFN, then decoded back to sequence space by cross-attention. Outputs from the three scales are concatenated along the feature dimension and fused with an FFN, with the full-resolution sequence serving as the residual anchor (Bandyopadhyay et al., 15 Nov 2025).

4. Representative formulations

EfficientViT. The EfficientViT MSLA block begins from $N \times N$6, projects to $N \times N$7, $N \times N$8, and $N \times N$9 with $O(N^2 d)$0 convolutions, applies per-scale aggregation $O(N^2 d)$1, $O(N^2 d)$2, and $O(N^2 d)$3 using depthwise-separable convolutions, computes linear attention independently for each scale with $O(N^2 d)$4, concatenates the per-scale outputs, and fuses them with a final $O(N^2 d)$5 projection. The block is followed by an FFN augmented with a depthwise convolution. EfficientViT inserts MSLA modules in Stages 3 and 4 of the backbone, where high-resolution context modeling is most useful (Cai et al., 2022).

MirrorLA. MirrorLA reframes the kernel feature map itself. Instead of applying ReLU directly, it learns a reflection vector $O(N^2 d)$6 and constructs a Householder reflection $O(N^2 d)$7, so that the feature map becomes $O(N^2 d)$8. The local mechanism uses per-block 2D reflections with $O(N^2 d)$9; the long-context mechanism perturbs $O(N^2)$0 using block variance across the sequence; and the global mechanism applies a single reflection in $O(N^2)$1 before splitting into heads. The stated pipeline is: global cross-head reflection, reshape into heads and 2D blocks, compute block variances and modulate angles, apply per-block reflections, then apply ReLU and proceed with linear attention contractions (Meng et al., 4 Feb 2026).

LinStereo. LinStereo’s PALA module applies a positive kernel feature map to queries and keys, then uses asymmetric 2D RoPE so that positional encoding affects the numerator but not the denominator. Let $O(N^2)$2 and $O(N^2)$3. One computes $O(N^2)$4 and $O(N^2)$5 once per head, then evaluates each output by global linear aggregation. The hidden state is updated through an adaptive gate,

$O(N^2)$6

so that confident locations integrate more new evidence while preserving accumulated estimates in uncertain regions. PALA is coupled to HSCV and to a monocular-depth-based initialization $O(N^2)$7 estimated from sparse stereo correspondences (Wang et al., 24 Jun 2026).

CLAReSNet. CLAReSNet’s MSLA operates on spectral embeddings of shape $O(N^2)$8 with $O(N^2)$9 and $\phi(\cdot)$0. Each scale executes three phases: encoding, where latent queries attend to the downsampled input sequence; processing, where the latents undergo self-attention and an FFN with GELU and expansion factor 2; and decoding, where the sequence queries attend back to the processed latents. The latent budget is allocated by a logarithmic rule with $\phi(\cdot)$1, $\phi(\cdot)$2, $\phi(\cdot)$3, and $\phi(\cdot)$4, producing $\phi(\cdot)$5–$\phi(\cdot)$6 latent tokens (Bandyopadhyay et al., 15 Nov 2025).

5. Empirical record

EfficientViT reports that, without performance loss on Cityscapes, it provides up to $\phi(\cdot)$7 and $\phi(\cdot)$8 GPU latency reduction over SegFormer and SegNeXt, respectively. For super-resolution it delivers up to $\phi(\cdot)$9x speedup over Restormer while providing $$\operatorname{sim}(\mathbf{q},\mathbf{k}) = \phi(\mathbf{q})\,\phi(\mathbf{k})^\top.$$0dB gain in PSNR, and for Segment Anything it delivers $$\operatorname{sim}(\mathbf{q},\mathbf{k}) = \phi(\mathbf{q})\,\phi(\mathbf{k})^\top.$$1x higher throughput on A100 GPU while achieving slightly better zero-shot instance segmentation performance on COCO. In its fixed-MAC ablation, removing either the multi-scale component or the global-attention component lowers Cityscapes mIoU from $$\operatorname{sim}(\mathbf{q},\mathbf{k}) = \phi(\mathbf{q})\,\phi(\mathbf{k})^\top.$$2 to $$\operatorname{sim}(\mathbf{q},\mathbf{k}) = \phi(\mathbf{q})\,\phi(\mathbf{k})^\top.$$3 or $$\operatorname{sim}(\mathbf{q},\mathbf{k}) = \phi(\mathbf{q})\,\phi(\mathbf{k})^\top.$$4, while removing both yields $$\operatorname{sim}(\mathbf{q},\mathbf{k}) = \phi(\mathbf{q})\,\phi(\mathbf{k})^\top.$$5 (Cai et al., 2022).

MirrorLA reports ImageNet-1K Top-1 results of $$\operatorname{sim}(\mathbf{q},\mathbf{k}) = \phi(\mathbf{q})\,\phi(\mathbf{k})^\top.$$6, $$\operatorname{sim}(\mathbf{q},\mathbf{k}) = \phi(\mathbf{q})\,\phi(\mathbf{k})^\top.$$7, $$\operatorname{sim}(\mathbf{q},\mathbf{k}) = \phi(\mathbf{q})\,\phi(\mathbf{k})^\top.$$8, and $$\operatorname{sim}(\mathbf{q},\mathbf{k}) = \phi(\mathbf{q})\,\phi(\mathbf{k})^\top.$$9 for MirrorLA-T/S/B/L. On COCO it reports RetinaNet $$\mathbf{O} = \frac{\phi(\mathbf{Q})\big(\phi(\mathbf{K})^\top \mathbf{V}\big)}{\phi(\mathbf{Q})\big(\phi(\mathbf{K})^\top \mathbf{1}\big)},$$0 performance of $$\mathbf{O} = \frac{\phi(\mathbf{Q})\big(\phi(\mathbf{K})^\top \mathbf{V}\big)}{\phi(\mathbf{Q})\big(\phi(\mathbf{K})^\top \mathbf{1}\big)},$$1 $$\mathbf{O} = \frac{\phi(\mathbf{Q})\big(\phi(\mathbf{K})^\top \mathbf{V}\big)}{\phi(\mathbf{Q})\big(\phi(\mathbf{K})^\top \mathbf{1}\big)},$$2 for MirrorLA-T and $$\mathbf{O} = \frac{\phi(\mathbf{Q})\big(\phi(\mathbf{K})^\top \mathbf{V}\big)}{\phi(\mathbf{Q})\big(\phi(\mathbf{K})^\top \mathbf{1}\big)},$$3 $$\mathbf{O} = \frac{\phi(\mathbf{Q})\big(\phi(\mathbf{K})^\top \mathbf{V}\big)}{\phi(\mathbf{Q})\big(\phi(\mathbf{K})^\top \mathbf{1}\big)},$$4 for MirrorLA-S, Mask R-CNN $$\mathbf{O} = \frac{\phi(\mathbf{Q})\big(\phi(\mathbf{K})^\top \mathbf{V}\big)}{\phi(\mathbf{Q})\big(\phi(\mathbf{K})^\top \mathbf{1}\big)},$$5 performance of $$\mathbf{O} = \frac{\phi(\mathbf{Q})\big(\phi(\mathbf{K})^\top \mathbf{V}\big)}{\phi(\mathbf{Q})\big(\phi(\mathbf{K})^\top \mathbf{1}\big)},$$6 $$\mathbf{O} = \frac{\phi(\mathbf{Q})\big(\phi(\mathbf{K})^\top \mathbf{V}\big)}{\phi(\mathbf{Q})\big(\phi(\mathbf{K})^\top \mathbf{1}\big)},$$7 for MirrorLA-S and improved $$\mathbf{O} = \frac{\phi(\mathbf{Q})\big(\phi(\mathbf{K})^\top \mathbf{V}\big)}{\phi(\mathbf{Q})\big(\phi(\mathbf{K})^\top \mathbf{1}\big)},$$8 as well, and Mask R-CNN $$\mathbf{O} = \frac{\phi(\mathbf{Q})\big(\phi(\mathbf{K})^\top \mathbf{V}\big)}{\phi(\mathbf{Q})\big(\phi(\mathbf{K})^\top \mathbf{1}\big)},$$9 performance of $S = \sum_j \phi(K_j)V_j^\top$0 $S = \sum_j \phi(K_j)V_j^\top$1 for MirrorLA-S and $S = \sum_j \phi(K_j)V_j^\top$2 $S = \sum_j \phi(K_j)V_j^\top$3 for MirrorLA-T. On ADE20K it reports $S = \sum_j \phi(K_j)V_j^\top$4 and $S = \sum_j \phi(K_j)V_j^\top$5 mIoU for MirrorLA-T and MirrorLA-S; on Cityscapes it reports $S = \sum_j \phi(K_j)V_j^\top$6 and $S = \sum_j \phi(K_j)V_j^\top$7 mIoU. In super-resolution with the DCTLSA backbone, $S = \sum_j \phi(K_j)V_j^\top$8 Urban100 improves to $S = \sum_j \phi(K_j)V_j^\top$9 and Manga109 to $Q, K, V \in \mathbb{R}^{N \times d}$00 in PSNR/SSIM; the paper also states memory reductions up to $Q, K, V \in \mathbb{R}^{N \times d}$01–$Q, K, V \in \mathbb{R}^{N \times d}$02 and latency reductions up to $Q, K, V \in \mathbb{R}^{N \times d}$03 (Meng et al., 4 Feb 2026).

LinStereo reports that PALA’s practical latency per iteration is comparable to a ConvGRU update, specifically $Q, K, V \in \mathbb{R}^{N \times d}$04 ms for PALA versus $Q, K, V \in \mathbb{R}^{N \times d}$05 ms ConvGRU at $Q, K, V \in \mathbb{R}^{N \times d}$06 on RTX 4500, while providing a global receptive field. In its PALA design ablation, a position-agnostic global linear-attention baseline gives KITTI 2015 EPE $Q, K, V \in \mathbb{R}^{N \times d}$07, TartanAir-UW AbsRel $Q, K, V \in \mathbb{R}^{N \times d}$08, and RMSE $Q, K, V \in \mathbb{R}^{N \times d}$09; adding global spatial encoding improves these to $Q, K, V \in \mathbb{R}^{N \times d}$10, $Q, K, V \in \mathbb{R}^{N \times d}$11, and $Q, K, V \in \mathbb{R}^{N \times d}$12; adding local spatial encoding gives $Q, K, V \in \mathbb{R}^{N \times d}$13, $Q, K, V \in \mathbb{R}^{N \times d}$14, and $Q, K, V \in \mathbb{R}^{N \times d}$15; and adding adaptive gating, i.e. full PALA, gives $Q, K, V \in \mathbb{R}^{N \times d}$16, $Q, K, V \in \mathbb{R}^{N \times d}$17, and $Q, K, V \in \mathbb{R}^{N \times d}$18. The paper summary states “28% lower AbsRel on TartanAir-UW, 26% on SQUID,” and also reports $Q, K, V \in \mathbb{R}^{N \times d}$19 lower EPE on occluded Middlebury (H) regions versus the previous best ViT-B baseline (Wang et al., 24 Jun 2026).

CLAReSNet reports overall accuracies of $Q, K, V \in \mathbb{R}^{N \times d}$20 on Indian Pines and $Q, K, V \in \mathbb{R}^{N \times d}$21 on Salinas. For Indian Pines it further reports BA $Q, K, V \in \mathbb{R}^{N \times d}$22, $Q, K, V \in \mathbb{R}^{N \times d}$23 $Q, K, V \in \mathbb{R}^{N \times d}$24, MCC $Q, K, V \in \mathbb{R}^{N \times d}$25, and ARI $Q, K, V \in \mathbb{R}^{N \times d}$26; for Salinas it reports BA $Q, K, V \in \mathbb{R}^{N \times d}$27, $Q, K, V \in \mathbb{R}^{N \times d}$28 $Q, K, V \in \mathbb{R}^{N \times d}$29, and ARI $Q, K, V \in \mathbb{R}^{N \times d}$30. The stated comparison is that these results surpass SSRN, HybridSN, and SpectralFormer in the reported setup, while the module’s complexity scales from $Q, K, V \in \mathbb{R}^{N \times d}$31 to $Q, K, V \in \mathbb{R}^{N \times d}$32 by adaptive latent token allocation (Bandyopadhyay et al., 15 Nov 2025).

Taken together, these results indicate that the empirical value of MSLA-like designs is not confined to one modality. The same general principle appears in semantic segmentation, super-resolution, promptable segmentation, image classification, object detection, stereo matching, diffusion transformers, and hyperspectral image classification, although the precise balance between accuracy, latency, and memory depends on the specific formulation.

6. Limitations, misconceptions, and open directions

A frequent misconception is that “MSLA” refers to a single standardized operator. The cited literature does not support that reading. EfficientViT uses ReLU linear attention plus small-kernel aggregation (Cai et al., 2022); MirrorLA uses Householder reflections before ReLU (Meng et al., 4 Feb 2026); LinStereo uses a positive-kernel linear attention with asymmetric 2D RoPE inside an iterative cost-volume pipeline (Wang et al., 24 Jun 2026); and CLAReSNet uses the same acronym for an exact-softmax latent bottleneck (Bandyopadhyay et al., 15 Nov 2025). This suggests that the label is architectural rather than canonical.

Another misconception is that linear attention is simply a drop-in replacement for softmax attention with no representational trade-off. EfficientViT explicitly notes that ReLU linear attention cannot produce sharp, highly localized attention maps, and compensates with multi-scale convolutional aggregation and depthwise convolution in the FFN (Cai et al., 2022). MirrorLA goes further by identifying non-negativity itself as a source of degradation when implemented by passive truncation, and proposes active reorientation as the remedy (Meng et al., 4 Feb 2026). LinStereo similarly acknowledges that linear attention compresses pairwise relations via $Q, K, V \in \mathbb{R}^{N \times d}$33 and $Q, K, V \in \mathbb{R}^{N \times d}$34, so that extremely repetitive textures could still challenge the normalization even with asymmetric RoPE (Wang et al., 24 Jun 2026).

The reported limitations are modality-specific. MirrorLA requires reliable variance estimates across tokens; extreme low-variance regimes rely on $Q, K, V \in \mathbb{R}^{N \times d}$35 and $Q, K, V \in \mathbb{R}^{N \times d}$36 to inject adequate perturbation, and improper cross-head configuration can underutilize global mixing (Meng et al., 4 Feb 2026). LinStereo notes that the frozen DA3 backbone dominates total parameters and runtime, that DPI relies on sparse handcrafted SIFT matches, and that performance may drop when insufficient inliers exist (Wang et al., 24 Jun 2026). CLAReSNet notes that extremely long sequences remain demanding because memory scales as $Q, K, V \in \mathbb{R}^{N \times d}$37, that too few latents may underfit while too many increase compute and can overfit, and that MSLA does not directly reweight classes under class imbalance (Bandyopadhyay et al., 15 Nov 2025).

The main open direction implied by these works is not the search for a single universal MSLA block, but the refinement of the trade-off between global interaction, scale structure, positional fidelity, and hardware behavior. EfficientViT points toward richer feature maps and hybrid position biases compatible with linear attention (Cai et al., 2022). MirrorLA suggests further exploration of geometric preprocessing inside non-negative kernels (Meng et al., 4 Feb 2026). LinStereo highlights adaptive, content-aware positional kernels and dynamic scale activation (Wang et al., 24 Jun 2026). CLAReSNet points to hierarchical latents, additional coarser scales, and extensions to temporal HSI sequences or multisensor fusion (Bandyopadhyay et al., 15 Nov 2025). Across these variants, MSLA remains a research program aimed at preserving long-range expressive power under the computational constraints of high-resolution or long-context vision systems.