Linear Attention Mamba (LAM) Overview

• Linear Attention Mamba (LAM) is a research direction that unifies linear attention with Mamba-inspired selective state-space models for efficient sequence processing.
• It embodies reformulation, hybridization, and modification patterns to balance computational efficiency with sophisticated token interactions across various domains.
• LAM leverages techniques like gating, decay, and topology-aware scanning to maintain linear-time, constant-memory performance while addressing traditional attention limitations.

Linear Attention Mamba (LAM) denotes, in current usage, not a single standardized architecture but a research direction at the interface of linear attention and Mamba-style selective state-space modeling. The literature uses the label in several related ways: as a reformulation of Mamba as a variant of linear attention, as a family of linear-attention blocks made “Mamba-like” through gating or selective mechanisms, and as hybrid models that combine Mamba with local, latent, or cross-modal attention. At the same time, multiple sources explicitly caution that Mamba is not linear attention in the strict query–key–value kernel sense, even when both serve as linear-complexity alternatives to quadratic self-attention (Han et al., 2024).

1. Conceptual scope and nomenclature

A central feature of the literature is that “Linear Attention Mamba” is better understood as an umbrella concept than as a single canonical model. One source states explicitly that a queried survey fragment does not present “Linear Attention Mamba (LAM)” as a named method, and that Mamba in that text is more defensibly interpreted as a state-space sequence model that can replace attention-style token mixing without thereby becoming linear attention in the technical kernelized sense (Ibrahim et al., 11 Feb 2025). This makes the term encyclopedically useful, but taxonomically loose.

Within that umbrella, three usages recur. First, some papers argue that Mamba can be rewritten in a linear-attention form and then ask which design differences matter. The clearest example is "Demystify Mamba in Vision: A Linear Attention Perspective" (Han et al., 2024). Second, some papers begin with linear attention and import Mamba-derived mechanisms such as decay masks, selective gating, or block design; "2Mamba2Furious: Linear in Complexity, Competitive in Accuracy" (Mongaras et al., 19 Feb 2026) is the most explicit case. Third, some models combine Mamba and attention as complementary submodules rather than identifying one with the other; representative examples include "LaMamba-Diff: Linear-Time High-Fidelity Diffusion Models Based on Local Attention and Mamba" (Fu et al., 2024), "Why mamba is effective? Exploit Linear Transformer-Mamba Network for Multi-Modality Image Fusion" (Zhu et al., 2024), and "PointLAMA: Latent Attention meets Mamba for Efficient Point Cloud Pretraining" (Lin et al., 23 Jul 2025).

This suggests a practical definition: LAM refers to architectures or analyses that attempt to retain Mamba’s linear-time, constant-memory, or scan-based benefits while importing some of the explicit token-interaction behavior associated with attention, or conversely to linear-attention systems that import Mamba’s selective or gated inductive biases.

2. Mathematical relationship between linear attention and selective state spaces

The most influential formal bridge is the unified formulation in "Demystify Mamba in Vision: A Linear Attention Perspective" (Han et al., 2024). There, causal linear attention is written in recurrent form as

$$S_i=S_{i-1}+k_i^{\top}v_i,\qquad Z_i=Z_{i-1}+k_i^{\top},\qquad y_i=q_iS_i/q_iZ_i,$$

while the selective SSM used for Mamba is written as

$$h_i=\widetilde{A}_i\odot h_{i-1}+B_i(\mathbf{\Delta}_i\odot x_i),\qquad y_i=C_i h_i+\mathbf{D}\odot x_i.$$

The paper then aligns the terms by treating $h_i$ as analogous to the accumulated linear-attention state $S_i$, $B_i$ as key-like, $C_i$ as query-like, and $x_i$ as value-like. In that reading, Mamba is a variant of linear attention with six major distinctions: input gate, forget gate, shortcut, no attention normalization, single-head design, and modified block design (Han et al., 2024).

A more language-oriented analysis sharpens the same point. "Exploring the Limitations of Mamba in COPY and CoT Reasoning" expands a simplified selective SSM into

$y_i = \sum_{j=1}^{i}\alpha_j (q_i^T q_j) v_j,$

where $\alpha_j$ is a monotone forgetting factor (Ren et al., 2024). This makes Mamba resemble unnormalized causal linear attention with decay: the sequence history is compressed into a finite recurrent state, and older contributions are multiplicatively attenuated. In this view, the difference from canonical linear attention is not merely implementation but memory semantics: Mamba is a decaying dynamical system rather than a normalized key–value accumulator.

Other papers adopt the same perspective more aggressively. "Differential Mamba" treats the S6 operator, and eventually the whole Mamba block, as an implicit lower-triangular attention-like matrix and then defines subtraction-based denoising at the operator level, replacing one implicit mixer $A$ with $$h_i=\widetilde{A}_i\odot h_{i-1}+B_i(\mathbf{\Delta}_i\odot x_i),\qquad y_i=C_i h_i+\mathbf{D}\odot x_i.$$0 (Schneider et al., 8 Jul 2025). By contrast, "VL-Mamba: Exploring State Space Models for Multimodal Learning" stresses that, despite similar linear-scaling motivation, Mamba is not linear attention in the standard kernelized-attention sense because there are no explicit $$h_i=\widetilde{A}_i\odot h_{i-1}+B_i(\mathbf{\Delta}_i\odot x_i),\qquad y_i=C_i h_i+\mathbf{D}\odot x_i.$$1 variables and no softmax or feature-map attention matrix approximation (Qiao et al., 2024).

The mathematical literature therefore supports two statements simultaneously. Mamba can often be rewritten in an attention-like form, which explains why LAM is a coherent research theme. But that rewrite does not erase the distinction between state propagation and query–key matching.

3. Main architectural patterns

The first major pattern is reformulation: Mamba is reinterpreted as linear attention plus a small number of decisive modifications. "Demystify Mamba in Vision" concludes that, in vision, the forget gate and block design are the main contributors to Mamba’s success, while input gate and shortcut are marginal, and dropping normalization or multi-head structure is actively harmful (Han et al., 2024). On that basis it proposes MILA, a Mamba-inspired linear attention model that preserves normalized multi-head linear attention, adds Mamba-style block design, and replaces the recurrent forget-gate function with positional encodings. This is arguably the canonical “linear attention first, Mamba second” formulation.

A second pattern reverses the direction: linear attention is strengthened by importing Mamba-derived masking or recurrence ideas. "2Mamba2Furious" explicitly treats Mamba-2 as a strong linear-attention-like architecture and simplifies it to a core consisting of short convolution, an $$h_i=\widetilde{A}_i\odot h_{i-1}+B_i(\mathbf{\Delta}_i\odot x_i),\qquad y_i=C_i h_i+\mathbf{D}\odot x_i.$$2-mask, and normalization, then introduces a second-order hidden state by replacing $$h_i=\widetilde{A}_i\odot h_{i-1}+B_i(\mathbf{\Delta}_i\odot x_i),\qquad y_i=C_i h_i+\mathbf{D}\odot x_i.$$3 with $$h_i=\widetilde{A}_i\odot h_{i-1}+B_i(\mathbf{\Delta}_i\odot x_i),\qquad y_i=C_i h_i+\mathbf{D}\odot x_i.$$4 (Mongaras et al., 19 Feb 2026). The resulting 2Mamba is presented as nearly as accurate as softmax attention while remaining more memory efficient for long contexts. "FMNet: Frequency-Assisted Mamba-Like Linear Attention Network for Camouflaged Object Detection" likewise builds on MLLA, uses the linear-attention recurrence

$$h_i=\widetilde{A}_i\odot h_{i-1}+B_i(\mathbf{\Delta}_i\odot x_i),\qquad y_i=C_i h_i+\mathbf{D}\odot x_i.$$5

and pairs it with a Mamba-like recurrence

$$h_i=\widetilde{A}_i\odot h_{i-1}+B_i(\mathbf{\Delta}_i\odot x_i),\qquad y_i=C_i h_i+\mathbf{D}\odot x_i.$$6

to justify a frequency-enhanced vision block (Deng et al., 14 Mar 2025).

A third pattern is hybridization rather than reformulation. "LaMamba-Diff" uses a visual state-space branch for global context and a local window-attention branch for explicit local token interaction inside each block (Fu et al., 2024). "Tmamba" runs a linear Transformer branch and a Mamba branch in parallel, then exchanges channel and positional information between them through asymmetric interaction modules (Zhu et al., 2024). "PointLAMA" inserts a single lightweight latent attention block into an otherwise Mamba-dominant point-cloud encoder, with the reported configuration using 11 Mamba blocks and 1 Latent Attention block (Lin et al., 23 Jul 2025). "MLLA-UNet" places Mamba-like linear attention blocks inside a U-shaped medical segmentation architecture, but explicitly describes them as linear attention with Mamba-inspired adaptive behavior rather than as true selective SSMs (Jiang et al., 2024).

A fourth pattern consists of Mamba-side modifications derived from the implicit-attention view. "Differential Mamba" subtracts two Mamba operators to reduce overallocation to irrelevant context (Schneider et al., 8 Jul 2025). "Mamba-3: Improved Sequence Modeling using State Space Principles" does not present itself as LAM, but it is important background because it extends the state-space side of the bridge with a richer three-term recurrence, complex-valued rotational state dynamics, and MIMO state updates while remaining within the same broader family of linear recurrent alternatives to attention (Lahoti et al., 16 Mar 2026).

4. Domain-specific instantiations

Vision is the dominant application domain. MILA is positioned as a scalable linear-attention backbone that outperforms various vision Mamba models in image classification and high-resolution dense prediction while preserving parallelizable computation and fast inference (Han et al., 2024). FMNet specializes the same Mamba-like linear-attention perspective for camouflaged object detection through multi-scale branches and frequency-domain enhancement (Deng et al., 14 Mar 2025). MLLA-UNet adapts MLLA to medical image segmentation in a U-shaped network and reports an average DSC of 88.32% across six datasets and 24 segmentation tasks (Jiang et al., 2024).

Generative modeling has produced a different family of LAM-style systems. LaMamba-Diff argues that pure Mamba loses fine local dependencies important for image synthesis, and therefore combines Mamba-like visual state-space modeling with fixed-size local attention in a latent-diffusion U-Net (Fu et al., 2024). Its abstract claims a reduction of up to 62% GFLOPs compared to DiT-XL/2 while achieving superior performance with comparable or fewer parameters. Here, LAM is not kernelized linear attention but a local-attention-plus-Mamba decomposition in which global context is assigned to the state-space branch and local detail to attention.

Multimodal work uses both replacement and hybrid strategies. VL-Mamba replaces the Transformer language backbone in a small multimodal LLM with a pretrained Mamba LLM and adds a Vision Selective Scan connector; its own description is careful that this is an attention-free linear-scaling alternative, not linear attention proper (Qiao et al., 2024). TransMamba focuses on transferring knowledge from pretrained Transformer systems into Mamba students through aligned latent spaces, weight subcloning, adaptive bidirectional distillation, and a cross-Mamba module for multimodal interaction (Chen et al., 21 Feb 2025). The relevance to LAM lies less in new mixers than in showing how Mamba can be operationalized as an attention replacement without full retraining.

Three-dimensional and structured-signal domains make the topology issue explicit. PointLAMA combines task-aware point serialization, sparse latent attention insertion, and a Mamba backbone to repair the local-geometry weakness of pure Mamba on point clouds (Lin et al., 23 Jul 2025). Omni-directional attention for speech separation takes a related route on spectrograms: instead of explicit token attention, it serializes the 2D time–frequency plane along ten directions and processes those scans with unidirectional Mamba while preserving linear complexity (Xue et al., 23 Jan 2026). This is adjacent to LAM because it addresses the same efficiency-versus-global-context problem, but does so through topology-aware scanning rather than QKV attention.

5. Efficiency, scaling, and systems considerations

The shared motivation of LAM research is the quadratic cost of standard self-attention and the desire for linear sequence scaling. The original "Mamba: Linear-Time Sequence Modeling with Selective State Spaces" identifies the inability to perform content-based reasoning as the key weakness of earlier efficient models, introduces input-dependent selective state spaces, and then recovers efficiency through a hardware-aware parallel scan algorithm; the paper reports fast inference, 5$$h_i=\widetilde{A}_i\odot h_{i-1}+B_i(\mathbf{\Delta}_i\odot x_i),\qquad y_i=C_i h_i+\mathbf{D}\odot x_i.$$7 higher throughput than Transformers, and linear scaling in sequence length (Gu et al., 2023).

Yet linearity is obtained by different mechanisms in different LAM subfamilies. In kernelized linear-attention systems such as MLLA or FMNet, linear complexity comes from associative factorization of the attention computation; FMNet states standard attention complexity as $$h_i=\widetilde{A}_i\odot h_{i-1}+B_i(\mathbf{\Delta}_i\odot x_i),\qquad y_i=C_i h_i+\mathbf{D}\odot x_i.$$8 and the MLLA-style linear attention module as $$h_i=\widetilde{A}_i\odot h_{i-1}+B_i(\mathbf{\Delta}_i\odot x_i),\qquad y_i=C_i h_i+\mathbf{D}\odot x_i.$$9 (Deng et al., 14 Mar 2025). In LaMamba-Diff, the attention branch is not kernelized at all: overall linearity comes from restricting softmax attention to fixed-size local windows while leaving long-range modeling to Mamba, so the per-token attention neighborhood is constant (Fu et al., 2024). In scan-based systems such as OA for speech, linearity comes from directional state-space passes without any token–token attention matrix; that paper contrasts the self-attention memory cost $h_i$0 with OA complexity $h_i$1 (Xue et al., 23 Jan 2026).

The systems question is increasingly central. "Mamba-3" argues that many theoretically linear recurrent models remain hardware-inefficient at decode time because of low arithmetic intensity, and responds with a more expressive recurrence, complex-valued state updates, and a MIMO formulation intended to improve quality without increasing decode latency materially (Lahoti et al., 16 Mar 2026). "2Mamba2Furious" makes the complementary point that improved expressivity can increase constant factors even when asymptotics remain favorable: its second-order hidden state improves accuracy, but the hidden-state memory is only more efficient than softmax attention beyond a derived crossover context length that is approximately 1058 for head dimension 64 (Mongaras et al., 19 Feb 2026).

A general systems conclusion follows. LAM is not a single complexity story but a design space in which linearity may come from scan algorithms, associative attention, fixed local windows, latent bottlenecks, or sparse insertion of attention blocks. The practical competitiveness of any given design depends as much on constants, kernels, and memory traffic as on asymptotic notation.

6. Limitations, misconceptions, and open problems

A persistent misconception is that Mamba is simply linear attention under another name. The literature does not support that simplification. VL-Mamba explicitly states that its mechanism is not linear attention in the standard kernelized-attention sense, because there are no $h_i$2 variables and no softmax approximation; dependence across positions is mediated through hidden-state propagation (Qiao et al., 2024). The survey fragment discussed under the title "A Survey on Mamba Architecture for Vision Applications" reaches the same cautionary conclusion for the material it actually contains (Ibrahim et al., 11 Feb 2025).

A second limitation is the fixed-state bottleneck. "Exploring the Limitations of Mamba in COPY and CoT Reasoning" shows that constant-sized Mamba can struggle on COPY tasks and arbitrary dynamic-programming-style chain-of-thought reasoning, and that exact COPY becomes possible only when state size grows linearly with sequence length, erasing the usual efficiency advantage (Ren et al., 2024). This critique is complementary to "Revealing and Mitigating the Local Pattern Shortcuts of Mamba," which argues that vanilla Mamba tends to rely on local pattern shortcuts and degrades on distributed-information tasks; its global-selection modification, implemented with only 4M extra parameters, raises a 130M Mamba model from failure-level performance to 80.54 on the hardest distributed-information setting reported in the paper (You et al., 2024).

The attention side has its own pathology. "SEMA: a Scalable and Efficient Mamba like Attention via Token Localization and Averaging" proves a dispersion result for generalized normalized attention, including linear attention, and argues that as the number of keys grows, normalized weights become increasingly uniform unless selectivity is localized (Tran et al., 10 Jun 2025). SEMA responds by combining token localization with arithmetic averaging. This critique matters for LAM because it implies that merely replacing softmax with a linear kernel does not automatically preserve focus.

The most constructive controversy concerns which Mamba ingredients are truly essential. "Demystify Mamba in Vision" finds that the forget gate and block design matter most in vision, while input gate and shortcut are marginal, and no normalization and single-head structure are actively detrimental (Han et al., 2024). This has direct architectural implications: some of what looks “Mamba-like” can be removed, replaced, or reinterpreted without loss, whereas other elements are the actual source of the gain.

Taken together, these results suggest several durable research directions. One is to improve global routing into recurrent memory without reverting to quadratic attention, as in global selection or topology-aware scanning. Another is to strengthen compressed state representations through higher-order interactions, differential denoising, or richer recurrences. A third is to exploit the pretrained Transformer ecosystem more effectively when building Mamba or LAM systems, as in TransMamba’s cross-architecture adaptation pipeline (Chen et al., 21 Feb 2025). The field is therefore converging less toward a single LAM recipe than toward a set of recurring design principles for reconciling selective recurrence with explicit or implicit attention.