Sparse Mamba Decoder: Efficient State Decoding

• Sparse Mamba Decoder (SMD) is a family of architectures that integrates a Mamba state-space core with domain-specific sparsity to efficiently process structured data.
• It employs mechanisms like local k-NN aggregation, Tanner graph edge selection, and token-pruning to reduce computational load and achieve up to 5×–10× speedup over dense methods.
• Replacing quadratic attention with linear state-space scans, SMD variants realize lower latency, reduced memory usage, and enhanced performance in error correction and classification tasks.

Sparse Mamba Decoder (SMD) denotes a family of decoder architectures that combine Mamba-style state-space modeling with an explicit sparsity mechanism that restricts computation to a structured subset of the full input or state space. In recent arXiv usage, the label appears in several distinct settings: 3D forest point-cloud segmentation, syndrome-based decoding for binary linear codes, defect-centric decoding for surface-code quantum error correction, sparse-token remote-sensing sequence modeling, and control-theoretic sparse state-space variants of Mamba itself (Nguyen et al., 1 Jun 2026, Gusev et al., 11 May 2026, Sayedsalehi et al., 16 May 2026, Alkayid et al., 7 May 2026, Hamdan et al., 2024). Across these settings, the shared design objective is to avoid dense global attention or dense full-volume processing by exploiting geometric neighborhoods, graph edges, active defects, top-$k$ token subsets, or sparse canonical state matrices.

1. Scope and usage of the term

The label “Sparse Mamba Decoder” does not identify a single standardized architecture. Rather, it is used for several related constructions that all retain a Mamba or Mamba-like state-space core while making the decoder sparse in a domain-specific manner. In ForestMamba, sparsity is imposed through local $\kappa$-NN voxel aggregation and vertically ordered query scans (Nguyen et al., 1 Jun 2026). In the Mamba message-passing decoder for LDPC codes, sparsity is defined by the nonzero edge set of the Tanner graph (Gusev et al., 11 May 2026). In the surface-code quantum decoder, sparsity is defect-centric: only the $k$ active detection events are processed (Sayedsalehi et al., 16 May 2026). In GDS-Mamba, sparsity is token-selective, with only the top-$k$ tokens entering the heavy decoder block (Alkayid et al., 7 May 2026). In Sparse-Mamba for sequence modeling, sparsity resides in the state-space parameterization itself, particularly in the $A$ matrix and companion-form $(A,B,C,D)$ structure (Hamdan et al., 2024).

Context	Sparse mechanism	Mamba role
Forest point clouds	Local $\kappa$-NN voxels, height-ordered query scans	Query refinement
LDPC decoding	Tanner-graph edgewise aggregation	Global stream mixing
Surface-code QEC	Active-defect tokens only	Defect-sequence backbone
MODIS time series	Top-$k$ token selection	Sparse branch decoder
Structural SSM	Sparse canonical $A$ matrices	Core sequence dynamics

This distribution of meanings suggests that “sparse” is an overloaded qualifier. It may refer to sparse inputs, sparse interactions, sparse token routing, or sparse system matrices, depending on the application.

2. Common architectural pattern

Despite domain differences, SMD variants share a recurring computational template. First, a sparse subset of relevant entities is identified or enforced. Second, local evidence is aggregated into token or node states. Third, a Mamba or bidirectional Mamba block performs long-range mixing with linear-time state-space scans rather than dense quadratic self-attention. Fourth, residual normalization and feed-forward updates produce refined decoder states. This pattern is explicit in the forest, coding, quantum, and remote-sensing formulations (Nguyen et al., 1 Jun 2026, Gusev et al., 11 May 2026, Sayedsalehi et al., 16 May 2026, Alkayid et al., 7 May 2026).

At the state-space level, the common recurrence is

$x_{t+1}=A x_t + B u_t,\qquad y_t = C x_t + D u_t,$

with Mamba variants modifying how $\kappa$0 are parameterized or scanned (Hamdan et al., 2024). What changes across SMD instantiations is not the use of state-space dynamics as such, but the mechanism by which relevant tokens, nodes, queries, or events are selected before or during the scan.

A second commonality is hybridization. ForestMamba retains a scaled dot-product weighting step over local neighborhoods before the dual-path Mamba scan (Nguyen et al., 1 Jun 2026). The LDPC decoder uses pairwise edge scoring and gated residual node updates before bidirectional Mamba refinement (Gusev et al., 11 May 2026). GDS-Mamba computes token-importance scores before sparse processing (Alkayid et al., 7 May 2026). Consequently, SMD should not be understood as a purely homogeneous SSM block in every usage; several realizations are hybrid sparse-local plus state-space-global decoders.

3. Geometry-aware and token-sparse decoders in remote sensing

In ForestMamba, the Sparse Mamba Decoder sits at the end of the network and refines $\kappa$1 instance query embeddings using the full voxel feature set produced by a sparse U-Net encoder (Nguyen et al., 1 Jun 2026). Each decoder layer performs three operations: local $\kappa$2-NN aggregation of nearby voxel features, global interaction among queries through a spatial dual-path Mamba scan, and a position-wise feed-forward network. The query set is initialized by a geometry-guided CHM+FPS module, and each query carries a 3D anchor position $\kappa$3. The local aggregation stage computes scaled dot-product weights over the $\kappa$4 nearest voxels to $\kappa$5, then forms a gated aggregation

$\kappa$6

followed by residual normalization. Global interaction is then modeled by two Mamba scans, one ordered bottom$\kappa$7top and one top$\kappa$8bottom, whose outputs are averaged and added back residually. The resulting decoder is linear in the number of queries, with overall complexity across $\kappa$9 layers reported as $k$0. On approximately $k$1M voxels with $k$2 queries, the decoder alone runs in approximately $k$3 ms versus $k$4 ms for a standard Transformer decoder and uses $k$5 GB peak GPU memory versus $k$6 GB.

The same paper ties sparsity to forest structural priors rather than to generic efficiency alone. Geometry-guided seeds concentrate queries on canopy maxima and understory regions, local $k$7-NN aggregation restricts evidence to nearby voxels, and the dual-path scan orders queries by height. This yields what the paper describes as ecologically informed context modeling and tree-centric grouping (Nguyen et al., 1 Jun 2026). A plausible implication is that the decoder’s sparsity pattern is not merely computationally economical but also an inductive bias aligned with forest geometry.

GDS-Mamba uses a different sparse-decoder design for MODIS time-series classification (Alkayid et al., 7 May 2026). Here the SMD is embedded in spectral, temporal, and spatial branches. The decoder first computes token-importance scores $k$8, selects the top $k$9 indices, gathers those tokens via a binary mask $k$0, applies a Mamba block to the reduced sequence $k$1, and then scatters the processed result back:

$k$2

In the spectral and temporal branches, $k$3 is the average incoming attention over all heads; in the spatial branch, it is based on cosine similarity to the central token. The sparse decoder therefore preserves the original sequence length while ensuring that only $k$4 tokens traverse the expensive block. The paper reports that when $k$5 with $k$6, the sparse attention cost is approximately $k$7 cheaper than full attention, and that the empirical choice of $k$8 typically yields $k$9–$A$0 overall speedup in the decoder.

4. Sparse Mamba decoders for classical and quantum error correction

For binary linear codes, the Mamba message-passing decoder (MMPD) is an SMD whose sparsity is graph-structural rather than token-pruned (Gusev et al., 11 May 2026). The decoder maintains a variable-node stream

$A$1

and a check-node stream

$A$2

At each iteration, messages are computed only along Tanner-graph edges. After linear projections, each edge $A$3 forms

$A$4

which is scored by a small MLP and normalized over neighbors to obtain $A$5. Aggregated messages then feed a gated residual node update

$A$6

after which each stream is refined by a bidirectional Mamba state-space block. The local steps scale with $A$7, the number of nonzero Tanner-graph edges, and the BiMamba global mixing scales linearly in sequence length rather than quadratically. On the $A$8 WiMAX LDPC code, the model reports $A$9M parameters, $(A,B,C,D)$0 GB train memory, and $(A,B,C,D)$1 GB inference memory, compared with $(A,B,C,D)$2 GB and $(A,B,C,D)$3 GB for CrossMPT; at a target BER of $(A,B,C,D)$4, it shows roughly a $(A,B,C,D)$5 dB gain over CrossMPT.

In surface-code quantum error correction, the Sparse Mamba Decoder is defect-centric (Sayedsalehi et al., 16 May 2026). Instead of processing the full syndrome volume of size $(A,B,C,D)$6, it extracts only the $(A,B,C,D)$7 active detection events and encodes each defect with a 13-dimensional feature vector

$(A,B,C,D)$8

These defect tokens are embedded and passed through $(A,B,C,D)$9 Mamba “Mixer” layers, each composed of a selective-scan state-space block and a gated dense feedforward sublayer with RMSNorm and residual connections. After masked mean pooling, a readout head produces a logit trained by standard cross-entropy. The central computational claim is that after one-time sparse extraction, neural processing is $\kappa$0 rather than $\kappa$1. The paper reports that on SI1000 noise the decoder reduces the MWPM logical error rate by up to $\kappa$2 at $\kappa$3, runs $\kappa$4–$\kappa$5 faster than Tesseract and $\kappa$6–$\kappa$7 faster than Belief Matching, and maintains nearly constant latency of $\kappa$8–$\kappa$9 $k$0s across $k$1–$k$2 under uniform circuit-level noise. On the Sycamore experimental dataset, the SMD ensemble matches or slightly surpasses the dense Mamba decoder of Varbanov et al.

Taken together, these two error-correction decoders show that SMD can mean either sparse message passing on a known graph or sparse event selection from a dense spacetime volume. The commonality is that state-space mixing is reserved for information that has already been structurally compressed.

5. Sparse state-space parameterization and control-theoretic formulations

A more foundational use of the term appears in “Sparse Mamba: Introducing Controllability, Observability, And Stability To Structural State Space Models” (Hamdan et al., 2024). Here the emphasis is not on sparse inputs or sparse token routing, but on sparse state-space parameterization. S-Mamba replaces dense or diagonal $k$3 matrices with sparse canonical forms: SC-Mamba uses controllable canonical form, SO-Mamba uses observable canonical form, and ST-Mamba2 enforces a stable diagonal $k$4. In the controllable and observable forms, the companion-structured $k$5 matrix has exactly $k$6 nonzeros and only $k$7 free parameters, while $k$8 and $k$9 take structured vector forms and $A$0 is scalar.

The control-theoretic significance is explicit. For controllability, the reachability matrix

$A$1

has rank $A$2 by construction in companion form. For observability, the observability matrix

$A$3

likewise has full rank in the observable form. Stability is enforced in ST-Mamba2 by constraining the diagonal entries of $A$4 to remain negative. The paper states that these constructions guarantee controllability, observability, or stability without auxiliary penalties.

The reported numerical effects are modest but concrete. Under a 1B-parameter setting, total parameter counts are $A$5 for Mamba, $A$6 for SO-Mamba, and $A$7 for SC-Mamba, saving approximately $A$8–$A$9k parameters. On CodeParrot, perplexity improves from $x_{t+1}=A x_t + B u_t,\qquad y_t = C x_t + D u_t,$0 for Mamba to $x_{t+1}=A x_t + B u_t,\qquad y_t = C x_t + D u_t,$1 for SO-Mamba and $x_{t+1}=A x_t + B u_t,\qquad y_t = C x_t + D u_t,$2 for SC-Mamba, while fill-in-middle training time drops from $x_{t+1}=A x_t + B u_t,\qquad y_t = C x_t + D u_t,$3 h $x_{t+1}=A x_t + B u_t,\qquad y_t = C x_t + D u_t,$4 m to $x_{t+1}=A x_t + B u_t,\qquad y_t = C x_t + D u_t,$5 h $x_{t+1}=A x_t + B u_t,\qquad y_t = C x_t + D u_t,$6 m and $x_{t+1}=A x_t + B u_t,\qquad y_t = C x_t + D u_t,$7 h $x_{t+1}=A x_t + B u_t,\qquad y_t = C x_t + D u_t,$8 m. For Mamba2, ST-Mamba2 reduces perplexity from $x_{t+1}=A x_t + B u_t,\qquad y_t = C x_t + D u_t,$9 to $\kappa$00. This is not a decoder in the same applied sense as the other SMDs, but it is structurally relevant because it shows that sparsity can be introduced inside the state transition itself rather than only in the data pathway.

6. Complexity regimes, empirical gains, and recurrent misconceptions

Across the literature, SMD architectures are primarily motivated by scaling laws. ForestMamba reports per-layer cost $\kappa$01, contrasted with Transformer costs such as $\kappa$02 cross-attention or $\kappa$03 self-attention (Nguyen et al., 1 Jun 2026). MMPD replaces $\kappa$04 attention storage with $\kappa$05 edge scores plus $\kappa$06 node states (Gusev et al., 11 May 2026). The surface-code decoder reduces neural cost from $\kappa$07 to $\kappa$08 after sparse extraction (Sayedsalehi et al., 16 May 2026). GDS-Mamba processes only $\kappa$09 tokens inside its heavy block and reports $\kappa$10–$\kappa$11 decoder speedup in practice (Alkayid et al., 7 May 2026). Sparse-Mamba reduces the state update itself from dense $\kappa$12 behavior to $\kappa$13 through structured matrices (Hamdan et al., 2024).

A recurrent misconception is that replacing attention with Mamba implies the complete disappearance of attention-like mechanisms. Several SMDs still use attention-style scoring locally: ForestMamba uses scaled dot-product weights over $\kappa$14-NN voxels, MMPD uses softmax-normalized edge scores, and GDS-Mamba uses attention-derived token-importance scores (Nguyen et al., 1 Jun 2026, Gusev et al., 11 May 2026, Alkayid et al., 7 May 2026). What is removed is dense global attention, not necessarily all weighting or selection based on pairwise similarity.

A second misconception is that sparsity has a uniform meaning across these models. The papers collectively show four distinct sparsity regimes: structured local neighborhoods, sparse graph edges, active-event extraction, and sparse state matrices. This suggests that SMD is better understood as a design family organized around constrained state-space decoding rather than as a single algorithm.

The empirical record reported in these papers is consistently tied to that family resemblance. ForestMamba reports decoder-level speed and memory gains over a Transformer decoder and overall $\kappa$15 times faster inference with $\kappa$16 times lower GPU memory than Transformer-based methods across seven forest regions (Nguyen et al., 1 Jun 2026). MMPD reports improved BER-memory tradeoffs on long LDPC codes (Gusev et al., 11 May 2026). The quantum SMD reports substantial latency gains and competitive or superior logical error rates on several noise models and on experimental Sycamore data (Sayedsalehi et al., 16 May 2026). GDS-Mamba reports high classification accuracy together with sparse-decoder acceleration in large-scale MODIS classification (Alkayid et al., 7 May 2026). Sparse-Mamba reports parameter, perplexity, and training-time improvements from control-theoretic structuring (Hamdan et al., 2024). The broad implication is that SMD has become a reusable pattern for building decoders in regimes where dense global processing is unnecessary, unaffordable, or poorly aligned with the underlying structure.