---
title: Multi-Head Selective SSM (MHSSM)
url: https://www.emergentmind.com/topics/multi-head-selective-state-space-model-mhssm
type: topic
---

# Multi-Head Selective SSM (MHSSM)

Searching arXiv for recent MHSSM-related papers to ground the article with up-to-date references.
arxiv_search(query="Multi-Head Selective State-Space Model OR MHSSM state space model multi-head Mamba", max_results=10, sort_by="relevance")
A Multi-Head Selective State-Space Model (MHSSM) is a family of architectures that extends selective state-space modeling by introducing multiple parallel heads, experts, or scan subspaces, each with its own selective dynamics, scan order, or parameter generator, and then fusing the resulting outputs. Across recent literature, the term covers several closely related constructions: directional experts that scan the same input in different orders, channel-partitioned selective scans over parallel subspaces, dual-branch or multi-branch selective projectors, and mixture-of-experts routing inside the state-space kernel itself. The unifying purpose is to increase directional diversity, operator span, or task-specific specialization while retaining the linear-time scan behavior associated with Mamba-like selective SSMs [2410.07528] [2506.22246] [2507.19778] [2510.26182].

## 1. Conceptual scope and nomenclature

The “multi-head” component of MHSSM is not defined by a single canonical implementation. In current practice, a head may be a direction-specific state-space branch, a channel-group subspace with its own selective scan, a frozen or plastic projector branch, or a routed state-space expert. CountMamba describes its Multi-directional State-Space Group as an MHSSM with \(H=4\) heads—Horizontal, Vertical, Diagonal, and Anti-diagonal—where each directional expert scans the same image under a different ordering [2410.07528]. EAMamba instead realizes MHSSM by splitting expanded channels into multiple groups and assigning distinct scan directions to different heads without increasing FLOPs or parameter count with the number of sequences [2506.22246]. HydraMamba defines MHS6 by partitioning channels into \(H\) heads, each running an input-dependent S6 with head-specific parameters and output fusion by concatenation and projection [2507.19778].

This broader usage also explains why the term should not be applied indiscriminately. PointABM integrates a bidirectional selective state-space model with Transformer multi-head self-attention, but its own SSM block is bidirectional rather than multi-head; the paper explicitly states that it “is not a Multi-Head Selective State-Space Model (MHSSM) in the strict sense” [2406.06069]. By contrast, MHS-VM, CountMamba, EAMamba, HydraMamba, and MossNet all instantiate multi-head structure inside or directly around the selective state-space mechanism itself [2406.05992] [2410.07528] [2506.22246] [2507.19778] [2510.26182].

A useful historical precursor is the earlier Multi-Head State Space Model for speech recognition, which introduced parallel state-space heads and inter-head gating, but not Mamba-style selectivity in the later sense. It is therefore best understood as antecedent rather than definitive MHSSM terminology [2305.12498].

## 2. Selective state-space foundations

Most MHSSM formulations inherit the standard continuous-time and discretized SSM framework. A canonical continuous-time system is
$$
\dot{h}(t)=A\,h(t)+B\,x(t),\qquad y(t)=C\,h(t)+D\,x(t),
$$
and Mamba-style models discretize this system with an input-dependent step size \(\Delta_t\), yielding discrete parameters such as
$$
\bar{A}_t = e^{\Delta_t A},\qquad \bar{B}_t = \Phi(\Delta_t,A)B.
$$
The resulting selective recurrence takes forms such as
$$
h_t=\bar{A}_t h_{t-1}+\bar{B}_t u_t,\qquad y_t=C_t h_t,
$$
or, in CountMamba’s Mamba-style formulation,
$$
h_k=g_k\odot(\bar{A}_k h_{k-1}+\bar{B}_k u_k)+(1-g_k)\odot h_{k-1},\qquad y_k=C_k h_k,
$$
with \(\Delta_k\), \(g_k\), \(B_k\), and \(C_k\) all input-dependent [2410.07528] [2512.19676].

The multi-head extension introduces independent per-head dynamics. HydraMamba states this explicitly:
$$
x_t^{(h)}=\bar{A}_t^{(h)}x_{t-1}^{(h)}+\bar{B}_t^{(h)}u_t^{(h)},\qquad
y_t^{(h)}=C_t^{(h)}x_t^{(h)},
$$
with
$$
y_t=W_o\,[y_t^{(1)}\Vert \cdots \Vert y_t^{(H)}].
$$
Its selectivity is implemented through head-specific \(B_t^{(h)}\), \(C_t^{(h)}\), and \(\Delta_t^{(h)}\), where \(\Delta_t^{(h)}\) is produced by a softplus-gated linear map [2507.19778]. The MRI super-resolution framework similarly uses \(m\) parallel heads, each with its own diagonal-stable \(A^{(i)}\), adaptive \(\Delta_t^{(i)}\), and merged output projection [2512.19676].

A broader operator-theoretic view is supplied by the unified framework of sequence modeling. There, a sequence map is written as
$$
y_i=\sum_{j=1}^{n} W_{ij}(X)x_j,
$$
where \(W_{ij}(X)\) is the input-dependent effective interaction operator. In the structured-dynamics case, selective SSMs induce operators of the form
$$
W_{ij}=C_i(X)\Big(\prod_{k=j+1}^{i}\bar{A}_k(X)\Big)\bar{B}_j(X),
$$
which makes explicit how selective dynamics define context-dependent lag interactions [2512.15115].

## 3. Head construction and fusion mechanisms

Recent MHSSM implementations differ mainly in how heads are defined and how their outputs are fused.

| System | Head construction | Fusion |
|---|---|---|
| CountMamba | Four directional experts: HSSM, VSSM, DSSM, ASSM | Softmax-weighted Global-Local Adaptive Fusion |
| EAMamba | Channel-group heads with distinct scan directions | Concat after inverse transforms, then gated projection |
| HydraMamba | Per-head S6 dynamics over channel partitions | Channel concatenation and linear projection |
| MHS-VM | Parallel subspaces, each with multiple scan routes | ESF and optional tail projection |
| Mamba-FSCIL | Frozen base branch and trainable incremental branch | Additive aggregation with identity branch |
| MossNet | MoE-routed time-mixing experts | Top-\(k\) weighted expert aggregation |

CountMamba’s Multi-directional State-Space Group uses four independent counting experts and finds that “Four experts” perform best, while Global-Local Adaptive Fusion computes sample-wise softmax weights \([\alpha_H,\alpha_V,\alpha_D,\alpha_A]\) and combines them with a local CNN branch [2410.07528]. EAMamba’s MHSSM instead groups channels into heads and applies head-specific transforms before a selective scan; its key claim is that this grouped design keeps FLOPs and parameters stable with respect to the number of scan directions, in contrast to repeated full-channel scans in 2DSS [2506.22246]. MHS-VM similarly projects features into multiple lower-dimensional subspaces, performs route-specific Mamba scans within each head, and then applies Embedding Section Fusion and Scan Route Attention based on coefficient-of-variation scaling [2406.05992].

A different form of multi-head specialization appears in Mamba-FSCIL. Its dual selective SSM projector consists of a frozen base branch and a trainable incremental branch, and the paper states that this can be viewed as a two-head instance of MHSSM in Few-Shot Class-Incremental Learning [2407.06136]. MossNet pushes the idea further by placing mixture-of-experts routing directly on \(B\), \(C\), and \(\Delta\) in the time-mixing SSM, so that active “heads” emerge through top-\(k\) expert selection per token [2510.26182].

These implementations suggest that “head” in MHSSM is best treated as a functional unit of selective dynamics rather than a single architectural primitive. A plausible implication is that the field is converging on a family resemblance definition: multiple selective state-space pathways, differentiated by scan order, parameterization, routing, or specialization, and fused downstream.

## 4. Spatial and visual instantiations

MHSSM has been especially prominent in vision, dense prediction, restoration, and structured spatial data. In high-resolution plant counting, CountMamba uses \(2\times2\) patches, four directional selective SSM heads, bidirectional scans within each head, and Global-Local Adaptive Fusion. It reports MAE \(4.6\) and RMSE \(7.9\) on maize tassels, MAE \(5.3\) and RMSE \(6.5\) on wheat ears, and MAE \(15.9\) with best RMSE \(18.6\) on sorghum heads, with ablations showing that progressively adding directions improves performance [2410.07528].

In low-level vision, EAMamba introduces an MHSSM with all-around scanning—horizontal, vertical, and diagonal routes—to address “local pixel forgetting.” Its reported outcome is a “31–89% reduction in FLOPs while maintaining favorable performance” across super resolution, denoising, deblurring, and dehazing [2506.22246]. MHS-VM uses multi-head scanning in parallel subspaces for VM-UNet, replacing SS2D with a multi-route head construction and Scan Route Attention; on ISIC18, its lightweight configuration improves mIoU from \(78.66\) to \(79.85\) and DSC from \(88.06\) to \(88.67\) while reducing parameters by about \(48\%\) and FLOPs by about \(56\%\) relative to VM-UNet [2406.05992].

Medical imaging has produced a more explicitly named MHSSM design. The MRI super-resolution framework combines multi-head selective SSMs with a lightweight channel MLP and hybrid scanning over horizontal, vertical, and diagonal directions. It reports \(0.9\)M parameters and \(57\) GFLOPs, with 7T brain performance of SSIM \(=0.951\pm0.021\), PSNR \(=26.90\pm1.41\) dB, LPIPS \(=0.076\pm0.022\), and GMSD \(=0.083\pm0.017\), and prostate performance of SSIM \(=0.770\pm0.049\), PSNR \(=27.15\pm2.19\) dB, LPIPS \(=0.190\pm0.095\), and GMSD \(=0.087\pm0.013\) [2512.19676].

Point-cloud learning uses a related but geometrically distinct formulation. HydraMamba’s MHS6 combines multi-head S6, bidirectional selective scans, a Conv1D local branch, and shuffle serialization via Hilbert variants. It reports \(94.0\%\) OA on ModelNet40, \(86.8\%\) instance mIoU on ShapeNet Part, and \(73.6\%\) mIoU on S3DIS, while its ablation shows that multi-head S6 improves over single-head S6 and peaks at \(H=6\) before slightly degrading at \(H=12\) due to overly small subspace width [2507.19778].

## 5. Sequential, language, and incremental-learning variants

Outside vision, MHSSM ideas have been used to separate stable and plastic dynamics, to emulate attention heads, and to organize temporal expertise. Mamba-FSCIL is the clearest class-incremental example. Its backbone and base selective SSM branch are frozen after the base session, while the incremental branch is trained across novel sessions and regularized by a suppression loss and a separation loss so that selective scan patterns differ between base and novel classes. The resulting two-head interpretation yields average accuracies of \(69.81\%\) on miniImageNet, \(68.12\%\) on CIFAR-100, and \(68.22\%\) on CUB-200, with corresponding final-session accuracies of \(59.36\%\), \(57.51\%\), and \(61.65\%\) [2407.06136].

MossNet provides the strongest language-model argument that MHSSM can approximate attention-like behavior from inside the SSM itself. It places MoE routing on the time-mixing SSM parameters \(\bar{B}_t\), \(C_t\), and \(\Delta_t\), and derives a constructive equivalence in which a mixture of state-space experts emulates a linear multi-head attention. The paper writes the induced output as
$$
y_t=\sum_{m,n}\sum_{i=1}^{t}\langle q_t^{m},k_i^{n}\rangle v_i,
$$
with \(q_t^{m}\) and \(k_i^{n}\) built from router probabilities and selective SSM parameters. This makes MossNet a practical MHSSM whose heads are realized by routed state-space experts rather than by explicit attention projections [2510.26182].

The speech-recognition precursor is structurally different but historically relevant. The Multi-Head State Space Model for Speech Recognition replaces multi-head attention with parallel SSM heads and an inter-head gating rule
$$
a^{(h)} = y^{(h)} \cdot \sigma\!\left(y^{(h + H/2)}\right),
$$
allowing half the heads to gate the other half. Its best Stateformer results are word error rates of \(1.76\%/4.37\%\) on the development sets and \(1.91\%/4.36\%\) on the test sets of LibriSpeech without an external language model [2305.12498]. Although this model is not selective in the later Mamba sense, it established the idea that multiple state-space heads can substitute for attention heads in sequence modeling.

## 6. Theory, misconceptions, and open problems

The most direct theoretical account of MHSSM is the unified framework that compares factorized attention-style mixing with structured dynamical models. Its central result for head counting is the Equivalence (Head-Count) Theorem: if a linear SSM induces lag operators spanning a \(k\)-dimensional subspace on length-\(n\) sequences, then \(H=k\) heads are necessary and sufficient in the corresponding multi-head factorized class to represent it exactly [2512.15115]. The same paper also proves an Interaction Rank Gap, showing that single-head factorized models cannot uniformly approximate certain structured dynamical maps, and a Gradient Highway Result, showing that attention admits distance-independent gradient paths whereas stable linear dynamics exhibit distance-dependent attenuation [2512.15115].

Several misconceptions follow from ignoring these distinctions. The first is that any model combining Mamba with a Transformer is automatically an MHSSM. PointABM directly contradicts this: its multi-head mechanism belongs to the Transformer front-end, while the selective state-space component is bidirectional rather than multi-head [2406.06069]. The second is that more heads monotonically improve performance. HydraMamba reports a non-linear relationship, with performance peaking at \(H=6\) and declining at \(H=12\) [2507.19778]. The third is that scan diversity is always beneficial. EAMamba’s ablations show that scan strategy matters, with all-around \(2D+\) diagonal scanning outperforming Zigzag, Z-order, and Hilbert variants in its restoration setting [2506.22246].

Current limitations recur across implementations. CountMamba notes the absence of explicit positional encodings and the trade-off imposed by very small \(2\times2\) patches [2410.07528]. Mamba-FSCIL highlights interpretability and possible capacity limits as distributions diversify [2407.06136]. MossNet identifies routing instability, expert collapse, and cross-head interference as practical failure modes [2510.26182]. Taken together, these results suggest that the central open problem is no longer whether selective SSMs can be made multi-head, but how best to allocate heads, constrain their specialization, preserve stability, and determine when operator-span gains justify added architectural complexity.

Source: https://www.emergentmind.com/topics/multi-head-selective-state-space-model-mhssm