Multi-Head Selective SSM (MHSSM)

• MHSSM is a family of architectures that extends selective state-space models by introducing multiple parallel heads with distinct scanning orders.
• It leverages independent head dynamics and fusion mechanisms to achieve improved performance and efficiency across tasks in vision, language, and incremental learning.
• Recent implementations like CountMamba and HydraMamba illustrate its practical benefits alongside challenges such as routing instability and capacity trade-offs.

Searching arXiv for recent MHSSM-related papers to ground the article with up-to-date references. arxiv_search(query="Multi-Head Selective State-Space Model OR MHSSM state space model multi-head Mamba", max_results=10, sort_by="relevance") A Multi-Head Selective State-Space Model (MHSSM) is a family of architectures that extends selective state-space modeling by introducing multiple parallel heads, experts, or scan subspaces, each with its own selective dynamics, scan order, or parameter generator, and then fusing the resulting outputs. Across recent literature, the term covers several closely related constructions: directional experts that scan the same input in different orders, channel-partitioned selective scans over parallel subspaces, dual-branch or multi-branch selective projectors, and mixture-of-experts routing inside the state-space kernel itself. The unifying purpose is to increase directional diversity, operator span, or task-specific specialization while retaining the linear-time scan behavior associated with Mamba-like selective SSMs (He et al., 2024, Lin et al., 27 Jun 2025, Qu et al., 26 Jul 2025, Tuli et al., 30 Oct 2025).

1. Conceptual scope and nomenclature

The “multi-head” component of MHSSM is not defined by a single canonical implementation. In current practice, a head may be a direction-specific state-space branch, a channel-group subspace with its own selective scan, a frozen or plastic projector branch, or a routed state-space expert. CountMamba describes its Multi-directional State-Space Group as an MHSSM with $H=4$ heads—Horizontal, Vertical, Diagonal, and Anti-diagonal—where each directional expert scans the same image under a different ordering (He et al., 2024). EAMamba instead realizes MHSSM by splitting expanded channels into multiple groups and assigning distinct scan directions to different heads without increasing FLOPs or parameter count with the number of sequences (Lin et al., 27 Jun 2025). HydraMamba defines MHS6 by partitioning channels into $H$ heads, each running an input-dependent S6 with head-specific parameters and output fusion by concatenation and projection (Qu et al., 26 Jul 2025).

This broader usage also explains why the term should not be applied indiscriminately. PointABM integrates a bidirectional selective state-space model with Transformer multi-head self-attention, but its own SSM block is bidirectional rather than multi-head; the paper explicitly states that it “is not a Multi-Head Selective State-Space Model (MHSSM) in the strict sense” (Chen et al., 2024). By contrast, MHS-VM, CountMamba, EAMamba, HydraMamba, and MossNet all instantiate multi-head structure inside or directly around the selective state-space mechanism itself (Ji, 2024, He et al., 2024, Lin et al., 27 Jun 2025, Qu et al., 26 Jul 2025, Tuli et al., 30 Oct 2025).

A useful historical precursor is the earlier Multi-Head State Space Model for speech recognition, which introduced parallel state-space heads and inter-head gating, but not Mamba-style selectivity in the later sense. It is therefore best understood as antecedent rather than definitive MHSSM terminology (Fathullah et al., 2023).

2. Selective state-space foundations

Most MHSSM formulations inherit the standard continuous-time and discretized SSM framework. A canonical continuous-time system is

$$\dot{h}(t)=A\,h(t)+B\,x(t),\qquad y(t)=C\,h(t)+D\,x(t),$$

and Mamba-style models discretize this system with an input-dependent step size $\Delta_t$, yielding discrete parameters such as

$$\bar{A}_t = e^{\Delta_t A},\qquad \bar{B}_t = \Phi(\Delta_t,A)B.$$

The resulting selective recurrence takes forms such as

$$h_t=\bar{A}_t h_{t-1}+\bar{B}_t u_t,\qquad y_t=C_t h_t,$$

or, in CountMamba’s Mamba-style formulation,

$$h_k=g_k\odot(\bar{A}_k h_{k-1}+\bar{B}_k u_k)+(1-g_k)\odot h_{k-1},\qquad y_k=C_k h_k,$$

with $\Delta_k$, $g_k$, $B_k$, and $H$0 all input-dependent (He et al., 2024, Safari et al., 22 Dec 2025).

The multi-head extension introduces independent per-head dynamics. HydraMamba states this explicitly:

$H$1

with

$H$2

Its selectivity is implemented through head-specific $H$3, $H$4, and $H$5, where $H$6 is produced by a softplus-gated linear map (Qu et al., 26 Jul 2025). The MRI super-resolution framework similarly uses $H$7 parallel heads, each with its own diagonal-stable $H$8, adaptive $H$9, and merged output projection (Safari et al., 22 Dec 2025).

A broader operator-theoretic view is supplied by the unified framework of sequence modeling. There, a sequence map is written as

$$\dot{h}(t)=A\,h(t)+B\,x(t),\qquad y(t)=C\,h(t)+D\,x(t),$$0

where $$\dot{h}(t)=A\,h(t)+B\,x(t),\qquad y(t)=C\,h(t)+D\,x(t),$$1 is the input-dependent effective interaction operator. In the structured-dynamics case, selective SSMs induce operators of the form

$$\dot{h}(t)=A\,h(t)+B\,x(t),\qquad y(t)=C\,h(t)+D\,x(t),$$2

which makes explicit how selective dynamics define context-dependent lag interactions (Ghodsi, 17 Dec 2025).

3. Head construction and fusion mechanisms

Recent MHSSM implementations differ mainly in how heads are defined and how their outputs are fused.

System	Head construction	Fusion
CountMamba	Four directional experts: HSSM, VSSM, DSSM, ASSM	Softmax-weighted Global-Local Adaptive Fusion
EAMamba	Channel-group heads with distinct scan directions	Concat after inverse transforms, then gated projection
HydraMamba	Per-head S6 dynamics over channel partitions	Channel concatenation and linear projection
MHS-VM	Parallel subspaces, each with multiple scan routes	ESF and optional tail projection
Mamba-FSCIL	Frozen base branch and trainable incremental branch	Additive aggregation with identity branch
MossNet	MoE-routed time-mixing experts	Top-$$\dot{h}(t)=A\,h(t)+B\,x(t),\qquad y(t)=C\,h(t)+D\,x(t),$$3 weighted expert aggregation

CountMamba’s Multi-directional State-Space Group uses four independent counting experts and finds that “Four experts” perform best, while Global-Local Adaptive Fusion computes sample-wise softmax weights $$\dot{h}(t)=A\,h(t)+B\,x(t),\qquad y(t)=C\,h(t)+D\,x(t),$$4 and combines them with a local CNN branch (He et al., 2024). EAMamba’s MHSSM instead groups channels into heads and applies head-specific transforms before a selective scan; its key claim is that this grouped design keeps FLOPs and parameters stable with respect to the number of scan directions, in contrast to repeated full-channel scans in 2DSS (Lin et al., 27 Jun 2025). MHS-VM similarly projects features into multiple lower-dimensional subspaces, performs route-specific Mamba scans within each head, and then applies Embedding Section Fusion and Scan Route Attention based on coefficient-of-variation scaling (Ji, 2024).

A different form of multi-head specialization appears in Mamba-FSCIL. Its dual selective SSM projector consists of a frozen base branch and a trainable incremental branch, and the paper states that this can be viewed as a two-head instance of MHSSM in Few-Shot Class-Incremental Learning (Li et al., 2024). MossNet pushes the idea further by placing mixture-of-experts routing directly on $$\dot{h}(t)=A\,h(t)+B\,x(t),\qquad y(t)=C\,h(t)+D\,x(t),$$5, $$\dot{h}(t)=A\,h(t)+B\,x(t),\qquad y(t)=C\,h(t)+D\,x(t),$$6, and $$\dot{h}(t)=A\,h(t)+B\,x(t),\qquad y(t)=C\,h(t)+D\,x(t),$$7 in the time-mixing SSM, so that active “heads” emerge through top-$$\dot{h}(t)=A\,h(t)+B\,x(t),\qquad y(t)=C\,h(t)+D\,x(t),$$8 expert selection per token (Tuli et al., 30 Oct 2025).

These implementations suggest that “head” in MHSSM is best treated as a functional unit of selective dynamics rather than a single architectural primitive. A plausible implication is that the field is converging on a family resemblance definition: multiple selective state-space pathways, differentiated by scan order, parameterization, routing, or specialization, and fused downstream.

4. Spatial and visual instantiations

MHSSM has been especially prominent in vision, dense prediction, restoration, and structured spatial data. In high-resolution plant counting, CountMamba uses $$\dot{h}(t)=A\,h(t)+B\,x(t),\qquad y(t)=C\,h(t)+D\,x(t),$$9 patches, four directional selective SSM heads, bidirectional scans within each head, and Global-Local Adaptive Fusion. It reports MAE $\Delta_t$0 and RMSE $\Delta_t$1 on maize tassels, MAE $\Delta_t$2 and RMSE $\Delta_t$3 on wheat ears, and MAE $\Delta_t$4 with best RMSE $\Delta_t$5 on sorghum heads, with ablations showing that progressively adding directions improves performance (He et al., 2024).

In low-level vision, EAMamba introduces an MHSSM with all-around scanning—horizontal, vertical, and diagonal routes—to address “local pixel forgetting.” Its reported outcome is a “31–89% reduction in FLOPs while maintaining favorable performance” across super resolution, denoising, deblurring, and dehazing (Lin et al., 27 Jun 2025). MHS-VM uses multi-head scanning in parallel subspaces for VM-UNet, replacing SS2D with a multi-route head construction and Scan Route Attention; on ISIC18, its lightweight configuration improves mIoU from $\Delta_t$6 to $\Delta_t$7 and DSC from $\Delta_t$8 to $\Delta_t$9 while reducing parameters by about $$\bar{A}_t = e^{\Delta_t A},\qquad \bar{B}_t = \Phi(\Delta_t,A)B.$$0 and FLOPs by about $$\bar{A}_t = e^{\Delta_t A},\qquad \bar{B}_t = \Phi(\Delta_t,A)B.$$1 relative to VM-UNet (Ji, 2024).

Medical imaging has produced a more explicitly named MHSSM design. The MRI super-resolution framework combines multi-head selective SSMs with a lightweight channel MLP and hybrid scanning over horizontal, vertical, and diagonal directions. It reports $$\bar{A}_t = e^{\Delta_t A},\qquad \bar{B}_t = \Phi(\Delta_t,A)B.$$2M parameters and $$\bar{A}_t = e^{\Delta_t A},\qquad \bar{B}_t = \Phi(\Delta_t,A)B.$$3 GFLOPs, with 7T brain performance of SSIM $$\bar{A}_t = e^{\Delta_t A},\qquad \bar{B}_t = \Phi(\Delta_t,A)B.$$4, PSNR $$\bar{A}_t = e^{\Delta_t A},\qquad \bar{B}_t = \Phi(\Delta_t,A)B.$$5 dB, LPIPS $$\bar{A}_t = e^{\Delta_t A},\qquad \bar{B}_t = \Phi(\Delta_t,A)B.$$6, and GMSD $$\bar{A}_t = e^{\Delta_t A},\qquad \bar{B}_t = \Phi(\Delta_t,A)B.$$7, and prostate performance of SSIM $$\bar{A}_t = e^{\Delta_t A},\qquad \bar{B}_t = \Phi(\Delta_t,A)B.$$8, PSNR $$\bar{A}_t = e^{\Delta_t A},\qquad \bar{B}_t = \Phi(\Delta_t,A)B.$$9 dB, LPIPS $$h_t=\bar{A}_t h_{t-1}+\bar{B}_t u_t,\qquad y_t=C_t h_t,$$0, and GMSD $$h_t=\bar{A}_t h_{t-1}+\bar{B}_t u_t,\qquad y_t=C_t h_t,$$1 (Safari et al., 22 Dec 2025).

Point-cloud learning uses a related but geometrically distinct formulation. HydraMamba’s MHS6 combines multi-head S6, bidirectional selective scans, a Conv1D local branch, and shuffle serialization via Hilbert variants. It reports $$h_t=\bar{A}_t h_{t-1}+\bar{B}_t u_t,\qquad y_t=C_t h_t,$$2 OA on ModelNet40, $$h_t=\bar{A}_t h_{t-1}+\bar{B}_t u_t,\qquad y_t=C_t h_t,$$3 instance mIoU on ShapeNet Part, and $$h_t=\bar{A}_t h_{t-1}+\bar{B}_t u_t,\qquad y_t=C_t h_t,$$4 mIoU on S3DIS, while its ablation shows that multi-head S6 improves over single-head S6 and peaks at $$h_t=\bar{A}_t h_{t-1}+\bar{B}_t u_t,\qquad y_t=C_t h_t,$$5 before slightly degrading at $$h_t=\bar{A}_t h_{t-1}+\bar{B}_t u_t,\qquad y_t=C_t h_t,$$6 due to overly small subspace width (Qu et al., 26 Jul 2025).

5. Sequential, language, and incremental-learning variants

Outside vision, MHSSM ideas have been used to separate stable and plastic dynamics, to emulate attention heads, and to organize temporal expertise. Mamba-FSCIL is the clearest class-incremental example. Its backbone and base selective SSM branch are frozen after the base session, while the incremental branch is trained across novel sessions and regularized by a suppression loss and a separation loss so that selective scan patterns differ between base and novel classes. The resulting two-head interpretation yields average accuracies of $$h_t=\bar{A}_t h_{t-1}+\bar{B}_t u_t,\qquad y_t=C_t h_t,$$7 on miniImageNet, $$h_t=\bar{A}_t h_{t-1}+\bar{B}_t u_t,\qquad y_t=C_t h_t,$$8 on CIFAR-100, and $$h_t=\bar{A}_t h_{t-1}+\bar{B}_t u_t,\qquad y_t=C_t h_t,$$9 on CUB-200, with corresponding final-session accuracies of $$h_k=g_k\odot(\bar{A}_k h_{k-1}+\bar{B}_k u_k)+(1-g_k)\odot h_{k-1},\qquad y_k=C_k h_k,$$0, $$h_k=g_k\odot(\bar{A}_k h_{k-1}+\bar{B}_k u_k)+(1-g_k)\odot h_{k-1},\qquad y_k=C_k h_k,$$1, and $$h_k=g_k\odot(\bar{A}_k h_{k-1}+\bar{B}_k u_k)+(1-g_k)\odot h_{k-1},\qquad y_k=C_k h_k,$$2 (Li et al., 2024).

MossNet provides the strongest language-model argument that MHSSM can approximate attention-like behavior from inside the SSM itself. It places MoE routing on the time-mixing SSM parameters $$h_k=g_k\odot(\bar{A}_k h_{k-1}+\bar{B}_k u_k)+(1-g_k)\odot h_{k-1},\qquad y_k=C_k h_k,$$3, $$h_k=g_k\odot(\bar{A}_k h_{k-1}+\bar{B}_k u_k)+(1-g_k)\odot h_{k-1},\qquad y_k=C_k h_k,$$4, and $$h_k=g_k\odot(\bar{A}_k h_{k-1}+\bar{B}_k u_k)+(1-g_k)\odot h_{k-1},\qquad y_k=C_k h_k,$$5, and derives a constructive equivalence in which a mixture of state-space experts emulates a linear multi-head attention. The paper writes the induced output as

$$h_k=g_k\odot(\bar{A}_k h_{k-1}+\bar{B}_k u_k)+(1-g_k)\odot h_{k-1},\qquad y_k=C_k h_k,$$6

with $$h_k=g_k\odot(\bar{A}_k h_{k-1}+\bar{B}_k u_k)+(1-g_k)\odot h_{k-1},\qquad y_k=C_k h_k,$$7 and $$h_k=g_k\odot(\bar{A}_k h_{k-1}+\bar{B}_k u_k)+(1-g_k)\odot h_{k-1},\qquad y_k=C_k h_k,$$8 built from router probabilities and selective SSM parameters. This makes MossNet a practical MHSSM whose heads are realized by routed state-space experts rather than by explicit attention projections (Tuli et al., 30 Oct 2025).

The speech-recognition precursor is structurally different but historically relevant. The Multi-Head State Space Model for Speech Recognition replaces multi-head attention with parallel SSM heads and an inter-head gating rule

$$h_k=g_k\odot(\bar{A}_k h_{k-1}+\bar{B}_k u_k)+(1-g_k)\odot h_{k-1},\qquad y_k=C_k h_k,$$9

allowing half the heads to gate the other half. Its best Stateformer results are word error rates of $\Delta_k$0 on the development sets and $\Delta_k$1 on the test sets of LibriSpeech without an external LLM (Fathullah et al., 2023). Although this model is not selective in the later Mamba sense, it established the idea that multiple state-space heads can substitute for attention heads in sequence modeling.

6. Theory, misconceptions, and open problems

The most direct theoretical account of MHSSM is the unified framework that compares factorized attention-style mixing with structured dynamical models. Its central result for head counting is the Equivalence (Head-Count) Theorem: if a linear SSM induces lag operators spanning a $\Delta_k$2-dimensional subspace on length-$\Delta_k$3 sequences, then $\Delta_k$4 heads are necessary and sufficient in the corresponding multi-head factorized class to represent it exactly (Ghodsi, 17 Dec 2025). The same paper also proves an Interaction Rank Gap, showing that single-head factorized models cannot uniformly approximate certain structured dynamical maps, and a Gradient Highway Result, showing that attention admits distance-independent gradient paths whereas stable linear dynamics exhibit distance-dependent attenuation (Ghodsi, 17 Dec 2025).

Several misconceptions follow from ignoring these distinctions. The first is that any model combining Mamba with a Transformer is automatically an MHSSM. PointABM directly contradicts this: its multi-head mechanism belongs to the Transformer front-end, while the selective state-space component is bidirectional rather than multi-head (Chen et al., 2024). The second is that more heads monotonically improve performance. HydraMamba reports a non-linear relationship, with performance peaking at $\Delta_k$5 and declining at $\Delta_k$6 (Qu et al., 26 Jul 2025). The third is that scan diversity is always beneficial. EAMamba’s ablations show that scan strategy matters, with all-around $\Delta_k$7 diagonal scanning outperforming Zigzag, Z-order, and Hilbert variants in its restoration setting (Lin et al., 27 Jun 2025).

Current limitations recur across implementations. CountMamba notes the absence of explicit positional encodings and the trade-off imposed by very small $\Delta_k$8 patches (He et al., 2024). Mamba-FSCIL highlights interpretability and possible capacity limits as distributions diversify (Li et al., 2024). MossNet identifies routing instability, expert collapse, and cross-head interference as practical failure modes (Tuli et al., 30 Oct 2025). Taken together, these results suggest that the central open problem is no longer whether selective SSMs can be made multi-head, but how best to allocate heads, constrain their specialization, preserve stability, and determine when operator-span gains justify added architectural complexity.