---
title: 'Mamba-SL: Multi-Domain Selective SSM'
url: https://www.emergentmind.com/topics/mamba-sl
type: topic
---

# Mamba-SL: Multi-Domain Selective SSM

Mamba-SL denotes a non-canonical family of task-specific Mamba adaptations rather than a single standardized architecture. In the available literature, the term is used most concretely for **staged point cloud sequence learning** in Serialized Point Mamba, can be read as **Mamba-based self-supervised learning** in HuBERT-style speech SSL, and is also a plausible shorthand for **Mamba-based segmentation for speaker labeling** in diarization. In the vision survey literature, by contrast, “Mamba-SL” is not defined explicitly; the closest survey-grounded interpretation is Mamba with **selective**, **linear-time** scanning, including bidirectional and 2D selective scan variants such as Vision Mamba and VMamba [2407.12319] [2506.12606] [2410.06459] [2410.03105].

## 1. Terminological status and scope

The literature suggests that “Mamba-SL” is best understood as an umbrella label for several domain-specific instantiations of Mamba’s selective structured state space modeling rather than as a single named model. This ambiguity is explicit in the vision survey, which states that it does not define or introduce a model explicitly named “Mamba-SL,” and in the diarization work, where the paper itself presents a “Mamba-based segmentation model for speaker diarization” rather than a formally named Mamba-SL system [2410.03105] [2410.06459].

| Context | Meaning of “Mamba-SL” | Source |
|---|---|---|
| Point clouds | staged point cloud sequence learning built on Mamba’s selective SSM | [2407.12319] |
| Speech SSL | Mamba-based HuBERT as a speech self-supervised learning pipeline | [2506.12606] |
| Speaker diarization | Mamba-based segmentation for speaker labeling in pyannote EEND-VC | [2410.06459] |
| Vision survey | not defined; closest reading is selective, linear-time scanning in vision Mamba variants | [2410.03105] |

This terminological dispersion is significant because the suffix “SL” does not designate a fixed module, loss, or benchmark. Instead, it indexes how Mamba’s selective scan is operationalized in a given modality: stage-wise serialization in point clouds, masked unit prediction in speech SSL, or frame-level speaker labeling in diarization. A plausible implication is that “Mamba-SL” functions more as a research shorthand for **Mamba plus task-structured sequence/label modeling** than as a stable architectural brand.

## 2. Shared state-space mechanics

Across these usages, the common substrate is the selective structured state space model. The standard continuous-time SSM is written as
$$
\dot{x}(t) = A x(t) + B u(t), \qquad y(t) = C x(t) + D u(t),
$$
with discrete-time parameters obtained through zero-order hold,
$$
\bar{A} = \exp(\Delta A), \qquad \bar{B} = \int_0^\Delta \exp(A\tau)\,B\,d\tau,
$$
yielding the recurrence
$$
x_t = \bar{A}x_{t-1} + \bar{B}u_t, \qquad y_t = Cx_t + Du_t.
$$
Mamba’s distinguishing step is to make parameters input-dependent, for example
$$
\Delta_t = \operatorname{softplus}(W_\Delta u_t + b_\Delta), \qquad B_t = W_B u_t, \qquad C_t = W_C u_t,
$$
so that scanning becomes content-aware while retaining linear complexity in sequence length [2410.03105] [2407.12319] [2506.12606].

The computational consequence is central across modalities. Mamba’s selective scan is reported as linear in sequence length, with complexity expressed as $\mathcal{O}(L\cdot N)$ or $\mathcal{O}(L)$ depending on formulation, whereas Transformer self-attention is quadratic, $\mathcal{O}(L^2)$, in both compute and memory for full attention maps [2410.03105] [2506.12606]. In practice, this linear-time recurrent scan is combined with depthwise 1D convolutions, gating, residual connections, normalization, and task-specific projections. The resulting blocks behave as recurrent scanners with attention-like selectivity, which is why they recur in long-sequence settings such as high-resolution vision, long-form speech, diarization, and serialized point clouds.

## 3. Mamba-SL as staged point cloud sequence learning

In Serialized Point Mamba, Mamba-SL refers explicitly to **staged point cloud sequence learning** built on Mamba’s selective SSM [2407.12319]. The method converts an unordered, sparse 3D point set into learnable 1D sequences, models them with linear-time SSM layers, and progressively pools and re-serializes to expand the receptive field from local neighborhoods to the whole scene. The backbone is U-Net–style: points are voxelized, serialized with space-filling curves such as Z-order or Hilbert curves and their “Trans-” variants, sliced into fixed-length subsequences such as $L=1024$, processed by Mamba blocks for local sequence modeling, reduced by grid pooling, re-serialized with a different order, and finally decoded to per-point predictions [2407.12319].

The stage structure is concrete. Each encoder stage consists of **grid pooling → Conditional Positional Encoding (CPE) → pre-norm → Mamba (1D conv + selective SSM) → post-norm → MLP → residual**. CPE is implemented with sparse submanifold convolutions,
$$
x \leftarrow x + \operatorname{norm}(\operatorname{SubMConv3d}(x)),
$$
and “Enhanced CPE” applies this before every Mamba block rather than only once per downsampling stage. Serialization rotates among `["z", "z-trans", "hilbert", "hilbert-trans"]` to induce cross-subsequence interaction. Two model variants are reported: **Serialized Point Mamba-tiny** with Mamba layers per stage $(2,2,2,6,2)$, channels fixed at $(192,192,192,192,192)$, and bidirectional Mamba enabled; and **Serialized Point Mamba (base)** with the same layer counts but channels $(32,64,128,256,512)$ and unidirectional Mamba [2407.12319].

The reported segmentation results are strong. Serialized Point Mamba achieves **76.8 mIoU on ScanNet**, **70.6 mIoU on S3DIS Area 5**, and **80.6 mIoU on nuScenes**; the abstract reports **70.3 mIoU on S3DIS**. On **Scannetv2 instance segmentation**, it records **40.0 mAP**, with **mAP@0.25 = 76.4** and **mAP@0.50 = 61.4**. Efficiency figures on ScanNetv2 are also explicit: **99 ms / 4.4 GB** inference latency/memory for Serialized Point Mamba, compared with **112 ms / 4.6 GB** for MinkUNet, **247 ms / 18.1 GB** for PTv2, **749 ms / 8.7 GB** for Swin3D, and **114 ms / 12.5 GB** for OctFormer [2407.12319].

The ablation studies clarify what “SL” contributes. Bidirectional Mamba improves the tiny variant by approximately **3.6 points** (**72.3 → 75.94**) but has minimal effect on the base variant (**76.67 vs 76.8**). Serialization diversity matters: **Single Hilbert: 74.64 mIoU; Hilbert+Z: 75.1; Hilbert+Trans-H+Z+Trans-Z: 76.37; random order across layers: 76.8**. Positional encoding is similarly decisive: **No PE: 69.75 mIoU; CPE once per stage: 76.21; Enhanced CPE (before every Mamba block): 76.8**. Pre-norm outperforms post-norm (**76.8 vs 73.35**), and an intermediate subsequence length works best (**L=512: 75.83; L=1024: 76.8; L=2048: 76.23**) [2407.12319]. These results make the point-cloud usage of Mamba-SL the clearest case where the suffix refers to a specific algorithmic design.

## 4. Mamba-SL as speech self-supervised learning

In speech, Mamba-SL can denote a **Mamba-based self-supervised learning** adaptation of HuBERT [2506.12606]. The model retains the HuBERT front-end and quantizer, but replaces Transformer blocks with Mamba blocks. Bidirectional variants include **ExtBiMamba** and **InnBiMamba**; the principal causal model is **Causal Mamba Base** with **78.2M parameters**, compared with **94.7M** for a causal Transformer baseline. The objective remains HuBERT-style masked unit prediction:
$$
L = - \sum_{t \in M} \log p(z_t \mid x_{1:L}),
$$
with **MFCC targets** in iteration 1 for **250k steps**, followed by retraining from scratch in iteration 2 using **k-means over the 6th-layer features** from iteration 1 for **400k steps** [2506.12606].

The main empirical motivation is long-context and causal speech processing. For inputs from **5 s to 320 s**, Mamba-based HuBERT’s **MACs/second remain nearly constant**, while causal Transformers’ MACs/second grow sharply; the causal Transformer runs **OOM beyond ~80 s on an RTX A6000 (48 GB)**. In long-context ASR on **TEDLIUM3**, **ExtBiMamba** fine-tuned on full talks achieves **document-level WER 11.08%**, improving from **utterance-level 13.37% (p = 0.001)**, while the same-size Transformer is **OOM on document-length input**. In streaming or causal ASR, a **78.2M Mamba model** achieves **WER 15.77%**, outperforming a **94.7M causal Transformer** at **16.66%** [2506.12606].

Representation quality is another major theme. In **SUPERB probing** under causal settings, **Mamba+MLP (94.7M)** achieves **PR 11.72, SID 73.48, ER 61.72, IC 89.62, SUPERB_S 823.15**, outperforming the **94.7M causal Transformer** at **PR 13.87, SID 60.04, ER 63.33, IC 94.23, SUPERB_S 805.44**. In small-scale bidirectional settings, **ExtBiMamba (23.2M)** yields **SUPERB_S 809.18**, above the **23.5M Transformer** at **802.63**; at base scale, however, the **94.7M Transformer** remains stronger with **SUPERB_S 868.93** versus **815.38** for **ExtBiMamba (94.3M)**. The paper also reports higher **phone purity peaks** for causal Mamba than for causal Transformer, stronger **CCA similarity to speaker embeddings**, and improved phonetic quality for quantized units derived from Mamba representations [2506.12606].

This domain therefore presents Mamba-SL not merely as a compute-efficient replacement for attention, but as an alternative speech SSL substrate whose strengths are clearest in causal, long-context, and unit-extraction regimes. At the same time, the bidirectional results show that linear-time selectivity does not uniformly dominate Transformer scaling.

## 5. Mamba-SL as speaker labeling in diarization

In speaker diarization, Mamba-SL is most naturally read as **Mamba-based segmentation for speaker labeling** within the pyannote EEND-VC pipeline [2410.06459]. The system replaces the BiLSTM core of the local EEND segmentation module with **7 External Bidirectional Mamba blocks**, uses **frozen WavLM Base** features of **768 dimensions** at approximately **49 fps** for **16 kHz** audio, and applies a **linear projection 768 → 256** before the Mamba stack. The internal state dimension is **$d_{\text{state}} = 64$**, and the output head consists of **two linear layers of hidden size 128** followed by a final layer to either multilabel or powerset outputs [2410.06459].

The diarization formulation is frame-wise. The multilabel head predicts up to **$N$** speakers with sigmoid activation and is trained with permutation-free binary cross-entropy; the powerset head predicts
$$
C = \sum_{i=0}^{K} {N \choose i}
$$
classes representing speaker subsets up to **$K=2$** simultaneous speakers, trained with permutation-free cross-entropy. The diarization error rate is defined as
$$
\text{DER} = \frac{T_{\text{Miss}} + T_{\text{FA}} + T_{\text{Conf}}}{T_{\text{speech}}}\times 100\%.
$$
Window length is a major design variable: **$W \in \{5,10,30,50\}$ seconds**, with **$N=\{4,4,5,6\}$** chosen to cover at least **97%** of training windows [2410.06459].

Training uses a compound dataset of **8 corpora**: **NOTSOFAR-1, MSDWild, VoxConverse, AliMeeting, AMI (ch1), MagicData-RAMC, AISHELL-4**, and a **simulated LibriSpeech-based set with MUSAN noise and RIR augmentation**. Optimization is specified as **80 epochs, 72,000 steps, batch size 32**, with **one warm-up epoch up to lr=0.002** and a **cyclic scheduler with 2-epoch period and 10-epoch half-life decay**. Domain adaptation uses **lr=5e-5**, early stopping after **10 epochs without improvement**, and retuning of clustering hyperparameters through **Optuna (300 trials with multivariate TPE)** [2410.06459].

The paper attributes Mamba’s advantage to its ability to support longer local windows without the memory burden of attention. The full pipeline DER on the validation macro average shows this pattern clearly. For **Mamba (multilabel)**, DER is **17.7** at **5 s**, **16.3** at **10 s**, **16.1** at **30 s**, and **16.9** at **50 s**. The corresponding **BiLSTM (multilabel)** values are **18.1, 17.7, 19.0, 20.1**. For powerset models, **Mamba** yields **17.6, 16.6, 16.4, 17.0**, again surpassing BiLSTM at **18.4, 18.2, 18.7, 19.5** [2410.06459].

Cross-dataset evaluation reinforces the trend. **Mamba 10 s (101M total parameters including WavLM)** consistently surpasses **BiLSTM 10 s (96.5M)** and **Attention 10 s (102M)** on several datasets, including **AISHELL-4: 11.0 vs 11.5 vs 11.9**, **AliMeeting far: 20.0 vs 21.2 vs 21.7**, **AMI ch1: 19.5 vs 20.1 vs 21.3**, **MSDWild: 22.5/16.0 vs 23.2/16.6 vs 24.6/18.2**, and **DIHARD III: 24.4 vs 25.7 vs 24.9**. With domain adaptation, **Mamba 30 s** reaches **AISHELL-4 10.5**, **AliMeeting far 16.2**, **AMI ch1 18.5**, **MSDWild 19.8/13.6**, and **DIHARD III 16.7**. The work reports state-of-the-art performance on **RAMC, AISHELL-4, and MSDWild**, with competitive results on **DIHARD III and AMI** after domain adaptation [2410.06459].

## 6. Adjacent “selective-linear” variants in vision and streaming ASR

The broader Mamba literature provides the context in which the suffix “SL” is often interpreted as **selective linear scanning**. In vision, images are tokenized into patches, serialized into 1D sequences, processed by Mamba blocks, and reshaped back to 2D. The survey catalogs **bidirectional H/V scanning**, **SS2D**, **cross scanning**, **local/windowed selective scanning**, **zigzag**, **spiral**, **radial**, **Hilbert curve**, **plain continuous 2D scanning**, and **atrous/skipping** variants. Representative models include **Vision Mamba (Vim)** with bidirectional scanning and position embeddings, **VMamba** with **SS2D**, **LocalMamba**, **EfficientVMamba**, **PlainMamba**, and video models such as **VideoMamba**, **VideoMambaPro**, and **SSSMLV** [2410.03105].

The survey emphasizes that these models preserve Mamba’s linear-time scan while adapting it to 2D or video structure. Reported benchmarks include **Heracles-C-L** at **85.9% Top-1** on **ImageNet-1K**, **VideoMambaPro-M** at **90.3% Top-1** and **98.5% Top-5** on **Kinetics-400**, within **0.6%** of **TubeViT-H** while using **~89% fewer parameters** and **~74% fewer FLOPs**, and competitive results from **VMamba-S/T** and **LocalVMamba-S** on **COCO** and **ADE20K**. The same survey also notes open issues: **scan anisotropy**, **stability/expressivity trade-offs**, limited large pretrained backbones, reduced interpretability, and adversarial robustness concerns for **VMamba** [2410.03105].

A related but distinct speech usage appears in streaming ASR, where the term Mamba-SL is not introduced, yet the architecture aligns with a “selective-linear” reading [2410.00070]. The system uses a causal Mamba encoder, a controllable convolutional lookahead with
$$
L = \frac{k-1}{2}\Delta,
$$
a streaming-style **Unimodal Aggregation (UMA)** module with
$$
\alpha_t = \sigma(We_t + b), \qquad
c_i = \frac{\sum_{t=\tau_i}^{\tau_{i+1}} \alpha_t e_t}{\sum_{t=\tau_i}^{\tau_{i+1}} \alpha_t},
$$
and an **Early Termination (ET)** mechanism for lower latency. On **AISHELL-1**, **Mamba UMA** achieves **CER 6.59%** with **FT 281 ms, LT 327 ms, Avg 271 ms** at **0 lookahead**, and **CER 5.55%** with **FT 605 ms, LT 453 ms, Avg 568 ms** at **$L=256$ ms**; ET reduces average latency from **271 ms to 196 ms** with **CER 6.82%**. On **AISHELL-2**, **Mamba UMA** yields **CER 7.02%** with **Avg 281 ms** at **0 lookahead**, and **CER 6.08%** with **Avg 764 ms** at **$L=448$ ms** [2410.00070]. This paper extends the broader interpretation of Mamba-SL toward online sequence labeling and emission control.

## 7. Limitations, misconceptions, and open directions

A recurrent misconception is that Mamba-SL names a single canonical architecture. The published evidence indicates otherwise. In point clouds it denotes staged sequence learning; in speech SSL it describes a HuBERT-style Mamba backbone; in diarization it corresponds to a Mamba local EEND segmentation model; in vision and streaming ASR it is, at most, an inferred shorthand for selective linear scanning [2407.12319] [2506.12606] [2410.06459] [2410.03105] [2410.00070].

Another misconception is that linear-time scanning by itself guarantees uniform superiority over Transformers or RNNs. The cross-domain record is more nuanced. In speech SSL, **ExtBiMamba** at base scale lags the best bidirectional Transformer on **SUPERB** despite its advantages in causal settings and long-context ASR [2506.12606]. In diarization, **powerset loss** can improve local DER yet harm the downstream pipeline because higher speaker confusion contaminates embedding extraction and clustering [2410.06459]. In point clouds, **bidirectional Mamba** substantially helps the tiny model but not the base model, suggesting that serialization diversity and stage-wise pooling already provide much of the long-range interaction [2407.12319]. In vision, the survey highlights **scan anisotropy**, **hidden-state accumulation**, limited pretraining resources, interpretability difficulties, and robustness degradation with scale [2410.03105].

The open research agenda is correspondingly heterogeneous. For point clouds, the stated directions include scaling to very large scenes, improving robustness to noise and extreme sparsity, extending to multimodal inputs, and introducing more explicit cross-subsequence communication [2407.12319]. For speech SSL, future work centers on scaling bidirectional Mamba, richer streaming setups, larger and multilingual pretraining, and deeper analysis of unit quality [2506.12606]. For diarization, unresolved issues include rapid speaker turns, heavy overlap, joint training with the embedding extractor, multilingual or domain-robust pretraining, and better overlap-aware losses [2410.06459]. For streaming ASR, the principal trade-off remains between lookahead-induced accuracy gains and emission latency, while ET can reduce latency at the cost of modest CER increases [2410.00070].

Taken together, these works suggest that Mamba-SL is best viewed as a research pattern: a selective state-space backbone combined with task-specific serialization, masking, labeling, or emission mechanisms to exploit Mamba’s linear-time scan without discarding domain structure. Its importance lies less in a fixed definition than in the convergent design principle it represents across 3D perception, speech representation learning, diarization, vision, and streaming recognition.

Source: https://www.emergentmind.com/topics/mamba-sl