Mamba-SL: Multi-Domain Selective SSM

• Mamba-SL is an umbrella term for selective state space model adaptations tailored for point cloud sequence learning, speech self-supervised learning, and speaker diarization.
• It employs input-dependent parameters to achieve content-aware, linear-time scanning, offering computational efficiency over traditional quadratic attention models.
• Empirical studies reveal strong segmentation, ASR, and diarization performance while highlighting trade-offs in bidirectional versus unidirectional designs and serialization strategies.

Mamba-SL denotes a non-canonical family of task-specific Mamba adaptations rather than a single standardized architecture. In the available literature, the term is used most concretely for staged point cloud sequence learning in Serialized Point Mamba, can be read as Mamba-based self-supervised learning in HuBERT-style speech SSL, and is also a plausible shorthand for Mamba-based segmentation for speaker labeling in diarization. In the vision survey literature, by contrast, “Mamba-SL” is not defined explicitly; the closest survey-grounded interpretation is Mamba with selective, linear-time scanning, including bidirectional and 2D selective scan variants such as Vision Mamba and VMamba (Wang et al., 2024, Lin et al., 14 Jun 2025, Plaquet et al., 2024, Rahman et al., 2024).

1. Terminological status and scope

The literature suggests that “Mamba-SL” is best understood as an umbrella label for several domain-specific instantiations of Mamba’s selective structured state space modeling rather than as a single named model. This ambiguity is explicit in the vision survey, which states that it does not define or introduce a model explicitly named “Mamba-SL,” and in the diarization work, where the paper itself presents a “Mamba-based segmentation model for speaker diarization” rather than a formally named Mamba-SL system (Rahman et al., 2024, Plaquet et al., 2024).

Context	Meaning of “Mamba-SL”	Source
Point clouds	staged point cloud sequence learning built on Mamba’s selective SSM	(Wang et al., 2024)
Speech SSL	Mamba-based HuBERT as a speech self-supervised learning pipeline	(Lin et al., 14 Jun 2025)
Speaker diarization	Mamba-based segmentation for speaker labeling in pyannote EEND-VC	(Plaquet et al., 2024)
Vision survey	not defined; closest reading is selective, linear-time scanning in vision Mamba variants	(Rahman et al., 2024)

This terminological dispersion is significant because the suffix “SL” does not designate a fixed module, loss, or benchmark. Instead, it indexes how Mamba’s selective scan is operationalized in a given modality: stage-wise serialization in point clouds, masked unit prediction in speech SSL, or frame-level speaker labeling in diarization. A plausible implication is that “Mamba-SL” functions more as a research shorthand for Mamba plus task-structured sequence/label modeling than as a stable architectural brand.

2. Shared state-space mechanics

Across these usages, the common substrate is the selective structured state space model. The standard continuous-time SSM is written as

$$\dot{x}(t) = A x(t) + B u(t), \qquad y(t) = C x(t) + D u(t),$$

with discrete-time parameters obtained through zero-order hold,

$$\bar{A} = \exp(\Delta A), \qquad \bar{B} = \int_0^\Delta \exp(A\tau)\,B\,d\tau,$$

yielding the recurrence

$$x_t = \bar{A}x_{t-1} + \bar{B}u_t, \qquad y_t = Cx_t + Du_t.$$

Mamba’s distinguishing step is to make parameters input-dependent, for example

$$\Delta_t = \operatorname{softplus}(W_\Delta u_t + b_\Delta), \qquad B_t = W_B u_t, \qquad C_t = W_C u_t,$$

so that scanning becomes content-aware while retaining linear complexity in sequence length (Rahman et al., 2024, Wang et al., 2024, Lin et al., 14 Jun 2025).

The computational consequence is central across modalities. Mamba’s selective scan is reported as linear in sequence length, with complexity expressed as $\mathcal{O}(L\cdot N)$ or $\mathcal{O}(L)$ depending on formulation, whereas Transformer self-attention is quadratic, $\mathcal{O}(L^2)$, in both compute and memory for full attention maps (Rahman et al., 2024, Lin et al., 14 Jun 2025). In practice, this linear-time recurrent scan is combined with depthwise 1D convolutions, gating, residual connections, normalization, and task-specific projections. The resulting blocks behave as recurrent scanners with attention-like selectivity, which is why they recur in long-sequence settings such as high-resolution vision, long-form speech, diarization, and serialized point clouds.

3. Mamba-SL as staged point cloud sequence learning

In Serialized Point Mamba, Mamba-SL refers explicitly to staged point cloud sequence learning built on Mamba’s selective SSM (Wang et al., 2024). The method converts an unordered, sparse 3D point set into learnable 1D sequences, models them with linear-time SSM layers, and progressively pools and re-serializes to expand the receptive field from local neighborhoods to the whole scene. The backbone is U-Net–style: points are voxelized, serialized with space-filling curves such as Z-order or Hilbert curves and their “Trans-” variants, sliced into fixed-length subsequences such as $L=1024$, processed by Mamba blocks for local sequence modeling, reduced by grid pooling, re-serialized with a different order, and finally decoded to per-point predictions (Wang et al., 2024).

The stage structure is concrete. Each encoder stage consists of grid pooling → Conditional Positional Encoding (CPE) → pre-norm → Mamba (1D conv + selective SSM) → post-norm → MLP → residual. CPE is implemented with sparse submanifold convolutions,

$$x \leftarrow x + \operatorname{norm}(\operatorname{SubMConv3d}(x)),$$

and “Enhanced CPE” applies this before every Mamba block rather than only once per downsampling stage. Serialization rotates among ["z", "z-trans", "hilbert", "hilbert-trans"] to induce cross-subsequence interaction. Two model variants are reported: Serialized Point Mamba-tiny with Mamba layers per stage $(2,2,2,6,2)$, channels fixed at $$\bar{A} = \exp(\Delta A), \qquad \bar{B} = \int_0^\Delta \exp(A\tau)\,B\,d\tau,$$0, and bidirectional Mamba enabled; and Serialized Point Mamba (base) with the same layer counts but channels $$\bar{A} = \exp(\Delta A), \qquad \bar{B} = \int_0^\Delta \exp(A\tau)\,B\,d\tau,$$1 and unidirectional Mamba (Wang et al., 2024).

The reported segmentation results are strong. Serialized Point Mamba achieves 76.8 mIoU on ScanNet, 70.6 mIoU on S3DIS Area 5, and 80.6 mIoU on nuScenes; the abstract reports 70.3 mIoU on S3DIS. On Scannetv2 instance segmentation, it records 40.0 mAP, with [email protected] = 76.4 and [email protected] = 61.4. Efficiency figures on ScanNetv2 are also explicit: 99 ms / 4.4 GB inference latency/memory for Serialized Point Mamba, compared with 112 ms / 4.6 GB for MinkUNet, 247 ms / 18.1 GB for PTv2, 749 ms / 8.7 GB for Swin3D, and 114 ms / 12.5 GB for OctFormer (Wang et al., 2024).

The ablation studies clarify what “SL” contributes. Bidirectional Mamba improves the tiny variant by approximately 3.6 points (72.3 → 75.94) but has minimal effect on the base variant (76.67 vs 76.8). Serialization diversity matters: Single Hilbert: 74.64 mIoU; Hilbert+Z: 75.1; Hilbert+Trans-H+Z+Trans-Z: 76.37; random order across layers: 76.8. Positional encoding is similarly decisive: No PE: 69.75 mIoU; CPE once per stage: 76.21; Enhanced CPE (before every Mamba block): 76.8. Pre-norm outperforms post-norm (76.8 vs 73.35), and an intermediate subsequence length works best (L=512: 75.83; L=1024: 76.8; L=2048: 76.23) (Wang et al., 2024). These results make the point-cloud usage of Mamba-SL the clearest case where the suffix refers to a specific algorithmic design.

4. Mamba-SL as speech self-supervised learning

In speech, Mamba-SL can denote a Mamba-based self-supervised learning adaptation of HuBERT (Lin et al., 14 Jun 2025). The model retains the HuBERT front-end and quantizer, but replaces Transformer blocks with Mamba blocks. Bidirectional variants include ExtBiMamba and InnBiMamba; the principal causal model is Causal Mamba Base with 78.2M parameters, compared with 94.7M for a causal Transformer baseline. The objective remains HuBERT-style masked unit prediction:

$$\bar{A} = \exp(\Delta A), \qquad \bar{B} = \int_0^\Delta \exp(A\tau)\,B\,d\tau,$$2

with MFCC targets in iteration 1 for 250k steps, followed by retraining from scratch in iteration 2 using k-means over the 6th-layer features from iteration 1 for 400k steps (Lin et al., 14 Jun 2025).

The main empirical motivation is long-context and causal speech processing. For inputs from 5 s to 320 s, Mamba-based HuBERT’s MACs/second remain nearly constant, while causal Transformers’ MACs/second grow sharply; the causal Transformer runs OOM beyond ~80 s on an RTX A6000 (48 GB). In long-context ASR on TEDLIUM3, ExtBiMamba fine-tuned on full talks achieves document-level WER 11.08%, improving from utterance-level 13.37% (p = 0.001), while the same-size Transformer is OOM on document-length input. In streaming or causal ASR, a 78.2M Mamba model achieves WER 15.77%, outperforming a 94.7M causal Transformer at 16.66% (Lin et al., 14 Jun 2025).

Representation quality is another major theme. In SUPERB probing under causal settings, Mamba+MLP (94.7M) achieves PR 11.72, SID 73.48, ER 61.72, IC 89.62, SUPERB_S 823.15, outperforming the 94.7M causal Transformer at PR 13.87, SID 60.04, ER 63.33, IC 94.23, SUPERB_S 805.44. In small-scale bidirectional settings, ExtBiMamba (23.2M) yields SUPERB_S 809.18, above the 23.5M Transformer at 802.63; at base scale, however, the 94.7M Transformer remains stronger with SUPERB_S 868.93 versus 815.38 for ExtBiMamba (94.3M). The paper also reports higher phone purity peaks for causal Mamba than for causal Transformer, stronger CCA similarity to speaker embeddings, and improved phonetic quality for quantized units derived from Mamba representations (Lin et al., 14 Jun 2025).

This domain therefore presents Mamba-SL not merely as a compute-efficient replacement for attention, but as an alternative speech SSL substrate whose strengths are clearest in causal, long-context, and unit-extraction regimes. At the same time, the bidirectional results show that linear-time selectivity does not uniformly dominate Transformer scaling.

5. Mamba-SL as speaker labeling in diarization

In speaker diarization, Mamba-SL is most naturally read as Mamba-based segmentation for speaker labeling within the pyannote EEND-VC pipeline (Plaquet et al., 2024). The system replaces the BiLSTM core of the local EEND segmentation module with 7 External Bidirectional Mamba blocks, uses frozen WavLM Base features of 768 dimensions at approximately 49 fps for 16 kHz audio, and applies a linear projection 768 → 256 before the Mamba stack. The internal state dimension is $$\bar{A} = \exp(\Delta A), \qquad \bar{B} = \int_0^\Delta \exp(A\tau)\,B\,d\tau,$$3, and the output head consists of two linear layers of hidden size 128 followed by a final layer to either multilabel or powerset outputs (Plaquet et al., 2024).

The diarization formulation is frame-wise. The multilabel head predicts up to $$\bar{A} = \exp(\Delta A), \qquad \bar{B} = \int_0^\Delta \exp(A\tau)\,B\,d\tau,$$4 speakers with sigmoid activation and is trained with permutation-free binary cross-entropy; the powerset head predicts

$$\bar{A} = \exp(\Delta A), \qquad \bar{B} = \int_0^\Delta \exp(A\tau)\,B\,d\tau,$$5

classes representing speaker subsets up to $$\bar{A} = \exp(\Delta A), \qquad \bar{B} = \int_0^\Delta \exp(A\tau)\,B\,d\tau,$$6 simultaneous speakers, trained with permutation-free cross-entropy. The diarization error rate is defined as

$$\bar{A} = \exp(\Delta A), \qquad \bar{B} = \int_0^\Delta \exp(A\tau)\,B\,d\tau,$$7

Window length is a major design variable: $$\bar{A} = \exp(\Delta A), \qquad \bar{B} = \int_0^\Delta \exp(A\tau)\,B\,d\tau,$$8 seconds, with $$\bar{A} = \exp(\Delta A), \qquad \bar{B} = \int_0^\Delta \exp(A\tau)\,B\,d\tau,$$9 chosen to cover at least 97% of training windows (Plaquet et al., 2024).

Training uses a compound dataset of 8 corpora: NOTSOFAR-1, MSDWild, VoxConverse, AliMeeting, AMI (ch1), MagicData-RAMC, AISHELL-4, and a simulated LibriSpeech-based set with MUSAN noise and RIR augmentation. Optimization is specified as 80 epochs, 72,000 steps, batch size 32, with one warm-up epoch up to lr=0.002 and a cyclic scheduler with 2-epoch period and 10-epoch half-life decay. Domain adaptation uses lr=5e-5, early stopping after 10 epochs without improvement, and retuning of clustering hyperparameters through Optuna (300 trials with multivariate TPE) (Plaquet et al., 2024).

The paper attributes Mamba’s advantage to its ability to support longer local windows without the memory burden of attention. The full pipeline DER on the validation macro average shows this pattern clearly. For Mamba (multilabel), DER is 17.7 at 5 s, 16.3 at 10 s, 16.1 at 30 s, and 16.9 at 50 s. The corresponding BiLSTM (multilabel) values are 18.1, 17.7, 19.0, 20.1. For powerset models, Mamba yields 17.6, 16.6, 16.4, 17.0, again surpassing BiLSTM at 18.4, 18.2, 18.7, 19.5 (Plaquet et al., 2024).

Cross-dataset evaluation reinforces the trend. Mamba 10 s (101M total parameters including WavLM) consistently surpasses BiLSTM 10 s (96.5M) and Attention 10 s (102M) on several datasets, including AISHELL-4: 11.0 vs 11.5 vs 11.9, AliMeeting far: 20.0 vs 21.2 vs 21.7, AMI ch1: 19.5 vs 20.1 vs 21.3, MSDWild: 22.5/16.0 vs 23.2/16.6 vs 24.6/18.2, and DIHARD III: 24.4 vs 25.7 vs 24.9. With domain adaptation, Mamba 30 s reaches AISHELL-4 10.5, AliMeeting far 16.2, AMI ch1 18.5, MSDWild 19.8/13.6, and DIHARD III 16.7. The work reports state-of-the-art performance on RAMC, AISHELL-4, and MSDWild, with competitive results on DIHARD III and AMI after domain adaptation (Plaquet et al., 2024).

6. Adjacent “selective-linear” variants in vision and streaming ASR

The broader Mamba literature provides the context in which the suffix “SL” is often interpreted as selective linear scanning. In vision, images are tokenized into patches, serialized into 1D sequences, processed by Mamba blocks, and reshaped back to 2D. The survey catalogs bidirectional H/V scanning, SS2D, cross scanning, local/windowed selective scanning, zigzag, spiral, radial, Hilbert curve, plain continuous 2D scanning, and atrous/skipping variants. Representative models include Vision Mamba (Vim) with bidirectional scanning and position embeddings, VMamba with SS2D, LocalMamba, EfficientVMamba, PlainMamba, and video models such as VideoMamba, VideoMambaPro, and SSSMLV (Rahman et al., 2024).

The survey emphasizes that these models preserve Mamba’s linear-time scan while adapting it to 2D or video structure. Reported benchmarks include Heracles-C-L at 85.9% Top-1 on ImageNet-1K, VideoMambaPro-M at 90.3% Top-1 and 98.5% Top-5 on Kinetics-400, within 0.6% of TubeViT-H while using ~89% fewer parameters and ~74% fewer FLOPs, and competitive results from VMamba-S/T and LocalVMamba-S on COCO and ADE20K. The same survey also notes open issues: scan anisotropy, stability/expressivity trade-offs, limited large pretrained backbones, reduced interpretability, and adversarial robustness concerns for VMamba (Rahman et al., 2024).

A related but distinct speech usage appears in streaming ASR, where the term Mamba-SL is not introduced, yet the architecture aligns with a “selective-linear” reading (Fang et al., 2024). The system uses a causal Mamba encoder, a controllable convolutional lookahead with

$$x_t = \bar{A}x_{t-1} + \bar{B}u_t, \qquad y_t = Cx_t + Du_t.$$0

a streaming-style Unimodal Aggregation (UMA) module with

$$x_t = \bar{A}x_{t-1} + \bar{B}u_t, \qquad y_t = Cx_t + Du_t.$$1

and an Early Termination (ET) mechanism for lower latency. On AISHELL-1, Mamba UMA achieves CER 6.59% with FT 281 ms, LT 327 ms, Avg 271 ms at 0 lookahead, and CER 5.55% with FT 605 ms, LT 453 ms, Avg 568 ms at $$x_t = \bar{A}x_{t-1} + \bar{B}u_t, \qquad y_t = Cx_t + Du_t.$$2 ms; ET reduces average latency from 271 ms to 196 ms with CER 6.82%. On AISHELL-2, Mamba UMA yields CER 7.02% with Avg 281 ms at 0 lookahead, and CER 6.08% with Avg 764 ms at $$x_t = \bar{A}x_{t-1} + \bar{B}u_t, \qquad y_t = Cx_t + Du_t.$$3 ms (Fang et al., 2024). This paper extends the broader interpretation of Mamba-SL toward online sequence labeling and emission control.

7. Limitations, misconceptions, and open directions

A recurrent misconception is that Mamba-SL names a single canonical architecture. The published evidence indicates otherwise. In point clouds it denotes staged sequence learning; in speech SSL it describes a HuBERT-style Mamba backbone; in diarization it corresponds to a Mamba local EEND segmentation model; in vision and streaming ASR it is, at most, an inferred shorthand for selective linear scanning (Wang et al., 2024, Lin et al., 14 Jun 2025, Plaquet et al., 2024, Rahman et al., 2024, Fang et al., 2024).

Another misconception is that linear-time scanning by itself guarantees uniform superiority over Transformers or RNNs. The cross-domain record is more nuanced. In speech SSL, ExtBiMamba at base scale lags the best bidirectional Transformer on SUPERB despite its advantages in causal settings and long-context ASR (Lin et al., 14 Jun 2025). In diarization, powerset loss can improve local DER yet harm the downstream pipeline because higher speaker confusion contaminates embedding extraction and clustering (Plaquet et al., 2024). In point clouds, bidirectional Mamba substantially helps the tiny model but not the base model, suggesting that serialization diversity and stage-wise pooling already provide much of the long-range interaction (Wang et al., 2024). In vision, the survey highlights scan anisotropy, hidden-state accumulation, limited pretraining resources, interpretability difficulties, and robustness degradation with scale (Rahman et al., 2024).

The open research agenda is correspondingly heterogeneous. For point clouds, the stated directions include scaling to very large scenes, improving robustness to noise and extreme sparsity, extending to multimodal inputs, and introducing more explicit cross-subsequence communication (Wang et al., 2024). For speech SSL, future work centers on scaling bidirectional Mamba, richer streaming setups, larger and multilingual pretraining, and deeper analysis of unit quality (Lin et al., 14 Jun 2025). For diarization, unresolved issues include rapid speaker turns, heavy overlap, joint training with the embedding extractor, multilingual or domain-robust pretraining, and better overlap-aware losses (Plaquet et al., 2024). For streaming ASR, the principal trade-off remains between lookahead-induced accuracy gains and emission latency, while ET can reduce latency at the cost of modest CER increases (Fang et al., 2024).

Taken together, these works suggest that Mamba-SL is best viewed as a research pattern: a selective state-space backbone combined with task-specific serialization, masking, labeling, or emission mechanisms to exploit Mamba’s linear-time scan without discarding domain structure. Its importance lies less in a fixed definition than in the convergent design principle it represents across 3D perception, speech representation learning, diarization, vision, and streaming recognition.