STSMamba: Spatial-Temporal-Spectral Architectures

• STSMamba is a family of Mamba-based architectures that explicitly organize spatial, temporal, and spectral information to enhance task-specific performance.
• Key variants include MambaST for cross-spectral pedestrian detection, a sparse deformable version for MODIS time series classification, and MCMamba for multichannel speech enhancement.
• Empirical studies show these architectures deliver improved efficiency and accuracy over traditional CNNs, Transformers, and LSTMs by leveraging structured tokenization and recurrence.

Spatial-Temporal-Spectral Mamba (STSMamba) denotes a family of Mamba-based architectures that extend state-space modeling to data in which spatial arrangement, temporal evolution, and spectral or modality structure must be processed jointly. In current arXiv usage, the term appears in at least three task-specific forms: as MambaST, a plug-and-play cross-spectral spatial-temporal fusion pipeline for RGB-thermal pedestrian detection (Gao et al., 2024); as a spatial-temporal-spectral Mamba with sparse deformable token sequence for MODIS time series classification (Dewis et al., 29 Jul 2025); and, in a related description, as MCMamba for multichannel speech enhancement, where Mamba blocks jointly model spatial and spectral information across microphones and time (Ren et al., 2024). The shared premise is that direct flattening of all dimensions into a single undifferentiated sequence is suboptimal; instead, STSMamba variants explicitly organize spatial, temporal, and spectral dependencies before or during state-space processing.

1. Terminological scope and family structure

The expression “Spatial-Temporal-Spectral Mamba” does not denote a single standardized network. The available literature uses it for distinct architectures that are unified by their reliance on Mamba blocks and by their explicit treatment of three structured axes: space, time, and spectrum or modality. In one case, “spectral” refers to RGB and thermal streams in cross-spectral perception; in another, it refers to multispectral MODIS bands; in a third, it refers to spectral representations of multichannel speech (Gao et al., 2024).

Work	Application	Characteristic modules
MambaST (Gao et al., 2024)	Cross-spectral pedestrian detection	MHHPA, OCF, recurrent temporal fusion
STSMamba (Dewis et al., 29 Jul 2025)	MODIS time series classification	TGS, SDMS, SDTM, SDSpeM, SDSpaM
MCMamba (Ren et al., 2024)	Multichannel speech enhancement	Full-band and narrow-band spatial modules; sub-band and full-band spectral modules

This distribution of meanings is important for interpretation. A common misconception is to treat STSMamba as a single plug-in block with fixed semantics. The published record instead shows a design pattern: Mamba is adapted so that structured tokenization, sequencing, and residual reconstruction align with the modality geometry of the task. A plausible implication is that “STSMamba” functions more as an architectural genre than as a single reproducible template.

2. State-space basis and recurrent modeling strategy

The pedestrian-detection formulation grounds STSMamba in the continuous-time linear time-invariant formulation of Mamba,

$$h'(t) \;=\; \mathbf{A}\,h(t)\;+\;\mathbf{B}\,x(t)\,, \qquad y(t)\;=\;\mathbf{W}\,h'(t)\,,$$

where $x(t)\in\mathbb{R}$, $h(t)\in\mathbb{R}^N$, $\mathbf{A}\in\mathbb{R}^{N\times N}$, $\mathbf{B}\in\mathbb{R}^{N\times1}$, and $\mathbf{W}\in\mathbb{R}^{1\times N}$ (Gao et al., 2024). In that setting, the scalar input is replaced by a concatenation of multi-scale patches from RGB and thermal feature maps at frame $t$, while the hidden state is carried forward in time through a simple recurrent connection.

The speech-enhancement formulation emphasizes the discrete-time structured state-space view,

$$h_n = \bar A\,h_{n-1} + \bar B\,x_n,\qquad y_n = C\,h_n,$$

and further describes a gated implementation,

$$Y' \;=\;\mathrm{SSM}\bigl(\mathrm{SiLU}(\mathrm{Conv}(\mathrm{Linear}(X)))\bigr) \odot \mathrm{SiLU}(\mathrm{Linear}(X)),\qquad Y \;=\;\mathrm{Linear}(Y'),$$

where the element-wise product implements input-selection gating (Ren et al., 2024). This makes explicit that STSMamba variants inherit not only recurrence, but also selective propagation and long-range dependency modeling from the underlying Mamba design.

The MODIS formulation pushes the idea further by inserting a sparse deformable sequencing stage before Mamba processing. For an input token matrix $U\in\mathbb R^{N\times d}$, it computes

$x(t)\in\mathbb{R}$0

then forms a shorter sparse sequence by ranking row-wise mean attention scores and keeping the top $x(t)\in\mathbb{R}$1 tokens before feeding them into a MambaBlock (Dewis et al., 29 Jul 2025). Here the state-space model is not merely a replacement for attention; it is coupled to a learned sparsification policy that changes token selection at every forward pass.

Across these versions, the recurring technical principle is structured sequencing rather than indiscriminate flattening. This suggests that the decisive design variable in STSMamba is often the construction of the sequence presented to the Mamba block, not the state-space kernel alone.

3. Cross-spectral pedestrian detection: MambaST

In pedestrian detection, STSMamba appears as MambaST, a plug-and-play cross-spectral spatial-temporal fusion pipeline for efficient pedestrian detection (Gao et al., 2024). The stated motivation is threefold: RGB cameras are unreliable under dark or low-light conditions; cross-spectral systems must integrate complementary information from thermal and visible cameras; and pedestrian detection is latency-sensitive, making efficient and easy-to-scale models with fewer parameters desirable. The architecture therefore couples spatial-temporal reasoning with RGB-thermal fusion.

Its core module is the Multi-head Hierarchical Patching and Aggregation (MHHPA) structure. For each frame $x(t)\in\mathbb{R}$2, a set of patch sizes $x(t)\in\mathbb{R}$3 is defined, typically $x(t)\in\mathbb{R}$4 for the first block and none for subsequent blocks. Given RGB and thermal feature maps $x(t)\in\mathbb{R}$5, non-overlapping patches of size $x(t)\in\mathbb{R}$6 are formed:

$x(t)\in\mathbb{R}$7

The paper distinguishes coarse-grained patches, which capture global context and suppress pixel-level noise, from fine-grained patches, which preserve high-resolution detail.

To preserve locality while fusing modalities, MambaST introduces Order-Aware Concatenation and Flattening (OCF). For patch size $x(t)\in\mathbb{R}$8,

$x(t)\in\mathbb{R}$9

with row-wise interleaving of RGB and thermal pixels and separate handling of even and odd columns (Gao et al., 2024). The projected sequence $h(t)\in\mathbb{R}^N$0 is then processed by $h(t)\in\mathbb{R}^N$1 cascaded MambaBlocks. The “multi-head” designation refers to the $h(t)\in\mathbb{R}^N$2 parallel patch scales; the paper states explicitly that no soft-attention is used.

Reconstruction is residual. The output $h(t)\in\mathbb{R}^N$3 is reshaped and split into $h(t)\in\mathbb{R}^N$4 and $h(t)\in\mathbb{R}^N$5, which are added back to the RGB and thermal patches, after which all scales are upsampled to $h(t)\in\mathbb{R}^N$6 and concatenated along the channel axis. Before patching, a learnable spectrum tag $h(t)\in\mathbb{R}^N$7 or $h(t)\in\mathbb{R}^N$8, together with positional embedding $h(t)\in\mathbb{R}^N$9, is added to each spatial feature. The paper describes this as spectral embeddings, joint tokenization through OCF, and residual fusion without explicit gating; cross-modal correction is learned implicitly by the recurrence (Gao et al., 2024).

Algorithm 1 provides full pseudocode. Architecturally, MambaST is therefore best understood as a cross-spectral sequence constructor around Mamba blocks, rather than as a direct transplant of sequence modeling from 1D language tasks.

4. MODIS time series classification: sparse deformable STSMamba

For MODIS time series classification, STSMamba is introduced as “Spatial-Temporal-Spectral Mamba with Sparse Deformable Token Sequence for Enhanced MODIS Time Series Classification” (Dewis et al., 29 Jul 2025). The input is a batch of MODIS time-series cubes

$\mathbf{A}\in\mathbb{R}^{N\times N}$0

with $\mathbf{A}\in\mathbb{R}^{N\times N}$1, $\mathbf{A}\in\mathbb{R}^{N\times N}$2, and spatial patch size $\mathbf{A}\in\mathbb{R}^{N\times N}$3. The model addresses high temporal dimensionality, mixed pixels, and spatial-temporal-spectral coupling effect.

The first stage is the Temporal Grouped Stem (TGS), designed to disentangle temporal-spectral feature coupling. The input is split into $\mathbf{A}\in\mathbb{R}^{N\times N}$4 groups $\mathbf{A}\in\mathbb{R}^{N\times N}$5, one per time step, and each group is processed by

$\mathbf{A}\in\mathbb{R}^{N\times N}$6

with $\mathbf{A}\in\mathbb{R}^{N\times N}$7. The outputs are restacked into

$\mathbf{A}\in\mathbb{R}^{N\times N}$8

The paper characterizes this as replacing a single giant convolution on all $\mathbf{A}\in\mathbb{R}^{N\times N}$9 channels by $\mathbf{B}\in\mathbb{R}^{N\times1}$0 parallel small-channel convolutions.

The main innovation is Sparse Deformable Mamba Sequencing (SDMS). Each of the three downstream modules follows the same sequence: reshape into tokens along one modality, compute a small modality-wise attention or similarity matrix, select the top $\mathbf{B}\in\mathbb{R}^{N\times1}$1-fraction of tokens, process the shorter sequence with a MambaBlock, and scatter outputs back as a residual (Dewis et al., 29 Jul 2025). The “deformable” terminology refers to the fact that the selected indices depend on the learned attention matrix and are updated at every forward pass. The paper notes that this is analogous to deformable attention in object detection, but the modality positions are reshuffled in a learnable way.

This sequencing policy underlies three explicit modules. Sparse Deformable Temporal Mamba (SDTM) reshapes $\mathbf{B}\in\mathbb{R}^{N\times1}$2 into $\mathbf{B}\in\mathbb{R}^{N\times1}$3 tokens, one per time step. Sparse Deformable Spectral Mamba (SDSpeM) treats $\mathbf{B}\in\mathbb{R}^{N\times1}$4 as a batch dimension and forms $\mathbf{B}\in\mathbb{R}^{N\times1}$5 spectral tokens of size $\mathbf{B}\in\mathbb{R}^{N\times1}$6 per sample. Sparse Deformable Spatial Mamba (SDSpaM) reshapes features into one token per pixel location with spectral length $\mathbf{B}\in\mathbb{R}^{N\times1}$7, then ranks locations by a cosine-angle similarity to the center pixel,

$\mathbf{B}\in\mathbb{R}^{N\times1}$8

before sparse Mamba processing (Dewis et al., 29 Jul 2025).

The paper states that this design disentangles the three modalities rather than “cramming all $\mathbf{B}\in\mathbb{R}^{N\times1}$9 into one long sequence.” A plausible implication is that the model treats sparsity and axis-specific factorization as coequal design goals: efficiency is obtained not only from Mamba’s linear-time properties, but also from aggressive reduction of the effective token count.

5. Multichannel speech enhancement: MCMamba as an STSMamba formulation

In multichannel speech enhancement, the related STSMamba formulation is described under the name MCMamba (Ren et al., 2024). The model processes a multichannel noisy STFT through four cascade modules: two for spatial modeling and two for spectral modeling. Each module is built around a Mamba block, and each block may be Uni-Mamba for real-time operation or Bi-Mamba for offline processing.

The spatial pathway is divided into full-band and narrow-band components. In the full-band spatial module, for each time $\mathbf{W}\in\mathbb{R}^{1\times N}$0 and frequency $\mathbf{W}\in\mathbb{R}^{1\times N}$1, the input is the real-imaginary vector across microphones,

$\mathbf{W}\in\mathbb{R}^{1\times N}$2

optionally augmented with inter-channel phase differences $\mathbf{W}\in\mathbb{R}^{1\times N}$3. This sequence is processed along the frequency axis. In the narrow-band spatial module, the model fixes frequency $\mathbf{W}\in\mathbb{R}^{1\times N}$4 and processes the time sequence $\mathbf{W}\in\mathbb{R}^{1\times N}$5 to capture how spatial cues evolve over time (Ren et al., 2024).

The spectral pathway is likewise split into sub-band and full-band components. The sub-band module operates on the magnitude of the reference channel, stacking $\mathbf{W}\in\mathbb{R}^{1\times N}$6 neighboring frequency bins with $\mathbf{W}\in\mathbb{R}^{1\times N}$7,

$\mathbf{W}\in\mathbb{R}^{1\times N}$8

and then running a Mamba block over time. The full-band spectral module aggregates $\mathbf{W}\in\mathbb{R}^{1\times N}$9 consecutive frames at each frequency with $t$0,

$t$1

again followed by Mamba processing and a fully connected layer (Ren et al., 2024).

After the fourth module, outputs are concatenated and passed through a final linear layer to predict a complex mask $t$2, which is applied to the reference-channel STFT:

$t$3

Training minimizes the mean-square error between the masked STFT and the clean STFT. The model thus instantiates spatial-temporal-spectral Mamba not through explicit multimodal token interleaving, as in MambaST, nor through deformable sparsification, as in the MODIS model, but through a cascade of axis-specific Mamba modules operating on STFT-derived spatial and spectral feature organizations.

6. Empirical profile, recurrent motifs, and interpretive issues

The reported empirical results differ by task and are therefore not directly comparable, but they establish a consistent pattern: STSMamba variants are presented as competitive or superior alternatives to Transformer-, CNN-, or LSTM-based baselines within their own domains.

For pedestrian detection on KAIST, MambaST reaches $t$4 LAMR in the “reasonable” setting, compared with $t$5 for vanilla CFT, $t$6 for T-CFT, and $t$7 for D-CFT; recall improves from $t$8 for D-CFT to $t$9. In the “reasonable small” setting, D-CFT records $$h_n = \bar A\,h_{n-1} + \bar B\,x_n,\qquad y_n = C\,h_n,$$0 LAMR and STSMamba records $$h_n = \bar A\,h_{n-1} + \bar B\,x_n,\qquad y_n = C\,h_n,$$1. The ablation study reports $$h_n = \bar A\,h_{n-1} + \bar B\,x_n,\qquad y_n = C\,h_n,$$2 median LAMR for single-scale patching $$h_n = \bar A\,h_{n-1} + \bar B\,x_n,\qquad y_n = C\,h_n,$$3, $$h_n = \bar A\,h_{n-1} + \bar B\,x_n,\qquad y_n = C\,h_n,$$4 for multi-scale patching $$h_n = \bar A\,h_{n-1} + \bar B\,x_n,\qquad y_n = C\,h_n,$$5 in block 1), $$h_n = \bar A\,h_{n-1} + \bar B\,x_n,\qquad y_n = C\,h_n,$$6 after inserting OCF, and $$h_n = \bar A\,h_{n-1} + \bar B\,x_n,\qquad y_n = C\,h_n,$$7 when increasing temporal frames $$h_n = \bar A\,h_{n-1} + \bar B\,x_n,\qquad y_n = C\,h_n,$$8 to $$h_n = \bar A\,h_{n-1} + \bar B\,x_n,\qquad y_n = C\,h_n,$$9. The same paper reports $$Y' \;=\;\mathrm{SSM}\bigl(\mathrm{SiLU}(\mathrm{Conv}(\mathrm{Linear}(X)))\bigr) \odot \mathrm{SiLU}(\mathrm{Linear}(X)),\qquad Y \;=\;\mathrm{Linear}(Y'),$$0 M parameters and $$Y' \;=\;\mathrm{SSM}\bigl(\mathrm{SiLU}(\mathrm{Conv}(\mathrm{Linear}(X)))\bigr) \odot \mathrm{SiLU}(\mathrm{Linear}(X)),\qquad Y \;=\;\mathrm{Linear}(Y'),$$1 GFLOPs for STSMamba, versus $$Y' \;=\;\mathrm{SSM}\bigl(\mathrm{SiLU}(\mathrm{Conv}(\mathrm{Linear}(X)))\bigr) \odot \mathrm{SiLU}(\mathrm{Linear}(X)),\qquad Y \;=\;\mathrm{Linear}(Y'),$$2 M and $$Y' \;=\;\mathrm{SSM}\bigl(\mathrm{SiLU}(\mathrm{Conv}(\mathrm{Linear}(X)))\bigr) \odot \mathrm{SiLU}(\mathrm{Linear}(X)),\qquad Y \;=\;\mathrm{Linear}(Y'),$$3 GFLOPs for CFT, and $$Y' \;=\;\mathrm{SSM}\bigl(\mathrm{SiLU}(\mathrm{Conv}(\mathrm{Linear}(X)))\bigr) \odot \mathrm{SiLU}(\mathrm{Linear}(X)),\qquad Y \;=\;\mathrm{Linear}(Y'),$$4 M and $$Y' \;=\;\mathrm{SSM}\bigl(\mathrm{SiLU}(\mathrm{Conv}(\mathrm{Linear}(X)))\bigr) \odot \mathrm{SiLU}(\mathrm{Linear}(X)),\qquad Y \;=\;\mathrm{Linear}(Y'),$$5 GFLOPs for D-CFT; inference latency is $$Y' \;=\;\mathrm{SSM}\bigl(\mathrm{SiLU}(\mathrm{Conv}(\mathrm{Linear}(X)))\bigr) \odot \mathrm{SiLU}(\mathrm{Linear}(X)),\qquad Y \;=\;\mathrm{Linear}(Y'),$$6 ms per frame, versus $$Y' \;=\;\mathrm{SSM}\bigl(\mathrm{SiLU}(\mathrm{Conv}(\mathrm{Linear}(X)))\bigr) \odot \mathrm{SiLU}(\mathrm{Linear}(X)),\qquad Y \;=\;\mathrm{Linear}(Y'),$$7 ms for CFT and $$Y' \;=\;\mathrm{SSM}\bigl(\mathrm{SiLU}(\mathrm{Conv}(\mathrm{Linear}(X)))\bigr) \odot \mathrm{SiLU}(\mathrm{Linear}(X)),\qquad Y \;=\;\mathrm{Linear}(Y'),$$8 ms for D-CFT (Gao et al., 2024).

For MODIS time series classification, STSMamba reports Saskatchewan results of OA $$Y' \;=\;\mathrm{SSM}\bigl(\mathrm{SiLU}(\mathrm{Conv}(\mathrm{Linear}(X)))\bigr) \odot \mathrm{SiLU}(\mathrm{Linear}(X)),\qquad Y \;=\;\mathrm{Linear}(Y'),$$9, AA $U\in\mathbb R^{N\times d}$0, and $U\in\mathbb R^{N\times d}$1, compared with the next best MambaHSI at OA $U\in\mathbb R^{N\times d}$2, AA $U\in\mathbb R^{N\times d}$3, and $U\in\mathbb R^{N\times d}$4. On Alberta cross-domain evaluation, it reports OA $U\in\mathbb R^{N\times d}$5, AA $U\in\mathbb R^{N\times d}$6, and $U\in\mathbb R^{N\times d}$7, compared with MambaHSI at OA $U\in\mathbb R^{N\times d}$8, AA $U\in\mathbb R^{N\times d}$9, and $x(t)\in\mathbb{R}$00. The best ablation setting uses $x(t)\in\mathbb{R}$01 and $x(t)\in\mathbb{R}$02 (Dewis et al., 29 Jul 2025).

For speech enhancement on CHiME-3, the non-causal Bi-Mamba version reports NB-PESQ $x(t)\in\mathbb{R}$03, WB-PESQ $x(t)\in\mathbb{R}$04, STOI $x(t)\in\mathbb{R}$05, and SDR $x(t)\in\mathbb{R}$06, compared with McNet at $x(t)\in\mathbb{R}$07, $x(t)\in\mathbb{R}$08, $x(t)\in\mathbb{R}$09, and $x(t)\in\mathbb{R}$10. The causal Uni-Mamba version records $x(t)\in\mathbb{R}$11, $x(t)\in\mathbb{R}$12, $x(t)\in\mathbb{R}$13, and $x(t)\in\mathbb{R}$14. An ablation study shows that replacing only the spectral modules with Mamba yields $x(t)\in\mathbb{R}$15, while replacing only the spatial modules yields $x(t)\in\mathbb{R}$16, which the paper interprets as evidence that Mamba performs exceptionally well in modeling spectral information (Ren et al., 2024).

Several interpretive clarifications follow directly from these reports. First, “multi-head” in MambaST should not be conflated with Transformer multi-head attention: the paper states that no soft-attention is used, and that the multi-head aspect arises from parallel patch scales (Gao et al., 2024). Second, “deformable” in the MODIS model refers to adaptive token selection driven by a learned attention matrix rather than geometric resampling (Dewis et al., 29 Jul 2025). Third, the literature uses the same umbrella term for architectures with materially different internal organizations. This suggests that the stable conceptual core of STSMamba is not one fixed block diagram, but the combination of Mamba-based recurrence with task-specific decomposition of spatial, temporal, and spectral structure.