---
title: Unified Spatio-Temporal State Space Model
url: https://www.emergentmind.com/topics/unified-spatio-temporal-state-space-model-ust-ssm
type: topic
---

# Unified Spatio-Temporal State Space Model

Unified Spatio-Temporal State Space Model (UST-SSM) most directly denotes the point cloud video backbone introduced in “UST-SSM: Unified Spatio-Temporal State Space Models for Point Cloud Video Modeling,” where selective state space models are adapted to unordered 4D point sequences through Temporal Interaction Sampling (TIS), Spatio-Temporal Selection Scanning (STSS), and Spatio-Temporal Structure Aggregation (STSA) [2508.14604]. In the surrounding literature, the term also functions as a broader research lens for SSM-based architectures that seek a single backbone, block, or training recipe that remains effective across spatial structure and temporal evolution, even when that unification is achieved by tokenization, scan design, local aggregation, or flexible training rather than by a single native multidimensional state equation [2407.08476].

## 1. Definition and scope

The strictest reading of UST-SSM is architectural: a model should treat space and time under one coherent state-space mechanism rather than as fully separate modules. The point cloud UST-SSM paper is the clearest direct exemplar because it is explicitly named as such and because its stated goal is to make selective SSMs suitable for point cloud videos, whose “spatio-temporal disorder” prevents effective unidirectional modeling under naive 1D unfolding [2508.14604]. In that formulation, a point cloud video is a sequence of dynamic 3D point sets,
$$
\mathbf{X}=\{T_1,T_2,\dots,T_n\},
$$
and the central problem is that unordered points, irregular frame-to-frame correspondences, and nonuniform 4D geometry make simple temporally sequential scanning a poor interface for a sequence model.

A broader, but still evidence-based, interpretation emerges across adjacent work. “VideoMamba” unifies space and time by flattening spatio-temporal tubelet tokens into a single sequence and applying bidirectional selective scans [2407.08476]. “PoseMamba” unifies them architecturally within one block by combining forward and backward spatial and temporal scans over reordered human-joint sequences [2408.03540]. “StretchySnake” is explicitly described as strongest on the training/inference flexibility side rather than on proposing a new intrinsically unified space-time SSM operator, which places it at the practical end of the UST-SSM spectrum rather than the equation-level end [2510.16209]. This suggests that the phrase “UST-SSM” is used in contemporary practice in two related senses: a named point cloud video model, and a family resemblance across SSM systems that attempt unified operation over spatial and temporal structure.

## 2. State-space foundations and what “unified” means

Across the cited works, the mathematical core is the standard continuous-time state space model,
$$
h'(t)=Ah(t)+Bx(t), \qquad y(t)=Ch(t),
$$
or the closely related form with an explicit skip term,
$$
\frac{dh(t)}{dt}=Ah(t)+Bx(t), \qquad y(t)=Ch(t)+Dx(t).
$$
With zero-order hold discretization, several papers write
$$
\bar A=\exp(\Delta A), \qquad
\bar B=(\Delta A)^{-1}(\exp(\Delta A)-I)\cdot \Delta B,
$$
followed by the recurrence
$$
h_t=\bar A h_{t-1}+\bar B x_t, \qquad y_t=\bar C h_t,
$$
or the equivalent convolutional form built from the discrete kernel [2510.16209][2407.08476][2409.16953][2601.14799]. In Mamba-style selective SSMs, the crucial modification is that \(B\), \(C\), and often \(\Delta\) become input-dependent, so the scan is no longer a fixed linear time-invariant operator but a selective, content-aware state transition [2407.08476][2408.03540][2409.16953].

Within that common foundation, the literature realizes “unification” in several distinct ways. In “VideoMamba,” space and time are unified by tubelet tokenization, flattening to a 1D sequence, and a Spatio-Temporal Forward and Backward SSM; this is sequence-unified, but not a native 3D state equation over \((t,h,w)\) [2407.08476]. In “PoseMamba,” unification is block-level and multi-branch: the same \(T\times J\) feature tensor is reorganized into spatial and temporal scan sequences, each in forward and backward directions, and the branch outputs are merged; this is more integrated than alternating spatial and temporal modules, but still not a single inseparable spatio-temporal recurrence [2408.03540]. In “StretchySnake,” the SSM recurrence itself is unchanged and the main contribution is a flexible-scale training paradigm for VideoMamba, which means its unification is deployment-oriented rather than operator-oriented [2510.16209]. A recurring misconception is therefore that every SSM-based video model with both spatial and temporal inputs is already a strict UST-SSM. The source literature does not support that claim. Several models are explicitly described as only partially unified because they keep separate spatial and temporal embeddings, separate scan orderings, or size-dependent interpolation at the input side rather than redefining the underlying state law [2510.16209][2408.03540].

## 3. Canonical formulation: UST-SSM for point cloud video

The explicit UST-SSM architecture for point cloud video modeling is organized around three modules: Temporal Interaction Sampling, Spatio-Temporal Selection Scanning, and Spatio-Temporal Structure Aggregation [2508.14604]. Its input point cloud sequence is represented as
$$
\mathbf{X}\in\mathbb{R}^{T\times N\times 3}, \qquad
\mathbf{Feat}\in\mathbb{R}^{T\times N\times C},
$$
and its central claim is that selective SSMs become effective only after the 4D data are reorganized into a sequence whose order respects semantic and geometric structure rather than naive framewise concatenation.

TIS addresses temporal sparsity introduced by standard stride-based downsampling. The paper contrasts standard single-stride sampling,
$$
\mathbf{Feat}_{T_{2i}}=\mathcal{F}\big(\mathcal{S}(T_{2i-1},T_{2i},T_{2i+1})\big),
$$
with a two-stage scheme that first performs stride-1 temporal sampling,
$$
\mathbf{Feat}'_{T_i}=\mathcal{F}\big(\mathcal{S}(T_{i-1},T_i,T_{i+1})\big),
$$
and then performs a stride-2 update over features that already encode local temporal neighborhoods. The stated effect is to use non-anchor frames and enlarge temporal receptive fields.

STSS is the sequence-construction mechanism. A prompt network produces
$$
\mathcal{P}=\mathrm{PromptNetwork}(\mathbf{Feat})\in\mathbb{R}^{N\times K},
$$
points are assigned to semantic clusters by \(\arg\max(p_i)\), each cluster is sorted by a 3D Hilbert curve,
$$
\widehat{\mathbf{X}_j},\widehat{\mathbf{Feat}_j}
=\mathrm{HilbertSort}(\mathbf{X}_j,\mathbf{Feat}_j),
$$
then chronologically ordered,
$$
\overline{\mathbf{X}_j},\overline{\mathbf{Feat}_j}
=\mathrm{ChronologicalOrdering}(\widehat{\mathbf{X}_j},\widehat{\mathbf{Feat}_j}),
$$
and finally concatenated:
$$
\overline{\mathbf{X}}=\bigcup_{j=1}^{k}\overline{\mathbf{X}_j}, \qquad
\overline{\mathbf{Feat}}=\bigcup_{j=1}^{k}\overline{\mathbf{Feat}_j}.
$$
The resulting serialized sequence has length \(L=T\times N\). The paper’s stated rationale is that semantically similar points that are spatially or temporally distant in the original data can be brought closer in scan order, reducing the long-range attenuation that harms unidirectional SSMs.

STSA then compensates for local 4D structure lost by serialization. Neighborhoods are formed with 4D KNN using both coordinates and temporal embeddings,
$$
K=\underset{\mathbf{X}_K\in X}{\arg\min}\left(|\mathbf{X}_C-\mathbf{X}_K|+|\mathbf{E}_C^t-\mathbf{E}_K^t|\right),
$$
and relative-plus-absolute features are assembled as
$$
\mathbf{F}_K'=
\frac{\mathbf{F}_K-\mathbf{F}_C}{\|\mathbf{F}_K-\mathbf{F}_C\|_2+\epsilon}
\oplus \mathbf{F}_C
\in\mathbb{R}^{L\times k\times 2C}.
$$
Adaptive aggregation then produces the refined center feature. The paper’s interpretation is explicit: ST-SSM handles global sequential dependencies, whereas STSA restores local 4D geometric and motion detail that the scan cannot preserve by itself.

Empirically, the model is validated on MSR-Action3D, NTU RGB+D, and Synthia 4D. On MSR-Action3D, UST-SSM reaches \(95.12\%\) with 36 frames, compared with \(93.23\%\) for MAMBA4D at 36 frames. On NTU RGB+D it reports \(92.1\) on cross-subject and \(97.8\) on cross-view. In the efficiency comparison on MSR-Action3D with 36 frames, UST-SSM is reported at \(0.138\)M parameters, \(2.403\) GB memory, \(2.824\) hours, and \(95.12\%\) accuracy, compared with PST-Transformer at \(0.289\)M, \(4.072\) GB, \(2.995\) hours, and \(91.15\%\) [2508.14604].

## 4. Recurring architectural patterns across related models

Across neighboring papers, several recurring design patterns appear. The table summarizes the most explicit ones.

| Model | Domain | Stated unification style |
|---|---|---|
| UST-SSM [2508.14604] | Point cloud video | Semantic-aware serialization + local 4D aggregation |
| VideoMamba [2407.08476] | Video recognition | Flattened spatio-temporal token stream + forward/backward SSM |
| PoseMamba [2408.03540] | 3D human pose estimation | Bidirectional global-local spatial scan + temporal scan in one block |
| StretchySnake [2510.16209] | Video action recognition | Flexible-scale VideoMamba training and inference |
| PAST-SSM [2409.16953] | Event recognition | Adaptive event-to-token front end + B-Mamba sequence modeling |
| UBATrack [2601.14799] | RGB-T/RGB-D/RGB-E tracking | Mamba adapter over multimodal spatio-temporal tokens |
| \(\Delta t\)-Mamba3D [2510.19003] | Longitudinal mammography | Time-aware selective scan + local 3D fusion |
| S\(^2\)M\(^2\)ECG [2509.03066] | 12-lead ECG | Per-lead bidirectional SSMs + lead fusion |
| STSSM optical flow [2506.07878] | Event optical flow | Spatio-temporal cuboid tokenization + SSM + reprojection |
| TrackSSM [2409.00487] | Multi-object tracking motion | Temporal SSM over spatial trajectory states |
| USTEP [2310.05829] | Predictive video learning | Micro/macro recurrent state evolution |
| Time-SSM [2405.16312] | Time-series forecasting | Dynamic spectral temporal operator |

A first pattern is sequence unification through tokenization. “VideoMamba” converts a video tensor \(V\in\mathbb{R}^{T\times H\times W\times C}\) into tubelet tokens, flattens the token grid of size \(n_tn_hn_w\), and applies forward and backward selective scans; “Spatio-Temporal State Space Model For Efficient Event-Based Optical Flow” does something closely analogous for event volumes by patching a 4D tensor \(\mathcal{I}\in\mathbb{R}^{C\times T\times H\times W}\), projecting 3D patches to tokens, processing them with Mamba, and reprojecting to a spatio-temporal volume [2407.08476][2506.07878]. This suggests a common recipe: represent local space-time structure at token level, then let a 1D SSM process that token stream.

A second pattern is multi-order scan fusion. “PoseMamba” is the clearest example: it keeps the Mamba/S6 recurrence but changes sequence construction, introducing a bidirectional global-local spatio-temporal Mamba block in which spatial scan order is redesigned to better follow body geometry while temporal scan remains forward/backward over time [2408.03540]. “VideoMamba” also shows that the reverse scan order is a first-order design choice: full spatio-temporal reversal outperforms reversing only space or only time [2407.08476].

A third pattern is explicit local compensation around the SSM. UST-SSM adds STSA to recover 4D neighborhoods after serialization [2508.14604]. \(\Delta t\)-Mamba3D adds multiscale depth-wise Conv3D over the reshaped \(d\times T\times H\times W\) latent tensor [2510.19003]. UBATrack combines its Spatio-Temporal Mamba Adapter with MultiFFT and a Dynamic Multi-modal Feature Mixer, indicating that sequence-state modeling alone is often paired with an auxiliary local or channel-mixing operator [2601.14799].

A fourth pattern is factorized rather than strict unification. PAST-SSM handles arbitrary-duration event streams by adaptive temporal tokenization in PEAS, then feeds selected event-frame patch tokens to stacked B-Mamba blocks [2409.16953]. S\(^2\)M\(^2\)ECG uses synchronized per-lead temporal segment tokens, bidirectional intra-lead SSMs, and a separate lead-fusion module [2509.03066]. TrackSSM models each object’s box state temporally with an encoder-decoder SSM but does not introduce dense scene-level spatial coupling [2409.00487]. These cases remain central to the UST-SSM discussion because the source texts explicitly frame them as partially unified or as useful design patterns, even when they stop short of a single space-time state law.

## 5. Domains, tasks, and empirical evidence

The literature now spans video action recognition, 3D human pose estimation, event-based recognition, multimodal tracking, multi-lead ECG classification, longitudinal medical imaging, event-based optical flow, and classical time-series forecasting. The empirical record is correspondingly heterogeneous.

In generic video understanding, “StretchySnake” reports gains across short-action and long-action benchmarks after flexible VideoMamba training. The paper states average gains over VideoMamba of \(+19.1\%\), \(+18.4\%\), \(+22.1\%\), and \(+24.8\%\) on Breakfast for \(f8,f16,f32,f64\), and best scores of \(60.2\) on Breakfast, \(80.0\) on COIN, \(94.5\) on UCF-101, and \(66.1\) on HMDB-51; it also reports \(54.7\) average retrieval across six datasets, above Hiera-L at \(50.2\) and CAST at \(51.6\) [2510.16209]. “VideoMamba” itself reports \(77.7\) top-1 on Kinetics-400 with 32 frames at 68 GFLOPs and \(64.2\) top-1 on Something-Something V2 with 32 frames, while showing that spatio-temporal reversal is the best backward scan design [2407.08476].

In articulated motion, “PoseMamba” achieves Human3.6M P1/P2 of \(38.1/32.5\) mm for PoseMamba-L, \(40.8/34.3\) for PoseMamba-B, and \(41.8/35.0\) for PoseMamba-S, and reports \(14.51\) mm on MPI-INF-3DHP with PoseMamba-B at \(T=81\). Its ablation isolates the scan design: unidirectional variants range from \(43.0\) to \(43.8\) mm MPJPE, bidirectional spatio-temporal gives \(42.4\) mm, and bidirectional global-local spatio-temporal gives \(41.8\) mm [2408.03540].

In event-based recognition, “PAST-SSM” emphasizes arbitrary duration and inference-frequency robustness rather than a new native multidimensional state operator. It reports \(98.28\) on DVS Action, \(66.38\) on SeAct, \(98.41\) on HARDVS, \(97.35\) on ArDVS100, \(100.00\) on Real-ArDVS10, and \(89.00\) on TemArDVS100. On DVS Action under train/test frequency mismatch, Event Counts baseline trained at 20 Hz drops from \(94.83\) to \(70.69\) at 100 Hz, whereas Event Counts + PAST-SSM-S drops from \(93.10\) to \(84.48\) [2409.16953]. In event-based optical flow, the STSSM network reports EPE \(1.11\), AE \(4.29\), 1PE \(26.4\), 3PE \(5.0\), and 32 GMACs on DSEC, compared with EV-FlowNet at EPE \(2.32\) and 62 GMACs, and TMA at EPE \(0.74\) and 522 GMACs; the paper states 4.5× faster inference and 8× lower computations than TMA [2506.07878].

In multimodal tracking, “UBATrack” reports strong results across RGB-T, RGB-D, and RGB-E. On LasHeR, UBATrack-384 gives SR \(60.1\), PR \(76.0\), NPR \(72.2\); on DepthTrack it gives \(67.3/66.9/67.7\) for F-score/Recall/Precision; on VOT-RGBD22 it gives \(77.8/82.5/93.7\) for EAO/Accuracy/Robustness; on VisEvent it gives MPR/MSR \(79.7/62.7\). The model is trained with a frozen OSTrack backbone and only STMA and DMFM are fine-tuned [2601.14799].

In biomedical sensing, “S\(^2\)M\(^2\)ECG” reports Chapman \(Acc=0.928, F_1=0.918, AUC=0.981\), PTB-XL \(Acc=0.757, F_1=0.637, AUC=0.883\), and SNPH \(Acc=0.913, F_1=0.921, AUC=0.985\), while using \(0.705\)M parameters and latency between \(1.049\) ms and \(10.571\) ms on a 10-second ECG at 250 Hz [2509.03066]. In longitudinal imaging, “\(\Delta t\)-Mamba3D” reports c-index \(0.742\) on CSAW and \(0.738\) on an independent dataset, together with higher 1–5 year AUC scores than recurrent, transformer, and prior state-space variants; its single-layer efficiency table reports 1.8M params, 0.32 GFLOPs, and 59.3M tokens/s for \(\Delta t\)-Mamba3D versus 7.1M params and 4.0 GFLOPs for a transformer [2510.19003].

A different but related strand is purely temporal forecasting. “Time-SSM” does not model space, but it reframes SSMs as Dynamic Spectral Operators and introduces Time-SSM with only one-seventh of the parameters compared to Mamba, while outperforming Mamba4TS across the eight-dataset main table average. “USTEP” likewise is not an SSM paper, but its micro/macro segment view offers a multi-timescale stateful perspective that can be mapped onto hierarchical spatio-temporal dynamics [2405.16312][2310.05829]. A plausible implication is that UST-SSM research now draws from both directions: explicitly spatio-temporal SSM backbones and temporally focused operator/state abstractions that can be extended to space-time settings.

## 6. Limitations, misconceptions, and open directions

The strongest repeated limitation is that many models are unified only in a qualified sense. “StretchySnake” is explicit that it is “not a new fundamentally unified spatio-temporal state space architecture,” but rather a flexible training framework for a VideoMamba backbone; it keeps separate spatial and temporal positional embeddings and does not redefine \(A,B,C,\Delta\) to be scale-aware [2510.16209]. “PoseMamba” is also explicit that its unification is architectural rather than equation-level: multiple 1D selective scans over different orderings of the same \(T\times J\) tensor are fused within a block, but there is no single inseparable spatio-temporal recurrence [2408.03540]. “PAST-SSM,” “S\(^2\)M\(^2\)ECG,” and the event optical-flow STSSM all preserve this pattern: spatial structure is injected by frame construction, patching, reordering, or branch organization, then flattened or projected into a sequence SSM, rather than being governed by one native multidimensional state operator [2409.16953][2509.03066][2506.07878].

A second limitation is dependence on serialization, scan topology, or factorized frontend design. UST-SSM for point cloud video exists precisely because naive serialization is inadequate; STSS is a learned repair to that mismatch, not the elimination of the mismatch itself [2508.14604]. VideoMamba’s performance depends strongly on reverse scan design [2407.08476]. TrackSSM remains object-centric and models each trajectory independently without explicit cross-object spatial coupling [2409.00487]. UBATrack serializes multimodal patch tokens into Mamba adapters, but the paper does not provide a generic spatio-temporal factorization theory or missing-modality formalism [2601.14799]. This suggests that scan topology, token order, and latent neighborhood reconstruction are first-class design choices in current UST-SSM practice.

A third limitation is incomplete treatment of irregular time and continuous dynamics. \(\Delta t\)-Mamba3D is the clearest irregular-time exemplar because it modifies the step size as
$$
\delta_i^{\mathrm{TA}}=\delta_i\left(1+\gamma\frac{\Delta t(i)}{\tau_{\min}}\right),
$$
which then changes the exact discretization of the transition. Yet the paper’s own synthesis makes clear that this is best understood as a semi-discrete irregular-time SSM, not a full latent ODE/CDE formulation [2510.19003]. Time-SSM provides a strong temporal operator interpretation and argues that structured \(A\) should remain constrained rather than fully selected from data, but it does not address spatial interaction directly [2405.16312]. USTEP provides a compelling micro/macro temporal decomposition, but not explicit \(A,B,C,D\)-style spatio-temporal state equations [2310.05829].

Open directions follow directly from these limits. The source literature repeatedly identifies the need for stronger native multidimensional state evolution, content-adaptive or topology-aware scan operators, more explicit spatial coupling inside the state transition rather than in post-hoc fusion, and principled handling of irregular temporal spacing beyond fixed or scaled discretization [2508.14604][2510.19003][2409.16953]. A plausible implication is that the next stage of UST-SSM research will be defined less by proving that SSMs can process spatial and temporal data at all, and more by determining how much of that unification should reside in tokenization, scan order, local fusion, structured discretization, or a genuinely joint space-time state law.

Source: https://www.emergentmind.com/topics/unified-spatio-temporal-state-space-model-ust-ssm