Unified Spatio-Temporal State Space Model

• UST-SSM is a unified model that integrates spatial and temporal data into a single state-space mechanism for point cloud video analysis.
• It employs Temporal Interaction Sampling, Spatio-Temporal Selection Scanning, and Spatio-Temporal Structure Aggregation to effectively serialize unordered 4D point sequences.
• Empirical results on benchmarks such as MSR-Action3D and NTU RGB+D demonstrate its superior efficiency with fewer parameters and enhanced accuracy compared to conventional methods.

Unified Spatio-Temporal State Space Model (UST-SSM) most directly denotes the point cloud video backbone introduced in “UST-SSM: Unified Spatio-Temporal State Space Models for Point Cloud Video Modeling,” where selective state space models are adapted to unordered 4D point sequences through Temporal Interaction Sampling (TIS), Spatio-Temporal Selection Scanning (STSS), and Spatio-Temporal Structure Aggregation (STSA) (Li et al., 20 Aug 2025). In the surrounding literature, the term also functions as a broader research lens for SSM-based architectures that seek a single backbone, block, or training recipe that remains effective across spatial structure and temporal evolution, even when that unification is achieved by tokenization, scan design, local aggregation, or flexible training rather than by a single native multidimensional state equation (Park et al., 2024).

1. Definition and scope

The strictest reading of UST-SSM is architectural: a model should treat space and time under one coherent state-space mechanism rather than as fully separate modules. The point cloud UST-SSM paper is the clearest direct exemplar because it is explicitly named as such and because its stated goal is to make selective SSMs suitable for point cloud videos, whose “spatio-temporal disorder” prevents effective unidirectional modeling under naive 1D unfolding (Li et al., 20 Aug 2025). In that formulation, a point cloud video is a sequence of dynamic 3D point sets,

$\mathbf{X}=\{T_1,T_2,\dots,T_n\},$

and the central problem is that unordered points, irregular frame-to-frame correspondences, and nonuniform 4D geometry make simple temporally sequential scanning a poor interface for a sequence model.

A broader, but still evidence-based, interpretation emerges across adjacent work. “VideoMamba” unifies space and time by flattening spatio-temporal tubelet tokens into a single sequence and applying bidirectional selective scans (Park et al., 2024). “PoseMamba” unifies them architecturally within one block by combining forward and backward spatial and temporal scans over reordered human-joint sequences (Huang et al., 2024). “StretchySnake” is explicitly described as strongest on the training/inference flexibility side rather than on proposing a new intrinsically unified space-time SSM operator, which places it at the practical end of the UST-SSM spectrum rather than the equation-level end (Siddiqui et al., 17 Oct 2025). This suggests that the phrase “UST-SSM” is used in contemporary practice in two related senses: a named point cloud video model, and a family resemblance across SSM systems that attempt unified operation over spatial and temporal structure.

2. State-space foundations and what “unified” means

Across the cited works, the mathematical core is the standard continuous-time state space model,

$h'(t)=Ah(t)+Bx(t), \qquad y(t)=Ch(t),$

or the closely related form with an explicit skip term,

$$\frac{dh(t)}{dt}=Ah(t)+Bx(t), \qquad y(t)=Ch(t)+Dx(t).$$

With zero-order hold discretization, several papers write

$$\bar A=\exp(\Delta A), \qquad \bar B=(\Delta A)^{-1}(\exp(\Delta A)-I)\cdot \Delta B,$$

followed by the recurrence

$$h_t=\bar A h_{t-1}+\bar B x_t, \qquad y_t=\bar C h_t,$$

or the equivalent convolutional form built from the discrete kernel (Siddiqui et al., 17 Oct 2025, Park et al., 2024, Zhou et al., 2024, Liang et al., 21 Jan 2026). In Mamba-style selective SSMs, the crucial modification is that $B$, $C$, and often $\Delta$ become input-dependent, so the scan is no longer a fixed linear time-invariant operator but a selective, content-aware state transition (Park et al., 2024, Huang et al., 2024, Zhou et al., 2024).

Within that common foundation, the literature realizes “unification” in several distinct ways. In “VideoMamba,” space and time are unified by tubelet tokenization, flattening to a 1D sequence, and a Spatio-Temporal Forward and Backward SSM; this is sequence-unified, but not a native 3D state equation over $(t,h,w)$ (Park et al., 2024). In “PoseMamba,” unification is block-level and multi-branch: the same $T\times J$ feature tensor is reorganized into spatial and temporal scan sequences, each in forward and backward directions, and the branch outputs are merged; this is more integrated than alternating spatial and temporal modules, but still not a single inseparable spatio-temporal recurrence (Huang et al., 2024). In “StretchySnake,” the SSM recurrence itself is unchanged and the main contribution is a flexible-scale training paradigm for VideoMamba, which means its unification is deployment-oriented rather than operator-oriented (Siddiqui et al., 17 Oct 2025). A recurring misconception is therefore that every SSM-based video model with both spatial and temporal inputs is already a strict UST-SSM. The source literature does not support that claim. Several models are explicitly described as only partially unified because they keep separate spatial and temporal embeddings, separate scan orderings, or size-dependent interpolation at the input side rather than redefining the underlying state law (Siddiqui et al., 17 Oct 2025, Huang et al., 2024).

3. Canonical formulation: UST-SSM for point cloud video

The explicit UST-SSM architecture for point cloud video modeling is organized around three modules: Temporal Interaction Sampling, Spatio-Temporal Selection Scanning, and Spatio-Temporal Structure Aggregation (Li et al., 20 Aug 2025). Its input point cloud sequence is represented as

$h'(t)=Ah(t)+Bx(t), \qquad y(t)=Ch(t),$0

and its central claim is that selective SSMs become effective only after the 4D data are reorganized into a sequence whose order respects semantic and geometric structure rather than naive framewise concatenation.

TIS addresses temporal sparsity introduced by standard stride-based downsampling. The paper contrasts standard single-stride sampling,

$h'(t)=Ah(t)+Bx(t), \qquad y(t)=Ch(t),$1

with a two-stage scheme that first performs stride-1 temporal sampling,

$h'(t)=Ah(t)+Bx(t), \qquad y(t)=Ch(t),$2

and then performs a stride-2 update over features that already encode local temporal neighborhoods. The stated effect is to use non-anchor frames and enlarge temporal receptive fields.

STSS is the sequence-construction mechanism. A prompt network produces

$h'(t)=Ah(t)+Bx(t), \qquad y(t)=Ch(t),$3

points are assigned to semantic clusters by $h'(t)=Ah(t)+Bx(t), \qquad y(t)=Ch(t),$4, each cluster is sorted by a 3D Hilbert curve,

$h'(t)=Ah(t)+Bx(t), \qquad y(t)=Ch(t),$5

then chronologically ordered,

$h'(t)=Ah(t)+Bx(t), \qquad y(t)=Ch(t),$6

and finally concatenated:

$h'(t)=Ah(t)+Bx(t), \qquad y(t)=Ch(t),$7

The resulting serialized sequence has length $h'(t)=Ah(t)+Bx(t), \qquad y(t)=Ch(t),$8. The paper’s stated rationale is that semantically similar points that are spatially or temporally distant in the original data can be brought closer in scan order, reducing the long-range attenuation that harms unidirectional SSMs.

STSA then compensates for local 4D structure lost by serialization. Neighborhoods are formed with 4D KNN using both coordinates and temporal embeddings,

$h'(t)=Ah(t)+Bx(t), \qquad y(t)=Ch(t),$9

and relative-plus-absolute features are assembled as

$$\frac{dh(t)}{dt}=Ah(t)+Bx(t), \qquad y(t)=Ch(t)+Dx(t).$$0

Adaptive aggregation then produces the refined center feature. The paper’s interpretation is explicit: ST-SSM handles global sequential dependencies, whereas STSA restores local 4D geometric and motion detail that the scan cannot preserve by itself.

Empirically, the model is validated on MSR-Action3D, NTU RGB+D, and Synthia 4D. On MSR-Action3D, UST-SSM reaches $$\frac{dh(t)}{dt}=Ah(t)+Bx(t), \qquad y(t)=Ch(t)+Dx(t).$$1 with 36 frames, compared with $$\frac{dh(t)}{dt}=Ah(t)+Bx(t), \qquad y(t)=Ch(t)+Dx(t).$$2 for MAMBA4D at 36 frames. On NTU RGB+D it reports $$\frac{dh(t)}{dt}=Ah(t)+Bx(t), \qquad y(t)=Ch(t)+Dx(t).$$3 on cross-subject and $$\frac{dh(t)}{dt}=Ah(t)+Bx(t), \qquad y(t)=Ch(t)+Dx(t).$$4 on cross-view. In the efficiency comparison on MSR-Action3D with 36 frames, UST-SSM is reported at $$\frac{dh(t)}{dt}=Ah(t)+Bx(t), \qquad y(t)=Ch(t)+Dx(t).$$5M parameters, $$\frac{dh(t)}{dt}=Ah(t)+Bx(t), \qquad y(t)=Ch(t)+Dx(t).$$6 GB memory, $$\frac{dh(t)}{dt}=Ah(t)+Bx(t), \qquad y(t)=Ch(t)+Dx(t).$$7 hours, and $$\frac{dh(t)}{dt}=Ah(t)+Bx(t), \qquad y(t)=Ch(t)+Dx(t).$$8 accuracy, compared with PST-Transformer at $$\frac{dh(t)}{dt}=Ah(t)+Bx(t), \qquad y(t)=Ch(t)+Dx(t).$$9M, $$\bar A=\exp(\Delta A), \qquad \bar B=(\Delta A)^{-1}(\exp(\Delta A)-I)\cdot \Delta B,$$0 GB, $$\bar A=\exp(\Delta A), \qquad \bar B=(\Delta A)^{-1}(\exp(\Delta A)-I)\cdot \Delta B,$$1 hours, and $$\bar A=\exp(\Delta A), \qquad \bar B=(\Delta A)^{-1}(\exp(\Delta A)-I)\cdot \Delta B,$$2 (Li et al., 20 Aug 2025).

4. Recurring architectural patterns across related models

Across neighboring papers, several recurring design patterns appear. The table summarizes the most explicit ones.

Model	Domain	Stated unification style
UST-SSM (Li et al., 20 Aug 2025)	Point cloud video	Semantic-aware serialization + local 4D aggregation
VideoMamba (Park et al., 2024)	Video recognition	Flattened spatio-temporal token stream + forward/backward SSM
PoseMamba (Huang et al., 2024)	3D human pose estimation	Bidirectional global-local spatial scan + temporal scan in one block
StretchySnake (Siddiqui et al., 17 Oct 2025)	Video action recognition	Flexible-scale VideoMamba training and inference
PAST-SSM (Zhou et al., 2024)	Event recognition	Adaptive event-to-token front end + B-Mamba sequence modeling
UBATrack (Liang et al., 21 Jan 2026)	RGB-T/RGB-D/RGB-E tracking	Mamba adapter over multimodal spatio-temporal tokens
$$\bar A=\exp(\Delta A), \qquad \bar B=(\Delta A)^{-1}(\exp(\Delta A)-I)\cdot \Delta B,$$3-Mamba3D (Zhou et al., 21 Oct 2025)	Longitudinal mammography	Time-aware selective scan + local 3D fusion
S$$\bar A=\exp(\Delta A), \qquad \bar B=(\Delta A)^{-1}(\exp(\Delta A)-I)\cdot \Delta B,$$4M$$\bar A=\exp(\Delta A), \qquad \bar B=(\Delta A)^{-1}(\exp(\Delta A)-I)\cdot \Delta B,$$5ECG (Zhang et al., 3 Sep 2025)	12-lead ECG	Per-lead bidirectional SSMs + lead fusion
STSSM optical flow (Humais et al., 9 Jun 2025)	Event optical flow	Spatio-temporal cuboid tokenization + SSM + reprojection
TrackSSM (Hu et al., 2024)	Multi-object tracking motion	Temporal SSM over spatial trajectory states
USTEP (Tan et al., 2023)	Predictive video learning	Micro/macro recurrent state evolution
Time-SSM (Hu et al., 2024)	Time-series forecasting	Dynamic spectral temporal operator

A first pattern is sequence unification through tokenization. “VideoMamba” converts a video tensor $$\bar A=\exp(\Delta A), \qquad \bar B=(\Delta A)^{-1}(\exp(\Delta A)-I)\cdot \Delta B,$$6 into tubelet tokens, flattens the token grid of size $$\bar A=\exp(\Delta A), \qquad \bar B=(\Delta A)^{-1}(\exp(\Delta A)-I)\cdot \Delta B,$$7, and applies forward and backward selective scans; “Spatio-Temporal State Space Model For Efficient Event-Based Optical Flow” does something closely analogous for event volumes by patching a 4D tensor $$\bar A=\exp(\Delta A), \qquad \bar B=(\Delta A)^{-1}(\exp(\Delta A)-I)\cdot \Delta B,$$8, projecting 3D patches to tokens, processing them with Mamba, and reprojecting to a spatio-temporal volume (Park et al., 2024, Humais et al., 9 Jun 2025). This suggests a common recipe: represent local space-time structure at token level, then let a 1D SSM process that token stream.

A second pattern is multi-order scan fusion. “PoseMamba” is the clearest example: it keeps the Mamba/S6 recurrence but changes sequence construction, introducing a bidirectional global-local spatio-temporal Mamba block in which spatial scan order is redesigned to better follow body geometry while temporal scan remains forward/backward over time (Huang et al., 2024). “VideoMamba” also shows that the reverse scan order is a first-order design choice: full spatio-temporal reversal outperforms reversing only space or only time (Park et al., 2024).

A third pattern is explicit local compensation around the SSM. UST-SSM adds STSA to recover 4D neighborhoods after serialization (Li et al., 20 Aug 2025). $$\bar A=\exp(\Delta A), \qquad \bar B=(\Delta A)^{-1}(\exp(\Delta A)-I)\cdot \Delta B,$$9-Mamba3D adds multiscale depth-wise Conv3D over the reshaped $$h_t=\bar A h_{t-1}+\bar B x_t, \qquad y_t=\bar C h_t,$$0 latent tensor (Zhou et al., 21 Oct 2025). UBATrack combines its Spatio-Temporal Mamba Adapter with MultiFFT and a Dynamic Multi-modal Feature Mixer, indicating that sequence-state modeling alone is often paired with an auxiliary local or channel-mixing operator (Liang et al., 21 Jan 2026).

A fourth pattern is factorized rather than strict unification. PAST-SSM handles arbitrary-duration event streams by adaptive temporal tokenization in PEAS, then feeds selected event-frame patch tokens to stacked B-Mamba blocks (Zhou et al., 2024). S$$h_t=\bar A h_{t-1}+\bar B x_t, \qquad y_t=\bar C h_t,$$1M$$h_t=\bar A h_{t-1}+\bar B x_t, \qquad y_t=\bar C h_t,$$2ECG uses synchronized per-lead temporal segment tokens, bidirectional intra-lead SSMs, and a separate lead-fusion module (Zhang et al., 3 Sep 2025). TrackSSM models each object’s box state temporally with an encoder-decoder SSM but does not introduce dense scene-level spatial coupling (Hu et al., 2024). These cases remain central to the UST-SSM discussion because the source texts explicitly frame them as partially unified or as useful design patterns, even when they stop short of a single space-time state law.

5. Domains, tasks, and empirical evidence

The literature now spans video action recognition, 3D human pose estimation, event-based recognition, multimodal tracking, multi-lead ECG classification, longitudinal medical imaging, event-based optical flow, and classical time-series forecasting. The empirical record is correspondingly heterogeneous.

In generic video understanding, “StretchySnake” reports gains across short-action and long-action benchmarks after flexible VideoMamba training. The paper states average gains over VideoMamba of $$h_t=\bar A h_{t-1}+\bar B x_t, \qquad y_t=\bar C h_t,$$3, $$h_t=\bar A h_{t-1}+\bar B x_t, \qquad y_t=\bar C h_t,$$4, $$h_t=\bar A h_{t-1}+\bar B x_t, \qquad y_t=\bar C h_t,$$5, and $$h_t=\bar A h_{t-1}+\bar B x_t, \qquad y_t=\bar C h_t,$$6 on Breakfast for $$h_t=\bar A h_{t-1}+\bar B x_t, \qquad y_t=\bar C h_t,$$7, and best scores of $$h_t=\bar A h_{t-1}+\bar B x_t, \qquad y_t=\bar C h_t,$$8 on Breakfast, $$h_t=\bar A h_{t-1}+\bar B x_t, \qquad y_t=\bar C h_t,$$9 on COIN, $B$0 on UCF-101, and $B$1 on HMDB-51; it also reports $B$2 average retrieval across six datasets, above Hiera-L at $B$3 and CAST at $B$4 (Siddiqui et al., 17 Oct 2025). “VideoMamba” itself reports $B$5 top-1 on Kinetics-400 with 32 frames at 68 GFLOPs and $B$6 top-1 on Something-Something V2 with 32 frames, while showing that spatio-temporal reversal is the best backward scan design (Park et al., 2024).

In articulated motion, “PoseMamba” achieves Human3.6M P1/P2 of $B$7 mm for PoseMamba-L, $B$8 for PoseMamba-B, and $B$9 for PoseMamba-S, and reports $C$0 mm on MPI-INF-3DHP with PoseMamba-B at $C$1. Its ablation isolates the scan design: unidirectional variants range from $C$2 to $C$3 mm MPJPE, bidirectional spatio-temporal gives $C$4 mm, and bidirectional global-local spatio-temporal gives $C$5 mm (Huang et al., 2024).

In event-based recognition, “PAST-SSM” emphasizes arbitrary duration and inference-frequency robustness rather than a new native multidimensional state operator. It reports $C$6 on DVS Action, $C$7 on SeAct, $C$8 on HARDVS, $C$9 on ArDVS100, $\Delta$0 on Real-ArDVS10, and $\Delta$1 on TemArDVS100. On DVS Action under train/test frequency mismatch, Event Counts baseline trained at 20 Hz drops from $\Delta$2 to $\Delta$3 at 100 Hz, whereas Event Counts + PAST-SSM-S drops from $\Delta$4 to $\Delta$5 (Zhou et al., 2024). In event-based optical flow, the STSSM network reports EPE $\Delta$6, AE $\Delta$7, 1PE $\Delta$8, 3PE $\Delta$9, and 32 GMACs on DSEC, compared with EV-FlowNet at EPE $(t,h,w)$0 and 62 GMACs, and TMA at EPE $(t,h,w)$1 and 522 GMACs; the paper states 4.5× faster inference and 8× lower computations than TMA (Humais et al., 9 Jun 2025).

In multimodal tracking, “UBATrack” reports strong results across RGB-T, RGB-D, and RGB-E. On LasHeR, UBATrack-384 gives SR $(t,h,w)$2, PR $(t,h,w)$3, NPR $(t,h,w)$4; on DepthTrack it gives $(t,h,w)$5 for F-score/Recall/Precision; on VOT-RGBD22 it gives $(t,h,w)$6 for EAO/Accuracy/Robustness; on VisEvent it gives MPR/MSR $(t,h,w)$7. The model is trained with a frozen OSTrack backbone and only STMA and DMFM are fine-tuned (Liang et al., 21 Jan 2026).

In biomedical sensing, “S$(t,h,w)$8M$(t,h,w)$9ECG” reports Chapman $T\times J$0, PTB-XL $T\times J$1, and SNPH $T\times J$2, while using $T\times J$3M parameters and latency between $T\times J$4 ms and $T\times J$5 ms on a 10-second ECG at 250 Hz (Zhang et al., 3 Sep 2025). In longitudinal imaging, “$T\times J$6-Mamba3D” reports c-index $T\times J$7 on CSAW and $T\times J$8 on an independent dataset, together with higher 1–5 year AUC scores than recurrent, transformer, and prior state-space variants; its single-layer efficiency table reports 1.8M params, 0.32 GFLOPs, and 59.3M tokens/s for $T\times J$9-Mamba3D versus 7.1M params and 4.0 GFLOPs for a transformer (Zhou et al., 21 Oct 2025).

A different but related strand is purely temporal forecasting. “Time-SSM” does not model space, but it reframes SSMs as Dynamic Spectral Operators and introduces Time-SSM with only one-seventh of the parameters compared to Mamba, while outperforming Mamba4TS across the eight-dataset main table average. “USTEP” likewise is not an SSM paper, but its micro/macro segment view offers a multi-timescale stateful perspective that can be mapped onto hierarchical spatio-temporal dynamics (Hu et al., 2024, Tan et al., 2023). A plausible implication is that UST-SSM research now draws from both directions: explicitly spatio-temporal SSM backbones and temporally focused operator/state abstractions that can be extended to space-time settings.

6. Limitations, misconceptions, and open directions

The strongest repeated limitation is that many models are unified only in a qualified sense. “StretchySnake” is explicit that it is “not a new fundamentally unified spatio-temporal state space architecture,” but rather a flexible training framework for a VideoMamba backbone; it keeps separate spatial and temporal positional embeddings and does not redefine $h'(t)=Ah(t)+Bx(t), \qquad y(t)=Ch(t),$00 to be scale-aware (Siddiqui et al., 17 Oct 2025). “PoseMamba” is also explicit that its unification is architectural rather than equation-level: multiple 1D selective scans over different orderings of the same $h'(t)=Ah(t)+Bx(t), \qquad y(t)=Ch(t),$01 tensor are fused within a block, but there is no single inseparable spatio-temporal recurrence (Huang et al., 2024). “PAST-SSM,” “S$h'(t)=Ah(t)+Bx(t), \qquad y(t)=Ch(t),$02M$h'(t)=Ah(t)+Bx(t), \qquad y(t)=Ch(t),$03ECG,” and the event optical-flow STSSM all preserve this pattern: spatial structure is injected by frame construction, patching, reordering, or branch organization, then flattened or projected into a sequence SSM, rather than being governed by one native multidimensional state operator (Zhou et al., 2024, Zhang et al., 3 Sep 2025, Humais et al., 9 Jun 2025).

A second limitation is dependence on serialization, scan topology, or factorized frontend design. UST-SSM for point cloud video exists precisely because naive serialization is inadequate; STSS is a learned repair to that mismatch, not the elimination of the mismatch itself (Li et al., 20 Aug 2025). VideoMamba’s performance depends strongly on reverse scan design (Park et al., 2024). TrackSSM remains object-centric and models each trajectory independently without explicit cross-object spatial coupling (Hu et al., 2024). UBATrack serializes multimodal patch tokens into Mamba adapters, but the paper does not provide a generic spatio-temporal factorization theory or missing-modality formalism (Liang et al., 21 Jan 2026). This suggests that scan topology, token order, and latent neighborhood reconstruction are first-class design choices in current UST-SSM practice.

A third limitation is incomplete treatment of irregular time and continuous dynamics. $h'(t)=Ah(t)+Bx(t), \qquad y(t)=Ch(t),$04-Mamba3D is the clearest irregular-time exemplar because it modifies the step size as

$h'(t)=Ah(t)+Bx(t), \qquad y(t)=Ch(t),$05

which then changes the exact discretization of the transition. Yet the paper’s own synthesis makes clear that this is best understood as a semi-discrete irregular-time SSM, not a full latent ODE/CDE formulation (Zhou et al., 21 Oct 2025). Time-SSM provides a strong temporal operator interpretation and argues that structured $h'(t)=Ah(t)+Bx(t), \qquad y(t)=Ch(t),$06 should remain constrained rather than fully selected from data, but it does not address spatial interaction directly (Hu et al., 2024). USTEP provides a compelling micro/macro temporal decomposition, but not explicit $h'(t)=Ah(t)+Bx(t), \qquad y(t)=Ch(t),$07-style spatio-temporal state equations (Tan et al., 2023).

Open directions follow directly from these limits. The source literature repeatedly identifies the need for stronger native multidimensional state evolution, content-adaptive or topology-aware scan operators, more explicit spatial coupling inside the state transition rather than in post-hoc fusion, and principled handling of irregular temporal spacing beyond fixed or scaled discretization (Li et al., 20 Aug 2025, Zhou et al., 21 Oct 2025, Zhou et al., 2024). A plausible implication is that the next stage of UST-SSM research will be defined less by proving that SSMs can process spatial and temporal data at all, and more by determining how much of that unification should reside in tokenization, scan order, local fusion, structured discretization, or a genuinely joint space-time state law.