MM-STT: Multi-Modal Spatio-Temporal Transformer

• MM-STT is a design pattern that integrates multiple modalities with unified spatio-temporal self-attention to jointly encode spatial, temporal, and cross-modal data.
• It employs modality-specific tokenization, learnable positional embeddings, and varied fusion strategies to capture long-range dependencies and enhance prediction accuracy.
• Empirical studies show MM-STT achieves significant performance gains in tasks like land subsidence forecasting and RGBT tracking compared to traditional CNN, LSTM, or segmented attention models.

Searching arXiv for the core paper and closely related MM-STT formulations to ground the article in current literature. A Multi-Modal Spatio-Temporal Transformer (MM-STT) denotes a class of architectures that model structured data distributed across space, time, and multiple information sources within a Transformer-centric framework. Across the literature represented here, MM-STT systems share a common objective: to encode long-range spatio-temporal dependencies while fusing heterogeneous modalities such as dynamic displacement fields and static physical priors, RGB and thermal imagery, audio and video, or graph-structured state trajectories with auxiliary context (Yao et al., 29 Sep 2025). In this sense, MM-STT is less a single canonical network than a recurring architectural pattern whose specific tokenization, attention factorization, and fusion strategy vary by application domain.

1. Definition and conceptual scope

Within the most explicit formulation, MM-STT is introduced for multi-step, high-resolution forecasting of land subsidence from InSAR data, where the model fuses dynamic displacement data with static physical priors and cyclical time features, then applies joint spatio-temporal self-attention over all tokens (patches × time) in a single Transformer encoder (Yao et al., 29 Sep 2025). The same underlying notion appears in other domains: RGBT tracking uses spatio-temporal multimodal tokens that mix RGB and thermal streams with temporal memory tokens (Sun et al., 2024); multi-view action recognition constructs per-view spatio-temporal multimodal tokens from audio and visual features before temporal and view-wise Transformer processing (Nguyen et al., 3 Apr 2025); and multimodal sequential learning factorizes intra-modal and inter-modal dynamics across language, vision, and acoustics (Zadeh et al., 2019).

The common denominator is the treatment of spatio-temporal prediction or representation learning as a token-based problem in which modality information is not external to sequence modeling but embedded into it. In some instances, fusion is early and pervasive, as in land subsidence forecasting, where all modalities are stacked as channels before tokenization (Yao et al., 29 Sep 2025). In others, fusion is structured and staged: RGBT tracking keeps modality-specific streams with shared encoder parameters and performs cross-modal attention inside inserted modules (Sun et al., 2024), whereas MultiTSF uses early multi-modal + late multi-view fusion (Nguyen et al., 3 Apr 2025).

This suggests that MM-STT is best understood as a design family rather than a single topology. Its defining property is not merely the presence of multiple modalities and attention layers, but the coupling of spatial, temporal, and cross-modal reasoning within a unified computational graph.

2. Core architectural primitives

A recurrent MM-STT pipeline begins with modality-specific preprocessing and tokenization. In the land-subsidence formulation, the input tensor is

$$\mathbf{X}_{\text{in}} \in \mathbb{R}^{B \times T_{\text{in}} \times C_{\text{in}} \times H \times W},$$

with $T_{\text{in}}=10$, $H=W=64$, and $C_{\text{in}}=6$, comprising one dynamic displacement channel, three static physical features (mean_velocity, acceleration, seasonality), and two cyclical time channels defined by day-of-year sin/cos encoding (Yao et al., 29 Sep 2025). These channels are patchified jointly, flattened, and linearly projected into embeddings, so that learned fusion is already present at the patch-embedding stage.

A comparable tokenization logic appears in RGBT tracking, where template and search images for visible and thermal modalities are split into $16 \times 16$ patches, linearly projected, and processed by shared ViT encoder layers (Sun et al., 2024). MultiTSF likewise forms per-view audio-visual tokens by concatenating AST and ViT features, yielding a sequence $$F_n^{\mathsf{AV}} \in \mathbb{R}^{T \times D_{\mathsf{AV}}}$$ for each view (Nguyen et al., 3 Apr 2025). In FMT, multimodal sequential data are aligned on a shared temporal grid and passed through unimodal embedding layers before factorized multimodal attentions are applied (Zadeh et al., 2019).

Another recurring primitive is explicit positional structure. Land-subsidence MM-STT adds a learnable spatio-temporal positional embedding over flattened patch–time tokens (Yao et al., 29 Sep 2025). STT for facial expression recognition uses separate learnable spatial and temporal positional embeddings after CNN tokenization (Ma et al., 2022). Video-language STGT adds 2D sinusoidal spatial embeddings and learnable temporal embeddings to vision tokens before graph-biased spatio-temporal attention (Zhang et al., 2024).

A final primitive is task-specific decoding. In the land-subsidence model, output tokens are projected back to patches, rearranged into a future tensor, and reduced to the displacement target by a final $1\times1$ convolution (Yao et al., 29 Sep 2025). In tracking and detection settings, decoder queries yield classification and localization outputs (Sun et al., 2024, Dai et al., 2022). In motion prediction, proposal features are decoded into future trajectories and confidence scores (Liu et al., 2021). The architectural pattern is therefore consistent: tokenization, positionalization, spatio-temporal transformation, and a domain-specific prediction head.

3. Attention and fusion strategies

The principal architectural distinction within MM-STT research concerns how spatial, temporal, and modal interactions are fused. The land-subsidence MM-STT adopts a fully joint mechanism: after concatenating patches across both space and time into a single sequence of length

$$N = T_{\text{in}} \times \frac{H}{P} \times \frac{W}{P},$$

multi-head self-attention is applied globally, so any token $(x,y,t)$ may attend directly to any other token $(x',y',t')$ (Yao et al., 29 Sep 2025). The paper emphasizes that there is no factorization into “spatial attention then temporal attention”; dependencies are learned together.

Other MM-STT variants factorize more explicitly. STT for facial expression recognition first applies spatial attention within frames and then temporal attention across frames for each spatial location (Ma et al., 2022). The 3D human motion ST-Transformer similarly uses a decoupled temporal and spatial self-attention mechanism, which reduces complexity from naive joint attention while preserving explicit temporal and structural dependencies (Aksan et al., 2020). MultiTSF separates temporal modeling per view from inter-view Transformer fusion (Nguyen et al., 3 Apr 2025). NaviSTAR splits spatial and temporal encoders, then fuses them with a multi-modal transformer that treats spatial and temporal streams as distinct modalities (Wang et al., 2023).

Cross-modal fusion also varies substantially. The land-subsidence model performs early and pervasive fusion by stacking all modalities before tokenization, with no modality-specific encoders (Yao et al., 29 Sep 2025). RGBT tracking instead uses cross-attention to enhance static templates and dynamic template tokens across modalities, then injects temporally fused information into search tokens (Sun et al., 2024). E$T_{\text{in}}=10$0-CSTP uses cross-modal attention with the spatio-temporal modality as query and text or image features as keys and values, followed by a gating mechanism over concatenated features (2505.17637). FMT adopts a factorized formulation in which distinct attention modules separately model unimodal, bimodal, and trimodal dynamics such as $T_{\text{in}}=10$1, $T_{\text{in}}=10$2, $T_{\text{in}}=10$3, $T_{\text{in}}=10$4, $T_{\text{in}}=10$5, $T_{\text{in}}=10$6, and $T_{\text{in}}=10$7 (Zadeh et al., 2019).

This suggests two major MM-STT design lineages. One lineage relies on fully fused tokens and global joint attention; the other preserves some modality-specific structure and introduces fusion through cross-attention, staged aggregation, or factorized attentions. The choice appears to depend on the domain’s inductive requirements, sequence length, and whether modality alignment is naturally shared or heterogeneous.

4. Representative instantiations across domains

The land-subsidence MM-STT is among the clearest end-to-end demonstrations of the pattern. It forecasts the next $T_{\text{in}}=10$8 displacement maps from the past $T_{\text{in}}=10$9 multi-modal maps on the EGMS dataset, using 16 Transformer encoder layers, Smooth L1 loss, AdamW, and a $H=W=64$0 rasterized spatial grid (Yao et al., 29 Sep 2025). The model is explicitly framed as addressing both a model gap—global, long-range spatio-temporal modeling—and a data paradigm gap—multi-modal exploitation of displacement, static priors, and temporal cycles.

RGBT tracking provides a different instantiation. Its Spatio-Temporal Multimodal Tokens module augments a ViT-based Siamese tracker with static template tokens, dynamic template tokens extracted from prior search regions, cross-modal enhancement between RGB and thermal templates, and temporal fusion into current search tokens (Sun et al., 2024). Here, MM-STT is not a monolithic encoder over a single flattened space-time grid; it is a modular insertion into a tracking backbone.

MultiTSF extends the pattern to human-centric multi-view and multi-modal action recognition. Shared AST and ViT encoders produce temporal audio tokens and frame-level visual tokens; these are concatenated into audio-visual tokens, processed by a shared temporal Transformer, and then fused across views by a multi-view Transformer (Nguyen et al., 3 Apr 2025). A Human Detection Module based on YOLOv10 provides frame-level binary pseudo-labels for human presence, adding an auxiliary supervision signal.

Other examples broaden the scope further. NaviSTAR uses a hybrid spatio-temporal graph transformer with a multi-modal fusion block, though “multi-modal” there refers to spatial and temporal interaction streams rather than separate sensors (Wang et al., 2023). TransSTAM fuses appearance and spatio-temporal features for multiple object tracking in a compact encoder–decoder Transformer (Dai et al., 2022). mmTransformer for autonomous driving uses stacked transformer encoder–decoder modules over motion history, HD map structure, and social context, with a fixed proposal set representing distinct future modes (Liu et al., 2021). STGT for video-language alignment applies spatio-temporal graph-guided attention to visual tokens before contrastive alignment with text embeddings (Zhang et al., 2024). E$H=W=64$1-CSTP incorporates text and images as auxiliary modalities for graph-based spatio-temporal forecasting while replacing Transformer temporal attention with Mamba (2505.17637).

Taken together, these systems indicate that MM-STT has become a transferable architectural pattern across remote sensing, tracking, action recognition, forecasting, navigation, and alignment problems.

5. Empirical behavior and reported performance

The strongest numerical evidence in the supplied literature comes from high-resolution land-subsidence prediction. On the public EGMS dataset, MM-STT reports, at the $H=W=64$2 horizon averaged over 10 runs, RMSE 0.0819, MAE 0.0624, $H=W=64$3, compared with CNN-LSTM (0.7499 RMSE), ConvLSTM (1.1987), MM-STGCN (0.9207), and MM-STAEformer (1.2771) (Yao et al., 29 Sep 2025). The paper characterizes this as roughly an order-of-magnitude RMSE reduction relative to all baselines. It further reports that at $H=W=64$4 and $H=W=64$5, MM-STT maintains RMSE around 0.08–0.09 and $H=W=64$6, while the baselines degrade substantially.

The same study also reports generalization to six unseen EGMS tiles. For continuous and periodic regimes averaged over four regions, MM-STT yields RMSE 0.1006, MAE 0.0957, and $H=W=64$7; for co-seismic regimes averaged over two regions, RMSE is 0.8254, MAE 0.5318, and $H=W=64$8 (Yao et al., 29 Sep 2025). The paper interprets the latter as evidence that the model cannot predict earthquakes as stochastic events but can model the post-event state once it enters the input window.

In RGBT tracking, the STMT formulation reports benchmark performance of 83.0 / 59.5 MPR / MSR on RGBT210, 86.5 / 63.8 on RGBT234, and 67.4 / 63.4 / 53.7 PR / NPR / SR on LasHeR, while running at 39.1 FPS (Sun et al., 2024). In multi-view action recognition, MultiTSF reports on MultiSensor-Home, under the multi-modal setting, sequence-level $H=W=64$9 and $C_{\text{in}}=6$0, with frame-level 76.12 and 91.45, and on MM-Office multi-modal sequence-level 85.65 and 93.03 (Nguyen et al., 3 Apr 2025).

The more general implication is that MM-STT formulations tend to be introduced where prior models are said to have local receptive fields, separate space/time modules, weak multi-modal integration, or inadequate long-range dependency modeling. The reported gains vary by domain, but the architectural rationale is consistent: stronger spatio-temporal context and deeper multi-modal fusion are associated with better long-horizon or ambiguity-sensitive performance.

6. Limitations, misconceptions, and future directions

A recurrent misconception is that any Transformer with multiple inputs qualifies as an MM-STT. The literature here suggests a narrower interpretation. In the land-subsidence setting, the claimed advantage is not merely “using a Transformer” but the combination of deep multi-modal fusion and truly joint spatio-temporal attention (Yao et al., 29 Sep 2025). Conversely, in RGBT tracking, preserving static template tokens while introducing dynamic temporal memory is central; naive replacement of the initial template is explicitly criticized for drift accumulation (Sun et al., 2024). These contrasts indicate that MM-STT design depends on the failure mode being addressed, not on a fixed template.

Scalability remains a persistent boundary. Joint attention over

$C_{\text{in}}=6$1

tokens may become costly for higher resolutions or longer sequences in land-subsidence forecasting (Yao et al., 29 Sep 2025). Video-language alignment addresses efficiency through graph-biased sparse attention and max-pooling sampling (Zhang et al., 2024). E$C_{\text{in}}=6$2-CSTP replaces temporal self-attention with Mamba specifically to obtain linear-time temporal complexity and reports 17.37%–56.11% reductions in computational overhead relative to several Transformer-based variants (2505.17637). This suggests that practical MM-STT systems will continue to explore structured sparsity, factorization, hierarchical tokenization, or alternative sequence operators.

Another limitation concerns causality and extrapolation. The land-subsidence model is described as a data-driven pattern forecaster, not a causal physical simulator (Yao et al., 29 Sep 2025). E$C_{\text{in}}=6$3-CSTP addresses confounding through a dual-branch causal design and a learned causal adjacency matrix, indicating one route toward more intervention-aware MM-STT formulations (2505.17637). More broadly, a plausible implication is that future MM-STT research will increasingly distinguish predictive correlation from causal dependency, especially in domains such as Earth observation, traffic, and environmental systems.

Future directions are explicitly stated in several works. For land subsidence these include hierarchical or multi-scale attention, larger geographic extents, additional modalities such as groundwater, land cover, DEM, and weather, and improved uncertainty modeling (Yao et al., 29 Sep 2025). For action recognition, the architecture highlights the role of human-centric auxiliary supervision and dynamic view weighting (Nguyen et al., 3 Apr 2025). For video-language alignment, graph construction and longer-range temporal reasoning remain open extensions (Zhang et al., 2024). Collectively, these directions indicate that MM-STT is evolving from a descriptive label for multi-modal Transformer architectures into a more principled design space centered on token structure, fusion depth, spatio-temporal factorization, and robustness under real-world heterogeneity.