---
title: Spatio-Temporal Memory Prediction
url: https://www.emergentmind.com/topics/spatio-temporal-memory-prediction-stmp
type: topic
---

# Spatio-Temporal Memory Prediction

Spatio-Temporal Memory Prediction (STMP) denotes, in the cited literature, a family of predictive formulations in which a model encodes historical temporal dependencies together with spatial structure and then uses that memory to forecast future states, detect deviations, retrieve reusable patterns, or support long-horizon planning. The term spans recurrent architectures such as stacked LSTMs and ConvLSTMs, attention-based transformers, replay and retrieval systems, state space models, dynamic knowledge graphs, and space-time projection methods. Its application range includes road traffic speed prediction, urban multi-attribute and flow prediction, video anomaly detection, weather forecasting, wireless-network metrics, person re-identification, turbulent-flow forecasting, and embodied task planning [2112.02409], [2205.08812], [2411.12972], [2502.10177].

## 1. Scope and formal task definitions

Across the literature, STMP is organized around the joint modeling of temporal history and spatial dependence. In traffic forecasting, this is expressed as a dynamically localised LSTM that combines a target road’s historical travel speed with aggregated dynamic spatial influences from neighboring roads [2112.02409]. In video prediction, the task is formulated as predicting future sequences from historical sequences, with input data represented as a 4D tensor in $\mathbb{R}^{T \times C \times H \times W}$ and a learned mapping
$$
\theta^\star = \arg \min_{\theta} \mathcal{L}\left(\mathcal{F}_\theta\left(\mathbf{X}_{in}^{t:T}\right), \mathbf{X}_{out}^{T+1:T+T'}\right)
$$
for future-frame prediction [2310.18698]. In embodied planning, the relevant memory state is explicitly factorized into temporal and spatial belief components,
$$
b_i = (b_i^t, b_i^s),
$$
where temporal belief is derived from summarized history and spatial belief from a dynamically updated knowledge graph [2502.10177].

The underlying notion of “memory” is therefore not uniform. In recurrent models it is the hidden state or cell state; in retrieval systems it is an external key-value store; in streaming systems it is a replay buffer; in urban flow models it can be a structured bank of spatio-temporal patterns; and in space-time projection it is the hindcast window embedded directly into a high-dimensional vector [2411.12972], [2404.14999], [2503.23686]. This suggests that STMP functions less as a single architecture than as a design principle: the predictive mechanism must preserve and exploit spatially organized temporal context.

The prediction target also varies. Some works forecast scalar or vector time series, such as temperature, traffic speed, flow, or Wi-Fi performance metrics [1811.06341], [2408.09423]. Others predict full future video frames or motion trajectories [2205.08812], [2310.18698]. Still others use spatio-temporal memory to support downstream inference rather than direct regression, as in video-based person re-identification and long-horizon embodied task planning [2108.09039], [2502.10177].

## 2. Recurrent and localized memory formulations

The recurrent lineage of STMP is exemplified by stacked LSTM, ConvLSTM, and memory-transfer variants. A 2-layer spatio-temporal stacked LSTM for weather forecasting assigns an independent LSTM to each location in the first layer and concatenates the resulting hidden states for the second layer, thereby separating local temporal encoding from cross-location fusion [1811.06341]. In that study, for 3-days ahead minimum temperature in “Nov/Dec,” the reported MAE changed from $3.04^\circ\mathrm{C}$ for the stacked LSTM to $1.72^\circ\mathrm{C}$ for the spatio-temporal stacked LSTM [1811.06341].

For road traffic speed prediction, a dynamically localised LSTM augments recurrent temporal memory with a localised dynamic spatial weight matrix $M_t$. Its entries are defined by relative speed differences, directionality, and inter-road distance:
$$
M_t(k,l)=
\begin{cases}
\displaystyle \frac{v_l-v_k}{v_l}\cdot \frac{1}{d_{lk}}, & \text{if road } k \text{ is behind road } l,\\[6pt]
\displaystyle \frac{v_k-v_l}{v_l}\cdot \frac{1}{d_{lk}}, & \text{if road } k \text{ is ahead of road } l,\\[6pt]
0, & \text{if road } k \text{ is out of the coverage of road } l.
\end{cases}
$$
The model receives both the historical travel speed of road $l$ and the column total of the corresponding column in the spatial weight matrix. On the main test set, the reported MAPE for route “B $\rightarrow$ Toll3” changed from $19.211$ and $19.186$ in the two baselines to $18.747$ in the proposed model; under abnormal conditions, the reported improvement over the best baseline for “C $\rightarrow$ Toll 3” was $0.324$ [2112.02409].

ConvLSTM-based STMP is especially prominent in video anomaly detection. A sequence-to-sequence architecture using spatio-temporal ConvLSTM predicts future frames from past normal frames, and anomalies are identified through prediction error rather than reconstruction error [2205.08812]. The error is defined as
$$
e(t) = \sum_{i=t+1}^{t+\tau} \|\hat{F}_i - F_i\|_2,
$$
with regularity score
$$
s(t) = 1 - \frac{e(t) - \min_{t'} e(t')}{\max_{t'} e(t')}.
$$
On CUHK Avenue with $\tau=5$, the reported prediction result was $84.8\%$ AUC / $22.4\%$ EER, versus $81.4\%$ AUC / $26.1\%$ EER for reconstruction [2205.08812]. A related ConvLSTM auto-encoder introduces bi-directionality, a spatial higher-order mechanism, and attention for enhanced spatio-temporal memory exchange; on UCSD Ped2, CUHK Avenue, and ShanghaiTech, the reported frame-level AUC values were $98.3$, $90.7$, and $79.7$, respectively [2206.12914].

A more explicit memory-transfer formulation appears in the transferable memory framework for unsupervised transfer learning. Its Transferable Memory Unit distills knowledge from the memory states of multiple pretrained RNNs and fuses those states through transfer gates:
$$
C_t = \hat{C}_t + \sum_{m=1}^M \left[ \alpha_t^m \odot \hat{c}_t^m + (1 - \alpha_t^m) \odot \hat{C}_t \right].
$$
On Flying Digits, the reported test MSE changed from $120.6$ for scratch training and $110.0$ for finetuning to $94.7$ for TMU transfer, while SSIM changed from $0.715/0.732$ to $0.777$ [2009.11763].

## 3. Attention, prompting, and transformer-based memory

A major shift in STMP is the replacement or augmentation of recurrent memory with attention and prompt mechanisms. PromptST addresses spatio-temporal multi-attribute prediction through a spatio-temporal transformer backbone and a parameter-sharing training scheme, followed by attribute-specific prompt tuning [2309.09500]. Its temporal encoder and spatial encoder both use positional embedding, multi-head attention, and feed-forward layers. Adaptation is performed by updating lightweight prompt tokens while freezing the pretrained backbone, with reported parameter savings of up to $99\%$ and good transferability on unseen spatio-temporal attributes [2309.09500].

The Triplet Attention Transformer removes recurrence entirely and alternates attention in temporal, spatial, and channel dimensions through the Triplet Attention Module. Temporal attention is causal and operates across frames; spatial attention uses grid unshuffle and relative positional embedding; channel attention groups channels for efficient intra-frame interaction [2310.18698]. On Moving MNIST $(10 \rightarrow 10)$, the reported result was MSE $17.55$ and SSIM $0.960$; on TaxiBJ $(4 \rightarrow 4)$, the reported result was $31.3 \times 100$ MSE and $0.984$ SSIM [2310.18698]. These results are presented as evidence that alternating temporal, spatial, and channel-level attention learns more complex short- and long-range spatiotemporal dependencies than recurrent baselines.

STAM emphasizes interpretability rather than only predictive accuracy. It aligns both spatial attention over variables and temporal attention over time steps directly to each output step:
$$
\beta_j^i = \frac{\exp(e_j^i)}{\sum_{o=1}^N \exp(e_j^o)}, \qquad
\alpha_j^t = \frac{\exp(a_j^t)}{\sum_{l=1}^{T_x} \exp(a_j^l)}.
$$
The model is described as causal and scalable, and it reported RMSE / MAE / $R^2$ of $47.78 / 29.85 / 0.739$ on the Pollution dataset, $0.0599 / 0.0415 / 0.9975$ on the Building dataset, and $4.591 / 3.281 / 0.412$ on EHR [2008.04882].

A distinct simplification strategy is represented by EasyST, which distills knowledge from spatio-temporal GNN teachers into an MLP student using a spatio-temporal information bottleneck, teacher-bounded regression loss, and spatial and temporal prompts [2409.06748]. The information bottleneck objective is written as
$$
\min_{\mathbb{P}(\mathbf{Z}|\mathbf{X})} \Big[ - \big( I(\mathbf{Y}, \mathbf{Z}) + I(\mathbf{Y}^T, \mathbf{Z}) \big) + (\beta_1 + \beta_2)\, I(\mathbf{X}, \mathbf{Z}) \Big].
$$
On the cited traffic benchmark, the reported MAE was $12.70$ for EasyST, compared with $12.87$ for STID and $13.53$ for MTGNN [2409.06748]. This suggests that explicit graph message passing is not a necessary condition for competitive STMP when distillation and prompt context are sufficiently strong.

## 4. External memory, retrieval, replay, and domain adaptation

Several recent formulations make memory explicit as an external structure rather than an implicit latent state. UniFlow introduces SpatioTemporal Memory Retrieval Augmentation (ST-MRA), a structured set of four key-value memory banks for time-domain, frequency-domain, time-derived spatial, and frequency-derived spatial patterns [2411.12972]. Queries are derived from self-attention, FFT, and GCN-based spatial embeddings, and retrieval is performed by
$$
\alpha_i = \text{softmax}(Q_i K_i^T), \qquad
P_i = \sum_j \alpha_{ij} V_i.
$$
The retrieved prompts are injected into the decoder. The reported outcome is that UniFlow outperforms existing models in both grid-based and graph-based flow prediction and achieves average RMSE/MAE reductions exceeding $10\%$ compared to the best individually trained specialized models [2411.12972].

Streaming-data STMP introduces a different memory problem: catastrophic forgetting. The unified replay-based continuous learning framework addresses this by combining a replay buffer, ranking-based maximally interfered retrieval, spatio-temporal mixup, a spatio-temporal autoencoder, and an STSimSiam objective [2404.14999]. The mixup rule is
$$
\mathcal{X}_{mix} = \lambda \cdot \mathcal{X}_M + (1-\lambda)\cdot \mathcal{X}_{\mathcal{B}},
$$
and the overall loss combines task and self-supervised terms,
$$
L_{all} = L_{task} + L_{ssl}.
$$
The framework reported up to $67\%$ improvement on some metrics for streaming data and identified data augmentation and the self-supervised loss as especially important for holistic retention and generalization [2404.14999].

Damba-ST recasts STMP in state space form. It argues that directly applying Mamba as a spatio-temporal backbone leads to negative transfer and severe performance degradation because of spatio-temporal heterogeneity and recursive hidden-state updates [2506.18939]. Its Domain-Adaptive State Space Model partitions latent space into shared and domain-specific subspaces and uses three Domain Adapters—Spatial Adapter, Temporal Adapter, and ST-Delay Adapter—to bridge domain distributions. The common component is obtained by projection onto the exchanged adapter direction,
$$
\mathbf{R}_{D \rightarrow C}^i = \text{proj}(\mathbf{R}_D^i, \hat{\mathcal{T}}^i)
= \left( \frac{\mathbf{R}_D^i \hat{\mathcal{T}}^{i\top}}{\|\hat{\mathcal{T}}^i\|^2} \right)\hat{\mathcal{T}}^i.
$$
The model retains $\mathcal{O}(LD)$ linear complexity and reported fast inference, including $0.9$ seconds per next-day forecast on a large urban dataset [2506.18939].

Taken together, these systems define an important branch of STMP in which memory is not merely “remembered” by recurrence. It is stored, sampled, aligned, replayed, and retrieved.

## 5. Spatial granularity, arbitrary regions, and non-neural formulations

A separate theme concerns the spatial unit over which memory is defined. One4All-ST addresses arbitrary modifiable areal units by learning multi-scale representations with a single model rather than training separate predictors for different partitions [2403.07022]. Its hierarchical spatial modeling is combined with scale normalization,
$$
\tilde{\mathbf{X}}^s = \frac{\mathbf{X}^s - \mathbb{E}[\mathbf{X}^s]}{\sqrt{\mathrm{Var}[\mathbf{X}^s]}},
$$
and its top-down feature fusion is written as
$$
\mathbf{H}^{P_i}_t = \mathbf{h}^{P_i}_t + \text{UpSample}(\mathbf{h}^{P_{i+1}}_t).
$$
To ensure consistency across scales, it uses a dynamic programming solution to the optimal combination problem and an extended quad-tree for indexing. The reported system uses $0.72$M parameters rather than $6\times$ the parameters of multi-model setups, supports query latency under $20$ ms, and achieved $10.6\%$ lower RMSE over the best baseline in some tasks [2403.07022].

A different response to high-dimensional spatio-temporal prediction is Space-Time Projection, which is rooted in Proper Orthogonal Decomposition rather than neural sequence modeling [2503.23686]. STP embeds the hindcast window directly:
$$
\mathbf{q}_- = [\mathbf{u}_1^T\ \mathbf{u}_2^T\ \cdots\ \mathbf{u}_n^T]^T,
$$
constructs extended space-time modes from the hindcast and forecast intervals, and predicts by projection:
$$
\mathbf{a}_-^\star = \mathbf{\Phi}_-^H \mathbf{W}\mathbf{q}_-^{\text{new}}, \qquad
\mathbf{q}_\pm^\star = \mathbf{\Phi}_\pm^\star \mathbf{a}_-^\star.
$$
For a fixed ensemble and prediction horizon, the only tunable parameter is the truncation rank, and the hindcast accuracy is presented as a reliable indicator for short-term forecast accuracy and as a lower bound on forecast errors [2503.23686]. In the reported comparison with a standard LSTM, STP consistently provided more accurate forecasts on the studied datasets [2503.23686].

Real Wi-Fi measurements provide yet another formulation. There, the prediction system first estimates Pearson correlations between access points and decides whether to perform only-temporal or spatio-temporal prediction based on whether neighbors exceed a threshold $Th_s$ [2408.09423]. For AP $i$ and neighbor $j$, the correlation is
$$
C_{i,j} = \frac{ N \sum_{n=1}^{N} x_{i, n}x_{j, n} - \left(\sum_{n=1}^{N} x_{i, n}\right)\left(\sum_{n=1}^{N} x_{j, n}\right) }{ \sqrt{ \left[N \sum_{n=1}^{N} x_{i, n}^2 - \left(\sum_{n=1}^{N} x_{i, n}\right)^2\right] \left[N \sum_{n=1}^{N} x_{j, n}^2 - \left(\sum_{n=1}^{N} x_{j, n}\right)^2\right] } }.
$$
A hybrid architecture then performs spatial processing with CNN and temporal prediction with RNN. The reported hybrid methodology improved prediction accuracy at the expense of a slight increase in Training Computational Time and negligible increase in Prediction Computational Time [2408.09423].

These examples show that STMP is not restricted to fixed graphs, regular grids, or end-to-end deep architectures. The memory carrier may be a hierarchical partition, a POD basis, or a correlation-conditioned hybrid network.

## 6. Extended applications, common misconceptions, and unresolved issues

STMP is often described as a forecasting problem, but the same memory logic is also used for planning, retrieval, and representation refinement. STMA, for example, couples a spatio-temporal memory module, a dynamic knowledge graph, and a planner-critic mechanism for long-horizon embodied task planning [2502.10177]. Its temporal summarizer maps history to temporal belief,
$$
\mathcal{S}: h_{[1:i-1]} \rightarrow b_i^t,
$$
the relation retriever extracts relational triples,
$$
R: b_i^t \rightarrow G' = \{(x^s, x^r, x^o)\},
$$
and the planner produces a subgoal and action sequence from $(b_i^t,b_i^s,o_i)$. On TextWorld, across 32 tasks, the reported result was a $31.25\%$ improvement in success rate and a $24.7\%$ increase in average score [2502.10177].

In video-based person re-identification, STMN stores frequent spatial distractors in a spatial memory and temporal attention patterns in a temporal memory [2108.09039]. Frame-level features are refined by subtracting a batch-normalized memory output,
$$
\mathbf{f}^{\text s}_{i,k} = \mathbf{f}^{\text o}_{i,k} - \mathrm{BN}(\mathbf{o}^{\text s}_{i,k}),
$$
and sequence-level aggregation is controlled by memory-derived temporal attention. With memory spread loss, the reported MARS results changed from $87.3 / 79.1$ Rank-1/mAP for the baseline to $89.9 / 83.7$ for the combined spatial and temporal memory model; on LS-VID, the reported result changed from $71.6 / 55.9$ to $80.6 / 66.6$ [2108.09039].

A biologically plausible interpretation appears in unsupervised learning of spatio-temporal patterns in spiking neuronal networks, where dendritic action potentials act as predictive traces and recurrent plasticity enables high-order sequence learning under noise and overlap [2410.08637]. The model is described as being able to learn and predict high-order sequences and as robust to different input settings and parameters [2410.08637]. This suggests that STMP can be formulated without gradient-based supervised training, provided that the mechanism retains context-sensitive predictive state.

Several misconceptions recur in the broader discussion. One is that STMP is inherently an LSTM problem; the cited literature shows competitive transformer, MLP-distillation, state space, retrieval, and POD alternatives [2310.18698], [2409.06748], [2506.18939], [2503.23686]. A second is that reconstruction is interchangeable with prediction in anomaly detection; the ConvLSTM anomaly-detection results explicitly report superior performance for prediction over reconstruction [2205.08812]. A third is that a single spatial partition is an innocuous prerequisite; the arbitrary-MAU literature frames this as a source of cost and inconsistency [2403.07022]. A fourth is that unified cross-domain models automatically generalize; Damba-ST reports that direct application of Mamba leads to negative transfer and severe performance degradation, while the streaming-data literature identifies catastrophic forgetting as a central obstacle [2506.18939], [2404.14999].

The unresolved issues identified by these works are correspondingly structural rather than incremental. They include how to preserve memory under data streams without catastrophic forgetting, how to prevent negative transfer across heterogeneous cities or tasks, how to maintain consistency across scales and regions, how to represent delay and causality without future leakage, and how to combine interpretability with scalability [2404.14999], [2506.18939], [2403.07022], [2008.04882]. In that sense, STMP is best understood as an active research area centered on the architecture of spatio-temporal memory itself: what is stored, where it is stored, how it is updated, and how prediction reads from it.

Source: https://www.emergentmind.com/topics/spatio-temporal-memory-prediction-stmp