Spatio-Temporal Memory Prediction

• Spatio-Temporal Memory Prediction is the integration of historical spatial and temporal data to forecast future states using recurrent, retrieval, and transformer-based methods.
• It employs diverse techniques such as LSTM, ConvLSTM, and attention-driven transformers to enhance accuracy in tasks like traffic forecasting, video anomaly detection, and embodied planning.
• Its practical insights address challenges like catastrophic forgetting and negative transfer while enabling scalable, interpretable, and precise predictions in various complex domains.

Spatio-Temporal Memory Prediction (STMP) denotes, in the cited literature, a family of predictive formulations in which a model encodes historical temporal dependencies together with spatial structure and then uses that memory to forecast future states, detect deviations, retrieve reusable patterns, or support long-horizon planning. The term spans recurrent architectures such as stacked LSTMs and ConvLSTMs, attention-based transformers, replay and retrieval systems, state space models, dynamic knowledge graphs, and space-time projection methods. Its application range includes road traffic speed prediction, urban multi-attribute and flow prediction, video anomaly detection, weather forecasting, wireless-network metrics, person re-identification, turbulent-flow forecasting, and embodied task planning (Lee et al., 2021, Tran et al., 2022, Yuan et al., 2024, Lei et al., 14 Feb 2025).

1. Scope and formal task definitions

Across the literature, STMP is organized around the joint modeling of temporal history and spatial dependence. In traffic forecasting, this is expressed as a dynamically localised LSTM that combines a target road’s historical travel speed with aggregated dynamic spatial influences from neighboring roads (Lee et al., 2021). In video prediction, the task is formulated as predicting future sequences from historical sequences, with input data represented as a 4D tensor in $\mathbb{R}^{T \times C \times H \times W}$ and a learned mapping

$$\theta^\star = \arg \min_{\theta} \mathcal{L}\left(\mathcal{F}_\theta\left(\mathbf{X}_{in}^{t:T}\right), \mathbf{X}_{out}^{T+1:T+T'}\right)$$

for future-frame prediction (Nie et al., 2023). In embodied planning, the relevant memory state is explicitly factorized into temporal and spatial belief components,

$b_i = (b_i^t, b_i^s),$

where temporal belief is derived from summarized history and spatial belief from a dynamically updated knowledge graph (Lei et al., 14 Feb 2025).

The underlying notion of “memory” is therefore not uniform. In recurrent models it is the hidden state or cell state; in retrieval systems it is an external key-value store; in streaming systems it is a replay buffer; in urban flow models it can be a structured bank of spatio-temporal patterns; and in space-time projection it is the hindcast window embedded directly into a high-dimensional vector (Yuan et al., 2024, Miao et al., 2024, Schmidt, 31 Mar 2025). This suggests that STMP functions less as a single architecture than as a design principle: the predictive mechanism must preserve and exploit spatially organized temporal context.

The prediction target also varies. Some works forecast scalar or vector time series, such as temperature, traffic speed, flow, or Wi-Fi performance metrics (Karevan et al., 2018, Shaabanzadeh et al., 2024). Others predict full future video frames or motion trajectories (Tran et al., 2022, Nie et al., 2023). Still others use spatio-temporal memory to support downstream inference rather than direct regression, as in video-based person re-identification and long-horizon embodied task planning (Eom et al., 2021, Lei et al., 14 Feb 2025).

2. Recurrent and localized memory formulations

The recurrent lineage of STMP is exemplified by stacked LSTM, ConvLSTM, and memory-transfer variants. A 2-layer spatio-temporal stacked LSTM for weather forecasting assigns an independent LSTM to each location in the first layer and concatenates the resulting hidden states for the second layer, thereby separating local temporal encoding from cross-location fusion (Karevan et al., 2018). In that study, for 3-days ahead minimum temperature in “Nov/Dec,” the reported MAE changed from $3.04^\circ\mathrm{C}$ for the stacked LSTM to $1.72^\circ\mathrm{C}$ for the spatio-temporal stacked LSTM (Karevan et al., 2018).

For road traffic speed prediction, a dynamically localised LSTM augments recurrent temporal memory with a localised dynamic spatial weight matrix $M_t$. Its entries are defined by relative speed differences, directionality, and inter-road distance:

$M_t(k,l)= \begin{cases} \displaystyle \frac{v_l-v_k}{v_l}\cdot \frac{1}{d_{lk}}, & \text{if road } k \text{ is behind road } l,\[6pt] \displaystyle \frac{v_k-v_l}{v_l}\cdot \frac{1}{d_{lk}}, & \text{if road } k \text{ is ahead of road } l,\[6pt] 0, & \text{if road } k \text{ is out of the coverage of road } l. \end{cases}$

The model receives both the historical travel speed of road $l$ and the column total of the corresponding column in the spatial weight matrix. On the main test set, the reported MAPE for route “B $\rightarrow$ Toll3” changed from $19.211$ and $$\theta^\star = \arg \min_{\theta} \mathcal{L}\left(\mathcal{F}_\theta\left(\mathbf{X}_{in}^{t:T}\right), \mathbf{X}_{out}^{T+1:T+T'}\right)$$0 in the two baselines to $$\theta^\star = \arg \min_{\theta} \mathcal{L}\left(\mathcal{F}_\theta\left(\mathbf{X}_{in}^{t:T}\right), \mathbf{X}_{out}^{T+1:T+T'}\right)$$1 in the proposed model; under abnormal conditions, the reported improvement over the best baseline for “C $$\theta^\star = \arg \min_{\theta} \mathcal{L}\left(\mathcal{F}_\theta\left(\mathbf{X}_{in}^{t:T}\right), \mathbf{X}_{out}^{T+1:T+T'}\right)$$2 Toll 3” was $$\theta^\star = \arg \min_{\theta} \mathcal{L}\left(\mathcal{F}_\theta\left(\mathbf{X}_{in}^{t:T}\right), \mathbf{X}_{out}^{T+1:T+T'}\right)$$3 (Lee et al., 2021).

ConvLSTM-based STMP is especially prominent in video anomaly detection. A sequence-to-sequence architecture using spatio-temporal ConvLSTM predicts future frames from past normal frames, and anomalies are identified through prediction error rather than reconstruction error (Tran et al., 2022). The error is defined as

$$\theta^\star = \arg \min_{\theta} \mathcal{L}\left(\mathcal{F}_\theta\left(\mathbf{X}_{in}^{t:T}\right), \mathbf{X}_{out}^{T+1:T+T'}\right)$$4

with regularity score

$$\theta^\star = \arg \min_{\theta} \mathcal{L}\left(\mathcal{F}_\theta\left(\mathbf{X}_{in}^{t:T}\right), \mathbf{X}_{out}^{T+1:T+T'}\right)$$5

On CUHK Avenue with $$\theta^\star = \arg \min_{\theta} \mathcal{L}\left(\mathcal{F}_\theta\left(\mathbf{X}_{in}^{t:T}\right), \mathbf{X}_{out}^{T+1:T+T'}\right)$$6, the reported prediction result was $$\theta^\star = \arg \min_{\theta} \mathcal{L}\left(\mathcal{F}_\theta\left(\mathbf{X}_{in}^{t:T}\right), \mathbf{X}_{out}^{T+1:T+T'}\right)$$7 AUC / $$\theta^\star = \arg \min_{\theta} \mathcal{L}\left(\mathcal{F}_\theta\left(\mathbf{X}_{in}^{t:T}\right), \mathbf{X}_{out}^{T+1:T+T'}\right)$$8 EER, versus $$\theta^\star = \arg \min_{\theta} \mathcal{L}\left(\mathcal{F}_\theta\left(\mathbf{X}_{in}^{t:T}\right), \mathbf{X}_{out}^{T+1:T+T'}\right)$$9 AUC / $b_i = (b_i^t, b_i^s),$0 EER for reconstruction (Tran et al., 2022). A related ConvLSTM auto-encoder introduces bi-directionality, a spatial higher-order mechanism, and attention for enhanced spatio-temporal memory exchange; on UCSD Ped2, CUHK Avenue, and ShanghaiTech, the reported frame-level AUC values were $b_i = (b_i^t, b_i^s),$1, $b_i = (b_i^t, b_i^s),$2, and $b_i = (b_i^t, b_i^s),$3, respectively (Shen et al., 2022).

A more explicit memory-transfer formulation appears in the transferable memory framework for unsupervised transfer learning. Its Transferable Memory Unit distills knowledge from the memory states of multiple pretrained RNNs and fuses those states through transfer gates:

$b_i = (b_i^t, b_i^s),$4

On Flying Digits, the reported test MSE changed from $b_i = (b_i^t, b_i^s),$5 for scratch training and $b_i = (b_i^t, b_i^s),$6 for finetuning to $b_i = (b_i^t, b_i^s),$7 for TMU transfer, while SSIM changed from $b_i = (b_i^t, b_i^s),$8 to $b_i = (b_i^t, b_i^s),$9 (Yao et al., 2020).

3. Attention, prompting, and transformer-based memory

A major shift in STMP is the replacement or augmentation of recurrent memory with attention and prompt mechanisms. PromptST addresses spatio-temporal multi-attribute prediction through a spatio-temporal transformer backbone and a parameter-sharing training scheme, followed by attribute-specific prompt tuning (Zhang et al., 2023). Its temporal encoder and spatial encoder both use positional embedding, multi-head attention, and feed-forward layers. Adaptation is performed by updating lightweight prompt tokens while freezing the pretrained backbone, with reported parameter savings of up to $3.04^\circ\mathrm{C}$0 and good transferability on unseen spatio-temporal attributes (Zhang et al., 2023).

The Triplet Attention Transformer removes recurrence entirely and alternates attention in temporal, spatial, and channel dimensions through the Triplet Attention Module. Temporal attention is causal and operates across frames; spatial attention uses grid unshuffle and relative positional embedding; channel attention groups channels for efficient intra-frame interaction (Nie et al., 2023). On Moving MNIST $3.04^\circ\mathrm{C}$1, the reported result was MSE $3.04^\circ\mathrm{C}$2 and SSIM $3.04^\circ\mathrm{C}$3; on TaxiBJ $3.04^\circ\mathrm{C}$4, the reported result was $3.04^\circ\mathrm{C}$5 MSE and $3.04^\circ\mathrm{C}$6 SSIM (Nie et al., 2023). These results are presented as evidence that alternating temporal, spatial, and channel-level attention learns more complex short- and long-range spatiotemporal dependencies than recurrent baselines.

STAM emphasizes interpretability rather than only predictive accuracy. It aligns both spatial attention over variables and temporal attention over time steps directly to each output step:

$3.04^\circ\mathrm{C}$7

The model is described as causal and scalable, and it reported RMSE / MAE / $3.04^\circ\mathrm{C}$8 of $3.04^\circ\mathrm{C}$9 on the Pollution dataset, $1.72^\circ\mathrm{C}$0 on the Building dataset, and $1.72^\circ\mathrm{C}$1 on EHR (Gangopadhyay et al., 2020).

A distinct simplification strategy is represented by EasyST, which distills knowledge from spatio-temporal GNN teachers into an MLP student using a spatio-temporal information bottleneck, teacher-bounded regression loss, and spatial and temporal prompts (Tang et al., 2024). The information bottleneck objective is written as

$1.72^\circ\mathrm{C}$2

On the cited traffic benchmark, the reported MAE was $1.72^\circ\mathrm{C}$3 for EasyST, compared with $1.72^\circ\mathrm{C}$4 for STID and $1.72^\circ\mathrm{C}$5 for MTGNN (Tang et al., 2024). This suggests that explicit graph message passing is not a necessary condition for competitive STMP when distillation and prompt context are sufficiently strong.

4. External memory, retrieval, replay, and domain adaptation

Several recent formulations make memory explicit as an external structure rather than an implicit latent state. UniFlow introduces SpatioTemporal Memory Retrieval Augmentation (ST-MRA), a structured set of four key-value memory banks for time-domain, frequency-domain, time-derived spatial, and frequency-derived spatial patterns (Yuan et al., 2024). Queries are derived from self-attention, FFT, and GCN-based spatial embeddings, and retrieval is performed by

$1.72^\circ\mathrm{C}$6

The retrieved prompts are injected into the decoder. The reported outcome is that UniFlow outperforms existing models in both grid-based and graph-based flow prediction and achieves average RMSE/MAE reductions exceeding $1.72^\circ\mathrm{C}$7 compared to the best individually trained specialized models (Yuan et al., 2024).

Streaming-data STMP introduces a different memory problem: catastrophic forgetting. The unified replay-based continuous learning framework addresses this by combining a replay buffer, ranking-based maximally interfered retrieval, spatio-temporal mixup, a spatio-temporal autoencoder, and an STSimSiam objective (Miao et al., 2024). The mixup rule is

$1.72^\circ\mathrm{C}$8

and the overall loss combines task and self-supervised terms,

$1.72^\circ\mathrm{C}$9

The framework reported up to $M_t$0 improvement on some metrics for streaming data and identified data augmentation and the self-supervised loss as especially important for holistic retention and generalization (Miao et al., 2024).

Damba-ST recasts STMP in state space form. It argues that directly applying Mamba as a spatio-temporal backbone leads to negative transfer and severe performance degradation because of spatio-temporal heterogeneity and recursive hidden-state updates (An et al., 22 Jun 2025). Its Domain-Adaptive State Space Model partitions latent space into shared and domain-specific subspaces and uses three Domain Adapters—Spatial Adapter, Temporal Adapter, and ST-Delay Adapter—to bridge domain distributions. The common component is obtained by projection onto the exchanged adapter direction,

$M_t$1

The model retains $M_t$2 linear complexity and reported fast inference, including $M_t$3 seconds per next-day forecast on a large urban dataset (An et al., 22 Jun 2025).

Taken together, these systems define an important branch of STMP in which memory is not merely “remembered” by recurrence. It is stored, sampled, aligned, replayed, and retrieved.

5. Spatial granularity, arbitrary regions, and non-neural formulations

A separate theme concerns the spatial unit over which memory is defined. One4All-ST addresses arbitrary modifiable areal units by learning multi-scale representations with a single model rather than training separate predictors for different partitions (Chen et al., 2024). Its hierarchical spatial modeling is combined with scale normalization,

$M_t$4

and its top-down feature fusion is written as

$M_t$5

To ensure consistency across scales, it uses a dynamic programming solution to the optimal combination problem and an extended quad-tree for indexing. The reported system uses $M_t$6M parameters rather than $M_t$7 the parameters of multi-model setups, supports query latency under $M_t$8 ms, and achieved $M_t$9 lower RMSE over the best baseline in some tasks (Chen et al., 2024).

A different response to high-dimensional spatio-temporal prediction is Space-Time Projection, which is rooted in Proper Orthogonal Decomposition rather than neural sequence modeling (Schmidt, 31 Mar 2025). STP embeds the hindcast window directly:

$M_t(k,l)= \begin{cases} \displaystyle \frac{v_l-v_k}{v_l}\cdot \frac{1}{d_{lk}}, & \text{if road } k \text{ is behind road } l,\[6pt] \displaystyle \frac{v_k-v_l}{v_l}\cdot \frac{1}{d_{lk}}, & \text{if road } k \text{ is ahead of road } l,\[6pt] 0, & \text{if road } k \text{ is out of the coverage of road } l. \end{cases}$0

constructs extended space-time modes from the hindcast and forecast intervals, and predicts by projection:

$M_t(k,l)= \begin{cases} \displaystyle \frac{v_l-v_k}{v_l}\cdot \frac{1}{d_{lk}}, & \text{if road } k \text{ is behind road } l,\[6pt] \displaystyle \frac{v_k-v_l}{v_l}\cdot \frac{1}{d_{lk}}, & \text{if road } k \text{ is ahead of road } l,\[6pt] 0, & \text{if road } k \text{ is out of the coverage of road } l. \end{cases}$1

For a fixed ensemble and prediction horizon, the only tunable parameter is the truncation rank, and the hindcast accuracy is presented as a reliable indicator for short-term forecast accuracy and as a lower bound on forecast errors (Schmidt, 31 Mar 2025). In the reported comparison with a standard LSTM, STP consistently provided more accurate forecasts on the studied datasets (Schmidt, 31 Mar 2025).

Real Wi-Fi measurements provide yet another formulation. There, the prediction system first estimates Pearson correlations between access points and decides whether to perform only-temporal or spatio-temporal prediction based on whether neighbors exceed a threshold $M_t(k,l)= \begin{cases} \displaystyle \frac{v_l-v_k}{v_l}\cdot \frac{1}{d_{lk}}, & \text{if road } k \text{ is behind road } l,\[6pt] \displaystyle \frac{v_k-v_l}{v_l}\cdot \frac{1}{d_{lk}}, & \text{if road } k \text{ is ahead of road } l,\[6pt] 0, & \text{if road } k \text{ is out of the coverage of road } l. \end{cases}$2 (Shaabanzadeh et al., 2024). For AP $M_t(k,l)= \begin{cases} \displaystyle \frac{v_l-v_k}{v_l}\cdot \frac{1}{d_{lk}}, & \text{if road } k \text{ is behind road } l,\[6pt] \displaystyle \frac{v_k-v_l}{v_l}\cdot \frac{1}{d_{lk}}, & \text{if road } k \text{ is ahead of road } l,\[6pt] 0, & \text{if road } k \text{ is out of the coverage of road } l. \end{cases}$3 and neighbor $M_t(k,l)= \begin{cases} \displaystyle \frac{v_l-v_k}{v_l}\cdot \frac{1}{d_{lk}}, & \text{if road } k \text{ is behind road } l,\[6pt] \displaystyle \frac{v_k-v_l}{v_l}\cdot \frac{1}{d_{lk}}, & \text{if road } k \text{ is ahead of road } l,\[6pt] 0, & \text{if road } k \text{ is out of the coverage of road } l. \end{cases}$4, the correlation is

$M_t(k,l)= \begin{cases} \displaystyle \frac{v_l-v_k}{v_l}\cdot \frac{1}{d_{lk}}, & \text{if road } k \text{ is behind road } l,\[6pt] \displaystyle \frac{v_k-v_l}{v_l}\cdot \frac{1}{d_{lk}}, & \text{if road } k \text{ is ahead of road } l,\[6pt] 0, & \text{if road } k \text{ is out of the coverage of road } l. \end{cases}$5

A hybrid architecture then performs spatial processing with CNN and temporal prediction with RNN. The reported hybrid methodology improved prediction accuracy at the expense of a slight increase in Training Computational Time and negligible increase in Prediction Computational Time (Shaabanzadeh et al., 2024).

These examples show that STMP is not restricted to fixed graphs, regular grids, or end-to-end deep architectures. The memory carrier may be a hierarchical partition, a POD basis, or a correlation-conditioned hybrid network.

6. Extended applications, common misconceptions, and unresolved issues

STMP is often described as a forecasting problem, but the same memory logic is also used for planning, retrieval, and representation refinement. STMA, for example, couples a spatio-temporal memory module, a dynamic knowledge graph, and a planner-critic mechanism for long-horizon embodied task planning (Lei et al., 14 Feb 2025). Its temporal summarizer maps history to temporal belief,

$M_t(k,l)= \begin{cases} \displaystyle \frac{v_l-v_k}{v_l}\cdot \frac{1}{d_{lk}}, & \text{if road } k \text{ is behind road } l,\[6pt] \displaystyle \frac{v_k-v_l}{v_l}\cdot \frac{1}{d_{lk}}, & \text{if road } k \text{ is ahead of road } l,\[6pt] 0, & \text{if road } k \text{ is out of the coverage of road } l. \end{cases}$6

the relation retriever extracts relational triples,

$M_t(k,l)= \begin{cases} \displaystyle \frac{v_l-v_k}{v_l}\cdot \frac{1}{d_{lk}}, & \text{if road } k \text{ is behind road } l,\[6pt] \displaystyle \frac{v_k-v_l}{v_l}\cdot \frac{1}{d_{lk}}, & \text{if road } k \text{ is ahead of road } l,\[6pt] 0, & \text{if road } k \text{ is out of the coverage of road } l. \end{cases}$7

and the planner produces a subgoal and action sequence from $M_t(k,l)= \begin{cases} \displaystyle \frac{v_l-v_k}{v_l}\cdot \frac{1}{d_{lk}}, & \text{if road } k \text{ is behind road } l,\[6pt] \displaystyle \frac{v_k-v_l}{v_l}\cdot \frac{1}{d_{lk}}, & \text{if road } k \text{ is ahead of road } l,\[6pt] 0, & \text{if road } k \text{ is out of the coverage of road } l. \end{cases}$8. On TextWorld, across 32 tasks, the reported result was a $M_t(k,l)= \begin{cases} \displaystyle \frac{v_l-v_k}{v_l}\cdot \frac{1}{d_{lk}}, & \text{if road } k \text{ is behind road } l,\[6pt] \displaystyle \frac{v_k-v_l}{v_l}\cdot \frac{1}{d_{lk}}, & \text{if road } k \text{ is ahead of road } l,\[6pt] 0, & \text{if road } k \text{ is out of the coverage of road } l. \end{cases}$9 improvement in success rate and a $l$0 increase in average score (Lei et al., 14 Feb 2025).

In video-based person re-identification, STMN stores frequent spatial distractors in a spatial memory and temporal attention patterns in a temporal memory (Eom et al., 2021). Frame-level features are refined by subtracting a batch-normalized memory output,

$l$1

and sequence-level aggregation is controlled by memory-derived temporal attention. With memory spread loss, the reported MARS results changed from $l$2 Rank-1/mAP for the baseline to $l$3 for the combined spatial and temporal memory model; on LS-VID, the reported result changed from $l$4 to $l$5 (Eom et al., 2021).

A biologically plausible interpretation appears in unsupervised learning of spatio-temporal patterns in spiking neuronal networks, where dendritic action potentials act as predictive traces and recurrent plasticity enables high-order sequence learning under noise and overlap (Feiler et al., 2024). The model is described as being able to learn and predict high-order sequences and as robust to different input settings and parameters (Feiler et al., 2024). This suggests that STMP can be formulated without gradient-based supervised training, provided that the mechanism retains context-sensitive predictive state.

Several misconceptions recur in the broader discussion. One is that STMP is inherently an LSTM problem; the cited literature shows competitive transformer, MLP-distillation, state space, retrieval, and POD alternatives (Nie et al., 2023, Tang et al., 2024, An et al., 22 Jun 2025, Schmidt, 31 Mar 2025). A second is that reconstruction is interchangeable with prediction in anomaly detection; the ConvLSTM anomaly-detection results explicitly report superior performance for prediction over reconstruction (Tran et al., 2022). A third is that a single spatial partition is an innocuous prerequisite; the arbitrary-MAU literature frames this as a source of cost and inconsistency (Chen et al., 2024). A fourth is that unified cross-domain models automatically generalize; Damba-ST reports that direct application of Mamba leads to negative transfer and severe performance degradation, while the streaming-data literature identifies catastrophic forgetting as a central obstacle (An et al., 22 Jun 2025, Miao et al., 2024).

The unresolved issues identified by these works are correspondingly structural rather than incremental. They include how to preserve memory under data streams without catastrophic forgetting, how to prevent negative transfer across heterogeneous cities or tasks, how to maintain consistency across scales and regions, how to represent delay and causality without future leakage, and how to combine interpretability with scalability (Miao et al., 2024, An et al., 22 Jun 2025, Chen et al., 2024, Gangopadhyay et al., 2020). In that sense, STMP is best understood as an active research area centered on the architecture of spatio-temporal memory itself: what is stored, where it is stored, how it is updated, and how prediction reads from it.