---
title: Spatiotemporal Maps in Research
url: https://www.emergentmind.com/topics/spatiotemporal-maps-stms
type: topic
---

# Spatiotemporal Maps in Research

Searching arXiv for recent and foundational papers on “spatiotemporal maps” and closely related STM usages.
Spatiotemporal maps (STMs) are representations that bind spatial structure to temporal evolution within a single formal object, but the term is used in several technically distinct ways across the literature. In some settings, an STM is a state representation over space and time, such as a stack of future occupancy grids for robot navigation [2108.10585] or a time-varying affordance field over an environment [1607.00354]. In others, it is a feature representation that converts a sequence into a map-like array for subsequent inference, as in VideoMap for action recognition [1809.03669], token-conditioned relevance maps in untrimmed video [2005.09183], or topological EEG image sequences processed by spatiotemporal pooling [2403.04353]. The term also appears in dynamical-systems and signal-processing contexts, where it denotes spatiotemporal structures in coupled maps [1309.4555, 1710.10077], symbolic encodings of chaotic lattices [1912.02940], or spatially varying temporal bases for dynamic MRI reconstruction [2507.14429]. Across these usages, the common principle is that spatial and temporal dependencies are not treated as separable afterthoughts, but are encoded jointly in a representation that supports prediction, reconstruction, control, or interpretation.

## 1. Conceptual scope and major usages

The STM concept is not a single standardized formalism. The literature instead shows several families of usage.

A first family treats an STM as a **time-indexed map of environment state or task-relevant likelihoods**. In robotics, a Spatio-Temporal Affordance Map (STAM) represents the likelihood that each area of an environment affords a task under the current world state [1607.00354]. In dynamic navigation, a Spatiotemporal Occupancy Grid Map (SOGM) is defined as a 3D grid map whose cells contain occupancy probabilities for given positions and future times [2108.10585].

A second family treats an STM as a **map-like feature representation of sequences**. Temporal-Spatial Mapping (TSM) converts a video into a 2D VideoMap by stacking frame-level CNN features in temporal order [1809.03669]. In cross-modal retrieval and action grounding, spatiotemporal relevance maps are tensors over time, height, and width that indicate where and when caption tokens are supported in an untrimmed video [2005.09183]. For EEG, the signal is converted into a sequence of 2D topological maps and then processed by a spatiotemporal pooling block that operates on a stacked temporal-feature plane [2403.04353].

A third family uses STM language in **dynamical systems and symbolic dynamics**. Coupled chaotic lattices are analyzed through spatiotemporal fixed points, regularity, chimera states, and synchronization of spatiotemporal structures [1309.4555, 1710.10077]. In the spatiotemporal cat map, local system states over finite spacetime windows are encoded by finite 2D symbol blocks that have both reconstructive and statistical meaning [1912.02940].

A fourth family uses STM as a **signal model or reconstruction prior**. In dynamic MRI, STMs generalize the partially separable functions model by allowing temporal basis functions to depend on spatial location [2507.14429]. In DSC-MRI perfusion estimation, a spatiotemporal CNN maps 4D image sequences to voxel-wise perfusion parameter maps while explicitly separating spatial and temporal modules [2312.05279]. In multitemporal satellite analysis, per-class probability maps across dates are refined by a 3D iterative spatiotemporal filter over space, time, and height [2107.00590].

This diversity suggests that “STM” is best understood as a class of representations rather than a single architecture. A plausible implication is that the defining criterion is not the data modality, but whether a method represents spatial structure and temporal evolution in a coupled form usable for downstream inference.

## 2. Formal representations and mathematical structures

The mathematical form of an STM depends on the application.

In robot affordance modeling, the formal object is a function
\[
f_{E,\boldsymbol{\theta}} : S \times T \rightarrow A_E
\]
where \(E\) is the environment, \(\boldsymbol{\theta}\) are affordance parameters, \(s_E(t)\in S\) is the environment state, \(\{\tau(t)\}\in T\) is the task set, and \(A_E\) is the affordance map over the environment [1607.00354]. Here the temporal aspect enters through the evolving state \(s_E(t)\), while the output is a spatial distribution of task support.

In dynamic navigation, the STM is explicitly a stack of future spatial grids. A SOGM is described as a 3D grid map, interpretable as a stack of 2D occupancy grids over future time steps, with spatial resolution \(dl_\mathrm{2D}=12\) cm, temporal resolution \(dt=0.1\) s, and prediction horizon \(T=3.0\) s, giving \(n_T=31\) time steps [2108.10585].

In video representation learning, the map can be a matrix
\[
\mathcal{M} = [f_1^{\top}; f_2^{\top}; \cdots; f_T^{\top}] \in \mathbb{R}^{T \times L},
\]
where each row is the vectorized feature representation of one frame [1809.03669]. In weakly supervised action grounding, the STM is a tensor
\[
\mathbf{M} \in \mathbb{R}^{T \times H \times W},
\]
and the map values are normalized relevance scores computed from cosine similarity between local video features and token embeddings [2005.09183].

In symbolic dynamics, the spatiotemporal cat map uses an integer symbol field \(m_z\) over \(z=(n,t)\in\mathbb{Z}^2\), related linearly to the state field by
\[
x_{z}=\sum_{z'\in\mathbb{Z}^2} g_{zz'}\, m_{z'}.
\]
For finite windows, a rectangular block \(M_R\) determines the local state with exponentially decaying boundary error [1912.02940].

In dynamic MRI, the STM model is
\[
p(x,t) \approx \sum_{l=1}^{L(x)} s_l(x,t)\,p_l(x),
\]
where the temporal functions \(s_l(x,t)\) depend on spatial location [2507.14429]. This differs from the PSF model
\[
p(x,t) \approx \sum_{l=1}^{L_{\mathrm{PSF}}} y_l(t)\,p_l(x),
\]
whose temporal basis functions are spatially invariant [2507.14429].

In chaotic-map lattices, the relevant structured state can be a spatiotemporal fixed point satisfying
\[
x^1=x^2=\dots=x^N,\qquad x^{i}_{n+1}=x^{i}_{n}=x^*,
\]
under the nearest-neighbor coupled-map evolution
\[
x^{i}_{n+1} = (1-\epsilon)f(x^{i}_n) + \frac{\epsilon}{2K} \sum_{j=1}^{K} \big(g(x^{i-j}_n) + g(x^{i+j}_n) \big),
\]
with \(K=1\) in the reported experiments [1309.4555].

These formalisms show that STMs may be tensors, grids, maps, latent bases, symbol lattices, or dynamical invariant structures. The unifying element is that temporal evolution is embedded in the same representational object as spatial organization.

## 3. Construction mechanisms and learning pipelines

STM construction varies from direct analytical derivation to learned prediction.

In STAM, the framework contains an affordance description module holding \(\boldsymbol{\theta}\) and an environment module holding \(s_E(t)\). The affordance map may be hand-designed or learned from observations; the paper uses Gaussian Mixture Models and Gaussian Mixture Regression for a following task, with candidate mixtures up to 8 Gaussian components, initialization via k-means, and model selection via BIC [1607.00354].

In SOGM prediction, ground-truth maps are generated automatically from prior navigation data. Lidar points are labeled as ground, permanent, movable, or dynamic through point cloud SLAM and point cloud ray tracing, obstacle points are projected to the ground plane, and temporally stacked 2D point clouds are converted into SOGMs [2108.10585]. A 3D–2D feedforward network then predicts future SOGM slices from \(n_\mathrm{f}=3\) aligned lidar frames using a KPConv 3D back-end and a 2D U-Net front-end [2108.10585].

In video TSM, per-frame CNN features are vectorized and stacked into a VideoMap, then processed by a shallow head ConvNet with three convolution blocks. A hierarchical temporal attention mechanism modulates head-ConvNet feature maps through attention vectors at multiple resolutions [1809.03669].

In token-conditioned action highlighting, SlowFast features are projected by \(1\times1\times1\) 3D convolutions into motion and visual embedding spaces, and relevance maps are computed by softmax-normalized cosine similarities against verb and noun embeddings [2005.09183]. The maps are then used to weight local triplet losses, and a global joint embedding is learned simultaneously for retrieval [2005.09183].

In EEG classification, the sequence begins with coordinate transformation of 3D electrode locations into 2D image coordinates using t-SNE, followed by scaling and nearest-neighbor interpolation to form topological maps [2403.04353]. InternImage extracts spatial features frame by frame, the feature vectors are stacked chronologically, and an ST-pooling block inspired by PoolFormer performs 2D pooling over the resulting spatiotemporal plane [2403.04353].

In multitemporal remote sensing, per-class probability distribution maps are first produced by a random forest with 500 trees from segmentation and features including PCA features, DMPs, and morphological top-hat reconstruction features [2107.00590]. These maps then enter a 3D iterative spatiotemporal filter operating on spatial neighborhood, temporal neighborhood, and nDSM consistency [2107.00590].

In dynamic MRI reconstruction, STMs are estimated from autocalibration data. The method builds a calibration convolution matrix \(C\), computes a nullspace projector from \(C^HC\), derives local matrices \(G(x)\), and obtains voxelwise nullspace bases through orthogonal iteration [2507.14429]. The implementation uses ellipsoidal filter support, FFT-based PISCO-style computation, and a sketched SVD approximation for efficiency [2507.14429].

The contrastive-learning literature shows a related but distinct mechanism: spatiotemporal structure is not represented as a single explicit map object, but decoupled into spatial and temporal attention maps. In SDS-CL, the Spatiotemporal-decoupling Intra-Inter Attention module computes spatial-decoupling and temporal-decoupling intra- and inter-attention maps over joint and motion features, which are then used in frame-level, joint-level, and skeleton-level contrastive objectives [2302.02316].

## 4. Spatial-temporal coupling strategies

A central design axis in STM research is how the spatial and temporal dimensions are coupled.

One strategy is **explicit stacking**. VideoMap stacks frame features row by row in temporal order, so temporal order is represented by row position and spatial semantics are inherited from the CNN feature vectors [1809.03669]. SOGMs stack future occupancy grids in time [2108.10585]. In EEG, temporal windows are converted into a sequence of topological images and then into a 2D feature plane for pooling [2403.04353].

A second strategy is **time-conditioned map generation**. STAM recomputes the affordance field as \(s_E(t)\) changes, so the map is spatial at any instant but temporally indexed through the environment state [1607.00354]. The same logic appears in satellite refinement, where each date’s probability map is a temporal slice of a larger probability volume refined by a 3D filter [2107.00590].

A third strategy is **joint spatiotemporal attention or relevance distributions**. In action highlighting, the motion and visual STMs are normalized distributions over \(T\times H\times W\), conditioned on verbs and nouns respectively [2005.09183]. In SDS-CL, spatial and temporal attention are explicitly decoupled into \(N\times N\) and \(T\times T\) maps rather than mixed into one global representation [2302.02316].

A fourth strategy is **spatially varying temporal subspaces**. Dynamic MRI STMs do not merely stack time frames; instead, they assign each spatial location its own temporal basis via \(s_l(x,t)\) [2507.14429]. This addresses the stated limitation of global low-rank models when temporal behavior is spatially heterogeneous.

A fifth strategy is **linear symbolic encoding over spacetime**. In the spatiotemporal cat map, a finite 2D symbol block both represents local spacetime structure and determines interior states with error bounded by
\[
|x_z-\bar{x}_z|\le C e^{-\nu\,\ell(z,\partial R)},
\]
where \(\cosh \nu=s/2\) [1912.02940]. This is a particularly strong form of coupling because the symbolic spacetime pattern is simultaneously descriptive and reconstructive.

These coupling strategies imply different inductive biases. Explicit stacking preserves order but may not model physical constraints. Time-conditioned maps support control and planning. Spatially varying temporal bases accommodate local heterogeneity. Symbolic codes support exact or asymptotically exact reconstruction in special dynamical systems.

## 5. Applications across domains

The STM literature spans a wide range of domains, with differing objectives and evaluation criteria.

### Representative STM formulations across domains

| Domain | STM object | Primary role |
|---|---|---|
| Robotics | STAM or SOGM | Task semantics and future occupancy for navigation [1607.00354, 2108.10585] |
| Video understanding | VideoMap or relevance tensor | Action recognition, retrieval, and grounding [1809.03669, 2005.09183] |
| Remote sensing | Time-indexed probability maps | Postclassification temporal refinement [2107.00590] |
| MRI | Spatially varying temporal basis maps | Accelerated dynamic reconstruction or voxelwise parameter mapping [2507.14429, 2312.05279] |
| Chaotic lattices | Spatiotemporal fixed points or symbolic blocks | Stability analysis and symbolic description [1309.4555, 1912.02940] |
| EEG | Sequence of topological maps | Motor imagery classification [2403.04353] |

In robotics, STAM is used to support robot tasks such as following by learning where a follower should be relative to a target pose \(\langle x_T,y_T,\alpha_T\rangle\) and \(\langle x_F,y_F,\alpha_F\rangle\) [1607.00354]. SOGMs are used to generate Spatiotemporal Risk Maps for a modified Timed Elastic Band planner, allowing proactive local planning in dynamic scenes [2108.10585].

In video analysis, TSM improves action recognition by exposing all frames jointly to a shallow ConvNet rather than averaging sparse per-segment scores [1809.03669]. Action highlighting uses caption supervision to produce different STMs for different nouns and verbs, allowing fine-grained “where and when” grounding in untrimmed videos [2005.09183].

In remote sensing, the STM-like object is a stack of per-class probabilities across dates, refined using spectral similarity, spatial proximity, and nDSM consistency to improve temporal robustness of land-cover classification [2107.00590].

In medical imaging, ST-Net maps DSC-MRI patches to CBV, CBF, and Tmax while combining 3D spatial context with temporal dynamics and a physics-informed loss [2312.05279]. Dynamic MRI STMs provide a reconstruction model for undersampled time series, with proof-of-concept results on 2D single-channel animal gastrointestinal MRI and 3D multichannel human fMRI [2507.14429].

In dynamical systems, the term refers less to data structures and more to emergent spacetime organization. Nonlinear coupling in coupled logistic-map lattices stabilizes a spatiotemporal fixed point over broader coupling intervals as the nonlinearity parameter \(q\) increases [1309.4555]. In coupled Hénon–Lozi ensembles, the network exhibits phase chimeras, amplitude chimeras, solitary states, a solitary-state chimera, and mutual synchronization of spatiotemporal structures [1710.10077].

This breadth complicates any attempt at a single universal definition. A plausible implication is that STM research is better organized by representational function—control map, feature map, latent basis, symbolic code, or emergent structure—than by terminology alone.

## 6. Empirical results and domain-specific evidence

Quantitative evidence for STM effectiveness is strongly domain-dependent.

In robot navigation, SOGM prediction on the Flow Followers dataset achieved, for the best reported model 3D-2D (4,4,3), \( \mathrm{AP}_{1.0}=31.0\% \), \( \mathrm{AP}_{2.0}=11.5\% \), \( \mathrm{AP}_{\mathrm{tot}}=27.9\% \), and \( \mathrm{MSE}=3.72 \) [2108.10585]. The chosen faster model (3,2,2) achieved \(28.1\% / 8.9\% / 26.1\% / 3.77\) [2108.10585]. The forward pass took less than 50 ms on an RTX 3090, preprocessing around 200 ms, and SRM conversion around 30 ms [2108.10585].

In TSM for action recognition, the full TSN+TSM model with attention reached 72.7% on HMDB51 and 94.3% on UCF101, with a 4.2% absolute improvement over TSN on HMDB51 [1809.03669]. On HMDB51, TSN (BN-Inception) achieved 68.5%, TSN+TSM without attention 72.2%, and TSN+TSM with attention 72.7% [1809.03669].

In action highlighting, the full model on MSR-VTT improved retrieval recall by about 2–3% over a baseline without alignment [2005.09183]. The reported R@1 values were 5.2 for video-to-caption and 5.3 for caption-to-video, versus 3.6 and 3.1 for the baseline without alignment [2005.09183].

In EEG classification, the proposed topological-map plus ST-pooling model achieved 88.57%, 80.65%, and 70.17% on two-, three-, and four-class motor imagery tasks in cross-individual validation [2403.04353]. The 4-class ablation showed 70.17% for t-SNE versus 69.30% for parallel projection, 68.71% for azimuthal equidistant projection, and 68.42% for UMAP [2403.04353]. ST-pooling achieved 70.17%, compared with 69.88% for multi-head attention and 69.74% for PoolFormer [2403.04353].

In multitemporal satellite classification, the 3D iterative spatiotemporal filter consistently improved individual classification results by 2% to 6% overall accuracy [2107.00590]. Average improvements by region were +4.24%, +5.29%, and +4.72%, and convergence occurred in about 5 iterations on average [2107.00590].

In DSC-MRI perfusion mapping, ST-Net achieved mean SSIM values of 0.952 for CBV, 0.943 for CBF, and 0.863 for Tmax, with a DICE score of 0.859 for the hypo-perfused region [2312.05279]. For 10,000 voxels, traditional deconvolution took 6.9 s and ST-Net 0.77 s [2312.05279].

In dynamic MRI reconstruction, STM representation was more parsimonious than PSF on rat GI MRI: STM with \(L=4\) components achieved the same representation quality that PSF needed about \(L_{\mathrm{PSF}}=6\) components to match [2507.14429]. In 3D fMRI, sketched SVD reduced STM computation time from about 115 minutes to about 0.18 minutes, about 640× faster, with nearly identical STM quality [2507.14429].

In coupled-map lattices, nonlinear coupling produced analytically derived stability windows for quadratic coupling: the zero fixed point was stable for \(0.75 \le \epsilon \le 1\), and the nonzero fixed point for \(0.61 \le \epsilon \le 0.75\) [1309.4555]. The synchronization range increased with \(q\), with a least-squares fit
\[
\epsilon_{sync} = \frac{a}{q^b} + c,
\]
where \(a=0.977\pm0.063\), \(b=1.1415\pm0.078\), and \(c=0.251\pm0.004\) [1309.4555].

These results show that STM methods are rarely evaluated by a common benchmark family. Instead, they are assessed against the practical objective of the domain: accuracy, AP, SSIM, DICE, NRMSE, stability interval, or symbolic precision.

## 7. Limitations, misconceptions, and open directions

A common misconception is that any method using both space and time automatically defines an STM. Several papers explicitly resist that simplification. The multi-camera motion-capture work based on Part Confidence Maps performs spatiotemporal filtering with multi-view spatial fusion, temporal IIR smoothing, and inverse kinematics, but does **not** define a map-like STM object analogous to an STM representation [1912.03880]. This distinction is important: spatiotemporal processing is broader than STM representation.

Another misconception is that STMs are always dense tensors over raw coordinates. In practice, they may instead be symbolic arrays [1912.02940], probability maps [2107.00590], occupancy stacks [2108.10585], attention maps [2302.02316], or local temporal bases indexed by spatial location [2507.14429].

The literature also makes clear that richer spatiotemporal structure brings additional assumptions and costs. STAM assumes the environment can be represented in a suitable underlying map and that the affordance signature can be learned or specified [1607.00354]. SOGMs assume fixed spatial and temporal discretization and were evaluated in simulation rather than real-world crowded deployments [2108.10585]. The satellite filter depends on accurate multitemporal registration and class-dependent nDSM bandwidths [2107.00590]. ST-Net relies on manually delineated arterial and venous references and on gold-standard parameter maps from RAPID [2312.05279]. Dynamic MRI STMs require autocalibration data and depend on local multiband or shift-invariant linear predictability assumptions in \((k,t)\)-space [2507.14429].

In representation learning, STM-like maps may preserve order but still compress away information. VideoMap is built from frame-level feature vectors rather than raw spatial fields, so fine spatial detail is lost before temporal modeling [1809.03669]. In action highlighting, motion alignment is noted to be harder than visual alignment because verbs do not always correspond to a single obvious local region [2005.09183].

In dynamical systems, the notion of an STM may refer to emergent structures rather than engineered representations. This suggests a conceptual divide between **descriptive STMs**, which explain observed spacetime organization, and **constructive STMs**, which are deliberately designed to support inference or control. A plausible implication is that future unification efforts will need to bridge these two traditions.

Several forward directions are explicit in the literature. STAM was demonstrated only on a simple following case and not yet on large-scale multi-task real-world validation [1607.00354]. SOGM work identifies future study of more complex human behavior, conditioning predictions on future robot actions, and visible-space reasoning [2108.10585]. EEG topological-map methods suggest evaluation on tasks beyond motor imagery [2403.04353]. Dynamic MRI STMs suggest combining the model with deep learning, spatially varying \(L(x)\), and regional encoding [2507.14429].

Taken together, these works show that STM research is not unified by a single tensor shape or algorithmic recipe. It is unified by an insistence that spatial organization and temporal evolution must be encoded together in a representation whose structure is consequential for inference. Whether the result is a gainmap, an occupancy volume, a symbolic lattice, a relevance tensor, or a voxelwise temporal basis, the central claim is the same: when the representation itself respects spacetime structure, downstream modeling can become more predictive, more stable, or more physically faithful [1607.00354, 2108.10585, 1912.02940, 2507.14429].

Source: https://www.emergentmind.com/topics/spatiotemporal-maps-stms