Spatiotemporal Maps in Research

• Spatiotemporal maps (STMs) are integrated representations that jointly encode spatial structure and temporal dynamics for predictive and control tasks.
• Construction strategies include explicit stacking, time-conditioned generation, joint attention mechanisms, and spatially varying temporal bases.
• Applications span robotics, video analysis, MRI, remote sensing, and dynamical systems, enhancing inference and reconstruction in each domain.

Searching arXiv for recent and foundational papers on “spatiotemporal maps” and closely related STM usages. Spatiotemporal maps (STMs) are representations that bind spatial structure to temporal evolution within a single formal object, but the term is used in several technically distinct ways across the literature. In some settings, an STM is a state representation over space and time, such as a stack of future occupancy grids for robot navigation (Thomas et al., 2021) or a time-varying affordance field over an environment (Riccio et al., 2016). In others, it is a feature representation that converts a sequence into a map-like array for subsequent inference, as in VideoMap for action recognition (Song et al., 2018), token-conditioned relevance maps in untrimmed video (Kasai et al., 2020), or topological EEG image sequences processed by spatiotemporal pooling (Fukushima et al., 2024). The term also appears in dynamical-systems and signal-processing contexts, where it denotes spatiotemporal structures in coupled maps (Kumar, 2013, Bukh et al., 2017), symbolic encodings of chaotic lattices (Gutkin et al., 2019), or spatially varying temporal bases for dynamic MRI reconstruction (Lobos et al., 19 Jul 2025). Across these usages, the common principle is that spatial and temporal dependencies are not treated as separable afterthoughts, but are encoded jointly in a representation that supports prediction, reconstruction, control, or interpretation.

1. Conceptual scope and major usages

The STM concept is not a single standardized formalism. The literature instead shows several families of usage.

A first family treats an STM as a time-indexed map of environment state or task-relevant likelihoods. In robotics, a Spatio-Temporal Affordance Map (STAM) represents the likelihood that each area of an environment affords a task under the current world state (Riccio et al., 2016). In dynamic navigation, a Spatiotemporal Occupancy Grid Map (SOGM) is defined as a 3D grid map whose cells contain occupancy probabilities for given positions and future times (Thomas et al., 2021).

A second family treats an STM as a map-like feature representation of sequences. Temporal-Spatial Mapping (TSM) converts a video into a 2D VideoMap by stacking frame-level CNN features in temporal order (Song et al., 2018). In cross-modal retrieval and action grounding, spatiotemporal relevance maps are tensors over time, height, and width that indicate where and when caption tokens are supported in an untrimmed video (Kasai et al., 2020). For EEG, the signal is converted into a sequence of 2D topological maps and then processed by a spatiotemporal pooling block that operates on a stacked temporal-feature plane (Fukushima et al., 2024).

A third family uses STM language in dynamical systems and symbolic dynamics. Coupled chaotic lattices are analyzed through spatiotemporal fixed points, regularity, chimera states, and synchronization of spatiotemporal structures (Kumar, 2013, Bukh et al., 2017). In the spatiotemporal cat map, local system states over finite spacetime windows are encoded by finite 2D symbol blocks that have both reconstructive and statistical meaning (Gutkin et al., 2019).

A fourth family uses STM as a signal model or reconstruction prior. In dynamic MRI, STMs generalize the partially separable functions model by allowing temporal basis functions to depend on spatial location (Lobos et al., 19 Jul 2025). In DSC-MRI perfusion estimation, a spatiotemporal CNN maps 4D image sequences to voxel-wise perfusion parameter maps while explicitly separating spatial and temporal modules (Cao et al., 2023). In multitemporal satellite analysis, per-class probability maps across dates are refined by a 3D iterative spatiotemporal filter over space, time, and height (Albanwan et al., 2021).

This diversity suggests that “STM” is best understood as a class of representations rather than a single architecture. A plausible implication is that the defining criterion is not the data modality, but whether a method represents spatial structure and temporal evolution in a coupled form usable for downstream inference.

2. Formal representations and mathematical structures

The mathematical form of an STM depends on the application.

In robot affordance modeling, the formal object is a function

$$f_{E,\boldsymbol{\theta}} : S \times T \rightarrow A_E$$

where $E$ is the environment, $\boldsymbol{\theta}$ are affordance parameters, $s_E(t)\in S$ is the environment state, $\{\tau(t)\}\in T$ is the task set, and $A_E$ is the affordance map over the environment (Riccio et al., 2016). Here the temporal aspect enters through the evolving state $s_E(t)$, while the output is a spatial distribution of task support.

In dynamic navigation, the STM is explicitly a stack of future spatial grids. A SOGM is described as a 3D grid map, interpretable as a stack of 2D occupancy grids over future time steps, with spatial resolution $dl_\mathrm{2D}=12$ cm, temporal resolution $dt=0.1$ s, and prediction horizon $T=3.0$ s, giving $E$0 time steps (Thomas et al., 2021).

In video representation learning, the map can be a matrix

$E$1

where each row is the vectorized feature representation of one frame (Song et al., 2018). In weakly supervised action grounding, the STM is a tensor

$E$2

and the map values are normalized relevance scores computed from cosine similarity between local video features and token embeddings (Kasai et al., 2020).

In symbolic dynamics, the spatiotemporal cat map uses an integer symbol field $E$3 over $E$4, related linearly to the state field by

$E$5

For finite windows, a rectangular block $E$6 determines the local state with exponentially decaying boundary error (Gutkin et al., 2019).

In dynamic MRI, the STM model is

$E$7

where the temporal functions $E$8 depend on spatial location (Lobos et al., 19 Jul 2025). This differs from the PSF model

$E$9

whose temporal basis functions are spatially invariant (Lobos et al., 19 Jul 2025).

In chaotic-map lattices, the relevant structured state can be a spatiotemporal fixed point satisfying

$\boldsymbol{\theta}$0

under the nearest-neighbor coupled-map evolution

$\boldsymbol{\theta}$1

with $\boldsymbol{\theta}$2 in the reported experiments (Kumar, 2013).

These formalisms show that STMs may be tensors, grids, maps, latent bases, symbol lattices, or dynamical invariant structures. The unifying element is that temporal evolution is embedded in the same representational object as spatial organization.

3. Construction mechanisms and learning pipelines

STM construction varies from direct analytical derivation to learned prediction.

In STAM, the framework contains an affordance description module holding $\boldsymbol{\theta}$3 and an environment module holding $\boldsymbol{\theta}$4. The affordance map may be hand-designed or learned from observations; the paper uses Gaussian Mixture Models and Gaussian Mixture Regression for a following task, with candidate mixtures up to 8 Gaussian components, initialization via k-means, and model selection via BIC (Riccio et al., 2016).

In SOGM prediction, ground-truth maps are generated automatically from prior navigation data. Lidar points are labeled as ground, permanent, movable, or dynamic through point cloud SLAM and point cloud ray tracing, obstacle points are projected to the ground plane, and temporally stacked 2D point clouds are converted into SOGMs (Thomas et al., 2021). A 3D–2D feedforward network then predicts future SOGM slices from $\boldsymbol{\theta}$5 aligned lidar frames using a KPConv 3D back-end and a 2D U-Net front-end (Thomas et al., 2021).

In video TSM, per-frame CNN features are vectorized and stacked into a VideoMap, then processed by a shallow head ConvNet with three convolution blocks. A hierarchical temporal attention mechanism modulates head-ConvNet feature maps through attention vectors at multiple resolutions (Song et al., 2018).

In token-conditioned action highlighting, SlowFast features are projected by $\boldsymbol{\theta}$6 3D convolutions into motion and visual embedding spaces, and relevance maps are computed by softmax-normalized cosine similarities against verb and noun embeddings (Kasai et al., 2020). The maps are then used to weight local triplet losses, and a global joint embedding is learned simultaneously for retrieval (Kasai et al., 2020).

In EEG classification, the sequence begins with coordinate transformation of 3D electrode locations into 2D image coordinates using t-SNE, followed by scaling and nearest-neighbor interpolation to form topological maps (Fukushima et al., 2024). InternImage extracts spatial features frame by frame, the feature vectors are stacked chronologically, and an ST-pooling block inspired by PoolFormer performs 2D pooling over the resulting spatiotemporal plane (Fukushima et al., 2024).

In multitemporal remote sensing, per-class probability distribution maps are first produced by a random forest with 500 trees from segmentation and features including PCA features, DMPs, and morphological top-hat reconstruction features (Albanwan et al., 2021). These maps then enter a 3D iterative spatiotemporal filter operating on spatial neighborhood, temporal neighborhood, and nDSM consistency (Albanwan et al., 2021).

In dynamic MRI reconstruction, STMs are estimated from autocalibration data. The method builds a calibration convolution matrix $\boldsymbol{\theta}$7, computes a nullspace projector from $\boldsymbol{\theta}$8, derives local matrices $\boldsymbol{\theta}$9, and obtains voxelwise nullspace bases through orthogonal iteration (Lobos et al., 19 Jul 2025). The implementation uses ellipsoidal filter support, FFT-based PISCO-style computation, and a sketched SVD approximation for efficiency (Lobos et al., 19 Jul 2025).

The contrastive-learning literature shows a related but distinct mechanism: spatiotemporal structure is not represented as a single explicit map object, but decoupled into spatial and temporal attention maps. In SDS-CL, the Spatiotemporal-decoupling Intra-Inter Attention module computes spatial-decoupling and temporal-decoupling intra- and inter-attention maps over joint and motion features, which are then used in frame-level, joint-level, and skeleton-level contrastive objectives (Xu et al., 2023).

4. Spatial-temporal coupling strategies

A central design axis in STM research is how the spatial and temporal dimensions are coupled.

One strategy is explicit stacking. VideoMap stacks frame features row by row in temporal order, so temporal order is represented by row position and spatial semantics are inherited from the CNN feature vectors (Song et al., 2018). SOGMs stack future occupancy grids in time (Thomas et al., 2021). In EEG, temporal windows are converted into a sequence of topological images and then into a 2D feature plane for pooling (Fukushima et al., 2024).

A second strategy is time-conditioned map generation. STAM recomputes the affordance field as $s_E(t)\in S$0 changes, so the map is spatial at any instant but temporally indexed through the environment state (Riccio et al., 2016). The same logic appears in satellite refinement, where each date’s probability map is a temporal slice of a larger probability volume refined by a 3D filter (Albanwan et al., 2021).

A third strategy is joint spatiotemporal attention or relevance distributions. In action highlighting, the motion and visual STMs are normalized distributions over $s_E(t)\in S$1, conditioned on verbs and nouns respectively (Kasai et al., 2020). In SDS-CL, spatial and temporal attention are explicitly decoupled into $s_E(t)\in S$2 and $s_E(t)\in S$3 maps rather than mixed into one global representation (Xu et al., 2023).

A fourth strategy is spatially varying temporal subspaces. Dynamic MRI STMs do not merely stack time frames; instead, they assign each spatial location its own temporal basis via $s_E(t)\in S$4 (Lobos et al., 19 Jul 2025). This addresses the stated limitation of global low-rank models when temporal behavior is spatially heterogeneous.

A fifth strategy is linear symbolic encoding over spacetime. In the spatiotemporal cat map, a finite 2D symbol block both represents local spacetime structure and determines interior states with error bounded by

$s_E(t)\in S$5

where $s_E(t)\in S$6 (Gutkin et al., 2019). This is a particularly strong form of coupling because the symbolic spacetime pattern is simultaneously descriptive and reconstructive.

These coupling strategies imply different inductive biases. Explicit stacking preserves order but may not model physical constraints. Time-conditioned maps support control and planning. Spatially varying temporal bases accommodate local heterogeneity. Symbolic codes support exact or asymptotically exact reconstruction in special dynamical systems.

5. Applications across domains

The STM literature spans a wide range of domains, with differing objectives and evaluation criteria.

Representative STM formulations across domains

Domain	STM object	Primary role
Robotics	STAM or SOGM	Task semantics and future occupancy for navigation (Riccio et al., 2016, Thomas et al., 2021)
Video understanding	VideoMap or relevance tensor	Action recognition, retrieval, and grounding (Song et al., 2018, Kasai et al., 2020)
Remote sensing	Time-indexed probability maps	Postclassification temporal refinement (Albanwan et al., 2021)
MRI	Spatially varying temporal basis maps	Accelerated dynamic reconstruction or voxelwise parameter mapping (Lobos et al., 19 Jul 2025, Cao et al., 2023)
Chaotic lattices	Spatiotemporal fixed points or symbolic blocks	Stability analysis and symbolic description (Kumar, 2013, Gutkin et al., 2019)
EEG	Sequence of topological maps	Motor imagery classification (Fukushima et al., 2024)

In robotics, STAM is used to support robot tasks such as following by learning where a follower should be relative to a target pose $s_E(t)\in S$7 and $s_E(t)\in S$8 (Riccio et al., 2016). SOGMs are used to generate Spatiotemporal Risk Maps for a modified Timed Elastic Band planner, allowing proactive local planning in dynamic scenes (Thomas et al., 2021).

In video analysis, TSM improves action recognition by exposing all frames jointly to a shallow ConvNet rather than averaging sparse per-segment scores (Song et al., 2018). Action highlighting uses caption supervision to produce different STMs for different nouns and verbs, allowing fine-grained “where and when” grounding in untrimmed videos (Kasai et al., 2020).

In remote sensing, the STM-like object is a stack of per-class probabilities across dates, refined using spectral similarity, spatial proximity, and nDSM consistency to improve temporal robustness of land-cover classification (Albanwan et al., 2021).

In medical imaging, ST-Net maps DSC-MRI patches to CBV, CBF, and Tmax while combining 3D spatial context with temporal dynamics and a physics-informed loss (Cao et al., 2023). Dynamic MRI STMs provide a reconstruction model for undersampled time series, with proof-of-concept results on 2D single-channel animal gastrointestinal MRI and 3D multichannel human fMRI (Lobos et al., 19 Jul 2025).

In dynamical systems, the term refers less to data structures and more to emergent spacetime organization. Nonlinear coupling in coupled logistic-map lattices stabilizes a spatiotemporal fixed point over broader coupling intervals as the nonlinearity parameter $s_E(t)\in S$9 increases (Kumar, 2013). In coupled Hénon–Lozi ensembles, the network exhibits phase chimeras, amplitude chimeras, solitary states, a solitary-state chimera, and mutual synchronization of spatiotemporal structures (Bukh et al., 2017).

This breadth complicates any attempt at a single universal definition. A plausible implication is that STM research is better organized by representational function—control map, feature map, latent basis, symbolic code, or emergent structure—than by terminology alone.

6. Empirical results and domain-specific evidence

Quantitative evidence for STM effectiveness is strongly domain-dependent.

In robot navigation, SOGM prediction on the Flow Followers dataset achieved, for the best reported model 3D-2D (4,4,3), $\{\tau(t)\}\in T$0, $\{\tau(t)\}\in T$1, $\{\tau(t)\}\in T$2, and $\{\tau(t)\}\in T$3 (Thomas et al., 2021). The chosen faster model (3,2,2) achieved $\{\tau(t)\}\in T$4 (Thomas et al., 2021). The forward pass took less than 50 ms on an RTX 3090, preprocessing around 200 ms, and SRM conversion around 30 ms (Thomas et al., 2021).

In TSM for action recognition, the full TSN+TSM model with attention reached 72.7% on HMDB51 and 94.3% on UCF101, with a 4.2% absolute improvement over TSN on HMDB51 (Song et al., 2018). On HMDB51, TSN (BN-Inception) achieved 68.5%, TSN+TSM without attention 72.2%, and TSN+TSM with attention 72.7% (Song et al., 2018).

In action highlighting, the full model on MSR-VTT improved retrieval recall by about 2–3% over a baseline without alignment (Kasai et al., 2020). The reported R@1 values were 5.2 for video-to-caption and 5.3 for caption-to-video, versus 3.6 and 3.1 for the baseline without alignment (Kasai et al., 2020).

In EEG classification, the proposed topological-map plus ST-pooling model achieved 88.57%, 80.65%, and 70.17% on two-, three-, and four-class motor imagery tasks in cross-individual validation (Fukushima et al., 2024). The 4-class ablation showed 70.17% for t-SNE versus 69.30% for parallel projection, 68.71% for azimuthal equidistant projection, and 68.42% for UMAP (Fukushima et al., 2024). ST-pooling achieved 70.17%, compared with 69.88% for multi-head attention and 69.74% for PoolFormer (Fukushima et al., 2024).

In multitemporal satellite classification, the 3D iterative spatiotemporal filter consistently improved individual classification results by 2% to 6% overall accuracy (Albanwan et al., 2021). Average improvements by region were +4.24%, +5.29%, and +4.72%, and convergence occurred in about 5 iterations on average (Albanwan et al., 2021).

In DSC-MRI perfusion mapping, ST-Net achieved mean SSIM values of 0.952 for CBV, 0.943 for CBF, and 0.863 for Tmax, with a DICE score of 0.859 for the hypo-perfused region (Cao et al., 2023). For 10,000 voxels, traditional deconvolution took 6.9 s and ST-Net 0.77 s (Cao et al., 2023).

In dynamic MRI reconstruction, STM representation was more parsimonious than PSF on rat GI MRI: STM with $\{\tau(t)\}\in T$5 components achieved the same representation quality that PSF needed about $\{\tau(t)\}\in T$6 components to match (Lobos et al., 19 Jul 2025). In 3D fMRI, sketched SVD reduced STM computation time from about 115 minutes to about 0.18 minutes, about 640× faster, with nearly identical STM quality (Lobos et al., 19 Jul 2025).

In coupled-map lattices, nonlinear coupling produced analytically derived stability windows for quadratic coupling: the zero fixed point was stable for $\{\tau(t)\}\in T$7, and the nonzero fixed point for $\{\tau(t)\}\in T$8 (Kumar, 2013). The synchronization range increased with $\{\tau(t)\}\in T$9, with a least-squares fit

$A_E$0

where $A_E$1, $A_E$2, and $A_E$3 (Kumar, 2013).

These results show that STM methods are rarely evaluated by a common benchmark family. Instead, they are assessed against the practical objective of the domain: accuracy, AP, SSIM, DICE, NRMSE, stability interval, or symbolic precision.

7. Limitations, misconceptions, and open directions

A common misconception is that any method using both space and time automatically defines an STM. Several papers explicitly resist that simplification. The multi-camera motion-capture work based on Part Confidence Maps performs spatiotemporal filtering with multi-view spatial fusion, temporal IIR smoothing, and inverse kinematics, but does not define a map-like STM object analogous to an STM representation (Ohashi et al., 2019). This distinction is important: spatiotemporal processing is broader than STM representation.

Another misconception is that STMs are always dense tensors over raw coordinates. In practice, they may instead be symbolic arrays (Gutkin et al., 2019), probability maps (Albanwan et al., 2021), occupancy stacks (Thomas et al., 2021), attention maps (Xu et al., 2023), or local temporal bases indexed by spatial location (Lobos et al., 19 Jul 2025).

The literature also makes clear that richer spatiotemporal structure brings additional assumptions and costs. STAM assumes the environment can be represented in a suitable underlying map and that the affordance signature can be learned or specified (Riccio et al., 2016). SOGMs assume fixed spatial and temporal discretization and were evaluated in simulation rather than real-world crowded deployments (Thomas et al., 2021). The satellite filter depends on accurate multitemporal registration and class-dependent nDSM bandwidths (Albanwan et al., 2021). ST-Net relies on manually delineated arterial and venous references and on gold-standard parameter maps from RAPID (Cao et al., 2023). Dynamic MRI STMs require autocalibration data and depend on local multiband or shift-invariant linear predictability assumptions in $A_E$4-space (Lobos et al., 19 Jul 2025).

In representation learning, STM-like maps may preserve order but still compress away information. VideoMap is built from frame-level feature vectors rather than raw spatial fields, so fine spatial detail is lost before temporal modeling (Song et al., 2018). In action highlighting, motion alignment is noted to be harder than visual alignment because verbs do not always correspond to a single obvious local region (Kasai et al., 2020).

In dynamical systems, the notion of an STM may refer to emergent structures rather than engineered representations. This suggests a conceptual divide between descriptive STMs, which explain observed spacetime organization, and constructive STMs, which are deliberately designed to support inference or control. A plausible implication is that future unification efforts will need to bridge these two traditions.

Several forward directions are explicit in the literature. STAM was demonstrated only on a simple following case and not yet on large-scale multi-task real-world validation (Riccio et al., 2016). SOGM work identifies future study of more complex human behavior, conditioning predictions on future robot actions, and visible-space reasoning (Thomas et al., 2021). EEG topological-map methods suggest evaluation on tasks beyond motor imagery (Fukushima et al., 2024). Dynamic MRI STMs suggest combining the model with deep learning, spatially varying $A_E$5, and regional encoding (Lobos et al., 19 Jul 2025).

Taken together, these works show that STM research is not unified by a single tensor shape or algorithmic recipe. It is unified by an insistence that spatial organization and temporal evolution must be encoded together in a representation whose structure is consequential for inference. Whether the result is a gainmap, an occupancy volume, a symbolic lattice, a relevance tensor, or a voxelwise temporal basis, the central claim is the same: when the representation itself respects spacetime structure, downstream modeling can become more predictive, more stable, or more physically faithful (Riccio et al., 2016, Thomas et al., 2021, Gutkin et al., 2019, Lobos et al., 19 Jul 2025).