Spatial-Temporal Extrapolation

• Spatial-temporal extrapolation is the process of predicting missing states by integrating spatial geometry, such as novel view synthesis, with temporal dynamics like video prediction.
• Researchers apply diverse methods—from self-supervised video synthesis to parallel encoding in radar nowcasting—to contend with challenges like error accumulation and nonstationarity.
• Practical applications span computer vision, sensor networks, and dynamical systems, demonstrating the value of merging spatial locality with temporal evolution.

Spatial-temporal extrapolation denotes the estimation, prediction, or synthesis of unobserved states across both space and time from partial observations. In computer vision, it includes generating a scene from a new point of view or a new point of time; in sensor, graph, and dynamical-system settings, it includes predicting values at target locations, future times, or both; in statistical formulations, it includes estimating future probability distributions or latent fields under explicit smoothness and sampling assumptions. A common thread is that spatial observations constrain geometry, structure, or locality, while temporal observations constrain dynamics, causality, or evolution. Recent work treats these signals as either complementary or partially independent, depending on the domain and representation (Zhang et al., 2022, Hu et al., 2023, Nayak et al., 22 May 2025).

1. Problem scope and formal definitions

A canonical formulation appears in “Video Extrapolation in Space and Time” (Zhang et al., 2022). There, Novel View Synthesis (NVS) synthesizes a scene from a new point of view, while Video Prediction (VP) synthesizes a scene from a new point of time; the joint problem is to synthesize appearance from new viewpoints and/or new timepoints. With images $\mathcal{I}_{v,t}$, observations $\mathcal{O} = \{\mathcal{I}_{v_i,t_j}\}_{i,j}$, and an unseen query $(v^*, t^*)$, the target is $\mathcal{I}_{v^*, t^*}$ (Zhang et al., 2022).

A graph-based formulation appears in “Graph Neural Processes for Spatio-Temporal Extrapolation” (Hu et al., 2023), which generates data at target locations from surrounding contexts in a graph. This setting is motivated by sparsely deployed sensors and the resulting lack of fine-grained information. A statistical forecasting formulation appears in “Learning Localized Spatio-Temporal Models From Streaming Data” (Osama et al., 2018), which predicts spatio-temporal processes with temporal patterns that vary across spatial regions when data is obtained as a stream. A theoretical distributional formulation appears in “A Temporal Spatial Minimax Rate for Smoothly-Varying Distributions in Wasserstein Space” (Kim, 5 Jun 2026), which estimates a future value $\mu_{t_n+h}$ of a curve $t \mapsto \mu_t$ in the $2$-Wasserstein space from finitely many noisy snapshots of its past.

A representative cross-domain summary is given below.

Setting	Extrapolated quantity	Representative work
Video synthesis	$\mathcal{I}_{v^*, t^*}$ from observed views and times	VEST (Zhang et al., 2022)
Graph sensing	Target-node values from context nodes	STGNP (Hu et al., 2023)
Radar nowcasting	Future radar echoes from past echo sequences	TempEE (Chen et al., 2023), SFTformer (Xu et al., 2024)
Dynamical systems	Future states beyond the training time horizon	TI-DeepONet (Nayak et al., 22 May 2025)
Streaming statistics	Missing values in spatial regions over time	Localized covariance model (Osama et al., 2018)
Wasserstein forecasting	Future distribution $\mu_{t_n+h}$	Minimax analysis (Kim, 5 Jun 2026)

In some later work, “spatial-temporal” is extended beyond Euclidean space. “STK-Adapter” models spatial structures and temporal patterns inherent in Temporal Knowledge Graphs through a Spatial-Temporal MoE (Zhao et al., 21 Apr 2026), and “Efficient Diffusion LLMs via Temporal-Spatial Parallel Decoding and Confidence Extrapolation” uses token position together with denoising trajectories as spatial and temporal signals for control during decoding (Li et al., 29 May 2026). This suggests that the concept has broadened from physical fields and images to structured and sequential state spaces.

2. Representations: coupling, decoupling, and locality

A central design choice is whether spatial and temporal structure should be learned jointly, separately, or in a partially decoupled form. VEST is motivated by the claim that NVS and VP provide complementary signals: viewpoint changes from spatial observations inform depth, and temporal observations inform the motion of cameras and individual objects (Zhang et al., 2022). In that formulation, spatial-temporal extrapolation is not merely multi-task learning; it is an attempt to recover a scene representation from both parallax and motion.

Radar nowcasting work makes the same issue explicit but reaches different architectural conclusions. “TempEE: Temporal-Spatial Parallel Transformer for Radar Echo Extrapolation Beyond Auto-Regression” uses a parallel encoder to extract spatio-temporal representations from continuous echo images, with a Multi-level Temporal-Spatial Attention mechanism to capture both global and local information and emphasize task-related regions, including sparse echo representations (Chen et al., 2023). By contrast, “SFTformer” starts from the claim that the spatial morphology and temporal evolution of radar echoes are correlated yet also possess independent characteristics; it therefore introduces a correlation-decoupling Transformer in which stacked SFT-Blocks model spatiotemporal correlation, spatial refinement, and temporal modeling separately, including a Frequency Enhancement Block in the temporal path (Xu et al., 2024).

Traffic forecasting offers a third stance. “Extralonger” argues that the prediction horizon is limited mainly due to the separation of temporal and spatial factors, and introduces a unified representation that reduces complexity from $\mathcal{O}(NT^2 + TN^2)$ to $\mathcal{O} = \{\mathcal{I}_{v_i,t_j}\}_{i,j}$0 while preserving a full receptive field (Zhang et al., 2024). Here the space-time distinction is not removed conceptually, but the computational pipeline is reorganized so that time and space are processed together at the representation level.

Localized statistical models address the same issue through basis structure rather than neural attention. The streaming covariance model in (Osama et al., 2018) constructs

$\mathcal{O} = \{\mathcal{I}_{v_i,t_j}\}_{i,j}$1

with localized cubic B-splines in space and periodic temporal bases, allowing spatially varying temporal periodicities. The diffusion-based spatio-temporal extension of Gaussian Matérn fields instead uses an SPDE whose parameters control the practical correlation range in time, the smoothness in time, and the type of non-separability of the spatio-temporal covariance (Lindgren et al., 2020). Across these lines of work, locality may arise from graph neighborhoods, attention windows, basis support, or differential operators, but the underlying question is the same: how much of space-time should be shared, and at what scale?

3. Learning and inference strategies

Training and inference schemes in spatial-temporal extrapolation are often dictated by long-horizon stability. VEST emphasizes self-supervision, withholding some views or times from the input and using them as targets, so that the model can learn from observations without requiring explicit ground truth for unseen viewpoints or times (Zhang et al., 2022). This is aligned with the broader practice of predicting missing spatial-temporal entries from partial context.

A recurring concern is error accumulation. TempEE explicitly avoids auto-regression and adopts a one-step forward strategy that predicts all target future frames in a single forward pass:

$\mathcal{O} = \{\mathcal{I}_{v_i,t_j}\}_{i,j}$2

thereby addressing cumulative error spreading (Chen et al., 2023). SFTformer uses a different remedy: a joint training paradigm for historical echo sequence reconstruction and future echo sequence prediction, motivated by the practice that weather forecast experts review historical echo evolution to make accurate predictions (Xu et al., 2024).

For operator learning in dynamical systems, “TI-DeepONet” reformulates temporal extrapolation as learning instantaneous time-derivative fields rather than direct state prediction:

$\mathcal{O} = \{\mathcal{I}_{v_i,t_j}\}_{i,j}$3

followed by numerical integration. During training and inference, the derivative estimate is advanced with numerical schemes such as RK4; the learnable variant TI(L)-DeepONet replaces fixed RK4 weights with state-dependent coefficients (Nayak et al., 22 May 2025). This preserves the Markovian structure of dynamical systems, supports continuous-time prediction, and allows higher-precision integrators during inference than those used during training (Nayak et al., 22 May 2025).

Signal-processing approaches implement extrapolation through explicit model fitting rather than end-to-end latent forecasting. Motion-adapted 3D-FSE reconstructs missing video content with a 3D sparse frequency model

$\mathcal{O} = \{\mathcal{I}_{v_i,t_j}\}_{i,j}$4

but shifts the weighting function per frame according to optical-flow-derived motion so that the maximal weight follows moving content (Spruck et al., 2022). In video communication, adaptive joint spatio-temporal error concealment performs a preliminary temporal extrapolation and then a spatial refinement using the spatial neighborhood of the lost block; DTER similarly treats temporal extrapolation as a noisy estimate and applies a modified Non-Local Means denoising stage (Seiler et al., 2022, Seiler et al., 2022). These methods show that “extrapolation” may refer not only to future forecasting but also to filling spatially missing regions with temporally guided estimates.

4. Uncertainty, stochastic structure, and theoretical limits

A major distinction among spatial-temporal methods is whether they provide calibrated uncertainty. STGNP is a neural latent variable model that combines deterministic spatio-temporal representations with latent variables for target locations and Graph Bayesian Aggregation, which aggregates contexts considering uncertainties in context data and graph structure (Hu et al., 2023). Its Bayesian aggregation step yields closed-form Gaussian updates for latent means and variances, and the reported empirical coverage for PM2.5 is 77%/92%/98% for $\mathcal{O} = \{\mathcal{I}_{v_i,t_j}\}_{i,j}$5 intervals (Hu et al., 2023). The associated claim is that neural models can achieve both strong learning capability and uncertainty estimates in graph extrapolation (Hu et al., 2023).

Classical stochastic models address uncertainty through covariance specification and conditional simulation. The localized streaming model of (Osama et al., 2018) uses a convex covariance-fitting criterion and recursive updates, enabling sequential prediction without retraining from scratch. The diffusion-based Matérn extension (Lindgren et al., 2020) gives a sparse finite element approximation implemented in R-INLA and supports separable and non-separable covariance structures, including Whittle-Matérn fields on curved manifolds. For high-frequency temperature data, the nonstationary Gaussian process model in (Guinness et al., 2013) decomposes observations into temporal mean, spatial mean, jump process, and residual GP, accounts for time-varying statistical characteristics and dependence on solar radiation, and uses conditional simulation to obtain well-calibrated confidence intervals; it also models spatial-temporal jumps to interpolate a slow-moving cold front.

Theoretical work in Wasserstein space formalizes the limits of extrapolation itself. Under the adiabatic bound $\mathcal{O} = \{\mathcal{I}_{v_i,t_j}\}_{i,j}$6, every estimator incurs $\mathcal{O} = \{\mathcal{I}_{v_i,t_j}\}_{i,j}$7-risk with $\mathcal{O} = \{\mathcal{I}_{v_i,t_j}\}_{i,j}$8-exponent

$\mathcal{O} = \{\mathcal{I}_{v_i,t_j}\}_{i,j}$9

over regular, locally transport-rich subclasses (Kim, 5 Jun 2026). The lower bound interpolates between a dimension-free extrapolation floor of order $(v^*, t^*)$0 and the spatial estimation curse $(v^*, t^*)$1, with a design-dependent effective sample size for arbitrary observation times (Kim, 5 Jun 2026). At $(v^*, t^*)$2 the matching upper bound is established; for $(v^*, t^*)$3 the unconditional general-$(v^*, t^*)$4 upper bound remains an open problem (Kim, 5 Jun 2026). This places spatial-temporal extrapolation within an explicit minimax framework: even with exact past information, an unobserved future carries irreducible cost.

5. Domains, benchmarks, and empirical patterns

Computer vision provides several distinct benchmark traditions. “Spatio-Temporal Image Boundary Extrapolation” studies future prediction of image boundaries in unobserved frames and reports, for the first time, spatio-temporal boundary extrapolation on a real-world video segmentation dataset (Bhattacharyya et al., 2016). It evaluates four neural architectures—RNN encoder-decoder, convolutional RNN encoder-decoder, fully convolutional model, and multi-scale model—and finds that the multi-scale model outperforms the others, especially for longer-term prediction, while recurrent variants often fail to beat copying the last input frame (Bhattacharyya et al., 2016). In a billiard scenario, the model predicts boundaries without assumptions of a strong parametric model or any object notion and is argued to have derived a notion of intuitive physics (Bhattacharyya et al., 2016).

Radar echo extrapolation has become a major application area. TempEE addresses cumulative error spreading, imprecise representation of sparsely distributed echoes, and inaccurate description of non-stationary motion processes, and reports superiority on a real-world dataset (Chen et al., 2023). SFTformer reports superior performance on the HKO-7 dataset and ChinaNorth-2021 dataset for short(1h), mid(2h), and long-term(3h) precipitation nowcasting (Xu et al., 2024). SteamCast, a deep probabilistic diffusion model for hail nowcasting, produces 30-minute nowcasts at 6-minute intervals for a single radar reflectivity variable, across 9 different vertical angles, on an approximately $(v^*, t^*)$5 latitude-longitude grid in Yan’an City, China, and reports MSE 0.02, PSNR 23.15, SSIM 0.81, ETS 0.18, and ACC 0.99, compared with PredRNN, CMS, and VMRNN (Shi et al., 26 Mar 2025).

Signal restoration results are often reported in PSNR. Motion-adapted 3D-FSE reports gains of up to 1.75 dB compared to existing 3D-FSE by compensating changing content due to motion (Spruck et al., 2022). Adaptive joint spatio-temporal error concealment reports a gain of up to 5.2 dB in PSNR compared to the unrefined pure temporal extrapolation (Seiler et al., 2022). DTER reports a visually noticeable average gain of 1 dB over pure temporal error concealment and is described as clearly outperforming other spatio-temporal error concealment algorithms (Seiler et al., 2022).

Long-horizon forecasting outside image domains shows similar trends. Extralonger extends traffic forecasting to a week on real-world benchmarks and reports up to 500$(v^*, t^*)$6 reduction in training time, 385$(v^*, t^*)$7 faster inference, and 172$(v^*, t^*)$8 reduction in memory usage, with 2.1 GB memory in the 2016-step scenario (Zhang et al., 2024). In operator learning, TI-DeepONet and TI(L)-DeepONet reduce relative $(v^*, t^*)$9 extrapolation errors by approximately 81% over autoregressive and 70% over fixed-horizon methods, while maintaining stability for temporal domains extending to about twice the training interval (Nayak et al., 22 May 2025). In toxicology, the multiscale spatial-temporal liver twin resolves liver microarchitecture at cellular resolution and is compared with homogeneous compartment pharmaco-dynamic models and classical Cmax/AUC extrapolation strategies, with the spatial-temporal model uniquely predicting sharper pericentral necrosis distributions and matching in vivo hepatotoxicity when appropriately calibrated (Dichamp et al., 2022).

6. Limitations, controversies, and open problems

The main empirical limitation is degradation under long horizons, complex motion, or strong nonstationarity. Boundary extrapolation shows decline in precision and recall over long extrapolation horizons, especially for complex or unstructured motion, and recursive prediction amplifies mistakes, producing reversals, deformation, or disappearance of objects over very long horizons (Bhattacharyya et al., 2016). TempEE identifies cumulative error spreading, sparse echo representation, and non-stationary motion as primary obstacles in radar nowcasting (Chen et al., 2023). Extralonger notes occasional performance drop during extreme fluctuations and does not handle sudden previously unseen events due to reliance only on historical data (Zhang et al., 2024).

A second limitation concerns the interaction between structural bias and computational feasibility. SFTformer is motivated by the claim that unified spatial-temporal representations in a highly coupled feature space may cause mutual interference between temporal modeling and spatial morphology refinement (Xu et al., 2024), whereas Extralonger argues almost the opposite in traffic forecasting, identifying the separation of temporal and spatial factors as a main source of high complexity (Zhang et al., 2024). These positions are not contradictory so much as domain-specific: one line emphasizes representational disentanglement, the other pipeline unification. A plausible implication is that the appropriate treatment of space-time depends on whether the dominant failure mode is interference, underfitting, or computational blow-up.

A third limitation concerns identification and theory. The liver digital twin is described as more computationally intensive than classical compartment models, and parameter identification remains difficult because multiple parameter sets can fit calibration data (Dichamp et al., 2022). In Wasserstein extrapolation, the unconditional matching upper bound for general $\mathcal{I}_{v^*, t^*}$0 remains open (Kim, 5 Jun 2026). In reduced-order modeling, naive graph autoencoder treatments of time as a parameter fail catastrophically in extrapolation, motivating time-consistent decompositions such as GCA-ROM with Tensor Train decomposition and Operator Inference (Chen et al., 28 Nov 2025).

Across the literature, stable spatial-temporal extrapolation increasingly depends on three requirements appearing in different guises: preservation of causality, explicit treatment of uncertainty, and representations that are rich enough to capture both cross-space structure and temporal evolution without introducing avoidable interference. Existing results show that no single architectural principle dominates across all settings, but they also show that purely spatial or purely temporal treatments are usually inadequate once the target lies outside the observed region in both dimensions (Zhang et al., 2022, Hu et al., 2023, Nayak et al., 22 May 2025).