Spatio-Temporal Fusion (STF): Methods & Applications

• Spatio-Temporal Fusion is a set of techniques that integrate spatial structures and temporal evolution to overcome resolution trade-offs, notably in remote sensing, video understanding, forecasting, and robotics.
• It employs diverse fusion strategies—from direct joint interactions via cross-attention to staged fusion using tokenization and aggregation—to capture complex spatio-temporal dependencies.
• Its effectiveness is demonstrated by significant performance improvements in resolution enhancement, anomaly detection, and predictive accuracy across multiple application domains.

Searching arXiv for the provided STF papers to ground the article with current records and citations. Spatio-Temporal Fusion (STF) denotes a family of methods that combine spatial structure and temporal evolution within a single predictive, reconstructive, or decision-making pipeline. Across the literature, the term is used in remote sensing to obtain data with both high spatial resolution and high temporal frequency, in video understanding to combine appearance and motion or multi-frame context, in forecasting to model sensor interactions over space and time, and in robotics to preserve object grounding and causal continuity across manipulation steps (Albanwan et al., 2021, Anwar et al., 2024, Liu et al., 2024, Huang et al., 13 Mar 2026). Rather than identifying one fixed architecture, STF names a recurrent design objective: exploit complementary information that is distributed across locations and across time.

1. Conceptual scope and taxonomy

In remote sensing, STF is framed as the response to the spatial–temporal resolution trade-off: fine-resolution sensors revisit infrequently, while coarse-resolution sensors provide dense temporal coverage but blur local structure. The chapter “Spatiotemporal Fusion in Remote Sensing” organizes this literature into pixel-level, feature-level, and decision-level fusion, and emphasizes that STF may combine spatial, spectral, temporal, semantic, and geometric redundancy rather than only raw pixel values (Albanwan et al., 2021). In action recognition, STF can instead mean the layerwise choice of spatial, spatiotemporal, or mixed convolutional units within a 3D CNN, so the fusion problem moves from data combination to operator allocation across depth (Zhou et al., 2020).

A second distinction is between direct joint interaction and staged fusion. Some methods explicitly mix spatial and temporal representations by cross-attention, global token attention, or graph propagation. Others keep appearance and motion or current and past frames separate, then merge them by concatenation, averaging, max pooling, or evidential aggregation. This suggests that STF is best understood as a spectrum of coupling strategies rather than a single intensity of integration.

Domain	STF formulation	Representative papers
Remote sensing	high spatial resolution and high temporal frequency prediction	(Cheng et al., 2016, Isono et al., 2023, Albanwan et al., 2021)
Video understanding	appearance–motion or multi-frame feature fusion	(Hu et al., 2022, Anwar et al., 2024, Thakre et al., 26 Jun 2025, Cho et al., 2019)
Forecasting	global sensor-time interaction and anomaly-aware fusion	(Liu et al., 2024, Wang et al., 23 Apr 2025)
Medical imaging	spatial branch + temporal branch + fusion bottleneck	(Zhang et al., 2024, Gou et al., 16 Jun 2026)
Robotics	object-centric tokens with persistent 3D grounding	(Huang et al., 13 Mar 2026)
LiDAR detection	spatial compensation + temporal merging + semantic supervision	(Wang et al., 13 Mar 2025)

2. Representations used by STF methods

STF methods differ first at the representation level. In classical remote sensing, the central variables are fine- and coarse-resolution reflectance images. STNLFFM models the target fine-resolution reflectance at date $t_p$ as

$$F(x,y,B,t_p) = a(x,y,B,\Delta t)\,F(x,y,B,t_k) + b(x,y,B,\Delta t),$$

then aggregates transformed similar pixels with non-local weights, so the fused object is still a reflectance image but the representation already encodes temporal conversion and patch similarity (Cheng et al., 2016). ROSTF keeps the same STF setting but promotes the latent clean HR images and sparse corruption variables to optimization variables, so STF becomes reconstruction of hidden images rather than direct prediction from observed ones (Isono et al., 2023).

In video anomaly detection, representation becomes object-centric. The method in “Spatio-Temporal-based Context Fusion for Video Anomaly Detection” detects targets, crops each object from the RGB frame and optical-flow map, and encodes them separately as $z_i^t$. Spatial context is then represented not by pairwise geometry or attention, but by the global average

$z^t = \frac{1}{n}\sum_{i=1}^{n} z_i^t,$

which is passed through a memory module and concatenated back to each target feature for reconstruction (Hu et al., 2022). In CAST, the representation is explicitly tokenized: spatial CNN maps become $S_i \in \mathbb{R}^{H'W' \times d}$, temporal frame tokens become $\mathcal{Z}\in\mathbb{R}^{F\times d}$, and STF is defined over interactions between these token sets (Thakre et al., 26 Jun 2025).

Other domains use yet different state spaces. GSTF for traffic prediction flattens all sensor-time positions into one sequence of $\alpha N$ tokens so that global attention can directly model all pairwise interactions among all sensors across all historical times (Liu et al., 2024). The DSA model DSANet fuses bottleneck features $F_m$ from a MIP-based spatial branch and $F_s$ from a temporal branch enhanced by TemporalFormer (Zhang et al., 2024). The trajectory-prediction model STF constructs an integrated 3D graph whose nodes are agent-time instances, so spatial and temporal information are fused simultaneously on one graph rather than sequentially in separate modules (Han et al., 2023). RoboStream turns each object into a token

$$\tau_i^t = \langle \mathbf{v}_i^t,\mathbf{c}_i^t,\mathbf{s}_i^t,t\rangle,$$

binding masked visual evidence, 3D centroid, geometric extent, and timestamp for persistent object grounding (Huang et al., 13 Mar 2026).

3. Fusion operators and architectural patterns

At the operator level, STF ranges from local weighted averaging to asymmetric cross-attention and constrained convex inference. In remote sensing, a representative formulation is the weighted non-local prediction

$$F(x,y,B,t_p) = a(x,y,B,\Delta t)\,F(x,y,B,t_k) + b(x,y,B,\Delta t),$$0

where each similar pixel is first temporally converted and then aggregated (Cheng et al., 2016). The remote-sensing review also describes STF as weighted combination over spatial, spectral, and temporal neighborhoods, including bilateral and 3D spatiotemporal filters, sparse representation, and decision-level probability-map fusion (Albanwan et al., 2021).

In video, several fusion patterns recur. The anomaly-detection model (Hu et al., 2022) performs intra-stream context fusion by concatenating each object feature with a memory-regularized frame context, but cross-stream fusion occurs only at scoring time by taking the maximum of normalized appearance and motion reconstruction errors. The action-recognition model STFN instead first extracts temporal dynamics in each modality with Residual Inception blocks, then fuses appearance and motion sequences by element-wise average, maximum, or multiplication; the best design is staged temporal modeling before and after fusion rather than early concatenation (Cho et al., 2019). CAST uses one-way cross-attention in which temporal tokens query a temporally averaged spatial token set: $$F(x,y,B,t_p) = a(x,y,B,\Delta t)\,F(x,y,B,t_k) + b(x,y,B,\Delta t),$$1 followed by residual normalization, so temporal context guides spatial evidence selection (Thakre et al., 26 Jun 2025).

Feature-level STF for video object detection can also be explicitly learnable yet shallow in temporal extent. The two-frame STF module for video detection uses Multi-Frame Attention, Single-Frame Attention, and a Dual-Frame Fusion Module with Adaptive Feature Pooling and deformable convolutions; the fusion stage outperformed concatenation, median, mean, and max on Cityscapes (Anwar et al., 2024). In DSA segmentation, the STF module computes branch-specific attention weights $$F(x,y,B,t_p) = a(x,y,B,\Delta t)\,F(x,y,B,t_k) + b(x,y,B,\Delta t),$$2 and $$F(x,y,B,t_p) = a(x,y,B,\Delta t)\,F(x,y,B,t_k) + b(x,y,B,\Delta t),$$3, sums them, reweights both value tensors, concatenates the refined outputs, upsamples them, and adds a residual connection: $$F(x,y,B,t_p) = a(x,y,B,\Delta t)\,F(x,y,B,t_k) + b(x,y,B,\Delta t),$$4 so dynamic vessel evidence refines a strong structural bottleneck (Zhang et al., 2024). In LiDAR detection, ST-Fusion combines a Spatial Aggregation module with progressively expanding kernels $$F(x,y,B,t_p) = a(x,y,B,\Delta t)\,F(x,y,B,t_k) + b(x,y,B,\Delta t),$$5 and a Temporal Merging module

$$F(x,y,B,t_p) = a(x,y,B,\Delta t)\,F(x,y,B,t_k) + b(x,y,B,\Delta t),$$6

thereby compensating for motion-induced misalignment before temporal merging (Wang et al., 13 Mar 2025).

A different architectural line treats STF itself as a search space or optimization object. The probabilistic 3D-CNN study represents a fusion strategy as a set of triplets $$F(x,y,B,t_p) = a(x,y,B,\Delta t)\,F(x,y,B,t_k) + b(x,y,B,\Delta t),$$7 and learns Bernoulli path variables with Variational DropPath, turning STF into a probability space over spatial, spatiotemporal, and mixed units (Zhou et al., 2020). ROSTF formulates STF as a constrained convex program over latent clean HR images, sparse-noise variables, edge-consistency constraints, brightness constraints, and $$F(x,y,B,t_p) = a(x,y,B,\Delta t)\,F(x,y,B,t_k) + b(x,y,B,\Delta t),$$8-ball fidelity sets, then solves it with preconditioned primal-dual splitting (Isono et al., 2023).

4. Learning, optimization, and inference regimes

STF spans unsupervised, supervised, distillation-based, transfer-learned, probabilistic, and training-free regimes. In unsupervised video anomaly detection, only normal training videos are used; spatial and temporal autoencoders are trained separately with reconstruction loss and an entropy regularizer on memory addressing, and abnormality is inferred from reconstruction error after late fusion of appearance and motion streams (Hu et al., 2022). In CAST for deepfake detection, training uses binary cross-entropy with logits on video-level predictions, while the STF gain is isolated by ablations that remove or reverse cross-attention (Thakre et al., 26 Jun 2025). The video object detector STF is trained end-to-end with a CenterNet-style objective consisting of heatmap focal loss, offset regression, and size regression (Anwar et al., 2024). DSANet uses deep supervision with a weighted sum of cross-entropy and Dice losses across decoder scales (Zhang et al., 2024).

Some STF methods add targeted supervision to strengthen fused features. In LiDAR detection, Semantic Injection augments points with point-wise semantic labels during training, a teacher model produces semantically enriched features, and the student fusion module is trained with scene-level feature distillation and an object-aware reconstruction loss (Wang et al., 13 Mar 2025). In ultra-high-speed x-ray radiography, EDVR-STF uses transfer learning: the model is pretrained on grayscale REDS and fine-tuned on x-ray sequences, then optimized with the Charbonnier loss while fusing three LR frames and two HR frames (Tang et al., 2024). In echocardiography, STFM combines a dual-stream CNN-LSTM with Re-EDL evidence outputs, an $$F(x,y,B,t_p) = a(x,y,B,\Delta t)\,F(x,y,B,t_k) + b(x,y,B,\Delta t),$$9-greedy segment-selection policy, and inference-time evidence fusion

$z_i^t$0

so representative clips receive stronger evidential influence (Gou et al., 16 Jun 2026).

Other works relax conventional end-to-end learning entirely. RoboStream is explicitly training-free: VLM-guided open-vocabulary object parsing, SAM3 masks, RGB-D point clouds, STF-Tokens, and a Causal Spatio-Temporal Graph are assembled online without fine-tuning, and planning is performed over persistent structured memory rather than latent weights adapted to the task (Huang et al., 13 Mar 2026). By contrast, the SST model STFM-V1 uses trend and seasonal decomposition, self-attention, residual connections, a benchmark/reference value, and anti-diagonal consistency regularization on the predicted delay-attractor matrix, so STF is supervised as structured state reconstruction rather than conventional next-step regression (Wang et al., 23 Apr 2025).

5. Application domains and empirical behavior

The empirical behavior of STF is strongly domain dependent, but a recurring pattern is that fusion is most valuable when the target phenomenon is only partially visible in any single frame, date, or sensor. In surveillance anomaly detection, the object-centric dual-stream model reached frame-level AUCs of 98.5% on UCSDped2 and 86.3% on Avenue; on UCSDped2, the dual network improved by 5.1% over the spatial stream and 0.3% over the temporal stream, and context-encoded memory improved the dual network from 97.5 to 98.5 AUC (Hu et al., 2022). In deepfake detection, CAST-B5 achieved 99.49% AUC and 97.57% accuracy in intra-dataset evaluation, while CAST-B0 reached 93.31% AUC on unseen DFD; the ablations showed that no cross-attention, separate self-attention, or reversed attention all reduced cross-dataset performance (Thakre et al., 26 Jun 2025).

In video object detection, the two-frame STF module reached 95.7 mAP on Cityscapes, 88.7 mAP on KITTI MOT, and 58.4 mAP on UAVDT, with especially clear gains on small and medium objects; on Cityscapes, the proposed dual-frame fusion scored 95.73, compared with 88.60 for concatenation and 91.89 for max fusion (Anwar et al., 2024). In DSA cerebral artery segmentation, DSANet achieved a Dice of 0.9033, and the ablation comparing TEB only with TEB + STF showed MTA JAC improving from 84.02 to 86.14 and MTA Dice from 88.25 to 89.99 (Zhang et al., 2024). In LiDAR 3D detection on nuScenes, Semantic-Supervised Spatial-Temporal Fusion yielded an improvement of approximately +2.8% in NDS on average, and for CenterPoint improved NDS from 65.58 to 68.32 (Wang et al., 13 Mar 2025).

Remote sensing shows two complementary STF stories. STNLFFM was generally more accurate and more robust than STARFM and ESTARFM, especially for heterogeneous landscapes and temporally dynamic areas such as flood or rapid crop growth (Cheng et al., 2016). ROSTF was comparable to several state-of-the-art STF methods in noiseless cases and outperformed them in noisy cases by explicitly modeling Gaussian noise, sparse corruption, and missing values within one convex optimization problem (Isono et al., 2023). A related dynamic-imaging analogue appears in x-ray radiography: with 3 subsequent images from the LR sequence of 4-time lower spatial resolution and 2 images from the HR sequence of 20-time lower frame rate, EDVR-STF achieved average PSNR of 37.57 dB and 35.15 dB on two x-ray datasets (Tang et al., 2024).

In forecasting, GSTF reported MAE 17.79, RMSE 28.80, MAPE 11.98% on PEMS04 and MAE 12.98, RMSE 21.90, MAPE 8.57% on PEMS08, attributing the gains to global long- and short-term spatio-temporal feature capture together with anomaly modeling (Liu et al., 2024). The SST forecasting model STFM-V1 reported RMSE 0.6939 and MAPE 2.3059 for one of its main $z_i^t$1, $z_i^t$2, $z_i^t$3 configurations, outperforming LSTM, XGBoost, DNN, and the Persistent model in the reported experiments (Wang et al., 23 Apr 2025). In trajectory prediction on ApolloScape, STF obtained the best WSFDE 1.707 while S2TNet retained lower WSADE, indicating a relative advantage at the final prediction horizon rather than uniformly across all forecast steps (Han et al., 2023).

The same principle extends beyond conventional perception tasks. In echocardiographic standard-view classification, STFM achieved 94.48% test accuracy and 91.14% macro F1 on EV9V, outperforming frame-only baselines and larger video architectures while using uncertainty-aware sampling and evidence-based fusion (Gou et al., 16 Jun 2026). In robotics, RoboStream achieved 90.5% on long-horizon RLBench and 44.4% on real-world block-building tasks, while both SoFar and VoxPoser scored 11.1%, indicating that persistent geometric anchoring and causal memory can be decisive in long-horizon manipulation (Huang et al., 13 Mar 2026).

6. Limitations, misconceptions, and open directions

A recurrent misconception is that STF necessarily implies deep, symmetric, all-to-all joint modeling of space and time. The literature shows many weaker but still effective forms. The anomaly-detection model (Hu et al., 2022) is a dual-stream architecture whose spatial and temporal networks are trained separately and coupled only by late score fusion. CAST is not bidirectional; its main design is explicitly one-way, with temporal tokens attending to a temporally averaged spatial token set (Thakre et al., 26 Jun 2025). DSANet’s named STF module is bottleneck-only rather than a full multi-scale fusion hierarchy (Zhang et al., 2024). This suggests that the term STF covers both tightly integrated and relatively loose interaction schemes.

Several limitations are domain specific but structurally recurrent. In anomaly detection, the spatial-context model is only global averaging over all targets in a frame, with no explicit relative-position encoding, adjacency graph, or learned attention over neighboring objects; the authors explicitly state that future work should improve object detection and the method for constructing spatial context “for the connection between the targets” (Hu et al., 2022). In deepfake detection, only 16 uniformly sampled frames are used, and the authors state that this may not capture longer-range temporal inconsistencies (Thakre et al., 26 Jun 2025). In STNLFFM, the linear reflectance-change assumption is explicitly acknowledged as a potential failure point over long time intervals (Cheng et al., 2016). In GSTF, anomaly labels are generated by a moving average anomaly identification method rather than independent event metadata, and several masked attention equations are malformed or omitted (Liu et al., 2024). In the SST model, the authors note that external environmental variables, time information, and relative position information are not yet included (Wang et al., 23 Apr 2025). In x-ray STF, performance degrades when HR anchors become very temporally sparse, and the paper does not provide explicit complexity analysis for full alternatives (Tang et al., 2024).

Across the surveyed work, two broader implications emerge. First, STF performance often depends less on the word “fusion” than on the quality of the representation being fused: object crops, tokens, graphs, attractors, or denoised latent images can make very different assumptions about stability, identity, and alignment. Second, the most recent works increasingly combine STF with memory, semantics, or uncertainty rather than treating it as a purely geometric merge. This suggests a continuing shift from shallow aggregation toward structured state maintenance: persistent object tokens in robotics, evidential clip fusion in echocardiography, semantic supervision in LiDAR, and anomaly-aware conditioning in traffic all treat STF as part of a larger problem of reliable long-horizon inference rather than only local feature mixing (Huang et al., 13 Mar 2026, Gou et al., 16 Jun 2026, Wang et al., 13 Mar 2025, Liu et al., 2024).