Spatiotemporal Fusion Training Strategy

• Spatiotemporal Fusion Training Strategy is a design doctrine that jointly optimizes spatial structure and temporal dynamics for enhanced video action recognition and multimodal tasks.
• It employs hierarchical fusion techniques, such as synchronized stream alignment and multi-level fusion, to integrate complementary features effectively.
• Empirical results demonstrate that carefully chosen fusion operators and temporal-context enrichment lead to significant improvements in accuracy and generalization.

Searching arXiv for the cited papers and closely related works to ground the article. arXiv Search: query="(Wang et al., 2019) Spatiotemporal Pyramid Network for Video Action Recognition" Spatiotemporal fusion training strategy denotes a class of model-building and optimization procedures that learn from spatial structure and temporal dynamics jointly rather than as isolated signals. In the literature covered here, it appears most prominently in video action recognition, where appearance and motion are fused across layers, time scales, and loss functions, but closely related formulations also occur in motion generation, multispectral detection, video fusion and restoration, optical flow, person re-identification, diarization, satellite imaging, micro-expression recognition, CAN-bus anomaly detection, and GPU-oriented spiking neural network training (Wang et al., 2019, Feichtenhofer et al., 2016, Cho et al., 2019, Tang et al., 30 Mar 2025). The central technical problem is how to align heterogeneous signals, choose a fusion operator that preserves informative interactions, and train the resulting hierarchy so that spatial and temporal cues reinforce rather than dilute one another.

1. Historical emergence and conceptual basis

Early modern formulations of spatiotemporal fusion in deep video models arose from dissatisfaction with late score averaging in two-stream networks. In that setting, an RGB stream and an optical-flow stream are trained separately and fused only at the prediction layer; this improves action recognition but leaves inter-stream dependencies under-modeled. "Convolutional Two-Stream Network Fusion for Video Action Recognition" formalized a stronger alternative: fuse appearance and motion at the last convolutional layer, preserve coarse spatial correspondence, and then extend the fused representation over time with 3D convolution and 3D pooling (Feichtenhofer et al., 2016).

The same basic concern is articulated more sharply in "Spatiotemporal Pyramid Network for Video Action Recognition". Two failure modes are emphasized. First, a spatial stream can be confused by similar backgrounds, as in FrontCrawl versus BreastStroke, where the swimming pool dominates appearance. Second, a temporal stream that uses 10 consecutive flow frames can be fooled by short snippets that share local motion but differ over longer time scales, as in Pull-ups versus RopeClimbing. The proposed remedy is hierarchical fusion over multiple abstraction levels: long-term temporal aggregation at the bottom, motion-guided spatial attention in the middle, and compact late fusion at the top, all trained end-to-end with a single spatiotemporal loss (Wang et al., 2019).

A related line of work shifted emphasis from architectural placement to temporal dynamics after fusion. "Spatio-Temporal Fusion Networks for Action Recognition" fused appearance and motion only after each stream had passed through temporal-dynamics extractors, then applied additional temporal modeling to the fused representation. This made fusion not merely cross-modal but explicitly video-wide, using 1D temporal convolutions over segment features (Cho et al., 2019).

Subsequent work generalized the notion of fusion from a fixed architectural choice to an object of optimization. "Spatiotemporal Fusion in 3D CNNs: A Probabilistic View" treated fusion strategies as random variables in a probability space over architectures and weights, approximated by variational DropPath. This reframed fusion design as posterior inference over candidate spatial, temporal, and mixed operators, rather than hand selection of one network per strategy (Zhou et al., 2020). A plausible implication is that spatiotemporal fusion matured from a local design trick into a broader training principle: the model should learn not only features, but also where and how fusion itself ought to occur.

2. Canonical architectural patterns

Across the cited literature, several recurring architectural motifs define spatiotemporal fusion training.

The first is stream decomposition with synchronized alignment. In action recognition, the usual split is spatial RGB versus temporal optical flow. SPN aligns both streams at the last convolutional block, such as inception5b, res5c, or conv5, where feature maps have matching spatial resolution, typically $7\times 7$. Spatial frames and optical-flow stacks are paired per snippet, and multiple temporal pathways share parameters while sampling variable temporal intervals $\tau\in\{1,\dots,10\}$ (Wang et al., 2019). The earlier two-stream fusion model similarly recommends ReLU5 or ReLU5_3 as the preferred fusion site, because coarse spatial layout is still present there, whereas earlier layers hurt accuracy and FC-layer fusion loses spatial information (Feichtenhofer et al., 2016).

The second is multi-level fusion rather than a single merge point. SPN is exemplary: bottom-level temporal fusion aggregates multiple optical-flow chunks into a long-term motion representation; middle-level fusion uses per-location motion to guide spatial attention; top-level fusion combines original spatial features, long-term temporal features, and attended spatial features into a $4096$-D representation for classification (Wang et al., 2019). A different but related hierarchy appears in STFN, where each stream first undergoes temporal modeling by Residual Inception blocks, then fused temporal dynamics are refined again by post-fusion temporal blocks before video-level aggregation (Cho et al., 2019).

The third is operator choice as an inductive bias. Some systems rely on simple operators such as element-wise sum, max, or concatenation followed by $1\times 1$ convolution (Feichtenhofer et al., 2016). Others use operators explicitly designed to capture multiplicative interactions. SPN introduces the spatiotemporal compact bilinear operator, a compact approximation to bilinear pooling. Given pathway features $\{v_i\}_{i=1}^m$, the fused feature is

$$z=\mathcal{F}^{-1}\!\left(\bigodot_{i=1}^m \mathcal{F}(g_{v_i})\right),$$

where $g_{v_i}$ is a Count-Sketch representation, $\mathcal{F}$ is the FFT, and memory remains $O(D)$ rather than $O(nm)$ for full bilinear fusion (Wang et al., 2019). This operator made it feasible to use interaction-rich fusion repeatedly inside one network.

The fourth is temporal-context enrichment around fusion. In 2016 two-stream fusion, temporal extension is supplied by 3D convolution and 3D max-pooling over chunks sampled at $\tau\in\{1,\dots,10\}$0 (Feichtenhofer et al., 2016). In SPN, long-term context is introduced by multi-path temporal CNNs sharing parameters across sampled flow chunks (Wang et al., 2019). In HiSTF Mamba, bidirectional temporal encoding is carried by forward and backward selective state-space scans, while a Dynamic Spatiotemporal Fusion Module filters redundant temporal signals before combining them with spatial features (Zhan et al., 10 Mar 2025). In VideoFusion, bi-temporal co-attention aggregates both $\tau\in\{1,\dots,10\}$1 and $\tau\in\{1,\dots,10\}$2 information so that only mutually relevant forward and backward cues are reinforced (Tang et al., 30 Mar 2025).

A final recurring motif is fusion under misalignment or modality gap. Strip-Fusion avoids rigid pixel-wise registration by alternating visible and thermal rows into strips, then mixing them with row- and column-oriented Strip-MLP modules (Kanu-Asiegbu et al., 25 Jan 2026). ComST-Flow bridges frame and event modalities through a common spatiotemporal gradient space, then uses reference boundaries to guide cross-modal correlation fusion for optical flow (Zhou et al., 10 Mar 2025). These works broaden the concept of spatiotemporal fusion beyond synchronized RGB-flow pipelines to settings where alignment itself is uncertain or heterogeneous.

3. Optimization regimes and training objectives

The defining feature of a spatiotemporal fusion training strategy is not only how features are merged, but how supervision propagates through the fused system.

In SPN, the entire pyramid is optimized with a single cross-entropy objective over the top fused representation,

$\tau\in\{1,\dots,10\}$3

Backpropagation passes through FFT-based STCB modules, the attention softmax, weighted pooling, and both CNN streams. Separate stream losses are not used (Wang et al., 2019). This unified-loss design directly contrasts with classical two-stream practice, where two losses are trained independently and only predictions are averaged afterward.

By contrast, "Convolutional Two-Stream Network Fusion for Video Action Recognition" uses a staged regime. Spatial and temporal streams are first trained separately. Fusion is then injected at ReLU5, often with identity initialization for the $\tau\in\{1,\dots,10\}$4 fusion convolution. During this phase, gradients are propagated only to the injected fusion layer, because full backpropagation into both towers did not improve results. Temporal extension with 3D convolution and 3D pooling is then trained, and optional FC8 fusion is kept for test-time prediction averaging rather than for joint training, to avoid bias toward the faster-adapting temporal tower (Feichtenhofer et al., 2016).

STFN adopts another two-stage schedule. First, backbones are frozen and only the temporal fusion network—Residual Inception stacks, fusion, post-fusion temporal blocks, and classifier—is trained with RMSProp. Then the full model is fine-tuned end-to-end with SGD and momentum (Cho et al., 2019). The effect is to separate learning of temporal fusion from readaptation of the underlying spatial and motion backbones.

The role of priors is explicit in "In Defense of Image Pre-Training for Spatiotemporal Recognition". There, image pre-training is reinterpreted as an appearance prior for the spatial component of 3D kernels, while temporal dynamics are learned during video fine-tuning. Spatial-Temporal Separable convolution splits channels into spatial and temporal groups, and center-slice or zero-init from 2D weights is preferred over uniform inflation for initializing the spatial component (Li et al., 2022). This is still a fusion strategy, but one whose optimization is structured by decomposition of what should be transferred and what should be learned from scratch.

Outside action recognition, training objectives become more heterogeneous while preserving the same principle of joint spatiotemporal supervision. VideoFusion combines self-supervised fusion consistency losses, supervised modality restoration, and a variational temporal consistency term to suppress flicker (Tang et al., 30 Mar 2025). HiSTF Mamba combines a diffusion denoising objective with position, velocity, and foot-contact losses, so that redundancy-free fusion serves a generative model rather than a classifier (Zhan et al., 10 Mar 2025). Strip-Fusion trains visible and thermal branches with detection losses plus a KL-divergence term that transfers relational structure from the more reliable modality to the less reliable one (Kanu-Asiegbu et al., 25 Jan 2026). In each case, fusion is not an isolated block but a path through which optimization pressure is distributed.

4. Empirical regularities and design trade-offs

A stable empirical pattern across the literature is that late score fusion is usually not the best trade-off. On UCF101 split 1, classical two-stream average achieved $\tau\in\{1,\dots,10\}$5, whereas SPN’s top-level compact bilinear fusion reached $\tau\in\{1,\dots,10\}$6, and the full pyramid reached $\tau\in\{1,\dots,10\}$7 in the stacking ablation before the final BN-Inception model achieved $\tau\in\{1,\dots,10\}$8 on UCF101 and $\tau\in\{1,\dots,10\}$9 on HMDB51 (Wang et al., 2019). In the 2016 two-stream fusion study, softmax averaging yielded $4096$0 top-1 on UCF101 split 1 with $4096$1M parameters, while ReLU5 conv fusion achieved $4096$2 with $4096$3M parameters, and the full spatiotemporal fusion model reached $4096$4 on UCF101 and $4096$5 on HMDB51 across splits (Feichtenhofer et al., 2016).

Another regularity is that the exact fusion operator matters, but only within an appropriate placement regime. In SPN’s late-fusion ablation on UCF101 split 1, average reached $4096$6, concatenation $4096$7, element-wise sum $4096$8, compact bilinear with $4096$9 also $1\times 1$0, with $1\times 1$1 $1\times 1$2, and with $1\times 1$3 $1\times 1$4, equal to $1\times 1$5 (Wang et al., 2019). The data support the view that interaction-rich fusion helps, but only up to a compactness threshold beyond which capacity saturates.

Temporal scale exhibits a similar non-monotonicity. In SPN’s temporal multi-path fusion, three temporal pathways with compact bilinear fusion achieved $1\times 1$6, outperforming one-path concatenation at $1\times 1$7 and five-path compact bilinear at $1\times 1$8 (Wang et al., 2019). In STFN, increasing the number of segments from $1\times 1$9 to $\{v_i\}_{i=1}^m$0 improved performance, but $\{v_i\}_{i=1}^m$1 segments caused a small drop, and stacking more than two Residual Inception blocks reduced accuracy substantially (Cho et al., 2019). HiSTF Mamba reports the same pattern in a different domain: full HiSTF with $\{v_i\}_{i=1}^m$2 achieved FID $\{v_i\}_{i=1}^m$3 on HumanML3D, while removing DSFM degraded FID to $\{v_i\}_{i=1}^m$4, and increasing stacking to $\{v_i\}_{i=1}^m$5 or $\{v_i\}_{i=1}^m$6 worsened FID to $\{v_i\}_{i=1}^m$7–$\{v_i\}_{i=1}^m$8 (Zhan et al., 10 Mar 2025). The repeated lesson is that more temporal machinery is not inherently better; redundancy and noise accumulation become limiting factors.

A third regularity is that post-fusion temporal refinement remains important. STFN’s proposed architecture with post-fusion Residual Inception blocks obtained $\{v_i\}_{i=1}^m$9 on HMDB51 and $$z=\mathcal{F}^{-1}\!\left(\bigodot_{i=1}^m \mathcal{F}(g_{v_i})\right),$$0 on UCF101 for split 1, while removing post-fusion temporal blocks dropped performance to $$z=\mathcal{F}^{-1}\!\left(\bigodot_{i=1}^m \mathcal{F}(g_{v_i})\right),$$1 and $$z=\mathcal{F}^{-1}\!\left(\bigodot_{i=1}^m \mathcal{F}(g_{v_i})\right),$$2 (Cho et al., 2019). This indicates that fusion should usually be treated as an intermediate representational event, not the terminal stage before classification.

Finally, careful decomposition can recover the value of image pre-training. On Kinetics-400, replacing 3D convolution with STS and fine-tuning from ImageNet improved a wide range of 3D CNNs without increasing parameters or computation. One representative result is SlowFast-50, where the reported top-1 increased from $$z=\mathcal{F}^{-1}\!\left(\bigodot_{i=1}^m \mathcal{F}(g_{v_i})\right),$$3 to $$z=\mathcal{F}^{-1}\!\left(\bigodot_{i=1}^m \mathcal{F}(g_{v_i})\right),$$4 while fine-tuning for $$z=\mathcal{F}^{-1}\!\left(\bigodot_{i=1}^m \mathcal{F}(g_{v_i})\right),$$5 epochs with $$z=\mathcal{F}^{-1}\!\left(\bigodot_{i=1}^m \mathcal{F}(g_{v_i})\right),$$6 GPUs rather than the $$z=\mathcal{F}^{-1}\!\left(\bigodot_{i=1}^m \mathcal{F}(g_{v_i})\right),$$7-epoch $$z=\mathcal{F}^{-1}\!\left(\bigodot_{i=1}^m \mathcal{F}(g_{v_i})\right),$$8-GPU baseline (Li et al., 2022). This directly contradicts the earlier assumption that video recognition necessarily favors from-scratch spatiotemporal training over image-derived priors.

5. Generalizations across domains

Although the earliest and most influential formulations emerged in action recognition, later work shows that spatiotemporal fusion training strategy is not confined to RGB-flow classification.

In text-driven motion generation, HiSTF Mamba fuses multi-granular spatial modeling and bidirectional temporal modeling inside a diffusion denoiser. Dual-Spatial Mamba runs part-based and whole-body branches in parallel, Bi-Temporal Mamba scans forward and backward, and DSFM removes redundant temporal information before fusion with spatial features. On HumanML3D, the full model achieved FID $$z=\mathcal{F}^{-1}\!\left(\bigodot_{i=1}^m \mathcal{F}(g_{v_i})\right),$$9 at $g_{v_i}$0 sampling steps and $g_{v_i}$1 at $g_{v_i}$2 steps, while removal of DSFM or part-based learning led to markedly worse scores (Zhan et al., 10 Mar 2025).

In multispectral pedestrian detection, Strip-Fusion trains visible and thermal YOLOv5 backbones in two stages, then injects temporally adaptive convolutions and strip-based feature mixers that tolerate misalignment. A KL-divergence loss aligns the less reliable modality toward the more informative one based on CIoU-derived reliability. The method reports state-of-the-art MR-All $g_{v_i}$3 on CVC-14 under the Reasonable setting with $g_{v_i}$4, $g_{v_i}$5, and strong heavy-occlusion performance on KAIST (Kanu-Asiegbu et al., 25 Jan 2026).

In multi-modal video fusion and restoration, VideoFusion fuses infrared and visible streams at the feature level while simultaneously restoring degraded inputs. Differential reinforcement modules isolate complementary signals, complete modality-guided fusion uses a common query built from both modalities, and bi-temporal co-attention stabilizes outputs across time. The model is trained on clips of length $g_{v_i}$6 from the M3SVD benchmark of $g_{v_i}$7 synchronized infrared-visible video pairs comprising $g_{v_i}$8 frames (Tang et al., 30 Mar 2025).

In scientific imaging, EDVR-STF reconstructs ultra-high-speed x-ray sequences by fusing a high-frame-rate low-resolution stream with a low-frame-rate high-resolution stream. With three subsequent low-resolution images at a $g_{v_i}$9-time lower spatial resolution and two high-resolution images at a $\mathcal{F}$0-time lower frame rate, it achieved average PSNR $\mathcal{F}$1 dB on additive manufacturing data and $\mathcal{F}$2 dB on friction stir welding data (Tang et al., 2024). Here the fusion strategy combines deformable alignment, temporal and spatial attention, and transfer learning from grayscale natural videos.

Other domains push the notion of fusion still further. TFusion for cross-dataset person re-identification fuses visual similarity with target-domain spatio-temporal transition patterns in a Bayesian framework, then iteratively improves the visual model through learning-to-rank (Lv et al., 2018). Audio-visual speaker diarization fuses tracked face locations with binaural spectral features inside a latent-variable temporal graphical model that permits simultaneous speakers (Gebru et al., 2016). Online satellite image fusion calibrates time-varying process covariance from only two historical high-resolution images per update, then combines modalities through distributed Kalman filtering and optional RTS smoothing (Li et al., 2023). In GPU-oriented SNN training, temporal fusion refers to fused execution of multiple time steps within a single kernel, reducing launch overhead from $\mathcal{F}$3 to $\mathcal{F}$4 while preserving the underlying leaky integrate-and-fire dynamics (Li et al., 2024). These cases differ strongly in architecture, but all treat spatial and temporal structure as signals to be jointly optimized rather than sequentially appended.

6. Limitations, misconceptions, and future directions

A common misconception is that spatiotemporal fusion is synonymous with any late combination of two modalities. The literature does not support that view. In action recognition, early convolutional placement generally outperforms softmax averaging, and hierarchical or post-fusion temporal modeling is often necessary for best performance (Feichtenhofer et al., 2016, Cho et al., 2019). Another misconception is that increasing temporal depth, segment count, pathway number, or stack depth will reliably improve performance. Multiple studies show diminishing returns or degradation once redundancy accumulates (Wang et al., 2019, Zhan et al., 10 Mar 2025).

The failure modes are equally recurrent. SPN notes persistent color and scene biases, such as Skiing versus SkateBoarding, and difficulty with fine-grained object details such as PizzaTossing versus Nunchucks (Wang et al., 2019). Strip-Fusion remains sensitive to longer temporal windows because the final-frame RoIs used for KL supervision may mismatch earlier frames under motion or ego-motion (Kanu-Asiegbu et al., 25 Jan 2026). HiSTF Mamba depends on the quality of Whole2Part segmentation and on the DSFM reduction factor $\mathcal{F}$5 (Zhan et al., 10 Mar 2025). TFusion for re-identification requires timestamps and camera IDs in the target domain, and its spatio-temporal stationarity assumption can break under abrupt traffic-pattern changes (Lv et al., 2018). Online satellite fusion relies on accurate sensor models and registration, while its identity dynamics $\mathcal{F}$6 are conservative by design (Li et al., 2023).

Future directions stated in the surveyed work are consistent across domains. SPN explicitly suggests enhanced attention-based fusion, longer-range temporal modeling, transformer-based or self-supervised pretraining, and fine-grained object detectors (Wang et al., 2019). HiSTF Mamba suggests explicit decorrelation regularization and broader cross-modal conditioning (Zhan et al., 10 Mar 2025). VideoFusion indicates that stronger temporal consistency mechanisms and degradation-aware fusion remain central for coherent video synthesis (Tang et al., 30 Mar 2025). ComST-Flow suggests that a common latent or gradient space can make multimodal fusion more interpretable, especially when direct feature fusion is undermined by heterogeneous representations (Zhou et al., 10 Mar 2025).

Taken together, these results suggest that a spatiotemporal fusion training strategy is best understood not as a single architecture but as a design doctrine. Spatial and temporal evidence must be aligned at an appropriate abstraction level, fused with an operator matched to the interaction structure of the problem, and trained under losses that let one modality or timescale correct the blind spots of another. The most successful formulations do not merely aggregate cues; they structure optimization so that appearance, motion, long-range dynamics, and modality reliability become mutually constraining components of one learned system.