Spatial-Temporal Feature Learning Module

• Spatial-Temporal Feature Learning modules are specialized neural architectures that jointly extract, encode, and propagate both spatial (intra-frame) and temporal (inter-frame) features from sequential data.
• They employ diverse techniques such as 3D CNNs, attention mechanisms, recurrent networks, and graph-based methods to fuse modalities and preserve detailed information across time.
• Empirical results demonstrate that combining spatial attention with temporal recurrence significantly improves performance in applications like video recognition and gaze estimation while optimizing computational efficiency.

Spatial-Temporal Feature Learning (STFL) Module

Spatial-Temporal Feature Learning (STFL) modules are specialized neural network architectures engineered to jointly extract, encode, and propagate both spatial (intra-frame) and temporal (inter-frame) information from sequential data such as video, event streams, or multichannel time-series. STFL serves as the architectural core in a broad range of video understanding, gaze tracking, action recognition, and neuromorphic sensing systems, enabling effective fusion of spatial structure and temporal dynamics at multiple scales. Across methodologies, STFL operates at various levels: from 3D CNNs and weight-shared convolutions to graph-based learning and attention-driven transformers, all with the aim of learning richer representations than those possible by spatial or temporal modeling alone.

1. Core Principles and Objectives

The principal function of an STFL module is to preserve and exploit both local spatial correlations (e.g., geometry of the human face and eyes in gaze estimation, or joint-to-joint relationships in pose tracking) and temporal dependencies (e.g., motion, action dynamics) in a unified manner. Primary design imperatives include:

• Spatial-Temporal Joint Representation: Avoid premature spatial pooling, instead deferring aggregation until after both spatial context and temporal propagation are modeled, thus maximizing information retention within and across frames (Personnic et al., 19 Dec 2025).
• Efficient Fusion of Heterogeneous Features: Facilitate the merging of multiple feature modalities (e.g., eye and face regions) via attention mechanisms and self-attention to allow flexible, data-dependent weighting across spatial and temporal domains.
• Propagation of Rich Context Over Time: Employ recurrent or attention-based mechanisms to maintain continuity and model short- and long-term temporal dependencies, essential for robust downstream performance (e.g., in video-based gaze estimation or gesture recognition).
• Parameter and FLOP Efficiency: Exploit architectural devices such as grouped heads, weight sharing, tensor-train decompositions, or dynamic adapters to facilitate scalability to long sequences and large input tensors without prohibitive computation (Yang et al., 17 Jan 2026, Liu et al., 2024).
• Modality- and Domain-Invariant Design: Enable robust transfer across modalities (e.g., visible and infrared in person re-identification), as well as efficient adaptation for new tasks or subjects.

2. Major Architectural Variants

A diverse set of STFL architectures has been developed, drawing from CNNs, transformers, RNNs, graph networks, and tailored attention modules. Notable representative designs include:

• Hybrid CNN-Attention-Recurrent Modules: STFL in ST-Gaze (Personnic et al., 19 Dec 2025) utilizes a dual-stream EfficientNet-B3 CNN backbone (separate for eye and face), followed by Efficient Channel Attention (ECA), transformer-style spatial self-attention (SAM), and a bi-level GRU recurrence. The module first encodes and fuses per-frame spatial information, then propagates the enriched feature state over time using GRUs that explicitly scan both intra-frame (spatial) and inter-frame (temporal) sequences.
• 3D CNNs with Collaborative Weight Sharing: CoST (Li et al., 2019) encodes spatial (XY) and temporal (XT, YT) slices of the video tensor via a shared 2D kernel, enforcing collaborative learning through weight tying across views. Resulting feature maps are fused using learned channelwise weights, supporting both interpretability and parameter efficiency.
• Attention-based Temporal Adaptation: The STFL modules in recent CLIP-based video frameworks (OmniCLIP (Liu et al., 2024), LSMRL (Yang et al., 17 Jan 2026)) incorporate lightweight temporal adapters (e.g., PTA—Parallel Temporal Adapter) and prompt generators (e.g., SPG) within transformer blocks; temporal heads, patch shift mechanisms, or grouped attention permit fusion of spatial and temporal cues with minimal increase in computational cost.
• Graph Neural Network STFL: In federated or neuromorphic settings, STFL constructs spatial-temporal graphs where node features represent channels or local patches (e.g., electrodes in EEG or events in neuromorphic vision) and edge weights encode spatial or temporal correlations. Message passing and readout in GNNs learn higher-order representations while respecting underlying graph structure, as in (Lou et al., 2021, Bi et al., 2019).
• Frequency- and Scale-Aware Modules: Frequency-based STFL modules (e.g., FSTA in SNNs (Yu et al., 2024)) synergize per-time-step attention, 2D Discrete Cosine Transform (DCT) spatial frequency decompositions, and adaptive channel fusion, motivated by empirical analysis of spatiotemporal frequency distributions in learned features.

3. Mathematical Formulations and Workflow

A typical state-of-the-art STFL instantiation, as in ST-Gaze (Personnic et al., 19 Dec 2025), progresses through the following computational steps:

1. Dual-Stream CNN Extraction: For each input frame $t$, independent CNN encoders transform each modality (e.g., left/right eye, face) into spatial feature maps $E_t$ and $F_t$, then concatenate to form $X_t \in \mathbb{R}^{C \times H \times W}$.
2. Channel and Spatial Self-Attention: Channel attention recalibrates $X_t$ via global average pooling and an MLP to re-weight channels, while spatial self-attention flattens $X_t$ into a spatial sequence, applies multi-head self-attention with positional encoding, and outputs an enriched set $Y_t$.
3. Bi-Level Spatio-Temporal Recurrence:
   • Intra-frame recurrence: A multi-layer GRU propagates through the sequence of spatial tokens within $Y_t$, yielding a contextually-aware hidden state $h_{64}^{(t)}$ per frame.
   • Inter-frame recurrence: The terminal hidden state $h_{64}^{(t-1)}$ initializes the next frame's GRU scan, enforcing temporal continuity and capturing frame-to-frame context.
4. Final Aggregation: The per-frame summary states are pooled or otherwise aggregated for downstream prediction or classification (e.g., gaze vector estimation).

This deferral of spatial pooling until after both attention and recurrence ensures intra-frame layouts are retained when modeling temporal dynamics.

4. Comparative Analysis and Empirical Evidence

Ablation studies across representative works consistently demonstrate the necessity of two critical aspects of STFL modules:

• Retention of Intra-frame Structure Prior to Temporal Modeling: Pooling spatial context before temporal encoding degrades performance. For example, ST-Gaze (Personnic et al., 19 Dec 2025) showed angular gaze error increases from 2.58° to 2.79° when early spatial pooling is used, and to 4.84° when self-attention is omitted.
• Synergy of Attention and Recurrence: Self-attention alone (without recurrence) or recurrence without spatial attention both yield inferior results. The calibrated combination is crucial for optimal generalization and adaptation, especially in tasks such as gaze estimation and fine-grained recognition (Sun et al., 2022, Personnic et al., 19 Dec 2025).

Representative quantitative results include:

Model Variant	Gaze Error (°)	AUROC (fine-grained)	Accuracy (gesture)
ST-Gaze (full STFL)	2.58	—	—
ST-Gaze (no SAM)	4.84	—	—
STAN STFL vs. concatenation	—	0.860 vs 0.839	—
Multi-branch STFL (sEMG)	—	—	96.41 (DB2, overall)

In video recognition tasks, collaborative weight-sharing STFL modules such as CoST (Li et al., 2019) outperform parameter-hungry 3D CNN baselines by up to 1–2% absolute in top-1 accuracy, with a roughly 2× reduction in kernel parameters.

5. Domain-Specific STFL Adaptations

STFL modules are specialized according to data characteristics and task constraints:

• Neuromorphic Event-Based Vision: STFL leverages PCA-reduced, temporally-smoothed features followed by slow feature analysis (SFA) or graph-based GNNs to learn invariance to translation, scale, and rotation in asynchronous event streams (Ghosh et al., 2019, Bi et al., 2019).
• Federated Learning Scenarios: In privacy-critical domains (e.g., multi-site EEG analysis), STFL enables each client to generate spatial-temporal graphs and train GNNs locally, with model synchronization performed via aggregation protocols such as FedAvg (Lou et al., 2021).
• CLIP-Based Video Understanding: Parameter- and FLOP-constrained STFL modules minimally augment pre-trained Vision Transformer backbones with temporal head grouping and patch-shift blocks to facilitate temporal modeling while maintaining computational efficiency (Yang et al., 17 Jan 2026, Liu et al., 2024).
• Fine-Grained and Multimodal Recognition: Context-aware LSTM blocks and skeleton-graph aggregation mechanisms are introduced to integrate spatial structure (e.g., part-based attention via GeM pooling) and video-wide dynamics, as seen in open-set recognition (Sun et al., 2022) and visible-infrared Re-ID (Jiang et al., 2024).

6. Implementation Considerations and Practical Impact

STFL module selection and integration must align with application-specific requirements, such as:

• Scalability: Modules like the 3D Conv-Siamese STFL (Paul et al., 2018) and federated GNN-based STFL (Lou et al., 2021) prioritize pairwise sampling and graph reduction mechanisms for handling massive unlabeled video or multi-agent datasets.
• Energy Efficiency: In spiking neural networks, frequency-based STFLs (FSTA) suppress redundant spikes and reduce average firing rates by over 33% compared to vanilla SNNs (Yu et al., 2024).
• Interpretability: Weight-sharing designs not only reduce overfitting risk, they afford quantifiable contributions of spatial and temporal cues (e.g., channelwise fusion coefficients in CoST (Li et al., 2019)), providing insight into dataset- or class-specific feature dependencies.
• Parameter Sharing and FLOP Reduction: Tensor-train factorizations, grouped attention, and omnimodal prompt mixing reduce overparameterization, with empirical metrics confirming superior generalization using only a fraction of the parameter budget.

7. Challenges, Limitations, and Future Directions

Despite their successes, STFL modules face enduring challenges:

• Trade-offs Between Spatial-Temporal Resolution and Efficiency: High spatial or temporal resolution can sharply increase model size and compute; thus, architectural innovations (attention head grouping, dynamic filter prediction) seek Pareto-optimal designs.
• Generalization Across Domains and Modalities: Robustness to new modalities (infrared, depth) and domain shift is an open area, with promising directions leveraging multi-modal or skeleton-guided integration (Jiang et al., 2024, Yang et al., 17 Jan 2026).
• Explicit Modeling of Long-Range Dependencies: Although temporal modeling has improved (e.g., dynamic temporal filtering via FFTs (Long et al., 2022)), designing STFL modules that can adaptively focus on long-term temporal trends and multi-scale spatial patterns remains an active research frontier.
• Interpretability and Adaptation: Modules that expose interpretable spatial-temporal weighting coefficients or afford efficient subject-specific adaptation are increasingly sought after, especially in applied systems (gaze tracking, prosthetics) (Personnic et al., 19 Dec 2025, Shin et al., 4 Apr 2025).

A plausible implication is that future STFL modules will increasingly unify efficient attention, dynamic recurrence, and parameter-shared convolutional architectures, often operating within video transformer backbones, to achieve both domain-agnostic transfer and deployment-level efficiency.

---

References

• Learning Spatio-Temporal Feature Representations for Video-Based Gaze Estimation (Personnic et al., 19 Dec 2025)
• Incorporating Scalability in Unsupervised Spatio-Temporal Feature Learning (Paul et al., 2018)
• Collaborative Spatio-temporal Feature Learning for Video Action Recognition (Li et al., 2019)
• STFL: A Temporal-Spatial Federated Learning Framework for Graph Neural Networks (Lou et al., 2021)
• Frequency-based Spatial-Temporal Attention Module for Spiking Neural Networks (Yu et al., 2024)
• Learning Language-Driven Sequence-Level Modal-Invariant Representations for Video-Based Visible-Infrared Person Re-Identification (Yang et al., 17 Jan 2026)
• OmniCLIP: Adapting CLIP for Video Recognition with Spatial-Temporal Omni-Scale Feature Learning (Liu et al., 2024)
• Spatial-Temporal Attention Network for Open-Set Fine-Grained Image Recognition (Sun et al., 2022)
• Graph-based Spatial-temporal Feature Learning for Neuromorphic Vision Sensing (Bi et al., 2019)
• Triplet Attention Transformer for Spatiotemporal Predictive Learning (Nie et al., 2023)
• Convolutional Tensor-Train LSTM for Spatio-temporal Learning (Su et al., 2020)
• Electromyography-Based Gesture Recognition: Hierarchical Feature Extraction for Enhanced Spatial-Temporal Dynamics (Shin et al., 4 Apr 2025)
• STARFlow: Spatial Temporal Feature Re-embedding with Attentive Learning for Real-world Scene Flow (Lu et al., 2024)

For implementation details and further empirical evaluation, see the cited works.