Temporal & Spatial Encoders

• Temporal and spatial encoders are foundational modules in spatiotemporal learning, capturing time-dependent dynamics and intra-frame spatial relationships.
• They employ diverse architectures such as CNNs, Transformers, and state-space models to extract and fuse features from videos, images, and time series.
• Hierarchical, dual-path, and adaptive fusion designs optimize both computational efficiency and model accuracy for tasks like action recognition and video understanding.

Temporal and spatial encoders are the foundational architectural elements in modern spatiotemporal machine learning, enabling models to integrate and disentangle space- and time-dependent information in sequential and structured data. They are core components of systems for action recognition, video-language reasoning, event detection, time series modeling, and beyond. Approaches to temporal and spatial encoding span state-space models (Mamba), hierarchical Transformers, convolutional architectures, and hybrid or dual-path networks, reflecting the diversity of data and requirements encountered in practice.

1. Conceptual Structure and Taxonomy

Spatial encoders are designed to capture intra-frame or within-instance dependencies—such as local or global relations between patches in an image, joints in a pose, or actors in a scene—while temporal encoders model inter-frame or sequential dynamics, tracking the evolution of features or entities over time. In advanced systems, these encoders can be arranged hierarchically, decoupled (dual-path), or tightly coupled via fusion/integration modules. A representative taxonomy includes:

• Pure spatial encoders: 2D or 3D CNNs, Vision Transformers; process single frames or pooled static features.
• Pure temporal encoders: 1D ConvNets over time, RNNs (LSTM/GRU/ConvLSTM), SSM-based structures (Mamba), or temporal Transformers operating on per-frame features.
• Hybrid or joint spatial-temporal encoders: Swin-Transformer with multi-video windows (Geng et al., 2022), Clustered Spatial-Temporal Transformers (Li et al., 2021), or joint cross-modal blocks.
• Dual-path/decoupled architectures: Separate spatial and temporal pathways, fused late (e.g., DeST (Lee et al., 2022), DiST (Qing et al., 2023), VideoGPT+ (Maaz et al., 2024)).
• Hierarchical multi-level encoders: Spatial hierarchy (from fine to coarse granularity), temporal hierarchy (short- to long-range), or body-part to whole-pose levels (Qian et al., 2023, Liu et al., 4 Nov 2025).

2. Architectural Designs and Mathematical Formalisms

Spatial Encoders

These modules typically operate on the spatial domain per time step or video frame via:

• CNNs: Stacked 2D convolutions, often with downsampling for multi-scale feature extraction (e.g., RegNet (Hu et al., 2022), ResNet-50 (Ding et al., 2022, Tan et al., 2022)).
• Transformers: Windowed or global self-attention over spatial tokens per frame (Swin-Transformer (Geng et al., 2022)), patch-based ViTs in image encoders (Lee et al., 2022, Qing et al., 2023).
• State-space layers: Scanning the spatial dimension as a "sequence" in Mamba or SSM layers, leveraging linear recurrence for full-channel/global field-of-view (Yang et al., 2024, Liu et al., 4 Nov 2025).
• Pooling-based bottlenecks: Pooling or dimensionality-reduction operations to summarize spatial content for cross-modal or temporal integration.

Temporal Encoders

Temporal encoders process sequences of spatial features, often with strategies to specifically address temporal locality, hierarchy, or efficiency:

• 1D/3D Convolutions: E.g., 3D I3D blocks (Hui et al., 2021), DWTempConv (Kim et al., 17 May 2026), temporal convolution banks (Yang et al., 2024).
• RNNs/ConvLSTMs: Explicit recurrence for modeling long-range dependencies, especially in regular time series (Rußwurm et al., 2018).
• Temporal Transformers: Attention mechanisms with time-indexed positional embeddings, able to capture non-local dependencies in time (Qian et al., 2023, Lee et al., 2022).
• State-Space and Mamba models: Linear or diagonal SSMs, which propagate state with low computational complexity and controllable memory depth (Yang et al., 2024, Liu et al., 4 Nov 2025).
• Frame/clip differencing: Extracting explicit motion cues via consecutive frame subtraction, potentially input to a temporal encoder (Liu et al., 4 Nov 2025, Ding et al., 2022).
• Temporal saliency-based modules: Attentive selection or clustering across time to focus on temporally relevant interactions (e.g., scene graph (Zhu, 15 Mar 2025)).

3. Hierarchical and Multi-Scale Spatial-Temporal Encoding

Hierarchical designs leverage the natural structure of the data:

• Local-to-global progression: HSTFormer’s sequence of encoders—STE (joint-level spatial), JTTE (joint trajectory temporal), BTTE (body-part group temporal), PTTE (whole-pose temporal)—mirrors the kinematic hierarchy, with progressively increasing context and abstraction (Qian et al., 2023).
• Multi-granularity partitioning: M²S²L partitions video clips into patch scales (fine, medium, coarse) and temporal windows (short, medium, long) with separate Mamba-based encoders for each, which are then adaptively fused (Liu et al., 4 Nov 2025).
• Adaptive fusion: Features are aligned (resize/reshape), weighted with soft selection modules, and fused along the channel or token axis to capture both fine- and coarse-scale spatiotemporal relationships.

Model or Method	Spatial Encoder	Temporal Encoder	Fusion/Integration
HSTFormer (Qian et al., 2023)	Joints/frame → STE → BTTE	JTTE → BTTE → PTTE	Linear fusion + projection
DiST (Qing et al., 2023)	CLIP ViT (frozen)	Efficient 3D CNN	Integration Branch (I-Block)
STMambaNet (Yang et al., 2024)	Mamba SSM on channel, CNN embedding	Mamba SSM on time, CNN emb.	Concatenate, FC classifier
M2S2L (Liu et al., 4 Nov 2025)	MS-VSSB (1/3/5 kernel, Mamba SSM)	Temporal Mamba Block (3 scales)	Learnable gating, bank fusion
RSTT (Geng et al., 2022)	Swin blocks (windowed, shifted)	Temporal axis in windowing	Implicit, via token-level attn
GroupFormer (Li et al., 2021)	Per-frame spatial attn, RoI+embedding	Per-actor temporal attn	Cross-individual/group decoders

Note: Cell contents are highly abridged; consult primary descriptions for full architectural details.

4. Encoder Fusion, Interaction, and Decoupling Strategies

Effective integration of spatial and temporal representations is a major architectural concern:

• Sequential/Stacked Encoders: Pass spatial features to a temporal encoder or vice versa, sometimes iterating this process (e.g., in classic RNN-CNN fusion).
• Parallel Decoupled Paths: In DiST (Qing et al., 2023) and DeST (Lee et al., 2022), spatial (ViT/image) and temporal (3D-CNN/video) branches operate independently on input streams, with features fused via a parameterized integration branch or simple late summation. This approach leverages pretrained spatial backbones with minimal finetuning overhead.
• Explicit Cross-Modal Bridging: Language-Bridged Duplex Transfer in (Ding et al., 2022) uses cross-attention blocks to exchange information between temporal and spatial encoders, mediated by language, at each encoder stage.
• Adaptive Channel/Token Fusion: E.g., Bilateral Channel Activation (BCA) applies channel-wise gating by cross-modality cues (Ding et al., 2022); LGFS in (Hui et al., 2021) modulates spatial/temporal contributions per-channel guided by language features.
• Hybrid Token Reshaping: The TIME preprocessing layer fuses $N^2$ temporal frames into an $N \times N$ spatial grid as the model input, resulting in each spatial token having a fixed local temporal field (Chen et al., 2024).

5. Modalities, Data Types, and Downstream Applications

The design of spatial and temporal encoders is influenced by the domain and downstream application:

• Video Action Recognition: Methods must encode complex spatial appearance and long-range temporal dynamics—hybrid models (e.g., TIME + ViT or VideoMAE) or joint windowed attention architectures are prominent (Chen et al., 2024, Geng et al., 2022).
• Video Understanding in LMMs: To balance context length and computational cost, models such as LiteFrame (Kim et al., 17 May 2026) collapse spatial-temporal redundancies within the encoder (e.g., via depthwise temporal convolutions + strided pooling), with supervised token distillation.
• EEG and Biosignal Analysis: SSM/Mamba-based dual encoders in STMambaNet (Yang et al., 2024) efficiently capture both spatial (across electrode grid) and temporal (long EEG sequences) patterns.
• Time-Series/Image Classification: Temporal correlations can be encoded into 2D matrices (GAF/MTF) for CNNs to exploit spatial invariance, effectively converting temporal tasks into spatial ones (Wang et al., 2015).
• Scene Graphs, Structured Prediction: Salient temporal encoders selectively link object pairs across frames, building explicit and sparse temporal graphs for structured scene modeling (Zhu, 15 Mar 2025).
• Referring Video Segmentation: Joint encoders and cross-modal bridges enable precise alignment between appearance, motion, and textual queries (Ding et al., 2022).

6. Computational Efficiency and Hierarchical Generalization

Architectural choices in spatial and temporal encoding have direct impact on computational efficiency, scalability, and generalization:

• Linear or near-linear complexity: Temporal encoding with SSM/Mamba (vs. quadratic self-attention) enables sequence scaling in EEG/video (Yang et al., 2024, Liu et al., 4 Nov 2025).
• Progressive aggregation: Hierarchical encoders prevent overfitting and preserve local cues without overwhelming models with global context, e.g., HSTFormer’s local-to-global progression (Qian et al., 2023).
• Spatial/temporal downsampling: Deploying depthwise or strided convolutions to reduce token count, as in LiteFrame (Kim et al., 17 May 2026), allows video LLMs to operate with large time windows under tight memory/latency constraints.
• Empirical findings: For hybrid models, spatial encoders provide substantial gains on spatially complex scenes, while temporal encoders are critical on structure-light, motion-focused domains (e.g., depth video, action sequences) (Chen et al., 2024).

7. Benchmarks, Empirical Gains, and Ablation Studies

The value of spatial/temporal encoders is substantiated in multiple domains and via rigorous ablation:

• 3D human pose estimation: HSTFormer achieves >13mm MPJPE improvement on MPI-INF-3DHP vs. non-hierarchical Transformers (Qian et al., 2023).
• Video QA: DeST achieves large absolute improvements (e.g., +10.6% on AGQA 2.0) versus non-decoupled architectures (Lee et al., 2022).
• Anomaly detection: M²S²L’s multi-scale design yields AUCs of 98.5% (Ped2), 92.1% (Avenue), 77.9% (ShanghaiTech), with real-time performance (45 FPS) (Liu et al., 4 Nov 2025).
• EEG classification: STMambaNet’s joint spatial-temporal encoding produces significant gains (82.37% avg accuracy vs. CNN-only 79.98%) (Yang et al., 2024).
• Action recognition with token fusion: TIME layer delivers 1.2–48% absolute accuracy gain on VideoMAE/ResNet/ViT backbones, strongly dependent on the relative importance of spatial vs. temporal cues for a given task (Chen et al., 2024).
• Contrastive learning: Spatial-temporal decoupling with supervised contrastive objectives yields across-the-board 0.4–0.7% improved accuracy in skeleton action recognition, with no test-time penalty (Zhang et al., 2023).
• Computational scales: Models like LiteFrame move the computational bottleneck from LLM-side visual token count to encoder efficiency, outperforming post-hoc token reduction at matching latency (Kim et al., 17 May 2026).

---

References:

• Hierarchical and hybrid Transformer-based: (Qian et al., 2023, Geng et al., 2022, Li et al., 2021)
• Salient temporal encoding and scene graph modeling: (Zhu, 15 Mar 2025)
• Mamba-based spatial/temporal encoding: (Yang et al., 2024, Liu et al., 4 Nov 2025)
• Decoupled and dual-encoder video-language: (Lee et al., 2022, Qing et al., 2023, Maaz et al., 2024)
• Temporal fusion and image-based encoding: (Chen et al., 2024, Wang et al., 2015)
• Contrasts and ablations: (Zhang et al., 2023, Hui et al., 2021, Ding et al., 2022, Tan et al., 2022, Rußwurm et al., 2018, Kim et al., 17 May 2026)

These advances demonstrate that the precise implementation and fusion strategy of temporal and spatial encoders are tightly coupled to the observed gains in both accuracy and efficiency across a wide spectrum of spatiotemporal learning tasks.