Frequency Decoupled Spatiotemporal Correlation Module

• Frequency decoupled spatiotemporal correlation modules separate low and high-frequency components to distinctly capture global trends and localized details.
• They leverage transforms like FFT, DWT, and Gaussian–Laplacian pyramids to effectively decompose and process multi-scale, multimodal data.
• Empirical results in applications such as video anomaly detection and speech separation demonstrate improved performance and enhanced interpretability.

A Frequency Decoupled Spatiotemporal Correlation Module (FDSCM) is a neural network component that explicitly separates (decouples) the frequency components of spatiotemporal data—across space, time, or both—and leverages this decoupling to more effectively capture and model both global and local correlations in complex signals. FDSCMs have emerged across modalities, including audio, video, radar, time series, and multivariate sensor data, and are motivated by the need to address interference, preserve localized details, and enable interpretable modeling under the constraints of signal diversity, multi-scale dependencies, and noise.

1. Fundamental Concepts and Motivation

Frequency decoupling in spatiotemporal modeling refers to the explicit decomposition and separate processing of information at different frequency bands in spatial, temporal, or joint spatiotemporal domains. The goal is to combat entangled dynamics (e.g., ego-motion vs. object motion in video (Liu et al., 16 Jan 2026)), prevent mutual interference between different types of dependencies (such as spatial morphology and temporal evolution in radar (Xu et al., 2024)), or robustify against domain shifts and modality mismatch (as in image-event fusion (Sun et al., 25 Mar 2025)).

Key justifications include:

• Low-frequency components typically encode global structures or trends (e.g., global ego-motion, background, scene layout).
• High-frequency components encode local, abrupt changes (e.g., moving objects, edges, anomalies).
• Treating these components separately allows for specialized modeling and robust, interpretable representations.

Empirical evidence demonstrates that FDSCMs outperform conventional attention or correlation modules in tasks such as video anomaly detection, precipitation nowcasting, continuous speech separation, zero-shot image–event depth estimation, and spatiotemporal anomaly detection (Shin et al., 20 Sep 2025, Xu et al., 2024, Sun et al., 25 Mar 2025, Ye et al., 25 Feb 2025, Liu et al., 16 Jan 2026, Shu et al., 13 Jan 2026, Meng et al., 2016).

2. Core Architectural Patterns

FDSCMs exhibit certain common design elements, but may be instantiated in various domains:

(a) Frequency Decomposition:

• Fast Fourier Transform (FFT)/Inverse FFT (iFFT), Discrete Wavelet Transform (DWT), or Gaussian–Laplacian pyramid separates input features into frequency bands (low vs. high; or multi-band) (Liu et al., 16 Jan 2026, Sun et al., 25 Mar 2025, Ye et al., 25 Feb 2025, Meng et al., 2016).
• Adaptive weighting schemes in the frequency domain (e.g., $w_k = l_k^2A_k^2$ in video (Liu et al., 16 Jan 2026)) allow per-band emphasis based on normalized frequency and band energy.

(b) Decoupled Processing Branches:

• Parallel or sequential branches process different bands or types of features separately, employing specialized modules—e.g., windowed attention for spatial morphology, temporal self-attention/Fourier blocks for periodicity (Xu et al., 2024), autocorrelation for long-range context (Liu et al., 16 Jan 2026), or cross-modal fusion via frequency-guided attention (Sun et al., 25 Mar 2025).

(c) Correlation and Attention Operators:

• Self-attention and cross-attention mechanisms are often executed within a frequency-specific channel or domain (Ye et al., 25 Feb 2025, Xu et al., 2024).
• Graph-based or spatial correlation is performed after frequency decomposition, often to exploit band-specific spatial relationships robust to sampling irregularity or channel mismatch (Sun et al., 2021, Ye et al., 25 Feb 2025, Shin et al., 20 Sep 2025).

(d) Phase Alignment and Low-Rank Decomposition (for interpretability):

• In scientific data analysis, phase-aligned spectral filtering clusters modes by phase slope, yielding interpretable, frequency-coherent components (Meng et al., 2016).

(e) Fusion Mechanisms and Residual Integration:

• Decoupled outputs are typically recombined by residual summation, projection, or multi-head attention, ensuring that global and local cues contribute additively without destructive interference (Xu et al., 2024, Liu et al., 16 Jan 2026, Sun et al., 25 Mar 2025).

3. Mathematical Formulations

Several canonical instantiations of FDSCM appear across domains:

Domain	Frequency Decoupling Definition	Key Correlation/Attention Mechanism
Video (UAV) (Liu et al., 16 Jan 2026)	1D FFT in time with adaptive $w_k$, plus 2D (space-time) autocorrelation	Spectral autocorrelation attention in frequency domain
Audio/speech (Shin et al., 20 Sep 2025)	Per-frequency–decoupled dual-path transformer; PHAT-$\beta_f$ weighting	IPD/power correlation, frequency-grouped attention, group conv
Radar (Xu et al., 2024)	SFT-block: decouples into (i) spatiotemporal, (ii) spatial, (iii) temporal/frequency	Windowed & shifted windowed attention; frequency-enhanced block (FEB)
Image-event depth (Sun et al., 25 Mar 2025)	Gaussian-Laplacian pyramid band-splitting, cross-modal attention per band	Top-down intra-band fusion, cross-branch multihead attention
WSN time series (Ye et al., 25 Feb 2025)	DWT (trend/seasonal split), frequency attention, temporal FFT/iFFT	Frequency-domain self-attention on high-pass seasonal bands
Traffic (Sun et al., 2021)	“Multi-fold” HSC/MCAN: speed/trend/deviation series aligned via Chebyshev polynomials	Per-band (fold) GCN + attention fusion
Scientific data (Meng et al., 2016)	Temporal DFT, phase-aligned spectral clustering by mode	Phase-aligned filtering, frequency-coherent mode extraction

4. Applied Example: FDSCM in Video Anomaly Detection

FTDMamba (Liu et al., 16 Jan 2026) uses a two-stage FDSCM:

• Temporal Frequency Decoupling: 1D FFT along the time axis of video features, apply frequency-adaptive weights $w_k$, take 1D iFFT to yield temporally frequency-enhanced signals. This separates global (ego-motion, low-frequency) and local (object, high-frequency) motions.
• Spatiotemporal Correlation Modeling: Reshape spatial axes to 1D, perform 2D FFT over $(T, S)$, derive power spectral density, take 2D iFFT to reconstruct a spatiotemporal autocorrelation map. This map serves as a dynamic attention mask highlighting globally coherent anomalies.
• Fusion and Output: The enhanced features are fused with a parallel TDMM block and decoded for anomaly classification.

Ablation demonstrates that omitting either stage of decoupling (frequency or correlation) degrades performance, confirming the importance of explicit frequency-wise separation and autocorrelation attention.

5. Domain-Specific Variants and Innovations

Certain FDSCM instantiations are specialized for modality or problem domain:

• PHAT-$\beta$-based spatial correlation input in continuous speech separation (Shin et al., 20 Sep 2025) tunes the phase magnitude balance per frequency, significantly boosting signal-to-distortion ratio improvement (SDRi) and reducing word error rate in challenging speech mixtures.
• Gaussian–Laplacian pyramids with cross-modal frequency attention for image-event fusion (Sun et al., 25 Mar 2025) resolve inherent frequency-mismatch; image features dominate low-freq global structure, events guide high-freq edge recovery.
• Wave equation-based frequency–time decoupling (Shu et al., 13 Jan 2026) achieves O(N log N) global interaction for visual signals and supports decomposable, physically motivated propagation of semantic features.

6. Impact, Limitations, and Empirical Results

Across benchmarks in video, radar, speech, sensor networks, and multimodal perception:

• FDSCMs consistently yield improved detection, separation, and prediction metrics compared to coupled or global-only attention mechanisms (Liu et al., 16 Jan 2026, Shin et al., 20 Sep 2025, Sun et al., 25 Mar 2025, Xu et al., 2024, Ye et al., 25 Feb 2025).
• Frequency decoupling mitigates mutual interference and enables interpretable multi-scale modeling, as evidenced by ablation test performance drops of 2–5% (in e.g., Micro-AUC, SDRi, CSI, HSS) when either branch is removed (Liu et al., 16 Jan 2026, Shin et al., 20 Sep 2025, Xu et al., 2024).
• In wireless sensor network anomaly detection, DWT + frequency attention + GCN yields F1 = 93.5%, outperforming classic and attention-based baselines (Ye et al., 25 Feb 2025).

A plausible implication is that in tasks where sources of spatial or temporal variation have distinct physical origins or semantic roles, frequency decoupling provides a principled approach for disentangling and robustly modeling these effects.

7. Interpretability and Generalization

Using explicit frequency separation, FDSCMs support:

• Extraction of interpretable, phase-coherent low-rank components corresponding to dynamical modes (e.g., traveling waves, trends) (Meng et al., 2016).
• Modularity: Frequency-specific processing blocks can be tailored for residual, group-convolutional, or attention-based integration, and stacked or parallelized to suit computation budgets and performance requirements (Shu et al., 13 Jan 2026, Xu et al., 2024, Shin et al., 20 Sep 2025).
• Generalization across modalities: The design pattern recurs in domains as diverse as speech, video, weather, remote sensing, and traffic time series.

Empirical evidence across tasks demonstrates that FDSCMs are a critical mechanism for robust and sample-efficient learning in structured spatiotemporal environments, particularly under heterogeneity, occlusion, noise, or domain shift (Ye et al., 25 Feb 2025, Sun et al., 25 Mar 2025, Sun et al., 2021, Shu et al., 13 Jan 2026, Xu et al., 2024).