FTDMamba: Frequency-Assisted UAV VAD

• FTDMamba is a novel neural architecture that decouples frequency and temporal dynamics to accurately detect anomalies in UAV videos.
• It employs frequency-decoupled spatiotemporal correlation to separate UAV ego-motion from local object movements, reducing false positives.
• Its two-branch encoder–decoder design with Mamba modules and multi-scale temporal dilation achieves state-of-the-art performance on the MUVAD dataset with near-real-time inference.

FTDMamba denotes the "Frequency-Assisted Temporal Dilation Mamba," a neural architecture designed for video anomaly detection (VAD) in unmanned aerial vehicle (UAV) videos, with particular efficacy in dynamic, moving-camera scenarios where multi-source motion coupling is present. FTDMamba innovatively integrates frequency-decoupled spatiotemporal correlation and temporally-dilated structured state-space sequence modeling using Mamba modules to explicitly disentangle and jointly model complex spatiotemporal dependencies, enabling robust anomaly detection in challenging aerial surveillance footage (Liu et al., 16 Jan 2026).

1. Problem Domain and Motivation

FTDMamba targets the unique challenge of VAD for UAV-captured video streams, where observed motion is a superposition of (a) global background flow induced by UAV ego-motion and (b) local object motion (vehicles, pedestrians, etc.). This "multi-source motion coupling" fundamentally differs from ground-based or static-camera setups, as standard frame differencing or optical flow-based approaches fail to separate these components, leading to elevated false positives (for normal UAV movement) and missed detection of foreground anomalies obscured by dominant background trends.

Conventional VAD architectures—predominantly reconstruction- or prediction-based, and reliant on monolithic spatiotemporal models—are inadequate for modeling this joint dynamics, as they lack mechanisms for frequency-domain decoupling and multi-scale temporal reasoning. FTDMamba is designed to resolve these deficiencies by coupling frequency analysis with multi-scale Mamba modeling in a unified end-to-end framework (Liu et al., 16 Jan 2026).

2. Network Architecture

FTDMamba utilizes a two-branch encoder–decoder architecture. The encoder is based on a 4-stage Pyramid Vision Transformer that extracts multi-scale feature tensors $\{f_i\}_{i=1}^4$. Each encoded feature $$f_i\in\mathbb{R}^{B\times T\times C_i\times H_i\times W_i}$$ is processed in parallel by two specialized modules:

• The Frequency Decoupled Spatiotemporal Correlation Module (FDSCM), which incorporates explicit frequency-based analysis and global attention.
• The Temporal Dilation Mamba Module (TDMM), which applies temporally-dilated, bidirectional Mamba state-space layers for local spatiotemporal modeling across a hierarchy of temporal scales.

Outputs from FDSCM and TDMM are concatenated at each level, projected, and passed through the decoder, which employs hierarchical upsampling with skip connections to reconstruct the predicted future frame $\hat{Y} = x_{T+1}$ (Liu et al., 16 Jan 2026).

3. Frequency Decoupled Spatiotemporal Correlation Module (FDSCM)

FDSCM is specifically engineered to disentangle ego-motion from object motion and to provide global spatiotemporal context:

• Temporal Frequency Decoupling: A 1D FFT is applied along the temporal axis for each feature at every spatial location:

$$\widehat f_k = \sum_{t=0}^{T-1} f_t \,e^{-j2\pi k t/T}$$

The amplitude spectrum $A_k=|\widehat f_k|$ is modulated with frequency weight $w_k=l_k^2A_k^2$, where $l_k=k/T$ (or $(k-T)/T$ if $k>T/2$). The spectrum is filtered and inverse FFT yields a temporally-weighted feature $f'(t)$.

• Spatiotemporal Correlation: Spatial dimensions are flattened and a 2D FFT is computed over (time, space). The power spectral density (PSD) $S_{f'}$ then gives an autocorrelation map $R_{f'}$ via Wiener–Khinchin theorem. This autocorrelation is used as an attention mask to enhance the feature: $\bar f = f' + R_{f'} \odot f'$.

This dual-step mechanism ensures separation of mixed motion sources and captures long-distance dependencies typically lost in local models (Liu et al., 16 Jan 2026).

4. Temporal Dilation Mamba Module (TDMM)

TDMM robustly encodes local spatiotemporal structure and continuity across variable timescales using Mamba's state-space modeling:

• STMamba Block: Features are projected into content and gate subspaces, then reorganized via two scan strategies:
  • Pixel-wise temporal-first scanning (row- and column-major) produces sequences of pixel evolution across frames.
  • Patch-wise spatial-first scanning gathers local patches across time, creating “history tubes” per patch.
  • Each sequence is processed by a 1D convolution followed by a Structured State Space Model (Mamba) layer and reconverted to the original arrangement. Outputs are fused via a gated linear combination and added to the input.
• Multi-Scale Temporal Dilation: The sequence is subsampled at varying rates $\eta\in\{1,2,3\}$ via reversible transforms $\Phi_\eta$. Each dilation yields subsequences processed independently by STMamba; outputs are merged after inverting $\Phi_\eta$. This multi-rate design captures both slow, global motion and fast, local changes.

This module jointly models long-term, smooth UAV trajectories and short-term, abrupt, object-centric anomalies, outperforming alternatives that lack such hierarchical temporal reasoning (Liu et al., 16 Jan 2026).

5. Training Setup and Loss Functions

The network is trained for next-frame prediction using the following setup:

• Data: 6 consecutive frames as input, resized to $256 \times 256$, normalized to $[-1, 1]$.
• Optimizer: AdamW with cosine-annealed learning rate, 200 epochs, batch size 8.
• Loss Function: A weighted sum of intensity loss ($L_{\rm int}$, MSE between predicted and ground-truth frames), gradient loss ($L_{\rm grl}$, difference of spatial gradients), and structural similarity ($L_{\rm ssim}$). The total loss is $$L = \alpha L_{\rm int} + \beta L_{\rm grl} + \gamma L_{\rm ssim}$$ (Liu et al., 16 Jan 2026).

6. Moving UAV VAD Dataset (MUVAD) and Benchmark Results

FTDMamba introduced the MUVAD dataset to address limitations in prior UAV-VAD benchmarks. MUVAD comprises 222,736 frames (46 normal train, 72 test clips, 12 anomaly types) with high-quality, cross-validated annotations. This resource uniquely enables evaluation of VAD under realistic drone motion and traffic-domain anomalies (Liu et al., 16 Jan 2026).

Across Drone-Anomaly, UIT-ADrone (static background), and MUVAD (dynamic background), FTDMamba achieves state-of-the-art results:

Dataset	Micro-AUC	Macro-AUC	EER	Best Prior (Micro/Macro/EER)
Drone-Anomaly	71.6%	72.3%	0.336	68.3% / 67.5% / 0.360
UIT-ADrone	70.7%	69.5%	0.368	68.6% / 66.5% / 0.403
MUVAD	71.4%	68.4%	0.372	67.9% / 65.1% / 0.413

FTDMamba demonstrates substantial robustness to severe input corruption, including Gaussian noise (AUC drop $<2\%$ at $\sigma=100$), and frame occlusions (AUC drop $<5\%$ at 50% missing frames). Ablation studies confirm all core components (frequency decoupling, temporal dilation, scan strategies) materially contribute to performance (Liu et al., 16 Jan 2026).

7. Analysis, Limitations, and Extensions

FTDMamba establishes a new paradigm for anomaly detection in airborne surveillance by providing the first explicit frequency-based decoupling of ego-motion and foreground activity, as well as comprehensive joint modeling of local and global spatiotemporal dynamics within the Mamba state-space framework. It achieves near–real-time inference (29 FPS) with a shallow STMamba, but deeper networks incur significant speed reductions.

Limitations include residual difficulty in detecting prolonged or highly camouflaged anomalies, and the absence of explicit scene semantics (traffic rules, object classes). The original work suggests that diffusion or autoregressive generative models could address the challenge of probabilistic event modeling, while integration of semantic priors or object detectors could enhance context-aware anomaly scoring. Deployment on resource-constrained UAVs may also require additional model compression or hardware optimization (Liu et al., 16 Jan 2026).

FTDMamba’s architecture and the MUVAD dataset provide a foundation for subsequent advances in frequency- and structure-informed spatiotemporal analysis in aerial surveillance and related sequential vision tasks.