SpikeTAD: Spiking Neural Networks for End-to-End Temporal Action Detection

Abstract: Video understanding is a crucial part of computer vision, with numerous application scenarios. With the increasing popularity of mobile devices, an increasing number of efforts are trying to deploy video understanding models on them. However, existing video understanding models are difficult to deploy due to their large size and prohibitive power consumption. Spiking Neural Networks (SNNs) have shown bioplausibility and low power advantages over Artificial Neural Networks (ANNs), especially on neuromorphic chips which are regarded as essential components of future mobile devices. However, excessively long conversion time-steps and severe performance degradation problems limit their application. To solve the problems above, we explore the application of SNNs on temporal action detection (TAD), which is an important task in video understanding, and propose the first SNN-based end-to-end TAD architecture coined as SpikeTAD. While maintaining extremely low power consumption, SpikeTAD achieves an average mAP of 67.2% in THUMOS14 and 37.42% in ActivityNet-1.3, demonstrating the feasibility of a low-power TAD model. Our code is available at https://github.com/MCG-NJU/SpikeTAD.

• The paper introduces SpikeTAD, the first end-to-end spiking neural network framework for temporal action detection, bridging the gap between conventional ANNs and energy-efficient SNNs.
• The method leverages novel Multi-Threshold Neurons and Expectation Compensation Modules within a compact Vision Transformer backbone to enable accurate ANN–SNN conversion with minimal timesteps.
• SpikeTAD demonstrates competitive mAP on THUMOS14 and ActivityNet-1.3 while reducing energy usage to as little as 20% of baseline ANN models, highlighting its potential for power-constrained applications.

SpikeTAD: Spiking Neural Networks for End-to-End Temporal Action Detection

Introduction and Motivation

Temporal Action Detection (TAD) in untrimmed videos is central to video understanding, with applications in mobile and edge environments where power efficiency is paramount. The primary limitation of prior state-of-the-art TAD systems has been their reliance on ANN models with high computational and energy requirements, making them impractical for power-constrained neuromorphic hardware. The advent of SNNs offers a bioplausible and energy-efficient solution, but previous SNN research has been limited to relatively simple tasks, suffering from information loss and performance degradation due to conversion inefficiencies and inadequate SNN architectures for complex, temporally extended tasks.

"SpikeTAD: Spiking Neural Networks for End-to-End Temporal Action Detection" (2606.12033) presents the first SNN-based, end-to-end TAD framework—SpikeTAD—demonstrating that energy-efficient, low-latency TAD systems with competitive accuracy are feasible using ANN–SNN conversion with targeted architectural and methodological innovations.

Architectural Innovations

SpikeTAD adapts a modern backbone-detector paradigm and exploits the temporal coding potential of SNNs for video. The backbone is based on a compact Vision Transformer (ViT-S), preceded by patch embedding and spatiotemporal positional encoding. Key architectural features include:

• Multi-Threshold Neurons (MTN): Nonlinearities such as GELU and LayerNorm in the backbone are replaced with MTNs, enabling multi-bit output and improved quantization, crucial for capturing semantic information in temporal domains.
• Expectation Compensation Modules (ECM): Essential for maintaining fidelity during ANN–SNN conversion with few timesteps, ECMs analytically restore the statistical expectation of cumulative spike-driven updates, particularly for matrix operations and sophisticated nonlinear functions in Transformers.
• Spiking Detector Head: The detection module employs an efficient max-pooling–based multi-scale feature extractor, leveraging Integrate-and-Fire neurons in lieu of ReLU activations, with additional quantized clip–floor–shift activations to mitigate quantization errors in the low-latency regime.

  Figure 2: Overview of SpikeTAD, depicting the ViT-based backbone, the MTN and ECM conversion paths, and the spike-driven detection architecture.

The framework maintains a clear separation of semantic time (video frame index) and computational time (simulation step), exploiting spike-based coding to accumulate rich features per frame at minimal energy cost.

Figure 4: Schematic differentiation between semantic time and computational time, highlighting the spatio-temporal feature accumulation process.

Methodology

ANN–SNN Conversion and Training Paradigm

SpikeTAD follows an ANN–SNN conversion strategy: the model is first trained as a conventional ANN, followed by systematic replacement of nonlinear elements by their spiking analogues. Conversion is formulated to minimize rate coding error across timesteps, using the following:

• Clip-Floor-Shift Activation: This specialized quantization function with a half-step shift and biased potential initialization theoretically zeroes expected quantization error under local uniformity, enabling accurate spike-based simulation with $T=4$–16 timesteps.
• MTN and ECM Integration: MTN enables richer signal quantization and multi-bit event representation per frame, while ECM ensures exact expectation propagation through nonlinear and matrix operations even in short simulation windows.
• SOP-based Power Estimation: Layerwise computation of active synaptic operations (SOPs) and rates of multiply–accumulate (MAC) and accumulate (AC) operations establish precise energy consumption benchmarks, with SNNs leveraging the sparsity of spiking events.

  Figure 3: MTN schematic, illustrating multi-threshold spike event emission for temporally dense signal representation.

Experimental Results

Benchmark experiments are conducted on THUMOS14 and ActivityNet-1.3, with raw video input and end-to-end training. The key empirical findings are:

• Performance: SpikeTAD achieves 67.2% average mAP on THUMOS14 and 37.42% on ActivityNet-1.3 with only 16 timesteps, matching or exceeding its ANN analog (ViT-S backbone with ReLU) and outperforming most lightweight ANN baselines.
• Energy Consumption: SpikeTAD’s energy usage is a small fraction of existing TAD methods—even versus compressed ANNs. At $T=4$, system-wide power is roughly 20% that of the baseline ANN and much less than 10% compared to common transformer-based detectors.

  Figure 1: Power-performance comparison of SpikeTAD with SOTA TAD methods, demonstrating dramatic energy savings under log scale normalization.

Ablation shows inferior performance for conventional SNN training approaches (direct supervised learning and simple ANN2SNN conversion) compared to the proposed ECM/MTN-based scheme, and validates the crucial role of the clip-floor-shift activation in maintaining accuracy with minimal temporal redundancy.

• Statistical significance analysis affirms highly stable results ($\sigma < 0.2$ mAP) across randomized seeds in spike-based inference.
• Error analysis highlights that background and localization errors dominate false positives, paralleling challenges in conventional TAD systems.

  Figure 5: Error decomposition for SpikeTAD on THUMOS14 revealing the primary sources of detection failure.

Limitations and Discussion

Two key limitations persist. First, dependency on pre-trained ANN backbones for high-level semantic representation places a ceiling on both energy savings and conversion fidelity. Second, spike-based video models experience computational overhead from the multiplication of semantic and computational timesteps, which is less pronounced in image-based SNNs. Nonetheless, the empirical trade-off in energy and accuracy is highly favorable under low-latency configurations.

These observations foreground promising future work along two axes: (1) architecting SNNs trainable end-to-end from scratch for TAD, eliminating dependence on pre-trained ANNs and (2) further optimizing spike coding schemes and neuron models to reach true binary or event-based low-power operation with minimal accuracy loss.

Conclusion

SpikeTAD (2606.12033) establishes the viability of spiking neural architectures for complex temporal video analysis, achieving high-accuracy, low-power TAD suitable for mobile and neuromorphic deployment. Through principled ANN–SNN conversion—including expectation compensation, MTN quantization, and low-error spiking heads—SpikeTAD closes the performance gap between SNNs and ANNs for long-sequence video understanding. The framework sets the foundation for future research into SNN-based video models and signals an inflection point for power-efficient neural vision systems.