SpikeTAD: SNN for Temporal Action Detection

• SpikeTAD is a spiking-neural-network architecture that converts an ANN TAD model to a spike-compatible system for low-power, end-to-end video action detection.
• It employs a ViT-style backbone with Multi-Threshold Neurons and Expectation Compensation Modules to accurately extract spatiotemporal features and detect precise action boundaries.
• Empirical evaluations on THUMOS14 and ActivityNet-1.3 demonstrate competitive mAP scores while substantially reducing energy consumption compared to conventional ANN TAD baselines.

SpikeTAD is a spiking-neural-network architecture for end-to-end temporal action detection (TAD) in untrimmed video, proposed as the first SNN-based end-to-end TAD framework. It targets a deployment regime in which power consumption is a primary constraint, especially on mobile, edge, and neuromorphic hardware. The model is not a directly trained pure SNN; rather, it is an ANN-to-SNN conversion system that trains an ANN TAD model end-to-end and then converts the backbone and detector into spike-compatible modules. In the reported configuration, SpikeTAD achieves an average mAP of 67.2% on THUMOS14 and 37.42% on ActivityNet-1.3 while maintaining substantially lower estimated energy than ANN TAD baselines (Yang et al., 10 Jun 2026).

1. Problem setting and conceptual scope

Temporal action detection aims to predict both temporal boundaries and action categories in untrimmed video, typically as triplets $(s_i, e_i, c)$, where $s_i$ is the start time, $e_i$ is the end time, and $c$ is the action class. The task is harder than video classification because it requires long-range semantic understanding and precise boundary localization in videos containing substantial background (Yang et al., 10 Jun 2026).

SpikeTAD addresses this problem under a low-power deployment objective. The paper motivates this direction by noting that existing video understanding models are difficult to deploy because of their large size and prohibitive power consumption, while SNNs offer low-power and neuromorphic-friendly computation through sparse spike-driven accumulation. The central difficulty is that TAD introduces a semantic temporal axis from video itself, and SNN inference introduces an additional computational temporal axis through spike simulation. The paper explicitly frames this as a distinction between semantic time and computational time, and it treats the resulting “time multiplication problem” as the main obstacle to applying SNNs to TAD (Yang et al., 10 Jun 2026).

This positioning distinguishes SpikeTAD from prior spiking work on other vision tasks. The paper states that prior SNN work focused mostly on image classification, simple vision tasks, and some object detection, whereas SpikeTAD targets end-to-end temporal action detection from raw RGB video. Related SNN literature in object detection, tracking, test-time adaptation, and temporal aggregation addresses adjacent problems, but not end-to-end TAD in this formulation (Luo et al., 2 Dec 2025, Yang et al., 27 May 2025, Luo et al., 3 Apr 2025, Qin, 14 Mar 2026).

2. Architecture and module organization

SpikeTAD consists of two major parts: a video backbone and a temporal detector. The pipeline starts from raw untrimmed video, performs patch embedding and positional embedding, extracts spatiotemporal features with a ViT-style backbone, collapses spatial dimensions, constructs multi-scale temporal features, and applies a lightweight detection head for classification and boundary regression (Yang et al., 10 Jun 2026).

Component	Role	Spiking mechanism
Backbone	Spatiotemporal feature extraction	MTN + ECM
Detector	Temporal detection head	IF neurons
Output heads	Classification and regression	Direct mapping from postsynaptic potential features

The backbone is ViT-S from VideoMAE v2. The input video of size $T \times H \times W \times 3$ is split into patches, projected to tokens, injected with spatiotemporal positional embedding and positional embedding, and passed through $L$ transformer blocks. Each block contains Multi-Threshold Spiking Self-Attention (MSSA), Multi-Threshold Feedforward Network (MFFN), LayerNorm, and residual connections. The block equations are given as

$$X^{l} = \text{MSSA}(X^{l-1}) + X^{l-1}, \quad l=1,\dots,L$$

and

$$X^{l} = \text{MFFN}(X^{l}) + X^{l}, \quad l=1,\dots,L.$$

Within MSSA, the key, query, and value streams are written as

$X_{K,Q,V} = \text{MTN}(W_{K,Q,V}(\text{MTN}(X))),$

and the attention computation is expressed as

$$X^{l} = \text{MP}(\text{MTN}(\text{Softmax}(\text{MP}(X^{l-1}_{K},X^{l-1}_{Q}))),X^{l-1}_{V}),$$

followed by

$s_i$0

The detector is intentionally lightweight. After spatial dimensions are removed, SpikeTAD builds multi-scale temporal features using max pooling. The detection head replaces the standard ANN pattern of Conv1d + LN + ReLU with

$s_i$1

It uses separate classification and regression branches, and the final outputs are temporal boundaries and action labels $s_i$2 (Yang et al., 10 Jun 2026).

This architectural split is central to the method. The backbone uses richer spike-compatible modules to preserve transformer behavior under short simulation horizons, while the detector uses simpler IF neurons for efficiency. This suggests that SpikeTAD treats conversion fidelity and spike efficiency as module-specific design problems rather than applying a uniform neuron substitution everywhere.

3. Spiking formulation and ANN-to-SNN conversion

SpikeTAD is explicitly described as a hybrid ANN-to-SNN converted architecture rather than a directly trained pure SNN. During training, the model remains an ANN. During inference and conversion, backbone nonlinearities are replaced by Multi-Threshold Neurons (MTN) and Expectation Compensation Modules (ECM), while detector ReLUs are replaced by Integrate-and-Fire (IF) neurons (Yang et al., 10 Jun 2026).

The detector uses the standard IF neuron with reset-by-subtraction: $s_i$3

$s_i$4

$s_i$5

$s_i$6

The paper emphasizes reset-by-subtraction as a way to reduce information loss.

For the backbone, a simple IF neuron is considered insufficient for nonlinear transformer components such as GELU, LayerNorm, and Softmax-related behavior. SpikeTAD therefore introduces the Multi-Threshold Neuron. MTN uses $s_i$7 thresholds, with positive thresholds

$s_i$8

and negative thresholds

$s_i$9

Its dynamics are written as

$e_i$0

$e_i$1

$e_i$2

$e_i$3

The paper states that MTN increases information capacity per time-step, but also increases communication bandwidth and energy relative to vanilla IF, and is therefore used only where necessary (Yang et al., 10 Jun 2026).

The conversion theory begins from an ANN layer

$e_i$4

and maps ANN activations to SNN firing rates. From IF dynamics, SpikeTAD derives

$e_i$5

and

$e_i$6

As $e_i$7 increases or the residual term becomes negligible, the paper states that

$e_i$8

A further conversion component is the quantization clip-floor activation with a shift term. The ANN detector uses a clip-floor-shift quantized activation, and the paper sets the shift parameter to

$e_i$9

with initial membrane potential

$c$0

The paper states that this makes the expected conversion error zero under a local uniformity assumption. In the detector, this shift is one of the strongest mechanisms for achieving low-step conversion with limited accuracy loss (Yang et al., 10 Jun 2026).

4. Expectation compensation, training, and inference

The Expectation Compensation Module is introduced to address short-horizon conversion error, especially in nonlinear layers. For a nonlinear function $c$1, and cumulative input spikes

$c$2

the exact spike output at step $c$3 is written as

$c$4

This lets the model reconstruct the current-step increment from cumulative expected outputs.

For matrix products in self-attention, the cumulative sums

$c$5

lead to

$c$6

with recursive update

$c$7

and expected output

$c$8

The paper uses ECM to preserve the behavior of Softmax, GELU, LayerNorm, and matrix products in self-attention (Yang et al., 10 Jun 2026).

Training proceeds in two stages. First, the model is trained end-to-end as an ANN TAD system using raw RGB input. Second, the trained ANN is converted into an SNN: backbone nonlinearities become MTN + ECM, detector ReLUs become IF neurons, and weights are transferred directly. The losses are standard TAD losses, with focal loss for classification and DIoU loss for regression: $c$9 The paper explicitly notes that SpikeTAD does not rely on direct surrogate-gradient SNN training (Yang et al., 10 Jun 2026).

At inference time, the detector feature tensor is replicated across simulation steps, the membrane is initialized at half-threshold, and IF neurons accumulate and fire across $T \times H \times W \times 3$0 steps. The accumulated detector output is

$T \times H \times W \times 3$1

This division of labor between ANN training and SNN deployment is a defining property of SpikeTAD. In relation to other spiking detectors, this is notably different from directly trained SNN object detection frameworks such as TDE-based detectors, which are optimized with direct spiking training rather than conversion (Luo et al., 2 Dec 2025).

5. Datasets, implementation, and empirical results

SpikeTAD is evaluated on THUMOS14 and ActivityNet-1.3. For THUMOS14, the paper reports 413 videos at 30 fps, with 200 validation videos, 213 test videos, and 20 categories. For ActivityNet-1.3, it reports 200 classes, 10,024 training videos, and 4,926 validation videos for testing (Yang et al., 10 Jun 2026).

Training and evaluation use 768-frame windows. On THUMOS14, the method randomly truncates a window of 768 frames with temporal stride 4. On ActivityNet-1.3, videos are resized to a fixed length of 768 frames. Training augmentation includes sampling 768 consecutive frames, resizing the shorter side to 182, random cropping from $T \times H \times W \times 3$2, crop area 90%–100%, resizing to $T \times H \times W \times 3$3, 50% horizontal flip, image enhancement, and color jitter. Testing uses a 768-frame sliding window, shorter side 160, and center crop $T \times H \times W \times 3$4. Optimization uses AdamW, warm-up, and cosine learning rate decay, with learning rate $T \times H \times W \times 3$5 for THUMOS14 and $T \times H \times W \times 3$6 for ActivityNet-1.3. Batch size is 2 on THUMOS14 and 16 on ActivityNet-1.3, training lasts 60 and 15 epochs respectively, and the random seed is 42 (Yang et al., 10 Jun 2026).

The reported spiking hyperparameters are: time steps $T \times H \times W \times 3$7, quantization step $T \times H \times W \times 3$8, scaling factor $T \times H \times W \times 3$9, number of thresholds in MTN $L$0, Softmax thresholds $L$1 and $L$2, and linear thresholds $L$3 and $L$4 (Yang et al., 10 Jun 2026).

Configuration	Avg mAP	Power (mJ)
Baseline(ANN), THUMOS14	67.3	400.4
SpikeTAD $L$5, THUMOS14	64.0	84.1
SpikeTAD $L$6, THUMOS14	66.5	184.2
SpikeTAD $L$7, THUMOS14	67.2	356.4
Baseline(ANN), ActivityNet-1.3	37.64	—
SpikeTAD $L$8, ActivityNet-1.3	37.42	—

On THUMOS14, SpikeTAD reports 64.0 average mAP at $L$9, 66.5 at $$X^{l} = \text{MSSA}(X^{l-1}) + X^{l-1}, \quad l=1,\dots,L$$0, and 67.2 at $$X^{l} = \text{MSSA}(X^{l-1}) + X^{l-1}, \quad l=1,\dots,L$$1. On ActivityNet-1.3, it reports 36.53, 37.05, and 37.42 at the same time-step settings. The $$X^{l} = \text{MSSA}(X^{l-1}) + X^{l-1}, \quad l=1,\dots,L$$2 model is therefore extremely close to the ANN counterpart, for which the paper reports 67.3 on THUMOS14 and 37.64 on ActivityNet-1.3 (Yang et al., 10 Jun 2026).

The IoU breakdown for THUMOS14 at $$X^{l} = \text{MSSA}(X^{l-1}) + X^{l-1}, \quad l=1,\dots,L$$3 is [email protected] = 82.1, [email protected] = 77.9, [email protected] = 70.5, [email protected] = 59.9, and [email protected] = 45.6. For ActivityNet-1.3 at $$X^{l} = \text{MSSA}(X^{l-1}) + X^{l-1}, \quad l=1,\dots,L$$4, the paper reports [email protected] = 56.04, [email protected] = 38.97, and [email protected] = 8.75 (Yang et al., 10 Jun 2026).

The ablations are central to the interpretation of the method. On THUMOS14, the paper compares an ANN baseline, a directly trained SNN, a naive ANN2SNN conversion, and the proposed conversion strategy. At $$X^{l} = \text{MSSA}(X^{l-1}) + X^{l-1}, \quad l=1,\dots,L$$5, the directly trained SNN reports 62.5 average mAP, naive ANN2SNN reports 62.1, and the proposed ANN2SNN strategy reports 67.2. A stronger detector-specific ablation shows that standard clip-floor activation is much worse and needs far more time-steps: the paper reports 13.9 average mAP at $$X^{l} = \text{MSSA}(X^{l-1}) + X^{l-1}, \quad l=1,\dots,L$$6 and 60.9 at $$X^{l} = \text{MSSA}(X^{l-1}) + X^{l-1}, \quad l=1,\dots,L$$7 for standard clip-floor, versus 67.3 at $$X^{l} = \text{MSSA}(X^{l-1}) + X^{l-1}, \quad l=1,\dots,L$$8 for clip-floor shift (Yang et al., 10 Jun 2026).

The detector-only conversion ablation shows that detector conversion is comparatively easy: on THUMOS14, detector-only conversion reaches 67.2 average mAP already at $$X^{l} = \text{MSSA}(X^{l-1}) + X^{l-1}, \quad l=1,\dots,L$$9, while the full system still exhibits a stronger time-step dependence. The paper interprets this as evidence that backbone conversion is the harder bottleneck (Yang et al., 10 Jun 2026).

6. Energy analysis, limitations, and place in the literature

SpikeTAD’s energy analysis uses 32-bit floating point energy assumptions at 45 nm: $$X^{l} = \text{MFFN}(X^{l}) + X^{l}, \quad l=1,\dots,L.$$0 The SNN/ANN energy ratio is defined as

$$X^{l} = \text{MFFN}(X^{l}) + X^{l}, \quad l=1,\dots,L.$$1

with

$$X^{l} = \text{MFFN}(X^{l}) + X^{l}, \quad l=1,\dots,L.$$2

The paper reports full-model energy ratios on THUMOS14 of 0.21 at $$X^{l} = \text{MFFN}(X^{l}) + X^{l}, \quad l=1,\dots,L.$$3, 0.46 at $$X^{l} = \text{MFFN}(X^{l}) + X^{l}, \quad l=1,\dots,L.$$4, and 0.89 at $$X^{l} = \text{MFFN}(X^{l}) + X^{l}, \quad l=1,\dots,L.$$5. In power terms, the reported values are 84.1 mJ, 184.2 mJ, and 356.4 mJ, compared with 400.4 mJ for the ANN baseline (Yang et al., 10 Jun 2026).

These numbers place SpikeTAD in a distinct operating regime: it is not the highest-accuracy TAD model in absolute terms, but it is explicitly designed for low-power end-to-end detection. The paper compares it with ANN TAD methods such as ActionFormer, ViT-TAD, AdaTAD, TALLFormer, and Progressive Block Drop, and argues that SpikeTAD is competitive in accuracy while using much less power. A plausible implication is that the method is best understood as a systems-oriented TAD architecture rather than a pure accuracy-maximization model (Yang et al., 10 Jun 2026).

The paper also reports stability statistics. Over 5 independent runs, SpikeTAD$$X^{l} = \text{MFFN}(X^{l}) + X^{l}, \quad l=1,\dots,L.$$6 on THUMOS14 achieves $$X^{l} = \text{MFFN}(X^{l}) + X^{l}, \quad l=1,\dots,L.$$7 average mAP with a 95% confidence interval of $$X^{l} = \text{MFFN}(X^{l}) + X^{l}, \quad l=1,\dots,L.$$8. Its false positive analysis identifies background errors and localization errors as the dominant remaining failure modes (Yang et al., 10 Jun 2026).

The stated limitations are equally specific. SpikeTAD depends on powerful pretrained ANN video backbones, so it currently relies on conversion rather than from-scratch SNN training. Video SNNs remain burdened by the coexistence of semantic time and computational time. The backbone uses high-bit MTN rather than purely binary spikes, which improves fidelity but increases communication and power relative to simpler binary SNNs. The paper suggests two future directions: end-to-end SNN TAD architectures trainable from scratch, and improved spike-firing or conversion mechanisms that preserve accuracy with lower power overhead (Yang et al., 10 Jun 2026).

Within the broader SNN literature, SpikeTAD occupies a specific niche. Unlike directly trained spiking object detectors enhanced by temporal modules such as TDE, it is a conversion-based video detector rather than a surrogate-gradient detector (Luo et al., 2 Dec 2025). Unlike SpikeFET, which is a fully spiking framework for unified frame-event object tracking, it addresses temporal localization in untrimmed RGB video rather than single-object tracking (Yang et al., 27 May 2025). Unlike TAC and TAC-TP, which study data-dependent temporal aggregation for SNN acceleration, SpikeTAD focuses on end-to-end TAD performance and power trade-offs rather than convolution-call reduction (Qin, 14 Mar 2026). This suggests that SpikeTAD’s main historical significance lies in opening low-power temporal action detection as a concrete SNN application domain rather than in solving the more general problems of SNN temporal modeling or acceleration.