Spiking Spatiotemporal Attention & Denoising

• Spiking spatiotemporal attention and denoising is a computational approach that combines spike-based neural models with mechanisms for selecting and filtering spatial and temporal information.
• It employs both neuron-level temporal integration via learnable membrane constants and network-level attention with strategies like power-of-two decay to enhance feature extraction.
• Empirical results demonstrate significant improvements in accuracy, sparsity, and energy efficiency on benchmarks such as CIFAR-10, CIFAR-100, and DVS-Gesture.

Spiking spatiotemporal attention and denoising integrates biologically inspired spiking neural computation with explicit mechanisms for selectively integrating and denoising information over both spatial and temporal dimensions. By exploiting the timing and accumulation of spikes in neural architectures, these methods enhance representational efficiency and robustness, enabling strong performance in static and event-driven vision tasks. Several recent frameworks—including DISTA, DS2TA, FSTA-SNN, and SCTFA—exemplify the state-of-the-art in operationalizing spatiotemporal attention and denoising in spiking networks and spiking transformers.

1. Spiking Spatiotemporal Attention: Mechanisms and Formulations

Core to spiking spatiotemporal attention is the realization of attention in both space and time, mapped onto event-based neural substrates. Architectures such as DISTA (Xu et al., 2023) and DS2TA (Xu et al., 2024) extend the classical transformer attention mechanism to work with spiking neuron models (typically leaky integrate-and-fire, LIF, with discretized time), with two principal approaches:

• Neuron-Level Temporal Attention: Each LIF neuron acts as an exponential integrator over past inputs, implemented via a learnable membrane time constant $\tau_m$. Larger $\tau_m$ values create longer temporal receptive windows, enabling neurons to integrate over multiple time steps. In DISTA, this parameter is made learnable and updated via backpropagation through time (BPTT), yielding per-neuron intrinsic spatiotemporal attention.
• Network-Level or Explicit Spatiotemporal Attention: Instead of restricting attention to the current step, spike histories over a window $W$ (or $T_{AW}$) are aggregated when forming Q/K/V projections. In DS2TA, synaptic contributions from previous timesteps are modulated by a learned attenuation function $df(m) = 2^{-\tau^{(l)} m}$, parameterized per layer, avoiding explicit storage of multiple projection matrices.

In both cases, spatiotemporal attention maps $A_{t}^{(l)}$ or $S^{(l)}[t]$ are constructed by (possibly) binary or low-precision dot products of spike-driven Q/K activations, yielding $N\times N$ attention tensors at each timestep $t$ and block $l$.

Table: Primary Spatiotemporal Attention Mechanisms

Framework	Neuron-Level	Network-Level	Attenuation Scheme
DISTA	Learnable $\tau_m$0	Temporal window $\tau_m$1	None (raw sum)
DS2TA	Fixed $\tau_m$2 (learned per block)	Attenuated window $\tau_m$3	Power-of-two decay $\tau_m$4

2. Nonlinear and Frequency-Domain Denoising Strategies

Denoising in spiking attention addresses the high variance and sparsity of spike-driven activations.

• Binary Thresholding: DISTA uses a simple threshold $\tau_m$5 for the $\tau_m$6 attention maps. Any entry below threshold is zeroed, enforcing sparsity and removing low-confidence correlations.

$\tau_m$7

• Hashmap-Based Nonlinear Denoising: DS2TA introduces a per-head, integer hashmap-based function (NSAD) for efficient, parameter-light denoising. A small look-up table captures a parameterized nonlinear mapping (thresholding and nonlinear amplification) applied to each attention score $\tau_m$8.

$\tau_m$9

The hashmap contains $W$0 entries per attention head and eliminates the need for continuous computation or large parameter stores.

• Frequency-Domain Denoising: FSTA-SNN (Yu et al., 2024) applies a 2D Discrete Cosine Transform (DCT) to average spike activity, generating a frequency map. Low-frequency redundant (background) components are selectively suppressed by learned gating in the DCT domain, and spatial/temporal weights modulate the spike map before residual addition. This explicitly denoises spatial frequencies, reducing spike noise and amplifying relevant features.

3. Integrated Architectures: Transformer and CNN Backbones

Several backbone architectures operationalize these attention and denoising modules in different contexts:

• Spiking Transformers: DISTA and DS2TA replace linear/MLP projections with LIF neuron arrays, allowing joint training of synaptic weights and intrinsic (biophysical) parameters. Self-attention layers process spike-driven sequences with temporal integration, followed by denoising and MLP blocks. DS2TA achieves parameter efficiency by reusing weight matrices and only adding tiny per-layer and per-head parameter sets.
• Spiking CNNs with Attention: FSTA-SNN and SCTFA (Cai et al., 2022) design attention modules that can be inserted after each convolutional/LIF block. FSTA-SNN employs frequency-domain spatial attention using non-trainable DCT bases and learned scalar gates, while SCTFA fuses spatial, channel, and temporal attention via gating mechanisms injected directly into the LIF state update equation.
• Temporal Attention Implementation: Temporal fusion can be “predictive-remapping” (SCTFA) where past attention guides LIF decay, or explicit cross-time aggregation as in transformers.

4. Training Procedures and Surrogate Gradient Learning

Spiking networks use surrogate gradient methods to bypass the non-differentiability of spike events during BPTT. DISTA and DS2TA use cross-entropy loss on time-aggregated firing rates or membrane potentials and update both synaptic weights and intrinsic parameters (e.g., $W$1, attenuation exponents, or denoiser parameters). FSTA-SNN and SCTFA leverage surrogate approximations for gradient flow through the spike function; e.g., arctan or piecewise linear functions approximate the Heaviside activation.

For energy- and spike-rate efficiency, sparsity-boosting mechanisms in attention/denoising directly feedback into forward and backward dynamics by modulating attention weights, LIF membrane traces, or convolutional features based on historical focus.

5. Empirical Results and Impact on Robustness, Accuracy, and Efficiency

Integrating spiking spatiotemporal attention with nonlinear denoising yields improvements across multiple metrics:

• Accuracy Gains: DISTA achieves 96.26% top-1 accuracy on CIFAR-10 and 79.15% on CIFAR-100 (6 time steps), outperforming spatial-only spiking vision transformers by +0.92% and +0.79% respectively (Xu et al., 2023). DS2TA achieves 94.92% (CIFAR-10, T=4) and 77.47% (CIFAR-100), and 94.44% on DVS-Gesture (Xu et al., 2024). FSTA-SNN raises CIFAR-10 accuracy to 96.52% (ResNet19, T=2), above previous SNN benchmarks (Yu et al., 2024). SCTFA boosts DVS Gesture accuracy from 90.87% to 97.33% (Cai et al., 2022).
• Noise Robustness and Data Incompleteness: SCTFA and FSTA-SNN demonstrate significant resilience under Poisson spike noise and event/frame dropping, with SCTFA maintaining high accuracy and hidden state stability even at high loss rates (Cai et al., 2022). Frequency attention in FSTA-SNN is shown to suppress redundant low-frequency activity and concentrate salient energy, further reducing vulnerability to random spike perturbations (Yu et al., 2024).
• Sparsity and Energy Savings: DS2TA's attention maps achieve >97% sparsity in all transformer blocks, reducing per-block attention energy consumption by up to 91.9% compared to spatial-only baselines (Xu et al., 2024). FSTA-SNN reduces average firing rates by 34% with essentially unchanged energy per inference (Yu et al., 2024).

6. Comparative Summary and Directions

Diverse implementations of spiking spatiotemporal attention and denoising share key principles: adaptation of temporal windows at the neuron or network level, lightweight or parameter-free integration schemes (e.g., power-of-two decay, DCT/frequency bases), and highly efficient denoising mechanisms (thresholding, hashmaps, spectral gating). Ablation studies consistently show that the combination of spatiotemporal attention and denoising is synergistic: omitting either diminishes both performance and robustness (Xu et al., 2023, Xu et al., 2024).

Table: Key Performance Metrics on Vision Benchmarks

Model	CIFAR-10 Top-1	CIFAR-100 Top-1	CIFAR10-DVS	DVS-Gesture	Attention Map Sparsity
DISTA	96.26%	79.15%	79.1%	—	—
DS2TA	94.92%	77.47%	79.1%	94.44%	>97%
FSTA-SNN	96.52%	80.42%	81.5%–82.7%	—	N/A (spike rate -34%)
SCTFA-SNN	—	—	—	97.33%	—

A plausible implication is that the systematic integration of spiking spatiotemporal attention and efficient denoising will become foundational for energy-efficient, noise-robust computing in neuromorphic vision and temporally complex learning domains. These schemes are also well-suited for real-world hardware deployment due to their parameter and compute efficiency.

7. Connections, Limitations, and Outlook

While current spiking spatiotemporal attention frameworks deliver compelling efficiency and robustness, limitations include the reliance on surrogate gradient methods and the challenge of scaling such discrete, event-driven methods to larger, more diverse datasets. Extensions include hybrid training methods, hardware-aware design considerations (e.g., for Loihi, TrueNorth), and the adaptation to continuous time-event streams or expanded sensory modalities (Cai et al., 2022). Incorporating advanced plasticity rules and biophysically realistic conductances remains an open research frontier.