Spatial-Channel-Temporal-Fused Attention (SCTFA)

• SCTFA is a biologically inspired attention module that fuses spatial, channel, and temporal signals within Spiking Neural Networks to enhance performance and data stability.
• It employs a plug-and-play design that integrates attention into convolutional SNN layers by gating LIF neuron voltage updates, leveraging predictive remapping.
• Experimental results demonstrate significant improvements in accuracy and robustness, with minimal computational overhead across diverse neuromorphic datasets.

The Spatial-Channel-Temporal-Fused Attention (SCTFA) module is a biologically inspired architectural component designed to enhance the performance of Spiking Neural Networks (SNNs) by fusing spatial, channel, and temporal saliency within the network’s processing pipeline. SCTFA operates as a plug-and-play block for convolutional SNN layers, propagating spatial-channel attention into subsequent time steps through the leaky integration mechanism, thereby mimicking predictive attentional remapping observed in biological perception. The method systematically integrates attention with native SNN temporal dynamics, resulting in improved accuracy, robustness to noise, and stability under incomplete data, with minimal computational overhead (Cai et al., 2022).

1. Architectural Motivation and Conceptual Overview

SNNs encode information via discrete spikes and capture temporal dependencies through membrane-potential decay in Leaky Integrate-and-Fire (LIF) neurons. However, standard SNN architectures lack explicit mechanisms to prioritize salient regions, channels, or temporal intervals. The SCTFA module addresses this gap by introducing an end-to-end differentiable @@@@1@@@@ that operates over the spatiotemporal spike activity and incorporates both spatial and channel cues.

SCTFA wraps each convolutional layer at each time step, transforming the layer’s binary spike output $S^{t,l} \in \{0,1\}^{H \times W \times C}$ into a real-valued attention tensor $U_{SE}^{t,l} \in [0,1]^{H \times W \times C}$. This tensor gates the membrane-potential updates of subsequent steps, so attention extracted at the current time influences the network’s future sensitivity, analogously to predictive attentional remapping in biological systems. The effect accumulates due to the natural temporal memory intrinsic to LIF neuron dynamics (Cai et al., 2022).

2. Mathematical Formulation of SCTFA Branches

At the core of SCTFA is a three-pathway calculation for spatial, channel, and temporal attention signals, followed by their fusion and direct modulation of the neuron voltage update.

2.1 Spatial Attention

Spatial attention is computed by “squeezing” the channel dimension at each spatial location using a $1 \times 1$ convolution followed by a sigmoid nonlinearity:

$$U_{sSE}^{t,l} = \sigma\left( \text{Conv}_{1\times1}(S^{t,l}; W_s^l, b^l) \right)$$

where $U_{sSE}^{t,l} \in [0,1]^{H \times W}$. Explicitly,

$$U_{sSE}^{t,l}(i,j) = \sigma \left( \sum_{c=1}^C W_{s,c}^l \cdot S^{t,l}(i, j, c) + b^l \right)$$

2.2 Channel Attention

Channel attention is extracted by spatially average-pooling each channel, then passing the result through a two-layer bottleneck network with reduction ratio $r$:

$$e^{t,l}(c) = \frac{1}{H \cdot W} \sum_{i=1}^H \sum_{j=1}^W S^{t,l}(i, j, c)$$

$$z = \text{ReLU}(W_{c1}^l e^{t,l}), \quad U_{cSE}^{t,l} = \sigma(W_{c2}^l z)$$

where $U_{cSE}^{t,l} \in [0,1]^C$.

2.3 Fusion Mechanism

Spatial and channel attention are fused by broadcasting and elementwise multiplication (Hadamard product), yielding the 3D attention tensor:

$$U_{SE}^{t,l}(i, j, c) = U_{sSE}^{t,l}(i, j) \cdot U_{cSE}^{t,l}(c)$$

2.4 Temporal Integration via Membrane Update

For each neuron $i$ in layer $l$, the standard LIF update is:

$$v_i^{t+1, l} = \kappa_{\tau} v_i^{t, l}(1-s_i^{t, l}) + \sum_j w_{ij}^{l, l-1} s_j^{t+1, l-1}$$

In SCTFA, the voltage is gated multiplicatively by the attention tensor:

$$v_i^{t+1, l} = \kappa_{\tau} v_i^{t, l} u_{SE;i}^{t, l}(1-s_i^{t, l}) + \sum_j w_{ij}^{l, l-1} s_j^{t+1, l-1}$$

with $\kappa_{\tau} = 1 - \Delta t / \tau$ the decay factor. This temporal accumulation means that attention effects “stick” to the membrane, propagating spatial-channel saliency across time.

3. Integration Algorithm and Training Pipeline

At each time step $t$ across all $L$ convolutional layers, the SCTFA algorithm follows these steps:

1. Compute spike activations $S^{t,l}$ using the surrogate gradient spike function.
2. For each convolutional layer:
   • Compute spatial attention $U_{sSE}^{t,l}$, pooled channel vector $e^{t,l}$, channel attention $U_{cSE}^{t,l}$, and fuse into $U_{SE}^{t,l}$.
   • Update neuron voltages via attention-gated LIF rule.
3. For non-convolutional layers, standard LIF update applies.
4. Accumulate output spikes for final-layer temporal voting.
5. After $T$ timesteps, decode class predictions by averaging spikes and computing mean-squared error loss.
6. Backpropagate through time using surrogate gradients, with differentiability ensured through $U_{SE}$ dependencies.

Crucial hyperparameters include leaky integration time constant, surrogate gradient (arctan with $\alpha = 2$), optimizer settings (Adam, exponential decay), channel attention reduction ratio ($r = 4$), and dataset-dependent architectural configurations. Training is performed with 100–200 epochs and batch sizes of 16–100. Each dataset’s convolutional/fc layer layout is detailed in the original work (Cai et al., 2022).

4. Computational Complexity and Performance Overhead

The inclusion of SCTFA introduces minimal computational burden:

• Parameter increase per convolutional layer is $0.2$\%–$1.0$\%, originating from the extra $1\times1$ conv and two FC layers per layer.
• Multiply–add operation count increases by $0.3$\%.
• Inference latency rises by $11$–$43$ ms per batch, dataset and model size dependent (see Tab. 4 in the source).
• No qualitative increase in model complexity arises, owing to the efficiency of the spatial and channel gating mechanisms.

5. Experimental Validation and Performance

Systematic evaluation on DVS-Gesture, SL-Animals-DVS, and MNIST-DVS event stream datasets demonstrates the efficacy of SCTFA:

• Full SCTFA module achieves DVS 97.3\% (+6.5\%), SL-Animals 86.6\% (+5.1\%), and MNIST-DVS 98.7\% (+1.0\%) compared to the baseline SNN, outperforming degenerate attention versions (spatial-temporal only, channel-temporal only).
• SCTFA-SNN retains 5–10\% higher accuracy than baseline and more stable activation drift under strong Poisson noise ($\lambda=0.5$ Hz), indicating robustness (Fig. 8).
• For randomly missing events or dropped frames up to 50\%, SCTFA-SNN exhibits significantly less degradation in accuracy compared to spatial- or channel-temporal-only modules (Fig. 9).
• Benchmarking against state-of-the-art on SL-Animals-DVS and MNIST-DVS, SCTFA-SNN achieves new leading scores (90.04\% and 98.90\%, respectively), and competitive accuracy on DVS-Gesture (97.92\%, up to 98.96\% with longer simulation; see Table 5).

6. Critical Implementation Factors and Reproducibility

• Neuron model: Leaky Integrate-and-Fire with dataset-calibrated $\kappa_{\tau}$.
• Decoder: temporal spike-rate voting with mean-squared error loss.
• Surrogate gradients: arctan-based for differentiability in spike function approximations.
• Training: Adam optimizer with exponential learning rate decay.
• Simulation: default $T$, $\Delta t$ time step widths per dataset; 100–200 epochs, batch size 16–100.
• Channel attention bottleneck: reduction ratio $r=4$.
• All architectural parameters, convolutional/fc layer layouts, and optimizer settings per dataset are specified in Table 1–2 of the source.

A direct implication is that, with careful selection of hyperparameters and adherence to the implementation details above, the SCTFA module can be integrated into a variety of convolutional SNN architectures without impacting their tractability or reproducibility.

7. Context, Significance, and Potential Directions

SCTFA marks an advance in the integration of biologically inspired attention with spike-based temporal computation. By unifying spatial and channel saliency with the intrinsic memory of LIF neurons, SCTFA delivers quantifiable improvements in accuracy, robustness, and data stability at negligible incremental cost. The approach illustrates how interpretive mechanisms from neuroscience—predictive attentional remapping in particular—can be fruitfully transposed into SNN architectures.

A plausible implication is that more finely resolved attention mechanisms, especially those that capitalize on the asymmetric and history-dependent properties of spike-driven computation, may yield further gains in event-driven perception or neuromorphic inference under constrained resources (Cai et al., 2022).