Temporal Retention Channel in SNNs

• Temporal Retention Channel is a mechanism that jointly preserves and amplifies informative spike features by integrating temporal evidence with channel-specific reweighting.
• It employs dual branches—identical attention (T-XA) for correlation and non-identical attention (T-NA) for dependency—to capture complex temporal-channel interactions.
• This approach improves recognition accuracy and efficiency in SNN architectures, with demonstrated performance gains on CIFAR10/100 and ImageNet benchmarks.

Temporal Retention Channel denotes a joint mechanism for preserving, aggregating, and selectively amplifying useful information across time steps and channels, rather than treating temporal and channel structure in isolation. In the recent spiking-neural-network literature, the term corresponds most closely to the Dual Temporal-channel-wise Attention mechanism, where temporal evidence is retained by attention over temporal slices and temporal-channel combinations, then reapplied to spike features through residual and multiplicative gating (Kim et al., 13 Mar 2025). In this sense, a Temporal Retention Channel is not a standalone recurrent memory state, but a structured pathway by which temporally informative and channel-specific spike responses are preserved, emphasized, and transmitted to downstream computation.

1. Conceptual scope

In spiking neural networks (SNNs), temporal information is intrinsic to the computation because neurons integrate inputs over multiple simulation steps and emit spikes according to membrane dynamics. The relevant neuron state evolves as

$$u^n(t) = \tau u^n(t-1) \odot (1-s^n(t-1)) + c^n(t),$$

with

$$c^n(t) = W^n \star s^{n-1}(t), \qquad s^n(t) = \mathbb{H}(u^n(t)-V_{th}).$$

Because the membrane state carries history, recognition quality depends on how effectively temporal evidence accumulated across steps is retained and emphasized (Kim et al., 13 Mar 2025).

Within this setting, “temporal retention” is not explicitly formalized as a separate theorem or scalar coefficient. Rather, it is implicit in mechanisms that preserve temporal resolution, compute attention over temporal slices or temporal-channel combinations, and avoid erasing baseline information through residual paths. A Temporal Retention Channel therefore refers to a soft retention scheme in which informative temporal-channel components are amplified and weak ones are attenuated, while the original spike content remains accessible (Kim et al., 13 Mar 2025).

The central distinction in the DTA formulation is between two complementary views of temporal-channel structure. Identical attention operations handle temporal and channel dimensions by the same form of operation under a unified strategy, and are described as suitable for learning correlation. Non-identical attention operations use different mechanisms for different dimensions or scales, and are described as suitable for learning dependency. The Temporal Retention Channel, in this reading, is the combined pathway that captures both correlation and dependency over time and channels (Kim et al., 13 Mar 2025).

2. Architectural realization in Dual Temporal-channel-wise Attention

The most explicit realization of a Temporal Retention Channel in the supplied literature is the Dual Temporal-channel-wise Attention (DTA) module for SNNs. DTA is inserted as a single plug-in attention block into an MS-ResNet-based SNN, and the paper emphasizes that a single DTA block is sufficient to outperform earlier SNNs that often relied on multiple attention blocks (Kim et al., 13 Mar 2025).

The input is a spiking feature tensor

$X \in \mathbb{R}^{T \times C \times H \times W},$

where $T$ is the number of simulation time steps, $C$ is the channel count, and $H,W$ are spatial dimensions. DTA operates over the full temporal-channel-spatial representation and outputs a reweighted tensor of the same semantic role (Kim et al., 13 Mar 2025).

DTA contains two parallel submodules:

1. T-XA: Temporal-channel-wise identical Cross Attention
2. T-NA: Temporal-channel-wise Non-identical Attention

Their outputs are fused as

$$O_{DTA} = \sigma(O_{T-XA} \odot O_{T-NA}) \odot Spikes,$$

where $O_{T-XA}$ and $O_{T-NA}$ are the branch outputs, $\odot$ is element-wise multiplication, and $$c^n(t) = W^n \star s^{n-1}(t), \qquad s^n(t) = \mathbb{H}(u^n(t)-V_{th}).$$0 is sigmoid. This equation is the most direct expression of the Temporal Retention Channel in the DTA framework: the module generates a joint temporal-channel weighting mask and applies it multiplicatively to the original spike features (Kim et al., 13 Mar 2025).

The architectural significance of this design lies in its division of labor. T-XA models temporal-channel correlation under identical operations, while T-NA models temporal-channel dependency under non-identical operations. Their fusion yields a richer joint mask than either branch alone (Kim et al., 13 Mar 2025).

3. Correlative retention and dependency-aware modulation

The first branch, T-XA, is described as a global attention branch composed of two local attention pathways:

• Temporal-wise Local Attention (TLA)
• Channel-wise Local Attention (CLA)

These are combined by

$$c^n(t) = W^n \star s^{n-1}(t), \qquad s^n(t) = \mathbb{H}(u^n(t)-V_{th}).$$1

Construction begins with Spatial Mean Pooling (SMP): $$c^n(t) = W^n \star s^{n-1}(t), \qquad s^n(t) = \mathbb{H}(u^n(t)-V_{th}).$$2 Here $$c^n(t) = W^n \star s^{n-1}(t), \qquad s^n(t) = \mathbb{H}(u^n(t)-V_{th}).$$3 and $$c^n(t) = W^n \star s^{n-1}(t), \qquad s^n(t) = \mathbb{H}(u^n(t)-V_{th}).$$4 denote the target and subtarget dimensions. In practice, TLA treats temporal dimension as target and channel as subtarget, while CLA does the converse. This establishes explicit cross-conditioning between time and channel axes (Kim et al., 13 Mar 2025).

Each local branch then applies a 1D convolution along the target dimension, followed by sigmoid gating, a learnable scaling coefficient, and a residual connection: $$c^n(t) = W^n \star s^{n-1}(t), \qquad s^n(t) = \mathbb{H}(u^n(t)-V_{th}).$$5 Here $$c^n(t) = W^n \star s^{n-1}(t), \qquad s^n(t) = \mathbb{H}(u^n(t)-V_{th}).$$6 is a 1D convolution along $$c^n(t) = W^n \star s^{n-1}(t), \qquad s^n(t) = \mathbb{H}(u^n(t)-V_{th}).$$7, $$c^n(t) = W^n \star s^{n-1}(t), \qquad s^n(t) = \mathbb{H}(u^n(t)-V_{th}).$$8 produces an attention map, $$c^n(t) = W^n \star s^{n-1}(t), \qquad s^n(t) = \mathbb{H}(u^n(t)-V_{th}).$$9 is a learnable parameter, and $X \in \mathbb{R}^{T \times C \times H \times W},$0 denotes residual addition. This residual form is central to retention: temporal and channel cues are preserved and reweighted rather than overwritten (Kim et al., 13 Mar 2025).

The second branch, T-NA, addresses dependency over the merged temporal-channel axis. The input is first reshaped from

$X \in \mathbb{R}^{T \times C \times H \times W},$1

to

$X \in \mathbb{R}^{T \times C \times H \times W},$2

so that temporal and channel indices are treated jointly as a 2D feature channel axis. This avoids expensive 3D attention or 3D convolution over the full tensor while still allowing interactions over all temporal-channel combinations (Kim et al., 13 Mar 2025).

The reshaped tensor is encoded as

$X \in \mathbb{R}^{T \times C \times H \times W},$3

where $X \in \mathbb{R}^{T \times C \times H \times W},$4 denotes a $X \in \mathbb{R}^{T \times C \times H \times W},$5 convolution plus GELU. T-NA then splits into:

• Local Temporal-Channel Attention (LTCA) for intra-dependency

$X \in \mathbb{R}^{T \times C \times H \times W},$6

with depth-wise, dilation-depth-wise, and point-wise convolutions;

• Global Temporal-Channel Attention (GTCA) for inter-dependency

$X \in \mathbb{R}^{T \times C \times H \times W},$7

with global average pooling and an MLP bottleneck of Linear-ReLU-Linear.

These are combined as

$X \in \mathbb{R}^{T \times C \times H \times W},$8

and decoded with residual addition: $X \in \mathbb{R}^{T \times C \times H \times W},$9

A plausible implication is that T-XA acts as a correlative retention mask, whereas T-NA acts as a dependency-aware retention mask over local and global contexts. In both cases, residual connections ensure that the original temporal-channel evidence is not erased (Kim et al., 13 Mar 2025).

4. Relation to prior SNN attention mechanisms

DTA is positioned as broader than temporal-only or channel-only attention in SNNs. The paper contrasts it with several earlier approaches. TA focuses on temporal-wise attention. Attention Spiking Neural Networks apply multi-dimensional attention sequentially over temporal, channel, and spatial dimensions using non-identical operations. TCJA-SNN jointly combines temporal and channel information, but is characterized as using an identical attention mechanism at the same stage. GAC-SNN uses a single attention block at the input stage and non-identical attention for temporal variation, improving efficiency over methods requiring multiple blocks. DTA differs by explicitly combining identical and non-identical operation types inside one module (Kim et al., 13 Mar 2025).

The earlier “Temporal-Channel Joint Attention” line is especially relevant because it already framed temporal-channel interaction as preferable to separate temporal-only or channel-only processing. TCJA-SNN compresses the spike tensor to a channel-by-time matrix, applies Temporal-wise Local Attention and Channel-Wise Local Attention, and fuses them with a Cross Convolutional Fusion layer. Its own characterization is that existing methods handled temporal and channel information separately, limiting joint feature extraction (Zhu et al., 2022). Relative to that design, DTA’s novelty is the explicit decomposition into correlation and dependency views, together with a single module that combines identical and non-identical operations (Kim et al., 13 Mar 2025).

This comparison also clarifies a common misconception. A Temporal Retention Channel in these SNN papers is not a long-horizon explicit memory store comparable to a recurrent cell or an external memory table. It is a feature-level retention mechanism: temporal slices and temporal-channel combinations are reweighted so that useful temporal evidence persists in the spike representation. This is why the DTA paper states that temporal information is retained by generating attention weights over temporal slices and temporal-channel combinations, then applying those weights as residual and multiplicative gates to the original spike features (Kim et al., 13 Mar 2025).

5. Empirical behavior, efficiency, and ablation evidence

The reported experimental results support the claim that combined temporal-channel retention matters. On CIFAR10/100, DTA-SNN with MS-ResNet-18 achieves:

Dataset	4 time steps	6 time steps
CIFAR10	$T$0	$T$1
CIFAR100	$T$2	$T$3

These results improve over GAC-SNN, which reports $T$4 on CIFAR10 and $T$5 on CIFAR100 for 4/6 steps (Kim et al., 13 Mar 2025).

On ImageNet-1k with MS-ResNet-34, DTA-SNN reaches $T$6 at 4 steps and $T$7 at 6 steps, exceeding GAC-SNN’s $T$8. On CIFAR10-DVS, DTA-SNN reaches $T$9 at 10 time steps, surpassing TCJA-SNN’s $C$0 and transformer-based SNN baselines such as Spikingformer ($C$1) and Spike-driven Transformer ($C$2) (Kim et al., 13 Mar 2025).

The ablation study isolates the retention contribution of each branch. On CIFAR100 at $C$3:

Configuration	Accuracy
baseline (no T-XA, no T-NA)	$C$4
T-XA only	$C$5
T-NA only	$C$6
T-XA + T-NA	$C$7

On CIFAR10-DVS at $C$8:

Configuration	Accuracy
baseline	$C$9
T-XA only	$H,W$0
T-NA only	$H,W$1
T-XA + T-NA	$H,W$2

These results are consistent with the paper’s thesis: temporal-channel correlation helps, temporal-channel dependency helps slightly more, and the combined mechanism performs best (Kim et al., 13 Mar 2025).

The method is also described as lightweight. Parameter overhead remains modest: for CIFAR10/100 backbones, GAC-SNN uses $H,W$3M parameters versus DTA-SNN $H,W$4M; for ImageNet backbones, GAC-SNN uses $H,W$5M versus DTA-SNN $H,W$6M. Training uses direct training with surrogate gradients, iterative LIF neurons with $H,W$7 and $H,W$8, SGD with momentum $H,W$9, cosine annealing, and standard cross-entropy loss rather than TET loss (Kim et al., 13 Mar 2025).

6. Broader interpretations across adjacent research areas

Outside SNNs, the phrase “temporal retention” appears in several adjacent literatures, but with different meanings. In wireless channel prediction, temporal retention is modeled as space-time covariance persistence of an electromagnetic field under motion, rather than as a discrete autoregressive memory. The STEM-KL and GEM-KL schemes treat the channel as a continuous spatio-temporal Gaussian random field, where past observations influence future estimates through a velocity-aware, direction-sensitive, polarization-aware kernel and Bayesian posterior conditioning (Li et al., 2024). This suggests a distinct, physics-grounded interpretation of a retention channel as covariance persistence through time.

In continual learning, temporal retention can denote a finite-capacity memory process rather than a feature gate. One analytical formulation treats memory as a Bridge Diffusion over a replay interval $$O_{DTA} = \sigma(O_{T-XA} \odot O_{T-NA}) \odot Spikes,$$0, updated by a Compress–Add–Smooth recursion. In that framework, the principal empirical law is

$$O_{DTA} = \sigma(O_{T-XA} \odot O_{T-NA}) \odot Spikes,$$1

where retention half-life scales linearly with temporal budget $$O_{DTA} = \sigma(O_{T-XA} \odot O_{T-NA}) \odot Spikes,$$2, and forgetting arises from lossy temporal compression rather than parameter interference (Chertkov, 31 Mar 2026). A different mechanistic study defines feature-level retention dynamically as

$$O_{DTA} = \sigma(O_{T-XA} \odot O_{T-NA}) \odot Spikes,$$3

with retention depending jointly on representation strength, superposition, and exposure age (Wasilewski et al., 18 Jun 2026).

In LLM-agent systems, temporal retention has been cast as a cross-session memory pathway. One recent formulation decomposes memory into working, episodic, and semantic layers with adaptive retrieval gating and retention regularization, using the state

$$O_{DTA} = \sigma(O_{T-XA} \odot O_{T-NA}) \odot Spikes,$$4

as the carrier of information across sessions (Tiwari et al., 31 Mar 2026). Another proposal introduces a Transformer Retention Layer built around a persistent memory matrix

$$O_{DTA} = \sigma(O_{T-XA} \odot O_{T-NA}) \odot Spikes,$$5

but leaves write dynamics and empirical validation largely unspecified (Yaslioglu, 15 Jan 2025).

These related usages show that “Temporal Retention Channel” is not a universally standardized term. In the DTA-SNN literature it denotes a joint temporal-channel reweighting pathway; in wireless modeling it denotes kernelized temporal correlation; in continual learning it denotes finite-capacity replay or feature-strength dynamics; and in agent architectures it denotes persistent memory across sessions. The common thread is preservation of temporally relevant information under resource, noise, or structural constraints, but the mathematical object carrying that information differs across domains (Kim et al., 13 Mar 2025).