Timestep-Level Backpropagation (TLBP)

• Timestep-Level Backpropagation is a design pattern that computes gradients at individual timesteps or short windows instead of propagating through entire sequences.
• It is applied across spiking neural network training, adversarial attack frameworks, and efficient online recurrent learning to reduce computational overhead.
• TLBP methods range from approximate truncation schemes to exact forward sensitivity and parallel linear-system reformulations, each balancing trade-offs in precision and efficiency.

Timestep-Level Backpropagation (TLBP) denotes a class of gradient-computation schemes that operate at the granularity of individual timesteps or short temporal windows rather than treating an entire temporal trajectory as a single monolithic backpropagation graph. In the narrow sense used in recent spiking-neural-network (SNN) adversarial-attack work, TLBP is a per-window perturbation-generation mechanism with runtime monitoring and early stopping (Kang et al., 19 Aug 2025). In a broader research usage, the same timestep-granular principle appears in spatial-only or timestep-subsampled SNN training, temporally truncated local BPTT, online synthetic-gradient learning in recurrent neural networks (RNNs), exact forward-sensitivity recursions for timestep-varying parameters, parallel triangular-system formulations of BPTT, and per-timestep detached optimization for diffusion-model inference (Meng et al., 2023).

1. Conceptual scope and defining idea

The common premise of TLBP is that temporal credit assignment need not always be performed by full backpropagation through time over every timestep in the sequence. Standard BPTT or STBP computes gradients by unrolling all $T$ steps and accumulating temporal dependencies globally. TLBP reorganizes that computation: it may restrict gradient flow to the current timestep, to a short window, to a sampled subset of timesteps, or to a per-timestep subgraph whose cross-time states are detached.

This reorganization serves different goals in different literatures. In adversarial SNNs, it is used to reduce attack latency by extracting gradients from the current window only and terminating once cross-entropy or misclassification indicates success (Kang et al., 19 Aug 2025). In efficient SNN training, it appears as pruning temporal routes entirely, or backpropagating on only $K$ selected timesteps, on the basis that temporal-domain contributions affect the final gradient direction only weakly (Meng et al., 2023). In temporally truncated local BPTT, setting the truncation length $k=1$ yields a strict per-timestep update rule, while larger $k$ interpolates toward standard truncated BPTT (Guo et al., 2021). In online RNN learning, synthetic gradients and eligibility traces provide per-timestep updates without any BPTT at all (Pemberton et al., 2024). In exact sensitivity-based treatments, timestep granularity is preserved without approximation by treating parameters as timestep-specific objects and propagating sensitivities forward (Bird et al., 2021).

A central implication is that TLBP is not a single algorithm. It is a design pattern for temporal gradient computation. Some instantiations are approximations that deliberately discard future-time dependencies; some are exact reformulations; some are online surrogates; and some are parallel solvers for the same underlying temporal system. The shared feature is that the timestep, rather than the full sequence, is the primary unit of backward computation.

2. Mathematical structure in spiking temporal models

In SNNs, TLBP is usually defined against the baseline of discrete-time Leaky Integrate-and-Fire dynamics with surrogate gradients. One formulation models the forward pass over $T$ timesteps as

$$f_{1:T}(x)=\frac{1}{T}\sum_{t=1}^{T} h \circ l^{L} \circ l^{L-1} \circ \cdots \circ l^{1}(x),$$

with each block $l^\ell(\cdot)$ containing Conv-BN-LIF layers. For block $\ell$ at timestep $t$, the membrane potential and spike are

$$u_{\ell,t}=\tau\,u_{\ell,t-1}+W_\ell\, s_{\ell-1,t}+\eta_{\ell,t}, \qquad s_{\ell,t}=H(u_{\ell,t}-V_{th}),$$

and the non-differentiable spike function is handled by a surrogate gradient during backpropagation (Kang et al., 19 Aug 2025).

Efficient-training work decomposes the full BPTT gradient into spatial and temporal components and argues that the temporal products are strongly attenuated. With $K$0, the temporal-sensitivity products decay with temporal distance, and for the rectangle surrogate with $K$1 the relevant diagonal entries are either $K$2 or $K$3. Empirically, cosine similarity between full BPTT gradients and spatial-only gradients remains high across CIFAR-10, DVS-CIFAR10, and ImageNet, motivating SLTT and SLTT-K as timestep-level pruning schemes (Meng et al., 2023).

A different SNN result reaches the opposite formal conclusion about completeness rather than efficiency. “Temporal Surrogate Back-propagation” shows that common surrogate-gradient treatments omit a reset-dependent temporal term in $K$4. For the membrane update with hard reset, the complete temporal Jacobian contains

$K$5

in addition to the usual leak-gated term. That paper’s empirical conclusion is not that the term is useless, but that on larger tasks its gains are negligible relative to the added overhead: NMNIST reaches 99.29% with the reset term and 99.28% without it; MNIST reaches 99.49% with it and 99.50% without it; CIFAR-10 reaches 88.96% with it and 88.98% without it (Yang, 2020).

Taken together, these results clarify a recurring misconception. TLBP is not inherently equivalent to “more exact temporal differentiation.” In some lines of work, timestep-level structure is exploited to remove or ignore temporal routes because they contribute little; in others, timestep-level structure is exploited to restore missing temporal routes and make the derivative complete.

3. TLBP as a low-latency adversarial mechanism in SNNs

The most explicit modern use of the term appears in the timestep-compressed attack (TCA), where TLBP is the perturbation-generation component of a latency-oriented adversarial framework for SNNs (Kang et al., 19 Aug 2025). The motivating observation is that state-of-the-art gradient-based SNN attacks largely extend FGSM and PGD from ANNs and therefore inherit the need to backpropagate through all timesteps. Early timesteps often form a warm-up phase with low spike activity and weak gradients, yet standard attacks still pay the full temporal cost.

TLBP replaces full-sequence gradient extraction with windowed computation. For window index $K$6 and window size $K$7,

$K$8

The per-window loss is

$K$9

and the gradient is computed only within that window,

$k=1$0

An FGSM-style update inside the window is

$k=1$1

The attack then performs runtime monitoring by accumulated cross-entropy,

$k=1$2

and stops when $k=1$3, when misclassification occurs at the end of a window, or when the sequence ends. The cost therefore scales with the timesteps actually needed until success, $k=1$4, rather than the full $k=1$5.

TLBP in this setting is coupled to Adversarial Membrane Potential Reuse (A-MPR). A-MPR precomputes a class-specific “membrane image” $k=1$6 that initializes the network into informative membrane states, bypassing the low-spike cold start. Its objective combines membrane similarity and adversarial loss,

$k=1$7

and updates $k=1$8 offline by

$k=1$9

Before runtime attack, the model is warmed up using the precomputed $k$0 for the true label, and membrane potentials are clamped to $k$1.

The reported operating point is explicitly real-time oriented. Experimental settings use untargeted attacks with $k$2; PGD comparisons are restricted to 2 iterations with step size $k$3; $k$4; $k$5; and A-MPR uses $k$6, $k$7, $k$8, $k$9 (Kang et al., 19 Aug 2025).

On VGG-11 and ResNet-17 trained with STBP on CIFAR-10, CIFAR-100, and CIFAR10-DVS, TCA reduces attack latency by up to 56.6% in white-box settings and 57.1% in black-box settings while maintaining comparable attack success rate. Under a 33 ms real-time threshold, many PGD settings are infeasible because of iteration overhead, whereas TLBP-based attacks can reach higher ASR within feasible latency by stopping early. The paper also reports that per-window TLBP often attains higher final ASR than full-timestep FGSM, and that many vulnerable inputs already succeed at $T$0 with $T$1 (Kang et al., 19 Aug 2025).

The ablation results make the warm-up mechanism operationally concrete. On CIFAR-10 with VGG-11/STBP and $T$2, $T$3, A-MPR variants yield 84.7% ASR with only $T$4, 85.6% with the addition of $T$5, and 87.8% with a class-specific $T$6. Without clamping, VGG-11/STBP at $T$7 falls to 14.8% ASR; with clamp bounds $T$8 it reaches 89.6%, and with $T$9 it reaches 87.8% (Kang et al., 19 Aug 2025).

4. Training-time TLBP in spiking neural networks

Training-oriented TLBP in SNNs follows two main lines: spatial-only pruning of temporal routes, and temporally truncated local backpropagation. The first is represented by SLTT and SLTT-K, which are explicit timestep-level interpretations of efficient SNN training (Meng et al., 2023). SLTT removes all temporal routes and retains only spatial routes at each timestep. The modified gradient is

$$f_{1:T}(x)=\frac{1}{T}\sum_{t=1}^{T} h \circ l^{L} \circ l^{L-1} \circ \cdots \circ l^{1}(x),$$0

where the per-timestep error term is computed using only same-timestep layerwise propagation. SLTT-K further restricts backward computation to a subset $$f_{1:T}(x)=\frac{1}{T}\sum_{t=1}^{T} h \circ l^{L} \circ l^{L-1} \circ \cdots \circ l^{1}(x),$$1, with $$f_{1:T}(x)=\frac{1}{T}\sum_{t=1}^{T} h \circ l^{L} \circ l^{L-1} \circ \cdots \circ l^{1}(x),$$2, sampled uniformly at random without replacement per training iteration.

The efficiency gains are substantial. On ImageNet with NF-ResNet-34 and $$f_{1:T}(x)=\frac{1}{T}\sum_{t=1}^{T} h \circ l^{L} \circ l^{L-1} \circ \cdots \circ l^{1}(x),$$3, BPTT uses about 28.41G memory, 73.8h wall-clock training, and achieves about 66.47% accuracy; SLTT uses about 8.47G, 66.9h, and reaches about 66.19%; SLTT-1 further reduces time to about 32.03h with about 66.17% accuracy. On DVS-CIFAR10 with VGG-11 and $$f_{1:T}(x)=\frac{1}{T}\sum_{t=1}^{T} h \circ l^{L} \circ l^{L-1} \circ \cdots \circ l^{1}(x),$$4, BPTT uses about 3.70G and 4.47h for 73.60% accuracy, whereas SLTT uses about 1.07G and 3.43h for 77.30% accuracy, and SLTT-2 uses 2.49h for 76.70% accuracy. Compared with OTTT, SLTT also reduces per-epoch cost: on CIFAR-10, OTTT uses 1.71G and 184.68s whereas SLTT uses 1.00G and 54.48s; on ImageNet, OTTT uses 19.38G and 7.52h जबकि SLTT uses 8.47G and 2.23h (Meng et al., 2023).

The second line is temporally truncated local BPTT, which combines temporal windows with blockwise local supervision (Guo et al., 2021). Time is partitioned into intervals of length $$f_{1:T}(x)=\frac{1}{T}\sum_{t=1}^{T} h \circ l^{L} \circ l^{L-1} \circ \cdots \circ l^{1}(x),$$5, backpropagation is restricted to each interval, and local classifiers attached to blocks of $$f_{1:T}(x)=\frac{1}{T}\sum_{t=1}^{T} h \circ l^{L} \circ l^{L-1} \circ \cdots \circ l^{1}(x),$$6 layers provide local losses. Within a window $$f_{1:T}(x)=\frac{1}{T}\sum_{t=1}^{T} h \circ l^{L} \circ l^{L-1} \circ \cdots \circ l^{1}(x),$$7, the recurrent error recursions are truncated by the boundary condition $$f_{1:T}(x)=\frac{1}{T}\sum_{t=1}^{T} h \circ l^{L} \circ l^{L-1} \circ \cdots \circ l^{1}(x),$$8. In the terminology of that work, setting $$f_{1:T}(x)=\frac{1}{T}\sum_{t=1}^{T} h \circ l^{L} \circ l^{L-1} \circ \cdots \circ l^{1}(x),$$9 yields strict TLBP: gradients are backpropagated only within a single timestep, and updates occur at every timestep.

The empirical trade-off is dataset dependent. On frame-based datasets, accuracy generally decreases as $l^\ell(\cdot)$0 shrinks. On DVS datasets, moderate truncation often improves accuracy before very small $l^\ell(\cdot)$1 becomes unstable. The headline result is on AlexNet for CIFAR10-DVS, where the method reports a 7.26% increase in accuracy together with 89.94% reduction in GPU memory, 10.79% reduction in memory access, and 99.64% reduction in MAC operations compared to standard end-to-end BPTT (Guo et al., 2021).

These two SNN-training literatures show that timestep-level gradient restriction can act either as a regularizer or as a systems optimization. They also identify its operational limits. SLTT’s instantaneous backward is incompatible with batch normalization aggregated across the full horizon, leading that work to use weight standardization or normalization-free ResNets (Meng et al., 2023). TT-LBPTT reports that excessively small truncation harms frame-based tasks that rely more strongly on long temporal accumulation (Guo et al., 2021).

5. Online, exact, and parallel recurrent formulations

Outside SNNs, TLBP appears in three mathematically distinct forms: synthetic-gradient online learning, exact forward sensitivities, and parallel linear-system reformulations.

In BP$l^\ell(\cdot)$2, a synthesiser network $l^\ell(\cdot)$3 predicts the future gradient $l^\ell(\cdot)$4, allowing the main RNN to update online at each timestep without waiting for BPTT (Pemberton et al., 2024). The synthesiser is trained by TD-style errors and eligibility traces: $l^\ell(\cdot)$5 with

$l^\ell(\cdot)$6

and

$l^\ell(\cdot)$7

The main RNN update is performed per timestep using the instantaneous loss gradient and the synthetic future-gradient estimate. As $l^\ell(\cdot)$8, the paper proves

$l^\ell(\cdot)$9

Empirically, BP(1) solves longer dependencies than the no-BPTT baseline: on sequential MNIST it reaches 90.6% test accuracy, and on copy-repeat it solves a maximum sequence length of 29.0, outperforming the reported n-step SG baselines (Pemberton et al., 2024).

A different exact formulation treats parameters as timestep-varying objects and propagates forward sensitivities rather than approximating long-term influence (Bird et al., 2021). For hidden-state sensitivity $\ell$0,

$\ell$1

where $\ell$2. The exact gradient with respect to timestep parameter $\ell$3 is

$\ell$4

and, in the common case where $\ell$5 affects $\ell$6 but not $\ell$7,

$\ell$8

This line of work explicitly rejects the short-dependency assumptions of truncated BPTT and Nth-order approximations.

A third line reformulates forward and backward propagation as triangular linear systems (Naumov, 2017). For a single-layer RNN across $\ell$9 timesteps, the backward error satisfies an upper block bi-diagonal system,

$t$0

with boundary $t$1. Recursive-doubling or scan-based algorithms then compute all $t$2 in $t$3 parallel depth. For multi-layer RNNs with $t$4 layers and $t$5 timesteps, the paper states that backward propagation can be performed in parallel in $t$6 steps.

These three viewpoints are complementary rather than competing. BP$t$7 is fully online and avoids BPTT entirely, but eligibility traces can be expensive, with memory complexity $t$8 and computational complexity $t$9 for a linear synthesiser (Pemberton et al., 2024). Forward sensitivities preserve exactness for long-term dependencies but require Jacobian machinery (Bird et al., 2021). Triangular-system formulations preserve exactness while exposing parallel depth reductions (Naumov, 2017).

6. Generalizations, misconceptions, and limits

TLBP is not confined to recurrent training or SNN attacks. In diffusion inference acceleration, per-timestep mask learning for pre-trained diffusion probabilistic models has been described in explicitly TLBP-style terms because optimization is performed independently for each timestep while all cross-timestep states are detached (He et al., 20 Mar 2026). For timestep $$u_{\ell,t}=\tau\,u_{\ell,t-1}+W_\ell\, s_{\ell-1,t}+\eta_{\ell,t}, \qquad s_{\ell,t}=H(u_{\ell,t}-V_{th}),$$0, the masked model uses

$$u_{\ell,t}=\tau\,u_{\ell,t-1}+W_\ell\, s_{\ell-1,t}+\eta_{\ell,t}, \qquad s_{\ell,t}=H(u_{\ell,t}-V_{th}),$$1

and optimizes

$$u_{\ell,t}=\tau\,u_{\ell,t-1}+W_\ell\, s_{\ell-1,t}+\eta_{\ell,t}, \qquad s_{\ell,t}=H(u_{\ell,t}-V_{th}),$$2

Because the Detach operation restricts gradients to the current timestep, $$u_{\ell,t}=\tau\,u_{\ell,t-1}+W_\ell\, s_{\ell-1,t}+\eta_{\ell,t}, \qquad s_{\ell,t}=H(u_{\ell,t}-V_{th}),$$3 for all $$u_{\ell,t}=\tau\,u_{\ell,t-1}+W_\ell\, s_{\ell-1,t}+\eta_{\ell,t}, \qquad s_{\ell,t}=H(u_{\ell,t}-V_{th}),$$4. This yields training-time memory $$u_{\ell,t}=\tau\,u_{\ell,t-1}+W_\ell\, s_{\ell-1,t}+\eta_{\ell,t}, \qquad s_{\ell,t}=H(u_{\ell,t}-V_{th}),$$5 for the current subgraph instead of $$u_{\ell,t}=\tau\,u_{\ell,t-1}+W_\ell\, s_{\ell-1,t}+\eta_{\ell,t}, \qquad s_{\ell,t}=H(u_{\ell,t}-V_{th}),$$6 for global full-chain optimization. Reported speedups reach 2.75× on LDM-4-G ImageNet 256×256 with 250 DDIM steps while preserving competitive FID/IS/sFID (He et al., 20 Mar 2026).

Several misconceptions recur across the literature. One is that TLBP always means an approximation. That is false: forward sensitivity equations and triangular-system reformulations are exact, whereas SLTT, SLTT-K, adversarial windowing, and truncated local BPTT deliberately discard temporal routes or temporal extent (Bird et al., 2021). Another is that ignoring global temporal dependencies necessarily harms outcomes. That is also too strong: SLTT reports high cosine similarity between full and spatial-only gradients and often retains or improves accuracy; TLBP-based attacks can equal or exceed full-timestep FGSM while reducing latency (Meng et al., 2023).

The converse misconception is equally problematic. TLBP is not universally superior to full BPTT. The 2025 attack work explicitly notes that architectures or tasks relying on long-range temporal integration or specialized temporal coding may benefit more from full BPTT, because critical evidence may emerge only at late timesteps (Kang et al., 19 Aug 2025). TT-LBPTT finds that shrinking the truncation length hurts frame-based datasets, even though DVS datasets may benefit from moderate truncation (Guo et al., 2021). BP$$u_{\ell,t}=\tau\,u_{\ell,t-1}+W_\ell\, s_{\ell-1,t}+\eta_{\ell,t}, \qquad s_{\ell,t}=H(u_{\ell,t}-V_{th}),$$7 notes that tasks with highly stochastic or weakly correlated temporal structure can degrade synthesiser-target predictability (Pemberton et al., 2024). Diffusion mask learning is schedule-specific, so changing the sampler or number of function evaluations may require re-optimization (He et al., 20 Mar 2026).

The general picture is therefore conditional. TLBP is most effective when temporal redundancy, low-information warm-up, or smooth adjacent-step dynamics make full-chain backpropagation unnecessarily expensive. It is least compelling when the target task demands fine-grained long-horizon credit assignment that cannot be reconstructed from local windows, local surrogates, or per-timestep detached subgraphs.