DelRec: Trainable Delays in Recurrent SNNs

• DelRec is a method for learning trainable axonal or synaptic delays in recurrent spiking neural networks, enabling precise temporal pattern detection through delay-aligned coincidence detection.
• It uses a differentiable interpolation scheme with surrogate-gradient optimization to tune real-valued delays during training, then rounds them to integers for inference.
• By acting as temporal skip connections, DelRec enhances network dynamics, mitigates gradient issues, and achieves state-of-the-art results on challenging temporal benchmarks.

DelRec is a method for learning axonal or synaptic delays in recurrent spiking neural networks. Introduced in "DelRec: learning delays in recurrent spiking neural networks" (Queant et al., 29 Sep 2025), it is described as the first surrogate-gradient learning (SGL)-based method to train axonal or synaptic delays in recurrent spiking layers, while remaining compatible with any spiking neuron model under a standard discrete-time formalism. Its central idea is to treat transmission delays as trainable quantities alongside synaptic weights, using differentiable interpolation during training and integer delays at inference. In the reported experiments, DelRec established new state-of-the-art results on Spiking Speech Commands and Permuted Sequential MNIST, and matched the state of the art on Spiking Heidelberg Digit using only vanilla Leaky-Integrate-and-Fire neurons with stateless synapses (Queant et al., 29 Sep 2025).

1. Motivation and conceptual basis

Delays are intrinsic to biological spiking systems: in addition to synaptic weights, biological synapses and axons exhibit transmission delays that shape network dynamics and spike-time relationships. DelRec is motivated by the claim that trainable axonal or synaptic delays increase expressivity by enabling neurons to detect arbitrary temporal patterns via delay-aligned coincidence detection. With heterogeneous delays, sequences of spike latencies can be transformed into synchronous events that are linearly separable downstream (Queant et al., 29 Sep 2025).

The method is also motivated by recurrent-network dynamics. In the DelRec formulation, trainable delays enable richer dynamical regimes such as sustained oscillations and polychronization, and effectively expand the solution space of the neuron’s differential equations. The paper further argues that delays behave as temporal skip connections that can mitigate vanishing or exploding gradients by bridging distant time steps, and that they support efficient temporal feature alignment by making relevant information arrive when membrane states are poised to integrate it (Queant et al., 29 Sep 2025).

A major practical target of DelRec is the set of limitations associated with fixed or integer-only delays. Those limitations include the need to predefine a maximum delay range, coarse temporal resolution, and non-differentiability at integer steps. Prior approaches mainly addressed feedforward delays and either used non-differentiable integer selection with annealed softmax over a finite delay set, relied on EventProp with exact gradients but limited scalability and suboptimal performance on modern benchmarks, or enforced Gaussian kernels centered on learned integer positions. DelRec positions itself against that background by combining recurrent delay learning with SGL and backpropagation through time.

2. Formal model and delayed recurrent parameterization

DelRec is agnostic to the specific spiking neuron model provided that it fits a standard discrete-time charge–fire–reset formalism. The generic dynamics are

$H[t] = f(V[t-1], I[t]),$

$S[t] = \Theta(H[t] - V_{th}),$

and

$$V[t] = \begin{cases} H[t]\cdot(1-S[t]) + V_{reset}\cdot S[t], & \text{hard reset} \ H[t] - V_{th}\cdot S[t], & \text{soft reset}. \end{cases}$$

Here, $H[t]$ is the pre-spike membrane potential, $V[t]$ the post-spike membrane potential, $S[t]\in\{0,1\}$ the spike, $V_{th}$ the threshold, $V_{reset}$ the reset potential, and $f$ the neuron charge function. All experiments in the original paper use Leaky-Integrate-and-Fire neurons, for which

$$f(V[t-1], I[t]) = \left(1 - \frac{1}{\tau_m}\right) V[t-1] + \frac{1}{\tau_m} I[t].$$

The synapses are stateless: inputs are integrated directly as currents without additional synaptic dynamics, so the total current is

$S[t] = \Theta(H[t] - V_{th}),$0

In a vanilla recurrent SNN baseline, recurrent connections deliver spikes from time $S[t] = \Theta(H[t] - V_{th}),$1 to $S[t] = \Theta(H[t] - V_{th}),$2:

$S[t] = \Theta(H[t] - V_{th}),$3

DelRec generalizes that recurrence by learning a delay per axon, denoted $S[t] = \Theta(H[t] - V_{th}),$4, so that all outgoing recurrent synapses of neuron $S[t] = \Theta(H[t] - V_{th}),$5 share the same delay:

$S[t] = \Theta(H[t] - V_{th}),$6

If neuron $S[t] = \Theta(H[t] - V_{th}),$7 spikes at time $S[t] = \Theta(H[t] - V_{th}),$8, neuron $S[t] = \Theta(H[t] - V_{th}),$9 receives $$V[t] = \begin{cases} H[t]\cdot(1-S[t]) + V_{reset}\cdot S[t], & \text{hard reset} \ H[t] - V_{th}\cdot S[t], & \text{soft reset}. \end{cases}$$0 at $$V[t] = \begin{cases} H[t]\cdot(1-S[t]) + V_{reset}\cdot S[t], & \text{hard reset} \ H[t] - V_{th}\cdot S[t], & \text{soft reset}. \end{cases}$$1. The effective minimum recurrent delay is one time step, so $$V[t] = \begin{cases} H[t]\cdot(1-S[t]) + V_{reset}\cdot S[t], & \text{hard reset} \ H[t] - V_{th}\cdot S[t], & \text{soft reset}. \end{cases}$$2 corresponds to a one-step recurrent delay. Delays are real-valued during training, and after the temporal spread width is decayed to zero they are rounded to the nearest integers for inference (Queant et al., 29 Sep 2025).

The original formulation emphasizes axonal delays, but it is also stated to be compatible with per-synapse delays. Initialization depends on the task: SSC and PS-MNIST recurrent delays are initialized from a half-Gaussian centered at $$V[t] = \begin{cases} H[t]\cdot(1-S[t]) + V_{reset}\cdot S[t], & \text{hard reset} \ H[t] - V_{th}\cdot S[t], & \text{soft reset}. \end{cases}$$3 with $$V[t] = \begin{cases} H[t]\cdot(1-S[t]) + V_{reset}\cdot S[t], & \text{hard reset} \ H[t] - V_{th}\cdot S[t], & \text{soft reset}. \end{cases}$$4; SHD recurrent delays are initialized uniformly in $$V[t] = \begin{cases} H[t]\cdot(1-S[t]) + V_{reset}\cdot S[t], & \text{hard reset} \ H[t] - V_{th}\cdot S[t], & \text{soft reset}. \end{cases}$$5; and feedforward delays, when used, are initialized uniformly in $$V[t] = \begin{cases} H[t]\cdot(1-S[t]) + V_{reset}\cdot S[t], & \text{hard reset} \ H[t] - V_{th}\cdot S[t], & \text{soft reset}. \end{cases}$$6.

3. Differentiable interpolation, surrogate gradients, and optimization

The core technical contribution of DelRec is a differentiable interpolation scheme for non-integer delays. Rather than forcing delays to remain integer-valued during optimization, DelRec schedules future recurrent inputs with a triangular spread function centered at $$V[t] = \begin{cases} H[t]\cdot(1-S[t]) + V_{reset}\cdot S[t], & \text{hard reset} \ H[t] - V_{th}\cdot S[t], & \text{soft reset}. \end{cases}$$7 and controlled by an epoch-dependent spread parameter $$V[t] = \begin{cases} H[t]\cdot(1-S[t]) + V_{reset}\cdot S[t], & \text{hard reset} \ H[t] - V_{th}\cdot S[t], & \text{soft reset}. \end{cases}$$8:

$$V[t] = \begin{cases} H[t]\cdot(1-S[t]) + V_{reset}\cdot S[t], & \text{hard reset} \ H[t] - V_{th}\cdot S[t], & \text{soft reset}. \end{cases}$$9

This spreads each recurrent contribution over neighboring integer time offsets. As training progresses, $H[t]$0 decays to $H[t]$1, tightening the spread until it converges to linear interpolation between the two nearest integer delays. The support is finite:

$H[t]$2

The buffer range used for scheduling is approximated layer-wise by

$H[t]$3

which removes the need to preset a fixed maximum delay range (Queant et al., 29 Sep 2025).

At time $H[t]$4, if neuron $H[t]$5 spikes, DelRec updates future recurrent inputs according to

$H[t]$6

In the limit $H[t]$7, the delayed signal reduces to linear interpolation. Let $H[t]$8, $H[t]$9, and $V[t]$0. Then

$V[t]$1

The derivative through the delay path is then

$V[t]$2

valid away from integer crossings of $V[t]$3.

Training uses SGL with backpropagation through time. The Heaviside spike derivative is replaced in the backward pass by a surrogate derivative. For SSC and PS-MNIST, DelRec uses a triangular surrogate,

$V[t]$4

and for SHD an arctan surrogate,

$V[t]$5

Because the triangular delay kernel is piecewise linear and the membrane equations are linear in $V[t]$6, the paper argues that gradients through the delay path are stable. The derivative of the triangular spread inside its support is

$V[t]$7

and is zero outside the support. This gives the delay-gradient expression

$V[t]$8

The paper also introduces, for SSC only, an optional per-neuron adaptive spread parameter $V[t]$9:

$S[t]\in\{0,1\}$0

with $S[t]\in\{0,1\}$1 initialized to $S[t]\in\{0,1\}$2 and learned jointly (Queant et al., 29 Sep 2025).

4. Architectures, implementation, and computational profile

The experimental study uses dataset-specific recurrent SNN architectures built from vanilla LIF neurons. For SSC, the network has three fully connected hidden layers of 256 neurons each, no batch normalization, dropout of $S[t]\in\{0,1\}$3 on feedforward connections and $S[t]\in\{0,1\}$4 on recurrent connections, axonal recurrent delays in hidden layers, and a linear readout to 35 classes. For PS-MNIST, the architecture has one hidden layer of 64 neurons followed by two hidden layers of 212 neurons, again with axonal recurrent delays and a linear readout to 10 classes. For SHD, large models use two hidden layers of 256 neurons each, recurrent delays only in the second layer, batch normalization, dropout, and augmented training; small ablation models preserve the two-layer topology with reduced neuron counts and approximately $S[t]\in\{0,1\}$5 parameters (Queant et al., 29 Sep 2025).

Temporal encoding and preprocessing follow event-stream binning. SSC and SHD both use spatial binning from 700 to 140 channels. The time step is $S[t]\in\{0,1\}$6 for SSC and $S[t]\in\{0,1\}$7 for SHD. Training configurations differ by dataset: SSC uses 100 epochs, batch size 128, Adam, one-cycle schedulers, $S[t]\in\{0,1\}$8, threshold 1, soft reset, the triangular surrogate, and an exponentially decaying $S[t]\in\{0,1\}$9 initialized at 10; PS-MNIST uses 200 epochs, batch size 256, AdamW, $V_{th}$0, soft reset, the triangular surrogate, and similar learning-rate schedules; SHD uses 150 epochs, batch size 256, AdamW, $V_{th}$1, threshold 1, hard reset, the arctan surrogate, batch normalization, and augmentations for large models (Queant et al., 29 Sep 2025).

Implementation is based on a circular scheduling buffer. The forward pass maintains a buffer $V_{th}$2 of size $V_{th}$3, where $V_{th}$4. At each time step, the model reads the current recurrent input from the buffer, computes the membrane update and spike emission, zeros the consumed slot, advances the buffer pointer modulo $V_{th}$5, and then schedules future recurrent contributions of emitted spikes across $V_{th}$6. The backward pass uses autograd with the surrogate derivative to compute gradients through the buffer, the spread kernel, the neuron charge function, the membrane state, and the spike outputs.

The computational profile depends on the learned delay distribution. Memory is dominated by the buffer and scales as $V_{th}$7, where

$V_{th}$8

For a dense recurrent layer, per-time-step compute is $V_{th}$9 in the worst case because scheduling touches all postsynaptic neurons and all offsets in $V_{reset}$0. The paper notes, however, that practical cost scales with the number of emitted spikes and fan-out, and that $V_{reset}$1 shrinks as $V_{reset}$2 decays. The method is reported to integrate seamlessly into PyTorch and SpikingJelly autograd, and no explicit gradient clipping was required in the reported experiments (Queant et al., 29 Sep 2025).

5. Benchmarks, ablations, and empirical interpretation

The empirical evaluation covers three temporal benchmarks. Spiking Speech Commands is described as a large audio spiking dataset with approximately 100k samples and 35 classes. Permuted Sequential MNIST is a long-range temporal integration task in which the input is a permuted pixel sequence. Spiking Heidelberg Digits contains approximately 10k recordings of spoken digits in English and German and is described as often saturated (Queant et al., 29 Sep 2025).

The principal test accuracies, with parameter counts described as comparable, are as follows:

Dataset	DelRec result	Comparison
SSC	82.58 ± 0.08% with only recurrent delays; 82.19 ± 0.16% with feedforward + recurrent delays	ASRC-SNN 81.54%; DCLS 80.69 ± 0.21%; SE-adLIF 80.44 ± 0.26%
PS-MNIST	96.21% with only recurrent delays	ASRC-SNN 95.77%; other LIF-family baselines 90–95%
SHD	93.73 ± 0.69% with feedforward + recurrent delays; 93.39 ± 0.45% with only recurrent delays	DCLS 93.77 ± 0.68%; SE-adLIF and BRF approximately 92.7–93.8%

These results support the paper’s claim that trainable recurrent delays outperform feedforward ones on the two more challenging temporal tasks, while matching the state of the art on SHD using only vanilla LIF neurons with instantaneous synapses (Queant et al., 29 Sep 2025).

The ablation study on SHD small models, with approximately $V_{reset}$3 parameters and no augmentations, isolates the role of recurrent delays. Models with learned recurrent delays consistently outperform vanilla SNNs and vanilla RSNNs with a uniform one-step delay. Random fixed recurrent delays already improve trainability relative to vanilla RSNNs, which the authors interpret as evidence that recurrent delays mitigate gradient issues. Under tight parameter budgets and firing-rate penalties, recurrent delays degrade less steeply than alternatives, suggesting better reuse or alignment of temporal information. The same ablations also indicate that feedforward delays can be more energy-efficient at their optimal accuracies under spike penalties, whereas recurrent delays achieve higher accuracy for a given parameter budget without strong sparsity constraints.

The paper offers three mechanistic explanations for recurrent-delay gains. First, recurrent delays implement temporal skip connections internal to the recurrent layer, improving credit assignment across distant time steps and reducing vanishing or exploding gradients. Second, they enable self-sustained patterns and resonance-like dynamics that exploit long-range dependencies. Third, they optimize temporal alignment so that spikes coincide at decision-relevant moments and leverage coincidence detection in LIF neurons. This suggests that the main contribution of DelRec is not merely additional tunable parameters, but a change in the temporal geometry of recurrent computation (Queant et al., 29 Sep 2025).

6. Position in the literature, neuromorphic implications, and later developments

DelRec is positioned against three main research lines: feedforward delay learning, recurrent delay learning with coarser parameterizations, and neuron-centric improvements that enrich intrinsic dynamics rather than connection timing. The paper explicitly contrasts DelRec with feedforward-delay methods such as SLAYER variants, DCLS, and related Gaussian-kernel or candidate-set approaches; with ASRC-SNN, which learns one recurrent delay per layer via softmax over a fixed set; and with EventProp-based recurrent delay optimization. It also distinguishes delay learning from approaches based on adaptive, resonant, or multi-compartment neuron models, which can produce strong performance but do not directly optimize recurrent transmission delays (Queant et al., 29 Sep 2025).

A major claimed advantage is hardware compatibility. DelRec rounds its trained real-valued delays to the nearest integers for inference, so the learned delays are directly deployable on neuromorphic systems with programmable delays, including Loihi, SpiNNaker, and custom ASICs. The paper further argues that properly aligned recurrent delays reduce unnecessary spiking and improve timing precision, lowering energy per correct classification under sparse activity. Axonal delays require one delay register per axon; synaptic delays require per-synapse delay storage but improve flexibility. DelRec is stated to support both forms (Queant et al., 29 Sep 2025).

The method was subsequently extended in "Combining Convolution and Delay Learning in Recurrent Spiking Neural Networks" (Zebendo et al., 17 Apr 2026). That work retains the DelRec delay-learning mechanism but replaces dense recurrent matrices with 1D convolutional recurrent connections. For a layer with $V_{reset}$4 neurons and kernel size $V_{reset}$5, the recurrent parameter count changes from $V_{reset}$6 to $V_{reset}$7, a 99.6% reduction in total recurrent parameters per layer, while recurrent weight parameters alone drop from 65,536 to 3. In that study’s SHD setup, the convolutional variant achieved 91.51% ± 0.70% test accuracy versus 91.72% ± 0.84% for the dense DelRec baseline, with inference reported as 52x faster (Zebendo et al., 17 Apr 2026).

The same extension also showed that learnable delays remain important under the convolutional parameterization. On SHD, fixed delays gave 86.28% ± 4.82 for $V_{reset}$8 and 88.19% ± 0.65 for $V_{reset}$9, whereas learnable delays reached 91.51% ± 0.70. On SSC, fixed delays gave 75.09% ± 2.82 for $f$0 and 77.65% ± 0.37 for $f$1, compared with 78.59% ± 0.39 for learnable delays (Zebendo et al., 17 Apr 2026). A plausible implication is that DelRec’s central idea—the optimization of recurrent transmission timing—can be decoupled from dense all-to-all recurrence and preserved in more structured recurrent operators.

Taken together, DelRec defines a recurrent SNN training framework in which delays are treated as first-class, differentiable parameters. Its reported contributions are a non-integer delay parameterization with well-defined gradients, dynamic scheduling without a preset maximum delay range, competitive or state-of-the-art temporal benchmark performance with simple LIF neurons, and direct compatibility with programmable-delay neuromorphic hardware (Queant et al., 29 Sep 2025).