Surrogate Gradient Descent in Neural Networks

• Surrogate GD is a family of optimization methods that replaces non-differentiable derivatives with smooth surrogate functions to enable effective gradient-based training.
• It is applied in diverse scenarios such as spiking neural networks, simulator calibration, and online hyperparameter optimization to improve stability and efficiency.
• Recent advancements like parametric surrogate gradients illustrate that tuning the surrogate function can materially enhance training performance and the induced kernel dynamics.

Surrogate Gradient Descent (Surrogate GD) denotes a family of optimization procedures in which descent is driven by a substitute gradient object rather than the exact derivative that would ordinarily be used. In contemporary machine learning, the term most commonly refers to training spiking or other discontinuous neural networks by keeping the exact non-differentiable forward dynamics while replacing the backward derivative with a smooth surrogate, thereby enabling backpropagation through time or related gradient-based updates (Luo et al., 2023, Eilers et al., 2024). The label is also used more broadly for methods that optimize through learned simulators, surrogate losses, target-space surrogates, or surrogate gradient directions when true gradients are unavailable, expensive, biased, or ill-conditioned (Atad et al., 2024, Zhuang et al., 2019, Léger et al., 2023).

1. Scope and terminology

In the spiking-neural-network literature, surrogate gradient learning is the standard setting in which the activation function’s derivative is substituted by a surrogate derivative during backpropagation, while the forward pass still uses the original discontinuous activation (Eilers et al., 2024). This usage is motivated by the fact that thresholding operations such as Heaviside spikes have zero derivative almost everywhere and are undefined at threshold, so literal gradient descent is either unusable or theoretically ill-posed.

The same phrase is not fully uniform across neighboring literatures. In finite-element calibration, for example, a neural network surrogate is trained to emulate an expensive simulator, and projected gradient descent is then applied to the surrogate rather than to the original model (Atad et al., 2024). In stochastic optimization, surrogate losses can be introduced over the learning rate itself, turning stepsize selection into an online convex optimization problem (Zhuang et al., 2019). In black-box optimization, surrogate gradient directions may be treated as correlated but biased signals that guide random search rather than replace it outright (Maheswaranathan et al., 2018, Meier et al., 2019).

A recurrent misconception is that these usages are interchangeable. They are not. In spiking networks, the surrogate usually replaces a non-differentiable local derivative inside a computational graph. In simulator calibration, the surrogate is a learned differentiable model of the objective. In online optimization, the surrogate may be a loss over a hyperparameter rather than over the model parameters. The commonality is structural: descent is organized around a proxy object that makes optimization tractable, but the mathematical role of the proxy differs substantially across settings (Atad et al., 2024, Zhuang et al., 2019, Léger et al., 2023).

2. Canonical formulation in spiking neural networks

A canonical Surrogate GD formulation appears in directly trained spiking neural networks built from integrate-and-fire dynamics. In one representative object-detection model, the membrane potential of neuron $j$ in layer $l$ evolves as

$$V_{mem,j}^l[t] = V_{mem,j}^l[t-1] + I_j^l[t] - V_{th}s_j^l[t],$$

with synaptic current

$$I_{j}^{l}[t] = \sum_i w_{i,j}^{l}s_i^{l-1}[t] + b_j^l,$$

and spike emission

$$s_j^l[t] = H\!\left(V_{mem,j}^{l}[t] - V_{th}\right).$$

The optimization obstruction is the hard threshold $H(\cdot)$, whose exact derivative is unusable for SGD or BPTT (Luo et al., 2023).

Surrogate GD resolves this by leaving the forward threshold unchanged and replacing only the backward derivative. In the same detector, the chosen surrogate is arctangent: $$g(x) = \frac{1}{\pi}\arctan\left(\frac{\pi}{2}\alpha x\right) + \frac{1}{2}, \qquad g'(x) = \frac{\alpha}{2\left(1+\left(\frac{\pi}{2}\alpha x\right)^2\right)}.$$ During backpropagation,

$$\frac{\partial s_j^l[t]}{\partial V_{mem,j}^l[t]} \approx g'\!\left(V_{mem,j}^{l}[t]-V_{th}\right).$$

The network is unrolled over time, the task loss is accumulated over $T$ steps, and BPTT is applied with surrogate derivatives inserted whenever threshold sensitivities are required (Luo et al., 2023).

Alternative surrogate forms are also used. For timing-based speech and synthetic spike tasks, the backward pass replaces the Heaviside derivative by

$$\frac{\partial H(x)}{\partial x} \approx \frac{1}{(\alpha |x| + 1)^2},\qquad \alpha=100,$$

showing that Surrogate GD is not tied to a single analytic family (Yu et al., 21 Jul 2025). In Loihi-compatible online learning, the hardware-level derivation starts from a rule of the form

$l$0

where $l$1 is the surrogate derivative of a sigmoid used in place of the derivative of the threshold function (Stewart et al., 2019).

The essential procedural invariant is therefore precise. Forward propagation uses the original spiking or discontinuous nonlinearity; backward propagation injects a smooth local gradient; temporal recurrence is handled by BPTT or an architecture-specific local approximation; and parameter updates remain gradient-based (Luo et al., 2023, Stewart et al., 2019).

3. Surrogate design as an optimization variable

A major development in the SNN literature is the shift from hand-chosen surrogate functions toward trainable surrogate parameterizations. “Membrane Potential Distribution Adjustment and Parametric Surrogate Gradient in Spiking Neural Networks” introduces Parametric Surrogate Gradient (PSG), where the surrogate shape parameter $l$2 is optimized jointly with the network weights during BPTT rather than fixed a priori (Wang et al., 2023).

For convolutional layers, PSG uses an arctangent auxiliary function

$l$3

with normalized derivatives

$l$4

For fully connected layers, where outputs depend on accumulated membrane potentials, PSG uses

$l$5

with normalized derivatives

$l$6

The weight and surrogate-parameter gradients are then computed jointly through BPTT (Wang et al., 2023).

This work also introduces Potential Distribution Adjustment (PDA), motivated by the claim that binary spike communication induces quantization error and shifts membrane-potential distributions into poor gradient regions. PDA penalizes the mismatch between the empirical membrane-potential distribution $l$7 and a Gaussian target $l$8 via a symmetric KL term: $l$9 The total training objective becomes

$$V_{mem,j}^l[t] = V_{mem,j}^l[t-1] + I_j^l[t] - V_{th}s_j^l[t],$$0

with $$V_{mem,j}^l[t] = V_{mem,j}^l[t-1] + I_j^l[t] - V_{th}s_j^l[t],$$1 in the reported experiments (Wang et al., 2023).

Empirically, the same paper argues that fixed surrogate choices are not innocuous. Different initial $$V_{mem,j}^l[t] = V_{mem,j}^l[t-1] + I_j^l[t] - V_{th}s_j^l[t],$$2 values lead to materially different final accuracies in plain SNNs, whereas PSG reduces sensitivity to initialization and learns layer-dependent values: shallow layers converge to smaller $$V_{mem,j}^l[t] = V_{mem,j}^l[t-1] + I_j^l[t] - V_{th}s_j^l[t],$$3, while the final layer converges to a much larger $$V_{mem,j}^l[t] = V_{mem,j}^l[t-1] + I_j^l[t] - V_{th}s_j^l[t],$$4 (Wang et al., 2023). This directly contradicts the common assumption that surrogate shape is merely a minor implementation detail.

4. Theoretical foundations: the surrogate-gradient NTK

The most explicit theory of Surrogate GD in discontinuous networks is the surrogate-gradient neural tangent kernel (SG-NTK). “A generalized neural tangent kernel for surrogate gradient learning” shows that a naive extension of ordinary NTK theory to jump activations is ill-posed: for sign activations, the infinite-width kernel has diverging diagonal, so ordinary gradient descent does not yield a well-behaved regression kernel in the infinite-width limit (Eilers et al., 2024).

The SG-NTK formalizes surrogate gradient learning by distinguishing the forward activation $$V_{mem,j}^l[t] = V_{mem,j}^l[t-1] + I_j^l[t] - V_{th}s_j^l[t],$$5, its true derivative $$V_{mem,j}^l[t] = V_{mem,j}^l[t-1] + I_j^l[t] - V_{th}s_j^l[t],$$6, and a surrogate derivative $$V_{mem,j}^l[t] = V_{mem,j}^l[t-1] + I_j^l[t] - V_{th}s_j^l[t],$$7. The key object is an asymmetric kernel built from a true Jacobian on one side and a quasi-Jacobian on the other: $$V_{mem,j}^l[t] = V_{mem,j}^l[t-1] + I_j^l[t] - V_{th}s_j^l[t],$$8 Under MSE loss, wide-network SGL obeys linearized function-space dynamics of the form

$$V_{mem,j}^l[t] = V_{mem,j}^l[t-1] + I_j^l[t] - V_{th}s_j^l[t],$$9

and the limiting predictor becomes kernel regression with the SG-NTK (Eilers et al., 2024).

The theoretical point is not that Surrogate GD recovers the “true” gradient of a discontinuous model. Rather, it induces its own coherent kernel dynamics. The ordinary NTK pathology arises from pairing two singular derivative factors; SGL replaces one of them with a bounded surrogate factor, which regularizes the limit. The paper proves kernel convergence at initialization for discontinuous activations under suitable assumptions, proves during-training kernel constancy for smooth approximants, and numerically shows that sign networks trained by SGL are well characterized by kernel regression with the limiting SG-NTK (Eilers et al., 2024).

A plausible implication is that surrogate choice affects not only optimization stability but also the induced function-space bias. The paper does not develop a full surrogate-selection theory, but its formulas make the dependence explicit through Gaussian expectations involving $$I_{j}^{l}[t] = \sum_i w_{i,j}^{l}s_i^{l-1}[t] + b_j^l,$$0 (Eilers et al., 2024).

5. Empirical domains and system-level results

Surrogate GD has been demonstrated in several technically distinct deployment regimes. In event-driven object detection, SNN-YOLOv3 combines direct surrogate-gradient training with Current Mean Decoding (CMD), a readout designed for regression rather than rate counting. On PASCAL VOC, the reported result is $$I_{j}^{l}[t] = \sum_i w_{i,j}^{l}s_i^{l-1}[t] + b_j^l,$$1 with only $$I_{j}^{l}[t] = \sum_i w_{i,j}^{l}s_i^{l-1}[t] + b_j^l,$$2 time steps. The comparison against Spiking-YOLO reports $$I_{j}^{l}[t] = \sum_i w_{i,j}^{l}s_i^{l-1}[t] + b_j^l,$$3 mAP at $$I_{j}^{l}[t] = \sum_i w_{i,j}^{l}s_i^{l-1}[t] + b_j^l,$$4 time steps, and the paper concludes that SNN-YOLOv3 is more than $$I_{j}^{l}[t] = \sum_i w_{i,j}^{l}s_i^{l-1}[t] + b_j^l,$$5 more energy efficient than YOLOv3 under the reported operation-based estimate (Luo et al., 2023). The technical lesson is that surrogate derivatives alone are insufficient for detection unless the output decoding is also compatible with regression.

In neuromorphic hardware, on-chip few-shot learning has been demonstrated on Intel’s Loihi processor by deriving a local plasticity rule from a surrogate-gradient update and then adapting it to Loihi’s constrained trace-based learning primitives. On IBM DvsGesture, the Loihi plasticity rule reaches $$I_{j}^{l}[t] = \sum_i w_{i,j}^{l}s_i^{l-1}[t] + b_j^l,$$6 test accuracy in the 11-way 1-shot setting and $$I_{j}^{l}[t] = \sum_i w_{i,j}^{l}s_i^{l-1}[t] + b_j^l,$$7 at 14 shots. In a 6+5-way transfer protocol, where five classes are unseen during pretraining, it reaches $$I_{j}^{l}[t] = \sum_i w_{i,j}^{l}s_i^{l-1}[t] + b_j^l,$$8 at 20 shots (Stewart et al., 2019). This shows that Surrogate GD can be transformed into hardware-compatible local learning rules rather than used only in offline simulation.

A different empirical question concerns coding regime. “Beyond Rate Coding: Surrogate Gradients Enable Spike Timing Learning in Spiking Neural Networks” reports that Surrogate GD-trained SNNs solve timing-only tasks in which spike-count information is removed. On SHD-norm, a plain Surrogate GD model reaches $$I_{j}^{l}[t] = \sum_i w_{i,j}^{l}s_i^{l-1}[t] + b_j^l,$$9 and a delay-learning version reaches $$s_j^l[t] = H\!\left(V_{mem,j}^{l}[t] - V_{th}\right).$$0, while the count-based MLP baseline is at chance, $$s_j^l[t] = H\!\left(V_{mem,j}^{l}[t] - V_{th}\right).$$1. On SSC-norm, the corresponding numbers are $$s_j^l[t] = H\!\left(V_{mem,j}^{l}[t] - V_{th}\right).$$2, $$s_j^l[t] = H\!\left(V_{mem,j}^{l}[t] - V_{th}\right).$$3, and $$s_j^l[t] = H\!\left(V_{mem,j}^{l}[t] - V_{th}\right).$$4 (Yu et al., 21 Jul 2025). The same study reports about $$s_j^l[t] = H\!\left(V_{mem,j}^{l}[t] - V_{th}\right).$$5 accuracy on a synthetic ISI task under clean conditions and a sharp performance drop under time reversal, especially when delays are learned. These results rebut the claim that Surrogate GD merely recovers firing-rate solutions.

For directly trained deep SNNs with learned surrogate shapes, PSG and PDA produce strong benchmark results at low timestep counts: $$s_j^l[t] = H\!\left(V_{mem,j}^{l}[t] - V_{th}\right).$$6 on MNIST at $$s_j^l[t] = H\!\left(V_{mem,j}^{l}[t] - V_{th}\right).$$7, $$s_j^l[t] = H\!\left(V_{mem,j}^{l}[t] - V_{th}\right).$$8 on CIFAR10 at $$s_j^l[t] = H\!\left(V_{mem,j}^{l}[t] - V_{th}\right).$$9, $H(\cdot)$0 on CIFAR100 at $H(\cdot)$1, $H(\cdot)$2 on NMNIST at $H(\cdot)$3, and $H(\cdot)$4 on DVS-CIFAR10 at $H(\cdot)$5 (Wang et al., 2023). The same paper reports that PSG adds only $H(\cdot)$6 training time per epoch over a standard SNN on CIFAR10, while PSG&PDA adds $H(\cdot)$7, and both add only $H(\cdot)$8 learnable parameters (Wang et al., 2023).

6. Broader surrogate-descent formulations beyond SNNs

Outside SNNs, the phrase Surrogate GD overlaps with several distinct optimization programs.

In simulator-based inverse problems, a neural surrogate can replace the original physical model inside a constrained descent loop. In finite-element calibration for an L4–L5 intervertebral disc model with $H(\cdot)$9 material parameters, a five-layer MLP surrogate is trained on Latin hypercube sampled simulations, then frozen, and Projected Gradient Descent with NN is applied to the inputs. The reported calibration time is under three seconds, versus up to 8 days for direct FE-based search in the worst case, and on synthetic data the method achieves $$g(x) = \frac{1}{\pi}\arctan\left(\frac{\pi}{2}\alpha x\right) + \frac{1}{2}, \qquad g'(x) = \frac{\alpha}{2\left(1+\left(\frac{\pi}{2}\alpha x\right)^2\right)}.$$0 versus $$g(x) = \frac{1}{\pi}\arctan\left(\frac{\pi}{2}\alpha x\right) + \frac{1}{2}, \qquad g'(x) = \frac{\alpha}{2\left(1+\left(\frac{\pi}{2}\alpha x\right)^2\right)}.$$1 for GA w/ NN and $$g(x) = \frac{1}{\pi}\arctan\left(\frac{\pi}{2}\alpha x\right) + \frac{1}{2}, \qquad g'(x) = \frac{\alpha}{2\left(1+\left(\frac{\pi}{2}\alpha x\right)^2\right)}.$$2 for the inverse model (Atad et al., 2024). Here the “surrogate gradient” is the gradient of a learned emulator rather than a backward replacement for a discontinuity.

In stochastic optimization, surrogate losses can be built over hyperparameters rather than model parameters. “Surrogate Losses for Online Learning of Stepsizes in Stochastic Non-Convex Optimization” defines

$$g(x) = \frac{1}{\pi}\arctan\left(\frac{\pi}{2}\alpha x\right) + \frac{1}{2}, \qquad g'(x) = \frac{\alpha}{2\left(1+\left(\frac{\pi}{2}\alpha x\right)^2\right)}.$$3

a convex surrogate over the stepsize $$g(x) = \frac{1}{\pi}\arctan\left(\frac{\pi}{2}\alpha x\right) + \frac{1}{2}, \qquad g'(x) = \frac{\alpha}{2\left(1+\left(\frac{\pi}{2}\alpha x\right)^2\right)}.$$4, and uses online convex optimization to adapt SGD learning rates with regret and non-convex convergence guarantees (Zhuang et al., 2019). “Target-based Surrogates for Stochastic Optimization” instead constructs a surrogate in output or target space,

$$g(x) = \frac{1}{\pi}\arctan\left(\frac{\pi}{2}\alpha x\right) + \frac{1}{2}, \qquad g'(x) = \frac{\alpha}{2\left(1+\left(\frac{\pi}{2}\alpha x\right)^2\right)}.$$5

so that one expensive target-gradient evaluation can be amortized across multiple parameter updates; in the stochastic case this leads to the SSO algorithm, interpretable as projected SGD in target space (Lavington et al., 2023). At a more abstract level, “Gradient descent with a general cost” constructs canonical surrogates $$g(x) = \frac{1}{\pi}\arctan\left(\frac{\pi}{2}\alpha x\right) + \frac{1}{2}, \qquad g'(x) = \frac{\alpha}{2\left(1+\left(\frac{\pi}{2}\alpha x\right)^2\right)}.$$6 via a $$g(x) = \frac{1}{\pi}\arctan\left(\frac{\pi}{2}\alpha x\right) + \frac{1}{2}, \qquad g'(x) = \frac{\alpha}{2\left(1+\left(\frac{\pi}{2}\alpha x\right)^2\right)}.$$7-transform and shows that gradient descent, mirror descent, natural gradient descent, and Newton’s method can all be recovered as instances of alternating minimization of cost-induced surrogates (Léger et al., 2023).

A third cluster uses surrogate directions inside black-box search. Guided Evolutionary Strategies builds a search covariance elongated along a subspace spanned by surrogate gradients and derives bias-variance tradeoffs for the resulting estimator (Maheswaranathan et al., 2018). “Improving Gradient Estimation in Evolutionary Strategies With Past Descent Directions” shows that past descent directions can themselves serve as surrogate directions and proves rapid convergence to the true gradient for linear functions (Meier et al., 2019). “Stein Variational Gradient Descent Without Gradient” replaces the unavailable score $$g(x) = \frac{1}{\pi}\arctan\left(\frac{\pi}{2}\alpha x\right) + \frac{1}{2}, \qquad g'(x) = \frac{\alpha}{2\left(1+\left(\frac{\pi}{2}\alpha x\right)^2\right)}.$$8 with a surrogate score $$g(x) = \frac{1}{\pi}\arctan\left(\frac{\pi}{2}\alpha x\right) + \frac{1}{2}, \qquad g'(x) = \frac{\alpha}{2\left(1+\left(\frac{\pi}{2}\alpha x\right)^2\right)}.$$9 and corrects the bias with the density ratio $$\frac{\partial s_j^l[t]}{\partial V_{mem,j}^l[t]} \approx g'\!\left(V_{mem,j}^{l}[t]-V_{th}\right).$$0 (Han et al., 2018). “A New Adaptive Gradient Method with Gradient Decomposition” defines a surrogate gradient through the transformed objective $$\frac{\partial s_j^l[t]}{\partial V_{mem,j}^l[t]} \approx g'\!\left(V_{mem,j}^{l}[t]-V_{th}\right).$$1, so that $$\frac{\partial s_j^l[t]}{\partial V_{mem,j}^l[t]} \approx g'\!\left(V_{mem,j}^{l}[t]-V_{th}\right).$$2, and then uses the complementary loss-based factor for adaptive scaling (Shao et al., 2021).

These formulations share a family resemblance but not a single formalism. The proxy may be a backward derivative, a learned emulator, a surrogate loss, a target-space upper model, a kernelized score, or a correlated search direction.

7. Limitations, misconceptions, and open questions

Surrogate GD is not exact gradient descent on the original discontinuous system. In wide-network theory, ordinary gradient descent for jump activations can be ill-posed, while Surrogate GD induces a distinct SG-NTK dynamics (Eilers et al., 2024). Treating the surrogate as a harmless implementation trick therefore obscures its algorithmic and statistical role.

Surrogate selection remains consequential. PSG was introduced precisely because most surrogate gradients were being chosen heuristically, and the reported ablations show that fixed $$\frac{\partial s_j^l[t]}{\partial V_{mem,j}^l[t]} \approx g'\!\left(V_{mem,j}^{l}[t]-V_{th}\right).$$3 choices can substantially alter final accuracy (Wang et al., 2023). Output mismatch can also dominate training: in spiking object detection, rate decoding performs poorly even with surrogate gradients, and CMD is introduced because object detection requires continuous-valued regression rather than only classification (Luo et al., 2023).

The empirical successes are also bounded. Timing-based SNNs trained with Surrogate GD degrade under perturbations such as per-neuron jitter, spike deletion, and time reversal, although the degradation is task- and perturbation-specific (Yu et al., 21 Jul 2025). The SG-NTK theory is strongest for wide multilayer perceptrons, MSE loss, and smooth approximants to discontinuous activations rather than for exact finite-width spiking systems (Eilers et al., 2024). Simulator-based surrogate descent depends on the local fidelity of the learned surrogate; the FE calibration study reports strong performance but does not directly validate gradient fidelity against FE sensitivities, and it still requires expensive upfront data generation (Atad et al., 2024).

A final misconception is that Surrogate GD is synonymous with rate-based learning in SNNs. The timing-only SHD and SSC results directly contradict that view (Yu et al., 21 Jul 2025). A more accurate characterization is that Surrogate GD is a flexible optimization principle whose effectiveness depends on three coupled design choices: what is replaced by a surrogate, how the surrogate is parameterized, and whether the rest of the training stack—readout, constraints, architecture, and hardware mapping—is compatible with the target task.