Sample-Wise Backpropagation

• Sample-wise backpropagation is a method that computes gradients for each individual sample, enabling explicit layerwise recursions and improved gradient transparency.
• It applies both in deterministic feedforward networks via two-step matrix recursions and in stochastic settings using trajectory-wise Hamiltonian gradients.
• This technique supports varied applications including online SGD, proximal updates, and spiking neural networks, thereby aiding in debugging and convergence analysis.

Searching arXiv for recent and foundational papers on sample-wise backpropagation and closely related variants. Sample-wise backpropagation denotes a family of training procedures in which forward states, adjoint variables, and parameter gradients are computed for each individual sample or trajectory, rather than being introduced only through aggregate batch formulas. In deterministic feedforward networks, this viewpoint yields an explicit layerwise matrix recursion for activations and gradients; in stochastic neural networks and stochastic optimal control, it appears as a trajectory-wise approximation of adjoint backward stochastic differential equations and Hamiltonian derivatives. Across these settings, the unifying theme is that the gradient signal is constructed locally along a single sample path and may then be used directly in stochastic gradient descent or averaged across a batch (Boughammoura, 2023, Archibald et al., 2022, Sun et al., 5 May 2025).

1. Conceptual scope and defining features

The defining feature of sample-wise backpropagation is that the computational graph is traversed on a per-sample basis. In the feedforward setting, each recursion and gradient step requires only the activations and weight matrices for a single data point. In the stochastic-control setting, the analogous object is a single simulated trajectory of the forward dynamics together with its associated backward adjoint variables. This makes the chain of computations explicit and modularized into local operations.

A central consequence is that sample-wise backpropagation is not restricted to purely online learning. The same per-sample gradients can be used in stochastic updates, averaged over a mini-batch, or incorporated into batch estimators. The literature therefore treats “sample-wise” as a computational organization of gradient construction rather than as a prohibition on batching. The deterministic two-step rule is “directly structured for sample-by-sample computation,” while the stochastic-control literature introduces explicit batch sample-wise estimators by averaging per-trajectory Hamiltonian gradients (Boughammoura, 2023, Sun et al., 5 May 2025).

Another recurrent feature is adjoint structure. In conventional feedforward networks, the backward pass is the reverse recursion of the forward pass. In stochastic neural networks and stochastic optimal control, the same role is played by backward SDEs or BSDEs. This establishes a common interpretation: sample-wise backpropagation propagates local sensitivity information backward along the realized sample path, whether that path is a deterministic activation chain, a diffusion trajectory, or a sequence of spike events.

2. Two-step matrix recursion in deterministic feedforward networks

In "A Two-Step Rule for Backpropagation" (Boughammoura, 2023), the forward pass for a feedforward network is written, for each sample, as

$$\begin{align*} X^0 &= x,\ Y^h &= W^h X^{h-1},\ X^h &= \sigma(Y^h), \qquad h=1,\ldots,L. \end{align*}$$

This decomposition isolates a linear map followed by an element-wise nonlinearity at every layer.

The backward pass is written in a parallel two-step form, but in the opposite direction. Using

$$\text{sup}_h = \frac{\partial J}{\partial X^h}, \qquad \text{sdown}_h = \frac{\partial J}{\partial Y^h},$$

the recursion is

$$\begin{align*} \text{sup}_L &= \frac{\partial J(f(x),y)}{\partial X^L},\ \text{sdown}_h &= \text{sup}_h \odot \sigma'(Y^h),\ \text{sup}_{h-1} &= (W^h)^T \text{sdown}_h,\ \frac{\partial J}{\partial W^h} &= \text{sdown}_h (X^{h-1})^T, \qquad h=L,\ldots,1. \end{align*}$$

Here the Hadamard product implements the nonlinear derivative step, and the transposed weight matrix propagates the upstream derivative to the previous layer.

This formulation emphasizes three properties. First, the backward pass mirrors the forward pass through a transposed linear map and an element-wise multiplication. Second, the gradient with respect to each weight matrix is an outer product of the local delta and the previous activation for the same sample. Third, the forward and backward passes are presented as adjoint mappings in the matrix diagrams discussed in the paper. The paper states that this organization is modular, sample-wise, and potentially more efficient and transparent than traditional recursive formulations. A plausible implication is that it is particularly useful when explicit per-sample gradients are needed for debugging, algorithmic analysis, or online updates.

The paper also contrasts this with standard presentations of backpropagation, which are often written as a single recursive delta equation or in a batch-oriented form. The two-step rule therefore does not change the gradient itself; it changes the decomposition and computational viewpoint.

3. Sample-wise adjoints in stochastic neural networks and stochastic optimal control

In "Convergence Analysis for Training Stochastic Neural Networks via Stochastic Gradient Descent" (Archibald et al., 2022), the network is formulated as a discretization of a stochastic differential equation,

$$X_{n+1} = X_n + h f(X_n,u_n) + \sqrt{h}\, g(u_n)\omega_n, \qquad n=0,\dots,N-1.$$

Training is posed as a stochastic optimal control problem with cost functional

$$J(u) = \left[\int_0^T r(X_t,u_t)\,dt + \Phi(X_T,\Gamma)\right].$$

The gradient is expressed through an adjoint backward SDE: $$\nabla J_u(u_t) = f_u(X_t,u_t)^\top Y_t + g_u(u_t)^\top Z_t + r_u(X_t,u_t)^\top.$$

The sample-wise step is to avoid solving the backward equation globally and instead approximate it along the realized forward trajectory. For iteration $k$, the sample-wise gradient estimator is

$$\nabla j^k_u(u^k_n) = f_u(X^k_n,u^k_n)^\top Y^k_n + g_u(u^k_n)^\top Z^k_n + r_u(X^k_n,u^k_n)^\top,$$

and the projected SGD update is

$$u^{k+1}_{t_n} = \mathcal{P}_{\mathcal{U}_N} \left( u^k_{t_n}-\eta_k \nabla j^k_u(u^k_{t_n}) \right).$$

The paper states that this is computationally similar to standard SGD, except that the gradient is formed from sample-wise backward SDE pseudo-data rather than from a full BSDE solver.

The convergence result in the strongly convex case is

$$\mathbb{E}\|u^{K+1}-u^*\|_2^2 \leq C\left(\frac{N}{K}+\frac{1}{N}\right).$$

The paper explicitly interprets this to mean that the number of training steps should be proportional to the square of the number of layers in the convex optimization case. In the non-convex case, it proves that for fixed depth $$\text{sup}_h = \frac{\partial J}{\partial X^h}, \qquad \text{sdown}_h = \frac{\partial J}{\partial Y^h},$$0,

$$\text{sup}_h = \frac{\partial J}{\partial X^h}, \qquad \text{sdown}_h = \frac{\partial J}{\partial Y^h},$$1

This stochastic-control formulation clarifies that sample-wise backpropagation is not confined to standard deterministic neural networks. It is equally a method for constructing unbiased or trajectory-wise gradient surrogates in settings where the state itself evolves stochastically and the adjoint object is a BSDE rather than a matrix transpose.

4. Batch sample-wise schemes, convergence-rate improvements, and high-order refinements

"Batch Sample-wise Stochastic Optimal Control via Stochastic Maximum Principle" (Sun et al., 5 May 2025) extends the sample-wise paradigm by averaging per-trajectory Hamiltonian derivatives over a batch. For the controlled SDE

$$\text{sup}_h = \frac{\partial J}{\partial X^h}, \qquad \text{sdown}_h = \frac{\partial J}{\partial Y^h},$$2

the Hamiltonian is

$$\text{sup}_h = \frac{\partial J}{\partial X^h}, \qquad \text{sdown}_h = \frac{\partial J}{\partial Y^h},$$3

and the sample-wise gradient at time $$\text{sup}_h = \frac{\partial J}{\partial X^h}, \qquad \text{sdown}_h = \frac{\partial J}{\partial Y^h},$$4 is

$$\text{sup}_h = \frac{\partial J}{\partial X^h}, \qquad \text{sdown}_h = \frac{\partial J}{\partial Y^h},$$5

Using $$\text{sup}_h = \frac{\partial J}{\partial X^h}, \qquad \text{sdown}_h = \frac{\partial J}{\partial Y^h},$$6 independent samples, the batch estimator is

$$\text{sup}_h = \frac{\partial J}{\partial X^h}, \qquad \text{sdown}_h = \frac{\partial J}{\partial Y^h},$$7

and the projected SGD step becomes

$$\text{sup}_h = \frac{\partial J}{\partial X^h}, \qquad \text{sdown}_h = \frac{\partial J}{\partial Y^h},$$8

The paper states that batch samples with a higher order scheme in the forward SDE improve the convergence rate from

$$\text{sup}_h = \frac{\partial J}{\partial X^h}, \qquad \text{sdown}_h = \frac{\partial J}{\partial Y^h},$$9

to

$$\begin{align*} \text{sup}_L &= \frac{\partial J(f(x),y)}{\partial X^L},\ \text{sdown}_h &= \text{sup}_h \odot \sigma'(Y^h),\ \text{sup}_{h-1} &= (W^h)^T \text{sdown}_h,\ \frac{\partial J}{\partial W^h} &= \text{sdown}_h (X^{h-1})^T, \qquad h=L,\ldots,1. \end{align*}$$0

and gives the theorem

$$\begin{align*} \text{sup}_L &= \frac{\partial J(f(x),y)}{\partial X^L},\ \text{sdown}_h &= \text{sup}_h \odot \sigma'(Y^h),\ \text{sup}_{h-1} &= (W^h)^T \text{sdown}_h,\ \frac{\partial J}{\partial W^h} &= \text{sdown}_h (X^{h-1})^T, \qquad h=L,\ldots,1. \end{align*}$$1

The same work also introduces a damped contraction update,

$$\begin{align*} \text{sup}_L &= \frac{\partial J(f(x),y)}{\partial X^L},\ \text{sdown}_h &= \text{sup}_h \odot \sigma'(Y^h),\ \text{sup}_{h-1} &= (W^h)^T \text{sdown}_h,\ \frac{\partial J}{\partial W^h} &= \text{sdown}_h (X^{h-1})^T, \qquad h=L,\ldots,1. \end{align*}$$2

with

$$\begin{align*} \text{sup}_L &= \frac{\partial J(f(x),y)}{\partial X^L},\ \text{sdown}_h &= \text{sup}_h \odot \sigma'(Y^h),\ \text{sup}_{h-1} &= (W^h)^T \text{sdown}_h,\ \frac{\partial J}{\partial W^h} &= \text{sdown}_h (X^{h-1})^T, \qquad h=L,\ldots,1. \end{align*}$$3

For a special case, the paper proves

$$\begin{align*} \text{sup}_L &= \frac{\partial J(f(x),y)}{\partial X^L},\ \text{sdown}_h &= \text{sup}_h \odot \sigma'(Y^h),\ \text{sup}_{h-1} &= (W^h)^T \text{sdown}_h,\ \frac{\partial J}{\partial W^h} &= \text{sdown}_h (X^{h-1})^T, \qquad h=L,\ldots,1. \end{align*}$$4

"A High-order Backpropagation Algorithm for Neural Stochastic Differential Equation Model" (Sheng et al., 8 Sep 2025) pushes this further for Neural SDEs. It states that improving the convergence order is challenging because of the loss of information of the $$\begin{align*} \text{sup}_L &= \frac{\partial J(f(x),y)}{\partial X^L},\ \text{sdown}_h &= \text{sup}_h \odot \sigma'(Y^h),\ \text{sup}_{h-1} &= (W^h)^T \text{sdown}_h,\ \frac{\partial J}{\partial W^h} &= \text{sdown}_h (X^{h-1})^T, \qquad h=L,\ldots,1. \end{align*}$$5 term in backward equations under sample-wise approximation and the limitations of the forward network form. Under convexity, the paper gives

$$\begin{align*} \text{sup}_L &= \frac{\partial J(f(x),y)}{\partial X^L},\ \text{sdown}_h &= \text{sup}_h \odot \sigma'(Y^h),\ \text{sup}_{h-1} &= (W^h)^T \text{sdown}_h,\ \frac{\partial J}{\partial W^h} &= \text{sdown}_h (X^{h-1})^T, \qquad h=L,\ldots,1. \end{align*}$$6

and states that first-order convergence is achieved when the number of training steps is proportional to the cubic number of layers. This places the $$\begin{align*} \text{sup}_L &= \frac{\partial J(f(x),y)}{\partial X^L},\ \text{sdown}_h &= \text{sup}_h \odot \sigma'(Y^h),\ \text{sup}_{h-1} &= (W^h)^T \text{sdown}_h,\ \frac{\partial J}{\partial W^h} &= \text{sdown}_h (X^{h-1})^T, \qquad h=L,\ldots,1. \end{align*}$$7-term approximation at the center of the accuracy bottleneck in high-order sample-wise stochastic training.

A further point in the batch SMP paper is that, when randomized neural networks are used, error backward propagation can be avoided and parameter update can be achieved via purely algebraic computation (vector algebra). The paper explicitly presents output-layer sample-wise gradients for parameters $$\begin{align*} \text{sup}_L &= \frac{\partial J(f(x),y)}{\partial X^L},\ \text{sdown}_h &= \text{sup}_h \odot \sigma'(Y^h),\ \text{sup}_{h-1} &= (W^h)^T \text{sdown}_h,\ \frac{\partial J}{\partial W^h} &= \text{sdown}_h (X^{h-1})^T, \qquad h=L,\ldots,1. \end{align*}$$8 and $$\begin{align*} \text{sup}_L &= \frac{\partial J(f(x),y)}{\partial X^L},\ \text{sdown}_h &= \text{sup}_h \odot \sigma'(Y^h),\ \text{sup}_{h-1} &= (W^h)^T \text{sdown}_h,\ \frac{\partial J}{\partial W^h} &= \text{sdown}_h (X^{h-1})^T, \qquad h=L,\ldots,1. \end{align*}$$9, and notes this as a future direction for improving efficiency.

5. Related algorithmic variants

Sample-wise backpropagation sits inside a larger landscape of backpropagation variants that preserve layerwise or eventwise locality while modifying the update rule, the state representation, or the loss structure.

Method	Core mechanism	Sample-wise relevance
ProxProp	Implicit proximal update for weights	Compatible with sample-wise or batch size 1 training
EventProp	Exact adjoint jumps at spike times	Event-based, temporally and spatially sparse
Double backpropagation	Loss depends on input derivatives or Jacobians	Uses extra backward recursions for sample-wise derivative terms
ZORB	Pseudoinverse-based target backpropagation	Contrast case: inherently batch-wise in this version

"Proximal Backpropagation" (Frerix et al., 2017) replaces explicit parameter-gradient steps by proximal mappings. For each linear layer, the update is

$$X_{n+1} = X_n + h f(X_n,u_n) + \sqrt{h}\, g(u_n)\omega_n, \qquad n=0,\dots,N-1.$$0

The paper proves that the ProxProp direction is always a descent direction, that fixed points coincide with stationary points of the original loss, and that the method remains compatible with mini-batch or sample-wise stochastic updates. Its sample-wise relevance is therefore not in redefining the backward signal, but in modifying the weight-update operator while retaining the same local layerwise structure.

"Event-Based Backpropagation can compute Exact Gradients for Spiking Neural Networks" (Wunderlich et al., 2020) establishes an exact event-based sample-wise backpropagation rule for continuous-time spiking neural networks. The central gradient formula is

$$X_{n+1} = X_n + h f(X_n,u_n) + \sqrt{h}\, g(u_n)\omega_n, \qquad n=0,\dots,N-1.$$1

with adjoint variables updated by explicit jump conditions at spike times. The backward pass traverses spike events in reverse order, accumulates synaptic gradients only at those events, and stores only spike times and relevant state variables. The paper reports $$X_{n+1} = X_n + h f(X_n,u_n) + \sqrt{h}\, g(u_n)\omega_n, \qquad n=0,\dots,N-1.$$2 test accuracy on Yin-Yang and $$X_{n+1} = X_n + h f(X_n,u_n) + \sqrt{h}\, g(u_n)\omega_n, \qquad n=0,\dots,N-1.$$3 test accuracy on MNIST.

"A Closer Look at Double Backpropagation" (Etmann, 2019) generalizes backpropagation to objectives containing input-derivative terms, such as

$$X_{n+1} = X_n + h f(X_n,u_n) + \sqrt{h}\, g(u_n)\omega_n, \qquad n=0,\dots,N-1.$$4

or Jacobian Frobenius penalties. In a Hilbert space framework, it derives additional backward-backward and forward-backward recursions for such sample-wise regularizers. For locally linear activation functions, the paper states a reduction of calculations for Frobenius-norm penalties on Jacobians by roughly a third. It also analyzes the discontinuous loss surface of ReLU networks in both inputs and parameters.

"ZORB: A Derivative-Free Backpropagation Algorithm for Neural Networks" (Ranganathan et al., 2020) is a useful contrast. It backpropagates targets using Moore-Penrose pseudoinverses instead of derivatives, with layerwise least-squares solutions such as

$$X_{n+1} = X_n + h f(X_n,u_n) + \sqrt{h}\, g(u_n)\omega_n, \qquad n=0,\dots,N-1.$$5

The paper states that this version is inherently batch-wise and less suited to online or very large-scale streaming data without further adaptation. It therefore illuminates a boundary of the sample-wise paradigm: locality of backward information does not by itself imply per-sample training if the solver is intrinsically batch linear algebra.

6. Misconceptions, limitations, and significance

A common misconception is that sample-wise backpropagation is merely an implementation detail of ordinary SGD. The literature indicates a broader role. In deterministic feedforward networks it reorganizes the backward recursion into explicit linear and nonlinear steps; in stochastic neural networks it replaces a global adjoint solve by trajectory-wise BSDE approximation; in spiking systems it yields exact event-based gradients at discontinuities. This suggests that “sample-wise” is a structural descriptor of how sensitivities are propagated, not only a statement about batch size.

Another misconception is that exact gradient propagation is impossible once the model contains discontinuities or event-driven dynamics. EventProp derives exact gradients through discrete spike events by combining the adjoint method with partial derivative jumps (Wunderlich et al., 2020). For ReLU networks with derivative-based losses, the discontinuities are analyzed as jump discontinuities on polytope boundaries, and the paper states that these do not pose a big problem in reality because the ill-defined points form a set of measure zero and batch averaging smooths the landscape (Etmann, 2019).

The main technical limitations in the stochastic literature are also explicit. In Neural SDE training, the $$X_{n+1} = X_n + h f(X_n,u_n) + \sqrt{h}\, g(u_n)\omega_n, \qquad n=0,\dots,N-1.$$6 term in the backward equation is identified as the main source of uncertainty under sample-wise approximation, and improving convergence order remains challenging because of loss of information in $$X_{n+1} = X_n + h f(X_n,u_n) + \sqrt{h}\, g(u_n)\omega_n, \qquad n=0,\dots,N-1.$$7 and the limitations of the forward network form (Sun et al., 5 May 2025, Sheng et al., 8 Sep 2025). In the 2022 stochastic neural network analysis, bounded and differentiable activation functions are necessary, while unbounded activations such as ReLU cause failures in the reported function-learning experiments (Archibald et al., 2022). In the damped contraction algorithm, effective contraction requires larger batch sizes such as $$X_{n+1} = X_n + h f(X_n,u_n) + \sqrt{h}\, g(u_n)\omega_n, \qquad n=0,\dots,N-1.$$8; otherwise, error plateaus (Sun et al., 5 May 2025).

The broader significance of sample-wise backpropagation is therefore methodological rather than purely terminological. It provides a common language for exact per-sample matrix gradients, pathwise adjoint methods for stochastic models, event-driven gradient transport in spiking networks, and higher-order or proximal refinements of local updates. A plausible implication is that future work will continue to treat sample-wise computation as a design axis for learning algorithms, especially where transparency of gradient flow, control-theoretic structure, sparse event processing, or nonstandard update operators are central.