Randomized Backward Propagation Process

• Randomized Backward Propagation Process is a family of learning procedures where backward error signals are delivered via fixed random or sign-concordant matrices instead of exact transposed weights.
• It offers alternative mechanisms like feedback alignment and meta-learned local plasticity rules, enhancing biological plausibility by decoupling the forward and backward passes.
• Empirical and theoretical studies show that despite using approximate feedback, these methods can converge effectively under overparameterized and regularized regimes with distinct trade-offs.

Randomized backwards propagation process denotes a family of learning procedures in which error information is communicated to earlier layers by mechanisms other than the exact transpose-weight reverse pass of standard backpropagation. In the most direct usage, it refers to random backpropagation and feedback alignment, where fixed random backward matrices replace transported forward weights in the hidden-layer update; in broader usage, it also includes sign-concordant random feedback realized by cortical-style microcircuits, meta-learned local plasticity rules operating with fixed random feedback pathways, and automatically searched backward equations that may contain random matrices, dropout, or Gaussian noise (Baldi et al., 2016). The term does not, however, cover every nonstandard backward process: several influential alternatives replace explicit reverse-mode transport by relaxation dynamics, wave propagation, or an additional deterministic gradient pass, but are not randomized in the relevant sense (Scellier et al., 2018).

1. Definition and scope

The expression is technically ambiguous because different literatures use “randomized,” “backward,” and “backpropagation” in different ways. The most stable core meaning is a backward credit-assignment mechanism in which the hidden-layer teaching signal is not computed with the exact transpose of the forward weights, but with a random or sign-concordant substitute. In that sense, the defining departure from standard backpropagation is the replacement of exact weight transport by a distinct backward channel whose numerical values are not constrained to equal the forward transpose (Song et al., 2021).

Several nearby constructions are often conflated with this core class. Some are genuinely random but not backward, such as randomized forward-mode gradient estimation. Others are genuinely backward but not random, such as equilibrium-style relaxation, deterministic forward-backward diffusion, or input-gradient penalties computed through an extra pass. Still others are stochastic only at the level of sampled data, random architectures, or exploratory search over update equations.

Formulation	Backward mechanism	Representative papers
Random backpropagation / feedback alignment	Fixed random feedback matrices replace transpose weights	(Baldi et al., 2016, Song et al., 2021)
Sign-concordant random feedback	Effective backward operator is random in magnitude but sign-concordant	(Yang et al., 2022, Shervani-Tabar et al., 2022)
Randomized forward alternative	One random directional derivative replaces the full reverse-mode gradient	(Shukla et al., 2023)
Deterministic nonstandard backward process	Relaxation, waves, or third-pass gradient penalties rather than random feedback	(Scellier et al., 2018, Betti et al., 2019, Xiong et al., 2022)

A persistent misconception is that any biologically plausible or asymmetric backward mechanism is thereby a randomized backward process. The literature does not support that identification. Asymmetry, relaxation, temporal delay, local Hebbian plasticity, and stochastic sampling are separable design choices, and only some combinations yield a genuinely randomized backward channel.

2. Canonical formulations: random feedback instead of weight transport

The canonical formulation appears in random backpropagation for layered networks. For an architecture ${\cal A}[N_0,\ldots,N_L]$ with forward dynamics

$$O_i^h=f_i^h(S_i^h), \qquad S_i^h=\sum_j w_{ij}^h O_j^{h-1},$$

standard backpropagation updates

$\Delta w_{ij}^h = \eta B_i^h O_j^{h-1},$

with backward recursion

$$B_i^h = (f_i^h)'\sum_k B_k^{h+1} w_{ki}^{h+1}, \qquad B_i^L=T_i-O_i^L.$$

Random Backpropagation replaces the transported forward weights by fixed random feedback coefficients: $\Delta w_{ij}^h=\eta R_i^h O_j^{h-1},$

$$R_i^h=(f_i^h)'\sum_k R_k^{h+1} c_{ki}^{h+1}, \qquad R_i^L=T_i-O_i^L.$$

The $c_{ki}^{h+1}$ are random, fixed matrices rather than transposes of $W^{h+1}$, so the backward pass becomes a distinct learning channel rather than an exact reverse-mode computation (Baldi et al., 2016).

This “learning channel” viewpoint is central. The forward pass already supplies the presynaptic term $O_j^{h-1}$; what must be delivered to a deep synapse is the target/output-dependent factor that modulates plasticity. Random backpropagation shows that this factor need not be the exact chain-rule derivative so long as it conveys sufficiently structured error information (Baldi et al., 2016).

The same idea is written in modern feedback-alignment form as

$$\boldsymbol{e}_{\ell}= \boldsymbol{B}_{\ell+1,\ell} \boldsymbol{e}_{\ell+1}\odot\sigma'(\boldsymbol{z}_{\ell}),$$

followed by a local weight update

$$O_i^h=f_i^h(S_i^h), \qquad S_i^h=\sum_j w_{ij}^h O_j^{h-1},$$0

Here $$O_i^h=f_i^h(S_i^h), \qquad S_i^h=\sum_j w_{ij}^h O_j^{h-1},$$1 is fixed and random under feedback alignment rather than set to $$O_i^h=f_i^h(S_i^h), \qquad S_i^h=\sum_j w_{ij}^h O_j^{h-1},$$2 as in exact backpropagation (Shervani-Tabar et al., 2022).

The literature derived several immediate variants. Skipped Random Backpropagation sends output error directly to each hidden layer through a random matrix $$O_i^h=f_i^h(S_i^h), \qquad S_i^h=\sum_j w_{ij}^h O_j^{h-1},$$3 rather than composing random matrices layer by layer. Adaptive Random Backpropagation updates the backward matrices themselves. Sparse variants replace dense random feedback by sparse random matrices. Across these families, one repeated empirical conclusion is that multiplication by the derivatives of the activation functions is important (Baldi et al., 2016).

Benchmark behavior is consistent with that interpretation. In one comparative study on MNIST, BP baseline was $$O_i^h=f_i^h(S_i^h), \qquad S_i^h=\sum_j w_{ij}^h O_j^{h-1},$$4, RBP baseline $$O_i^h=f_i^h(S_i^h), \qquad S_i^h=\sum_j w_{ij}^h O_j^{h-1},$$5, and SRBP baseline $$O_i^h=f_i^h(S_i^h), \qquad S_i^h=\sum_j w_{ij}^h O_j^{h-1},$$6; on CIFAR-10, BP baseline was $$O_i^h=f_i^h(S_i^h), \qquad S_i^h=\sum_j w_{ij}^h O_j^{h-1},$$7, RBP $$O_i^h=f_i^h(S_i^h), \qquad S_i^h=\sum_j w_{ij}^h O_j^{h-1},$$8, and SRBP $$O_i^h=f_i^h(S_i^h), \qquad S_i^h=\sum_j w_{ij}^h O_j^{h-1},$$9 (Baldi et al., 2016). These results established that random backward communication can be effective, but usually not identical to exact backpropagation.

3. Convergence, alignment, and the role of regularization

Theoretical analysis sharpened the distinction between successful learning and exact gradient transport. For two-layer networks under squared error loss, feedback alignment can drive training error to zero exponentially fast in the overparameterized regime even though the backward pass uses random fixed weights. A representative result proves

$\Delta w_{ij}^h = \eta B_i^h O_j^{h-1},$0

with high probability under Gaussian initialization and suitable width conditions (Song et al., 2021).

That result does not imply that the learned forward weights become aligned with the random backward weights. The same analysis defines alignment through the asymptotic cosine between the learned output weights $\Delta w_{ij}^h = \eta B_i^h O_j^{h-1},$1 and the fixed random feedback vector $\Delta w_{ij}^h = \eta B_i^h O_j^{h-1},$2: $\Delta w_{ij}^h = \eta B_i^h O_j^{h-1},$3 In the overparameterized regime without regularization, the theory gives

$\Delta w_{ij}^h = \eta B_i^h O_j^{h-1},$4

so learning can succeed while alignment vanishes as width grows (Song et al., 2021). This separates two ideas that are often merged in informal discussions: random backward weights can support optimization without necessarily becoming geometrically aligned to the learned forward channel.

Regularization changes that picture. For linear two-layer networks with a suitable $\Delta w_{ij}^h = \eta B_i^h O_j^{h-1},$5-type schedule and near-isometry assumptions on the data matrix, the same work proves persistent positive alignment,

$\Delta w_{ij}^h = \eta B_i^h O_j^{h-1},$6

for all sufficiently large $\Delta w_{ij}^h = \eta B_i^h O_j^{h-1},$7 (Song et al., 2021). In that regime, regularization damps the random initialization component of $\Delta w_{ij}^h = \eta B_i^h O_j^{h-1},$8 while preserving a $\Delta w_{ij}^h = \eta B_i^h O_j^{h-1},$9-dependent component accumulated through learning.

A distinct theoretical line studies the dynamics of random backpropagation directly, rather than its asymptotic alignment to exact backprop. For linear chains of arbitrary length, linear autoencoders with decorrelated data, and certain nonlinear chains when the derivative of the activation functions is included, Random Backpropagation converges to fixed points; for linear chains, the convergence target is a global minimum of the corresponding error surface (Baldi et al., 2016). In the simplest three-node linear chain, for example,

$$B_i^h = (f_i^h)'\sum_k B_k^{h+1} w_{ki}^{h+1}, \qquad B_i^L=T_i-O_i^L.$$0

and the critical points satisfy

$$B_i^h = (f_i^h)'\sum_k B_k^{h+1} w_{ki}^{h+1}, \qquad B_i^L=T_i-O_i^L.$$1

The remarkable feature is that the fixed random feedback coefficient $$B_i^h = (f_i^h)'\sum_k B_k^{h+1} w_{ki}^{h+1}, \qquad B_i^L=T_i-O_i^L.$$2 shapes the dynamics but not the location of the optimum in this elementary case (Baldi et al., 2016).

These results jointly establish the modern theoretical interpretation of randomized backward propagation: it is not exact gradient descent, but it can still be convergent, exponentially effective in overparameterized regimes, and in some settings partially alignable through additional structure such as regularization.

4. Biological and local implementations

A major motivation for randomized backward propagation is the biological implausibility of weight transport. One line of work asks whether random backpropagation can be realized using only local circuitry and local plasticity. A prominent answer proposes a cortical microcircuit with excitatory pyramidal neurons, inhibitory parvalbumin interneurons, and inhibitory somatostatin interneurons, organized with basal and apical dendritic compartments. In that framework, basal compartments carry feedforward drive, apical compartments carry error-like currents, and a neuron’s emitted postsynaptic current is modeled as a basal component plus an apical error modulation (Yang et al., 2022).

The optimization interpretation is explicit. If output-layer apical current encodes

$$B_i^h = (f_i^h)'\sum_k B_k^{h+1} w_{ki}^{h+1}, \qquad B_i^L=T_i-O_i^L.$$3

then the microcircuit is arranged so that paired excitatory and inhibitory backward pathways cancel the basal component and leave only an error-related signal. Under ideal local balancing, the effective backward operator is a random matrix $$B_i^h = (f_i^h)'\sum_k B_k^{h+1} w_{ki}^{h+1}, \qquad B_i^L=T_i-O_i^L.$$4 that is not equal to $$B_i^h = (f_i^h)'\sum_k B_k^{h+1} w_{ki}^{h+1}, \qquad B_i^L=T_i-O_i^L.$$5, but whose sign structure is concordant with the forward pathway because of Dale’s law. In that sense, the framework is equivalent to sign-concordant feedback alignment rather than exact backpropagation (Yang et al., 2022).

Its synaptic rules are fully local: $$B_i^h = (f_i^h)'\sum_k B_k^{h+1} w_{ki}^{h+1}, \qquad B_i^L=T_i-O_i^L.$$6 with basal synapses updated by Hebbian correlation between presynaptic postsynaptic currents and local error variables, and apical synapses updated anti-Hebbianly to balance competing backward branches (Yang et al., 2022). On benchmarks, the model reached $$B_i^h = (f_i^h)'\sum_k B_k^{h+1} w_{ki}^{h+1}, \qquad B_i^L=T_i-O_i^L.$$7 best accuracy on MNIST with 5 discrete time steps and $$B_i^h = (f_i^h)'\sum_k B_k^{h+1} w_{ki}^{h+1}, \qquad B_i^L=T_i-O_i^L.$$8 on CIFAR10 (Yang et al., 2022).

A complementary strategy keeps the random backward matrices fixed and instead meta-learns a better local plasticity rule for the forward weights. In that setting, hidden errors are still propagated by

$$B_i^h = (f_i^h)'\sum_k B_k^{h+1} w_{ki}^{h+1}, \qquad B_i^L=T_i-O_i^L.$$9

but the forward update is no longer restricted to the vanilla pseudo-gradient. The learned sparse rule is

$\Delta w_{ij}^h=\eta R_i^h O_j^{h-1},$0

The three components are, respectively, an FA-style pseudo-gradient term, an error-Hebbian term, and an Oja term (Shervani-Tabar et al., 2022).

The error-Hebbian component is especially relevant to randomized backward propagation because, under simplifying assumptions,

$\Delta w_{ij}^h=\eta R_i^h O_j^{h-1},$1

This suggests a direct channel by which information about the random feedback matrix is written into the forward weights, accelerating effective alignment (Shervani-Tabar et al., 2022). The meta-learning experiments are explicitly online, with batch size $\Delta w_{ij}^h=\eta R_i^h O_j^{h-1},$2, $\Delta w_{ij}^h=\eta R_i^h O_j^{h-1},$3 classes per task, $\Delta w_{ij}^h=\eta R_i^h O_j^{h-1},$4 training examples per class, and thus $\Delta w_{ij}^h=\eta R_i^h O_j^{h-1},$5 training examples total in each episode (Shervani-Tabar et al., 2022).

5. Search-based and stochastic extensions of backward rules

Randomization in the backward process is not limited to fixed feedback matrices. Another direction treats the backward equation itself as a search space. One study introduces a domain-specific language for layerwise backward signals $\Delta w_{ij}^h=\eta R_i^h O_j^{h-1},$6 built from operands, unary functions, and binary functions, and then uses an evolution-based method to discover new propagation rules (Alber et al., 2018).

The search space explicitly includes standard gradient terms, feedback-alignment-like terms, direct-feedback-alignment-like terms, Gaussian random matrices $\Delta w_{ij}^h=\eta R_i^h O_j^{h-1},$7, Bernoulli random matrices $\Delta w_{ij}^h=\eta R_i^h O_j^{h-1},$8, Gaussian noise injection, dropout, clipping, sign operations, and several norm-based normalizations. A candidate backward rule can therefore be deterministic, stochastic, or mixed. The discovered rules are not constrained to remain exact chain-rule derivatives, which makes the procedure a genuine search over alternative backward computational graphs rather than a small perturbation of standard backpropagation (Alber et al., 2018).

The empirical pattern is specific. The search was performed on CIFAR-10 using Wide ResNet child models. For WRN 16-2 trained for 20 epochs, the baseline $\Delta w_{ij}^h=\eta R_i^h O_j^{h-1},$9 with SGD achieved $$R_i^h=(f_i^h)'\sum_k R_k^{h+1} c_{ki}^{h+1}, \qquad R_i^L=T_i-O_i^L.$$0, whereas discovered equations reached $$R_i^h=(f_i^h)'\sum_k R_k^{h+1} c_{ki}^{h+1}, \qquad R_i^L=T_i-O_i^L.$$1; with SGD+Momentum, the baseline was $$R_i^h=(f_i^h)'\sum_k R_k^{h+1} c_{ki}^{h+1}, \qquad R_i^L=T_i-O_i^L.$$2 and the best discovered equation reached $$R_i^h=(f_i^h)'\sum_k R_k^{h+1} c_{ki}^{h+1}, \qquad R_i^L=T_i-O_i^L.$$3 (Alber et al., 2018). Under longer training horizons, however, discovered rules generally performed similarly to standard backpropagation at convergence rather than clearly surpassing it (Alber et al., 2018).

This line of work broadens the meaning of randomized backward propagation. The random element may lie in fixed backward matrices, in stochastic transformations applied to a gradient-like signal, or in the search process that discovers the update equation. It also shows that short-horizon acceleration and asymptotic superiority are distinct objectives: a randomized backward rule can improve early training without becoming the best converged optimizer.

Stochasticity of the model itself should not be confused with stochasticity of the backward pass. A useful counterexample is the Backpropagation Neural Tree, described as a stochastic computational dendritic tree with random repeated inputs through its leaves and stochastic architecture generation, yet trained by a deterministic recursive backpropagation procedure executed in pre-order traversal during the backward phase (Ojha et al., 2022). Here the model is stochastic, but the backward rule is not.

6. Distinct neighboring concepts and terminological pitfalls

Several methods are adjacent to randomized backward propagation but technically different. A mathematically clean example is the randomized forward mode gradient, defined by

$$R_i^h=(f_i^h)'\sum_k R_k^{h+1} c_{ki}^{h+1}, \qquad R_i^L=T_i-O_i^L.$$4

This estimator uses forward-mode automatic differentiation to compute one random directional derivative and lifts it back to a vector estimator. If $$R_i^h=(f_i^h)'\sum_k R_k^{h+1} c_{ki}^{h+1}, \qquad R_i^L=T_i-O_i^L.$$5 is isotropic, it is unbiased up to a scalar factor, and the paper shows that among the candidate distributions Bernoulli directions are preferred because smaller kurtosis yields smaller expected relative squared error (Shukla et al., 2023). Despite the randomization, this is not a randomized backward pass; it replaces reverse mode by a randomized forward-mode surrogate.

Other neighboring methods are nonrandom alternatives to backpropagation. Generalized equilibrium propagation replaces explicit backward transport by a two-phase relaxation process for fixed-point recurrent networks. The update direction depends on

$$R_i^h=(f_i^h)'\sum_k R_k^{h+1} c_{ki}^{h+1}, \qquad R_i^L=T_i-O_i^L.$$6

so its approximation error is tied to asymmetry of the state Jacobian rather than to randomness (Scellier et al., 2018). Backprop diffusion likewise interprets backpropagation as a deterministic process of forward and backward waves in a network with signaling delays, with local updates generated where the waves meet (Betti et al., 2019). Reverse Back Propagation adds a third deterministic pass that penalizes the input-end gradient through an auxiliary loss

$$R_i^h=(f_i^h)'\sum_k R_k^{h+1} c_{ki}^{h+1}, \qquad R_i^L=T_i-O_i^L.$$7

again without any randomized backward operator (Xiong et al., 2022).

Even ordinary stochastic gradient descent does not qualify merely because samples are random. In the convergence theorem for sample-wise backpropagation, the backward recursion itself is deterministic once a sample is fixed; the stochasticity lies only in the i.i.d. sampling of inputs $$R_i^h=(f_i^h)'\sum_k R_k^{h+1} c_{ki}^{h+1}, \qquad R_i^L=T_i-O_i^L.$$8 (Wu, 2021). Likewise, outside neural learning altogether, “backwards analysis” studies a randomized reverse exposure process on permutations and asks how much entropy is required for expected incremental-cost bounds; it is a combinatorial random process, not a neural credit-assignment rule (Knudsen et al., 2017).

The central conceptual boundary is therefore clear. A randomized backwards propagation process, in the technical sense developed in the neural-learning literature, is a credit-assignment procedure in which backward teaching signals are conveyed by random or sign-concordant surrogate pathways rather than by transported forward transposes. Its most established forms are Random Backpropagation, feedback alignment, sign-concordant microcircuit realizations, and local plasticity rules designed to exploit fixed random feedback. Its principal theoretical questions are convergence, alignment, locality, and the degree to which exact gradient transport is actually necessary for learning (Baldi et al., 2016).