Random Backpropagation: Theory & Variants

• Random Backpropagation (RBP) is a family of supervised learning algorithms that replaces exact transposes with fixed random feedback matrices to address the weight-transport problem.
• Variants like SRBP, DFA, and adaptive feedback offer distinct methods of propagating error signals, each affecting convergence and performance differently.
• Empirical and theoretical studies show that while RBP can achieve near-backprop accuracy in shallow networks, challenges persist in deep, online, and biologically realistic settings.

Random Backpropagation (RBP), often identified with Feedback Alignment (FA), is a family of supervised credit-assignment algorithms in which the transpose of the forward weight matrices used by standard backpropagation is replaced by fixed random feedback matrices, thereby removing the weight-transport requirement while preserving local weight updates driven by presynaptic activity and a postsynaptic error-like signal (Baldi et al., 2016, Shervani-Tabar et al., 2022). Within the broader “deep learning channel” framework, RBP treats deep learning as the problem of communicating non-local target information to deep synapses through a separate feedback pathway rather than through exact reuse of the forward pathway (Baldi et al., 2017). It has consequently become a central reference point in research on biologically plausible learning, with extensions to direct-feedback, adaptive-feedback, sparse, sign-concordant, spiking, neuromorphic, and meta-learned local-plasticity settings (Yang et al., 2022, Kaiser et al., 2019).

1. Conceptual basis and the weight-transport problem

Standard backpropagation computes hidden-layer error signals by explicitly reusing downstream forward weights. For a feedforward network with layers $l = 1,\dots,L$, forward weights $W_l$, pre-activations $a_l$, activations $h_l$, and loss $L(h_L,y)$, the forward dynamics are

$a_l = W_l h_{l-1}, \qquad h_l = f(a_l),$

and standard backpropagation uses

$$\delta_L = \partial L / \partial a_L, \qquad \delta_l = (W_{l+1}^\top \delta_{l+1}) \odot f'(a_l), \qquad \Delta W_l = -\eta \delta_l h_{l-1}^\top.$$

These equations imply that hidden-layer updates require exact access to $W_{l+1}^\top$, which is the weight-transport problem (Shervani-Tabar et al., 2022).

RBP replaces the transposed forward matrix with a fixed random matrix $B_{l+1}$ sampled at initialization and then held constant:

$$\delta_L = \partial L / \partial a_L, \qquad \delta_l = (B_{l+1}\delta_{l+1}) \odot f'(a_l), \qquad \Delta W_l = -\eta \delta_l h_{l-1}^\top.$$

In this sense, RBP preserves the local product structure of gradient descent while discarding exact backward symmetry (Shervani-Tabar et al., 2022). The same basic idea is presented in the “deep learning channel” language as a way to communicate non-local target information to deep synapses under locality constraints in space and time (Baldi et al., 2016, Baldi et al., 2017).

This formulation is closely tied to the claim that deep local learning alone is insufficient for complex tasks. In the 2016 analysis, local synaptic rules of the canonical form

$W_l$0

for hidden layers and

$W_l$1

at the output layer do not, by themselves, solve the critical equations of deep supervised learning; some channel must deliver error-related information to deep weights (Baldi et al., 2016). RBP’s distinct contribution is the claim that this channel need not carry exact transposes of forward weights.

2. Algorithmic forms and main variants

RBP is best understood as a family rather than a single recursion. The core variants differ primarily in how the feedback signal is routed and whether the backward pathway is fixed or adaptive (Baldi et al., 2016, Baldi et al., 2017).

Variant	Backward signal	Distinctive property
RBP / FA	$W_l$2	Layerwise fixed random feedback
SRBP / DFA-style direct feedback	$W_l$3 or $W_l$4	Direct output-to-hidden error broadcast
ARBP / ASRBP	Feedback matrices adapted locally	Same local rules can be used in the learning channel
SCFA	$W_l$5	Sign-concordant feedback improves alignment

Skipped Random Backpropagation (SRBP) sends output-layer error directly to each hidden layer via a fixed random matrix, bypassing staged layer-by-layer propagation. In the 2016 treatment, SRBP addresses architectural and derivative-symmetry issues more aggressively than staged RBP because each hidden layer needs only the local derivative at the receiving neuron, not the transport of derivatives from higher layers (Baldi et al., 2016). Direct Feedback Alignment (DFA) in the later terminology is the closely related rule

$W_l$6

with $W_l$7 mapping output errors into layer-$W_l$8 coordinates (Shervani-Tabar et al., 2022).

Adaptive variants update the learning-channel weights locally rather than freezing them. In the “symmetries of the deep learning channel” formulation, a feedback weight $W_l$9 can be updated by

$a_l$0

and matrix forms such as $a_l$1 are used to define ARBP (Baldi et al., 2017). The 2016 paper also introduced sparse feedback and combinations such as ASRBP, showing that random feedback does not need to be dense to remain effective (Baldi et al., 2016).

A persistent empirical point across these variants is the importance of multiplying the feedback signal by the local activation derivative. In the 2016 simulations, removing $a_l$2 reduced MNIST test accuracy from 97.9% to 89.9% for BP, from 97.2% to 88.3% for RBP, and from 97.2% to 88.4% for SRBP; on CIFAR-10 the corresponding no-$a_l$3 results were 54.8% for BP, 32.7% for RBP, and 39.9% for SRBP (Baldi et al., 2016). The later symmetry analysis likewise concluded that omitting all derivatives does not work, whereas using the same nonlinear neurons in both forward and learning channels can work provided the nonlinearities are not purely non-negative (Baldi et al., 2017).

3. Convergence, alignment, and symmetry properties

The main modern mathematical analysis of RBP focuses on two-layer networks in the overparameterized regime. For

$a_l$4

with squared error loss, standard backprop updates the hidden weights using the forward coefficient $a_l$5, while FA/RBP replaces it with a fixed random feedback weight $a_l$6:

$a_l$7

Under bounded smooth activation, Assumption $a_l$8, bounded data, Gaussian initialization, and sufficiently large width $a_l$9, the residual error contracts exponentially:

$h_l$0

The same analysis shows that the weights remain close to initialization, which is the basis for the NTK-style argument used in the proof (Song et al., 2021).

The central theoretical controversy concerns alignment. In scalar-output regression, the alignment measure is

$h_l$1

Without regularization, the theory gives

$h_l$2

so alignment vanishes as width grows. With L2 regularization on $h_l$3, however, the linear-case theory proves that a schedule $h_l$4 for $h_l$5 and $h_l$6 for $h_l$7 can induce positive asymptotic alignment, with

$h_l$8

for sufficiently large $h_l$9, where $L(h_L,y)$0 does not depend on $L(h_L,y)$1, $L(h_L,y)$2, $L(h_L,y)$3, or $L(h_L,y)$4 (Song et al., 2021). This establishes that convergence of training error and alignment of forward and feedback pathways are distinct phenomena: the former can occur in wide networks without the latter.

Earlier analytical work approached the same question through continuous-time polynomial ODEs. For linear chains such as $L(h_L,y)$5, $L(h_L,y)$6, arbitrary-depth chains $L(h_L,y)$7, expansive $L(h_L,y)$8, and compressive $L(h_L,y)$9, RBP, SRBP, and adaptive variants were shown to converge to fixed points corresponding to global minima of quadratic error under broad initial conditions (Baldi et al., 2016). The 2017 symmetry analysis further proved, in general linear adaptive cases, identities such as

$a_l = W_l h_{l-1}, \qquad h_l = f(a_l),$0

showing that backward weights can track the transpose of forward weights up to an initialization-dependent offset, with small initialization producing “essentially symmetric” channels during learning (Baldi et al., 2017).

These results collectively restrict a common misconception. RBP is not simply “random backpropagation that mysteriously works”; rather, the published theory supports exponential error decay in wide two-layer settings, while also showing that alignment is conditional and may require regularization or adaptive structure (Song et al., 2021).

4. Meta-learned local plasticity rules with random feedback

A major recent development is the use of meta-learning to improve RBP under biologically constrained online learning. In the 2022 study on fixed random feedback pathways, the objective was not to meta-learn initial weights but to meta-learn a sparse, interpretable local plasticity rule shared across all layers, with networks reinitialized each episode to isolate the effect of the rule itself (Shervani-Tabar et al., 2022).

The inner-loop protocol was deliberately restrictive: forward weights and feedback matrices were Xavier-initialized at the start of each episode, feedback matrices were fixed random matrices, hidden units used softplus with $a_l = W_l h_{l-1}, \qquad h_l = f(a_l),$1, the output used softmax, training was online with batch size 1, and each episode contained only 250 training examples. The architecture used for the principal experiments was a 5-layer fully connected network with dimensions 784-170-130-100-70-47, and the outer loop used 600 episodes of ADAM with meta-learning rate $a_l = W_l h_{l-1}, \qquad h_l = f(a_l),$2 and an L1 penalty on the plasticity coefficients (Shervani-Tabar et al., 2022).

The discovered rule was

$a_l = W_l h_{l-1}, \qquad h_l = f(a_l),$3

with $a_l = W_l h_{l-1}, \qquad h_l = f(a_l),$4. Here the first term is the FA/DFA-like pseudo-gradient, the second is an error-based Hebbian component (“eHebb”), and the third is Oja’s rule (Shervani-Tabar et al., 2022).

The significance of the rule is mechanistic rather than merely empirical. Under mild assumptions in a linear model,

$a_l = W_l h_{l-1}, \qquad h_l = f(a_l),$5

so the eHebb term tends to push $a_l = W_l h_{l-1}, \qquad h_l = f(a_l),$6 toward the transpose of the fixed feedback matrix and thereby improve alignment of FA’s error directions with those of backpropagation. By contrast, Oja’s term has stable fixed points with $a_l = W_l h_{l-1}, \qquad h_l = f(a_l),$7 in compression layers, improving feature separability even when alignment angles change only slightly (Shervani-Tabar et al., 2022).

The reported empirical improvements were specific. On MNIST, FA only begins to learn after approximately 2000 iterations, whereas BP improves immediately. On EMNIST meta-training, FA stabilized around approximately 25% accuracy versus BP around 70% when both used meta-optimized learning rates. The meta-learned $a_l = W_l h_{l-1}, \qquad h_l = f(a_l),$8 rule outperformed DFA and approached BP-level accuracy by approximately 300 episodes, with loss curves following the same pattern. eHebb reduced alignment angles $a_l = W_l h_{l-1}, \qquad h_l = f(a_l),$9 across layers, especially in deeper layers, while Oja’s term reduced the orthonormality proxy

$$\delta_L = \partial L / \partial a_L, \qquad \delta_l = (W_{l+1}^\top \delta_{l+1}) \odot f'(a_l), \qquad \Delta W_l = -\eta \delta_l h_{l-1}^\top.$$0

across hidden layers (Shervani-Tabar et al., 2022). Mann-Whitney U tests yielded $$\delta_L = \partial L / \partial a_L, \qquad \delta_l = (W_{l+1}^\top \delta_{l+1}) \odot f'(a_l), \qquad \Delta W_l = -\eta \delta_l h_{l-1}^\top.$$1-values below 0.05 within fewer than 100 episodes when comparing FA with eHebb, Oja, $$\delta_L = \partial L / \partial a_L, \qquad \delta_l = (W_{l+1}^\top \delta_{l+1}) \odot f'(a_l), \qquad \Delta W_l = -\eta \delta_l h_{l-1}^\top.$$2, and full-pool rules (Shervani-Tabar et al., 2022).

5. Spiking, neuromorphic, and microcircuit realizations

RBP has also been reformulated for non-differentiable neurons and event-driven sensory processing. One line of work replaces the explicit derivative $$\delta_L = \partial L / \partial a_L, \qquad \delta_l = (W_{l+1}^\top \delta_{l+1}) \odot f'(a_l), \qquad \Delta W_l = -\eta \delta_l h_{l-1}^\top.$$3 by iterative temporal differencing (ITD) driven by fixed random feedback. In that construction, a network first computes a “free” forward pass, then receives top-down random feedback over several temporal relaxation steps, and finally updates weights by a local correlation between presynaptic activity and the temporal difference in postsynaptic activity:

$$\delta_L = \partial L / \partial a_L, \qquad \delta_l = (W_{l+1}^\top \delta_{l+1}) \odot f'(a_l), \qquad \Delta W_l = -\eta \delta_l h_{l-1}^\top.$$4

The motivation is that differentiability of the activation function is no longer required if the needed sensitivity is approximated by temporal differences between free and nudged states (Dargazany, 2019).

A second line implements event-driven Random Backpropagation (eRBP) in spiking neural networks. The hidden-layer update takes the three-factor form

$$\delta_L = \partial L / \partial a_L, \qquad \delta_l = (W_{l+1}^\top \delta_{l+1}) \odot f'(a_l), \qquad \Delta W_l = -\eta \delta_l h_{l-1}^\top.$$5

where $$\delta_L = \partial L / \partial a_L, \qquad \delta_l = (W_{l+1}^\top \delta_{l+1}) \odot f'(a_l), \qquad \Delta W_l = -\eta \delta_l h_{l-1}^\top.$$6 is a presynaptic spike, $$\delta_L = \partial L / \partial a_L, \qquad \delta_l = (W_{l+1}^\top \delta_{l+1}) \odot f'(a_l), \qquad \Delta W_l = -\eta \delta_l h_{l-1}^\top.$$7 is a boxcar surrogate gradient, and $$\delta_L = \partial L / \partial a_L, \qquad \delta_l = (W_{l+1}^\top \delta_{l+1}) \odot f'(a_l), \qquad \Delta W_l = -\eta \delta_l h_{l-1}^\top.$$8 is a feedback error delivered by fixed random synapses (Kaiser et al., 2019). In the DvsGesture benchmark, a dense four-layer SNN with two hidden layers of 200 neurons reached 92.7% accuracy with a motion-centered covert attention window and 86.1% without attention. The same rule was embedded in a robotic setup with a DVS mounted on a microsaccading robotic head, where the system classified visual affordances within 100 ms after microsaccade onset (Kaiser et al., 2019).

A third line attempts to realize sign-concordant random backpropagation directly in cortical-style microcircuits. The 2022 microcircuit framework used pyramidal neurons together with PV and SOM interneurons, with basal dendrites carrying feedforward activity and apical dendrites carrying error signals. Under local anti-Hebbian alignment of backward excitatory and inhibitory feedback and strict Dale’s principle, the hidden-layer error recursion becomes

$$\delta_L = \partial L / \partial a_L, \qquad \delta_l = (W_{l+1}^\top \delta_{l+1}) \odot f'(a_l), \qquad \Delta W_l = -\eta \delta_l h_{l-1}^\top.$$9

which is equivalent to sign-concordant feedback alignment under the paper’s assumptions (Yang et al., 2022). The reported best accuracies were 99.64% on MNIST and 86.88% on CIFAR-10, described as BP-comparable on MNIST and competitive on CIFAR-10 under the stated biological constraints (Yang et al., 2022).

Taken together, these results show that RBP has become a template rather than a fixed algorithm: fixed random feedback can be combined with surrogate gradients, temporal differencing, or microcircuit-level compartmentalization, provided the resulting rule preserves locality and an error-carrying feedback pathway (Dargazany, 2019, Yang et al., 2022).

6. Empirical profile, limitations, and relation to later local-learning frameworks

The original large-scale empirical profile of RBP was mixed rather than uniform. In the 2016 benchmarks on fully connected MNIST, BP reached 97.9% test accuracy, while RBP and SRBP each reached 97.2%, ARBP and ASRBP reached 97.3%, and “top layer only” reached 84.7%. On a convolutional CIFAR-10 model, BP reached 83.4%, whereas RBP reached 70.2%, SRBP 72.7%, and “top layer only” 47.9% (Baldi et al., 2016). Sparse feedback remained usable over a wide range of expected nonzeros per receiving neuron, and low-precision communication degraded gracefully: with 1-bit error signals on MNIST, BP reached 94.6%, RBP 89.8%, and SRBP 91.6% (Baldi et al., 2016).

Later work clarified that these results depend strongly on regime. RBP and DFA can learn effectively in shallow or moderately deep settings, but their performance degrades in deep networks, in online or low-data regimes, and when alignment angles become large in deeper layers (Shervani-Tabar et al., 2022). The theory in overparameterized two-layer models does not yet extend to full generality for deep networks, cross-entropy losses, unbounded activations such as ReLU, or stochastic mini-batch training (Song et al., 2021). The meta-learning work likewise notes that broader architectures such as convnets were not exhaustively explored, that softplus was used because ReLU non-differentiability would break meta-gradients, and that learning rules for plastic backward pathways remain an open direction (Shervani-Tabar et al., 2022).

The biological interpretation is also deliberately constrained. The microcircuit framework states that FA, DFA, and sign-concordant variants address the weight-transport problem but remain controversial because of other unresolved issues, and its own equivalence to sign-concordant FA depends on perfect local alignment of excitatory and inhibitory feedback and strict positivity of synapses under Dale’s principle (Yang et al., 2022). The meta-learned rule assumes that neurons can multiplex forward activations and backward error signals in different time windows, dendritic compartments, or burst-versus-single-spike codes; this is presented as a testable hypothesis rather than an established fact (Shervani-Tabar et al., 2022).

Subsequent block-local methods make RBP’s historical role explicit by treating it as a special case of a broader probabilistic framework. In Block-Local Learning, fixed random feedback with one block per layer recovers RBP/DFA-style updates, while learned feedback and local KL-based objectives substantially outperform FA on harder benchmarks. For example, on Fashion-MNIST with ResNet-18, BLL achieved top-1 accuracy 94.2 versus BP 92.7 versus FA 87.9; on CIFAR-10 with ResNet-50, BLL achieved 92.6 versus BP 94.0 versus FA 70.3 (Kappel et al., 2023). This suggests that later local-learning systems increasingly treat RBP not as a final algorithmic endpoint but as a foundational relaxation of weight transport from which richer local-learning frameworks can be derived (Kappel et al., 2023).

RBP therefore occupies a specific position in the learning-theory landscape. It shows that exact transpose transport is not necessary for useful credit assignment, but it does not establish that random fixed feedback is universally sufficient. The strongest available picture is conditional: random feedback can support convergence, sometimes near-BP accuracy, and increasingly sophisticated local rules; yet alignment, depth scaling, stochastic training behavior, and biological realizability remain active research questions (Song et al., 2021, Shervani-Tabar et al., 2022).