---
title: 'Random Backpropagation: Theory & Variants'
url: https://www.emergentmind.com/topics/random-backpropagation-rbp
type: topic
---

# Random Backpropagation: Theory & Variants

Random Backpropagation (RBP), often identified with Feedback Alignment (FA), is a family of supervised credit-assignment algorithms in which the transpose of the forward weight matrices used by standard backpropagation is replaced by fixed random feedback matrices, thereby removing the weight-transport requirement while preserving local weight updates driven by presynaptic activity and a postsynaptic error-like signal [1612.02734][2210.16414]. Within the broader “deep learning channel” framework, RBP treats deep learning as the problem of communicating non-local target information to deep synapses through a separate feedback pathway rather than through exact reuse of the forward pathway [1712.08608]. It has consequently become a central reference point in research on biologically plausible learning, with extensions to direct-feedback, adaptive-feedback, sparse, sign-concordant, spiking, neuromorphic, and meta-learned local-plasticity settings [2205.07292][1904.04805].

## 1. Conceptual basis and the weight-transport problem

Standard backpropagation computes hidden-layer error signals by explicitly reusing downstream forward weights. For a feedforward network with layers $l = 1,\dots,L$, forward weights $W_l$, pre-activations $a_l$, activations $h_l$, and loss $L(h_L,y)$, the forward dynamics are
$$
a_l = W_l h_{l-1}, \qquad h_l = f(a_l),
$$
and standard backpropagation uses
$$
\delta_L = \partial L / \partial a_L, \qquad
\delta_l = (W_{l+1}^\top \delta_{l+1}) \odot f'(a_l), \qquad
\Delta W_l = -\eta \delta_l h_{l-1}^\top.
$$
These equations imply that hidden-layer updates require exact access to $W_{l+1}^\top$, which is the weight-transport problem [2210.16414].

RBP replaces the transposed forward matrix with a fixed random matrix $B_{l+1}$ sampled at initialization and then held constant:
$$
\delta_L = \partial L / \partial a_L, \qquad
\delta_l = (B_{l+1}\delta_{l+1}) \odot f'(a_l), \qquad
\Delta W_l = -\eta \delta_l h_{l-1}^\top.
$$
In this sense, RBP preserves the local product structure of gradient descent while discarding exact backward symmetry [2210.16414]. The same basic idea is presented in the “deep learning channel” language as a way to communicate non-local target information to deep synapses under locality constraints in space and time [1612.02734][1712.08608].

This formulation is closely tied to the claim that deep local learning alone is insufficient for complex tasks. In the 2016 analysis, local synaptic rules of the canonical form
$$
\Delta w_{ij}^h = F(O_i^h, O_j^{h-1}, w_{ij}^h)
$$
for hidden layers and
$$
\Delta w_{ij}^L = F(T_i, O_i^L, O_j^{L-1}, w_{ij}^L)
$$
at the output layer do not, by themselves, solve the critical equations of deep supervised learning; some channel must deliver error-related information to deep weights [1612.02734]. RBP’s distinct contribution is the claim that this channel need not carry exact transposes of forward weights.

## 2. Algorithmic forms and main variants

RBP is best understood as a family rather than a single recursion. The core variants differ primarily in how the feedback signal is routed and whether the backward pathway is fixed or adaptive [1612.02734][1712.08608].

| Variant | Backward signal | Distinctive property |
|---|---|---|
| RBP / FA | $\delta_l = (B_{l+1}\delta_{l+1}) \odot f'_l(h_l)$ | Layerwise fixed random feedback |
| SRBP / DFA-style direct feedback | $\delta_l = (B_l \delta_L) \odot f'_l(h_l)$ or $\delta_l = (B_l \nabla_{h_L}L)\odot f'(a_l)$ | Direct output-to-hidden error broadcast |
| ARBP / ASRBP | Feedback matrices adapted locally | Same local rules can be used in the learning channel |
| SCFA | ${\rm sign}(B) = {\rm sign}(W^\top)$ | Sign-concordant feedback improves alignment |

Skipped Random Backpropagation (SRBP) sends output-layer error directly to each hidden layer via a fixed random matrix, bypassing staged layer-by-layer propagation. In the 2016 treatment, SRBP addresses architectural and derivative-symmetry issues more aggressively than staged RBP because each hidden layer needs only the local derivative at the receiving neuron, not the transport of derivatives from higher layers [1612.02734]. Direct Feedback Alignment (DFA) in the later terminology is the closely related rule
$$
\delta_l = (B_l \nabla_{h_L}L)\odot f'(a_l),
$$
with $B_l$ mapping output errors into layer-$l$ coordinates [2210.16414].

Adaptive variants update the learning-channel weights locally rather than freezing them. In the “symmetries of the deep learning channel” formulation, a feedback weight $c_{rs}^l$ can be updated by
$$
\Delta c_{rs}^l = \eta R_s^{l+1} O_r^l,
$$
and matrix forms such as $\Delta C_l = \eta O_l (R_{l+1})^\top$ are used to define ARBP [1712.08608]. The 2016 paper also introduced sparse feedback and combinations such as ASRBP, showing that random feedback does not need to be dense to remain effective [1612.02734].

A persistent empirical point across these variants is the importance of multiplying the feedback signal by the local activation derivative. In the 2016 simulations, removing $f'(\cdot)$ reduced MNIST test accuracy from 97.9% to 89.9% for BP, from 97.2% to 88.3% for RBP, and from 97.2% to 88.4% for SRBP; on CIFAR-10 the corresponding no-$f'$ results were 54.8% for BP, 32.7% for RBP, and 39.9% for SRBP [1612.02734]. The later symmetry analysis likewise concluded that omitting all derivatives does not work, whereas using the same nonlinear neurons in both forward and learning channels can work provided the nonlinearities are not purely non-negative [1712.08608].

## 3. Convergence, alignment, and symmetry properties

The main modern mathematical analysis of RBP focuses on two-layer networks in the overparameterized regime. For
$$
f(x)=\frac{1}{\sqrt{p}}\sum_{r=1}^p \beta_r \psi(w_r^\top x),
$$
with squared error loss, standard backprop updates the hidden weights using the forward coefficient $\beta_r$, while FA/RBP replaces it with a fixed random feedback weight $b_r$:
$$
w_r \leftarrow w_r - \eta \frac{1}{\sqrt{p}}\sum_{i=1}^n e_i\, b_r\, \psi'(w_r^\top x_i) x_i.
$$
Under bounded smooth activation, Assumption $G$, bounded data, Gaussian initialization, and sufficiently large width $p$, the residual error contracts exponentially:
$$
\|e(t+1)\| \leq \left(1-\frac{\eta\gamma}{4}\right)\|e(t)\|.
$$
The same analysis shows that the weights remain close to initialization, which is the basis for the NTK-style argument used in the proof [2106.06044].

The central theoretical controversy concerns alignment. In scalar-output regression, the alignment measure is
$$
\cos(\angle(b,\beta(t)))=\frac{\langle b,\beta(t)\rangle}{\|b\|\,\|\beta(t)\|}.
$$
Without regularization, the theory gives
$$
\cos(\angle(b,\beta(t))) = O(n/\sqrt{p}),
$$
so alignment vanishes as width grows. With L2 regularization on $\beta$, however, the linear-case theory proves that a schedule $\lambda(t)=\lambda$ for $t\leq T$ and $\lambda(t)=0$ for $t>T$ can induce positive asymptotic alignment, with
$$
\cos(\angle(b,\beta(t))) \ge c > 0
$$
for sufficiently large $t$, where $c$ does not depend on $n$, $d$, $p$, or $\eta$ [2106.06044]. This establishes that convergence of training error and alignment of forward and feedback pathways are distinct phenomena: the former can occur in wide networks without the latter.

Earlier analytical work approached the same question through continuous-time polynomial ODEs. For linear chains such as $A[1,1,1]$, $A[1,1,1,1]$, arbitrary-depth chains $A[1,\dots,1]$, expansive $A[1,N,1]$, and compressive $A[N,1,N]$, RBP, SRBP, and adaptive variants were shown to converge to fixed points corresponding to global minima of quadratic error under broad initial conditions [1612.02734]. The 2017 symmetry analysis further proved, in general linear adaptive cases, identities such as
$$
C_i = A_{i+1}^\top + K_i,
$$
showing that backward weights can track the transpose of forward weights up to an initialization-dependent offset, with small initialization producing “essentially symmetric” channels during learning [1712.08608].

These results collectively restrict a common misconception. RBP is not simply “random backpropagation that mysteriously works”; rather, the published theory supports exponential error decay in wide two-layer settings, while also showing that alignment is conditional and may require regularization or adaptive structure [2106.06044].

## 4. Meta-learned local plasticity rules with random feedback

A major recent development is the use of meta-learning to improve RBP under biologically constrained online learning. In the 2022 study on fixed random feedback pathways, the objective was not to meta-learn initial weights but to meta-learn a sparse, interpretable local plasticity rule shared across all layers, with networks reinitialized each episode to isolate the effect of the rule itself [2210.16414].

The inner-loop protocol was deliberately restrictive: forward weights and feedback matrices were Xavier-initialized at the start of each episode, feedback matrices were fixed random matrices, hidden units used softplus with $\beta = 10$, the output used softmax, training was online with batch size 1, and each episode contained only 250 training examples. The architecture used for the principal experiments was a 5-layer fully connected network with dimensions 784-170-130-100-70-47, and the outer loop used 600 episodes of ADAM with meta-learning rate $10^{-3}$ and an L1 penalty on the plasticity coefficients [2210.16414].

The discovered rule was
$$
F^{bio}(\Theta) =
- \theta_0 e_l h_{l-1}^\top
- \theta_2 e_l e_{l-1}^\top
+ \theta_9 \big(h_l h_{l-1}^\top - (h_l h_l^\top)W_l\big),
$$
with $\Delta W_l = F^{bio}(\Theta)$. Here the first term is the FA/DFA-like pseudo-gradient, the second is an error-based Hebbian component (“eHebb”), and the third is Oja’s rule [2210.16414].

The significance of the rule is mechanistic rather than merely empirical. Under mild assumptions in a linear model,
$$
E[e_l e_{l-1}^\top \mid B_{l+1,l}] = \sigma_l^2 B_{l+1,l}^\top,
$$
so the eHebb term tends to push $W_l$ toward the transpose of the fixed feedback matrix and thereby improve alignment of FA’s error directions with those of backpropagation. By contrast, Oja’s term has stable fixed points with $W_l W_l^\top \approx I$ in compression layers, improving feature separability even when alignment angles change only slightly [2210.16414].

The reported empirical improvements were specific. On MNIST, FA only begins to learn after approximately 2000 iterations, whereas BP improves immediately. On EMNIST meta-training, FA stabilized around approximately 25% accuracy versus BP around 70% when both used meta-optimized learning rates. The meta-learned $F^{bio}$ rule outperformed DFA and approached BP-level accuracy by approximately 300 episodes, with loss curves following the same pattern. eHebb reduced alignment angles $\alpha_l$ across layers, especially in deeper layers, while Oja’s term reduced the orthonormality proxy
$$
E_W = \|z_l - W_l W_l^\top \sigma(z_l)\|_2^2
$$
across hidden layers [2210.16414]. Mann-Whitney U tests yielded $p$-values below 0.05 within fewer than 100 episodes when comparing FA with eHebb, Oja, $F^{bio}$, and full-pool rules [2210.16414].

## 5. Spiking, neuromorphic, and microcircuit realizations

RBP has also been reformulated for non-differentiable neurons and event-driven sensory processing. One line of work replaces the explicit derivative $\phi'(z_l)$ by iterative temporal differencing (ITD) driven by fixed random feedback. In that construction, a network first computes a “free” forward pass, then receives top-down random feedback over several temporal relaxation steps, and finally updates weights by a local correlation between presynaptic activity and the temporal difference in postsynaptic activity:
$$
\Delta W_l = -\eta \Delta a_l\, a_{l-1}^{(-)T}.
$$
The motivation is that differentiability of the activation function is no longer required if the needed sensitivity is approximated by temporal differences between free and nudged states [1907.07255].

A second line implements event-driven Random Backpropagation (eRBP) in spiking neural networks. The hidden-layer update takes the three-factor form
$$
\Delta w_{ij}(t)=\eta\, s_j(t)\, g(v_i(t))\, \delta_i^{fb}(t), \qquad
\delta_i^{fb}(t)=\sum_k B_{ik} e_k(t),
$$
where $s_j(t)$ is a presynaptic spike, $g(v_i(t))$ is a boxcar surrogate gradient, and $\delta_i^{fb}(t)$ is a feedback error delivered by fixed random synapses [1904.04805]. In the DvsGesture benchmark, a dense four-layer SNN with two hidden layers of 200 neurons reached 92.7% accuracy with a motion-centered covert attention window and 86.1% without attention. The same rule was embedded in a robotic setup with a DVS mounted on a microsaccading robotic head, where the system classified visual affordances within 100 ms after microsaccade onset [1904.04805].

A third line attempts to realize sign-concordant random backpropagation directly in cortical-style microcircuits. The 2022 microcircuit framework used pyramidal neurons together with PV and SOM interneurons, with basal dendrites carrying feedforward activity and apical dendrites carrying error signals. Under local anti-Hebbian alignment of backward excitatory and inhibitory feedback and strict Dale’s principle, the hidden-layer error recursion becomes
$$
\boldsymbol{\delta}^{(l)}(t) \approx \sigma'(\boldsymbol{u}^{(l)}(t))\, \boldsymbol{B}^{(l+1)}\, \boldsymbol{\delta}^{(l+1)}(t),
\qquad
{\rm sign}(\boldsymbol{B}^{(l+1)}) = {\rm sign}((\boldsymbol{W}^{(l+1)})^\top),
$$
which is equivalent to sign-concordant feedback alignment under the paper’s assumptions [2205.07292]. The reported best accuracies were 99.64% on MNIST and 86.88% on CIFAR-10, described as BP-comparable on MNIST and competitive on CIFAR-10 under the stated biological constraints [2205.07292].

Taken together, these results show that RBP has become a template rather than a fixed algorithm: fixed random feedback can be combined with surrogate gradients, temporal differencing, or microcircuit-level compartmentalization, provided the resulting rule preserves locality and an error-carrying feedback pathway [1907.07255][2205.07292].

## 6. Empirical profile, limitations, and relation to later local-learning frameworks

The original large-scale empirical profile of RBP was mixed rather than uniform. In the 2016 benchmarks on fully connected MNIST, BP reached 97.9% test accuracy, while RBP and SRBP each reached 97.2%, ARBP and ASRBP reached 97.3%, and “top layer only” reached 84.7%. On a convolutional CIFAR-10 model, BP reached 83.4%, whereas RBP reached 70.2%, SRBP 72.7%, and “top layer only” 47.9% [1612.02734]. Sparse feedback remained usable over a wide range of expected nonzeros per receiving neuron, and low-precision communication degraded gracefully: with 1-bit error signals on MNIST, BP reached 94.6%, RBP 89.8%, and SRBP 91.6% [1612.02734].

Later work clarified that these results depend strongly on regime. RBP and DFA can learn effectively in shallow or moderately deep settings, but their performance degrades in deep networks, in online or low-data regimes, and when alignment angles become large in deeper layers [2210.16414]. The theory in overparameterized two-layer models does not yet extend to full generality for deep networks, cross-entropy losses, unbounded activations such as ReLU, or stochastic mini-batch training [2106.06044]. The meta-learning work likewise notes that broader architectures such as convnets were not exhaustively explored, that softplus was used because ReLU non-differentiability would break meta-gradients, and that learning rules for plastic backward pathways remain an open direction [2210.16414].

The biological interpretation is also deliberately constrained. The microcircuit framework states that FA, DFA, and sign-concordant variants address the weight-transport problem but remain controversial because of other unresolved issues, and its own equivalence to sign-concordant FA depends on perfect local alignment of excitatory and inhibitory feedback and strict positivity of synapses under Dale’s principle [2205.07292]. The meta-learned rule assumes that neurons can multiplex forward activations and backward error signals in different time windows, dendritic compartments, or burst-versus-single-spike codes; this is presented as a testable hypothesis rather than an established fact [2210.16414].

Subsequent block-local methods make RBP’s historical role explicit by treating it as a special case of a broader probabilistic framework. In Block-Local Learning, fixed random feedback with one block per layer recovers RBP/DFA-style updates, while learned feedback and local KL-based objectives substantially outperform FA on harder benchmarks. For example, on Fashion-MNIST with ResNet-18, BLL achieved top-1 accuracy 94.2 versus BP 92.7 versus FA 87.9; on CIFAR-10 with ResNet-50, BLL achieved 92.6 versus BP 94.0 versus FA 70.3 [2305.14974]. This suggests that later local-learning systems increasingly treat RBP not as a final algorithmic endpoint but as a foundational relaxation of weight transport from which richer local-learning frameworks can be derived [2305.14974].

RBP therefore occupies a specific position in the learning-theory landscape. It shows that exact transpose transport is not necessary for useful credit assignment, but it does not establish that random fixed feedback is universally sufficient. The strongest available picture is conditional: random feedback can support convergence, sometimes near-BP accuracy, and increasingly sophisticated local rules; yet alignment, depth scaling, stochastic training behavior, and biological realizability remain active research questions [2106.06044][2210.16414].

Source: https://www.emergentmind.com/topics/random-backpropagation-rbp