Random Contrastive Hebbian Learning (rCHL)

• Random Contrastive Hebbian Learning (rCHL) is a variant of contrastive Hebbian learning that eliminates weight symmetry by using fixed random feedback matrices.
• It employs a fully local Hebbian/anti-Hebbian update rule with continuous-time dynamics to perform credit assignment without explicit error backpropagation.
• Empirical benchmarks on logic, classification, and autoencoding tasks demonstrate that rCHL achieves competitive performance while enhancing biological plausibility.

Random Contrastive Hebbian Learning (rCHL) is a variant of contrastive Hebbian learning that dispenses with biologically implausible weight symmetries by employing fixed random matrices for feedback during learning. In rCHL, synaptic weight updates occur via a contrastive, fully local Hebbian/anti-Hebbian rule using continuous-time neural dynamics, enabling effective credit assignment in multilayer feed-forward networks without requiring the transpose of feed-forward weights. rCHL has demonstrated competitive empirical performance on logic, classification, and autoencoding tasks, while enhancing biological plausibility for computational neuroscience modeling (Detorakis et al., 2018).

1. Network Structure and Training Phases

rCHL operates on a purely feed-forward architecture of $L$ layers, with each layer $k$ ($k=0,\ldots,L$) maintaining an activity vector $x_k \in \mathbb{R}^{n_k}$. Inputs $x_0$ and outputs $x_L$ are clamped during training as appropriate. The synaptic structure consists of:

• Feed-forward weights $W_k \in \mathbb{R}^{n_k\times n_{k-1}}$, $k=1 \ldots L$
• Fixed random feedback matrices $G_k \in \mathbb{R}^{n_{k-1}\times n_k}$, $k=2 \ldots L$
• Optional biases $k$0

Each training example $k$1 invokes two distinct phases:

1. Free (positive) phase: Input $k$2 is clamped; $k$3 is unconstrained.
2. Clamped (negative) phase: $k$4, output $k$5 is clamped.

All hidden units exchange activity via bottom-up ($k$6) and top-down ($k$7) pathways in each phase.

2. Continuous-Time Neural Dynamics

At every hidden layer ($k$8), neural state dynamics in either phase are described by a first-order non-linear ordinary differential equation:

• Free phase: For $k$9,

$k=0,\ldots,L$0

with $k=0,\ldots,L$1, $k=0,\ldots,L$2.

• Clamped phase: For $k=0,\ldots,L$3,

$k=0,\ldots,L$4

with $k=0,\ldots,L$5, $k=0,\ldots,L$6, $k=0,\ldots,L$7.

$k=0,\ldots,L$8 is a Lipschitz continuous nonlinearity (e.g., sigmoid). Both phases are integrated numerically (e.g., forward Euler) for a fixed time duration $k=0,\ldots,L$9 until a steady state is reached.

3. Hebbian-Contrastive Weight Update and Locality

After both phases reach equilibrium, synaptic updates are performed as follows:

$x_k \in \mathbb{R}^{n_k}$0

with analogous updates for the biases if learnable. Here, $x_k \in \mathbb{R}^{n_k}$1 is the learning rate and $x_k \in \mathbb{R}^{n_k}$2 scales the magnitude across layers. This update is local (depends only on pre- and post-synaptic activity in the two phases) and requires no explicit gradient calculation, weight symmetry, or error backpropagation.

4. Random Feedback Matrices and Algorithm Implementation

Unlike classical contrastive Hebbian learning, which requires top-down feedback via transposed weights ($x_k \in \mathbb{R}^{n_k}$3), rCHL replaces these with fixed random matrices $x_k \in \mathbb{R}^{n_k}$4 drawn at initialization from a specified distribution (e.g., $x_k \in \mathbb{R}^{n_k}$5 or $x_k \in \mathbb{R}^{n_k}$6). These random projections are held constant during all training epochs and propagate “error-like” signals from the output layer to the hidden layers.

A high-level pseudocode excerpt for rCHL includes:

• Initialization of weights $x_k \in \mathbb{R}^{n_k}$7 and random feedback matrices $x_k \in \mathbb{R}^{n_k}$8
• For each training epoch:
  1. Integrate network for free phase with $x_k \in \mathbb{R}^{n_k}$9
  2. Integrate for clamped phase with $x_0$0
  3. Update $x_0$1 and $x_0$2 using the difference of outer products

This execution is asynchronous and event-driven, supporting continuous state evolution and local computations (Detorakis et al., 2018).

5. Parameter Sensitivity and Pseudospectra Analysis

rCHL’s convergence and stability depend on key hyperparameters:

• Feedback gain $x_0$3: Stable and rapid learning is achieved for $x_0$4 in the range $x_0$5–$x_0$6; larger values ($x_0$7) induce instability due to the scaling effect in the weight updates.

• Learning rate $x_0$8: Values $x_0$9 produce slow or underfitted learning, while $x_L$0 in $x_L$1–$x_L$2 accelerates convergence, though excessive $x_L$3 may cause oscillations.
• Random feedback matrix spectrum: Wider support in $x_L$4 or higher $x_L$5 in $x_L$6 increases convergence speed but can increase oscillatory behavior.
• Pseudospectra of $x_L$7: Matrices with broader $x_L$8-pseudospectra contours (i.e., minimal singular value contours further from the origin) yield faster and less oscillatory convergence; narrower, sub-Gaussian pseudospectra provide slower yet more robust learning.

This suggests that proper tuning of the feedback matrix distribution and hyperparameters is essential for optimal rCHL performance.

6. Empirical Benchmarks

Empirical evaluation of rCHL covers Boolean logic, image classification, and autoencoding:

Task	Architecture	Test MSE	Accuracy	Notes
Bars-and-Stripes (4x4)	16-50-2	$x_L$9	100%	$W_k \in \mathbb{R}^{n_k\times n_{k-1}}$05,000 epochs, $W_k \in \mathbb{R}^{n_k\times n_{k-1}}$1, $W_k \in \mathbb{R}^{n_k\times n_{k-1}}$2
XOR	2-2-1	$W_k \in \mathbb{R}^{n_k\times n_{k-1}}$3	1.0	Equal/better than BP/FDA/CHL speed
MNIST digits	784-128-64-10	$W_k \in \mathbb{R}^{n_k\times n_{k-1}}$4	97.5%	Comparable to CHL (97.7%), outperforms BP/FDA
eMNIST letters	784-256-128-26	$W_k \in \mathbb{R}^{n_k\times n_{k-1}}$5	84.6%	Slightly below CHL/BP/FDA (89–90%)
Autoencoder	784-36-784	–	–	rCHL codebook slightly noisier than CHL

On the MNIST benchmark, rCHL effectively matches classical CHL and outperforms backpropagation (BP) and feedback alignment (FDA) in both MSE and accuracy. On eMNIST letters, rCHL lags by several percent compared to BP, CHL, and FDA, indicating sensitivity to the choice of feedback matrices and possible avenues for further optimization. For autoencoding, both CHL and rCHL achieve reasonable reconstruction, though rCHL encodings are marginally noisier (Detorakis et al., 2018).

7. Biological Plausibility and Computational Implications

rCHL’s design eliminates the biologically dubious requirement for bidirectional (symmetric) synapses by substituting fixed random feedback pathways, paralleling possible unknown or nonspecific feedback routes in biological cortex. The synaptic update rule is strictly local, utilizing only co-activation of pre/post-synaptic neurons in both free and clamped states, and is compatible with both rate-coded and spiking-neuron implementations via first-order ODE dynamics. The use of random feedback can be interpreted as a computational proxy for inter-laminar or higher-order feedback signals or background network activity. The strictly local computations and absence of global error signals make rCHL inherently suited to neuromorphic implementations, including spiking networks governed by STDP-like rules.

Overall, rCHL sustains the favorable convergence behavior of contrastive Hebbian learning and backpropagation while achieving a higher level of neurobiological fidelity. This renders rCHL a promising methodology for integrating learning theory and systems neuroscience (Detorakis et al., 2018).