---
title: Random Contrastive Hebbian Learning (rCHL)
url: https://www.emergentmind.com/topics/random-contrastive-hebbian-learning-rchl
type: topic
---

# Random Contrastive Hebbian Learning (rCHL)

Random Contrastive Hebbian Learning (rCHL) is a variant of contrastive Hebbian learning that dispenses with biologically implausible weight symmetries by employing fixed random matrices for feedback during learning. In rCHL, synaptic weight updates occur via a contrastive, fully local Hebbian/anti-Hebbian rule using continuous-time neural dynamics, enabling effective credit assignment in multilayer feed-forward networks without requiring the transpose of feed-forward weights. rCHL has demonstrated competitive empirical performance on logic, classification, and autoencoding tasks, while enhancing biological plausibility for computational neuroscience modeling [1806.07406].

## 1. Network Structure and Training Phases

rCHL operates on a purely feed-forward architecture of $L$ layers, with each layer $k$ ($k=0,\ldots,L$) maintaining an activity vector $x_k \in \mathbb{R}^{n_k}$. Inputs $x_0$ and outputs $x_L$ are clamped during training as appropriate. The synaptic structure consists of:

- Feed-forward weights $W_k \in \mathbb{R}^{n_k\times n_{k-1}}$, $k=1 \ldots L$
- Fixed random feedback matrices $G_k \in \mathbb{R}^{n_{k-1}\times n_k}$, $k=2 \ldots L$
- Optional biases $b_k \in \mathbb{R}^{n_k}$

Each training example $(I, T)$ invokes two distinct phases:

1. **Free (positive) phase:** Input $x_0=I$ is clamped; $x_L$ is unconstrained.
2. **Clamped (negative) phase:** $x_0 = I$, output $x_L = T$ is clamped.

All hidden units exchange activity via bottom-up ($W_k$) and top-down ($G_{k+1}$) pathways in each phase.

## 2. Continuous-Time Neural Dynamics

At every hidden layer ($1 \leq k \leq L-1$), neural state dynamics in either phase are described by a first-order non-linear ordinary differential equation:

- **Free phase:** For $x̌_k(t)$,
  $$
  \frac{dx̌_k}{dt} = -x̌_k + f_k( W_k x̌_{k-1} + \gamma G_{k+1} x̌_{k+1} + b_k)
  $$
  with $x̌_0 = I$, $x̌_{L+1}=0$.
- **Clamped phase:** For $x̂_k(t)$,
  $$
  \frac{dx̂_k}{dt} = -x̂_k + f_k( W_k x̂_{k-1} + \gamma G_{k+1} x̂_{k+1} + b_k)
  $$
  with $x̂_0 = I$, $x̂_L = T$, $x̂_{L+1}=0$.

$f_k(\cdot)$ is a Lipschitz continuous nonlinearity (e.g., sigmoid). Both phases are integrated numerically (e.g., forward Euler) for a fixed time duration $t_f$ until a steady state is reached.

## 3. Hebbian-Contrastive Weight Update and Locality

After both phases reach equilibrium, synaptic updates are performed as follows:

$$
\Delta W_k = \eta \gamma^{k-L} ( x̂_k x̂_{k-1}^T - x̌_k x̌_{k-1}^T )
$$

with analogous updates for the biases if learnable. Here, $\eta > 0$ is the learning rate and $\gamma^{k-L}$ scales the magnitude across layers. This update is local (depends only on pre- and post-synaptic activity in the two phases) and requires no explicit gradient calculation, weight symmetry, or error backpropagation.

## 4. Random Feedback Matrices and Algorithm Implementation

Unlike classical contrastive Hebbian learning, which requires top-down feedback via transposed weights ($V_{k+1} = W_{k+1}^T$), rCHL replaces these with fixed random matrices $G_{k+1}$ drawn at initialization from a specified distribution (e.g., $Uniform(-a,a)$ or $Normal(0,\sigma^2)$). These random projections are held constant during all training epochs and propagate “error-like” signals from the output layer to the hidden layers.

A high-level pseudocode excerpt for rCHL includes:

- Initialization of weights $W_k$ and random feedback matrices $G_k$
- For each training epoch:
  1. Integrate network for free phase with $x_0 = I$
  2. Integrate for clamped phase with $x_L = T$
  3. Update $W_k$ and $b_k$ using the difference of outer products

This execution is asynchronous and event-driven, supporting continuous state evolution and local computations [1806.07406].

## 5. Parameter Sensitivity and Pseudospectra Analysis

rCHL’s convergence and stability depend on key hyperparameters:

- **Feedback gain $\gamma$:** Stable and rapid learning is achieved for $\gamma$ in the range $0.01$–$0.1$; larger values ($\gamma\approx 1$) induce instability due to the scaling effect in the weight updates.
- **Learning rate $\eta$:** Values $\lesssim 0.01$ produce slow or underfitted learning, while $\eta$ in $0.1$–$0.5$ accelerates convergence, though excessive $\eta$ may cause oscillations.
- **Random feedback matrix spectrum:** Wider support in $Uniform(-\ell,\ell)$ or higher $\sigma$ in $Normal(0, \sigma^2)$ increases convergence speed but can increase oscillatory behavior.
- **Pseudospectra of $G_k$:** Matrices with broader $\epsilon$-pseudospectra contours (i.e., minimal singular value contours further from the origin) yield faster and less oscillatory convergence; narrower, sub-Gaussian pseudospectra provide slower yet more robust learning.

This suggests that proper tuning of the feedback matrix distribution and hyperparameters is essential for optimal rCHL performance.

## 6. Empirical Benchmarks

Empirical evaluation of rCHL covers Boolean logic, image classification, and autoencoding:

| Task                          | Architecture          | Test MSE      | Accuracy   | Notes                                                      |
|-------------------------------|----------------------|---------------|------------|------------------------------------------------------------|
| Bars-and-Stripes (4x4)        | 16-50-2              | $\to 0$       | 100%       | $\approx$5,000 epochs, $\eta=0.1$, $\gamma=0.05$           |
| XOR                           | 2-2-1                | $1.1\times10^{-6}$ | 1.0        | Equal/better than BP/FDA/CHL speed                         |
| MNIST digits                  | 784-128-64-10        | $0.0044$      | 97.5%      | Comparable to CHL (97.7%), outperforms BP/FDA              |
| eMNIST letters                | 784-256-128-26       | $0.0073$      | 84.6%      | Slightly below CHL/BP/FDA (89–90%)                         |
| Autoencoder                   | 784-36-784           | –             | –          | rCHL codebook slightly noisier than CHL                    |

On the MNIST benchmark, rCHL effectively matches classical CHL and outperforms backpropagation (BP) and feedback alignment (FDA) in both MSE and accuracy. On eMNIST letters, rCHL lags by several percent compared to BP, CHL, and FDA, indicating sensitivity to the choice of feedback matrices and possible avenues for further optimization. For autoencoding, both CHL and rCHL achieve reasonable reconstruction, though rCHL encodings are marginally noisier [1806.07406].

## 7. Biological Plausibility and Computational Implications

rCHL’s design eliminates the biologically dubious requirement for bidirectional (symmetric) synapses by substituting fixed random feedback pathways, paralleling possible unknown or nonspecific feedback routes in biological cortex. The synaptic update rule is strictly local, utilizing only co-activation of pre/post-synaptic neurons in both free and clamped states, and is compatible with both rate-coded and spiking-neuron implementations via first-order ODE dynamics. The use of random feedback can be interpreted as a computational proxy for inter-laminar or higher-order feedback signals or background network activity. The strictly local computations and absence of global error signals make rCHL inherently suited to neuromorphic implementations, including spiking networks governed by STDP-like rules.

Overall, rCHL sustains the favorable convergence behavior of contrastive Hebbian learning and backpropagation while achieving a higher level of neurobiological fidelity. This renders rCHL a promising methodology for integrating learning theory and systems neuroscience [1806.07406].

Source: https://www.emergentmind.com/topics/random-contrastive-hebbian-learning-rchl