Bayesian Hebbian Plasticity in Neural Networks

• Bayesian Hebbian plasticity is a framework that combines Bayesian inference with Hebbian learning to update synaptic weights based on the statistical correlations between pre- and postsynaptic activity.
• It employs local, online estimation using leaky integrator dynamics that capture both short-term coincidence detection and long-term normalization, aligning weight updates with mutual information measures.
• The approach supports unsupervised representation learning with structural plasticity and sparse connectivity, yielding competitive benchmark performance and robustness to noise in neural network models.

Bayesian Hebbian plasticity denotes a class of synaptic learning rules that integrate Bayesian inference principles with Hebbian co-activation statistics to achieve online, local, and biologically plausible learning in neural networks. These mechanisms characterize synaptic change as the incremental update of log-probabilities or mutual information between presynaptic and postsynaptic activity, yielding weights that encode statistical dependencies as prescribed by Bayesian theory. The framework provides a normative justification for Hebbian and spike-timing-dependent plasticity (STDP) rules, realizing statistical learning through neural substrates and supporting robust unsupervised representation learning in both spiking and rate-based networks.

1. Bayesian-Hebbian Principle and Weight Dynamics

Bayesian Hebbian plasticity formalizes synaptic update rules underpinning co-activation-driven potentiation and depression by relating them to the statistical structure of neural activity. The canonical example is found in the Bayesian Confidence Propagation Neural Network (BCPNN), where each synapse $i \to j$ maintains estimates of presynaptic and postsynaptic activation probabilities and their co-activation (joint probability):

• $P_i(t)$: marginal probability of presynaptic activation
• $P_j(t)$: marginal probability of postsynaptic activation
• $P_{ij}(t)$: joint activation probability.

The synaptic weight is $W_{ij}(t) = \log \frac{P_{ij}(t)}{P_i(t) P_j(t)},$ representing the log-odds or pointwise mutual information between pre and postsynaptic units. The postsynaptic bias is simply $b_j(t) = \log P_j(t)$ (Ravichandran et al., 2023, Ravichandran et al., 2024).

The underlying Hebbian aspect arises because $P_{ij}(t)$ is updated via an online low-pass filter of the product of pre- and postsynaptic traces, precisely capturing coincidence detection:

$$\tau_z \frac{dZ_i}{dt} = -Z_i + S_i(t),\qquad \tau_z \frac{dZ_j}{dt} = -Z_j + S_j(t),$$

$\tau_p \frac{dP_{ij}}{dt} = -P_{ij} + Z_i Z_j,$

where $S_i(t) \in \{0,1\}$ denotes spike events (spiking models) or binary/rate-normalized activity (rate-based models).

2. STDP, Bernoulli Message Passing, and Gradient Alignment

A variant grounded in spike-timing-dependent plasticity (STDP) employs classical pair-based updates:

$P_i(t)$0

where potentiation and depression depend on the temporal order of spikes. This rule is shown to implement a stochastic gradient ascent on the log-likelihood of synaptic efficacy under Bayesian inference, aligning the expected update $P_i(t)$1 with the gradient of the log-posterior on the weights. When applied to modules encoding sum-product updates in factor graphs with Bernoulli random variables (e.g., AND, OR, XOR), this STDP rule enables local, event-driven learning of exact Bayesian message-passing relations. Parameter mappings exist between STDP hyperparameters and those controlling the Bayesian fixed point or integration window (Adamiat et al., 19 Dec 2025).

3. Local Online Estimation and Normalization

Key to Bayesian Hebbian plasticity is the online, local estimation of marginals and joint statistics from spike or rate traces using leaky integrator dynamics. These are organized in two-stage filtering:

• Short-term "z-traces" (for coincident detection on the order of tens of milliseconds)
• Long-term "p-traces" (for running averages over seconds)

Weights are continuously updated using $P_i(t)$2 where $P_i(t)$3 ensures numerical stability. This normalization ensures weight homeostasis and bounds, since increased firing raises the denominator, counteracting runaway potentiation. Divisive normalization is implicit in the multiplicative denominator structure, supporting robust scaling and avoidance of saturation (Ravichandran et al., 2024, Ravichandran et al., 2023).

4. SoftHebb, Winner-Take-All Circuits, and Cross-Entropy Minimization

The SoftHebb model extends Bayesian Hebbian plasticity to soft winner-take-all (WTA) circuits operating in rate-based networks. Here, observations $P_i(t)$4 are generated by $P_i(t)$5 latent causes, modeled as a mixture:

$P_i(t)$6

Parametric densities are chosen such that the neural softmax activation implements the posterior over causes:

$P_i(t)$7

The synaptic update aligns with the gradient of log-likelihood:

$P_i(t)$8

with $P_i(t)$9. This yields a strictly local, online learning rule minimizing the Kullback-Leibler divergence $P_j(t)$0 between the modeled and true data distributions. Cross-entropy minimization emerges as a consequence, offering a direct statistical interpretation of Hebbian learning in unsupervised regimes (Moraitis et al., 2021).

5. Structural Plasticity and Sparse Connectivity

Bayesian Hebbian learning frameworks often incorporate structural plasticity to maintain sparse and efficient representations, as observed in neocortical circuits. Connections are maintained in a binary connectivity mask $P_j(t)$1, and synaptic rewiring occurs at fixed intervals or pattern presentations. Local usage scores, typically proportional to $P_j(t)$2 or weighted combinations thereof, are used to prune the least informative synapses and replace them with candidates sampled from the larger potential pool. This process ensures that each neuron samples its most informative input subset while keeping overall connection density fixed (e.g., $P_j(t)$3), resulting in sparse and patchy receptive fields and enhancing scalability and neuromorphic viability (Ravichandran et al., 2023, Ravichandran et al., 2024).

6. Empirical Results and Functional Properties

Empirical studies using Bayesian Hebbian plasticity, particularly the spiking and feedforward BCPNN implementations, demonstrate:

• Competitive unsupervised representation learning on MNIST and F-MNIST benchmarks, with spiking BCPNN achieving $P_j(t)$4 on MNIST and $P_j(t)$5 on F-MNIST (vs. $P_j(t)$6 and $P_j(t)$7 for non-spiking BCPNN, respectively); performance comparable to STDP-based spiking networks (Ravichandran et al., 2023).
• Robustness to noise and adversarial attacks in the SoftHebb model, exceeding that of standard backpropagation-trained MLPs under strong perturbations (Moraitis et al., 2021).
• Exact realization of sum-product belief propagation in Bernoulli factor graphs by SNN modules with STDP, with root mean square errors on message probabilities on the order of $P_j(t)$8 and marginal posteriors within $P_j(t)$9–$P_{ij}(t)$0 of analytic solutions (Adamiat et al., 19 Dec 2025).
• Capacity for generative modeling and smooth interpolation of object classes, suggesting a link between biological plasticity and sample-efficient, robust, and interpretable learning (Moraitis et al., 2021).

7. Significance, Context, and Theoretical Impact

Bayesian Hebbian plasticity constitutes a rigorous synthesis of statistical learning theory and biophysically plausible neural plasticity, operationalized in both spiking and rate-based architectures. This establishes a normative (ML-grounded) role for local, unsupervised Hebbian rules—solving weight transport and update-locking challenges inherent in backpropagation-based learning. The framework's ability to unify Bayesian inference, homeostatic normalization, synaptic and structural plasticity, and robust unsupervised learning makes it foundational for neuromorphic computation and the scalable synthesis of brain-like systems. Theoretical analyses, such as in SoftHebb, clarify why these rules offer cross-entropy minimization and Kullback-Leibler optimality without explicit supervision, and empirical studies consistently confirm their competitive and robust learning dynamics (Moraitis et al., 2021, Ravichandran et al., 2024, Ravichandran et al., 2023, Adamiat et al., 19 Dec 2025).

---

Key References:

• Spiking BCPNN unsupervised learning: (Ravichandran et al., 2023)
• Message-passing with STDP: (Adamiat et al., 19 Dec 2025)
• SoftHebb WTA Bayesian learning: (Moraitis et al., 2021)
• Feedforward BCPNN, structural plasticity: (Ravichandran et al., 2024)