Synaptic Sampling in Neural Computation

• Synaptic sampling is a computational framework where synapses perform Monte Carlo sampling using stochastic neurotransmitter release to represent probabilistic information.
• It integrates Bernoulli-based synaptic dropout, local learning rules, and population coding to implement Bayesian inference and capture both epistemic and aleatoric uncertainties.
• This approach underpins robust, energy-efficient neural computation and has inspired neuromorphic implementations that tolerate synaptic pruning and device variability.

Synaptic sampling is a theoretical and computational framework positing that biological synapses, through their inherent stochasticity and unreliability, implement Monte Carlo sampling over possible neural and synaptic states. This mechanism enables neural circuits to represent, propagate, and act upon probabilistic information in an online, energy-efficient, and biologically plausible manner. Synaptic sampling provides an explanatory link between experimentally observed synaptic variability, local learning dynamics, and the brain’s capacity for uncertainty estimation and flexible probabilistic computation (Neftci et al., 2015, McKee et al., 2021, Aitchison et al., 2015, McKee et al., 2022).

1. Synaptic Stochasticity and the Sampling Paradigm

Biological synapses frequently exhibit stochastic transmitter release, with synaptic vesicle fusion occurring with probability $p\in[0,1]$ per presynaptic spike. Empirical measurements report vesicle release probabilities in cortex and hippocampus typically between 10–50%, often lower than 20%, and substantial trial-to-trial variability even under fixed stimulation (McKee et al., 2022). Synaptic sampling interprets this unreliability not as noise to be averaged out, but as a functional resource exploited for probabilistic computation.

Under this view, each presynaptic spike initiates a Bernoulli trial at each synapse: with probability $p$, a postsynaptic response (PSP) is generated. The actual amplitude of the PSP can itself be treated as a random variable drawn from a synapse-specific posterior over synaptic efficacy, $p(w\mid\mathcal{D})$, where $\mathcal{D}$ denotes the history of observed activity (Aitchison et al., 2015). Thus, across both structural (epistemic) and output (aleatoric) levels, the network evolves under multiplicative noise, naturally providing samples from distributions over weights, outputs, and decisions (Neftci et al., 2015).

2. Theoretical Formalizations: Energy Models and Population Codes

Synaptic sampling is instantiated in multiple formal models. In the Synaptic Sampling Machine (SSM), randomness is introduced exclusively at the synapses via a multiplicative “blank-out mask” $m_{ij}(t)\sim \mathrm{Bernoulli}(p)$ over connection $w_{ij}$, so $w_{ij}\to w_{ij} m_{ij}(t)$. Neural units themselves can be deterministic threshold or leaky integrate-and-fire (LIF) neurons. The joint network state across neurons and synaptic masks is described by an energy function (similar to Hopfield or Boltzmann machines):

$$E(s;w,b) = -\frac12\sum_{i,j} w_{ij}s_is_j - \sum_i b_i s_i,$$

with the probability of a particular neuron state $s$ proportional to a mixture over all synaptic mask configurations: $$P(s\mid w,b) \propto \sum_{m} P(m)\exp(-E(s;w\odot m, b))$$ (Neftci et al., 2015).

In population code models, the mapping from encoded variables $x$ (input) to $y$ (output) across population-tuned neurons is represented as a conditional distribution $P(y|x,W)$. The competition among output neurons, typically via lateral inhibition or winner-take-all (WTA) dynamics, turns the stochastic synaptic inputs into samples from target probability distributions, with synaptic transmission probabilities tuned to encode arbitrary learned posteriors (McKee et al., 2021). For linear population codes, the “subset-max” synaptic dropout mapping ensures that each output neuron $i$ is selected with probability

$p_i = \frac{w_i}{\sum_j w_j}$

achieved by setting dropout probabilities as

$q_i = \frac{w_i}{\sum_{j=i}^n w_j}$

for ordered weights (McKee et al., 2021).

3. Bayesian Inference, Uncertainty, and Learning Rules

Synaptic sampling unifies representations of both epistemic (parametric/model) and aleatoric (residual/output) uncertainties.

• Epistemic uncertainty is captured by stochastic sampling of synaptic efficacies from their learned posteriors $P(W|\text{data})$, reflecting uncertainty due to limited experience.
• Aleatoric uncertainty arises from noise and uncertainty in the inputs or outputs even when synaptic weights are fixed; realized as variability in network output for a given input.

To achieve complete Bayesian inference, synaptic dropout masks can combine two factors: (1) an epistemic term $\phi_i$, by matching the variance of a Bernoulli-masked weight to a posterior (e.g., Beta or Dirichlet), and (2) an aleatoric term $Q_i$ implementing the sampling of the output distribution conditional on weights. The combined mask probability $\bar{Q}_i = \phi_i \cdot Q_i$ yields correct sampling from the full posterior predictive (McKee et al., 2021, McKee et al., 2022).

Learning rules for synaptic release probabilities must be biologically plausible, local, and consistent with online adaptation. For the “subset-max” dropout, a synapse $i$ can update its probability after each trial via:

$$\hat{q}_{i,t} = \hat{q}_{i, t-1} + \gamma (q^*_{i,t} - \hat{q}_{i,t-1}),$$

with $q^*_{i,t} = w_i / \sum_{j\in S_t} w_j$, where $S_t$ is the set of surviving synapses in that trial, and only the synapse associated with the “winning” output is updated (McKee et al., 2021). For policy-gradient-based learning in reinforcement tasks, synaptic weights undergo stochastic differential equation (Langevin) updates in parameter space, e.g.,

$$d\theta_i = \beta[c_p(\mu-\theta_i) + c_g g_i]dt + \sqrt{2\beta T} dW_i$$

where $g_i$ is a local estimate of the reward gradient (Kaiser et al., 2020).

4. Biological and Neuromorphic Implementations

The biological substrate for synaptic sampling relies on documented probabilistic vesicle release, modulation of release probabilities via plasticity and neuromodulation, and the prevalence of lateral inhibition and population coding in cortical circuits (Neftci et al., 2015, McKee et al., 2022). Synaptic sampling mechanisms only require locally accessible signals: pre- and post-synaptic activity, local counts or eligibility traces, and local variables tracking transmission probability.

Neuromorphic realizations of synaptic sampling include memristor-based Synaptic Sampling Machines, wherein each synaptic crosspoint implements a local, hardware Bernoulli sampler (Synaptic Sampling Cell, SSC) that stochastically gates analog synaptic conductances (Dolzhikova et al., 2018). This permits high-speed, low-power sampling on-chip, with parallel sampling rates up to tens of MHz and energy budgets on the order of 10–50 pJ per sample. These physical instantiations of synaptic randomness enable massive parallelism, energy and area efficiency, and are well-matched to large spiking architectures for edge sensing and adaptive control (Dolzhikova et al., 2018, Yepes et al., 2016).

5. Empirical Results and Performance Characteristics

Benchmarks on standard datasets (e.g., MNIST) demonstrate that SSMs and related synaptic-sampling schemes not only achieve classification accuracy competitive with (and often exceeding) traditional RBMs or deterministic spiking models, but also yield representations that are naturally sparse and robust to pruning and quantization (Neftci et al., 2015). The learned networks tolerate the removal of 75–80% of weak synapses with negligible accuracy loss and benefit from stochastic regularization akin to DropConnect or Dropout.

In reinforcement settings, such as visuomotor tasks, synaptic sampling with policy-gradient learning rules enables fast online adaptation, with synaptic sparsity emerging naturally under appropriate priors (Kaiser et al., 2020). Neuromorphic platforms implementing binary or stochastic crossbars report substantial reductions in energy per inference, the minimization of ensemble size required for stable performance, and increased robustness to device-level variability (Yepes et al., 2016).

6. Interpretations, Implications, and Open Questions

Synaptic sampling offers a unified Bayesian account of synaptic variability, learning, and probabilistic inference in neural systems (Aitchison et al., 2015, McKee et al., 2022). Its key features are:

• Local Uncertainty Representation: Each synapse manifests its confidence via its variance of transmission, directly communicating uncertainty to downstream targets.
• Monte Carlo Population Decoding: Neurons sum sampled PSPs, naturally integrating over uncertainty and permitting risk-sensitive or probability-matching behavior without explicit uncertainty parameterization (Aitchison et al., 2015).
• Biological Plausibility: All mechanisms are implementable with locally available signals, scalable to network sizes appropriate for cortex, and observable in vivo.
• Hardware-Efficient Implementability: Synaptic sampling maps favorably to emerging hardware with low-precision, local stochasticity, and scalable parallelism (Aimone et al., 2023).

Open challenges include characterizing the mixing speed and convergence properties of synaptic sampling in deep recurrent networks, quantifying trade-offs between synaptic reliability and representational capacity, and devising scalable local learning rules that maintain efficiency and stability across biologically plausible operating regimes.

7. Summary Table: Key Synaptic Sampling Frameworks

Framework / Paper	Synaptic Stochasticity Modeled	Learning Rule
SSM (Neftci et al., 2015)	Bernoulli blank-out mask, all synapses	Contrastive divergence / local STDP
Memristor-SSM (Dolzhikova et al., 2018)	Hardware coin-flip at each crossbar	In situ pulses, data–recon difference
Bayesian Dropout (McKee et al., 2021)	Transmission probability set by $w_i$	Online local update of dropout probabilities
Adaptive Failure (McKee et al., 2022)	Product of epistemic ($\phi$) and aleatoric ($q$) dropout	Hebbian count-based moment-matching
Reinforcement SPORE (Kaiser et al., 2020)	Langevin SDE in parameter space	Local eligibility trace with drift/diffusion

These frameworks, spanning from theoretical neuroscience to neuromorphic engineering, collectively establish synaptic sampling as a foundational principle in neural Bayesian computation, with broad applicability to artificial and biological intelligence.