Plasticity-Adjusted Sampling (PAS)

• Plasticity-Adjusted Sampling (PAS) is an adaptive framework that calibrates neural learning by leveraging stochastic plasticity rules and dynamic replay ratios.
• It integrates Bayesian inference with continuous-time stochastic dynamics to maintain parameter diversity and suppress overfitting while reusing data efficiently.
• PAS adjusts plasticity via metrics like the Fraction of Active Units to modify update-to-data ratios in reinforcement learning, thereby enhancing network robustness.

Plasticity-Adjusted Sampling (PAS) is a class of adaptive sampling algorithms and stochastic plasticity rules designed to calibrate learning, adaptation, and robustness in neural systems by explicitly monitoring and responding to the plasticity state of the system. PAS is employed both as a general theoretical framework for Bayesian inference in neural circuits and as a practical algorithm for optimizing data reuse in deep reinforcement learning by modulating update frequencies based on functional plasticity metrics. This entry synthesizes both lines of development as established in (Kappel et al., 2015) and (Ma et al., 2023), delineating formal criteria, algorithmic structures, biological parallels, and empirical properties.

1. Theoretical Foundations: PAS as Bayesian Inference

PAS originated as a theory of brain plasticity formulated in terms of Bayesian inference over synaptic parameters and architectures. Within this perspective, a neural network maintains, not a single maximally likely set of parameters, but a posterior distribution $p(\theta|D)$, with $\theta$ the vector of all synaptic strengths and binary connection variables, and $D$ the dataset of observations—inputs or spike patterns. The posterior is defined as:

$$p(\theta|D) \propto p_S(\theta) \cdot p_N(D|\theta),$$

where $p_S(\theta)$ encodes structural priors (such as sparsity or specific weight distributions), and $p_N(D|\theta)$ is the likelihood of data under the parameterization. Rather than greedily maximizing $p(\theta|D)$ (MAP) or $p_N(D|\theta)$ (MLE), PAS employs continuous-time stochastic differential equations (overdamped Langevin dynamics):

$$d\theta_i = b \left[ \nabla_{\theta_i} \log p_S(\theta) + \nabla_{\theta_i} \log p_N(D|\theta) \right] dt + \sqrt{2b} dW_i,$$

ensuring long-term sampling from $p(\theta|D)$. The decomposition of the drift into activity-dependent (Hebbian-like) and structural (prior-driven) terms unifies synaptic and structural plasticity.

2. PAS in Visual Reinforcement Learning: Adaptive Replay Ratio

PAS was later adopted as a practical algorithm in off-policy visual reinforcement learning, where sample efficiency and catastrophic loss of plasticity during critic training were identified as core bottlenecks (Ma et al., 2023). In this context, PAS denotes an automated schedule for the update-to-data (replay) ratio (RR) anchored to an online metric of critic plasticity, enabling dynamic adjustment of data reuse frequency.

Key formalism centers on the Fraction of Active Units (FAU):

$$\Phi_{\mathcal{M}}(t) = \frac{1}{N} \sum_{n\in \mathcal{M}} \mathbf{1}(a_n(x_t) > 0),$$

where $a_n(x_t)$ is the post-activation value for neuron $n$ in module $\mathcal{M}$ on data $x_t$. Changes in critic FAU over an interval $I$,

$\Delta\Phi_C(t) = |\Phi_C(t) - \Phi_C(t-I)|,$

quantify plasticity recovery or stagnation. The PAS algorithm switches from a low to high RR when $\Delta\Phi_C(t)$ drops below a threshold $\tau$.

3. Algorithmic Structure and Implementation

Bayesian PAS: Stochastic Dynamics for Neural Parameters

• All synaptic parameters $\theta_i$ evolve according to Langevin dynamics, incorporating both prior-driven and activity-dependent forces, combined with Gaussian noise perturbations.
• In discrete time:

$$\Delta \theta_i = \eta \left[ \partial_{\theta_i} \log p_S(\theta) + N \partial_{\theta_i} \log p_N(x^n|\theta) \right] + \sqrt{2\eta} \xi_i,$$

where $\eta$ is the step size, $N$ the sample count, and $\xi_i \sim \mathcal{N}(0,1)$.

• Biological mapping: $\theta_i < 0$ denotes a silent synapse; $\theta_i > 0$, an active synapse with efficacy $w_i = \exp(\theta_i - \theta_0)$.

Practical PAS in RL: Adaptive RR Control

• Maintain two values for RR: $\mathrm{RR}_{\text{low}}$ (e.g., $0.5$), and $\mathrm{RR}_{\text{high}}$ (e.g., $2.0$).
• Plasticity is periodically measured (every $I$ env steps). When $\Delta\Phi_C(t) < \tau$, switch to $\mathrm{RR}_{\text{high}}$, otherwise remain at $\mathrm{RR}_{\text{low}}$.
• No changes to replay sampling, model architecture, or loss definitions are required; PAS only governs the data reuse cadence.

Component	Bayesian PAS (Kappel et al., 2015)	RL PAS (Ma et al., 2023)
Plasticity metric	Posterior over synaptic & structural parameters	Fraction of Active Units
Adaptation driver	Langevin dynamics, prior+likelihood gradients, noise	Dynamic replay ratio (RR)
Biological mapping	Synapse rewiring/turnover, STDP, spine motility	N/A
Key update rule	$$\Delta\theta = \eta\nabla_\theta\log p(\theta|D)+\sqrt{2\eta}\xi$$	RR(t): stepwise switching

4. Empirical Properties and Validation

Theoretical PAS

• PAS resist overfitting in small-data, high-dimensional settings (e.g., Boltzmann machines trained on few MNIST samples): inclusion of informative priors via PAS maintains high test log-likelihood and suppresses overfitting, while sampling solely from likelihood (omitting prior) fails to generalize.
• In spiking/structural models, PAS dynamics yield power-law survival curves for newly formed synapses, closely matching in vivo statistics.
• PAS recovers from distributed functional/structural lesions in simulated WTA circuits, automatically reorganizing network structure for self-repair.

RL PAS

• Across DeepMind Control and Atari-100K benchmarks, PAS outperforms static RR schedules both in sample efficiency and final performance, particularly by circumventing catastrophic early-stage critic plasticity loss.
• Empirically, the switch to high RR occurs anywhere from 0.6 to 1.2 million environment steps depending on seed and environment.

5. Biological Interpretation and Predictions

In the Bayesian PAS framework, intrinsic synaptic noise (Wiener-like terms) and molecular-level dynamics (receptor trafficking, actin remodeling) provide the substrate for continuous, stochastic synaptic parameter changes. The theory predicts region- and state-dependent modulation of effective "temperature" $T$ controlling the exploration-exploitation tradeoff in synaptic turnover, potentially mediated by neuromodulators. Survival statistics of synaptic spines and slow drift of tuning curves are formalized as natural consequences of PAS. Structural and synaptic plasticity are unified, as both the existence and strength of connections are governed by single-parameter dynamics.

6. Practical Guidelines and Integration

• In reinforcement learning agents, monitor critic FAU every $I$ steps using batches of recent data.
• Default hyperparameters: $I = 5 \times 10^4$ steps, $\tau=10^{-3}$, $\mathrm{RR}_{\text{low}} = 0.5$, $\mathrm{RR}_{\text{high}}=2.0$.
• The PAS (Adaptive RR) mechanism is lightweight, requiring only an additional monitoring pass and simple logic to set the update-to-data ratio. No other elements of the pipeline are modified.
• Extensive evaluations substantiate the sufficiency and robustness of the single FAU-based switching criterion.

7. Related Models and Extensions

PAS distinguishes itself from classical approaches seeking MAP or MLE solutions by its explicit maintenance of parameter diversity and resilience via posterior sampling. It offers a normative basis for stochasticity in neural dynamics, reinterpreting empirical findings on synaptic turnover and plasticity. In applied domains (such as RL), PAS circumvents manual scheduling of critical hyperparameters (RR) by directly responding to online measurements of functional plasticity. A plausible implication is that variants of PAS could be generalized to other domains where adaptive control of training or sampling regimens is linked to module-level functional diversity (Kappel et al., 2015, Ma et al., 2023).