Nonlinear Synaptic Pruning (NSP)

• NSP is a family of structural plasticity mechanisms where synaptic changes depend nonlinearly on local activity signals and connectivity metrics.
• NSP research spans co-evolving neural network models, developmental pruning curves, and applications in deep learning and spiking networks, demonstrating phase transitions and efficiency gains.
• NSP frameworks integrate competitive and homeostatic controls with nonlinear scoring rules to yield sparse architectures while preserving network functionality.

Searching arXiv for the cited NSP-related papers to ground the synthesis. Nonlinear Synaptic Pruning (NSP) denotes a class of structural plasticity mechanisms in which synapse addition, removal, survival, or effective strength depends nonlinearly on local activity, connectivity, or learned latent variables, rather than on uniform or purely linear decay. In the cited literature, NSP appears in co-evolving Hopfield-type networks, degree-dependent developmental models, noise-probing rules for recurrent circuits, triplet-STDP accounts of thalamocortical refinement, stochastic gate formulations for deep networks, synaptic-strength pruning in CNNs, and dendrite-inspired sparse SNNs. A recurring motif is a feedback loop in which dynamics shape topology and topology reshapes dynamics, yielding phase transitions, bistability, hub formation, receptive-field refinement, or strong compression with limited performance loss (Millán et al., 2017, Millán et al., 2018, Moore et al., 2020, Lin et al., 2018, Han et al., 2022, Cai et al., 29 Aug 2025).

1. Conceptual scope and defining characteristics

The term is not used uniformly across the literature. Several papers formulate “synaptic pruning,” “adaptive pruning,” “plasticity networks,” or “preferential detachment” without using NSP as an explicit label, while still instantiating the same underlying logic: pruning depends on a nonlinear map from local signals to structural change. The relevant nonlinearities include power laws in physiological current, exponential dependence on degree, BCM-like thresholded rate terms, stochastic binary gates, survival functions with asymmetric updates, and piecewise thresholding coupled to dendritic gain parameters. This suggests that NSP is best regarded as a mechanistic family rather than a single canonical algorithm (Millán et al., 2018, Kazu et al., 2024, Li et al., 2019, Crodelle et al., 26 Apr 2025, Cai et al., 29 Aug 2025).

Across these formulations, four features recur. First, pruning is local in the sense that each synapse or unit is scored by quantities available at or near that site: pre/post activity traces, covariance, degree, norm-scaled weight, or a gate variable. Second, the scoring rule is nonlinear, e.g. $I_i^\alpha$, $\exp[-k f(\cdot)]$, $r(r-r_{\text{th}})$, or $\sigma(k\phi)$. Third, pruning is coupled to a competitive or homeostatic constraint, so potentiation or preservation of some connections increases the pressure on others. Fourth, developmental history matters: transient over-connectivity, wave statistics, frozen-density intervals, or epoch-dependent schedules can determine the eventual stationary phase or sparse architecture (Millán et al., 2017, Millán et al., 2018, Han et al., 2022, Crodelle et al., 26 Apr 2025, Vos et al., 12 Aug 2025).

2. Co-evolving neural-network models and developmental pruning curves

A foundational formulation appears in adaptive auto-associative networks that combine Amari–Hopfield neural dynamics with evolving topology. In these models, neurons are binary stochastic units with local field $h_i(t)=\sum_j w_{ij} e_{ij}(t) s_j(t)$, threshold $\theta_i(t)=\frac{1}{2}\sum_j w_{ij} e_{ij}(t)$, and memory overlap $m(t)$ as the macroscopic retrieval order parameter. Structural dynamics are specified by gain and loss probabilities

$$P_i^g=u(\kappa)\,\pi(I_i), \qquad P_i^l=d(\kappa)\,\eta(I_i),$$

with mean degree $\kappa(t)=\frac{1}{N}\sum_i k_i(t)$ and local physiological variable $I_i=|h_i-\theta_i|$. In the simplified power-law form, effective local preferences are

$\exp[-k f(\cdot)]$0

so the topology evolves by a nonlinear function of activity. These models exhibit a homogeneous memory phase, a heterogeneous memory phase with hubs and strong disassortativity, and a homogeneous noisy phase; near the structural critical line, the network acquires scale-free statistics such as $\exp[-k f(\cdot)]$1, $\exp[-k f(\cdot)]$2, and $\exp[-k f(\cdot)]$3 (Millán et al., 2017).

The 2018 extension makes the developmental trajectory itself central. Mean connectivity obeys

$\exp[-k f(\cdot)]$4

and a realistic pruning curve is obtained with an early growth term, yielding

$\exp[-k f(\cdot)]$5

with an early peak $\exp[-k f(\cdot)]$6 followed by pruning to $\exp[-k f(\cdot)]$7. In the frozen-density approximation, the network is held at $\exp[-k f(\cdot)]$8 for a transient interval $\exp[-k f(\cdot)]$9, after which pruning begins. The critical observation is that the onset heterogeneity $r(r-r_{\text{th}})$0, not $r(r-r_{\text{th}})$1 alone, predicts the stationary state: small $r(r-r_{\text{th}})$2 leads to a heterogeneous memory phase, large $r(r-r_{\text{th}})$3 to a homogeneous noisy phase. In the bistable regime, varying $r(r-r_{\text{th}})$4 induces a discontinuous transition in stationary overlap $r(r-r_{\text{th}})$5 and homogeneity $r(r-r_{\text{th}})$6, and intermediate transient density is optimal both for memory and for achieving stable memory states with a minimum energy consumption. The same framework was proposed as a possible explanation for characteristic synaptic pruning curves and for anomalies such as autism and schizophrenia associated, respectively, with a deficit or an excess of pruning (Millán et al., 2018).

3. Analytical variants in neuroscience: covariance, degree dependence, and spontaneous waves

A distinct NSP mechanism uses background noise to probe recurrent-network structure. In linear rate networks with dynamics $r(r-r_{\text{th}})$7, the stationary covariance $r(r-r_{\text{th}})$8 is used to define a synapse-specific preservation probability. For excitatory synapses,

$r(r-r_{\text{th}})$9

and for inhibitory synapses the sign of the covariance term is reversed. Surviving synapses are strengthened by $\sigma(k\phi)$0, whereas non-surviving synapses are set to zero. For a subset of linear and rectified-linear networks, this rule preserves the spectrum of the original matrix and hence preserves network dynamics even when the fraction of pruned synapses asymptotically approaches $\sigma(k\phi)$1. Here the nonlinearity lies in the multiplicative dependence on weight and covariance, together with the stochastic strengthen-or-prune operation (Moore et al., 2020).

A complementary developmental abstraction treats pruning as preferential detachment in a directed graph. Neuronal death is governed by

$\sigma(k\phi)$2

and synaptic pruning by

$\sigma(k\phi)$3

Because pruning probability depends exponentially on the product $\sigma(k\phi)$4, synapses between well-connected neurons are strongly protected, whereas edges involving poorly connected neurons are removed much more often. In predominantly feed-forward networks this preferential detachment generates heavy-tailed, near scale-free degree distributions, with reported power-law exponents in the range $\sigma(k\phi)$5, and reproduces a decreasing pruning rate over developmental time. The authors explicitly note that the algorithm is not intended to be a realistic model of neuronal network formation, but rather an existence proof that selective deletion alone can generate heavy-tailed connectivity (Kazu et al., 2024).

A third line of work studies thalamocortical development under stage II retinal waves. In the full model, LGN spikes drive AdEx V1 neurons, and LGN$\sigma(k\phi)$6V1 synapses evolve under triplet STDP with fast rate homeostasis. In a reduced rate description, the effective weight update is

$\sigma(k\phi)$7

while the firing-rate dynamics separate into an input-gain term and a flux term: $\sigma(k\phi)$8 Stage II waves drive shrinkage of initially broad LGN$\sigma(k\phi)$9V1 receptive fields into a central subset of strong weights; varying wave speed and width changes the amount of pruning; and changes in initial weight or LTD ratio can produce ring-like or periodic receptive fields through bifurcation-like changes in the phase portrait. Adding gap junctions between neighboring V1 neurons promotes precise local retinotopy. The stage II end state also conditions later stage III development: broad stage II receptive fields can support ON/OFF segregation and orientation selectivity, whereas overly narrow stage II receptive fields bias the system toward ON-dominated, iso-oriented structure (Crodelle et al., 26 Apr 2025).

4. NSP in artificial neural networks and deep-learning regularization

In CNN compression, a prominent formulation defines a connection-level importance variable called Synaptic Strength. With convolution, BN, and homogeneous nonlinearity, each kernel $h_i(t)=\sum_j w_{ij} e_{ij}(t) s_j(t)$0 is reparameterized as $h_i(t)=\sum_j w_{ij} e_{ij}(t) s_j(t)$1, and the strength of the connection from input channel $h_i(t)=\sum_j w_{ij} e_{ij}(t) s_j(t)$2 to output channel $h_i(t)=\sum_j w_{ij} e_{ij}(t) s_j(t)$3 is

$h_i(t)=\sum_j w_{ij} e_{ij}(t) s_j(t)$4

Training applies an $h_i(t)=\sum_j w_{ij} e_{ij}(t) s_j(t)$5 penalty to all $h_i(t)=\sum_j w_{ij} e_{ij}(t) s_j(t)$6, then prunes kernels whose synaptic strength falls below a global threshold. The method prunes connections between input and output feature maps rather than entire filters or individual weights. Reported results include up to $h_i(t)=\sum_j w_{ij} e_{ij}(t) s_j(t)$7 pruning on CIFAR-10 and, for ImageNet ResNet-50, a synaptic-pruned model with $h_i(t)=\sum_j w_{ij} e_{ij}(t) s_j(t)$8M parameters, Top-1 error $h_i(t)=\sum_j w_{ij} e_{ij}(t) s_j(t)$9, and Top-5 error $\theta_i(t)=\frac{1}{2}\sum_j w_{ij} e_{ij}(t)$0, compared with a $\theta_i(t)=\frac{1}{2}\sum_j w_{ij} e_{ij}(t)$1M-parameter baseline at Top-1 error $\theta_i(t)=\frac{1}{2}\sum_j w_{ij} e_{ij}(t)$2 and Top-5 error $\theta_i(t)=\frac{1}{2}\sum_j w_{ij} e_{ij}(t)$3 (Lin et al., 2018).

Neural Plasticity Networks formulate pruning and expansion through an $\theta_i(t)=\frac{1}{2}\sum_j w_{ij} e_{ij}(t)$4-regularized objective with stochastic binary gates: $\theta_i(t)=\frac{1}{2}\sum_j w_{ij} e_{ij}(t)$5 The activation probability of each gate is parameterized by a nonlinear function such as $\theta_i(t)=\frac{1}{2}\sum_j w_{ij} e_{ij}(t)$6 or a centered-scaled hard sigmoid. The single parameter $\theta_i(t)=\frac{1}{2}\sum_j w_{ij} e_{ij}(t)$7 modulates the plasticity regime: $\theta_i(t)=\frac{1}{2}\sum_j w_{ij} e_{ij}(t)$8 recovers dropout with probability $\theta_i(t)=\frac{1}{2}\sum_j w_{ij} e_{ij}(t)$9, $m(t)$0 recovers conventional dense training, and finite $m(t)$1 yields adaptive pruning or expansion. The formulation is intrinsically nonlinear because small changes in $m(t)$2 can sharply change the activation probability of a unit, and the cost-benefit tradeoff is imposed directly on expected gate activity (Li et al., 2019).

Self-building Neural Networks apply a different logic. The architecture is initially dense, but all weights start at zero, so functional synaptogenesis occurs through the local Hebbian rule

$m(t)$3

At a chosen pruning episode $m(t)$4, the absolute values $m(t)$5 are thresholded globally by percentile, weak connections are removed, Hebbian learning stops, and the remaining graph is simplified by topological operations that collapse cycles into “fake nodes.” Across classical control tasks, performance decay with increasing pruning rate is smaller than in conventional neural networks, and validation on unseen tasks showed better adaptation, especially when over $m(t)$6 of the weights were pruned (Ferigo et al., 2023).

A more direct training-time regularizer replaces dropout with permanent magnitude pruning controlled by a cubic sparsity schedule. After warmup,

$m(t)$7

with $m(t)$8, $m(t)$9, $$P_i^g=u(\kappa)\,\pi(I_i), \qquad P_i^l=d(\kappa)\,\eta(I_i),$$0, and $$P_i^g=u(\kappa)\,\pi(I_i), \qquad P_i^l=d(\kappa)\,\eta(I_i),$$1. At fixed intervals, low-magnitude active weights are masked out permanently, with pruning applied globally across layers. On RNN, LSTM, and PatchTST models for four time-series datasets, the method ranked best overall; Friedman tests reported statistically significant improvements with $$P_i^g=u(\kappa)\,\pi(I_i), \qquad P_i^l=d(\kappa)\,\eta(I_i),$$2 in many setups; and reported gains included Mean Absolute Error reductions of up to $$P_i^g=u(\kappa)\,\pi(I_i), \qquad P_i^l=d(\kappa)\,\eta(I_i),$$3 over models with no or standard dropout, and up to $$P_i^g=u(\kappa)\,\pi(I_i), \qquad P_i^l=d(\kappa)\,\eta(I_i),$$4 in select transformer models (Vos et al., 12 Aug 2025).

5. Spiking networks and developmental-plasticity implementations

Developmental Plasticity-inspired Adaptive Pruning (DPAP) implements online NSP through trace-based BCM plasticity, dendritic-spine-style neuronal importance, and nonlinear survival functions. For SNNs, spiking traces follow

$$P_i^g=u(\kappa)\,\pi(I_i), \qquad P_i^l=d(\kappa)\,\eta(I_i),$$5

and synaptic importance is defined by

$$P_i^g=u(\kappa)\,\pi(I_i), \qquad P_i^l=d(\kappa)\,\eta(I_i),$$6

with a sliding threshold $$P_i^g=u(\kappa)\,\pi(I_i), \qquad P_i^l=d(\kappa)\,\eta(I_i),$$7. After normalization, synapses and neurons update survival functions $$P_i^g=u(\kappa)\,\pi(I_i), \qquad P_i^l=d(\kappa)\,\eta(I_i),$$8 and $$P_i^g=u(\kappa)\,\pi(I_i), \qquad P_i^l=d(\kappa)\,\eta(I_i),$$9 through asymmetric rules with a positive bonus $\kappa(t)=\frac{1}{N}\sum_i k_i(t)$0 for nonnegative importance and an epoch-dependent exponential term $\kappa(t)=\frac{1}{N}\sum_i k_i(t)$1. Pruning occurs only when the survival function becomes negative, so elimination is gradual rather than instantaneous. Reported SNN results include $\kappa(t)=\frac{1}{N}\sum_i k_i(t)$2 pruning on MNIST with a $\kappa(t)=\frac{1}{N}\sum_i k_i(t)$3 accuracy improvement, $\kappa(t)=\frac{1}{N}\sum_i k_i(t)$4 compression on N-MNIST with $\kappa(t)=\frac{1}{N}\sum_i k_i(t)$5, and $\kappa(t)=\frac{1}{N}\sum_i k_i(t)$6 compression on DVS-Gesture with $\kappa(t)=\frac{1}{N}\sum_i k_i(t)$7, with average compression around $\kappa(t)=\frac{1}{N}\sum_i k_i(t)$8 and overall speedup of about $\kappa(t)=\frac{1}{N}\sum_i k_i(t)$9 (Han et al., 2022).

Adaptive Sparse Structure Development for SNNs (SD-SNN) combines dendritic-spine-inspired synaptic boundaries, neuron pruning, and synaptic regeneration. Each synapse has adaptive bounds $I_i=|h_i-\theta_i|$0 and $I_i=|h_i-\theta_i|$1; persistent boundary violation expands them, persistent decay contracts them by a factor $I_i=|h_i-\theta_i|$2, and neuron importance is defined as

$I_i=|h_i-\theta_i|$3

Low-$I_i=|h_i-\theta_i|$4 neurons are pruned at layer-wise rates $I_i=|h_i-\theta_i|$5 modulated by a neurotrophic-like factor $I_i=|h_i-\theta_i|$6, while pruned synapses can regenerate if their gradients remain within the top $I_i=|h_i-\theta_i|$7 for more than $I_i=|h_i-\theta_i|$8 epochs. Reported results include $I_i=|h_i-\theta_i|$9 accuracy at $\exp[-k f(\cdot)]$00 pruning on spatial MNIST and $\exp[-k f(\cdot)]$01 accuracy with a $\exp[-k f(\cdot)]$02 compression rate on DVS-Gesture (Han et al., 2022).

NSPDI-SNN couples nonlinear synaptic pruning to nonlinear dendritic integration. The effective post-transition weight is

$\exp[-k f(\cdot)]$03

followed by hard pruning at threshold $\exp[-k f(\cdot)]$04, giving the final rule

$\exp[-k f(\cdot)]$05

Here $\exp[-k f(\cdot)]$06 is a learnable transition gain that can vary by synapse, channel, or layer. The same layers may also include nonlinear dendritic integration,

$\exp[-k f(\cdot)]$07

or its convolutional analogue. Reported sparsity–accuracy points include about $\exp[-k f(\cdot)]$08 sparsity on DVS128 Gesture with average loss $\exp[-k f(\cdot)]$09, $\exp[-k f(\cdot)]$10 sparsity with average loss $\exp[-k f(\cdot)]$11, $\exp[-k f(\cdot)]$12 sparsity on CIFAR10-DVS with average loss $\exp[-k f(\cdot)]$13, and $\exp[-k f(\cdot)]$14 sparsity on CIFAR10 with average loss about $\exp[-k f(\cdot)]$15. The method also reported the best experimental results on all three event-stream datasets considered in the study (Cai et al., 29 Aug 2025).

6. Recurring principles, efficiency claims, and open problems

Several principles recur across these disparate formulations. First, transient developmental structure matters: in co-evolving Hopfield networks, onset heterogeneity at the beginning of pruning determines the eventual memory phase; in wave-driven thalamocortical development, the stage II receptive-field profile determines whether stage III can generate orientation selectivity; and in adaptive SNN pruning, early survival trajectories strongly constrain later sparsity (Millán et al., 2018, Crodelle et al., 26 Apr 2025, Han et al., 2022). Second, pruning is rarely an isolated deletion event: it is usually embedded in homeostatic control, whether through fixed-$\exp[-k f(\cdot)]$16 growth–death balance, fast rate homeostasis, additive synaptic scaling, $\exp[-k f(\cdot)]$17 penalties, or regeneration mechanisms (Millán et al., 2017, Li et al., 2019, Han et al., 2022). Third, selective heterogeneity often functions as an objective in itself: hub-rich and disassortative connectivity stabilizes memory; preferential detachment yields heavy tails and parsimonious wiring; and dendritic gain parameters can preserve function at very high sparsity (Millán et al., 2017, Kazu et al., 2024, Cai et al., 29 Aug 2025).

A common misconception is that NSP always means hard elimination of small weights. The cited work shows a broader landscape: in some models pruning is effectively a depression to near-minimal weight rather than literal deletion; in others it is a stochastic survive-or-strengthen rule, a binary gate that can hibernate or reactivate, a survival function crossing zero after gradual decay, or a structured neuron/channel removal accompanied by synaptic regeneration. This suggests that “pruning” in the NSP literature frequently denotes a dynamical structural-selection process, not merely one-shot sparsification (Moore et al., 2020, Li et al., 2019, Han et al., 2022).

The literature also leaves clear open problems. The preferential-detachment model is explicitly not intended as a realistic model of neuronal network formation; the strongest spectral guarantees for noise-based pruning are derived only for restricted linear settings; Synaptic Strength assumes BN before convolution and homogeneous activations; the dropout-replacement pruning schedule was demonstrated on time-series architectures and notes that scalability to huge vision or LLMs remains to be explored; Self-building Neural Networks identify automatic decisions about when and how much to prune as future work; DPAP points toward combining pruning with developmental growth and probabilistic survival rules; and NSPDI-SNN suggests extensions toward attention mechanisms and meta-learning of pruning schedules. Taken together, these constraints indicate that NSP is already a rich formal framework, but not yet a unified theory spanning developmental neuroscience, recurrent dynamics, deep CNNs, and large-scale sparse training (Kazu et al., 2024, Moore et al., 2020, Lin et al., 2018, Vos et al., 12 Aug 2025, Ferigo et al., 2023, Han et al., 2022, Cai et al., 29 Aug 2025).