NeuroPlastic: A Plasticity-Modulated Optimizer for Biologically Inspired Learning Dynamics

Abstract: Optimization algorithms are fundamental to modern deep learning, yet most widely used methods rely on update rules based primarily on local gradient statistics. We introduce NeuroPlastic, a plasticity-modulated optimizer that augments gradient-based updates with an adaptive multi-signal modulation mechanism inspired by multi-factor synaptic plasticity, a concept from neurobiology. NeuroPlastic dynamically scales gradient updates using interacting components that capture gradient, activity-like, and memory-like statistics, forming a lightweight modulation layer compatible with standard deep learning training pipelines. Across image classification benchmarks, NeuroPlastic consistently improves over a controlled gradient-only ablation, with more pronounced gains on the Fashion-MNIST benchmark and in reduced-data regimes. In transfer experiments on CIFAR-10 with ResNet-18, the method remains stable and competitive without retuning. These results suggest that multi-signal plasticity-inspired modulation can provide a useful extension to conventional gradient-driven optimization, particularly when learning signals are limited or noisy, and offer a promising direction for gradient-based methods in deep learning.

• The paper presents NeuroPlastic, an optimizer that modulates gradient updates using normalized gradient, activity, and memory signals.
• Empirical results show modest accuracy gains on MNIST and notable improvements on Fashion-MNIST, particularly in low-data regimes.
• The optimizer integrates as a drop-in layer without extra parameters, maintaining robust and stable performance even in transfer tasks like CIFAR-10.

NeuroPlastic: A Plasticity-Modulated Optimizer for Biologically Inspired Learning Dynamics

Introduction

This work proposes NeuroPlastic, a biologically inspired optimizer that augments standard gradient-based updates with a multi-signal, plasticity-modulation mechanism. Drawing explicitly on principles of multi-factor synaptic plasticity from computational neuroscience, NeuroPlastic combines information from gradient, activity-like, and memory-like filtered signals to construct a per-parameter plasticity coefficient. This modulation layer is designed to adaptively scale gradient updates, potentially improving optimization dynamics, especially in data- or signal-limited regimes. The optimizer is integrated as a drop-in replacement without introducing additional trainable parameters or modifying model architectures.

NeuroPlastic Formulation

The NeuroPlastic update rule is grounded in the augmentation of classical stochastic gradient descent with a biologically motivated modulation architecture. For parameters $\theta_t$ at optimization step $t$, the raw stochastic gradient $g_t = \nabla_\theta \mathcal{L}(\theta_t)$ is transformed by a learned plasticity signal. The per-parameter modulated update $u_t$ is given by

$u_t = \alpha_t \odot g_t,$

where $\odot$ denotes element-wise multiplication, and $\alpha_t$ is the plasticity coefficient. $\alpha_t$ is constructed as a weighted combination of three normalized statistics:

• Normalized gradient magnitude ($g_t^{(s)}$)
• Activity trace (exponential moving average of $|g_t|$)
• Memory trace (ratio of first to second moment, Adam-style)

Each component is z-score normalized, and the result is stabilized by a two-stage homeostatic operation: norm clipping and RMS-based gain control, constraining $t$0 and matching an RMS target.

Figure 1: The NeuroPlastic update process—gradient signals are modulated by plasticity coefficients formed from normalized gradient, activity, and memory signals before stabilization and parameter updates.

In ablation, the gradient-only condition computes updates by removing the activity and memory contributions, only retaining per-parameter, bounded direct modulation of gradient magnitude.

Empirical Evaluation

NeuroPlastic is benchmarked against both its gradient-only ablation and widely used optimizers (SGD, Adam, AdamW) on MNIST, Fashion-MNIST, and CIFAR-10, including a systematically controlled low-data regime.

MNIST and Fashion-MNIST

On the classical MNIST dataset, NeuroPlastic yields consistent, though modest, improvements in test accuracy and loss relative to the gradient-only version. The gains are more pronounced on the more difficult Fashion-MNIST task, with NeuroPlastic showing a mean test accuracy improvement of approximately +0.006 and reduced loss across independent seeds.

Figure 2: NeuroPlastic yields improvements in test accuracy over a gradient-only ablation throughout training on both MNIST and Fashion-MNIST, with larger gains in the more complex Fashion-MNIST.

Crucially, mechanistic diagnostic analyses show that the raw gradient distribution remains similar across baselines. Performance gains emerge from the modulation and stabilization of updates (as indicated by divergent trajectories for effective plasticity and update norms), not from a simple rescaling of gradients.

Low-Data Regime Robustness

A systematic reduction of training set sizes (10%, 25%, 50%, 100%) on Fashion-MNIST demonstrates that NeuroPlastic's advantage expands as the data regime becomes more scarce, with larger improvements in both final accuracy and loss in low-sample conditions.

Figure 3: Across all data fractions, NeuroPlastic outperforms gradient-only ablation, showing greater improvements under increased data constraints.

CIFAR-10 and Transfer

When transferred to CIFAR-10 with ResNet-18 and without retuning (using MNIST-trained hyperparameters), NeuroPlastic maintains stable convergence and competitive performance with baseline optimizers. While SGD and Adam/AdamW reach higher final accuracy under well-tuned learning rates, NeuroPlastic closes the gap at higher learning rate schedules and does not destabilize when applied out-of-domain.

Figure 4: On CIFAR-10, NeuroPlastic achieves competitive accuracy under specific learning rates and maintains stable optimization dynamics similar to mainstream optimizers.

Transfer experiments further confirm that while the NeuroPlastic mechanism alone does not outperform gradient-only or classical optimizers under substantial distribution shift, it exhibits strong stability properties.

Figure 5: Both NeuroPlastic and the gradient-only ablation exhibit stable convergence when transferred, suggesting the robustness of modulated plasticity updates.

Mechanistic Dissection and Ablation

Mechanistic diagnostics reveal that NeuroPlastic maintains higher and more stable plasticity coefficients across training, modifying effective update dynamics while keeping raw gradient scales similar to baselines. Update norms are larger in early training and decay more smoothly.

Figure 6: Full NeuroPlastic produces distinct update and plasticity dynamics while preserving raw gradient statistics, pointing to modulation as the source of improved optimization.

Design space exploration on CIFAR-10 demonstrates that naive or alternative modulations (e.g., simple residual or hybrid modulations) usually underperform, while carefully controlled, controller-style formulations can approach classical SGD performance. This underscores the need for principled integration of plasticity signals.

Figure 7: Effectiveness of plasticity modulation is highly formulation-dependent; controller-style mechanisms approach baseline performance, while naive variants may underperform.

Theoretical and Practical Implications

The study demonstrates that biologically inspired, multi-signal modulus can deliver compatible, stable enhancements to standard optimizers under restricted learning signals—most saliently in low-data or noisy-signal regimes. This supports the theoretical prospect that optimizing with richer, multi-factor update statistics yields more robust adaptation, in line with synaptic and neuromodulatory processes in biological systems.

Practically, NeuroPlastic is deployed as a lightweight modulation layer without introducing new parameters or model changes, making it suitable for straightforward integration into existing pipelines. Notably, its benefits are not uniformly positive and depend on both the structure of the plasticity coefficient and the matching of hyperparameters to task and data regime.

Limitations and Future Directions

The main limitations are scope (predominantly light-to-moderate scale image benchmarks) and the hand-crafted nature of the modulation architecture. While results suggest benefit in data-sparse regimes, the advantage is modest and sometimes negated under strong baselines. The current fixed weighting and lack of adaptive scheduling may also constrain maximal gains and generality.

Future research may target several axes:

• Adaptive or learned weighting of plasticity signals
• Broader validation on large-scale, high-noise, or sparse-data domains
• Scheduling modulation strength dynamically as training proceeds
• Integration with modern architectural and preconditioning techniques

Conclusion

NeuroPlastic operationalizes multi-signal plasticity in a practical, modular optimizer. It delivers tangible, though moderate, performance improvements in data- and signal-constrained settings without sacrificing robustness or requiring architectural overhauls. The study establishes the viability of plasticity-inspired update modulation as a direction for optimizers, motivating adaptive, biologically grounded extensions in gradient-based learning frameworks.