Self-Motivated Growing Neural Networks

• SMGrNNs are adaptive neural networks that autonomously evolve their topology during training based on internal metrics like loss and gradient statistics.
• They employ self-memory supervision and generative replay mechanisms to mitigate catastrophic forgetting in continual learning scenarios.
• Empirical studies show that SMGrNNs achieve competitive performance on tasks like generative modeling and reinforcement learning by dynamically balancing growth and pruning.

A Self-Motivated Growing Neural Network (SMGrNN) is a neural learning architecture in which network size or topology evolves autonomously during training based on internal task signals or local statistics. Rather than relying on static architectures or externally-driven architecture search, SMGrNN frameworks grow (and in some cases prune) their own structure in response to observed loss, residual errors, or locally measured plasticity signals. This concept has emerged independently across conditional generative modeling, feedforward and policy networks, and reinforcement learning controllers, providing a principled mechanism for capacity adaptation and addressing phenomena such as catastrophic forgetting or overfitting. Key SMGrNN variants incorporate self-memory supervision via circulatory replay (Huang et al., 2020), residual-driven expansion (Ford et al., 2023), and local statistical plasticity with online topology adaptation (Jia et al., 14 Dec 2025).

1. Architectural Principles in SMGrNN

SMGrNN models can be characterized by their self-tuning architecture, where network capacity grows in alignment with learning dynamics. The growth triggers are tightly coupled to measurable quantities internal to the model's own training or microstates rather than dataset metadata or external controller commands.

• In circulatory CVAE SMGrNNs, topology growth is linked to the appearance of novel categories, with new output “private” parameters instantiated per class (Huang et al., 2020).
• In residual-driven SMGrNN, hidden-layer widths are expanded when statistically significant residual error remains unmodeled, using dual-threshold tests on MSE improvement (Ford et al., 2023).
• In online plasticity-based SMGrNN, neuron and edge insertion—and pruning—are based on short-term statistics such as edge weight-update variance, monitored locally by structural modules embedded within the network (Jia et al., 14 Dec 2025).

The common principle is that network growth is triggered by self-assessed insufficiency in representational capacity, often resolved by adding minimally-invasive new parameters or nodes.

2. Memory, Replay, and Catastrophic Forgetting

A key challenge addressed by SMGrNN initiatives is the mitigation of catastrophic forgetting in continual or class-incremental learning. The SMGrNN based on circulatory variational autoencoders employs a self-memory supervision mechanism, where generative replay recreates pseudo-samples of previously seen classes and circulatory encoding ensures memory of old categories persists as new ones are introduced (Huang et al., 2020):

• For observed categories $i<j$, memory playback reintroduces generated samples by

$$x_{\rm mem}^{(i)} \sim p_\theta(x|z^{(i)}, y^{(i)}), \quad z^{(i)}\sim\mathcal{N}(0,I)$$

and applies a memory-supervision loss summed over past labels.

• The circulatory mechanism involves re-encoding the generated sample and enforcing latent consistency with additional KL and BCE losses, establishing robust embedding for replayed data.

Empirically, this strategy preserves generative accuracy and classifier performance on earlier categories even in the absence of explicit replay for specific labels (Huang et al., 2020).

3. Formal Algorithms and Growth Criteria

The operational protocols for SMGrNN vary by model class, but share key themes:

• Residual-Fitting SMGrNN (Ford et al., 2023):
  • Computes base network loss $\alpha$, fits a smaller residual network on current error, computes new combined loss $\beta$.
  • Growth is triggered if both $\beta/\alpha < 1 - \gamma$ and $\alpha/\alpha_{\text{prev}} < 1 - \gamma$ hold, for user threshold $\gamma$.
  • Upon each growth, weights are block-diagonal fused; new connections are initialized with small magnitude.
• Circulatory CVAE SMGrNN (Huang et al., 2020):
  • On each novel class, a new decoder column for $W_p$ is appended and initialized with random weights for class-conditional decoding.
  • Only the relevant column is updated per input, preventing parameter starvation.
  • The total loss $\mathcal{L}_{\text{total}}$ incorporates standard CVAE, cyclical replay, and memory-replay supervision terms, each weighted by hyperparameters.
• Plasticity-Driven SMGrNN (Jia et al., 14 Dec 2025):
  • Maintains rolling buffers for short-window average $\mu_k^\Delta$ and variance $\sigma_k^\Delta$ of edge-wise gradient steps.
  • Node or edge growth occurs when

  $$|\mu_k^\Delta| < \frac{1}{2} \sigma_k^\Delta,\quad (\sigma_k^\Delta)^2 > \lambda_{\rm edge} |\mu_k^\Delta|$$

  indicating optimizer indecision about the edge's sign or magnitude. - Pruning removes edges with both small absolute weight and low average recent update, ensuring capacity does not explode.

Pseudocode in each reference specifies precise stepwise mechanics, always centering trigger conditions on statistics derived from the network's intrinsic behavior (Huang et al., 2020, Ford et al., 2023, Jia et al., 14 Dec 2025).

4. Empirical Evaluation and Performance

Experimental comparisons appear across several tasks and domains:

• Generative Modeling (MNIST, Fashion-MNIST): SMGrNN with circulatory CVAE achieves near-identical classifier accuracy on generative and real data as joint training, with $99.6\%$ vs. $99.2\%$ accuracy on MNIST and $81.5\%$ vs. $82.1\%$ on Fashion-MNIST (Huang et al., 2020). Private parameter growth enables continual addition of categories without accuracy loss.
• Classification, Imitation, and RL: Residual-fitting SMGrNN performs comparably or better than large fixed-size nets, with notably smaller final average width and better capacity utilization. On tasks such as CIFAR-10 histogram classification and MuJoCo imitation, adaptive networks can occasionally outperform statically allocated larger models (Ford et al., 2023).
• Control Policy Distillation: Plasticity-based SMGrNN attains fast learning, lower return variance, and smaller or task-appropriate final sizes on CartPole, Acrobot, and LunarLander compared to static MLP baselines (Jia et al., 14 Dec 2025).

Ablation studies consistently reveal the necessity of both growth and pruning: disabling growth harms learning and increases variance, while omitting pruning leads to explosive but unhelpful increase in network complexity (Jia et al., 14 Dec 2025).

5. Underlying Mechanisms and Theoretical Perspectives

All SMGrNN models decouple the condition for structural change from global signals not tied directly to task error or local edge/node statistics. Key mechanisms include:

• Local plasticity modules (e.g., SPM in (Jia et al., 14 Dec 2025)) are implemented via simple windows over gradient histories, allowing edge-wise adaptation with minimal global coordination.
• Residual-driven fusion avoids over-parametrization by requiring statistically meaningful error improvement, with fusion initialized to avoid dead connections (Ford et al., 2023).
• Disentanglement via selective activation in circulatory CVAE ensures private and common features are robust to catastrophic interference, even with partial replay omission (Huang et al., 2020).

This locally triggered adaptability, often achieved with only small hyperparameter sets and minimal external supervision, enables networks to scale indentifiably with problem difficulty or task complexity.

6. Extensions, Applications, and Future Directions

The SMGrNN paradigm is extensible along several research frontiers (Jia et al., 14 Dec 2025):

• Integration of Hebbian or spike-timing-dependent plasticity to further align SPM events with local co-activity or spike pattern rules.
• Application to continual and lifelong multi-task learning, where the ability to retain sub-networks for prior tasks may mitigate interference and forgetting.
• Meta-optimization of structural plasticity parameters via higher-level search to automate regime selection.
• Neuromorphic deployment, where local-rule modularity is conducive to hardware-constrained, event-driven compute paradigms.

A plausible implication is that broad adoption of adaptive, self-motivated structural plasticity could reduce the need for exhaustive architecture search, allowing neural systems to better calibrate capacity to data and task structure during deployment.

7. Comparative Summary of SMGrNN Implementations

Reference	Growth Trigger	Pruning	Task Domains	Adaptivity Mechanism
(Huang et al., 2020)	Arrival of new category	No	Generative modeling (CVAE, continual learning)	Per-category param expansion, memory replay
(Ford et al., 2023)	Residual error reduction	No	Classification, imitation, reinforcement learning	Residual fusion with dual-threshold
(Jia et al., 14 Dec 2025)	Local edge update statistics	Yes	RL policy distillation, control	Edge/node insertion & pruning via SPM

Each approach exemplifies the central concept of self-motivated structural evolution based on task- or plasticity-internal signals, representing a significant advance in adaptive neural modeling.