Self-Abstraction Learning for Effective and Stable Training of Deep Neural Networks

Abstract: Training large-scale deep neural networks effectively and stably is essential for applying deep learning across various fields. However, conventional methods, which rely on training a single large network, often encounter challenges such as gradient vanishing, overfitting and unstable learning. To overcome these limitations, we introduce Self-Abstraction Learning (SAL), a hierarchical framework. In SAL, networks are arranged by structural complexity, where the simplest topmost network is trained first and its hidden and output layers serve as guidance for the successively more complex networks below. This top-down sequential guidance effectively mitigates optimization issues, enabling stable training of deep architectures. Various experiments across MLP, CNN, and RNN architectures demonstrate that SAL consistently outperforms conventional methods, ensuring robust generalization even in data-scarce and complex network regimes.

• The paper demonstrates that a hierarchical guidance pipeline using coarse-grained representations stabilizes and regularizes deep neural network training.
• The method leverages MSE loss to transfer hidden activations from simpler models, effectively mitigating gradient vanishing and reducing overfitting.
• Empirical results across various architectures and datasets, including challenging data-scarce regimes, confirm significant accuracy improvements and robust generalization.

Self-Abstraction Learning: A Hierarchical Framework for Stable and Effective Deep Neural Network Training

Motivation and Related Work

Deep neural networks (DNNs), including MLPs, CNNs, and RNNs, are routinely scaled in pursuit of performance improvements. However, increased architectural complexity and limited dataset size exacerbate instability, gradient vanishing/explosion, and severe overfitting. Traditional mitigation strategies involve activation function selection (e.g., ReLU), residual connections, and regularization, but often fail to comprehensively address optimization pathologies.

Self-Abstraction Learning (SAL) introduces a hierarchical training pipeline that progressively guides the optimization of deep architectures from simple to complex, leveraging coarse-grained representation spaces as teacher signals. This approach differs fundamentally from conventional Knowledge Distillation, which operates in a complex-to-simple direction for model compression, and curriculum learning, which reorders data presentation or progressively grows models. Instead, SAL transfers hidden layer representations from structurally simpler “upper floor” models, sequentially guiding increasingly complex “lower floor” models within a unified training pipeline.

Figure 1: SAL Framework illustrated with a 1:2 hidden layer width ratio; upper floor guidance precedes lower floor modeling.

The SAL Hierarchical Architecture

SAL’s training proceeds as follows:

1. The topmost ANN, with minimal depth and width, is trained using CE loss.
2. Hidden and output layer activations from this model serve as targets for a deeper, wider model (“lower floor”), which is trained using MSE loss to match those abstraction targets.
3. Subsequently, the lower floor model is directly trained with conventional CE loss.
4. This sequence is recursively applied through multiple floors, ending with the most complex model at the bottom.

The hierarchical guidance leverages coarse representations, regularizing the optimization trajectory and constraining search space—enabling effective learning and generalization, especially in data-scarce regimes.

Figure 2: SAL learning process for a three-floor structure; solid arrows denote standard CE training, dashed arrows denote MSE-based guidance.

Theoretical Generalization and Implicit Regularization

SAL theoretically inherits superior generalization from simpler models. For deeper models $f$ and simpler guiding models $h$, if $\mathbb{E}[\|f(x)-h(x)\|_2] < \epsilon$, the generalization error satisfies:

$R(f) \leq R(h) + C \cdot \epsilon$

for constant $C > 0$; see Appendix for proof.

Moreover, SAL provides a tighter upper bound on generalization via the Rademacher complexity of the simpler model's function class. The hierarchical guidance acts as implicit regularization, preventing the optimization of high-capacity models from degenerating into overfitting, especially with limited $m$.

Empirical Validation Across Architectures and Datasets

SAL was evaluated on a spectrum of architectures (MLP, CNN, LSTM) and datasets (MNIST, CIFAR10, CIFAR100, IMAGENETTE, CARER, TREC, TREC_FINE) with identical bottom floor configurations to plain baselines. Experiments spanned full datasets and extreme subsampling (1/10, 1/100), targeting the data-scarce regime. SAL's guidance parameters (direct/guidance epochs) were varied to assess robustness.

Gradient Flow and Vanishing Mitigation

SAL stabilizes gradient flow across deep networks. In contrast to plain training, which rapidly attenuates gradients away from the output, SAL maintains stable backward propagation through top-down guidance. This is especially pronounced in deep MLPs with 16 layers, as evidenced by log-gradient ratio analysis.

Figure 3: Log-scale gradient ratio across layers on image datasets—SAL preserves gradient magnitude in deep MLP settings.

Generalization Performance and Overfitting Suppression

SAL consistently outperforms plain training in accuracy across both full and partial datasets. Notable improvements are seen in challenging configurations, e.g., CNNs on CIFAR100 (subset) and LSTMs on TREC_FINE, often with marked reductions in performance variance.

SAL also exhibits competitive training efficiency: under time-matched evaluation, SAL surpasses plain training in accuracy throughout the training duration, due to early and stable convergence driven by representational guidance.

Figure 4: Test accuracy over wall-clock training time—SAL achieves superior accuracy in MLP (CIFAR100), CNN (Imagenette), LSTM (CARER).

Hyperparameter and Architectural Sensitivity

SAL’s gains are robust to hyperparameter variations (training/guidance epochs $t$, $r$; floor size $F$) with minimal sensitivity attributable to these settings. The optimal number of floors (typically $F=3$) balances guidance strength with architectural complexity.

SAL incurs modest VRAM overhead (3–24%) and longer training times, reflecting the recursive multi-floor pipeline. The tradeoff for performance and stability is typically favorable.

Practical Deployment: BUSI Medical Imaging

SAL demonstrates strong empirical performance in real-world, data-scarce medical imaging. On the BUSI dataset (780 images across three classes), both MLP and CNN models guided by SAL substantially outperform plain baselines, with mean AUROC improvements of 17.61% (MLP) and 16.46% (CNN), and higher scores on F1-Score, Sensitivity, Specificity, and Accuracy.

Figure 5: SAL vs plain baseline on BUSI, with substantial improvements across all metrics for both MLP and CNN.

Practical and Theoretical Implications

SAL's hierarchical guidance mechanism effectively mitigates deep model variance, overcoming critical limitations of conventional training. The reduced dependency on activation modifications and explicit architectural interventions (e.g., skip connections) enables flexible design and stable optimization in diverse settings. SAL’s favorable bias-variance trade-off and empirically observed overfitting suppression position it as a compelling paradigm for deep model training, particularly when data is scarce or architectural complexity is required.

Future Directions

Potential SAL extensions include:

• Application to advanced Transformer architectures, with implications for NLP and vision tasks.
• Integration of attention-based or relational feature transfer mechanisms to enable more flexible guidance.
• Combining SAL with Knowledge Distillation and residual guidance for synergistic improvement.
• Real-time deployment optimization via residual target definition to exploit speed-accuracy trade-offs.

These avenues may further strengthen SAL’s utility, broadening its impact in deep learning and practical adoption.

Conclusion

Self-Abstraction Learning constitutes a hierarchical, guidance-driven approach for the stable and effective training of deep neural networks. The empirical and theoretical evidence demonstrates consistent improvements over plain training, with robust generalization, mitigated gradient vanishing, and suppressed overfitting. The flexibility and efficacy of SAL render it a valuable strategy for DNN optimization in both data-rich and data-scarce environments, with substantial practical relevance across multiple domains (2604.24313).