$ε$-rank and the Staircase Phenomenon: New Insights into Neural Network Training Dynamics

Abstract: Understanding the training dynamics of deep neural networks (DNNs), particularly how they evolve low-dimensional features from high-dimensional data, remains a central challenge in deep learning theory. In this work, we introduce the concept of $ε$-rank, a novel metric quantifying the effective feature of neuron functions in the terminal hidden layer. Through extensive experiments across diverse tasks, we observe a universal staircase phenomenon: during training process implemented by the standard stochastic gradient descent methods, the decline of the loss function is accompanied by an increase in the $ε$-rank and exhibits a staircase pattern. Theoretically, we rigorously prove a negative correlation between the loss lower bound and $ε$-rank, demonstrating that a high $ε$-rank is essential for significant loss reduction. Moreover, numerical evidences show that within the same deep neural network, the $ε$-rank of the subsequent hidden layer is higher than that of the previous hidden layer. Based on these observations, to eliminate the staircase phenomenon, we propose a novel pre-training strategy on the initial hidden layer that elevates the $ε$-rank of the terminal hidden layer. Numerical experiments validate its effectiveness in reducing training time and improving accuracy across various tasks. Therefore, the newly introduced concept of $ε$-rank is a computable quantity that serves as an intrinsic effective metric characteristic for deep neural networks, providing a novel perspective for understanding the training dynamics of neural networks and offering a theoretical foundation for designing efficient training strategies in practical applications.

• The paper introduces ε-rank to measure neuron function diversity and demonstrates its link to a stepwise loss reduction during training.
• It theoretically proves that higher ε-rank is necessary for significant loss minimization and faster convergence in deep neural networks.
• The study proposes a novel pre-training strategy that enhances initial ε-rank, resulting in improved performance across various tasks.

$\epsilon$-Rank and the Staircase Phenomenon in Neural Network Training Dynamics

Introduction

The paper "$\epsilon$-rank and the Staircase Phenomenon: New Insights into Neural Network Training Dynamics" (2412.05144) introduces the concept of $\epsilon$-rank to quantify the effective feature diversity of neuron functions in deep neural networks (DNNs), particularly in the terminal hidden layer. The authors demonstrate, both theoretically and empirically, that the $\epsilon$-rank is intimately linked to the training loss and exhibits a universal staircase pattern: loss decreases stepwise as the $\epsilon$-rank increases during stochastic gradient descent. This phenomenon appears across a variety of tasks, architectures, and activation functions.

Further, the paper establishes the theoretical correlation between the loss lower bound and the $\epsilon$-rank: a high $\epsilon$-rank is necessary for significant loss minimization. Based on these findings, the authors propose a pre-training strategy to enhance the initial $\epsilon$-rank, resulting in faster convergence and improved accuracy.

Definition and Universality of $\epsilon$-Rank and Staircase Phenomenon

$\epsilon$-rank is defined for a neural network by examining the Gram matrix of neuron functions in the final hidden layer. Given a set of neuron functions $\{\phi_j(x; \theta)\}_{j=1}^n$, the Gram matrix $M_u$ is constructed with entries $$M_{ij} = \int_{\Omega} \phi_i(x; \theta)\phi_j(x; \theta)dx$$. The $\epsilon$-rank is the number of eigenvalues of $M_u$ greater than a threshold $\epsilon$. This extension allows for practical quantification of linear independence among neuron functions even in the presence of numerical or approximation errors.

Empirical evidence is provided for the universality of the staircase phenomenon. Across multiple domains—including function fitting, PDE solving, and high-dimensional tasks like handwriting recognition—the loss function initially plateaus due to low $\epsilon$-rank and subsequently drops sharply when functional diversity (as measured by $\epsilon$-rank) increases. This pattern is robust with respect to network architectures (depth/width), activation functions (ReLU, tanh, ELU, cosine), and task types.

A layerwise analysis indicates that $\epsilon$-rank increases with network depth, suggesting hierarchical enhancement of feature diversity.

Theoretical Foundations

The paper rigorously proves that the lower bound of the loss function is a decreasing function of $\epsilon$-rank, using classical matrix analysis and spectral theory. Specifically, for general DNNs, the loss can be bounded below by a function depending on $\text{dist}(u^*, \mathcal{F}_p)$, where $\mathcal{F}_p$ denotes a function space generated by $p$ $\epsilon$-linearly independent neuron functions. If the $\epsilon$-rank is limited, the network cannot escape high-loss regions regardless of parameter optimization.

The proofs leverage orthogonality and rank-revealing factorization results, establishing that to reduce loss beyond a threshold, expansion of functional diversity (increase in $\epsilon$-rank) is necessary. This mathematically explains the observed staircase phenomenon and provides a formal criterion for effective training.

Practical Implications: Pre-training via $\epsilon$-Rank Enhancement

A key implication is the role of parameter initialization in functional diversity. Typical initialization schemes (such as Xavier/Glorot) yield near-linear neuron functions with low $\epsilon$-rank, leading to loss plateaus and slow convergence. The authors propose pre-training the first hidden layer to enforce $\epsilon$-linear independence.

Their deterministic initialization in 1D constructs $\tanh$ basis functions centered on a uniform grid, yielding high $\epsilon$-rank at startup. For high-dimensional networks, they generate neuron functions as $\tanh(\gamma(a_j\cdot x + b_j))$ with $a_j$ sampled uniformly on the unit sphere and $b_j$ sampled uniformly from the domain, achieving domain-wide representation and functional independence.

Numerical experiments confirm that such strategies immediately elevate $\epsilon$-rank to $O(n)$ and eliminate early loss plateaus—converging an order of magnitude faster and achieving superior final accuracy in both function approximation and PDE solving.

Approaches such as random feature methods and partition of unity further validate the premise: higher initial $\epsilon$-rank correlates strongly with lower final error.

Theoretical and Practical Implications

The $\epsilon$-rank framework creates a bridge between classical numerical analysis and modern deep learning, suggesting the utility of mathematical tools from spectral theory, finite elements, and functional analysis to deepen understanding of neural representations and training bottlenecks.

Practically, the findings challenge and refine standard initialization, feature extraction, and architecture design principles for DNNs, especially in scientific computing contexts (PDEs, PINNs). Enhancing $\epsilon$-rank at initialization is more effective than increasing network width or depth, offering better resource efficiency and reduced training times.

Theoretically, the explicit dependency of loss reduction on $\epsilon$-rank motivates deeper exploration of spectral properties in neural function spaces and their impact on generalization and optimization landscapes.

Future Directions

This research suggests several future avenues:

• Further exploration of $\epsilon$-rank as a diagnostic tool for model selection, pruning, and regularization.
• Integration of $\epsilon$-rank metrics with neural kernel frameworks (NTK, spectral bias) to quantify inductive bias and representational flow.
• Extension to adaptive architecture designs, domain decomposition, and multimodal neural operators for high-dimensional, irregular domains.
• Investigation of dynamic $\epsilon$-rank tracking during training to inform curriculum learning and transfer learning strategies.
• Formal analysis of $\epsilon$-rank in convolutional and transformer architectures.

Conclusion

The introduction of $\epsilon$-rank provides a robust, intrinsic metric for effective feature diversity in deep neural networks, offering a theoretical and practical framework for understanding and optimizing training dynamics. The identification and explanation of the universal staircase phenomenon advance the mathematical foundation of deep learning and provide actionable guidelines for initialization and architecture design, enabling faster convergence and improved accuracy in diverse applications (2412.05144).