Emergent Low-Rank Training Dynamics in DNNs

• Emergent low-rank training dynamics are phenomena where gradients and weight updates concentrate in low-dimensional subspaces, despite high nominal parameter dimensions.
• Analyses based on stable rank measures and gradient decomposition illuminate how training trajectories confine within invariant subspaces across various architectures.
• Adaptive algorithms like Cuttlefish and OIALR leverage these dynamics to achieve substantial parameter compression and improved training speed without sacrificing performance.

Emergent low-rank training dynamics refer to the widespread and theoretically grounded phenomenon in deep neural network (DNN) optimization wherein gradients, weight updates, and even entire training trajectories concentrate within low-dimensional subspaces—well below the nominal parameter-space dimensionality. This emergent property has far-reaching implications for efficiency, generalization, network design, and the development of adaptive low-rank learning algorithms. The foundational insight is that, whether as a product of optimization geometry, architectural inductive bias, or the statistical structure of data, low-rank dynamics are not imposed but arise naturally throughout training and fine-tuning processes.

1. Mathematical Foundations of Low-Rank Training Dynamics

The low-rank character of neural training dynamics has been formalized from several complementary perspectives:

Stable Rank and Effective Dimensionality

For a weight matrix $W\in\mathbb{R}^{m\times n}$, the stable rank, defined as $$\mathrm{srank}(W) = \|W\|_F^2 / \|W\|_2^2 = \sum_{i}\sigma_i^2/\sigma_1^2$$, provides a smooth measure of "effective" matrix dimensionality that discounts negligibly small singular values (Wang et al., 2023). Empirical studies find that, after an initial phase of volatility, each layer’s stable rank rapidly converges and remains nearly constant throughout the remainder of optimization.

Gradient Decomposition and Rank Evolution

In DNNs, per-layer gradients for a batch of size $B$ are sums of rank-one outer products: $\nabla_W \mathcal{L} = \sum_{i=1}^B \delta_i x_i^T$, so $\operatorname{rank}(\nabla_W \mathcal{L}) \le B$ (Baker et al., 2024). This structure extends across architectures (fully-connected, convolutional, recurrent) and is shaped by bottlenecks, activation function non-linearity, input size, and other inductive constraints.

Theoretical Results in Linear/Nonlinear Regimes

In multilayer perceptrons (MLPs) with smooth activations and properly scaled orthogonal initialization, it is possible to explicitly identify invariant low-dimensional subspaces—parameterized by input and output dimensionality—within which the entire trajectory of weight updates remains confined (Xu et al., 5 Feb 2026). In three-layer linear models, the cumulative weight updates grow incremental rank in a sequential, mode-wise fashion, mirroring singular value learning (Zhao et al., 2023).

In recurrent neural networks (RNNs), stacking all weight matrices across training steps into a third-order tensor, the tensor rank remains controlled and low, with theoretical upper bounds determined by the initial weight rank and task complexity (Pellegrino et al., 2023).

2. Empirical Evidence Across Architectures

Evidence for emergent low-rank dynamics has been collected for a range of network types and scales:

• Convolutional and ResNet/VGG architectures: Layerwise stable ranks typically plateau within the first 20–80 epochs. For example, in ResNet-18 on CIFAR-10, convolutional block stable ranks stabilize near epoch 80 (Wang et al., 2023).
• Transformers and LLMs: The phenomenon is non-uniform; attention projections and certain MLP sub-layers exhibit significant compressibility, while other layers are closer to full rank (Jaiswal et al., 2024).
• Recurrent Networks (biological and artificial): RNN training on motor tasks produces weight tensors of very low tensor rank, with less than 5 components explaining nearly all connectivity changes over learning (Pellegrino et al., 2023).

Empirical validation shows that low-rank subspaces determined at initialization or soon after can predict and restrict the locus of substantial training activity, whether in synthetic, computer vision, or language modeling settings.

3. Dynamical Mechanisms and Low-Rank Structure Emergence

Several mechanisms underlie the concentration of training in low-rank subspaces:

• Gradient Descent Spectral Concentration: Gradient flow dynamics, even without explicit rank truncation, favor movement along leading singular vector directions, resulting in an implicit bias toward low-rank solutions (Balzano et al., 25 Mar 2025). This bias is amplified by the contraction effect of the spectrum under repeated application of the Hessian in local quadratic approximation.
• Sequential Mode Learning: In linear architectures, stochastic gradient descent causes singular modes to "light up" sequentially, with the cumulative update trajectory increasing rank at most one at a time (Zhao et al., 2023).
• Subspace Freezing: After a rapid early phase in which orthogonal factor bases (from SVD of weights) evolve quickly, these bases stabilize and further optimization occurs predominantly in the subspace spanned by the largest singular vectors (Coquelin et al., 2024). Empirically, basis-alignment metrics show early drop and subsequent saturation.
• Invariant Update Confinement: For two-layer MLPs with smooth activations, gradient updates in bulk dimensions (above a threshold determined by task output size and data dimension) are provably negligible, and full training is confined to a 2K-dimensional subspace where K is the output dimension (Xu et al., 5 Feb 2026).

4. Practical Algorithms Leveraging Emergent Low-Rank Dynamics

Recognizing these emergent properties, a variety of algorithms have been proposed to exploit low-rank structure for efficiency and adaptive compression:

Adaptive Low-Rank Training

• Cuttlefish: Tracks full-rank weights’ stable ranks during an initial warmup, then automatically switches to low-rank parameterizations with factor dimensions set to measured stable ranks. This process yields 4–10× reductions in parameter count and >1.2× training speedups with no accuracy loss, and operates entirely without manual rank tuning (Wang et al., 2023).
• Orthogonality-Informed Adaptive Low-Rank Training (OIALR): Integrates SVD factorization once basis-stability is detected, then performs updates in low-rank form, dynamically retruncating based on singular value thresholds. This achieves up to 83% compression with <2% accuracy loss (Coquelin et al., 2024).
• Incremental Low-Rank Learning (InRank): Maintains explicit low-rank factorizations of cumulative weight updates, incrementing rank only as needed to explain variance in learned modes. GPT-2 models trained with InRank achieve ∼1.3–1.6× speedup and use only 25–30% of full-rank parameter capacity (Zhao et al., 2023).

Memory-Efficient Fine-Tuning and Gradient Compression

• WeLore: Analyses gradient subspace stabilization to categorize each layer as low-rank (LRC) or non-low-rank (N-LRC), then applies data-agnostic SVD-based projection and restricts fine-tuning to LRCs, preserving or even improving generalization at dramatically reduced resource cost (Jaiswal et al., 2024).
• Fira: Observes that Adam's per-matrix scaling factors are nearly invariant under low-rank gradient projections in LLM training. Fira utilizes norm-based scaling to perform full-rank weight updates using only low-rank optimizer state, paired with a norm-growth limiter to ensure stable dynamics, thus matching or exceeding full-rank performance at 61% lower memory cost (Chen et al., 2024).

Geometry-Aware Optimization

• Riemannian Momentum and Low-Rank Adam: Standard (Euclidean) momentum and Adam algorithms applied naively to factorized weights can induce instability or spurious solutions. Geometry-aware optimizers project both the parameter velocity and gradient onto the tangent-space of the fixed-rank manifold, guaranteeing rank invariance, subspace stability, and robust convergence (Schotthöfer et al., 20 Jun 2025). This leads to improved validation metrics and convergence rates compared to factor-wise methods.

5. Architectural and Task Dependence

The extent and utility of emergent low-rank dynamics depends sensitively on architectural and task factors:

• Activation Function: Strongly nonlinear activations (e.g., ReLU) induce more pronounced gradient rank collapse. Slope-parameterized activations (Leaky-ReLU) interpolate between full and low-rank regimes (Baker et al., 2024).
• Bottleneck and Downsampling Structures: Narrow intermediate layers or downsampling operations cap rank capacity for gradients and thus weight updates. Convolutional stride, patch count, recurrent BPTT window, and output dimension all provide explicit control handles for manipulating emergent low-rankness.
• Task Dimensionality: When the function to be learned is intrinsically low-dimensional, weight updates in, e.g., RNNs, are confined provably and empirically to low-tensor-rank trajectories (Pellegrino et al., 2023).

This sensitivity allows both for intentional exploitation (targeted compression layers, subspace-adaptive training) and for mitigation when excessive rank collapse would hinder learning rich representations.

6. Implications and Open Directions

The recognition of universal, architecture- and data-driven emergent low-rank training dynamics has sparked both practical advances and new theoretical investigations:

• Parameter Efficiency: The possibility to match full-rank performance using only a small fraction of trainable parameters (via adaptive low-rank switching) presents substantial benefits in training large-scale models and in edge or federated contexts (Wang et al., 2023, Chen et al., 2024).
• Theoretical Underpinning for LoRA and PETL: Kernel regime limitations do not capture the full behavior of low-rank adaptation and fine-tuning. Nonlinear, feature-rich regimes admit rapid convergence of low-rank adapters, especially when initialized on top of a strong base model (Dayi et al., 2024, Balzano et al., 25 Mar 2025).
• Interpretability and Compression: Low-rank parametrizations not only speed training but yield more interpretable model updates, as changes are concentrated in a small number of meaningful subspaces (Jaiswal et al., 2024).
• Extension to Optimizer Design and Nonlinear Settings: Geometry-aware momentum and adaptive optimizers offer robust, theoretically justified improvements when training in low-rank or mixed-rank regimes (Schotthöfer et al., 20 Jun 2025). Recent results begin to extend explicit subspace confinement theory from linear networks to smooth, nonlinear MLPs (Xu et al., 5 Feb 2026).
• Task-Specific Adaptivity: Layerwise, data-driven adaptation of rank (as in Cuttlefish, WeLore) optimizes the capacity/redundancy tradeoff and avoids over-compressing critical modules.

Future work will refine the theoretical connection between initialization choice, data geometry, and emergent subspace structure; extend provable results to convolutional and transformer architectures; and unify automated rank-adaptive training with dynamic optimizer schemes. The emergent low-rank perspective thus provides a rigorous foundation for engineering efficient, scalable, and adaptive deep learning systems.