Dynamic Two-Layer MLPs

• Dynamic two-layer MLPs are neural networks with smooth activations where gradient updates concentrate in a fixed low-dimensional subspace established at initialization.
• This emergent subspace behavior supports low-rank training methods that reduce memory and compute demands while preserving model performance.
• Theoretical analysis and empirical validation highlight that proper initialization and smooth nonlinearity are crucial for maintaining subspace invariance during training.

Dynamic two-layer multilayer perceptrons (MLPs) with smooth activations exhibit an emergent phenomenon whereby gradient-based training drives almost all weight changes within a fixed low-dimensional subspace determined at initialization. This behavior, observed under both full-batch and stochastic training regimes, underlies the success of low-rank training, compression, and adaptation methods and can be exploited via explicit architectural parameterizations to yield substantial reductions in both memory and compute cost while achieving performance parity with fully-parameterized models (Xu et al., 5 Feb 2026).

1. Mathematical Formulation and Model Setup

Consider a two-layer MLP of the form

$f_{W_1}(X) = W_2\,\phi\big(W_1 X\big)$

where $X \in \mathbb{R}^{d \times N}$ denotes whitened input data ($XX^\top = I_d$), $Y \in \mathbb{R}^{K \times N}$ the target labels ($K < d/2$), $W_1 \in \mathbb{R}^{m \times d}$ the learned first-layer weights, and $W_2 \in \mathbb{R}^{K \times m}$ a fixed, full-row-rank second-layer matrix. The entrywise nonlinearity $\phi: \mathbb{R} \to \mathbb{R}$ is assumed smooth ($|\phi'|\le\beta,\,|\phi''|\le\mu$). The loss function is

$L(W_1) = \frac{1}{2}\|W_2\,\phi(W_1 X) - Y\|_F^2,$

and training proceeds by gradient descent (GD) or its stochastic variants on $X \in \mathbb{R}^{d \times N}$0, with updates

$X \in \mathbb{R}^{d \times N}$1

2. Training Dynamics and Subspace Invariance

Backpropagation yields the gradients

$X \in \mathbb{R}^{d \times N}$2

so

$X \in \mathbb{R}^{d \times N}$3

A key finding is that GD updates to $X \in \mathbb{R}^{d \times N}$4 concentrate in a fixed $X \in \mathbb{R}^{d \times N}$5-dimensional subspace determined by the initialization and the initial gradient structure. Explicitly, let the SVD of the initial gradient be $X \in \mathbb{R}^{d \times N}$6, with the top-$X \in \mathbb{R}^{d \times N}$7 subspaces specified by $X \in \mathbb{R}^{d \times N}$8 and $X \in \mathbb{R}^{d \times N}$9. The complement, denoted $XX^\top = I_d$0, has dimension $XX^\top = I_d$1 and admits bases $XX^\top = I_d$2 and $XX^\top = I_d$3 for input and output spaces, with

$XX^\top = I_d$4

The magnitude of the projected gradients and updates within $XX^\top = I_d$5 remains uniformly small throughout training: $XX^\top = I_d$6 Thus, the dynamics of $XX^\top = I_d$7 are confined to the orthogonal active subspace orthogonal to $XX^\top = I_d$8. Perturbation-theoretic arguments (e.g., Wedin’s sin Θ theorem) ensure that this subspace drifts only minimally during training.

3. Theoretical Conditions and Guarantees for Low-Rank Dynamics

Several conditions are required to guarantee the emergence and invariance of the low-dimensional subspace:

• Input normalization: inputs must be whitened ($XX^\top = I_d$9), with $Y \in \mathbb{R}^{K \times N}$0.
• Smooth nonlinearities: $Y \in \mathbb{R}^{K \times N}$1, $Y \in \mathbb{R}^{K \times N}$2.
• Initialization: first-layer weights are small and semi-orthogonal ($Y \in \mathbb{R}^{K \times N}$3), with $Y \in \mathbb{R}^{K \times N}$4 suitably bounded.
• Learning rate: step size $Y \in \mathbb{R}^{K \times N}$5 is at most $Y \in \mathbb{R}^{K \times N}$6.

Under these hypotheses:

• The initial gradient is approximately rank-$Y \in \mathbb{R}^{K \times N}$7.
• The singular subspaces of $Y \in \mathbb{R}^{K \times N}$8 associated with large singular values change at most exponentially slowly ($Y \in \mathbb{R}^{K \times N}$9).
• Projected updates into $K < d/2$0 are $K < d/2$1 initially and decay as $K < d/2$2 throughout training.

4. Low-Rank Parameterization: Construction and Initialization

These findings motivate an explicit low-rank reparameterization of the two-layer MLP. If $K < d/2$3, $K < d/2$4 are orthonormal bases of the “active” $K < d/2$5-dimensional subspace, then

$K < d/2$6

where $K < d/2$7 contains the only learned parameters at this layer. This parameterization can be generalized to each intermediate layer of deeper MLPs.

The construction of $K < d/2$8 leverages the initial gradient: a backward pass at $K < d/2$9 identifies the top-$W_1 \in \mathbb{R}^{m \times d}$0 subspaces of $W_1 \in \mathbb{R}^{m \times d}$1; the orthogonal complement to $W_1 \in \mathbb{R}^{m \times d}$2 is designated as $W_1 \in \mathbb{R}^{m \times d}$3, and $W_1 \in \mathbb{R}^{m \times d}$4 is set via $W_1 \in \mathbb{R}^{m \times d}$5. Proper initialization within this subspace (“$W_1 \in \mathbb{R}^{m \times d}$6”) is crucial; random subspace initialization leads to training failure.

5. Empirical Validation Across Architectures and Tasks

Extensive empirical results support the theoretical framework:

• Synthetic two-layer networks ($W_1 \in \mathbb{R}^{m \times d}$7): with smooth nonlinearities (ELU, GELU, SiLU), the orthogonal complement subspaces ($W_1 \in \mathbb{R}^{m \times d}$8 dimensions) experience minimal drift (sub-degree rotation) and near-constant singular values after thousands of GD steps. In contrast, non-smooth activations (ReLU variants) lead to significant subspace and singular value instability.
• Deeper MLPs ($W_1 \in \mathbb{R}^{m \times d}$9): identical concentration of intermediate-layer weight changes to the active subspace.
• Optimization variants: the phenomenon persists under minibatch SGD and Adam, unwhitened data, and cross-entropy loss.
• Low-rank MLP on Fashion-MNIST ($W_2 \in \mathbb{R}^{K \times m}$0): low-rank MLP ($W_2 \in \mathbb{R}^{K \times m}$1) initialized via the prescribed method matches both test-loss and accuracy of the full-width model over 1500 epochs, whereas random projection initialization fails to converge.
• VGG-16 head on CIFAR-10: with the convolutional backbone frozen, a low-rank head ($W_2 \in \mathbb{R}^{K \times m}$2) matches full-head performance within ±0.5% accuracy for full fine-tuning; for classifier-only tuning, gap narrows to ∼2% when doubling $W_2 \in \mathbb{R}^{K \times m}$3 to $W_2 \in \mathbb{R}^{K \times m}$4.

6. Architectural and Practical Implications

This subspace-concentration phenomenon enables practical architectural modifications:

• Memory and compute reduction: wrapping each layer with low-rank factors about the “dead” directions reduces both resource requirements and parameter counts without accuracy loss, provided initialization is performed properly.
• Compatibility with deep architectures: insertions of low-rank wrappers generalize to multi-layer MLP settings, with all intermediate weight changes remaining concentrated as in the two-layer case.
• Fine-tuning and adaptation: provides theoretical explanation for empirical successes of low-rank fine-tuning techniques (e.g., LoRA), where restricting optimization to a small, precomputed subspace is sufficient for high performance.

7. Open Problems and Research Directions

Several open theoretical and practical questions are prompted by these results:

• The mechanism by which smooth activations stabilize the subspace, as opposed to non-smooth options (e.g., ReLU), warrants further characterization.
• Relaxations of input whitening or small-initialization assumptions could broaden applicability.
• Effects of additional forms of stochasticity (dropout, quantization, SGD noise) on subspace invariance remain to be systematically studied.
• Connections to phenomena such as neural collapse, feature learning dynamics, and implicit bias in deep learning are not fully elucidated.
• Extensions to architectures beyond MLPs—including convolutional networks, residual blocks, transformers—as well as online (incremental) subspace tracking methods are promising avenues for future research.

In summary, the analysis of dynamic two-layer MLPs with smooth nonlinearities reveals that the effective learning dynamics are confined to a sharply delimited, initialization-determined subspace. This behavior is preserved under typical training regimes and can be operationalized via low-rank parameterizations to achieve efficient, high-performing models across tasks and architectures (Xu et al., 5 Feb 2026).