No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Training

Abstract: Memory-efficient optimizers such as GaLore train LLMs by projecting gradients onto a rank-r subspace recomputed every T steps, assuming this subspace is a slowly drifting object that can be tracked. We show that beyond a small reproducible core, there is no such object. Two estimates of the top-r subspace computed at the same step from disjoint minibatches disagree as much as estimates computed T steps apart (0.73 vs 0.74 of the maximal chordal distance sqrt(2r), at Pythia-160M with r=128): the apparent rotation at each refresh is dominated by estimator noise. This holds across four model families in three architecture classes from 70M to 6.9B parameters, strengthening with scale, and more weakly in a vision transformer. Only ~39 of 128 directions are reproducible across minibatches, and averaging cannot recover the rest: under N-fold averaging the gradient's spectral tail shrinks as N^-1/4 rather than the N^-1/2 of pure noise, so no averaging budget makes the subspace well defined. What helps instead follows from treating each refresh as a change of coordinates for Adam's state. Carrying the second moment blindly is provably about (r-k*)/2 worse than the best rotation-blind estimator, while the first moment transports exactly through the rotation, the optimal linear map under isotropic gradients and the rule LDAdam uses. At 1B over 40k steps (3 seeds), full LDAdam reaches 18.7 perplexity at beta2=0.999, beating untransported GaLore after its best beta2 fix (19.3); shortening the second-moment memory to beta2=0.99 helps the refreshing optimizers, though for canonical GaLore the effect is small and a full-rank control reverses it. One measurable fact, subspace non-identifiability, clarifies why GaLore works, which patches work, and what to check before trusting a low-rank assumption: the reproducible rank k*.

• The paper identifies that beyond a small stable core (k* ≈ 39), the top-r gradient subspace behaves as a near-random sample at each refresh.
• It demonstrates through spectral and statistical analysis that averaging gradients does not stabilize the subspace due to a flat, power-law spectral tail.
• The study shows that transporting optimizer moments and lowering β₂ effectively mitigate misalignment, improving performance in low-rank training.

Non-Identifiability of Low-Rank Gradient Subspaces in GaLore-Family Optimizers

Summary and Motivation

This work provides a detailed empirical and theoretical investigation of the geometric assumptions underlying memory-efficient optimizers such as GaLore, LDAdam, and their variants, which are widely used for low-rank training of large neural LLMs. These optimizers frequently project gradients onto a top-$r$ singular subspace, periodically refreshed, with the premise that this subspace encodes stable, slowly evolving directions of descent. The central thesis of the paper is that this premise is incorrect at the ranks used in practice: above a small stable core of directions, the top-$r$ gradient subspace is essentially a random sample at each refresh, dominated by noise, and cannot be meaningfully tracked across time. This non-identifiability is demonstrated to be a robust empirical phenomenon across architectures, model families, and scales, and the study provides both spectral and statistical explanations for its occurrence.

Empirical Investigation of Subspace Stability

The measured rotation between top-$r$ subspaces before and after a refresh nearly saturates the geometric maximum ($96.7$–$99.6\%$ of $\sqrt{2r}$ across ranks), indicating that each refresh returns an almost orthogonal frame with respect to the previous one. This instability is clearly visualized based on principal-angle analysis.

Figure 1: Subspace rotation per refresh (Pythia-1B) as a function of rank: the rotation nearly attains the maximal distance between two random orthonormal frames.

A critical control experiment computes two estimates of the top-$r$ subspace at the same step from disjoint minibatches and finds their disagreement is nearly as large as that observed across time—corroborating that the lack of stability is due to estimator noise, not dynamics.

Figure 2: Same-step subspace disagreement (normalized) demonstrates that the majority of subspace mismatch is due to estimator noise, not temporal drift.

Direct measurement establishes that of $r=128$ directions, on average only $k^\star \approx 39$ are reproducible across minibatch splits; the remainder are essentially re-sampled at each evaluation, confirming a strong non-identifiability beyond this stable core.

Spectral Analysis and the Failure of Averaging

The singular spectrum of gradient matrices is found to have a small dominant spike (order $k^\star$), followed by a flat, power-law-tailed bulk with no spectral gap at the cut-off rank $r$0 typically used for projection.

Figure 3: Gradient singular spectrum in two Pythia models; at the chosen ranks, the absence of a spectral gap precludes statistical identifiability of the subspace.

Because of the lack of separation, the subspace projection is maximally sensitive to noise. The Davis–Kahan/Wedin $r$1 theorems provide theoretical backing: when the gap between singular values vanishes, the SVD output becomes a Haar-random draw in the degenerate subspace, explaining the observed redraw.

Efforts to stabilize the subspace by averaging across multiple gradients or refresh steps do not succeed. Averaging only shrinks the spectral tail as $r$2 (where $r$3 is the averaging budget), instead of $r$4 as for pure noise, because the tail comprises genuine gradient signal with a power-law distribution.

Figure 4: Averaging over multiple matrices fails to make the top-$r$5 subspace identifiable; the subspace disagreement decays much more slowly than would be predicted by noise averaging.

Implications for Optimizer State and Training Performance

While the instability of the subspace might suggest catastrophic performance, GaLore-family optimizers retain a nearly constant fraction of gradient energy (typically $r$6–$r$7), regardless of which directions are selected within the flat tail. This explains why optimizer efficacy is robust to basis turnover.

However, the optimizer moments (e.g., Adam's moving averages) are affected differently:

• The first moment can be optimally transported across basis changes via the rotation matrix, which underpins the approach in LDAdam.
• The second moment, when simply carried (unchanged) across refreshes, can become badly misaligned, provably incurring an error $r$8 times above the optimal transportable estimator.

Empirical tests show that transporting optimizer moments across refreshes (as in LDAdam), and lowering the second-moment decay parameter ($r$9) from the standard $r$0 to $r$1, both yield significantly lower final validation perplexities. These effects are optimizer-specific and disappear in full-rank Adam, which does not experience redraws.

Figure 5: Lowering $r$2 from $r$3 to $r$4 for various refreshing optimizers improves performance substantially; the effect is absent in a full-rank baseline, confirming its connection to redraws.

Notably, basis averaging/EMA of the subspace does not improve performance—the real engineering lever is the tracking of optimizer state, not the subspace itself.

Generality and Limitations

The lack of gradient subspace identifiability is shown to be general across:

• Transformer architectures (GPT-NeoX, GPT-2, Llama-class, Vision Transformers)
• Model scales (70M to 6.9B parameters)
• Training runs and hardware backends

It becomes more pronounced with model scale, as the reproducible core $r$5 shrinks in absolute terms. The phenomenon is intrinsic to the gradient geometry and not introduced by the use of low-rank optimizers.

Theoretical and Practical Implications

Theoretical: The results clarify that no finite averaging budget or sophisticated tracking scheme can extract a persistent low-rank subspace due to the intrinsic gapless structure of the gradient spectrum at practical ranks. Instead, only a small core is statistically recoverable.

Practical:

• Reliance on subspace tracking for momentum and variance accumulation (e.g. in Adam) leads to stale or misaligned state.
• Remedy for the instability consists of transporting moments through each basis rotation, and reducing $r$6 to reduce the memory span of the moving average, aligning the effective timescale of the state with the frequency of subspace redraws.
• Basis averaging is ineffective.
• Measurement of $r$7 (reproducible directions) should precede any reliance on a rank assumption in practice.

Future Outlook: The identified non-identifiability implies that future low-rank optimizers should abandon persistent subspace-tracking at high ranks in favor of dynamically re-drawn or randomized subspaces, with state transportation as a necessary feature. Adaptations that estimate and match $r$8 or incorporate spectral information could avoid overfitting projection rank. The observed robustness of energy capture to subspace turnover may motivate further exploration of randomized or hybrid projections for efficient LLM training.

Conclusion

This paper refutes the common geometric intuition underlying low-rank projection optimizers by demonstrating the statistical non-identifiability of the gradient subspace at practical ranks. The refresh at each interval effectively re-draws a new, nearly orthogonal subspace, with only a minor stable core. The efficacy of GaLore-type methods is thus not due to successful subspace tracking, but to persistent capture of gradient energy by arbitrary frames within the bulk. Optimizer state, not frame stability, is the bottleneck. Remedies consisting of state transport and appropriate memory decay constants (smaller $r$9) are theoretically motivated and empirically validated, providing clear practical recommendations for scalable optimizer design in large-scale neural models.

---

Reference:

Noel Thomas, "No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Training" (2607.05872)