LDAdam: Memory-Efficient Adaptive Optimizer

• LDAdam is a memory-efficient optimizer that compresses Adam states into dynamically selected low-dimensional subspaces, reducing memory costs without altering trainable parameters.
• It employs projection-aware update rules and generalized error feedback to transport optimizer states accurately across basis redraws during training.
• Empirical results and theoretical analysis demonstrate that LDAdam matches or exceeds Adam’s performance while significantly reducing memory usage in both fine-tuning and pre-training.

Searching arXiv for the specified LDAdam papers and closely related low-rank optimizer work. LDAdam is a memory-efficient optimizer for training large models that performs adaptive optimization steps within lower dimensional subspaces while consistently exploring the full parameter space during training. It compresses Adam-type optimizer states into dynamically selected rank-$r$ subspaces, uses projection-aware update rules for transitions between subspaces, and integrates a generalized error feedback mechanism that explicitly accounts for both gradient and optimizer state compression (Robert et al., 2024). A later analysis recast its central mechanism in terms of non-identifiability: beyond a small reproducible core, the top-$r$ gradient subspace is not a slowly drifting object that can be tracked, so LDAdam’s practical advantage comes from transporting optimizer state correctly across repeated basis redraws rather than from tracking a stable low-rank subspace (Thomas, 7 Jul 2026).

1. Definition, scope, and memory model

Training very large models is memory-intensive mainly because optimizers such as Adam store first- and second-order gradient statistics for every parameter, doubling or more the memory required for the model parameters themselves. LDAdam seeks to resolve this bottleneck by compressing optimizer states, retaining adaptivity, and supporting both full pre-training and fine-tuning. Unlike many PEFT/adapter approaches, it does not restrict or alter which parameters can be trained (Robert et al., 2024).

The method stores optimizer information in low-dimensional subspaces of rank $r \ll d$. This reduces auxiliary memory cost while preserving an Adam-like adaptive update. In the 2024 formulation, the optimizer is presented as a convergence-guaranteed, memory-efficient Adam-type method for practical language-model pre-training and fine-tuning; in the 2026 analysis, it is also presented as a correction to the stale-state pathology of earlier low-rank optimizers when the projection basis is refreshed frequently (Robert et al., 2024).

The memory comparison reported for Llama-2 7B at $r=32$ is as follows.

Optimizer	Optimizer State Size	Example
Adam	$2nm$	25.1 GB
LDAdam	$nr + 2rm$	1.2 GB
GaLore	$nr + 2rm$	1.2 GB

This compression model is central to LDAdam’s design: the memory saving is obtained not by freezing parameters or moving to a decentralized communication regime, but by maintaining adaptive statistics in a low-dimensional coordinate system that changes during training (Robert et al., 2024).

2. Low-dimensional adaptive update rule

For a parameter matrix $\theta \in \mathbb{R}^{n \times m}$, LDAdam defines at step $t$ an orthonormal projection matrix $U_t \in \mathbb{R}^{n \times r}$ selecting a rank-$r$0 subspace. If the stochastic gradient is $r$1, the optimizer works with a projected quantity

$r$2

where $r$3 is an accumulated gradient/error buffer (Robert et al., 2024).

The first-moment update is projection-aware:

$r$4

The parameter update then lifts the low-dimensional adaptive step back to the original parameter space:

$r$5

The second-moment estimator is likewise translated across basis changes:

$r$6

These formulas are intended to preserve the statistical meaning of Adam’s moments when the coordinate system itself changes (Robert et al., 2024).

Subspace adaptation is part of the algorithm rather than a preprocessing step. The reported implementation uses block power iteration with warm starts rather than a costly exact SVD. The interpolation

$r$7

balances past optimizer state and the new gradient-like signal, and $r$8 is set to approximate the left singular vectors of $r$9 via block power iteration. This makes LDAdam a dynamically reprojected optimizer: the subspace may change every step, yet the adaptive state remains meaningful because it is explicitly translated into the new basis (Robert et al., 2024).

3. Projection-aware state transport and generalized error feedback

The algorithmic novelty of LDAdam lies in how it treats optimizer state when the low-rank basis changes. In the low-rank optimizer template used by the GaLore family, at step $r \ll d$0 a gradient $r \ll d$1 is projected onto the top-$r \ll d$2 left singular subspace: one computes $r \ll d$3, projects each gradient as $r \ll d$4 for the next $r \ll d$5 steps, and applies Adam in the low-dimensional space. The difficulty is that Adam’s moments are coordinate-dependent, so a refreshed basis creates a state-transport problem (Thomas, 7 Jul 2026).

LDAdam addresses that problem directly. For the first moment, the transported state follows the coordinate change exactly:

$r \ll d$6

For the second moment, the reported rule maps variances through the squared entries of the rotation:

$r \ll d$7

where $r \ll d$8 are the previous variances. In the 2026 analysis, this is justified as the minimum-variance unbiased estimator for the new-basis variances, while the first-moment rule is identified as the minimum-MSE linear estimator under isotropic gradients (Thomas, 7 Jul 2026).

LDAdam also incorporates generalized error feedback. It maintains a high-dimensional error buffer $r \ll d$9 that stores the residual discarded in projection for both gradients and optimizer-state projections. The update is

$r=32$0

The first term is the current step’s projection error for the accumulated gradient. The second term corrects the loss of information that occurs when the momentum buffer itself is projected into a new subspace. To avoid extra allocation, the error buffer is stored in place of the gradient accumulator in standard training frameworks, with a tradeoff that unsupported features such as gradient clipping may arise in some cases (Robert et al., 2024).

This combination of state transport and generalized error feedback distinguishes LDAdam from low-rank methods that refresh the basis but carry optimizer state blindly.

4. Non-identifiability of the low-rank subspace

A central result of the 2026 analysis is that the top-$r=32$1 subspace of the gradient is not statistically identifiable beyond a small reproducible core. Two estimates of the top-$r=32$2 subspace computed at the same step from disjoint minibatches disagree almost as much as estimates computed $r=32$3 steps apart: at Pythia-160M with $r=32$4, the same-step disagreement is $r=32$5, while the across-time disagreement is $r=32$6, where $r=32$7 is the maximal chordal distance between two orthonormal frames (Thomas, 7 Jul 2026).

The chordal distance is reported as

$r=32$8

with $r=32$9 the principal angles between consecutive bases. The observed near-maximal disagreement indicates that the apparent basis rotation at refresh time is dominated by estimator noise rather than by coherent temporal drift (Thomas, 7 Jul 2026).

The same phenomenon is reported across four model families in three architecture classes from 70M to 6.9B parameters, strengthening with scale, and more weakly in a vision transformer. Only about $2nm$0 of 128 directions are reproducible across batches. The paper calls this reproducible component the “reproducible core,” and proposes the reproducible rank $2nm$1 as the measurable quantity to check before trusting a low-rank assumption (Thomas, 7 Jul 2026).

A related observation is that the gradient spectrum has no gap at the working rank. The spectral tail decays under $2nm$2-fold averaging as $2nm$3 rather than the $2nm$4 expected for pure noise, so no averaging budget makes the subspace well defined. Empirically, even a 10,000-step basis average reduces perplexity by at most 0.36 and still leaves basis rotation at approximately 84% of the geometric maximum per refresh. This directly contradicts the common assumption that a stable top-$2nm$5 subspace can be recovered by more aggressive averaging (Thomas, 7 Jul 2026).

5. Theoretical characterization

The 2024 LDAdam paper proves convergence under standard assumptions: the loss is $2nm$6-smooth and lower bounded, stochastic gradients are unbiased and uniformly bounded, gradient variance is bounded, and the compression operator satisfies

$2nm$7

for some $2nm$8. In the non-convex case, the reported bound is

$2nm$9

Under a Polyak–Lojasiewicz condition,

$nr + 2rm$0

the paper reports

$nr + 2rm$1

The stated interpretation is that LDAdam matches the asymptotic rates of established optimizers such as AMSGrad up to compression-dependent constants, without restrictive stable-rank assumptions (Robert et al., 2024).

The 2026 paper adds a distinct theoretical message. If the subspace is redrawn at refresh time and shares few directions with the previous one, blind carry of the optimizer state is erroneous. For second moments, blind carry is provably about

$nr + 2rm$2

worse than the best rotation-blind estimator; for $nr + 2rm$3 and $nr + 2rm$4, this is approximately 45 times suboptimal. By contrast, first-moment transport through $nr + 2rm$5 is exact through the rotation and is the optimal linear map under isotropic gradients, while the squared-rotation rule is the optimal estimator for the new-basis variances (Thomas, 7 Jul 2026).

Taken together, these results support two distinct but compatible views of LDAdam. The 2024 paper presents a compressed adaptive optimizer with convergence guarantees and generalized error feedback. The 2026 paper suggests a more specific interpretation: when the top-$nr + 2rm$6 subspace is non-identifiable, the essential requirement is correct coordinate transport of optimizer state rather than any attempt to track a persistent low-rank object.

6. Empirical behavior, comparisons, and related terminology

The reported empirical record covers both fine-tuning and pre-training. In fine-tuning, RoBERTa-base on GLUE is reported to reach an average score of 86.48 for LDAdam versus 86.28 for Adam, with memory cost reduced from 0.46GB to 0.15GB. On Llama-2 7B for GSM8K, LDAdam at $nr + 2rm$7 and $nr + 2rm$8 is reported to achieve accuracy close to Adam and to outperform GaLore at small ranks, with peak memory reduced from 55.34GB for Adam to approximately 32GB for LDAdam with $nr + 2rm$9. In pre-training on C4 with Llama 130M, 350M, and 1.3B, validation perplexity is reported as on par with or slightly better than Adam, including 17.30 for LDAdam at $nr + 2rm$0 versus 18.08 for Adam on Llama 350M; at ultra-high compression $nr + 2rm$1, only LDAdam with error feedback is reported to converge, while GaLore and LDAdam without error feedback diverge or stall (Robert et al., 2024).

The 2026 analysis gives a more targeted comparison on Pythia-1B over 40k steps and three seeds. At the basis refresh, carrying state blindly and transporting it by the LDAdam rule yield the following validation perplexities (Thomas, 7 Jul 2026).

State at refresh	$nr + 2rm$2	$nr + 2rm$3
Carry (GaLore)	22.07 ± 0.17	19.28 ± 0.18
Transport (LDAdam rule)	18.73 ± 0.13	16.92 ± 0.16

This comparison supports several practical prescriptions stated in the paper: transport optimizer state through the basis rotation at every refresh; lower the second moment’s memory, for example from $nr + 2rm$4 to $nr + 2rm$5, because relevant statistics become obsolete quickly; do not expect subspace averaging to succeed; and measure $nr + 2rm$6 before relying on a low-rank assumption (Thomas, 7 Jul 2026). A plausible implication is that the effective rank for optimization is not the nominal working rank $nr + 2rm$7 but the identifiable portion of it.

A common source of confusion is nomenclature. LDAdam should be distinguished from DADAM, the consensus-based distributed adaptive moment estimation method for online optimization over a decentralized network. DADAM is a peer-to-peer decentralized optimizer with a mixing matrix $nr + 2rm$8, local gradients, and dynamic-regret guarantees; it addresses communication topology and decentralized data access rather than low-dimensional optimizer-state compression (Nazari et al., 2019). The similarity of names does not indicate a shared mechanism.

Within the low-rank optimizer literature, LDAdam is therefore best understood as an Adam-type method whose distinctive feature is projection-aware state transport supplemented by generalized error feedback. The later non-identifiability results suggest that its success does not validate a strong low-rank tracking hypothesis; instead, they identify the reproducible rank $nr + 2rm$9 as the quantity that determines whether low-rank optimizer statistics can be trusted and why transport-based corrections matter.