---
title: Low-Memory Optimization (LOMO) Methods
url: https://www.emergentmind.com/topics/low-memory-optimization-lomo-bc6c1774-9fa1-4a9c-8ba1-928ea5517524
type: topic
---

# Low-Memory Optimization (LOMO) Methods

Low-Memory Optimization (LOMO) encompasses a diverse set of algorithmic, architectural, and systems-level techniques designed to enable efficient optimization of modern large-scale machine learning models under stringent memory constraints. LOMO methods span a wide range of domains, including deep learning, nonsmooth optimization, embedded systems, and resource-constrained distributed settings. They are unified by the principle of reducing auxiliary memory required for optimizer state, activations, intermediate buffers, or system-level data structures, while preserving accuracy and convergence guarantees. Recent advances in LOMO have enabled full-parameter fine-tuning of multi-billion parameter models on commodity GPUs, on-device adaptation of large language models, memory-optimal federated learning, and highly resource-efficient inference and training regimes.

## 1. Historical Foundations in Subgradient and Convex Optimization

Classical low-memory optimization strategies originated in large-scale convex and nonsmooth optimization, where memory limits necessitated algorithms with only O(n) or O(n+m) auxiliary storage—far less than required for Hessians, full gradient history, or large cutting-plane bundles. Notable developments include:

- **Shor’s Normalized Subgradient Method:** Maintains only the current iterate and one subgradient vector; requires O(n) storage and guarantees suboptimality within λ(1+ε)/2 for any ε>0 [1902.01572].
- **Nesterov’s Smoothing and Accelerated Methods:** Applies adaptive smoothing and acceleration to nonsmooth objectives, storing only the current point, a dual variable, and a subgradient, yielding O(n+m) memory and O(1/ε) complexity [1902.01572].
- **Universal Accelerated and Mirror-Prox Methods:** Provide optimal black-box rates for Hölder continuous objectives and monotone variational inequalities, with minimal per-iteration storage [1902.01572].
- **Adaptive Mirror Descent (Primal-Dual):** Solves constrained and variational problems using only a running iterate, (sub)gradient, and small auxiliary state, achieving O(1/ε²) iteration complexity with O(n+m) storage [1902.01572].

These foundational techniques established that, even for high-dimensional and nonsmooth problems, efficiency and optimality could be attained with sub-linear or linear memory overhead.

## 2. System-Level and Architectural LOMO: Memory Partitioning and Reversible Networks

System-level LOMO methods exploit operating system or hardware-level mechanisms to partition, isolate, or reconstruct memory resources:

- **Virtual Memory Node Partitioning:** Lim et al. introduced a scheme that partitions physical memory into k virtual nodes (“vnodes”), each dedicated to a process class (e.g., critical vs non-critical applications) [2101.10707]. Formally, $\sum_{i=1}^k M_i = M$ (total memory), with strict caps $T_i = M_i$ per vnode. Local memory pressure is handled within vnode boundaries via selective reclamation. This prevents the activation of global low-memory killers, dramatically increasing free memory (+670 MB), reducing process launch latency (–1015 ms), and fully eliminating system-wide LMK/OOMK events under workload [2101.10707].
- **Reversible Transformer Networks:** In large model fine-tuning, RevFFN introduces reversible blocks that split activations, performing coupled forward updates $Y_1 = X_1 + f(X_1, X_2)$, $Y_2 = X_2 + g(Y_1)$, enabling reconstruction of original activations during the backward pass without caching all intermediates [2512.20920]. This achieves a 2× reduction in activation memory and enables full-model fine-tuning (e.g., 2.7B Mixture-of-Experts LLMs) on a single 80 GB GPU, outperforming activation checkpointing and distributed-sharding frameworks [2512.20920].

These approaches shift parts of the memory management problem to system primitives or invertible computation, offloading or reconstructing information as needed to stay within hardware bounds.

## 3. Optimizer-Centric LOMO: State Compression, Fusion, and Block Selection

Mainstream LOMO research focuses on redesigning optimizer state management and parameter updates to minimize memory overhead without sacrificing statistical adaptivity:

- **Optimizer State Compression and Factorization:**
  - *Extreme Tensoring (ET):* Diagonal preconditioners (e.g., AdaGrad) are factorized as Kronecker products of mode-wise marginal accumulators, reducing O(d) memory to O(k·d^{1/k}). With $k=2$ or 3, ET matches Adam’s performance at 1000× less memory on ResNet-18 and transformer tasks [1902.04620].
  - *Adafactor and SMMF:* Both maintain low-rank or factored statistics for second moments (e.g., NNMF on square-matricized tensors in SMMF, minimizing auxiliary state to $\sim$2($\hat n{+}\hat m$) per tensor) with up to 96% memory reduction compared to Adam, and maintain performance parity across CNNs and transformers [2412.08894].
  - *CAME:* Supplements Adafactor with a confidence-guided momentum scaling, stabilizing updates via per-block moving-average factorization of instantaneous instability, storing only four vectors per block [2307.02047].
  - *Low-rank Momentum Factorization (MoFaSGD):* Approximates first order moment with an adaptive rank-$r$ SVD, updating thin factors via efficient QR/SVD operations, requiring O((m+n)r) memory (e.g., 66% saving over AdamW for r=32) [2507.08091].
- **Fused Backward-Update Loops:** LOMO fuses gradient computation and parameter update in a single backward pass, so that no full gradient tensor is ever materialized; peak memory is O(1) in the number of layers for gradient storage [2306.09782]. AdaLomo extends this by adding an NNMF-compressed adaptive learning rate per parameter; grouped update normalization ensures stability and recovers AdamW-level convergence [2310.10195].
- **Block-Coordinate Descent (BlockLLM):** Updates are performed only on selected parameter blocks (typically transformer layers) with highest normalized gradient magnitude, updating <$5\%$ of parameters and optimizer state per step. Memory scales as $[1+3(1-s)]n\,b$ for sparsity $s$, with empirical VRAM savings of 13% vs. rank-8 GaLore and accuracy parity [2406.17296].

Combinations of these methods—statistical adaptivity, low-rank/matrix compressions, per-block or vnodal isolation, fusion—underpin almost all current LOMO optimizers.

## 4. Activation and Forward Memory Minimization: Token/Fine-Grained Sparsity and Low-Precision

Beyond optimizer state, LOMO targets reduction of activation and intermediate buffer memory, critical in long-context and extreme classification:

- **Token Elimination and Contextual Sparsity (LeMo):** By profiling attention scores at the block level and training tiny predictors of block informativeness, only $r\approx0.52$ token fraction is retained per layer; memory and step speed improve by up to 1.93× and 1.36× over LoRA/LongLoRA without notable loss in perplexity or accuracy for sequences up to 16K [2501.09767].
- **Gradient and Update Fusion in Low Precision (ELMO, ME-MPO):** ELMO eliminates both FP32 master weights and DRAM-resident gradient/momentum buffers for gigantic classification heads (e.g., 3M labels). All forward/updates are carried out in BF16/FP8 with Kahan summation/stochastic rounding, and chunked kernel-fused updates reduce peak memory to 6.6 GB (vs 39.7 GB baseline) with less than 1% accuracy loss [2510.11168]. ME-MPO generalizes this by representing parameters as FP16+$e$ “extra bits” and fusing optimizer steps into backward hooks, yielding 16–25% lower memory [2309.12381].

These approaches exploit the redundancy in token and label space, as well as modern hardware support for low-precision arithmetic, to minimize transient memory use.

## 5. LOMO in Distributed, Federated, and Zeroth-Order Environments

LOMO directly addresses federated and on-device learning, where training must proceed under strict per-device memory envelopes:

- **Federated Foresight Pruning + BP-Free Training:** Devices compute per-parameter saliency via Monte Carlo NTK-based scores; pruned networks retain only the most influential parameters [2405.04765]. Zeroth-order (Stein’s identity) optimization then runs on the sparse subnet, reducing peak memory up to 9× over full backprop and lowering FLOPs by 5–10× [2405.04765]. Empirically, accuracy loss relative to FedAvg is within 3–5% at 90% sparsity.
- **Low-Rank Jacobian Backpropagation with Error Feedback (GradLite):** LLM fine-tuning under severe memory budgets employs low-rank projections $J_t \approx U_t V_t^T$, storing only O(k) activations per layer and compensating for projection bias via error feedback [2510.22467]. Empirical peak VRAM drops 41–50% vs. checkpointing; theoretical convergence guarantees are preserved.
- **Memory Partitioning in Embedded Systems:** In Android environments, fixed vnode boundaries and CPU affinity ensure applications never trigger system-level OOM, crucial for reliability [2101.10707].

These advances have made memory-optimal collaborative learning feasible for edge, mobile, and federated networks.

## 6. Formal Analysis, Regret, and Convergence Properties

Theoretical guarantees are central to the credibility of LOMO:

- **Regret and Convergence Bounds:**
  - *SMMF, Adafactor, Extreme Tensoring, MoFaSGD*: All match the best-known O(1/√T) or O(1/ε) rates of their full memory-consuming analogues under standard stochastic or deterministic oracles [1902.04620, 2507.08091, 2412.08894].
  - *GradLite*: Low-rank backprop with error feedback maintains unbiasedness and achieves the same mean convergence guarantee as vanilla SGD/Adam [2510.22467].
- **Empirical-Statistical Trade-offs:** Adafactor, AdaLomo, MoFaSGD, and CAME show performance degradation only if the factorization/compression rank is too aggressive relative to parameter structure; all provide detailed ablation studies justifying memory/accuracy trade-offs [2310.10195, 2507.08091, 2412.08894, 2307.02047].
- **Practical Memory Formulas:** For optimizers, per-layer state scales as (parameters) + (O(factorization state)), with auxiliary buffer reduction typically 2–10× depending on technique and rank/pattern. For activations, reductions of 2–10× are common with sparse token involvement or reversible architectures.

LOMO thus bridges the gap between theoretical optimization and practical system limitations.

## 7. Applications, Limitations, and Future Outlook

LOMO is now critical to full-parameter fine-tuning of LLMs, on-device adaptation, federated learning, and large-output extreme classification. Standard practices include:

- *Combining LOMO with ZeRO/factorized sharding or activation checkpointing* for end-to-end minimization of parameter, state, and activation memory [2306.09782, 2310.10195].
- *Practical tuning*: memory/fidelity can be dialed in via factorization rank, block size, or token sparsity threshold; factorization overhead is negligible for suitable k, and practical implementations routinely profile or ablate these parameters [2507.08091, 2412.08894, 2501.09767].
- *Limitations*: Most LOMO schemes trade memory for additional compute, especially in backprop recomputation or matrix/tensor factorization. Some methods (e.g., activation pruning) may degrade compatibility with inflexible hardware layouts or black-box accelerator stacks [2501.09767]. Certain extremely large-batch/federated scenarios still demand further research in stabilizing convergence.

A plausible implication is that continued progress in LOMO, including hybrid quantization, dynamic adaptive partitioning, and hierarchical or multi-scale factorization, will be foundational for the next generation of resource- and privacy-aware AI systems at all scales.

Source: https://www.emergentmind.com/topics/low-memory-optimization-lomo-bc6c1774-9fa1-4a9c-8ba1-928ea5517524