---
title: APOLLO Optimizer Innovations
url: https://www.emergentmind.com/topics/apollo-optimizer
type: topic
---

# APOLLO Optimizer Innovations

The APOLLO Optimizer refers to a set of modern optimization frameworks and algorithms sharing the “APOLLO” designation, advancing state-of-the-art training efficiency, memory scaling, and solution quality for both deep learning and structured combinatorial optimization. Notable APOLLO optimizers include (i) APOLLO for nonconvex stochastic optimization, a diagonal quasi-Newton scheme, and (ii) APOLLO for memory-efficient large-scale language model pre-training, which achieves near-SGD memory costs while matching AdamW’s convergence. APOLLO also refers to an alternating predictor-corrector framework (Apollo-MILP) for mixed-integer linear programming. Each variant addresses distinct, domain-specific challenges via tailored algorithmic innovations.

## 1. APOLLO for Memory-Efficient Large-Scale Neural Network Optimization

The “APOLLO” optimizer for large language models (LLMs) [2412.05270] solves the critical bottleneck of optimizer state memory when training transformer-based architectures with towering parameter counts. AdamW requires storage of full first ($M_t$) and second moment ($V_t$) estimates per parameter, incurring a $2mn$-float overhead for parameter matrices $W \in \mathbb{R}^{m \times n}$. APOLLO approximates such element-wise adaptive scaling by tracking first and second moment statistics in a much lower-dimensional subspace, leveraging structured channel-wise scaling and random projection for efficiency.

### Algorithmic Structure

For parameter matrix $W \in \mathbb{R}^{m \times n}$ and batch gradient $G_t$, APOLLO executes the following core steps:
1. Project gradient: $R_t = P_t G_t$, $P_t \in \mathbb{R}^{r\times m}$, sampled as a Gaussian random projection, $r \ll m$.
2. Update AdamW-like low-rank moments: $M^R_t, V^R_t \in \mathbb{R}^{r\times n}$.
3. Compute channel-wise scaling: for each $j$, $s^R_{t,j} = \|\hat{R}_t[:,j]\|_2 / \|R_t[:,j]\|_2$ using $\hat{R}_t = M^R_t / (\sqrt{V^R_t} + \epsilon)$.
4. Apply adaptive update: $W_{t+1} = W_t - \eta\alpha G_t S_t$, where $S_t = \mathrm{diag}(s^R_t)$, $\alpha \approx \sqrt{n/r}$ corrects random projection bias.

A “rank-1” variant, APOLLO-Mini, collapses the projection to $r=1$, producing a single scaling factor per parameter matrix and reducing the optimizer state to $2n+2$ floats, matching that of SGD with momentum.

### Memory and Computational Analysis

| Optimizer      | Optimizer state size   | SVD required | Per-iteration complexity     |
|---------------|-----------------------|--------------|------------------------------|
| AdamW         | $2mn$                  | No           | $O(mn)$                      |
| GaLore/Fira   | $mr + 2nr$ (plus SVD)  | Yes          | $O(mr) + O(r^3)$ per SVD     |
| APOLLO        | $2nr + 2$              | No           | $O(nr)$ (projection/update)   |
| APOLLO-Mini   | $2n + 2$               | No           | $O(n)$                       |

APOLLO eliminates full-matrix moment tracking and periodic SVD costs (as in GaLore or Fira), maintaining computational simplicity and parallelizability.

### Empirical Performance and System Impact

Across LLaMA pre-training experiments (7B/13B models), APOLLO matches or exceeds AdamW in perplexity, enables 3× larger batch sizes (and corresponding throughput) on the same GPUs, and permits naively distributed (DDP) or single-GPU 7B pre-training under 12GB of memory with INT8 quantization. Performance remains robust even at $r=1$ (APOLLO-Mini), where the optimizer state cost approaches zero.

APOLLO’s channel-wise scaling empirically matches or slightly outperforms full element-wise AdamW on language modeling and fine-tuning (e.g., MMLU), even under aggressive memory constraints.

## 2. APOLLO: Diagonal Quasi-Newton for Stochastic Nonconvex Optimization

A separate “APOLLO” optimizer targets nonconvex stochastic objectives in deep learning [2009.13586], introducing a diagonally parameterized quasi-Newton update for scalable curvature adaptation.

### Core Update Rule

Classical quasi-Newton algorithms (e.g., BFGS) maintain a dense Hessian approximation, which is infeasible ($O(d^2)$) and potentially indefinite for high-dimensional, nonconvex objectives. APOLLO’s diagonal approximation updates, given by
\[
B_{t+1} = B_t + \Lambda, \quad \Lambda = \frac{s_t^\top y_t - s_t^\top B_t s_t}{\lVert s_t \rVert_4^4} \mathrm{Diag}(s_t^2),
\]
enforce a “weak” secant condition parameter-wise. Negative or near-zero eigenvalues are addressed using the rectification
\[
D_t = \mathrm{rectify}(B_t, \sigma) = \max(|B_t|, \sigma),
\]
ensuring positive-definite curvature for all updates.

### Full Algorithm

Each iteration involves:
- Momentum and exponential moving average (EMA) for gradients.
- Diagonal quasi-Newton update of $B_t$ as above.
- Gradient preconditioning by $D_t^{-1}$.
- Parameter update $\theta_{t+1} = \theta_t - \eta_{t+1} d_{t+1}$ with $d_{t+1}=D_{t+1}^{-1} m_{t+1}$.

Memory overhead is $O(4d)$ vs. $O(3d)$ for Adam, $O(2d)$ for SGD.

### Convergence and Generalization

In convex online settings, APOLLO achieves $O(\sqrt{T})$ regret, matching adaptive optimizers such as Adam and RMSProp. In nonconvex stochastic optimization, APOLLO attains expected gradient norm minimization rates of $O(\log T/\sqrt{T})$, also matching Adam-type algorithms. The scale hyperparameters $\eta$ and $\sigma$ are coupled, simplifying tuning.

Across CIFAR-10 (ResNet-110), ImageNet (ResNeXt-50), One Billion Word (2-layer LSTM), and WMT’14 En→De (Transformer), APOLLO delivers fast convergence and competitive or superior test accuracy, test perplexity, or BLEU versus SGD, Adam, RAdam, AdaBelief, and AdaHessian.

## 3. APOLLO-MILP: Alternating Prediction-Correction for Mixed-Integer Linear Programming

In mixed-integer linear programming (MILP), the APOLLO framework [2503.01129] refers to Apollo-MILP, which combines machine learning–driven prediction with combinatorial correction steps, iteratively reducing problem size while preserving solution quality and feasibility.

### Predict-and-Correct Workflow

At each iteration $k$, maintain reduced instance $\mathcal{I}^{(k)}$ with subset of variables $P^{(k)}$ fixed:
1. **Prediction**: Use GNN-based predictor $p_\theta$ to compute marginal probabilities for unfixed variables. Extract partial solution $\hat{x}^{(k)}[P]$ via thresholding $k_1$ largest and $k_0$ smallest.
2. **Correction**: Solve trust-region sub-MILP around $\hat{x}^{(k)}[P]$,
   \[
   \min_x c^\top x \quad s.t.\quad Ax \leq b,\, \ell \leq x \leq u,\, \|x[P] - \hat{x}[P]\|_1 \leq \Delta,
   \]
to obtain a reference solution $\tilde{x}^{(k)}$.
3. **Fixing/Reduction**: Fix indices where $\hat{x}_i = \tilde{x}_i$ (high confidence), then update $P^{(k+1)}$, further reducing $\mathcal{I}^{(k+1)}$.

This cycle continues until all integer variables are fixed or resource limits are met.

### Uncertainty-Based Error Upper Bound (UEBO)

The uncertainty of predictions is evaluated through an upper bound on the (intractable) KL divergence between predicted and true marginals:
\[
D_{KL}\bigl(p_\theta(x_i|\mathcal{I}) \| q(x_i|\mathcal{I})\bigr) \leq H(p_\theta(x_i|\mathcal{I})) + d(p_\theta, q),
\]
where $H(\cdot)$ is entropy and $d(\cdot,\cdot)$ is prediction-correction discrepancy. In practice, single-point prediction-correction consistency $\mathbb{I}[\hat{x}_i=\tilde{x}_i]$ is used to guide variable fixing.

### Empirical Results and Guarantees

On combinatorial auctions, set covering, item placement, and workload apportionment benchmarks, Apollo-MILP achieves up to 77.8% reduction in absolute gap versus Gurobi, 70.2% versus SCIP, and closes 100% of the gap on hard real-world datasets. The algorithm maintains feasibility at each reduction step and achieves faster and lower primal gap convergence versus all tested baselines.

## 4. Comparison of Methodologies and Domains

| Variant         | Domain            | Main Technique             | Core Memory/Complexity Benefit                        |
|-----------------|------------------|----------------------------|------------------------------------------------------|
| APOLLO (LLM)    | Deep learning    | Channel/tensor-wise scaling via low-rank random projection | $O(nr)$ or $O(n)$ optimizer state; no SVD            |
| APOLLO (quasi-Newton) | Deep learning | Diagonal quasi-Newton with rectified update          | $O(4d)$ state; diagonal curvature adaptation         |
| APOLLO-MILP     | MILP/combinatorial| Alternating ML prediction and trust-region correction | Problem-size reduction while preserving feasibility  |

While all variants address scaling—memory or combinatorial dimension—they are specialized for different mathematical problem classes: APOLLO for LLMs prioritizes memory-optimal adaptive learning; APOLLO (quasi-Newton) targets curvature adaptation in stochastic optimization; APOLLO-MILP applies prediction-correction with uncertainty quantification for MILP.

## 5. Practical Implications and System-Level Benefits

The memory and computational gains of APOLLO (LLM) have significant practical consequences:
- Training of foundation models (e.g., LLaMA-13B) becomes feasible on mainstream (A100-80GB) hardware without aggressive model sharding or optimizer state offloading.
- On low-end GPUs, pre-training models up to 7B parameters with INT8 quantization is achieved using a single 12GB card.
- Batch size scaling is unlocked: APOLLO enables up to 3× throughput versus AdamW owing to optimizer state reduction and removal of SVD-induced stalls (GaLore/Fira).

APOLLO-MILP’s predictor-corrector logic yields substantial problem-size reductions and improved solution quality without risking infeasibility—a critical property for industrial and scientific MILP deployments.

## 6. Limitations, Compatibility, and Hyperparameter Tuning

- APOLLO (LLM) requires choosing subspace rank $r$; low $r$ (APOLLO-Mini) suffices for most settings but extremely aggressive compression may trade off some adaptivity.
- No hyper-parameter tuning beyond AdamW defaults is generally required; APOLLO operates robustly with $\alpha=1$ or $\sqrt{n/r}$.
- The framework is compatible with activation checkpointing, ZeRO, quantization, and arbitrary large-batch distributed training.
- APOLLO-MILP presupposes access to high-performance MILP solvers for trust-region correction and requires ML predictor pretraining.

In summary, the APOLLO optimizer family provides complementary, scalable approaches to training efficiency and high-quality solution finding in both deep neural learning and combinatorial domains, anchoring their distinctive advantages on principled memory reduction, statistical uncertainty, and scalable correction mechanisms [2412.05270, 2009.13586, 2503.01129].

Source: https://www.emergentmind.com/topics/apollo-optimizer