---
title: 'OmniOpt: Unified Optimizer Benchmark'
url: https://www.emergentmind.com/topics/omniopt
type: topic
---

# OmniOpt: Unified Optimizer Benchmark

Searching arXiv for the specified paper and closely related optimizer benchmarking/survey work.
OmniOpt is a unified survey and benchmark cookbook of modern optimizers that treats optimizer selection for large-scale model training as a system-level design decision constrained jointly by compute, memory, tuning budget, and task diversity. The framework organizes a fragmented landscape of over one hundred methods through four coupled components: a five-stage meta-pipeline for optimizer updates, a norm-constrained linear minimization oracle (LMO) view of descent geometry, a dual-dimension taxonomy over mechanisms and effect objectives, and a unified cross-domain benchmark spanning language and vision settings [2607.04033]. Its central contribution is not a single new optimizer, but an operational coordinate system for selecting optimizers under explicit mechanism and objective assumptions.

## 1. Scope and conceptual framing

OmniOpt presents a unified exposition of modern optimizers built on three layers of abstraction—an operational “meta-pipeline,” a geometric “LMO-driven” view, and a dual-dimension taxonomy—culminating in a cross-domain benchmark and practical selection guidelines [2607.04033]. In this formulation, optimizer design is analyzed simultaneously as an update transformation, as a geometric direction-selection problem, and as a measurable trade-off across training objectives.

A key premise is that the optimizer landscape remains fragmented despite the existence of many method families, including element-wise adaptive moments, matrix-level structural methods, discretized directions, state-compression approaches, and curvature-aware or regularization-oriented variants. OmniOpt addresses this fragmentation by placing these families in a common representational frame. This suggests that optimizer comparison is most informative when the mechanism of an update and the intended training effect are both made explicit rather than conflated.

The paper’s definition of the problem also rejects the narrow view that optimizer choice is only about asymptotic convergence speed. Instead, the relevant criteria include validation loss or perplexity, extra FLOPs or milliseconds per step, optimizer-state memory, training stability, hyperparameter robustness, and generalization [2607.04033]. A plausible implication is that optimizer evaluation must be multi-objective even when a single scalar metric dominates a particular deployment.

## 2. Five-stage meta-pipeline

OmniOpt models every optimizer update as a structured transformation of a stochastic signal through five ordered stages, denoted \(S0\)–\(S5\). For a matrix-parameter block \(W_t\in\mathbb{R}^{m\times n}\) and mini-batch gradient \(G_t\), one full update is written as
$$
\Delta_t \;=\;  \mathbf{S5}\Bigl(\; \mathbf{S4}\bigl(\; \mathbf{S3}\bigl(\; \mathbf{S2}\bigl(\mathbf{S1}(G_t)\bigr)\;;\,\mathcal{S}_{t-1}\bigr)\;;\,\mathcal{S}_{t-1}\Bigr)\;;\,\mathcal{S}_{t-1},W_t\Bigr), 
\quad W_{t+1}=W_t+\Delta_t.
$$
Here \(\mathcal{S}_{t-1}\) denotes the optimizer state [2607.04033].

The five stages are defined concretely. \(S0\) is training signal acquisition, where \(G_t\) is formed by first-order backprop, variance-reduced estimators, or Hessian-vector products. \(S1\) is parameter scoping and routing, where coordinates are grouped, such as matrices versus vectors, and routed to separate sub-rules. \(S2\) is gradient transformation, applying an analysis map \(\hat{G}_t=\mathcal{T}(G_t;\mathcal{S}_{t-1})\); examples include identity for AdamW, sign discretization for Lion, Newton–Schulz orthogonalization for Muon, Kronecker-factored whitening for Shampoo, and low-rank projection for GaLore. \(S3\) is state evolution, updating memory via \(\mathcal{S}_t=f(\mathcal{S}_{t-1},\hat{G}_t)\); examples include first and second moments \(m_t,v_t\), Kronecker factors \(L_t,R_t\), quantized accumulators, or streamed updates. \(S4\) is update reconstruction, inverting any \(S2\) compression or basis change, such as \(\Delta_t=P_t\,\tilde\Delta_t\,P_t^\top\) for low-rank representations or \(Q_L\,\tilde\Delta_t\,Q_R^\top\) for orthogonal bases. \(S5\) is update finalization, applying global or layer-wise learning rate \(\eta_t\), decoupled weight decay \(\lambda\), clipping or trust-ratio filters, and other post-processing [2607.04033].

The paper emphasizes that most methods leave four of five stages identity-like and that their active stages pinpoint where their novelty lives. This is a strong normalization of optimizer design space: rather than treating methods as monolithic algorithms, OmniOpt decomposes them into loci of intervention. A plausible implication is that many apparently distinct optimizers can be compared more precisely by asking which stage they modify and what measurable objective that modification targets.

## 3. LMO-driven geometry and the four-axis decomposition

OmniOpt uses norm-constrained linear minimization oracles as a unifying geometric primitive for direction selection. Given a convex constraint set \(\mathcal{D}\) and a signal \(s\), the oracle solves
$$
\mathrm{lmo}_{\mathcal{D}}(s) \;=\;\arg\min_{x\in\mathcal{D}} \langle s,x\rangle.
$$
When \(\mathcal{D}\) is a norm ball \(\{x:\|x\|\le\rho\}\), the oracle becomes
$$
\mathrm{lmo}_{\|\,\cdot\,\|\le\rho}(s) =-\rho\,u^{\sharp}(s),\quad
u^{\sharp}(s)=\arg\max_{\|u\|\le1}\langle s,u\rangle,
$$
so the descent direction is \(u^{\sharp}(s)\) [2607.04033].

Three canonical cases are used to relate familiar optimizer behaviors to explicit geometries. For the Euclidean ball \(\|x\|_2\le\rho\), the oracle yields \(-\rho\,s/\|s\|_2\). For the max ball \(\|x\|_\infty\le\rho\), it yields \(-\rho\,\mathrm{sign}(s)\). For the spectral-norm ball on matrices, \(\|X\|_2\le\rho\), it yields \(-\rho\,UV^\top\) when \(s=U\Sigma V^\top\) [2607.04033]. Adaptive methods such as Adam are then interpreted as realizing an LMO over a dynamic anisotropic box \(\{|x_i|\le\rho\,|m_{t,i}|/\sqrt{v_{t,i}}\}\), while preconditioned updates \(\Phi_t(s)=H_t^{-1/2}s\) coincide with LMO directions on metric balls \(\{x:\|x\|_{H_t}\le\rho\}\).

This geometric view is paired with a four-axis decomposition:
$$
(M_t,H_t,\mathcal{D}_t)=\text{StateEstimator}_t(g_t,\mathcal{S}_{t-1}),\quad
D_t=\Phi_t(M_t;H_t,\mathcal{D}_t),\quad
W_{t+1}=W_t-\eta_t\,D_t.
$$
The four axes are: update domain or support; state estimator \((M_t,H_t)\); geometry and precondition operator \(\Phi_t\); and finalization wrapper including global learning rate, weight decay, clipping, trust ratios, and filters [2607.04033].

The significance of this construction is that “LMO” and “preconditioner” views are explicitly unified rather than treated as separate optimizer traditions. This suggests that optimizer geometry can be specified either through a constraint set \(\mathcal{D}_t\) or through an operator \(H_t^{-\alpha}\) without changing the underlying comparative frame.

## 4. Dual-dimension taxonomy of optimizer families and effect objectives

OmniOpt’s taxonomy has two dimensions. Dimension A groups approximately one hundred or more optimizers into five non-overlapping families by primary mechanism. Dimension B records the measurable training objectives each method aims to improve [2607.04033].

The five methodology families are as follows.

| Family | Primary mechanism | Active stages |
|---|---|---|
| T1 | Element-wise adaptive moments & scalar control | Mainly \(S3/S5\) |
| T2 | Matrix-level structural methods | \(S1\)–\(S4\) |
| T3 | Discretized directions | \(S2/S3\) |
| T4 | State compression & structural aggregation | \(S3/S4\) |
| T5 | Curvature-aware & geometric regularization | Near \(S0/S5\) |

T1 includes SGD, AdamW, RAdam, and AdaBelief, and is characterized by diagonal geometry with activity concentrated in state evolution and finalization. T2 includes Muon, Shampoo, SOAP, and GaLore, and acts by changing bases or subspaces. T3 includes SignSGD, Lion, RLion, and FOCUS, using irreversible sign or quantization maps. T4 includes AdaFactor, 8-bit Adam, Adam-mini, APOLLO, Conda, and LOMO, shrinking or sharing state. T5 includes the SAM family, Sophia, AdamP, LAMB, and Cautious, acting through perturbations, Hessian clipping, post-filters, or trust ratios [2607.04033].

The six effect objectives are \(O1\) convergence efficiency, \(O2\) step cost, \(O3\) memory overhead, \(O4\) training stability, \(O5\) hyperparameter robustness, and \(O6\) generalization. More specifically, \(O1\) is measured by quantities such as validation loss or perplexity versus training tokens; \(O2\) by extra FLOPs or milliseconds per step; \(O3\) by optimizer-state gigabytes; \(O4\) by gradient-norm coefficient of variation and spike rate; \(O5\) by sensitivity to learning-rate or weight-decay perturbations; and \(O6\) by downstream or cross-architecture transfer [2607.04033].

The taxonomy’s stated purpose is to provide a cross-matrix of families versus objectives that guides which effects each family is expected to improve or must be evaluated on. A common misconception is that optimizer categories are sufficiently described by implementation details alone. OmniOpt instead treats mechanism family and effect objective as separate coordinates, making it possible to distinguish, for example, a memory-oriented state-compression method from a geometry-changing matrix method even when both alter runtime behavior.

## 5. Cross-domain benchmark design

At the core of OmniOpt is a unified benchmark that instantiates the full taxonomy across language and vision tasks [2607.04033]. The benchmark is divided into two language-model stages and one vision ablation, each aligned to specified subsets of the six effect objectives.

Stage 1 is short-context LLM pretraining targeting \(O1\)–\(O3\). It uses tokenized C4, LLaMA blocks with sequence length \(256\), model scales of \(60\) M for \(10\) k steps, \(130\) M for \(20\) k, \(350\) M for \(60\) k, and \(1\) B for \(100\) k. The benchmark evaluates \(24\) representative optimizers spanning \(T1\)–\(T5\), with hyperparameter sweeps over learning rate, betas, epsilon, and method-specific knobs, while fixed decay and clipping are turned off. The reported metrics are validation perplexity for \(O1\), per-step runtime for \(O2\), and optimizer-state memory for \(O3\) [2607.04033].

Stage 2 is long-context generalization targeting \(O1\) and \(O4\)–\(O6\). It uses FineWeb-Edu, described as higher-quality text, with sequence length \(32\) k tokens. The models are \(340\) M and \(1\) B across four backbones—Transformer++, Gated DeltaNet, DeltaNet, and GLA. The same \(24\) optimizers are used, but decoupled decay and global clipping are now enabled. Metrics include WikiText test PPL for \(O1\), gradient-norm CV for \(O4\), a learning-rate perturbation test for \(O5\), and cross-scenario PPL rank stability together with CS-harness average accuracy for \(O6\) [2607.04033].

The vision ablation uses CIFAR100 with three backbones: ResNet50, DeiT-S, and CAFormer-S12. It employs a single training protocol, measures Top-1 accuracy as \(O1\), and is used to confirm architecture-dependent optimizer behavior [2607.04033].

| Benchmark component | Configuration | Metrics |
|---|---|---|
| Stage 1 | C4, LLaMA blocks, seq \(=256\), \(60\) M/\(130\) M/\(350\) M/\(1\) B | Validation PPL, runtime, state memory |
| Stage 2 | FineWeb-Edu, seq \(=32\) k, \(340\) M/\(1\) B, Transformer++/Gated DeltaNet/DeltaNet/GLA | WikiText PPL, gradient-norm CV, lr-perturbation, rank stability, CS-harness accuracy |
| Vision ablation | CIFAR100, ResNet50/DeiT-S/CAFormer-S12 | Top-1 accuracy |

The benchmark is designed to expose trade-offs rather than to force a single ranking. This is especially clear in the separation of short-context efficiency measurements from long-context generalization, robustness, and transfer measurements. A plausible implication is that optimizer conclusions derived from one training regime should not be assumed to transfer without direct measurement.

## 6. Empirical trade-offs, family profiles, and selection guidelines

In Stage 1, AdamW is reported as the stable \(T1\) baseline, with moderate perplexity, runtime, and memory [2607.04033]. Within \(T2\), methods split into “heavy” variants such as SOAP and Shampoo, which achieve the best perplexity at high cost, and “balanced” variants such as RMNP, which provides strong perplexity with moderate runtime and memory. \(T4\) methods yield extreme memory reduction in the case of AdaFactor or mixed short-context perplexity in the case of APOLLO, but require caution. \(T3\) methods such as Lion are the fastest and learning-rate tolerant, yet lag in perplexity. At \(1\) B parameters, the reported Pareto frontiers show RMNP and APOLLO as frontier points for quality versus runtime below heavy SOAP and Muon, while APOLLO and AdaFactor dominate the low-memory edge of the quality-versus-memory frontier at a quality cost [2607.04033].

In Stage 2, SOAP is described as the most transferable quality leader and is never outside the top two ranks. MARS-AdamW is the most stable AdamW extension and often tops CS accuracy. Muon, labeled \(T2.1\), shows cross-architecture sensitivity, performing strongly on GLA and more weakly on others. APOLLO collapses at \(32\) k, which the paper presents as confirmation of state-compression’s rank limitation. Lion and Adan remain locally robust but lag in final perplexity [2607.04033].

Auxiliary \(O4\) and \(O5\) tests refine this picture. \(T2\) methods, specifically Muon and \(T2.2\) MARS-Shampoo, improve smoothness as measured by gradient-norm CV, although heavy methods can suffer rare spikes under certain architectures. Lion and MARS-Lion have the flattest local learning-rate response, whereas APOLLO is most fragile to a \(5\times\) learning rate [2607.04033]. These findings directly counter the assumption that long-context performance and local robustness necessarily coincide.

The family-level objective profile summarizes the benchmark results at higher resolution. \(T1\) methods are strong on \(O1\) and \(O4\), moderate on \(O2\) and \(O3\), and good on \(O5\) and \(O6\), making them the reliable baseline family. \(T2\) methods are the \(O1/O4\) winners, but their \(O2/O3\) costs vary widely and \(O6\) varies by method. \(T3\) methods are efficient and tolerant on \(O2/O5\) but weak on \(O1\), with mixed \(O6\). \(T4\) methods are \(O3\) champions but drop on \(O1/O6\) at long context. \(T5\) methods achieve occasional \(O6\) wins, such as SAM, but are overall situational under the reported protocol [2607.04033].

OmniOpt concludes with tiered recommendations and action-oriented defaults. Tier I consists of AdamW, RMNP, and Muon. Tier II includes SOAP, MARS-AdamW, AdamP, Adan, Lion, APOLLO, Conda, and MARS-Shampoo. Tier III contains RAdam, NAdam, Prodigy, AdaBelief, GaLore, Shampoo, 8-bit Adam, CAME, AdaFactor, Adam-mini, LAMB, and Sophia. The practical guidance is then keyed to a single binding constraint: AdamW for a balanced default, SOAP for a long-context quality ceiling, AdaFactor for a memory efficiency ceiling, RMNP for a balanced matrix method with moderate cost and quality, Muon as a transparent spectral method for mechanistic studies with per-architecture validation, Lion for fast exploratory runs with awareness of a perplexity gap, and APOLLO for a short-context memory boost with re-validation in long context [2607.04033].

The paper’s final position is therefore not that one optimizer dominates universally, but that optimizer choice must be driven by the single binding constraint of a run—quality, runtime, memory, stability, or generalization. The meta-pipeline, LMO view, and dual taxonomy together are intended to supply a shared coordinate system for making and explaining that choice [2607.04033].

Source: https://www.emergentmind.com/topics/omniopt