OmniOpt: Unified Optimizer Benchmark

• OmniOpt is a unified survey and benchmark framework that organizes over 100 modern optimizers as a system-level design problem considering compute, memory, tuning, and task diversity.
• It decomposes optimizer updates into a structured five-stage meta-pipeline, employs LMO-driven geometric insights, and uses a dual-dimension taxonomy to clarify intervention points.
• The framework’s cross-domain benchmark evaluates perplexity, runtime, memory footprint, stability, and generalization, informing practical selection guidelines for varied training constraints.

Searching arXiv for the specified paper and closely related optimizer benchmarking/survey work. OmniOpt is a unified survey and benchmark cookbook of modern optimizers that treats optimizer selection for large-scale model training as a system-level design decision constrained jointly by compute, memory, tuning budget, and task diversity. The framework organizes a fragmented landscape of over one hundred methods through four coupled components: a five-stage meta-pipeline for optimizer updates, a norm-constrained linear minimization oracle (LMO) view of descent geometry, a dual-dimension taxonomy over mechanisms and effect objectives, and a unified cross-domain benchmark spanning language and vision settings (Li et al., 4 Jul 2026). Its central contribution is not a single new optimizer, but an operational coordinate system for selecting optimizers under explicit mechanism and objective assumptions.

1. Scope and conceptual framing

OmniOpt presents a unified exposition of modern optimizers built on three layers of abstraction—an operational “meta-pipeline,” a geometric “LMO-driven” view, and a dual-dimension taxonomy—culminating in a cross-domain benchmark and practical selection guidelines (Li et al., 4 Jul 2026). In this formulation, optimizer design is analyzed simultaneously as an update transformation, as a geometric direction-selection problem, and as a measurable trade-off across training objectives.

A key premise is that the optimizer landscape remains fragmented despite the existence of many method families, including element-wise adaptive moments, matrix-level structural methods, discretized directions, state-compression approaches, and curvature-aware or regularization-oriented variants. OmniOpt addresses this fragmentation by placing these families in a common representational frame. This suggests that optimizer comparison is most informative when the mechanism of an update and the intended training effect are both made explicit rather than conflated.

The paper’s definition of the problem also rejects the narrow view that optimizer choice is only about asymptotic convergence speed. Instead, the relevant criteria include validation loss or perplexity, extra FLOPs or milliseconds per step, optimizer-state memory, training stability, hyperparameter robustness, and generalization (Li et al., 4 Jul 2026). A plausible implication is that optimizer evaluation must be multi-objective even when a single scalar metric dominates a particular deployment.

2. Five-stage meta-pipeline

OmniOpt models every optimizer update as a structured transformation of a stochastic signal through five ordered stages, denoted $S0$–$S5$. For a matrix-parameter block $W_t\in\mathbb{R}^{m\times n}$ and mini-batch gradient $G_t$, one full update is written as

$$\Delta_t \;=\; \mathbf{S5}\Bigl(\; \mathbf{S4}\bigl(\; \mathbf{S3}\bigl(\; \mathbf{S2}\bigl(\mathbf{S1}(G_t)\bigr)\;;\,\mathcal{S}_{t-1}\bigr)\;;\,\mathcal{S}_{t-1}\Bigr)\;;\,\mathcal{S}_{t-1},W_t\Bigr), \quad W_{t+1}=W_t+\Delta_t.$$

Here $\mathcal{S}_{t-1}$ denotes the optimizer state (Li et al., 4 Jul 2026).

The five stages are defined concretely. $S0$ is training signal acquisition, where $G_t$ is formed by first-order backprop, variance-reduced estimators, or Hessian-vector products. $S1$ is parameter scoping and routing, where coordinates are grouped, such as matrices versus vectors, and routed to separate sub-rules. $S2$ is gradient transformation, applying an analysis map $S5$0; examples include identity for AdamW, sign discretization for Lion, Newton–Schulz orthogonalization for Muon, Kronecker-factored whitening for Shampoo, and low-rank projection for GaLore. $S5$1 is state evolution, updating memory via $S5$2; examples include first and second moments $S5$3, Kronecker factors $S5$4, quantized accumulators, or streamed updates. $S5$5 is update reconstruction, inverting any $S5$6 compression or basis change, such as $S5$7 for low-rank representations or $S5$8 for orthogonal bases. $S5$9 is update finalization, applying global or layer-wise learning rate $W_t\in\mathbb{R}^{m\times n}$0, decoupled weight decay $W_t\in\mathbb{R}^{m\times n}$1, clipping or trust-ratio filters, and other post-processing (Li et al., 4 Jul 2026).

The paper emphasizes that most methods leave four of five stages identity-like and that their active stages pinpoint where their novelty lives. This is a strong normalization of optimizer design space: rather than treating methods as monolithic algorithms, OmniOpt decomposes them into loci of intervention. A plausible implication is that many apparently distinct optimizers can be compared more precisely by asking which stage they modify and what measurable objective that modification targets.

3. LMO-driven geometry and the four-axis decomposition

OmniOpt uses norm-constrained linear minimization oracles as a unifying geometric primitive for direction selection. Given a convex constraint set $W_t\in\mathbb{R}^{m\times n}$2 and a signal $W_t\in\mathbb{R}^{m\times n}$3, the oracle solves

$W_t\in\mathbb{R}^{m\times n}$4

When $W_t\in\mathbb{R}^{m\times n}$5 is a norm ball $W_t\in\mathbb{R}^{m\times n}$6, the oracle becomes

$W_t\in\mathbb{R}^{m\times n}$7

so the descent direction is $W_t\in\mathbb{R}^{m\times n}$8 (Li et al., 4 Jul 2026).

Three canonical cases are used to relate familiar optimizer behaviors to explicit geometries. For the Euclidean ball $W_t\in\mathbb{R}^{m\times n}$9, the oracle yields $G_t$0. For the max ball $G_t$1, it yields $G_t$2. For the spectral-norm ball on matrices, $G_t$3, it yields $G_t$4 when $G_t$5 (Li et al., 4 Jul 2026). Adaptive methods such as Adam are then interpreted as realizing an LMO over a dynamic anisotropic box $G_t$6, while preconditioned updates $G_t$7 coincide with LMO directions on metric balls $G_t$8.

This geometric view is paired with a four-axis decomposition:

$G_t$9

The four axes are: update domain or support; state estimator $$\Delta_t \;=\; \mathbf{S5}\Bigl(\; \mathbf{S4}\bigl(\; \mathbf{S3}\bigl(\; \mathbf{S2}\bigl(\mathbf{S1}(G_t)\bigr)\;;\,\mathcal{S}_{t-1}\bigr)\;;\,\mathcal{S}_{t-1}\Bigr)\;;\,\mathcal{S}_{t-1},W_t\Bigr), \quad W_{t+1}=W_t+\Delta_t.$$0; geometry and precondition operator $$\Delta_t \;=\; \mathbf{S5}\Bigl(\; \mathbf{S4}\bigl(\; \mathbf{S3}\bigl(\; \mathbf{S2}\bigl(\mathbf{S1}(G_t)\bigr)\;;\,\mathcal{S}_{t-1}\bigr)\;;\,\mathcal{S}_{t-1}\Bigr)\;;\,\mathcal{S}_{t-1},W_t\Bigr), \quad W_{t+1}=W_t+\Delta_t.$$1; and finalization wrapper including global learning rate, weight decay, clipping, trust ratios, and filters (Li et al., 4 Jul 2026).

The significance of this construction is that “LMO” and “preconditioner” views are explicitly unified rather than treated as separate optimizer traditions. This suggests that optimizer geometry can be specified either through a constraint set $$\Delta_t \;=\; \mathbf{S5}\Bigl(\; \mathbf{S4}\bigl(\; \mathbf{S3}\bigl(\; \mathbf{S2}\bigl(\mathbf{S1}(G_t)\bigr)\;;\,\mathcal{S}_{t-1}\bigr)\;;\,\mathcal{S}_{t-1}\Bigr)\;;\,\mathcal{S}_{t-1},W_t\Bigr), \quad W_{t+1}=W_t+\Delta_t.$$2 or through an operator $$\Delta_t \;=\; \mathbf{S5}\Bigl(\; \mathbf{S4}\bigl(\; \mathbf{S3}\bigl(\; \mathbf{S2}\bigl(\mathbf{S1}(G_t)\bigr)\;;\,\mathcal{S}_{t-1}\bigr)\;;\,\mathcal{S}_{t-1}\Bigr)\;;\,\mathcal{S}_{t-1},W_t\Bigr), \quad W_{t+1}=W_t+\Delta_t.$$3 without changing the underlying comparative frame.

4. Dual-dimension taxonomy of optimizer families and effect objectives

OmniOpt’s taxonomy has two dimensions. Dimension A groups approximately one hundred or more optimizers into five non-overlapping families by primary mechanism. Dimension B records the measurable training objectives each method aims to improve (Li et al., 4 Jul 2026).

The five methodology families are as follows.

Family	Primary mechanism	Active stages
T1	Element-wise adaptive moments & scalar control	Mainly $$\Delta_t \;=\; \mathbf{S5}\Bigl(\; \mathbf{S4}\bigl(\; \mathbf{S3}\bigl(\; \mathbf{S2}\bigl(\mathbf{S1}(G_t)\bigr)\;;\,\mathcal{S}_{t-1}\bigr)\;;\,\mathcal{S}_{t-1}\Bigr)\;;\,\mathcal{S}_{t-1},W_t\Bigr), \quad W_{t+1}=W_t+\Delta_t.$$4
T2	Matrix-level structural methods	$$\Delta_t \;=\; \mathbf{S5}\Bigl(\; \mathbf{S4}\bigl(\; \mathbf{S3}\bigl(\; \mathbf{S2}\bigl(\mathbf{S1}(G_t)\bigr)\;;\,\mathcal{S}_{t-1}\bigr)\;;\,\mathcal{S}_{t-1}\Bigr)\;;\,\mathcal{S}_{t-1},W_t\Bigr), \quad W_{t+1}=W_t+\Delta_t.$$5–$$\Delta_t \;=\; \mathbf{S5}\Bigl(\; \mathbf{S4}\bigl(\; \mathbf{S3}\bigl(\; \mathbf{S2}\bigl(\mathbf{S1}(G_t)\bigr)\;;\,\mathcal{S}_{t-1}\bigr)\;;\,\mathcal{S}_{t-1}\Bigr)\;;\,\mathcal{S}_{t-1},W_t\Bigr), \quad W_{t+1}=W_t+\Delta_t.$$6
T3	Discretized directions	$$\Delta_t \;=\; \mathbf{S5}\Bigl(\; \mathbf{S4}\bigl(\; \mathbf{S3}\bigl(\; \mathbf{S2}\bigl(\mathbf{S1}(G_t)\bigr)\;;\,\mathcal{S}_{t-1}\bigr)\;;\,\mathcal{S}_{t-1}\Bigr)\;;\,\mathcal{S}_{t-1},W_t\Bigr), \quad W_{t+1}=W_t+\Delta_t.$$7
T4	State compression & structural aggregation	$$\Delta_t \;=\; \mathbf{S5}\Bigl(\; \mathbf{S4}\bigl(\; \mathbf{S3}\bigl(\; \mathbf{S2}\bigl(\mathbf{S1}(G_t)\bigr)\;;\,\mathcal{S}_{t-1}\bigr)\;;\,\mathcal{S}_{t-1}\Bigr)\;;\,\mathcal{S}_{t-1},W_t\Bigr), \quad W_{t+1}=W_t+\Delta_t.$$8
T5	Curvature-aware & geometric regularization	Near $$\Delta_t \;=\; \mathbf{S5}\Bigl(\; \mathbf{S4}\bigl(\; \mathbf{S3}\bigl(\; \mathbf{S2}\bigl(\mathbf{S1}(G_t)\bigr)\;;\,\mathcal{S}_{t-1}\bigr)\;;\,\mathcal{S}_{t-1}\Bigr)\;;\,\mathcal{S}_{t-1},W_t\Bigr), \quad W_{t+1}=W_t+\Delta_t.$$9

T1 includes SGD, AdamW, RAdam, and AdaBelief, and is characterized by diagonal geometry with activity concentrated in state evolution and finalization. T2 includes Muon, Shampoo, SOAP, and GaLore, and acts by changing bases or subspaces. T3 includes SignSGD, Lion, RLion, and FOCUS, using irreversible sign or quantization maps. T4 includes AdaFactor, 8-bit Adam, Adam-mini, APOLLO, Conda, and LOMO, shrinking or sharing state. T5 includes the SAM family, Sophia, AdamP, LAMB, and Cautious, acting through perturbations, Hessian clipping, post-filters, or trust ratios (Li et al., 4 Jul 2026).

The six effect objectives are $\mathcal{S}_{t-1}$0 convergence efficiency, $\mathcal{S}_{t-1}$1 step cost, $\mathcal{S}_{t-1}$2 memory overhead, $\mathcal{S}_{t-1}$3 training stability, $\mathcal{S}_{t-1}$4 hyperparameter robustness, and $\mathcal{S}_{t-1}$5 generalization. More specifically, $\mathcal{S}_{t-1}$6 is measured by quantities such as validation loss or perplexity versus training tokens; $\mathcal{S}_{t-1}$7 by extra FLOPs or milliseconds per step; $\mathcal{S}_{t-1}$8 by optimizer-state gigabytes; $\mathcal{S}_{t-1}$9 by gradient-norm coefficient of variation and spike rate; $S0$0 by sensitivity to learning-rate or weight-decay perturbations; and $S0$1 by downstream or cross-architecture transfer (Li et al., 4 Jul 2026).

The taxonomy’s stated purpose is to provide a cross-matrix of families versus objectives that guides which effects each family is expected to improve or must be evaluated on. A common misconception is that optimizer categories are sufficiently described by implementation details alone. OmniOpt instead treats mechanism family and effect objective as separate coordinates, making it possible to distinguish, for example, a memory-oriented state-compression method from a geometry-changing matrix method even when both alter runtime behavior.

5. Cross-domain benchmark design

At the core of OmniOpt is a unified benchmark that instantiates the full taxonomy across language and vision tasks (Li et al., 4 Jul 2026). The benchmark is divided into two language-model stages and one vision ablation, each aligned to specified subsets of the six effect objectives.

Stage 1 is short-context LLM pretraining targeting $S0$2–$S0$3. It uses tokenized C4, LLaMA blocks with sequence length $S0$4, model scales of $S0$5 M for $S0$6 k steps, $S0$7 M for $S0$8 k, $S0$9 M for $G_t$0 k, and $G_t$1 B for $G_t$2 k. The benchmark evaluates $G_t$3 representative optimizers spanning $G_t$4–$G_t$5, with hyperparameter sweeps over learning rate, betas, epsilon, and method-specific knobs, while fixed decay and clipping are turned off. The reported metrics are validation perplexity for $G_t$6, per-step runtime for $G_t$7, and optimizer-state memory for $G_t$8 (Li et al., 4 Jul 2026).

Stage 2 is long-context generalization targeting $G_t$9 and $S1$0–$S1$1. It uses FineWeb-Edu, described as higher-quality text, with sequence length $S1$2 k tokens. The models are $S1$3 M and $S1$4 B across four backbones—Transformer++, Gated DeltaNet, DeltaNet, and GLA. The same $S1$5 optimizers are used, but decoupled decay and global clipping are now enabled. Metrics include WikiText test PPL for $S1$6, gradient-norm CV for $S1$7, a learning-rate perturbation test for $S1$8, and cross-scenario PPL rank stability together with CS-harness average accuracy for $S1$9 (Li et al., 4 Jul 2026).

The vision ablation uses CIFAR100 with three backbones: ResNet50, DeiT-S, and CAFormer-S12. It employs a single training protocol, measures Top-1 accuracy as $S2$0, and is used to confirm architecture-dependent optimizer behavior (Li et al., 4 Jul 2026).

Benchmark component	Configuration	Metrics
Stage 1	C4, LLaMA blocks, seq $S2$1, $S2$2 M/$S2$3 M/$S2$4 M/$S2$5 B	Validation PPL, runtime, state memory
Stage 2	FineWeb-Edu, seq $S2$6 k, $S2$7 M/$S2$8 B, Transformer++/Gated DeltaNet/DeltaNet/GLA	WikiText PPL, gradient-norm CV, lr-perturbation, rank stability, CS-harness accuracy
Vision ablation	CIFAR100, ResNet50/DeiT-S/CAFormer-S12	Top-1 accuracy

The benchmark is designed to expose trade-offs rather than to force a single ranking. This is especially clear in the separation of short-context efficiency measurements from long-context generalization, robustness, and transfer measurements. A plausible implication is that optimizer conclusions derived from one training regime should not be assumed to transfer without direct measurement.

6. Empirical trade-offs, family profiles, and selection guidelines

In Stage 1, AdamW is reported as the stable $S2$9 baseline, with moderate perplexity, runtime, and memory (Li et al., 4 Jul 2026). Within $S5$00, methods split into “heavy” variants such as SOAP and Shampoo, which achieve the best perplexity at high cost, and “balanced” variants such as RMNP, which provides strong perplexity with moderate runtime and memory. $S5$01 methods yield extreme memory reduction in the case of AdaFactor or mixed short-context perplexity in the case of APOLLO, but require caution. $S5$02 methods such as Lion are the fastest and learning-rate tolerant, yet lag in perplexity. At $S5$03 B parameters, the reported Pareto frontiers show RMNP and APOLLO as frontier points for quality versus runtime below heavy SOAP and Muon, while APOLLO and AdaFactor dominate the low-memory edge of the quality-versus-memory frontier at a quality cost (Li et al., 4 Jul 2026).

In Stage 2, SOAP is described as the most transferable quality leader and is never outside the top two ranks. MARS-AdamW is the most stable AdamW extension and often tops CS accuracy. Muon, labeled $S5$04, shows cross-architecture sensitivity, performing strongly on GLA and more weakly on others. APOLLO collapses at $S5$05 k, which the paper presents as confirmation of state-compression’s rank limitation. Lion and Adan remain locally robust but lag in final perplexity (Li et al., 4 Jul 2026).

Auxiliary $S5$06 and $S5$07 tests refine this picture. $S5$08 methods, specifically Muon and $S5$09 MARS-Shampoo, improve smoothness as measured by gradient-norm CV, although heavy methods can suffer rare spikes under certain architectures. Lion and MARS-Lion have the flattest local learning-rate response, whereas APOLLO is most fragile to a $S5$10 learning rate (Li et al., 4 Jul 2026). These findings directly counter the assumption that long-context performance and local robustness necessarily coincide.

The family-level objective profile summarizes the benchmark results at higher resolution. $S5$11 methods are strong on $S5$12 and $S5$13, moderate on $S5$14 and $S5$15, and good on $S5$16 and $S5$17, making them the reliable baseline family. $S5$18 methods are the $S5$19 winners, but their $S5$20 costs vary widely and $S5$21 varies by method. $S5$22 methods are efficient and tolerant on $S5$23 but weak on $S5$24, with mixed $S5$25. $S5$26 methods are $S5$27 champions but drop on $S5$28 at long context. $S5$29 methods achieve occasional $S5$30 wins, such as SAM, but are overall situational under the reported protocol (Li et al., 4 Jul 2026).

OmniOpt concludes with tiered recommendations and action-oriented defaults. Tier I consists of AdamW, RMNP, and Muon. Tier II includes SOAP, MARS-AdamW, AdamP, Adan, Lion, APOLLO, Conda, and MARS-Shampoo. Tier III contains RAdam, NAdam, Prodigy, AdaBelief, GaLore, Shampoo, 8-bit Adam, CAME, AdaFactor, Adam-mini, LAMB, and Sophia. The practical guidance is then keyed to a single binding constraint: AdamW for a balanced default, SOAP for a long-context quality ceiling, AdaFactor for a memory efficiency ceiling, RMNP for a balanced matrix method with moderate cost and quality, Muon as a transparent spectral method for mechanistic studies with per-architecture validation, Lion for fast exploratory runs with awareness of a perplexity gap, and APOLLO for a short-context memory boost with re-validation in long context (Li et al., 4 Jul 2026).

The paper’s final position is therefore not that one optimizer dominates universally, but that optimizer choice must be driven by the single binding constraint of a run—quality, runtime, memory, stability, or generalization. The meta-pipeline, LMO view, and dual taxonomy together are intended to supply a shared coordinate system for making and explaining that choice (Li et al., 4 Jul 2026).