LMO-IGT: Implicit Gradient Transport in LMO Methods

• LMO-IGT is a stochastic optimization method that evaluates gradients at transported lookahead points to enhance convergence without extra gradient evaluations.
• It employs dual momentum sequences and a regularized support function to balance variance reduction and the geometry-aware update process.
• Empirical results on benchmarks like CIFAR-10 demonstrate that LMO-IGT achieves improved performance over standard stochastic methods with minimal computational overhead.

Searching arXiv for papers on “LMO-IGT” and related LMO-based optimization.

LMO-IGT is a class of stochastic linear-minimization-oracle-based optimization methods that leverages implicit gradient transport to accelerate modern LMO-based updates while preserving the single-gradient-per-iteration structure of standard stochastic LMO. It was introduced for problems of the form

$$\min_{w\in\mathcal P} F(w), \qquad F(w):=\mathbb E_\xi[f(w;\xi)],$$

with a unified treatment of unconstrained normalized methods and constrained Frank–Wolfe-type formulations. Its defining innovation is to evaluate stochastic gradients at transported points along the update trajectory rather than at the current iterate, and its analysis is organized around the regularized support function, a stationarity measure that bridges gradient-norm and Frank–Wolfe-gap viewpoints (Jang et al., 7 May 2026).

1. Optimization setting and oracle geometry

The underlying primitive is a linear minimization oracle over a compact convex set $\mathcal C\ni 0$: $$\operatorname{LMO}_{\mathcal C}(g)\in \arg\min_{v\in\mathcal C}\langle g,v\rangle.$$ Given a gradient or momentum estimate $g_t$, an LMO-based method chooses

$v_t=\operatorname{LMO}_{\mathcal C}(g_t),$

and updates by

$w_{t+1}=(1-\lambda\eta_t)w_t+\eta_t v_t.$

When $\lambda=0$, this is the unconstrained case. When $\lambda>0$, with $s_t=\lambda^{-1}v_t\in\mathcal P$ and $\gamma_t=\lambda\eta_t$, the same rule becomes

$\mathcal C\ni 0$0

which is exactly a conditional-gradient update over $\mathcal C\ni 0$1 (Jang et al., 7 May 2026).

This formulation subsumes several geometry-aware optimizers. If $\mathcal C\ni 0$2 is an $\mathcal C\ni 0$3-ball, then

$\mathcal C\ni 0$4

recovering sign-type updates such as signSGD, Signum, and Lion. If $\mathcal C\ni 0$5 is the operator-norm ball for matrices,

$\mathcal C\ni 0$6

and $\mathcal C\ni 0$7 is an SVD, then

$\mathcal C\ni 0$8

which is the orthogonalized matrix update underlying Muon (Jang et al., 7 May 2026). Subsequent layer-wise LMO formulations, notably Gluon, made this geometry explicit for Muon- and Scion-style training of large neural networks by assigning blockwise norms and blockwise radii $\mathcal C\ni 0$9 (Riabinin et al., 19 May 2025).

2. Implicit gradient transport

The motivation for LMO-IGT is the gap between plain stochastic LMO and variance-reduced LMO. Standard stochastic momentum updates evaluate gradients at $$\operatorname{LMO}_{\mathcal C}(g)\in \arg\min_{v\in\mathcal C}\langle g,v\rangle.$$0, so the exponential moving average lags behind $$\operatorname{LMO}_{\mathcal C}(g)\in \arg\min_{v\in\mathcal C}\langle g,v\rangle.$$1. Variance-reduced LMO improves the rate by using corrections such as

$$\operatorname{LMO}_{\mathcal C}(g)\in \arg\min_{v\in\mathcal C}\langle g,v\rangle.$$2

but this requires gradients at both the current and previous iterates, effectively using two stochastic gradient evaluations per iteration (Jang et al., 7 May 2026).

LMO-IGT keeps one stochastic gradient per iteration, but changes the query point. At iteration $$\operatorname{LMO}_{\mathcal C}(g)\in \arg\min_{v\in\mathcal C}\langle g,v\rangle.$$3, it uses two momentum-like sequences,

$$\operatorname{LMO}_{\mathcal C}(g)\in \arg\min_{v\in\mathcal C}\langle g,v\rangle.$$4

$$\operatorname{LMO}_{\mathcal C}(g)\in \arg\min_{v\in\mathcal C}\langle g,v\rangle.$$5

with $$\operatorname{LMO}_{\mathcal C}(g)\in \arg\min_{v\in\mathcal C}\langle g,v\rangle.$$6. The paper interprets $$\operatorname{LMO}_{\mathcal C}(g)\in \arg\min_{v\in\mathcal C}\langle g,v\rangle.$$7 as a more responsive “fast” momentum and $$\operatorname{LMO}_{\mathcal C}(g)\in \arg\min_{v\in\mathcal C}\langle g,v\rangle.$$8 as a more stable “slow” buffer. The update direction remains

$$\operatorname{LMO}_{\mathcal C}(g)\in \arg\min_{v\in\mathcal C}\langle g,v\rangle.$$9

but both the actual iterate and the transported point are updated via

$g_t$0

$g_t$1

The theoretical choice

$g_t$2

yields the lookahead relation

$g_t$3

Thus $g_t$4 is a transported point obtained by extrapolating the parameter trajectory. The analysis shows that this construction cancels a first-order drift term in the tracking-error recursion for the momentum buffer, which is why the improved convergence guarantee for LMO-IGT requires a second-order smoothness assumption

$g_t$5

in addition to $g_t$6-smoothness and bounded stochastic-gradient variance (Jang et al., 7 May 2026).

3. Regularized support function and convergence theory

A central contribution of the framework is the regularized support function

$g_t$7

This unifies the unconstrained and constrained cases. When $g_t$8 and $g_t$9,

$v_t=\operatorname{LMO}_{\mathcal C}(g_t),$0

where $v_t=\operatorname{LMO}_{\mathcal C}(g_t),$1 is the Frank–Wolfe gap. When $v_t=\operatorname{LMO}_{\mathcal C}(g_t),$2,

$v_t=\operatorname{LMO}_{\mathcal C}(g_t),$3

the support function of $v_t=\operatorname{LMO}_{\mathcal C}(g_t),$4 at $v_t=\operatorname{LMO}_{\mathcal C}(g_t),$5. For centrally symmetric $v_t=\operatorname{LMO}_{\mathcal C}(g_t),$6, this reduces to the dual norm induced by $v_t=\operatorname{LMO}_{\mathcal C}(g_t),$7; for a Euclidean ball of diameter $v_t=\operatorname{LMO}_{\mathcal C}(g_t),$8,

$v_t=\operatorname{LMO}_{\mathcal C}(g_t),$9

The paper also states that $w_{t+1}=(1-\lambda\eta_t)w_t+\eta_t v_t.$0 is an exact stationarity certificate: it is equivalent to KKT stationarity when $w_{t+1}=(1-\lambda\eta_t)w_t+\eta_t v_t.$1, and to $w_{t+1}=(1-\lambda\eta_t)w_t+\eta_t v_t.$2 when $w_{t+1}=(1-\lambda\eta_t)w_t+\eta_t v_t.$3 (Jang et al., 7 May 2026).

The one-step descent analysis is organized by the mismatch

$w_{t+1}=(1-\lambda\eta_t)w_t+\eta_t v_t.$4

For one LMO step,

$w_{t+1}=(1-\lambda\eta_t)w_t+\eta_t v_t.$5

Averaging this inequality reduces convergence to bounding the estimator mismatch. The resulting comparison among the three principal regimes is as follows (Jang et al., 7 May 2026).

Method	Average stationarity rate	Gradient evaluations per iteration
Stochastic LMO	$w_{t+1}=(1-\lambda\eta_t)w_t+\eta_t v_t.$6	1
LMO-VR	$w_{t+1}=(1-\lambda\eta_t)w_t+\eta_t v_t.$7	2
LMO-IGT	$w_{t+1}=(1-\lambda\eta_t)w_t+\eta_t v_t.$8	1

Equivalently, the iteration complexities for driving

$w_{t+1}=(1-\lambda\eta_t)w_t+\eta_t v_t.$9

below $\lambda=0$0 are $\lambda=0$1 for stochastic LMO, $\lambda=0$2 for variance-reduced LMO, and $\lambda=0$3 for LMO-IGT. The theory therefore places LMO-IGT between plain stochastic LMO and variance-reduced LMO: it improves the stochastic rate without paying the extra gradient cost of variance reduction (Jang et al., 7 May 2026).

4. Instantiations, empirical behavior, and practical constraints

The principal practical instantiations studied are Lion-IGT and Muon-IGT. On CIFAR-10 with ResNet-18, the experiments compare AdamW, NIGT, Lion, Muon, Lion-VR, Muon-VR, Lion-IGT, and Muon-IGT over 200 epochs, with curves averaged over five runs and one-standard-deviation bands. The reported metrics are test accuracy over epochs and over wall-clock time. Among the tested methods, Muon-IGT achieves the best final accuracy, while Lion-IGT and Muon-IGT run at nearly the same speed as their plain baselines because IGT introduces minimal computation and memory overhead (Jang et al., 7 May 2026).

The appendices extend the empirical picture. Additional experiments include an ablation on Muon-IGT showing that adding double momentum improves over Muon and that adding IGT on top improves further, robustness studies over learning rate and weight decay, language modeling with nanoGPT (10M) on Shakespeare, and larger-scale language modeling with nanoGPT (124M) on OpenWebText. In the reported results, Muon-IGT outperforms baselines in both training and test loss on Shakespeare and retains the same qualitative advantage on OpenWebText (Jang et al., 7 May 2026).

The practical caveats are explicit. The improved convergence guarantee depends on second-order smoothness, the results are stated in expectation for average first-order stationarity rather than for last-iterate or high-probability guarantees, and the framework does not cover biased or approximate LMOs. Implementation also requires maintaining both $\lambda=0$4 and $\lambda=0$5, and the paper’s own hyperparameters show that IGT variants may need retuning rather than direct reuse of the base optimizer’s settings. For CIFAR-10 with ResNet-18, for example, the reported Muon setting is

$\lambda=0$6

whereas Muon-IGT uses

$\lambda=0$7

A plausible implication is that the transport mechanism changes the effective optimization geometry enough that hyperparameter transfer from the base method is not automatic (Jang et al., 7 May 2026).

5. Relation to adjacent LMO-based optimizer research

LMO-IGT is part of a broader recent effort to reinterpret deep-learning optimizers through LMO geometry. Gluon formalizes Muon and Scion as layer-wise LMO methods over block norms $\lambda=0$8, introduces a layer-wise $\lambda=0$9-smoothness condition,

$\lambda>0$0

and derives deterministic layer-specific radii

$\lambda>0$1

thereby shifting the emphasis from transported query points to layer-wise geometry and theory–practice alignment in LLM training (Riabinin et al., 19 May 2025).

A second neighboring direction adds second-order information. “Better LMO-based Momentum Methods with Second-Order Information” extends Hessian-corrected momentum to arbitrary-norm LMO methods and proves an

$\lambda>0$2

rate under relaxed smoothness, explicitly treating LMO-based extrapolated momentum/IGT as an

$\lambda>0$3

baseline in the same framework (Khirirat et al., 15 Dec 2025). In that sense, LMO-IGT occupies an intermediate position: it improves over plain momentum without Hessian-vector products, while later second-order methods improve the asymptotic rate further.

A different, but easily confused, optimization development is Local LMO. There the update is

$\lambda>0$4

with the oracle localized to the intersection of the feasible set and a small ball around the current iterate. Local LMO is presented as a projection-free generalization of gradient descent or projected gradient descent, not as an implicit-gradient-transport method, and its theory targets PGD-like rates in convex, strongly convex, nonconvex, stochastic, and nonsmooth regimes (Richtárik et al., 9 May 2026). The shared acronym “LMO” therefore masks materially different algorithmic ideas.

6. Terminological scope and common ambiguities

The label “LMO-IGT” is unusually prone to ambiguity because both components are overloaded acronyms in arXiv-indexed research. In optimization, “LMO” denotes a linear minimization oracle and “IGT” denotes implicit gradient transport, giving the specific optimizer family discussed above (Jang et al., 7 May 2026). Outside optimization, however, “LMO” also denotes $\lambda>0$5 buffer layers in coated-conductor architectures (Troshyn et al., 2020), and it denotes the Le–Murakami–Ohtsuki invariant, functor, and spectrum in quantum topology (Kuriya, 26 Aug 2025). Likewise, “IGT” appears as the acronym for the inductive graph transformer in delivery-time estimation (Zhou et al., 2022), and in NLP-oriented documentation research it refers to interlinear glossed text, as in multilingual gloss generation with GlossLM (Ginn et al., 2024).

For that reason, “LMO-IGT” should not be read as a generic cross-disciplinary phrase. Within contemporary optimization, it has a narrow and technical meaning: an LMO-based stochastic optimizer that evaluates gradients at transported lookahead points, uses the regularized support function as its unifying stationarity measure, and achieves $\lambda>0$6 average-stationarity complexity with one stochastic gradient evaluation per iteration (Jang et al., 7 May 2026).