---
title: 'LMO-IGT: Implicit Gradient Transport in LMO Methods'
url: https://www.emergentmind.com/topics/lmo-igt
type: topic
---

# LMO-IGT: Implicit Gradient Transport in LMO Methods

Searching arXiv for recent papers on “LMO-IGT” and related LMO-based optimization.

LMO-IGT is a class of stochastic linear-minimization-oracle-based optimization methods that leverages implicit gradient transport to accelerate modern LMO-based updates while preserving the single-gradient-per-iteration structure of standard stochastic LMO. It was introduced for problems of the form
\[
\min_{w\in\mathcal P} F(w), \qquad F(w):=\mathbb E_\xi[f(w;\xi)],
\]
with a unified treatment of unconstrained normalized methods and constrained Frank–Wolfe-type formulations. Its defining innovation is to evaluate stochastic gradients at transported points along the update trajectory rather than at the current iterate, and its analysis is organized around the regularized support function, a stationarity measure that bridges gradient-norm and Frank–Wolfe-gap viewpoints [2605.05577].

## 1. Optimization setting and oracle geometry

The underlying primitive is a linear minimization oracle over a compact convex set \(\mathcal C\ni 0\):
\[
\operatorname{LMO}_{\mathcal C}(g)\in \arg\min_{v\in\mathcal C}\langle g,v\rangle.
\]
Given a gradient or momentum estimate \(g_t\), an LMO-based method chooses
\[
v_t=\operatorname{LMO}_{\mathcal C}(g_t),
\]
and updates by
\[
w_{t+1}=(1-\lambda\eta_t)w_t+\eta_t v_t.
\]
When \(\lambda=0\), this is the unconstrained case. When \(\lambda>0\), with \(s_t=\lambda^{-1}v_t\in\mathcal P\) and \(\gamma_t=\lambda\eta_t\), the same rule becomes
\[
w_{t+1}=(1-\gamma_t)w_t+\gamma_t s_t,
\]
which is exactly a conditional-gradient update over \(\mathcal P=\lambda^{-1}\mathcal C\) [2605.05577].

This formulation subsumes several geometry-aware optimizers. If \(\mathcal C\) is an \(\ell_\infty\)-ball, then
\[
\operatorname{LMO}_{\mathcal C}(g)=-\operatorname{sign}(g),
\]
recovering sign-type updates such as signSGD, Signum, and Lion. If \(\mathcal C\) is the operator-norm ball for matrices,
\[
\mathcal C=\{X:\|X\|_{\mathrm{op}}\le 1\},
\]
and \(g=U\operatorname{diag}(\sigma)V^\top\) is an SVD, then
\[
\operatorname{LMO}_{\mathcal C}(g)=-UV^\top,
\]
which is the orthogonalized matrix update underlying Muon [2605.05577]. Subsequent layer-wise LMO formulations, notably Gluon, made this geometry explicit for Muon- and Scion-style training of large neural networks by assigning blockwise norms and blockwise radii \(t_i^k\) [2505.13416].

## 2. Implicit gradient transport

The motivation for LMO-IGT is the gap between plain stochastic LMO and variance-reduced LMO. Standard stochastic momentum updates evaluate gradients at \(w_t\), so the exponential moving average lags behind \(\nabla F(w_t)\). Variance-reduced LMO improves the rate by using corrections such as
\[
\nabla f(w_t;\xi_t)-\nabla f(w_{t-1};\xi_t),
\]
but this requires gradients at both the current and previous iterates, effectively using two stochastic gradient evaluations per iteration [2605.05577].

LMO-IGT keeps one stochastic gradient per iteration, but changes the query point. At iteration \(t\), it uses two momentum-like sequences,
\[
g_t = \beta_{1,t} m_{t-1} + (1-\beta_{1,t}) \nabla f(x_t;\xi_t),
\]
\[
m_t = \beta_{2,t} m_{t-1} + (1-\beta_{2,t}) \nabla f(x_t;\xi_t),
\]
with \(0\le \beta_{1,t}\le \beta_{2,t}<1\). The paper interprets \(g_t\) as a more responsive “fast” momentum and \(m_t\) as a more stable “slow” buffer. The update direction remains
\[
v_t=\operatorname{LMO}_{\mathcal C}(g_t),
\]
but both the actual iterate and the transported point are updated via
\[
x_{t+1}=(1-\lambda\eta_{1,t})w_t+\eta_{1,t}v_t,
\]
\[
w_{t+1}=(1-\lambda\eta_{2,t})w_t+\eta_{2,t}v_t.
\]

The theoretical choice
\[
\eta_{1,t}=\frac{\eta_{2,t}}{1-\beta_{2,t}}
\]
yields the lookahead relation
\[
x_{t+1}=w_{t+1}+\frac{\beta_2}{1-\beta_2}(w_{t+1}-w_t)
      =w_t+\frac{1}{1-\beta_2}(w_{t+1}-w_t).
\]
Thus \(x_t\) is a transported point obtained by extrapolating the parameter trajectory. The analysis shows that this construction cancels a first-order drift term in the tracking-error recursion for the momentum buffer, which is why the improved convergence guarantee for LMO-IGT requires a second-order smoothness assumption
\[
\|\nabla^2F(x)-\nabla^2F(y)\|_{\mathrm{op}}\le \rho\|x-y\|
\]
in addition to \(L\)-smoothness and bounded stochastic-gradient variance [2605.05577].

## 3. Regularized support function and convergence theory

A central contribution of the framework is the regularized support function
\[
\Psi_{\mathcal C,\lambda}(w) := \sup_{v\in\mathcal C}\langle -\nabla F(w),\, v-\lambda w\rangle.
\]
This unifies the unconstrained and constrained cases. When \(\lambda>0\) and \(\mathcal P=\lambda^{-1}\mathcal C\),
\[
\Psi_{\mathcal C,\lambda}(w)=\lambda G_{\mathcal P}(w),
\]
where \(G_{\mathcal P}(w)\) is the Frank–Wolfe gap. When \(\lambda=0\),
\[
\Psi_{\mathcal C,0}(w)=h_{\mathcal C}(-\nabla F(w)),
\]
the support function of \(\mathcal C\) at \(-\nabla F(w)\). For centrally symmetric \(\mathcal C\), this reduces to the dual norm induced by \(\mathcal C\); for a Euclidean ball of diameter \(R\),
\[
\Psi_{\mathcal C,0}(w)=\frac{R}{2}\|\nabla F(w)\|.
\]
The paper also states that \(\Psi_{\mathcal C,\lambda}(w)=0\) is an exact stationarity certificate: it is equivalent to KKT stationarity when \(\lambda>0\), and to \(\nabla F(w)=0\) when \(\lambda=0\) [2605.05577].

The one-step descent analysis is organized by the mismatch
\[
\hat\epsilon_t:=g_t-\nabla F(w_t).
\]
For one LMO step,
\[
F(w_{t+1}) \le F(w_t) -\eta_t\Psi_{\mathcal C,\lambda}(w_t) +\eta_t R\|\hat\epsilon_t\| +\frac{L}{2}\eta_t^2R^2.
\]
Averaging this inequality reduces convergence to bounding the estimator mismatch. The resulting comparison among the three principal regimes is as follows [2605.05577].

| Method | Average stationarity rate | Gradient evaluations per iteration |
|---|---|---|
| Stochastic LMO | \(O(T^{-1/4})\) | 1 |
| LMO-VR | \(O(T^{-1/3})\) | 2 |
| LMO-IGT | \(O(T^{-2/7})\) | 1 |

Equivalently, the iteration complexities for driving
\[
\frac1T\sum_{t=0}^{T-1}\mathbb E[\Psi_{\mathcal C,\lambda}(w_t)]
\]
below \(\varepsilon\) are \(\mathcal O(\varepsilon^{-4})\) for stochastic LMO, \(\mathcal O(\varepsilon^{-3})\) for variance-reduced LMO, and \(\mathcal O(\varepsilon^{-3.5})\) for LMO-IGT. The theory therefore places LMO-IGT between plain stochastic LMO and variance-reduced LMO: it improves the stochastic rate without paying the extra gradient cost of variance reduction [2605.05577].

## 4. Instantiations, empirical behavior, and practical constraints

The principal practical instantiations studied are Lion-IGT and Muon-IGT. On CIFAR-10 with ResNet-18, the experiments compare AdamW, NIGT, Lion, Muon, Lion-VR, Muon-VR, Lion-IGT, and Muon-IGT over 200 epochs, with curves averaged over five runs and one-standard-deviation bands. The reported metrics are test accuracy over epochs and over wall-clock time. Among the tested methods, Muon-IGT achieves the best final accuracy, while Lion-IGT and Muon-IGT run at nearly the same speed as their plain baselines because IGT introduces minimal computation and memory overhead [2605.05577].

The appendices extend the empirical picture. Additional experiments include an ablation on Muon-IGT showing that adding double momentum improves over Muon and that adding IGT on top improves further, robustness studies over learning rate and weight decay, language modeling with nanoGPT (10M) on Shakespeare, and larger-scale language modeling with nanoGPT (124M) on OpenWebText. In the reported results, Muon-IGT outperforms baselines in both training and test loss on Shakespeare and retains the same qualitative advantage on OpenWebText [2605.05577].

The practical caveats are explicit. The improved convergence guarantee depends on second-order smoothness, the results are stated in expectation for average first-order stationarity rather than for last-iterate or high-probability guarantees, and the framework does not cover biased or approximate LMOs. Implementation also requires maintaining both \(x_t\) and \(w_t\), and the paper’s own hyperparameters show that IGT variants may need retuning rather than direct reuse of the base optimizer’s settings. For CIFAR-10 with ResNet-18, for example, the reported Muon setting is
\[
\eta=5\times 10^{-2},\ \beta_1=0.99,\ \beta_2=0.99,\ \text{weight decay}=5\times 10^{-4},
\]
whereas Muon-IGT uses
\[
\eta=5\times 10^{-4},\ \beta_1=0.9,\ \beta_2=0.99,\ \text{weight decay}=5\times 10^{-1}.
\]
A plausible implication is that the transport mechanism changes the effective optimization geometry enough that hyperparameter transfer from the base method is not automatic [2605.05577].

## 5. Relation to adjacent LMO-based optimizer research

LMO-IGT is part of a broader recent effort to reinterpret deep-learning optimizers through LMO geometry. Gluon formalizes Muon and Scion as layer-wise LMO methods over block norms \(\|\cdot\|_{(i)}\), introduces a layer-wise \((L^0,L^1)\)-smoothness condition,
\[
\|\nabla_i f(X)-\nabla_i f(Y)\|_{(i)\star} \le \left(L_i^0 + L_i^1 \|\nabla_i f(X)\|_{(i)\star}\right)\|X_i-Y_i\|_{(i)},
\]
and derives deterministic layer-specific radii
\[
t_i^k = \frac{\|\nabla_i f(X^k)\|_{(i)\star}}{L_i^0 + L_i^1\|\nabla_i f(X^k)\|_{(i)\star}},
\]
thereby shifting the emphasis from transported query points to layer-wise geometry and theory–practice alignment in LLM training [2505.13416].

A second neighboring direction adds second-order information. “Better LMO-based Momentum Methods with Second-Order Information” extends Hessian-corrected momentum to arbitrary-norm LMO methods and proves an
\[
O(K^{-1/3})
\]
rate under relaxed smoothness, explicitly treating LMO-based extrapolated momentum/IGT as an
\[
O(K^{-2/7})
\]
baseline in the same framework [2512.13227]. In that sense, LMO-IGT occupies an intermediate position: it improves over plain momentum without Hessian-vector products, while later second-order methods improve the asymptotic rate further.

A different, but easily confused, optimization development is Local LMO. There the update is
\[
x_{k+1}\in \arg\min_{z\in\mathcal X\cap\mathcal B(x_k,t_k)}\langle \nabla f(x_k), z\rangle,
\]
with the oracle localized to the intersection of the feasible set and a small ball around the current iterate. Local LMO is presented as a projection-free generalization of gradient descent or projected gradient descent, not as an implicit-gradient-transport method, and its theory targets PGD-like rates in convex, strongly convex, nonconvex, stochastic, and nonsmooth regimes [2605.08850]. The shared acronym “LMO” therefore masks materially different algorithmic ideas.

## 6. Terminological scope and common ambiguities

The label “LMO-IGT” is unusually prone to ambiguity because both components are overloaded acronyms in arXiv-indexed research. In optimization, “LMO” denotes a linear minimization oracle and “IGT” denotes implicit gradient transport, giving the specific optimizer family discussed above [2605.05577]. Outside optimization, however, “LMO” also denotes \(\mathrm{La}_{1-x}\mathrm{Mn}_{1-y}\mathrm{O}_{1\pm\delta}\) buffer layers in coated-conductor architectures [2012.01318], and it denotes the Le–Murakami–Ohtsuki invariant, functor, and spectrum in quantum topology [2508.18985]. Likewise, “IGT” appears as the acronym for the inductive graph transformer in delivery-time estimation [2211.02863], and in NLP-oriented documentation research it refers to interlinear glossed text, as in multilingual gloss generation with GlossLM [2403.06399].

For that reason, “LMO-IGT” should not be read as a generic cross-disciplinary phrase. Within contemporary optimization, it has a narrow and technical meaning: an LMO-based stochastic optimizer that evaluates gradients at transported lookahead points, uses the regularized support function as its unifying stationarity measure, and achieves \(\mathcal O(\varepsilon^{-3.5})\) average-stationarity complexity with one stochastic gradient evaluation per iteration [2605.05577].

Source: https://www.emergentmind.com/topics/lmo-igt