AdamO: Decoupled, Geometry-Aware Optimizers

• AdamO is a label for multiple distinct Adam-derived optimizers that decouple conventional adaptive updates from specialized geometry, norm control, or regularization mechanisms.
• Variants include decoupled orthogonal dynamics for vision/language tasks, isometry-promoting updates for continual learning, and collapse suppression for offline RL, each with unique mathematical designs and objectives.
• Empirical results show that AdamO methods can achieve higher accuracies, more robust convergence, and improved stability compared to standard AdamW across various benchmarks.

AdamO is a non-canonical name used in recent machine-learning literature for several distinct Adam-derived optimization methods rather than for a single standardized algorithm. The shared motif is not a fixed update rule but a design pattern: each method preserves an Adam-style adaptive component while decoupling an additional control mechanism that is intended to regulate geometry, norms, or regularization more explicitly than AdamW. In 2026, the name was used for Decoupled Orthogonal Dynamics in general deep learning, for an isometry-promoting optimizer in continual learning, and for a collapse-suppressed optimizer for offline reinforcement learning (Chen et al., 4 Feb 2026, Rosseau et al., 8 Jun 2026, Qiao et al., 3 May 2026).

1. Terminological scope and principal variants

The term has acquired multiple meanings in a short period. The variants differ in objective, mathematical construction, and empirical target regime.

Variant	Core decoupling mechanism	Application regime
AdamO as Decoupled Orthogonal Dynamics	Radial norm control separated from tangential Adam preconditioning	Vision and language tasks
AdamO for continual learning	Isometry-promoting regularization decoupled from Adam updates	Supervised and RL continual learning
AdamO for offline RL	Orthogonality correction decoupled from Adam moments with a task-alignment budget	Offline critic optimization

This multiplicity is central to interpreting the name correctly. A common misconception is to treat AdamO as a single optimizer comparable to Adam, AdamW, or RMSProp. The current literature instead uses the label for several optimizer constructions that share an Adam-like backbone but target different failure modes: radial oscillation and norm control, plasticity loss via dynamical-isometry degradation, or temporal-difference instability and critic collapse.

2. Decoupled Orthogonal Dynamics

In "Decoupled Orthogonal Dynamics: Regularization for Deep Network Optimizers" (Chen et al., 4 Feb 2026), AdamO is defined by an explicit orthogonal decomposition of each parameter-block update into a radial component, which changes the parameter norm, and a tangential component, which changes direction while remaining orthogonal to the parameter vector. For a parameter block $w$, the projections are

$$\varphi^{\rho}_{w}(z) :=\frac{\langle z,w\rangle}{\langle w,w\rangle}\,w, \qquad \varphi^{\theta}_{w}(z) :=z-\varphi^{\rho}_{w}(z).$$

The gradient is decomposed as $g_t=g_t^\rho+g_t^\theta$, with $g_t^\rho \parallel w$ and $g_t^\theta \perp w$. AdamO assigns norm control to an SGD-style radial channel with curvature-adaptive step size and purely radial weight decay,

$w^{\text{decay}} = (1 - \eta_{\rho,t} \lambda)\,w,$

while confining Adam-style adaptive preconditioning to the tangential subspace,

$$\Delta w_t^\theta = \eta_\theta\,\varphi_w^\theta\!\left(\frac{\hat m_t^\theta}{\sqrt{\hat v_t^\theta}+\epsilon}\right).$$

The radial step size is itself curvature-adaptive through

$$\kappa_t := \|g_t - g_{t-1}\|^2,\qquad \tau_t := \beta_\tau \tau_{t-1} + (1 - \beta_\tau)\,\kappa_t,$$

and

$$\eta_{\rho,t} := \frac{\eta_{\rho}}{\sqrt{\tau_t/\tau_{\text{target}} + \epsilon}}.$$

The motivation is the "Radial Tug-of-War" attributed to AdamW: task gradients tend to increase parameter norms, whereas weight decay suppresses norm growth, and the resulting radial oscillations enter Adam’s second-moment estimate $v_t$, thereby polluting adaptive scaling for tangential feature learning. AdamO attempts to remove that interference by separating magnitude and direction dynamics. The method also includes architecture-aware rules: low-dimensional parameters use a simplified Adam update, and scale-invariant layers receive tangential updates only.

Empirically, the paper reports substantial gains on CIFAR-100 with ResNet-18, where test accuracy is $$\varphi^{\rho}_{w}(z) :=\frac{\langle z,w\rangle}{\langle w,w\rangle}\,w, \qquad \varphi^{\theta}_{w}(z) :=z-\varphi^{\rho}_{w}(z).$$0 for AdamW and $$\varphi^{\rho}_{w}(z) :=\frac{\langle z,w\rangle}{\langle w,w\rangle}\,w, \qquad \varphi^{\theta}_{w}(z) :=z-\varphi^{\rho}_{w}(z).$$1 for full AdamO. Ablations indicate that AdamO-Isotropic is $$\varphi^{\rho}_{w}(z) :=\frac{\langle z,w\rangle}{\langle w,w\rangle}\,w, \qquad \varphi^{\theta}_{w}(z) :=z-\varphi^{\rho}_{w}(z).$$2, essentially indistinguishable from AdamW, and that removing curvature adaptation reduces performance to $$\varphi^{\rho}_{w}(z) :=\frac{\langle z,w\rangle}{\langle w,w\rangle}\,w, \qquad \varphi^{\theta}_{w}(z) :=z-\varphi^{\rho}_{w}(z).$$3. On modular-arithmetic grokking, AdamO reaches $$\varphi^{\rho}_{w}(z) :=\frac{\langle z,w\rangle}{\langle w,w\rangle}\,w, \qquad \varphi^{\theta}_{w}(z) :=z-\varphi^{\rho}_{w}(z).$$4 test accuracy, compared with $$\varphi^{\rho}_{w}(z) :=\frac{\langle z,w\rangle}{\langle w,w\rangle}\,w, \qquad \varphi^{\theta}_{w}(z) :=z-\varphi^{\rho}_{w}(z).$$5 for AdamW and $$\varphi^{\rho}_{w}(z) :=\frac{\langle z,w\rangle}{\langle w,w\rangle}\,w, \qquad \varphi^{\theta}_{w}(z) :=z-\varphi^{\rho}_{w}(z).$$6 for Adam; the first epoch with test accuracy $$\varphi^{\rho}_{w}(z) :=\frac{\langle z,w\rangle}{\langle w,w\rangle}\,w, \qquad \varphi^{\theta}_{w}(z) :=z-\varphi^{\rho}_{w}(z).$$7 is 2785 for AdamO and 2508 for AdamW. The paper also reports that AdamO has a broader high-accuracy region in 2D grid searches over $$\varphi^{\rho}_{w}(z) :=\frac{\langle z,w\rangle}{\langle w,w\rangle}\,w, \qquad \varphi^{\theta}_{w}(z) :=z-\varphi^{\rho}_{w}(z).$$8 than AdamW has over learning rate and weight decay, which it interprets as greater hyperparameter robustness.

3. Dynamical-isometry AdamO for continual learning

In "Preserving Plasticity in Continual Learning via Dynamical Isometry" (Rosseau et al., 8 Jun 2026), AdamO is a different optimizer whose purpose is to preserve plasticity under non-stationarity. The paper defines plasticity through the reduction of excess achievable loss and relates local progress under first-order methods to the empirical Neural Tangent Kernel,

$$\varphi^{\rho}_{w}(z) :=\frac{\langle z,w\rangle}{\langle w,w\rangle}\,w, \qquad \varphi^{\theta}_{w}(z) :=z-\varphi^{\rho}_{w}(z).$$9

with first-order loss change governed by

$g_t=g_t^\rho+g_t^\theta$0

Within that framework, task-agnostic plasticity is associated with keeping the NTK spectrum well conditioned, and dynamical isometry—layer-wise Jacobian singular values remaining close to one—is proposed as a practical surrogate.

The method regularizes each weight matrix toward orthogonality or pseudo-orthogonality with

$g_t=g_t^\rho+g_t^\theta$1

and then decouples the corresponding gradient from Adam’s moment estimates. If $g_t=g_t^\rho+g_t^\theta$2 is the task gradient and $g_t=g_t^\rho+g_t^\theta$3 is the gradient of the summed isometry penalty, AdamO updates parameters by

$g_t=g_t^\rho+g_t^\theta$4

This is explicitly presented as analogous to AdamW: Adam’s adaptive statistics are built only from the task loss, while the isometry-promoting step acts separately.

A distinctive claim of this work is that the regularizer can reactivate dormant ReLU units because it acts directly on weight matrices even when task gradients vanish for dead units. The paper evaluates the method on random-label memorization, pixel-permutation, label-shuffling, continual PPO on MinAtar, and continual Octax. Across these supervised and reinforcement-learning benchmarks, the proposed methods "consistently match or outperform existing approaches." Diagnostics are reported to show tighter weight spectra and higher effective rank, better-conditioned empirical NTKs, Jacobian singular values concentrated near 1, and fewer dormant neurons. For practical use, the authors recommend AdamO + ReLU as the default option, and report only a few percent overhead in wall-clock time and approximately $g_t=g_t^\rho+g_t^\theta$5–$g_t=g_t^\rho+g_t^\theta$6 additional memory.

4. Collapse-suppressed AdamO for offline reinforcement learning

In "AdamO: A Collapse-Suppressed Optimizer for Offline RL" (Qiao et al., 3 May 2026), the name designates yet another optimizer, this time for suppressing critic collapse in offline temporal-difference learning. The paper models Adam-based TD learning as a feedback system and analyzes a frozen local regime in which TD errors satisfy a second-order linear recurrence. Stability is characterized exactly by the spectral radius of an augmented update operator: $g_t=g_t^\rho+g_t^\theta$7 with

$g_t=g_t^\rho+g_t^\theta$8

and exponential convergence of TD error if and only if

$g_t=g_t^\rho+g_t^\theta$9

The induced TD dynamics operator is

$g_t^\rho \parallel w$0

where $g_t^\rho \parallel w$1 is the preconditioned Gram operator. The paper argues that Adam’s coordinate-wise preconditioning can distort parameter geometry so that the damping term $g_t^\rho \parallel w$2 no longer dominates the bootstrapped self-excitation term $g_t^\rho \parallel w$3. AdamO addresses this by adding a decoupled orthogonality correction to selected weight blocks. For a block $g_t^\rho \parallel w$4,

$g_t^\rho \parallel w$5

with orthogonality gradient $g_t^\rho \parallel w$6. Adam moments are computed only from the task gradient $g_t^\rho \parallel w$7, while the orthogonality step is scaled relative to the Adam step $g_t^\rho \parallel w$8 and restricted by a task-alignment budget

$g_t^\rho \parallel w$9

The final update is

$g_t^\theta \perp w$0

The method comes with explicit task-safety guarantees: for $g_t^\theta \perp w$1, AdamO is non-inferior to Adam at the next step under a stated smoothness-dependent condition, and for $g_t^\theta \perp w$2 the worst-case single-step degradation is bounded. The paper also shows that AdamO preserves Adam’s continuous-time dissipative Hamiltonian structure up to a controlled perturbation determined by $g_t^\theta \perp w$3.

Empirically, AdamO is reported to improve stability and normalized return across MuJoCo locomotion, AntMaze, Adroit, and FrankaKitchen, and to remain broadly compatible with TD3+BC, IQL, ReBRAC, ACTIVE, PARS, and SQOG. Average TD3+BC scores are reported as $g_t^\theta \perp w$4 on AntMaze, $g_t^\theta \perp w$5 on MuJoCo Locomotion, and $g_t^\theta \perp w$6 on Adroit when replacing Adam with AdamO. On pen-human, Adam is reported to collapse to approximately $g_t^\theta \perp w$7, whereas AdamO attains approximately $g_t^\theta \perp w$8. The computational overhead is described as roughly $g_t^\theta \perp w$9 more wall-clock time than Adam, with an example of $w^{\text{decay}} = (1 - \eta_{\rho,t} \lambda)\,w,$0 versus $w^{\text{decay}} = (1 - \eta_{\rho,t} \lambda)\,w,$1 minutes for TD3+BC over $w^{\text{decay}} = (1 - \eta_{\rho,t} \lambda)\,w,$2M steps.

5. Relation to Adam, AdamW, and Adam-like optimizer design

All three AdamO constructions are intelligible against the broader literature on Adam-like methods. The underlying Adam template remains the familiar first- and second-moment recursion with bias correction, while the differentiating feature is the decoupling of an auxiliary structural control from the adaptive preconditioner. In the Decoupled Orthogonal Dynamics paper, the conflict is between radial norm control and tangential feature learning; in the continual-learning paper, it is between isometry regularization and task gradients; in the offline-RL paper, it is between orthogonality correction and critic-loss descent. This suggests a recurring design pattern: AdamO methods are geometry-aware extensions of AdamW-style decoupling rather than a single optimizer family in the strict algorithmic sense (Chen et al., 4 Feb 2026, Rosseau et al., 8 Jun 2026, Qiao et al., 3 May 2026).

A broader analytical lens appears in "Beyond Adam: Disentangling Optimizer Effects in the Fine-Tuning of Atomistic Foundation Models" (Liu et al., 5 Dec 2025). That work writes first-order optimization generically as

$w^{\text{decay}} = (1 - \eta_{\rho,t} \lambda)\,w,$3

treats the optimizer as a data-dependent linear map applied to the gradient, and interprets $w^{\text{decay}} = (1 - \eta_{\rho,t} \lambda)\,w,$4 as a preconditioned Hessian that imposes a spectral filter on curvature. It then states, specifically for a hypothetical "AdamO," that one should specify the preconditioner $w^{\text{decay}} = (1 - \eta_{\rho,t} \lambda)\,w,$5, characterize its spectral filter, decouple curvature and regularization, test spectra- and dynamics-sensitive tasks, and possibly combine the method with a brief second-order refinement stage. A plausible implication is that this preconditioning viewpoint provides a common language for the otherwise heterogeneous AdamO proposals.

The contemporaneous theory paper "Adam Converges in Nonsmooth Nonconvex Optimization" gives a different kind of context: it proves the first finite-time guarantee for the classical form of Adam, with bias correction and without further algorithmic modifications, in nonsmooth nonconvex optimization under a randomly scaled learning rate, establishing a convergence rate of $w^{\text{decay}} = (1 - \eta_{\rho,t} \lambda)\,w,$6 and emphasizing the parameter choice $w^{\text{decay}} = (1 - \eta_{\rho,t} \lambda)\,w,$7 (Liu, 21 Jun 2026). That result does not analyze any AdamO variant directly, but it clarifies that recent AdamO proposals are not attempts to repair an optimizer with no theoretical footing; rather, they target specific geometric or dynamical pathologies that arise beyond basic convergence questions.

6. Homographs, adjacent names, and scope delimitations

The optimizer literature is not the only source of similarly named research objects. "AdaMo" in automatic code summarization is a transfer-learning model that assembles CodeBERT and GPT-2 into a single encoder-decoder architecture, adds Gaussian noise to latent representations, and introduces continuous pretraining and intermediate finetuning schemes; it is not an optimizer, despite the near-identity of the name in some queries (Gu et al., 2022).

Likewise, "ADAMO" in astroparticle physics denotes "Anisotropic detectors for DArk Matter Observation," an R&D program centered on radiopure ZnWO$w^{\text{decay}} = (1 - \eta_{\rho,t} \lambda)\,w,$8 anisotropic scintillators for directional dark-matter detection. In that setting the term refers to detector development, quenching-factor measurements, and directional modulation concepts, not to stochastic optimization (Belli et al., 2020).

For that reason, any use of "AdamO" requires local context. Within current machine-learning usage, the term is best treated as a label for several independent Adam-derived optimizers that share a decoupling philosophy but differ substantially in mathematics, target failure modes, and empirical claims.