DDCAdam: Gauge-Equivariant Adam Optimizer

• DDCAdam is a gauge-equivariant variant of Adam that restores symmetry covariance by splitting gradients into orbit-tangent and horizontal components.
• It leverages a G-invariant metric to construct adaptive preconditioners, ensuring that optimization occurs on the quotient manifold rather than drifting along gauge orbits.
• Empirical tests demonstrate that DDCAdam significantly reduces loss gaps and improves dead-direction rate readability compared to standard Adam.

Searching arXiv for the specified papers and closely related work to ground the article. Searching arXiv for “Dead-Direction Conditioner DDCAdam” and related singular-learning papers. DDCAdam, short for Dead-Direction Conditioner Adam, is a gauge-equivariant variant of Adam for deep-network optimization. It is designed for models whose loss is invariant under continuous parameter symmetries, so that optimization effectively lives on the quotient manifold $\bar\Theta=\Theta/G$ rather than on the full parameter space $\Theta$. The method constructs a preconditioner in the orbit decomposition of a $G$-invariant metric, splitting updates into orbit-tangent and quotient-relevant components, with the stated goal that the trajectory remain a preconditioned gradient flow on the symmetry quotient rather than drifting along gauge orbits (Shirodkar, 28 Jun 2026). Within the accompanying theory of dead directions, DDCAdam is presented as the Adam-family construction that restores the symmetry covariance needed to read singular-learning rates directly from training trajectories (Shirodkar, 4 Jun 2026).

1. Concept and problem setting

In the formulation used for DDCAdam, $\Theta\subset\mathbb{R}^p$ is the open parameter set of a network computing $p_\theta(y\mid x)$ with loss

$$L(\theta)=\mathbb{E}_{(x,y)\sim \mathrm{data}}[-\log p_\theta(y\mid x)].$$

A Lie group $G$ acts freely and properly on $\Theta$ by architectural gauges that leave $L$ invariant. The paper identifies four main continuous symmetries: cross-entropy row-shift, ReLU and SwiGLU rescaling, LayerNorm and RMSNorm scale, and a per-head $O(d_{\rm head})$ attention rotation matched to RoPE (Shirodkar, 28 Jun 2026).

The motivation is that these symmetries generate orbit directions along which the loss remains constant, while the Fisher metric develops zero modes. In the associated singular-learning framework, such directions are “dead directions”: parameter-space directions in which Fisher curvature decays to zero as one approaches the singular set. DDCAdam is therefore not merely an optimizer modification for numerical stability; it is a geometric intervention aimed at preserving the quotient structure that standard per-coordinate adaptive preconditioning obscures (Shirodkar, 4 Jun 2026).

2. Quotient geometry and dead directions

The geometric setup introduces a $\Theta$0-invariant Riemannian metric $\Theta$1 on $\Theta$2 and the quotient manifold

$\Theta$3

At each $\Theta$4, the tangent space splits into a vertical subspace $\Theta$5, tangent to the group orbit, and a horizontal complement $\Theta$6, so that

$\Theta$7

Here $\Theta$8 and $\Theta$9 are the $G$0-orthogonal projectors onto the vertical and horizontal subspaces (Shirodkar, 28 Jun 2026).

This decomposition matters because dead directions are defined through the vanishing rate of KL divergence and Fisher curvature near singularities. Along a smooth path $G$1 with $G$2,

$G$3

and $G$4 is then a dead direction of KL-order $G$5. The “Fisher rate decay” theorem states that along this path

$G$6

so the log-log slope of the directional Fisher curvature recovers the KL order. The quotient theorem further states that this rate exponent is intrinsic on $G$7, that SGD on any $G$8-invariant metric realizes gradient flow on the quotient, and that standard Adam does not; DDCAdam is introduced specifically to restore this property (Shirodkar, 4 Jun 2026).

3. Algorithmic construction

Standard Adam maintains

$G$9

and updates

$\Theta\subset\mathbb{R}^p$0

The cited work argues that this per-coordinate second-moment construction breaks $\Theta\subset\mathbb{R}^p$1-equivariance under non-axis-aligned symmetries, producing “gauge-mode drift” (Shirodkar, 4 Jun 2026).

DDCAdam replaces that coordinatewise treatment by a gauge decomposition. Each gradient is split into vertical and horizontal parts,

$\Theta\subset\mathbb{R}^p$2

Two second-moment EMAs are then maintained: one on a collapsed representation of the vertical component and one on the horizontal coordinates,

$\Theta\subset\mathbb{R}^p$3

Here $\Theta\subset\mathbb{R}^p$4 collapses a vertical vector to one scalar per orbit dimension, and $\Theta\subset\mathbb{R}^p$5 is the corresponding lift back to parameter space. After bias correction, the update is

$\Theta\subset\mathbb{R}^p$6

Equivalently, the method defines a block-diagonal conditioner

$\Theta\subset\mathbb{R}^p$7

so that $\Theta\subset\mathbb{R}^p$8 (Shirodkar, 28 Jun 2026).

A closely related presentation uses a gauge specification for each parameter block $\Theta\subset\mathbb{R}^p$9: a projector $p_\theta(y\mid x)$0 onto orbit directions, a projector $p_\theta(y\mid x)$1 onto horizontals, a collapse map $p_\theta(y\mid x)$2, and its inverse $p_\theta(y\mid x)$3. In that form, DDCAdam applies decoupled weight decay, decomposes $p_\theta(y\mid x)$4 and $p_\theta(y\mid x)$5, updates separate Adam moments for the vertical scalars and horizontal coordinates, chooses a vertical update mode, and recombines via

$p_\theta(y\mid x)$6

The paper explicitly allows three vertical modes: frozen, sgd, and adam; for multiplicative gauges such as ReLU-rescale and LN-scale, recombination is performed in log-norm coordinates so that the update commutes with rescaling (Shirodkar, 4 Jun 2026).

4. Equivariance properties and quotient dynamics

The central theoretical claim is exact $p_\theta(y\mid x)$7-equivariance on an Adam base. Writing the DDCAdam preconditioner as $p_\theta(y\mid x)$8, the construction is stated to make $p_\theta(y\mid x)$9, $$L(\theta)=\mathbb{E}_{(x,y)\sim \mathrm{data}}[-\log p_\theta(y\mid x)].$$0, the vertical collapser, the lift, and the per-frame second-moment EMAs all $$L(\theta)=\mathbb{E}_{(x,y)\sim \mathrm{data}}[-\log p_\theta(y\mid x)].$$1-equivariant. Consequently, for all $$L(\theta)=\mathbb{E}_{(x,y)\sim \mathrm{data}}[-\log p_\theta(y\mid x)].$$2,

$$L(\theta)=\mathbb{E}_{(x,y)\sim \mathrm{data}}[-\log p_\theta(y\mid x)].$$3

The associated continuous-time ODE $$L(\theta)=\mathbb{E}_{(x,y)\sim \mathrm{data}}[-\log p_\theta(y\mid x)].$$4 is then $$L(\theta)=\mathbb{E}_{(x,y)\sim \mathrm{data}}[-\log p_\theta(y\mid x)].$$5-equivariant and descends under $$L(\theta)=\mathbb{E}_{(x,y)\sim \mathrm{data}}[-\log p_\theta(y\mid x)].$$6 to a well-defined preconditioned gradient flow of the quotient loss $$L(\theta)=\mathbb{E}_{(x,y)\sim \mathrm{data}}[-\log p_\theta(y\mid x)].$$7 on $$L(\theta)=\mathbb{E}_{(x,y)\sim \mathrm{data}}[-\log p_\theta(y\mid x)].$$8; the vertical summand projects to zero (Shirodkar, 28 Jun 2026).

This quotient interpretation is the bridge to geometric singular learning. On such a quotient preconditioned flow, the smallest Fisher eigenvalue is stated to decay as $$L(\theta)=\mathbb{E}_{(x,y)\sim \mathrm{data}}[-\log p_\theta(y\mid x)].$$9 along any KL-order-$G$0 horizontal canonical direction, which is presented as recovering the singular learning coefficient from the trajectory. The discrete Euler step inherits the same rate up to $G$1, with $G$2 the per-step condition number of the preconditioner. In the companion theory paper, the corresponding conclusion is that DDCAdam restores the symmetry covariance needed for direct trajectory-rate readout on $G$3, whereas standard Adam does not (Shirodkar, 4 Jun 2026).

A useful distinction follows from the construction itself. DDCAdam does not remove gauge directions from the optimizer state; it conditions them jointly through a reduced set of orbit scalars while keeping ordinary adaptive conditioning on the horizontal remainder. This suggests that the method aims to preserve gauge structure without turning the optimizer into a purely projected first-order method.

5. Empirical behavior

The reported experiments focus on three settings: over-training in a LLM, training a vision transformer from scratch, and composition with Muon on grokking transformers. The headline claim is that respecting the symmetry changes both the minimum the optimizer reaches and what it leaves measurable there (Shirodkar, 28 Jun 2026).

Setting	Baseline	DDC result
Depth-12 LM over-train	AdamW: gap $G$4	DDCAdam: gap $G$5
Depth-8 ViT-100	AdamW: val loss $G$6	DDCAdam: val loss $G$7
Depth-24 modular-addition grok	Muon: $G$8 seeds grok	DDCMuon: $G$9

In the depth-12 LLM of approximately 75 M parameters, AdamW is reported to fall into over-training collapse with validation–train loss gap approximately $\Theta$0, whereas DDCAdam resists collapse with gap approximately $\Theta$1. For “dead-direction rate” readability, among $\Theta$2 layer-by-observable cells, AdamW reads a finite-slope decay in only $\Theta$3, while DDCAdam reads it in $\Theta$4, with no overlap (Shirodkar, 28 Jun 2026).

On ImageNet-100 with a vision transformer trained from scratch, matched-decay DDCAdam reaches lower validation loss, $\Theta$5 nats versus $\Theta$6 nats, with an accuracy tie. At the same decay, DDCAdam is reported to compress spare MLP capacity much more strongly, with activation $\Theta$7 approximately $\Theta$8 smaller and thousands of dead units versus single-digit for the matched AdamW run (Shirodkar, 28 Jun 2026).

On a Muon base, where the rotation gauge is stated to compose exactly, DDCMuon groks ten of eleven seeds at depth $\Theta$9 that a plain Muon never reaches. In the curvature comparison, DDCMuon settles at a measurably less singular minimum, with smallest quotient-Fisher eigenvalue approximately $L$0 versus $L$1 (Shirodkar, 28 Jun 2026). These latter results are not DDCAdam proper, but they delimit the broader DDC construction and indicate which components of the theory extend beyond Adam.

6. Integration, cost, and limitations

The implementation prescription is operationally simple: wrap a base optimizer by providing $L$2, $L$3, and the collapse and lift maps for each gauge; track two EMAs $L$4 and $L$5; and replace the usual preconditioned update by the DDC recombination. The paper states that, in popular frameworks, this is a 200–400 LoC addition covering the four architectural gauges (Shirodkar, 28 Jun 2026).

The stated computational overhead depends on the gauge. Abelian gauges such as cross-entropy shift and ReLU/LN rescale add $L$6 work per layer. The non-abelian rotation gauge requires an eigendecomposition per head; with an adaptive trigger that recomputes only when the Gram matrix drifts, the overhead is approximately $L$7–$L$8 per step, falling to approximately $L$9 at large scale. On a Muon base, the scaled-polar orthogonalizer adds approximately $O(d_{\rm head})$0–$O(d_{\rm head})$1 wall-clock, at parity on many GPUs, versus $O(d_{\rm head})$2 cost for exact per-head SVD (Shirodkar, 28 Jun 2026).

The scope of the guarantees is narrower than the overall DDC template. Exact equivariance and the step-bias guarantee are stated to hold only for the Adam base; on a Muon base, the method still holds the orbit but loses the per-step metric conditioning. The rigorous discrete-step bias bound $O(d_{\rm head})$3 is stated for $O(d_{\rm head})$4, with standard Adam hyperparameters keeping one safely in that regime, while larger $O(d_{\rm head})$5 incurs $O(d_{\rm head})$6 corrections that remain equivariant. For the body-frame rotation gauge, equivariance is reported up to $O(d_{\rm head})$7 over $O(d_{\rm head})$8 steps because of eigenvector sign ambiguity, and sign-pinning fixes this to machine precision. The same source describes extension to Shampoo, KFAC, and other second-order methods as straightforward by replacing the per-coordinate step with an orbit-decomposed split, while also noting that reading full dead-direction order and multiplicity from canonical misalignment remains open (Shirodkar, 28 Jun 2026).