Cross-Optimizer Invariant Diagnostics

• Cross-Optimizer Invariant Diagnostics use stable invariant quantities—such as NC₀, layer gradient attributions, and quotient-space observables—to compare optimizer behaviors beyond traditional loss and accuracy metrics.
• The method leverages tests based on geometric, spectral, and misclassification signals to identify necessary conditions for phenomena like Neural Collapse and to isolate optimizer-specific effects.
• Empirical studies show that factors like coupled weight decay and invariant preservation correlate with faster convergence, accurate error localization, and efficient retraining strategies.

In recent arXiv literature, a cross-optimizer invariant diagnostic denotes a diagnostic construction that compares optimizer behavior by anchoring the comparison to an invariant object or to a measurement whose interpretation remains stable across optimizer choices. The invariant may be a necessary condition for a geometric phenomenon, a semantic-equivalence criterion, a per-layer error-localization score, an optimizer-regularized norm or spectral signature, a data-spectrum predictor of scaling behavior, or a quotient-space quantity defined by gauge symmetry. Taken together, these diagnostics show that optimizer comparisons need not be restricted to train loss or final accuracy; they can instead be organized around invariants of representation geometry, compiler correctness, spectral structure, or reparameterization class (Zhao et al., 18 Feb 2026, Pasichnyk, 30 Mar 2026, Louloudakis et al., 3 May 2025, Zhang et al., 11 May 2026, Ramani et al., 28 May 2026, Shirodkar, 28 Jun 2026).

1. Conceptual scope and recurring structures

A recurring pattern across these works is that the word invariant does not mean that all optimizers produce the same trajectory or the same minimum. Rather, it refers to a quantity or criterion whose meaning is stable enough to support comparison: the requirement that an optimized ONNX graph remain semantically equivalent to the original graph, the identification of the same “problem layers” under SGD and Adam, the vanishing of a necessary condition for Neural Collapse, the preservation of stable rank along same-optimizer interpolation paths, the prediction of exponent shift from the data spectrum alone, or the reparameterization-invariance of a quotient-space observable (Zhao et al., 18 Feb 2026, Pasichnyk, 30 Mar 2026, Louloudakis et al., 3 May 2025, Zhang et al., 11 May 2026, Ramani et al., 28 May 2026, Shirodkar, 28 Jun 2026).

Diagnostic	Core quantity	Reported role
NC₀	$\|W1\|_2^2$	Necessary condition for NC
Layer attribution score	$$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$	Identifies “problem layers”
OODTE invariants	$\mathrm{Valid}(M')$ and task-specific agreement	Checks crash-freedom and semantic equivalence
Mode-connectivity diagnostics	loss barrier, stable rank, $R_d(W,\alpha)$	Separates same- and cross-optimizer structure
Scaling-law diagnostic	$\Delta\alpha_P(s)$	Forecasts optimizer gain from spectrum
Dead-Direction Rate	$\mathrm{DDR}(t)=\frac1k\#\{i:v^V_{t,i}<\delta\}$	Gauge-invariant quotient diagnostic

This breadth matters because optimizer dependence appears in several distinct forms. In some settings the invariant is expected to hold and its violation signals a bug or a structural impossibility. In others, the invariant is what remains unchanged while optimizer-specific effects appear in the residual degrees of freedom. The result is not a single metric but a family of diagnostics organized around optimizer-robust interpretation.

2. NC₀ as a necessary-condition diagnostic for Neural Collapse

For Neural Collapse, the proposed cross-optimizer diagnostic is NC₀. Let $W\in\mathbb R^{K\times P}$ be the last-layer weight matrix of a $K$-way classifier with no bias, and let $1\in\mathbb R^P$ be the all-ones vector. NC₀ is defined by

$\mathrm{NC}_0(W)\;\coloneqq\;\|W1\|_2^2.$

Equivalently, if $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$0 is the $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$1-th row of $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$2, then

$$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$3

so $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$4 precisely when each row-sum $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$5. The diagnostic is motivated by a necessity result: if the centered class means $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$6 and the weight rows $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$7 both converge, up to normalization, to the same simplex equiangular tight frame, then $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$8, hence $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$9. In the paper’s proof sketch, NC₂ gives $\mathrm{Valid}(M')$0 after centering, while NC₃ gives $\mathrm{Valid}(M')$1, so $\mathrm{Valid}(M')$2 (Zhao et al., 18 Feb 2026).

The point of NC₀ is tractability. Unlike NC₁–NC₃, which involve class-conditional feature covariances and singular values of $\mathrm{Valid}(M')$3 or $\mathrm{Valid}(M')$4, NC₀ is the simple squared $\mathrm{Valid}(M')$5-norm of $\mathrm{Valid}(M')$6. This permits closed-form optimizer-dependent dynamics under the last-layer linear model $\mathrm{Valid}(M')$7 and cross-entropy loss. For SGD with decoupled weight decay,

$\mathrm{Valid}(M')$8

and for $\mathrm{Valid}(M')$9, if $R_d(W,\alpha)$0, then

$R_d(W,\alpha)$1

exponentially. For SGD with coupled weight decay,

$R_d(W,\alpha)$2

and if $R_d(W,\alpha)$3, then $R_d(W,\alpha)$4 again decays exponentially. By contrast, in the unconstrained-features model with fixed features $R_d(W,\alpha)$5 being a simplex ETF, SignGD with decoupled weight decay,

$R_d(W,\alpha)$6

has constant sign-pattern $R_d(W,\alpha)$7, keeps $R_d(W,\alpha)$8 proportional to $R_d(W,\alpha)$9, and yields $\Delta\alpha_P(s)$0 that increases from $\Delta\alpha_P(s)$1 and converges to

$\Delta\alpha_P(s)$2

With SignGD and coupled weight decay,

$\Delta\alpha_P(s)$3

and a suitable learning-rate schedule $\Delta\alpha_P(s)$4, one can prove $\Delta\alpha_P(s)$5 as $\Delta\alpha_P(s)$6.

These results isolate the role of weight-decay coupling. Under SGD, NC₀ tends to zero with either decoupled or coupled weight decay. Under decoupled-decay adaptive methods, described in the paper as AdamW/SignumW, NC₀ does not tend to zero. Under coupled-decay adaptive methods, described as Adam/Signum with in-gradient weight decay, NC₀ may tend to zero, often more slowly or nonmonotonically.

The empirical study consists of 3,900+ runs with ResNet9 and VGG9 on MNIST, FashionMNIST, and CIFAR-10, together with an unconstrained-feature toy model and preliminary ViT results. The reported observations are that only methods with coupled weight decay produce $\Delta\alpha_P(s)$7 in finite epochs; AdamW and SignumW yield NC₀ values $\Delta\alpha_P(s)$8–$\Delta\alpha_P(s)$9 larger and never collapse to zero despite high train accuracy; momentum in SGD accelerates the decay of NC₀ beyond simply speeding loss convergence; and NC₀ correlates with NC₃ with $\mathrm{DDR}(t)=\frac1k\#\{i:v^V_{t,i}<\delta\}$0–$\mathrm{DDR}(t)=\frac1k\#\{i:v^V_{t,i}<\delta\}$1, less so with NC₁ and NC₂. The caveat is explicit: vanishing NC₀ is necessary but not sufficient for full NC, whereas NC₀ divergence is a definitive sign that NC cannot occur.

3. Misclassification-only gradient attribution and optimizer-invariant layer localization

A different use of the phrase appears in the diagnostic built from beta-scheduling and misclassification-only gradient attribution. The momentum schedule is

$\mathrm{DDR}(t)=\frac1k\#\{i:v^V_{t,i}<\delta\}$2

derived from the critically damped harmonic oscillator, with clamping to $\mathrm{DDR}(t)=\frac1k\#\{i:v^V_{t,i}<\delta\}$3 for numerical stability. Under this $\mathrm{DDR}(t)=\frac1k\#\{i:v^V_{t,i}<\delta\}$4-schedule, the training dynamics remain near the critically damped regime, and the diagnostic is obtained by collecting all test inputs that the model misclassifies, back-propagating their losses, and measuring, for each layer $\mathrm{DDR}(t)=\frac1k\#\{i:v^V_{t,i}<\delta\}$5, the aggregate gradient magnitude

$\mathrm{DDR}(t)=\frac1k\#\{i:v^V_{t,i}<\delta\}$6

Layers whose gradients dominate this misclassification-only signal are declared “problem layers.” In the ResNet-18/CIFAR-10 experiments, ranking by $\mathrm{DDR}(t)=\frac1k\#\{i:v^V_{t,i}<\delta\}$7 and thresholding at the median or a small percentile produced a sparse cut with 3 of 7 layer-groups above threshold (Pasichnyk, 30 Mar 2026).

The reported cross-optimizer invariance is exact at the level of the identified layer set. Applied to a ResNet-18 trained with SGD and constant $\mathrm{DDR}(t)=\frac1k\#\{i:v^V_{t,i}<\delta\}$8, and to the same architecture trained with Adam, the top-scoring layers are identical: $\mathrm{DDR}(t)=\frac1k\#\{i:v^V_{t,i}<\delta\}$9, giving 3/3 shared and 100% overlap. The paper’s interpretation is that $W\in\mathbb R^{K\times P}$0 measures an architectural bottleneck rather than an artifact of the optimizer’s trajectory. This holds despite different error counts: 453 errors for the SGD baseline and 647 for the Adam model.

The same work reports convergence effects under the schedule. On ResNet-18/CIFAR-10, epochs to 90% accuracy are 100 for the baseline with $W\in\mathbb R^{K\times P}$1, 144 for 1cycle, and 52 for the Physics $W\in\mathbb R^{K\times P}$2-schedule, corresponding to a 1.9$W\in\mathbb R^{K\times P}$3 speedup over baseline. The qualitative damping analysis states that the Physics schedule stays near critical damping throughout, whereas the baseline spends 85% of training underdamped. The paper treats these speedups as a by-product; the main contribution is the use of the schedule as a principled, parameter-free tool for localizing and correcting specific failure modes in trained networks.

The correction procedure is explicitly “surgical.” After identifying the three “sick” layers, all other layers are frozen, while $W\in\mathbb R^{K\times P}$4, $W\in\mathbb R^{K\times P}$5, and $W\in\mathbb R^{K\times P}$6 are unfrozen, amounting to approximately 29% of parameter tensors. Retraining for 30 epochs with reduced max-LR $W\in\mathbb R^{K\times P}$7 under the same $W\in\mathbb R^{K\times P}$8-schedule fixes 62 errors, introduces 55 new ones, and yields a net $W\in\mathbb R^{K\times P}$9 improvement while retraining only 15% of the original 200-epoch budget, for 82% compute savings versus full retraining. When the iKFAD per-parameter friction metric is used to select layers instead, the same correction fixes 62 errors and introduces only 40 new ones, giving net $K$0. Parameter-level correction of the top 15–30% of parameters by friction fixes only 42–46 errors and yields negative net improvement.

4. Semantic invariants and differential testing of model optimizers

In OODTE, the invariant is semantic preservation under graph optimization. Two invariants are defined between an original model $K$1 and its optimized variant $K$2. Invariant $K$3 is crash-freedom and graph validity: $K$4 Invariant $K$5 is semantic equivalence on a test set $K$6, with task-specific comparators. For regression or dense-tensor outputs, OODTE uses an $K$7-norm bound; for classification top-$K$8 lists it uses Kendall’s Tau and requires $K$9 on each input; for object detection and segmentation it requires precision, recall, F1, mAP, and average IoU at thresholds $1\in\mathbb R^P$0 to differ by at most a small $1\in\mathbb R^P$1, often $1\in\mathbb R^P$2; and for text generation it requires $1\in\mathbb R^P$3 (Louloudakis et al., 3 May 2025).

The differential-testing pipeline has four named modules. The Model Orchestrator fetches ONNX models and their configurations. The Optimizer Module applies either the default bundle of “fuse” + “eliminate” passes or all 47 passes in one atomic sweep. The Runner Module splits the test dataset into chunks to fit memory and executes both $1\in\mathbb R^P$4 and $1\in\mathbb R^P$5 under ONNX Runtime. The Metrics Comparison Module selects the task-appropriate comparator and flags a violation if any per-input or aggregate metric falls outside the requirement. If a violation or crash is found, OODTE re-invokes the Optimizer Module in per-pass mode, applying each individual pass $1\in\mathbb R^P$6 and re-running evaluation to localize the offending pass. In practice, OODTE stops early if it finds the first pass that triggers the failure.

The decision rule is intentionally strict. For classification, any input with $1\in\mathbb R^P$7 counts as a discrepancy, and a pass is flagged if the affected percentage exceeds 0%. For detection and segmentation, any nonzero difference in average precision, recall, F1, or IoU is treated as a violation. For text, BLEU strictly below 1 is flagged. For binary sentiment, any label flip is flagged. No formal $1\in\mathbb R^P$8-values are computed; OODTE uses strict equality or tolerance zero wherever full equivalence is expected.

The evaluation covers 130 ONNX models: 93 image-classification networks, 30 object-detection or segmentation networks, and 7 transformers. Under the all-pass run, 31 out of 130 model instances, approximately 23.8%, either caused the optimizer to crash or yielded an invalid ONNX graph. Restricting to the default fuse+eliminate bundle, 12 instances, 9.2%, crashed or produced invalid models. For accuracy deviations without crash, 28 of 93 classification models, 30.1%, saw a drop in top-5 or top-10 accuracy; 5 of 30 detection/segmentation models, 16.7%, saw metric breaks; and transformers were described as extremely robust, with only GPT-2 exhibiting a trivial BLEU drop below 0.001% for a handful of token predictions. OODTE discovered 15 distinct issues, 14 previously unknown, affecting 9 of 47 optimization passes and the optimizer overall.

The broader methodological significance is explicit. OODTE is described as a “cross-optimizer invariant diagnostics” approach reusable for any graph- or IR-based optimizer, provided that the optimizer exposes a runtime API to apply passes in bulk or one-by-one, the optimized artifact is executable under a common runner, and a task-appropriate comparator exists to quantify semantic difference.

5. Optimizer-constrained geometry, spectra, and scaling exponents

In work on optimizer-induced mode connectivity, the invariant is attached to the optimizer-regularized zero-loss set rather than to raw parameter space. For a two-layer ReLU network

$1\in\mathbb R^P$9

with zero-loss set

$\mathrm{NC}_0(W)\;\coloneqq\;\|W1\|_2^2.$0

the optimizer-regularized solution set is

$\mathrm{NC}_0(W)\;\coloneqq\;\|W1\|_2^2.$1

The implicit-bias constraints differ by optimizer: AdamW corresponds to $\mathrm{NC}_0(W)\;\coloneqq\;\|W1\|_2^2.$2, while Muon corresponds to $\mathrm{NC}_0(W)\;\coloneqq\;\|W1\|_2^2.$3 with dual norm $\mathrm{NC}_0(W)\;\coloneqq\;\|W1\|_2^2.$4. Theorems 3.1 and 3.2 state that, for sufficiently large width, same-optimizer solution sets are path-connected for normalized GD and Muon, and also for Signum and AdamW. Theorem 3.3 then characterizes inter-optimizer interaction: at large width, two optimizer-induced regions can overlap or be disjoint depending on regularization, and the union is connected precisely when the regions touch. The finite-width construction is sharper: for a handcrafted dataset with width $\mathrm{NC}_0(W)\;\coloneqq\;\|W1\|_2^2.$5, AdamW and Muon converge to disconnected zero-loss components, and any path joining them must leave the zero-loss set and incur a loss barrier at least $\mathrm{NC}_0(W)\;\coloneqq\;\|W1\|_2^2.$6. In GPT-2 pretraining, same-optimizer paths have tiny loss barrier, reported as below $\mathrm{NC}_0(W)\;\coloneqq\;\|W1\|_2^2.$7, stable rank invariant up to $\mathrm{NC}_0(W)\;\coloneqq\;\|W1\|_2^2.$8, and norms approximately $\mathrm{NC}_0(W)\;\coloneqq\;\|W1\|_2^2.$9; cross-optimizer AdamW$$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$00Muon paths still have low loss but stable rank increases monotonically, the maximum singular-value outliers of AdamW shrink, the spectrum becomes more isotropic, and the path visits novel spectral configurations not reachable by either optimizer alone (Zhang et al., 11 May 2026).

In work on optimizer dependence of neural scaling laws, the invariant diagnostic is the mapping from data-spectrum steepness to optimizer-induced exponent shift. The scaling law is

$$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$01

with $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$02 estimated by ordinary least squares on $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$03 over the largest five widths, $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$04. The data covariance has eigenvalues

$$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$05

and natural-language embedding covariances are reported around $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$06–$$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$07. Five optimizer variants are studied: GD with preconditioner $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$08, Diagonal with $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$09, Full NG with $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$10, Sign-GD, and Matrix-Sign with $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$11. The central quantity is

$$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$12

The informal spectral heuristic states that $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$13 for any $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$14 that down-weights high-variance modes, that $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$15 grows with $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$16 until finite-size or budget saturation, and that $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$17 as $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$18. Numerically, at $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$19, the paper reports $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$20 and $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$21, a $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$22 larger fitted exponent for Full NG, with Matrix-Sign at approximately $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$23 and the Diagonal method at approximately $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$24. The practical rule is three-step: estimate $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$25 from the data spectrum; if $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$26, then $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$27 and SGD-like optimizers suffice; if $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$28, then $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$29 grows roughly monotonically up to $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$30, and one can read off or interpolate the expected exponent shift. Because a nonzero $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$31 compounds with each doubling of model size, the diagnostic forecasts optimizer benefit without retraining a full suite of models (Ramani et al., 28 May 2026).

These two strands address different invariants but are structurally related. The mode-connectivity work tracks what is preserved inside an optimizer’s own implicit-bias region and what changes along cross-optimizer interpolation. The scaling-law work uses the data spectrum as an optimizer-agnostic predictor of when advanced preconditioning should alter asymptotic behavior. In both cases, the comparison is organized around a quantity that is more stable than raw parameter coordinates.

6. Gauge-equivariant quotient diagnostics and recurrent caveats

The Dead-Direction Conditioner defines a cross-optimizer invariant diagnostic on the quotient manifold of gauge-equivalent parameters. Let $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$32 be parameter space and let a Lie group $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$33 act so that the loss $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$34 and model $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$35 are $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$36-invariant. The examples listed are the cross-entropy logit shift, ReLU-pair rescaling, LayerNorm-scale, and per-head attention rotation. Passing to the quotient $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$37, and choosing a $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$38-invariant Riemannian metric $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$39, one decomposes the tangent space as

$$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$40

with projectors $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$41 and $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$42. DDC tracks separate EMAs of squared gradients in the vertical and horizontal subspaces and uses them in a $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$43-equivariant update. The diagnostic observable is the Dead-Direction Rate: $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$44 where $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$45 and $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$46 are the vertical EMAs. Because $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$47 is $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$48-invariant and $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$49 is intrinsic to $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$50, DDR is invariant under any reparameterization $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$51. The paper argues that under any $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$52-equivariant optimizer, including DDCAdam and DDCMuon, DDR measures how many gauge directions the preconditioner has collapsed, whereas under non-equivariant optimizers such as AdamW or plain Muon, coordinate leaks inflate some orbit directions and keep DDR artificially low (Shirodkar, 28 Jun 2026).

The empirical results are reported for several settings. In a depth-12 LLM on an Adam base in an over-training regime, DDCAdam holds a validation–train loss gap of $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$53 nats against $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$54 for AdamW, and reads the dead-direction rate in 32 of 65 layer-by-observable cells where AdamW reads it in 7. In a ViT-S trained from scratch on ImageNet-100, validation loss is $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$55 nats for DDCAdam and $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$56 nats for AdamW, with activation smallest singular value $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$57 against $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$58, and “dead feed-forward units” reported as thousands versus handful. On a Muon base in grokking arithmetic at depth 24, DDCMuon reaches grok in 10 of 11 seeds whereas plain Muon reaches 0 of 11, with validation loss $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$59 bpb against $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$60 bpb.

Several caveats recur across the literature. Vanishing NC₀ is necessary but not sufficient for full Neural Collapse, even though its divergence guarantees that full Neural Collapse cannot emerge (Zhao et al., 18 Feb 2026). OODTE’s semantic-equivalence tests deliberately use strict equality or tolerance zero wherever full equivalence is expected, and no formal $$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$61-values are computed (Louloudakis et al., 3 May 2025). In optimizer-induced mode connectivity, low loss along a cross-optimizer path does not imply preservation of optimizer-specific spectral invariants; the AdamW$$A_\ell=\sum_{i\in\mathcal D_{\rm err}}\|g_i^{(\ell)}\|_2$$62Muon path can remain smooth in loss while stable rank drifts monotonically and the spectrum becomes more isotropic (Zhang et al., 11 May 2026). In scaling-law experiments, whether and how the measured exponent shift transfers to large-scale LLM training remains an important open question (Ramani et al., 28 May 2026).

The broader implication is methodological. These diagnostics do not erase optimizer dependence; they make it legible. Some do so by identifying a quantity that should be preserved and then detecting its failure. Others do so by locating an optimizer-independent coordinate system—semantic outputs, misclassification-only layer gradients, spectral exponents, or quotient-space orbit coordinates—from which optimizer-specific structure becomes measurable rather than confounded with parameterization.