Approx. Orthogonalized Gradient Oracle

• The paper demonstrates that the approximately orthogonalized gradient oracle corrects stochastic gradients to suppress nuisance errors, yielding a higher-order bias reduction.
• It employs a residualization strategy by subtracting the nuisance-predictable gradient component, thereby stabilizing the optimization trajectory in plug-in SGD.
• The approach theoretically interpolates between nuisance-sensitive and nuisance-insensitive regimes, achieving bias rates of order O(||hat g - g₀||⁴) under precise conditions.

An Approximately Orthogonalized Gradient Oracle is a stochastic gradient oracle modified to reduce sensitivity to nuisance-parameter estimation error in optimization problems whose objective depends on an unknown nuisance $g_0$. In "Stochastic Gradients under Nuisances" (Yu et al., 28 Aug 2025), it is defined as the estimated Neyman-orthogonalized score

$$\hat S^{\mathrm{no}}(\theta,g;z)=S_\theta(\theta,g;z)-\hat\Gamma\,\nabla_g\ell(\theta,g;z),$$

where $S_\theta$ is the ordinary $\theta$-gradient score, $\nabla_g\ell$ is the nuisance gradient, and $\hat\Gamma$ estimates an ideal orthogonalizing operator $\Gamma_0$. The construction does not alter the nuisance estimate itself; instead, it changes the stochastic update direction so that the optimization trajectory is less distorted by plug-in nuisance error. In the paper’s formulation, this produces an interpolation between nuisance-sensitive and nuisance-insensitive stochastic optimization regimes (Yu et al., 28 Aug 2025).

1. Formal setting

The relevant optimization problem is a population risk

$$\min_{\theta\in\Theta} L_0(\theta),\qquad L_0(\theta):=\mathbb{E}_{Z\sim P}[\ell_0(\theta;Z)],$$

but in practice the learner only has access to a partially specified class

$$L(\theta,g)=\mathbb{E}_{Z\sim P}[\ell(\theta,g;Z)],$$

with target parameter $\theta\in\Theta\subseteq\mathbb{R}^d$ and nuisance parameter $$\hat S^{\mathrm{no}}(\theta,g;z)=S_\theta(\theta,g;z)-\hat\Gamma\,\nabla_g\ell(\theta,g;z),$$0, where $$\hat S^{\mathrm{no}}(\theta,g;z)=S_\theta(\theta,g;z)-\hat\Gamma\,\nabla_g\ell(\theta,g;z),$$1 may be infinite-dimensional. The target of interest is

$$\hat S^{\mathrm{no}}(\theta,g;z)=S_\theta(\theta,g;z)-\hat\Gamma\,\nabla_g\ell(\theta,g;z),$$2

yet the learner observes only an estimate $$\hat S^{\mathrm{no}}(\theta,g;z)=S_\theta(\theta,g;z)-\hat\Gamma\,\nabla_g\ell(\theta,g;z),$$3, often computed from data independent of the SGD stream (Yu et al., 28 Aug 2025).

A stochastic gradient oracle $$\hat S^{\mathrm{no}}(\theta,g;z)=S_\theta(\theta,g;z)-\hat\Gamma\,\nabla_g\ell(\theta,g;z),$$4 is any map satisfying

$$\hat S^{\mathrm{no}}(\theta,g;z)=S_\theta(\theta,g;z)-\hat\Gamma\,\nabla_g\ell(\theta,g;z),$$5

For the standard plug-in method, the score is

$$\hat S^{\mathrm{no}}(\theta,g;z)=S_\theta(\theta,g;z)-\hat\Gamma\,\nabla_g\ell(\theta,g;z),$$6

and the SGD recursion is

$$\hat S^{\mathrm{no}}(\theta,g;z)=S_\theta(\theta,g;z)-\hat\Gamma\,\nabla_g\ell(\theta,g;z),$$7

The nuisance parameter creates two distinct difficulties. Replacing $$\hat S^{\mathrm{no}}(\theta,g;z)=S_\theta(\theta,g;z)-\hat\Gamma\,\nabla_g\ell(\theta,g;z),$$8 by $$\hat S^{\mathrm{no}}(\theta,g;z)=S_\theta(\theta,g;z)-\hat\Gamma\,\nabla_g\ell(\theta,g;z),$$9 changes the population objective from $S_\theta$0 to $S_\theta$1, so the drift is toward the wrong vector field. In addition, the entire optimization trajectory is perturbed because the dynamics are driven by $S_\theta$2 rather than $S_\theta$3. In the non-asymptotic analysis of (Yu et al., 28 Aug 2025), these effects appear as a bias radius in the convergence bound.

2. Neyman orthogonality and nuisance insensitivity

The paper formulates Neyman orthogonality directly at the level of the population gradient oracle. With directional derivative notation

$S_\theta$4

the oracle is Neyman orthogonal at $S_\theta$5 if

$S_\theta$6

and equivalently, at the risk level,

$S_\theta$7

This means that the first derivative of the $S_\theta$8-gradient with respect to nuisance perturbations vanishes at the target (Yu et al., 28 Aug 2025).

The resulting optimization consequence is that nuisance perturbations enter the drift field only at second order. The paper states this as the source of a “nuisance insensitive” regime. Under standard smoothness, strong convexity, and moment assumptions, baseline plug-in SGD satisfies

$S_\theta$9

whereas under Neyman orthogonality plus a higher-order smoothness condition,

$\theta$0

Accordingly, without orthogonality the bias radius is typically of order $\theta$1, while with orthogonality it improves to order $\theta$2. The Approximately Orthogonalized Gradient Oracle is introduced to recover this higher-order nuisance suppression even when the original loss is not exactly Neyman orthogonal.

3. Construction of the orthogonalized oracle

The construction begins from an ideal orthogonalization. In finite-dimensional nuisance spaces $\theta$3, the orthogonalizing matrix is motivated by the least-squares regression

$\theta$4

In the negative log-likelihood case $\theta$5, this yields

$\theta$6

where $\theta$7 and $\theta$8 (Yu et al., 28 Aug 2025).

The ideal orthogonalized score is then

$\theta$9

The paper generalizes this construction to infinite-dimensional Hilbert nuisance spaces $\nabla_g\ell$0. By Riesz representation,

$\nabla_g\ell$1

The cross-derivative object $\nabla_g\ell$2 is defined coordinatewise by

$\nabla_g\ell$3

and the nuisance Hessian operator $\nabla_g\ell$4 by

$\nabla_g\ell$5

Assuming $\nabla_g\ell$6 is invertible, the orthogonalizing operator is

$\nabla_g\ell$7

The ideal Neyman-orthogonalized oracle is therefore

$\nabla_g\ell$8

and the paper proves that this oracle is Neyman orthogonal at $\nabla_g\ell$9.

The Approximately Orthogonalized Gradient Oracle is obtained by replacing $\hat\Gamma$0 with an estimator $\hat\Gamma$1: $\hat\Gamma$2 Its approximation quality is measured by

$\hat\Gamma$3

The key conceptual point is that the correction is applied to the gradient oracle itself. The nuisance estimate remains plug-in, but the update direction is adjusted by subtracting the component of the $\hat\Gamma$4-gradient that is predictable from nuisance perturbations through $\hat\Gamma$5. The paper explicitly characterizes this as an influence-function, control-variate, or residualization style correction.

4. Orthogonalized SGD and convergence behavior

Using the approximate oracle, the orthogonalized SGD update is

$\hat\Gamma$6

Algorithmically, one first estimates $\hat\Gamma$7, often from an auxiliary sample independent of the SGD stream, and separately estimates $\hat\Gamma$8. Then, on fresh optimization samples $\hat\Gamma$9, one computes $\Gamma_0$0, computes $\Gamma_0$1, forms the corrected score, and updates $\Gamma_0$2 (Yu et al., 28 Aug 2025).

The theorem for the approximately orthogonalized method requires, among other conditions, first-order optimality for the orthogonalized score, smoothness and strong monotonicity of the orthogonalized drift, second-moment control for nuisance-gradient terms, second-order smoothness, and a higher-order smoothness bound for $\Gamma_0$3. It also requires

$\Gamma_0$4

so that the corrected vector field retains the required contraction structure.

The resulting convergence bound contains three terms: a geometric contraction term, a stochastic term proportional to $\Gamma_0$5, and a nuisance term of the form

$\Gamma_0$6

This is the paper’s precise interpolation result. Ordinary plug-in SGD has nuisance bias $\Gamma_0$7; exact orthogonality has nuisance bias $\Gamma_0$8; approximate orthogonalization yields

$\Gamma_0$9

If

$$\min_{\theta\in\Theta} L_0(\theta),\qquad L_0(\theta):=\mathbb{E}_{Z\sim P}[\ell_0(\theta;Z)],$$0

then the mixed term is also $$\min_{\theta\in\Theta} L_0(\theta),\qquad L_0(\theta):=\mathbb{E}_{Z\sim P}[\ell_0(\theta;Z)],$$1, so approximate orthogonalization recovers the orthogonal rate up to constants. If instead

$$\min_{\theta\in\Theta} L_0(\theta),\qquad L_0(\theta):=\mathbb{E}_{Z\sim P}[\ell_0(\theta;Z)],$$2

the mixed term behaves like $$\min_{\theta\in\Theta} L_0(\theta),\qquad L_0(\theta):=\mathbb{E}_{Z\sim P}[\ell_0(\theta;Z)],$$3, resembling the non-orthogonal regime. In that sense, the quality of $$\min_{\theta\in\Theta} L_0(\theta),\qquad L_0(\theta):=\mathbb{E}_{Z\sim P}[\ell_0(\theta;Z)],$$4 determines how close the oracle comes to exact nuisance insensitivity.

5. Canonical example: the partially linear model

The paper’s worked example is the non-orthogonal partially linear model with loss

$$\min_{\theta\in\Theta} L_0(\theta),\qquad L_0(\theta):=\mathbb{E}_{Z\sim P}[\ell_0(\theta;Z)],$$5

where $$\min_{\theta\in\Theta} L_0(\theta),\qquad L_0(\theta):=\mathbb{E}_{Z\sim P}[\ell_0(\theta;Z)],$$6. Its ordinary score is

$$\min_{\theta\in\Theta} L_0(\theta),\qquad L_0(\theta):=\mathbb{E}_{Z\sim P}[\ell_0(\theta;Z)],$$7

In this model, the orthogonalizing operator is

$$\min_{\theta\in\Theta} L_0(\theta),\qquad L_0(\theta):=\mathbb{E}_{Z\sim P}[\ell_0(\theta;Z)],$$8

and the ideal Neyman-orthogonalized gradient oracle becomes

$$\min_{\theta\in\Theta} L_0(\theta),\qquad L_0(\theta):=\mathbb{E}_{Z\sim P}[\ell_0(\theta;Z)],$$9

This is exactly a residualization correction: the feature $$L(\theta,g)=\mathbb{E}_{Z\sim P}[\ell(\theta,g;Z)],$$0 is replaced by the residual $$L(\theta,g)=\mathbb{E}_{Z\sim P}[\ell(\theta,g;Z)],$$1 (Yu et al., 28 Aug 2025).

An approximate oracle substitutes an estimator $$L(\theta,g)=\mathbb{E}_{Z\sim P}[\ell(\theta,g;Z)],$$2, equivalently an estimated $$L(\theta,g)=\mathbb{E}_{Z\sim P}[\ell(\theta,g;Z)],$$3, for the true regression. The example clarifies the general mechanism. The correction removes the nuisance-predictable component of the ordinary gradient rather than constructing a new nuisance estimator. It also shows why the oracle can be easier to implement than deriving an analytically orthogonalized loss for the full problem.

6. Assumptions, limitations, and frequent misunderstandings

The construction requires sufficient regularity to define $$L(\theta,g)=\mathbb{E}_{Z\sim P}[\ell(\theta,g;Z)],$$4, $$L(\theta,g)=\mathbb{E}_{Z\sim P}[\ell(\theta,g;Z)],$$5, and $$L(\theta,g)=\mathbb{E}_{Z\sim P}[\ell(\theta,g;Z)],$$6, and ideally to invert $$L(\theta,g)=\mathbb{E}_{Z\sim P}[\ell(\theta,g;Z)],$$7. The theorem for the approximate method additionally requires that the orthogonalized drift remain strongly monotone after replacing $$L(\theta,g)=\mathbb{E}_{Z\sim P}[\ell(\theta,g;Z)],$$8 by $$L(\theta,g)=\mathbb{E}_{Z\sim P}[\ell(\theta,g;Z)],$$9. If $\theta\in\Theta\subseteq\mathbb{R}^d$0 is too inaccurate, the corrected field may lose contraction and the theorem no longer applies (Yu et al., 28 Aug 2025).

The method also does not remove nuisance dependence unconditionally. If $\theta\in\Theta\subseteq\mathbb{R}^d$1 converges slowly, the quartic term may still be large. If orthogonality holds only weakly or the second-order expansion is poor, first-order nuisance effects can reappear. Estimating $\theta\in\Theta\subseteq\mathbb{R}^d$2 may itself be nontrivial in high-dimensional or infinite-dimensional nuisance classes. The paper states that the main theorem is in expectation and does not present a separate high-probability theorem for orthogonalized SGD in its main result.

A recurrent misunderstanding is to treat approximate orthogonalization as a modification of the nuisance estimate. The paper rejects that interpretation. The nuisance estimate remains plug-in; what changes is the stochastic update direction. A second misunderstanding is to identify the method with a fully orthogonalized loss. The paper instead emphasizes an oracle-level correction, which can be easier to deploy because it only requires the $\theta\in\Theta\subseteq\mathbb{R}^d$3-gradient, a nuisance derivative or nuisance gradient, and an estimate of the linear operator $\theta\in\Theta\subseteq\mathbb{R}^d$4.

7. Relation to adjacent orthogonalization paradigms

Several other papers study orthogonalization of first-order information, but they do not define the same object. "Statistically Optimal First Order Algorithms: A Proof via Orthogonalization" (Montanari et al., 2022) does not literally define an approximately orthogonalized gradient oracle; its central orthogonalized object is an orthogonal AMP transformation in which, after Onsager correction and an $\theta\in\Theta\subseteq\mathbb{R}^d$5 Gram–Schmidt step, effective first-order directions behave as asymptotically orthogonal Gaussian observations. This is a theory of orthogonalized first-order information streams rather than nuisance-robust plug-in SGD.

"Orthogonalising gradients to speed up neural network optimisation" (Tuddenham et al., 2022) orthogonalizes per-layer component gradients by projecting each layer’s gradient matrix onto its nearest orthonormal factor via SVD. There, exact orthogonality is imposed only within the chosen layerwise blocks, so the overall procedure is best viewed as a practical approximate orthogonalized-gradient transform for neural-network training rather than a Neyman-orthogonal nuisance correction.

"Orthogonalized SGD and Nested Architectures for Anytime Neural Networks" (Wan et al., 2020) performs sequential projection of lower-priority task gradients against higher-priority task gradients in multitask training. The orthogonalization is exact only relative to the currently observed minibatch vectors; with respect to true task subspaces or expected gradients it is explicitly approximate, order-dependent, and local.

"Faster Adaptive Optimization via Expected Gradient Outer Product Reparameterization" (DePavia et al., 3 Feb 2025) constructs an orthonormal change of basis from the expected gradient outer product, estimated from full-batch or stochastic gradient samples. The transformed gradient oracle has approximately diagonal expected outer product, so its coordinates are decorrelated in expectation, but the paper does not whiten gradients and does not address nuisance parameters.

These neighboring lines of work suggest a broader taxonomy of orthogonalized first-order methods: nuisance-robust oracle correction, AMP-style decorrelation of information streams, layerwise orthogonalization of component updates, priority-based multitask gradient projection, and EGOP-based reparameterization. A plausible implication is that “approximately orthogonalized gradient oracle” is best treated as a family resemblance term across multiple literatures, whereas its strict technical meaning is the oracle

$\theta\in\Theta\subseteq\mathbb{R}^d$6

introduced for stochastic optimization under nuisance parameters (Yu et al., 28 Aug 2025).