Neyman Orthogonality: Robust Estimation

• Neyman orthogonality is defined as the property where small changes in nuisance parameters have no first-order effect on the estimated target parameter.
• It underpins robust, √n-consistent inference even when nuisance estimates converge at slower high-dimensional rates, typical in doubly robust models.
• Constructing orthogonal scores by adjusting naive moment equations reduces linear bias from nuisance estimation, thereby enhancing inference efficiency.

Neyman orthogonality is a structural property of statistical functionals or estimating equations that guarantees first-order insensitivity of an estimator to small perturbations in nuisance parameters. This property lies at the core of modern semiparametric inference and orthogonal machine learning, enabling robust inference about parameters of interest even in the presence of high-dimensional or complex nuisance components. Neyman orthogonality ensures that the impact of nuisance estimation errors on the target parameter enters only at second order, resulting in estimators that can achieve $\sqrt{n}$-consistency and valid inference under much weaker regularity and convergence conditions than would be possible otherwise.

1. Formal Definition and Characterization

At the heart of Neyman orthogonality is the moment or score equation identifying a finite-dimensional parameter $\theta_0$ in the presence of a (possibly infinite-dimensional) nuisance parameter $\eta_0$:

$\mathbb{E}\big[\psi(W; \theta_0, \eta_0)\big] = 0,$

where $W$ represents the observed data, $\psi$ is a real-valued influence function (score), and $\eta_0$ encodes the true value of the nuisance components. Neyman orthogonality holds if the Gateaux (directional) derivative of the expected score with respect to $\eta$ vanishes at $\eta_0$:

$$\partial_\eta \, \mathbb{E}[\psi(W; \theta_0, \eta)]|_{\eta_0}[h - \eta_0] = 0 \qquad \forall\, h,$$

or, equivalently,

$$\partial_\eta\,\mathbb{E}[\psi(W; \theta_0, \eta)]|_{\eta = \eta_0} = 0.$$

This requirement means that to first order, small perturbations of the nuisance parameter $\eta$ leave the moment equation unchanged. For differentiable loss-based risks, this can be equivalently phrased as the vanishing of the mixed (cross) derivatives $D_g D_\theta L(\theta^*, g_0)$ for the population risk $L(\theta, g)$ at its oracle solution $(\theta^*, g_0)$ (Foster et al., 2019).

2. Motivation and Impact on Estimation Robustness

The conceptual significance of Neyman orthogonality is its role in eliminating the leading-order (linear) bias resulting from estimating $\eta_0$ with an estimated $\hat{\eta}$. A Taylor or von Mises expansion of the sample moment equation shows that, so long as the orthogonality condition holds, the bias term

$$\partial_\eta \, \mathbb{E}\big[\psi(W; \theta_0, \eta)\big]|_{\eta_0} [\hat{\eta} - \eta_0]$$

disappears, leaving only higher-order ($O(\|\hat{\eta} - \eta_0\|^2)$) contributions. This allows for $\sqrt{n}$-consistent inference on $\theta_0$ even when the nuisance estimates converge at slower, high-dimensional rates such as $o(n^{-1/4})$, rather than the much faster $o(n^{-1/2})$ rate required in the non-orthogonal case (Chernozhukov et al., 2017, Mackey et al., 2017).

3. Construction of Orthogonal Scores

The construction of orthogonal scores typically begins with an unbiased but non-orthogonal estimating equation, which is then augmented by correction terms to ensure the cross-derivative vanishes. A general approach involves:

• Identifying the naive (possibly non-orthogonal) moment equation.
• Isolating the influence of nuisance estimation errors on the moment.
• Augmenting the score by adding terms (typically involving residuals and projections) that exactly cancel the leading linear effect of these errors.

For instance, in the estimation of average treatment effects (ATE) in potential outcomes models, standard procedures augment outcome regression estimators with inverse-propensity score terms, yielding doubly robust, Neyman-orthogonal scores (Chernozhukov et al., 2017). Similar constructions are used in partially linear models and various settings involving high-dimensional regression (Sabbagh et al., 23 Feb 2026).

4. Statistical Learning and Sample-Splitting Meta-Algorithms

Neyman orthogonality underpins a broad class of two-stage or meta-algorithms in statistical learning:

• Stage 1: Estimate the nuisance parameter $\eta_0$ using an arbitrary machine learning algorithm on a subset of the data.
• Stage 2: Plug in $\hat{\eta}$ and estimate $\theta$ by solving the empirical estimating equation, or by minimizing an empirical risk with $\hat{\eta}$ held fixed, on another subset (often via cross-fitting).

In these designs, Neyman orthogonality ensures that the impact of the nuisance estimation enters only at second order, yielding bounds of the form:

$$\|\hat{\theta} - \theta^*\|^2 \leq O(\text{target rate}) + O(\text{nuisance rate}^4),$$

and allows for non-asymptotic excess risk guarantees, provided the nuisance component is sufficiently less complex than the target (Foster et al., 2019). This phenomenon is a core rationale for sample splitting and cross-fitting in high-dimensional inference.

5. Higher-Order Orthogonality

The theory of Neyman orthogonality extends to $k$-th order orthogonality, in which not only the first but also all cross-derivatives of order up to $k$ vanish in appropriate directions:

$$E[D^\alpha m(Z, \theta_0, h_0(X)) | X] = 0 \quad\text{for all } |\alpha| \leq k,$$

where $D^\alpha$ denotes mixed partial derivatives with respect to components of the nuisance. For $k$-orthogonal moments, the required convergence rate of the nuisance estimate weakens to $o(n^{-1/(2k+2)})$. In partially linear regression models, second-order ($k=2$) Neyman orthogonality can be constructed when the treatment residual is non-Gaussian, yielding estimators that are robust to even slower nuisance estimation rates (Mackey et al., 2017).

6. Bayesian Inference and Neyman Orthogonality

In semi-parametric Bayesian inference with high-dimensional or nonparametric nuisance components, Neyman orthogonality legitimizes "cutting feedback"—fixing a plug-in estimator $\hat{\eta}$ for the nuisance and focusing Bayesian updating solely on the target parameter. Under orthogonality, the marginal posterior of $\theta$—obtained by solving the weighted estimating equation $\sum w_i \psi(O_i; \theta, \hat{\eta}) = 0$ with bootstrap or Dirichlet weights—has the same $\sqrt{n}$-asymptotic coverage and variance as in oracle settings where $\eta$ is known (Sabbagh et al., 23 Feb 2026). This establishes a Bayesian-frequentist duality and provides strong justification for two-step plug-in posteriors in causal inference and other semi-parametric settings.

7. Limitations, Non-Orthogonality, and Remediation

When Neyman orthogonality fails ($$\partial_\eta \mathbb{E}[\psi(W;\theta_0,\eta)]|_{\eta_0} \neq 0$$), first-order bias persists unless the nuisance estimator is exceptionally accurate ($o(n^{-1/2})$ rates), which is typically impractical in complex or high-dimensional regimes. In such cases, the resulting estimator exhibits non-negligible bias, and inference is not robust. Remedies include explicit debiasing procedures, construction of higher-order orthogonal moments where feasible, or full joint modeling/updating of both $\theta$ and $\eta$ at the cost of computational complexity (Sabbagh et al., 23 Feb 2026). In specific models, impossibility results dictate when higher-order orthogonality can or cannot be achieved, such as in partially linear regression models with Gaussian treatment residuals, for which second-order orthogonality is provably impossible (Mackey et al., 2017).

---

Key Results Across Principal Methodological Domains

Domain	Primary Role of Neyman Orthogonality	Representative Paper
Double/debiased machine learning	Enables $\sqrt{n}$-valid inference with slow nuisance learning	(Chernozhukov et al., 2017)
Semi-parametric Bayesian inference	Validates two-step "cut feedback" posteriors	(Sabbagh et al., 23 Feb 2026)
Statistical learning excess risk	Second-order impact of nuisance on oracle rates	(Foster et al., 2019)
Higher-order orthogonal moments	Weakens nuisance rate requirements via $k$-orthogonality	(Mackey et al., 2017)

Neyman orthogonality thus constitutes a unifying structural tool, enabling valid inference in the presence of high-dimensional or adaptive nuisance estimation across frequentist and Bayesian, parametric and semiparametric domains. Its practical impact includes robust, computationally efficient inference strategies that exploit sample splitting, cross-fitting, and moment augmentation, as well as a rigorous framework for understanding the statistical cost of nuisance learning.