Neyman-Orthogonal Scores

• Neyman-orthogonal scores are estimating equations that nullify first-order bias from nuisance parameter errors, ensuring debiased estimation of target parameters.
• They are constructed via tangent-space projection and underpin double machine learning methods, yielding efficient and robust estimators in complex models.
• Applications include causal inference, average treatment effect estimation, and ultrahigh-dimensional regression, all benefiting from reduced plug-in bias and improved inferential performance.

A Neyman-orthogonal score is an estimating equation whose mean is first-order insensitive to perturbations in the nuisance parameter. This property underlies the construction of debiased and double machine learning estimators for high-dimensional and semiparametric models, enabling reliable inference for target parameters when the nuisance component is estimated with machine learning or flexible nonparametric methods. Neyman orthogonality ensures plug-in bias only enters at second order, which is essential for the robustness and efficiency of modern inference strategies across statistics, econometrics, and causal inference.

1. Formal Definition and Motivating Principles

Given observations $W\sim P$, let $\theta:\mathcal{P}\to\mathbb{R}$ be the finite-dimensional target and $\eta:\mathcal{P}\to\mathcal{H}$ a (possibly infinite-dimensional) nuisance parameter. Suppose an estimating function $$\psi(W;\theta, \eta):\mathcal{W}\times\mathbb{R}\times\mathcal{H}\to\mathbb{R}$$ has the property that its expectation is zero at the true values $(\theta_0, \eta_0)$,

$\mathbb{E}_{P_0}[\psi(W;\theta_0, \eta_0)] = 0.$

The score $\psi$ is Neyman-orthogonal at $(\theta_0, \eta_0)$ if the Gateaux derivative in any admissible nuisance direction $h$ vanishes:

$$\partial_\eta \mathbb{E}_{P_0}\left[\psi(W; \theta_0, \eta)\right]\Big|_{\eta=\eta_0}[h] = 0 \ \forall h\in \mathcal{H}.$$

Small first-order errors in the estimation of $\eta_0$ do not affect the leading bias in the score for $\theta$; this underpins the double robustness and bias-reduction of the resulting estimator (Chen et al., 16 Mar 2026, Chernozhukov et al., 2017, Foster et al., 2019).

The Neyman-orthogonality condition also appears as

$$\frac{d}{dr} \mathbb{E}_{P_0} \left[\psi\left(W; \theta_0, \eta_0 + r(h - \eta_0)\right)\right]\biggr|_{r=0} = 0$$

for each $h$. This formulation applies generally across moment equations, M-estimation, likelihood, and GMM settings (Chen et al., 16 Mar 2026, Nekipelov et al., 2018).

2. Equivalence to Pathwise Differentiability

A central development is the equivalence between Neyman orthogonality and pathwise differentiability within a local product structure framework. Pathwise differentiability is a geometric property: for any regular (quadratic mean differentiable) parametric submodel $\{P_t\}$ with score $s$, the target admits an influence function $\varphi$ such that

$$\frac{d}{dt}\theta(P_t)\biggr|_{t=0} = \mathbb{E}_{P_0}[\varphi(W) s(W)],$$

with $\varphi$ in the orthogonal complement of the nuisance tangent space.

Under a local product structure assumption—where the target and nuisance can be perturbed independently via submodels—Neyman-orthogonal estimating equations coincide with influence function-based scores for pathwise differentiable targets. This provides a rigorous foundation for the widely observed coincidence of efficient influence functions and orthogonalized scores in debiased ML procedures (Chen et al., 16 Mar 2026).

The equivalence theorem states:

• Neyman-orthogonality (plus correct specification and nondegeneracy) implies pathwise differentiability, with an explicit influence function.
• Conversely, if the target is pathwise differentiable, local product structure ensures the influence equation corresponds to a Neyman-orthogonal estimating function.

This equivalence is fundamental for the design, justification, and theoretical guarantees of semiparametric and double/debiased machine learning estimators (Chen et al., 16 Mar 2026).

3. Construction and Properties of Neyman-Orthogonal Scores

The systematic construction of orthogonal scores proceeds by projecting naive scores onto the orthocomplement of the nuisance tangent space. Specifically, given a working moment function $\psi(W;\theta,\eta)$ and nuisance score $\ell_h(W;\eta)$, the orthogonalized score is

$$S(W;\theta,\eta) = \psi(W;\theta,\eta) - \mathbb{E}_{P_0}[\psi(W;\theta,\eta)\ell_h(W;\eta)^\top] \cdot \mathbb{E}_{P_0}[\ell_h(W;\eta)\ell_h(W;\eta)^\top]^{-1} \ell_h(W;\eta)$$

(Sabbagh et al., 23 Feb 2026). This generic procedure yields a score whose expectation is first-order insensitive to perturbations in the nuisance.

In practical settings, bias-corrected or doubly robust scores are constructed to satisfy this property. For instance, in missing data, IPW, and causal inference, scores typically combine regression and inverse-propensity components to achieve orthogonality (Chernozhukov et al., 2017, Kato, 27 Oct 2025). In high-dimensional or high-noise settings, this construction is pivotal for valid inference and achieving oracle rates (Nekipelov et al., 2018, Kato, 27 Oct 2025, Yang et al., 2022).

Summary Table: Key Structural Aspects

Aspect	Description	Reference
Insensitivity	1st-order bias in $\eta$ has zero effect on $\theta$	(Chen et al., 16 Mar 2026)
Pathwise differentiable	Influence function matches orthogonal score	(Chen et al., 16 Mar 2026)
Efficient estimation	Yields minimal-variance unbiased estimator	(Chen et al., 16 Mar 2026)
General construction	Via tangent-space projection	(Sabbagh et al., 23 Feb 2026)

In all cases, Neyman-orthogonality safeguards the estimator’s leading bias against nuisance estimation errors.

4. Higher-Order Orthogonality and Robustness

When nuisance parameters are estimated at slow rates—e.g., fixed effects with limited panel length—first-order orthogonality may be insufficient. Generalizing, a score is said to be $q$-th order Neyman-orthogonal if all moments of derivatives up to order $q$ vanish:

$$\mathbb{E}[\nabla_\eta^{(p)} \psi(Z; \theta_0, \eta_0)] = 0, \qquad \text{for } p = 1, \dots, q,$$

so that Taylor expansions to order $q$ kill all lower-order bias terms, and only $o(\|\hat\eta-\eta_0\|^{q+1})$ remains (Bonhomme et al., 2024, Mackey et al., 2017). The required convergence rate for $\hat\eta$ is thus relaxed to $o(n^{-1/[2(q+1)]})$ for root-$n$ consistency.

Notably, higher-order orthogonalization is crucial in settings with pronounced incidental parameter bias or weak identification. Explicit constructive procedures for such scores have been developed using projections in conditional likelihood models, as well as generalized moment conditions in the partially linear and fixed-effect models (Bonhomme et al., 2024, Mackey et al., 2017).

5. Canonical Examples and Applications

Average Treatment Effect (ATE)

In causal inference, the ATE is identified by

$$\theta(P) = \mathbb{E}_P[\mu_1(X) - \mu_0(X)], \quad \mu_a(x) = \mathbb{E}_P[Y | X=x, A=a].$$

The doubly robust (Augmented-IPW) score

$$\psi(W;\theta,\eta) = \frac{A}{\pi(X)}\{Y-\mu_1(X)\} - \frac{1-A}{1-\pi(X)}\{Y-\mu_0(X)\} + \mu_1(X) - \mu_0(X) - \theta$$

is correctly specified, Neyman-orthogonal, and corresponds to the efficient influence function under regularity conditions (Chen et al., 16 Mar 2026, Chernozhukov et al., 2017, Kato, 27 Oct 2025).

In two-stage procedures, estimation proceeds by:

1. Estimating $\mu_a$, $\pi$ using machine learning or flexible nonparametrics.
2. Plugging these into $\psi$ and solving $\mathbb{E}_n[\psi] = 0$.

The resulting estimator is $\sqrt{n}$-consistent as long as nuisance estimators achieve $o(n^{-1/4})$ norm rates—directly due to Neyman-orthogonality (Chernozhukov et al., 2017, Kato, 27 Oct 2025, Foster et al., 2019).

Ultrahigh-Dimensional Linear Regression

Score-based testing in the model $Y = X^\top\beta + Z^\top\gamma + \varepsilon$ with high-dimensional $\gamma$ can be invalid due to large first-stage bias if $X$ is correlated with $Z$. The orthogonally debiased score

$$S_\mathrm{orth}(\beta,\gamma) = -\mathbb{E}[ (X - W^\top Z)(Y - X^\top\beta - Z^\top\gamma) ]$$

with $W = \mathbb{E}[ZZ^\top]^{-1}\mathbb{E}[ZX^\top]$, is Neyman-orthogonal and reduces both bias and variance in resulting test statistics (Yang et al., 2022).

Signal Recovery, Heterogeneous Effects, and Beyond

Neyman-orthogonal scores have been used to construct DR- and R-learners for CATE estimation, weighted orthogonal estimators, and for robust inference in NPIV, panel data, and complex semiparametric learning. These frameworks all systematically leverage orthogonality for bias-robustness and double robustness (Melnychuk et al., 6 Feb 2025, Morzywolek et al., 2023, Foster et al., 2019, Kato, 27 Oct 2025, Nekipelov et al., 2018).

6. Implications for Statistical Learning and Machine Learning

In double/debiased machine learning (DML) meta-algorithms, Neyman-orthogonality is the keystone for second-order bias control. Excess risk decompositions show that, under orthogonality, the cross-term between target and nuisance errors vanishes, and only a second-order remainder remains:

$$\| \hat{\theta} - \theta^* \|^2 \lesssim \mathrm{(oracle\ risk)} + O( \|\hat{g} - g_0 \|^2 ),$$

so the estimator achieves the same statistical rate as if the nuisance were known, under mild complexity constraints on the target and nuisance classes (Foster et al., 2019, Melnychuk et al., 6 Feb 2025, Morzywolek et al., 2023). This principle guarantees quasi-oracle excess risk bounds in a wide array of high-dimensional and nonparametric models.

Sample splitting and cross-fitting can be used to handle overfitting and dependence between stages, exploiting orthogonality to avoid empirical process complications (Chernozhukov et al., 2017). Higher-order orthogonality further relaxes the required rates of nuisance estimation, a crucial feature in models with slow nuisance convergence, such as short panels or network data (Bonhomme et al., 2024, Mackey et al., 2017).

7. Bayesian Inference, Higher-Order Extensions, and Limitations

In partially (semi-)parametric Bayesian models, Neyman-orthogonality validates the use of "cut" or plug-in Bayesian procedures, where the marginal posterior for $\theta$ is centered at the two-step estimator and obtains frequentist coverage under conditions analogous to the frequentist DML setting. Orthogonality ensures that the leading effect of nuisance estimation is nullified, so that even Bayesian bootstrap or Dirichlet-process plug-in approaches are asymptotically valid for credible intervals (Sabbagh et al., 23 Feb 2026).

The extension to higher-order orthogonality further reduces plug-in bias and ensures valid inference when nuisance estimates are much less precise (Bonhomme et al., 2024, Mackey et al., 2017). However, certain limitations remain: e.g., in the partially linear regression model, the existence of nontrivial higher-order orthogonal moments may be precluded by Gaussianity of the treatment residual, establishing a boundary for further bias correction (Mackey et al., 2017).

In summary, Neyman-orthogonal scores provide the analytic, geometric, and algorithmic backbone for robust inference under high-dimensional, semiparametric, and modern ML-aided statistical models, with applications extending from causal inference to model selection and debiasing in contemporary data science (Chen et al., 16 Mar 2026, Chernozhukov et al., 2017, Kato, 27 Oct 2025, Foster et al., 2019, Nekipelov et al., 2018, Melnychuk et al., 6 Feb 2025, Bonhomme et al., 2024, Yang et al., 2022, Morzywolek et al., 2023, Sabbagh et al., 23 Feb 2026, Mackey et al., 2017).