Hypothesis Testing for Penalized Estimating Equations with Cross-Fitted Covariance Calibration

Abstract: We study hypothesis testing for penalized estimators in settings where the full marginal distribution of a multivariate response is difficult to specify, such as longitudinal data with correlated measurements or high-dimensional heteroscedastic regression. Assuming that the conditional mean model is correctly specified, we establish that the penalized estimating equations admit a $\sqrt{n}$-consistent solution, even when the working covariance structure is misspecified. Our inferential target is a low-dimensional subvector of parameters associated with the mean model. We show that the resulting test statistic converges to a $χ^2$ distribution, and that its asymptotic power depends on the nuisance covariance function. To mitigate this dependence, we propose estimating the covariance function via cross-fitting, which provides a calibrated and robust procedure for inference.

• The paper introduces a framework that decouples mean and covariance estimation, enabling valid inference in high-dimensional models.
• It proposes a cross-fitting procedure for nonparametric covariance estimation, ensuring efficiency even when covariance is misspecified.
• The study establishes asymptotic normality and improved power for hypothesis testing despite complex, covariate-dependent covariance structures.

Hypothesis Testing for Penalized Estimating Equations with Cross-Fitted Covariance Calibration

Introduction

This paper addresses the problem of valid hypothesis testing for penalized estimators in high-dimensional conditional mean models, particularly settings with complex, possibly misspecified and covariate-dependent covariance structures. Unlike classical likelihood or quasi-likelihood-based inference, the work formalizes inference for sparse parameter vectors using penalized estimating equations (EE), and introduces a cross-fitted procedure for nonparametric covariance function estimation. This decouples mean and covariance specification, enabling valid and efficient inference even when the marginal distribution of a multivariate response is intractable, variance is heteroscedastic, or the covariance structure is not constant across covariates.

Penalized Estimating Equation Framework

The parameter of interest is an $s$-sparse vector $\beta_0 \in \mathbb{R}^p$ from a conditional mean model

$\mathbb{E}(Y_i| X_i) = g(X_i^\top \beta_0)$

where $g(\cdot)$ is a known (possibly nonlinear) link function. The dimension $p$ may exceed $n$. The primary inferential focus is a low-dimensional subvector $\beta_{0,M}$ associated with selected features. Importantly, there are minimal assumptions or structure imposed on the conditional covariance of $Y_i$, and $Y_i$ may be multivariate and not identically distributed over $i$.

Estimation is conducted by constructing partial penalized estimating equations:

$\beta_0 \in \mathbb{R}^p$0

where $\beta_0 \in \mathbb{R}^p$1 aggregates subject-level score functions weighted by the inverse covariance, and $\beta_0 \in \mathbb{R}^p$2 applies a nonconvex penalty (e.g., SCAD, MCP) to components outside $\beta_0 \in \mathbb{R}^p$3. Importantly, the mean specification can be correct under misspecified or heteroscedastic covariance, and the penalty structure enables high-dimensional model selection.

Covariance Misspecification and Consistency

A key theoretical result is that, under mild regularity (boundedness of inverse weights), consistent estimation at the oracle rate is achieved even when the working covariance $\beta_0 \in \mathbb{R}^p$4 is misspecified. That is, the estimator $\beta_0 \in \mathbb{R}^p$5 solving the penalized EE remains $\beta_0 \in \mathbb{R}^p$6-consistent for $\beta_0 \in \mathbb{R}^p$7. This detaches mean model inference from the difficulties of joint density specification and provides robustness akin to sandwich/heteroscedasticity-consistent literature but in high-dimensional, possibly non-Gaussian, and non-longitudinal settings.

Cross-Fitted Covariance Calibration

Although $\beta_0 \in \mathbb{R}^p$8-consistency holds under misspecification, the efficiency and asymptotic distribution of the estimator are determined by the unknown nuisance covariance function. To address this, the paper proposes a two-stage cross-fitting workflow inspired by ideas from (DML) [chernozhukov2018DML]. The process is as follows:

• The data are split into two folds. On each fold, an initial estimator is constructed (using a working covariance).
• Residuals from the initial fit are used to nonparametrically estimate the covariate-dependent covariance function using kernel regression and active set variable screening via sufficient dimension reduction techniques.
• The estimated covariance from one fold is used as the weight structure for solving penalized EEs on the other fold.
• The final estimator is averaged across both cross-fitted solutions.

This design eliminates dependency between the estimated covariance weights and the test statistic, thus mitigating the bias and non-orthogonality that would distort first-order asymptotics in the presence of nuisance estimation.

Active Set Selection for Covariance Structure

The paper further addresses the identification of the minimal subset of covariates (active set $\beta_0 \in \mathbb{R}^p$9) on which the covariance structure depends. A sufficient dimension reduction scheme is used, leveraging penalized empirical loss minimization with LASSO for high-dimensional decorrelation screening. The procedure provides adaptive, data-driven selection of the relevant dimensions for covariance modeling, ensuring the estimated covariance function remains parsimonious and interpretable. Theorems establish the consistency of both residual estimation and active set recovery under moderate signal strength and sparsity assumptions at high dimensions.

Asymptotic Theory and Power

The primary inferential result is that, under regularity and rate conditions on the sparsity and dimension ($\mathbb{E}(Y_i| X_i) = g(X_i^\top \beta_0)$0), the cross-fitted estimator is asymptotically normal:

$\mathbb{E}(Y_i| X_i) = g(X_i^\top \beta_0)$1

where $\mathbb{E}(Y_i| X_i) = g(X_i^\top \beta_0)$2 and $\mathbb{E}(Y_i| X_i) = g(X_i^\top \beta_0)$3 are the sandwich matrices incorporating the data-dependent covariance function. The corresponding Wald test statistic for general linear hypotheses has an asymptotic $\mathbb{E}(Y_i| X_i) = g(X_i^\top \beta_0)$4 distribution under the null and a non-central $\mathbb{E}(Y_i| X_i) = g(X_i^\top \beta_0)$5 distribution under local alternatives.

A formal power comparison theorem shows that the Wald test based on the cross-fitted estimator with estimated covariance always has (asymptotically) greater or equal noncentrality parameter and power relative to any test based on an arbitrary working covariance, i.e., exploiting covariance structure estimation is always efficiency-improving.

Implications and Future Directions

This framework effectively generalizes robust and efficient inference in high-dimensional models to cases where neither the likelihood nor quasi-likelihood formulation is available and the covariance structure is complex, unknown, and may vary nonlinearly with covariates. The cross-fitting ensures the removal of first-order bias from nuisance estimation, which is a critical technical advance for high-dimensional semiparametric inference.

Practically, the methods are applicable to high-dimensional, heterogeneous, or multivariate data scenarios common in econometrics, biology, or longitudinal studies, especially when heteroscedasticity or complex dependence is present and the primary target is low-dimensional inference or testing after feature selection. The results enable rigorous uncertainty quantification and valid $\mathbb{E}(Y_i| X_i) = g(X_i^\top \beta_0)$6-values without needing the correct likelihood or parametric covariance model.

On the theoretical side, the analysis raises the potential for further developments in double/debiased machine learning for general estimating equation contexts, extensions to dependent data and time series, or adaptation to more aggressive sparsity regimes. Moreover, data-driven nonparametric covariance modeling combined with penalized semiparametric scores points toward general frameworks for inference in "model-free" high-dimensional environments.

Conclusion

This work formalizes efficient and valid hypothesis testing for sparse parameters in high-dimensional models using penalized estimating equations with cross-fitted, nonparametric covariance calibration. The cross-fitting delivers robustness to covariance misspecification and optimal power, while a novel active set selection procedure for the covariance function ensures consistent nuisance estimation. The result is a flexible, theoretically rigorous inferential framework for high-dimensional mean models under general covariance heterogeneity.