Increasing Missingness to Reduce Bias: Richardson-SGD with Missing Data

Abstract: Stochastic gradient methods are central to modern large-scale learning, but their use with incomplete covariates remains delicate since imputation schemes generally introduce systematic gradient biases, as shown for linear models. In this work, we prove that all parametric models exhibit similar gradient bias for various imputation procedures and characterize exactly the dependence on the missingness ratio vector $p$, with $O(|p|)$ as the leading term. We exploit this analysis to propose a simple debiasing procedure for stochastic gradient descent (SGD) with missing values based on Richardson extrapolation, which leverages the exact expression of the gradient bias. The key idea is to \emph{deliberately add missingness}: from an already incomplete observation, we generate a further-thinned version at a higher, controlled missingness level, and combine the two resulting stochastic gradients to cancel the leading bias term. We prove that one Richardson step reduces the gradient bias from $O(|p|)$ to $O(|p|^2)$ under several missingness scenarios. Our proposed method is computationally efficient, model-agnostic and applies to any parametric loss whose stochastic gradient can be computed after imputation. Furthermore, when missing indicators are independent, the population gradient bias is a multilinear polynomial in $p$ and depends only on population gradient errors induced by declaring a single coordinate missing. In this case, our method generalizes to a multi-step Richardson procedure which recursively cancels higher-order terms. Empirically, Richardson debiasing improves optimization and estimation across several generalized linear models and combines positively with widely used imputation procedures such as MICE. These results suggest that, somewhat counter-intuitively, adding controlled missingness on top of existing missing data can make stochastic learning from incomplete data more accurate.

• The paper introduces Richardson-SGD, which cancels first-order gradient bias by synthesizing estimators from samples with different missingness scales.
• The method achieves bias reduction from O(‖p‖) to O(‖p‖²) and guarantees improved convergence rates compared to plain imputation-based SGD.
• Empirical validation across various GLMs and imputation strategies shows Richardson-SGD consistently enhances parameter accuracy and predictive performance.

Debiasing SGD for Missing Data via Controlled Missingness: Analysis of Richardson-SGD

Overview and Motivation

This paper addresses the central challenge in parametric supervised learning with missing covariates: the bias introduced by imputation-based stochastic gradients. It demonstrates that imputation-induced biases in the stochastic gradients are not only systematic but, under broad conditions, have a fixed first-order structure in the missingness vector. The key insight is that this bias can be cancelled by deliberate "controlled missingness" via Richardson extrapolation, which exploits the functional dependence of bias on the missingness ratio.

First-Order Structure of Gradient Bias

The authors prove under heterogeneous MCAR and scalable MAR mechanisms that the population gradient bias for any data-independent imputation scheme admits an expansion

$B(w, p) = A(w)\,p + R(w, p)$

where $A(w)$ is a matrix of population gradient gaps and $p$ is the vector of marginal missingness probabilities. The leading bias is thus linear in $p$ under conditional independence of the mask components, with the remainder $R(w, p)$ being $O(\|p\|^2)$. This decomposition is verified across general parametric losses and generic imputation schemes, including MICE, mean, zero, and $k$-NN imputations.

Figure 1: Multi-order Richardson correction in 4-covariate linear and logistic regression under hMCAR. Each Richardson step removes an additional order of the gradient bias.

This structural insight immediately suggests that imputation alone cannot remove the leading-order bias unless the imputer achieves perfect recovery, which is typically impossible in high-dimensional or genuinely missing-data regimes.

Richardson-SGD: Algorithmic Construction

The paper proposes Richardson-SGD, which cancels the leading $O(\|p\|)$ gradient bias by synthesizing an estimator evaluated at two missingness scales (original and further thinned). The procedure is as follows:

1. Thinning: For each sample with observed mask at scale $p$, generate a further-thinned mask at scale $Cp$ by additional Bernoulli draws.
2. One Imputation: Impute only once at the higher scale $A(w)$0, then restore artificially hidden entries to construct the imputed input corresponding to the original mask.
3. Richardson Combination: Form the linear combination

$A(w)$1

to eliminate the linear bias.

For independent masks, iterating this procedure—i.e., higher-order Richardson steps—successively cancels higher terms of the bias polynomial. In linear regression, where the bias is quadratic in $A(w)$2, two Richardson steps are sufficient for exact debiasing.

Theoretical Guarantees

The main theoretical results include:

• First-Order Cancellation: Under independent hMCAR or sMAR, Richardson-SGD reduces gradient bias from $A(w)$3 to $A(w)$4.
• Higher-Order Cancellation: In settings with a limited number of missing features, the multi-step Richardson procedure gives exact debiasing in at most $A(w)$5 steps.
• SGD Convergence: For one-pass SGD with strongly convex losses:
  • Plain Imputed SGD: $A(w)$6
  • Richardson-SGD: $A(w)$7

Empirical Validation

The paper validates Richardson-SGD on synthetic and real-world datasets using a variety of GLMs (linear, logistic, Poisson) and imputation strategies (zero, mean, $A(w)$8-NN, MICE, MICE-RF). Across all settings, Richardson-SGD systematically improves parameter estimation and predictive accuracy compared to the corresponding plain imputation-based SGD.

Figure 2: Test loss and parameter MSE for logistic regression on Covertype as average missingness $A(w)$9 varies. Richardson correction consistently improves over base imputers across missingness levels.

Figure 3: Final parameter MSE comparison (logistic regression, hMCAR) across datasets. Richardson correction dominates (lower is better), with pronounced gains as imputation quality improves.

The methodology proves robust to estimation errors in the missingness mechanism parameters and performs competitively even under misspecification (e.g., using hMCAR approximation when the true mechanism is sMAR).

Practical and Theoretical Implications

This work has several noteworthy implications:

• Model- and Imputation-Agnostic: Richardson-SGD does not require parametric assumptions on the data distribution, nor analytic knowledge of the loss surface; it only requires access to an imputation pipeline and the missingness ratios.
• Lightweight and Modular: The method wraps around standard SGD with only one extra draw and gradient computation per sample.
• Variance-Bias Tradeoff: The variance inflation from Richardson corrections is quantified and remains moderate for first-order correction. Higher-order correction is computationally feasible and statistically justified when few features are partially observed.

Notably, the counterintuitive result that increasing missingness in a controlled manner can improve SGD-based estimation reflects a fundamental aspect of systematic bias: leveraging structural dependence on a nuisance parameter (missingness) enables its cancellation via extrapolation.

Limitations and Future Directions

A primary limitation is the requirement for conditional independence of missingness across features to realize higher-order cancellations. For correlated mask structures, constructing synthetic further-thinned masks that match the true conditional distribution becomes infeasible. Extending the methodology to arbitrarily dependent missingness mechanisms or to multi-epoch training regimes remains open.

Conclusion

The analysis and empirical evidence provided in this paper demonstrate that Richardson-SGD is an effective, theoretically grounded mechanism for debiasing gradient estimation in the presence of missing covariates. Its modularity ensures compatibility with modern imputation and learning pipelines, and its principled design yields significant gains in parameter accuracy. As high-dimensional, incomplete datasets proliferate, controlled-extrapolation techniques such as Richardson-SGD provide a robust, easy-to-adopt addition to the statistical learning toolbox.