Model Assisted Data Integration: An unbiased sampling strategy to use nonprobability data

Abstract: The aim of survey statistics is to produce estimates with a minimal bias and a corresponding acceptable variance given a specific budget, preferable with a minor response burden for the participants. In recent years, considerable efforts have been made to achieve this through the extended use of found or non-probability data. However, to be able to safely utilize such data, rigorous theoretical foundations is needed, where one main concern is the of lack control due to not having access to the selection mechanism for the data. Several methods have been proposed in the literature to deal with this, though often relying on assumptions that may be difficult or impossible to verify in practice. Extending on the Data Integrated (DI) estimator introduced by Kim and Tam (2021), this paper introduce the Model Assisted Data Integration (MADI) sampling strategy. The proposed sampling strategy includes an estimator that has the desired properties: it is design-unbiased, has a design-unbiased variance estimator and is suitable for the intense production cycle of the statistical agency. The estimator uses nonprobability data combined with a probability sample that has a sampling design which aims to include individuals not captured by the nonprobability data. The estimator can use arbitrary machine learning models to produce unbiased estimates. A main conclusion of the paper is that the proposed sampling strategy can produce estimates with much lower variances compared to traditional survey estimators, and we use real empirical data to illustrate this point.

• The paper presents MADI, a model-assisted estimator that integrates nonprobability data using machine learning and auxiliary information to ensure design unbiasedness.
• It leverages a controlled probability sample for error correction, significantly reducing variance and sample size requirements compared to traditional methods.
• Empirical evaluations show robust performance under selection bias and missing data conditions, highlighting its scalability for official statistics.

Model Assisted Data Integration (MADI): An Unbiased Approach for Leveraging Nonprobability Data

Introduction and Motivation

The integration of nonprobability data (NPD) into official statistics holds the promise of reducing costs, enhancing efficiency, and lowering response burdens. However, a major barrier is the lack of control over the underlying selection mechanism, leading to potential selection bias and challenging the traditional design-based inference paradigm. The literature proposes various estimators for NPD integration—imputation, propensity adjustments, calibration, and doubly-robust estimators—yet these typically rely on assumptions (e.g., missing at random) that are rarely testable in operational settings. The Data Integration (DI) estimator introduced by Kim and Tam (2021) relaxed some of these assumptions, but limitations in efficiency and adaptability remain.

This paper introduces the Model Assisted Data Integration (MADI) sampling strategy, extending DI by exploiting arbitrary machine learning models and population-level auxiliary information to yield design-unbiased point and variance estimators. The MADI approach is tailored to the operational constraints of National Statistical Offices (NSOs) and is particularly adept at efficiently utilizing available NPD, even with strong selection bias and small accompanying probability samples.

Traditional and Data Integration Estimators: Limitations

The Horvitz-Thompson (HT) estimator and the Generalized Regression (GREG) estimator form the backbone of traditional survey statistics. While the HT estimator is design-unbiased, it exhibits excessive variance, especially for small samples. The GREG estimator is design-consistent but not unbiased and suffers when sample size is small relative to the number of auxiliary variables—a common scenario with data integration. In such cases, GREG may yield unstable or undefined weights due to matrix singularity, further limiting its practicality.

The DI estimator by Kim and Tam leverages a partition of the population into a nonprobability sample (A) and its complement (B), using a probability sample from B and the observed study variable values in A. The estimator is design-unbiased if the probability sample is only drawn from B, but the structure is rigid and does not directly leverage flexible modeling approaches or large sets of auxiliary information.

The MADI Sampling Strategy and Estimator

The MADI strategy systematically unifies traditional survey design concepts with advances in model-assisted estimation:

• Data Partitioning: The population $U$ is divided into $A$ (units with observed $y_i$ from NPD) and $B = U \setminus A$.
• Auxiliary Information: Complete auxiliary vectors $x_i$ are assumed available for all units in $U$.
• Model Estimation: A machine learning model $u(x, A)$ is trained on $A$ to predict $y_i$ using $x_i$.
• Sample Design: A probability sample is drawn only from $A$0, not $A$1, with strict control on inclusion probabilities, minimizing sample size by focusing resources on undercovered subdomains.

The MADI estimator for the population total of $A$2 is:

$A$3

where $A$4 denotes the probability sample from $A$5 and $A$6 the respective inclusion probabilities.

Key properties:

• Design Unbiasedness: The estimator corrects for prediction error in the probability sample, ensuring design unbiasedness “by construction.”
• Design-unbiased Variance Estimator: An explicit formula is derived and empirically validated, supporting robust variance estimation regardless of the complexity of $A$7.
• Model Flexibility: Any predictive model (e.g., random forest, OLS, neural net) can be employed; overfitting does not compromise design validity due to the error-correction mechanism.
• Variance Reduction: Use of a prediction function trained on a typically large NPD markedly reduces estimator variance, particularly when probability sample sizes are restricted.

Empirical Evaluation and Numerical Results

Simulations on Swedish Income and Taxation register data (N ≈ 10,000) with a subset (NPD) proportion of 70% illustrate MADI’s practical advantages. Various estimator configurations—including HT, GREG, naïve random forests, DI with HT or GREG, and MADI with OLS and random forest (RF) base models—are compared for bias, root mean squared error (RMSE), and coverage.

Numerical findings:

• Variance Reduction and Efficiency: SRSB-MADIRF demonstrates the lowest RMSE across all sample sizes (from $A$8 to $A$9), vastly outperforming comparators. For example, at $y_i$0, SRSB-MADIRF achieves RMSE of 5,959, compared to 56,056 for DIHT and 180,482 for SRSU-HT.
• Bias and Coverage: MADI estimators exhibit unbiasedness regardless of the underlying selection bias or sample size; coverage probabilities consistently approach nominal levels (≈95%). Naïve machine learning estimators (without error correction) display substantial bias.
• Sample Size Requirements: For a fixed coefficient of variation (CV), MADI achieves target precision with markedly smaller samples compared to GREG or HT, except under extreme selection bias (e.g., entirely deterministic NPD selection on $y_i$1).
• Robustness: Performance remains strong under high selection bias and various missingness mechanisms, including MNAR conditions.

Theoretical and Practical Implications

The MADI estimator decouples the impact of model misspecification and overfitting from design-bias by explicitly correcting prediction errors within a controlled probability sample. This enables:

• Safe Integration of Rich, High-Dimensional Models: Unlike GREG, which falters as the number of auxiliary variables increases, MADI leverages large NPDs to fit complex models without jeopardizing inference validity.
• Scalability and Automation: The method is amenable to industrialized official statistics production, as model selection and training can be fully automated and standardized across survey cycles.
• Broad Applicability: The strategy applies to business registers, administrative datasets, web panels, and other sources where partial coverage and auxiliary information are prevalent. Its design-based protections facilitate adoption in settings with rigid audit and validation requirements (e.g., NSOs).

Future Directions

The research points toward several avenues:

• Extending to Multiple Study Variables: MADI’s flexibility with auxiliary variables and modeling strategies supports survey designs where different variables require different model specifications.
• Nonresponse and Measurement Error: The present work suggests further investigation into MADI’s behavior when probability samples themselves are affected by nonresponse, and when NPD contains measurement errors on the study variable.
• Advanced Model Integration: As NPD sizes increase, the use of neural networks and other high-capacity models becomes feasible—potentially extending the methodological frontier of model-assisted survey inference.
• Operational Implementation: Questions remain about best practices for model selection, diagnostics, and computational efficiency for very large populations and complex NPDs.

Conclusion

Model Assisted Data Integration (MADI) constitutes a statistically rigorous and computationally practical solution for integrating nonprobability data into official statistics. By ensuring design-unbiased estimation and variance calculation, irrespective of the models used for prediction, MADI addresses key challenges in multi-source data integration. Empirical evidence demonstrates substantial reductions in required probability sample sizes, robust performance under selection bias, and compatibility with arbitrary machine learning models. The framework is positioned as a viable and scalable path for enabling broader NPD use within the statistical system, with future research opportunities spanning nonresponse adjustment, error modeling, and complex variable structures.