Imputation Scores (I-Scores)

• Imputation Scores (I-Scores) are a framework that evaluates missing data imputation methods by comparing the estimated conditional distributions to the true data-generating distribution.
• They leverage density-ratio and energy-based approaches to avoid the limitations of RMSE, ensuring that methods replicating full distributional properties are ranked higher.
• The estimation process uses classifier-based log-ratio techniques and U-statistics, facilitating robust and objective comparisons across different imputation strategies.

Searching arXiv for papers on Imputation Scores and related ranking methods. Imputation Scores (I-Scores) are a framework for assessing missing value imputations in settings where the objective is not merely to minimize pointwise prediction error, but to rank highest an imputation method that samples from the correct conditional distribution. Introduced in "Imputation Scores" (Näf et al., 2021), the framework replaces the standard practice of artificially masking observed entries and evaluating them with predictive losses such as RMSE, a practice that can favor conditional-mean imputers under quadratic loss. The original framework develops a density-ratio I-Score applicable to discrete and continuous data, does not require to mask additional observations for evaluations, and is also applicable if there are no complete observations. A later extension, "How to rank imputation methods?" (Näf et al., 15 Jul 2025), follows the concept of I-Scores and develops an energy-based score under a different missingness assumption, with particular emphasis on why additional masking can be misleading under MAR.

1. Motivation and problem formulation

Given the prevalence of missing data in modern statistical research, a broad range of methods is available for any given imputation task. The central problem addressed by I-Scores is how to choose the “best” imputation method in a given application. The standard approach is to select some observations, set their status to missing, and compare prediction accuracy of the methods under consideration on these observations. Besides having to somewhat artificially mask observations, a shortcoming of this approach is that imputations based on the conditional mean will rank highest if predictive accuracy is measured with quadratic loss (Näf et al., 2021).

The original I-Score framework is designed to alter the ranking criterion. Rather than rewarding an imputation because it predicts held-out entries well under squared error, it is intended to rank highest an imputation that can sample from the true conditional distributions. This makes the target of evaluation explicitly distributional rather than exclusively predictive. The paper therefore develops a proper scoring framework for incomplete-data settings in which the “true” imputation law $P^*$ should achieve the maximum population score (Näf et al., 2021).

This orientation also clarifies a common misconception in imputation evaluation. RMSE and related losses can be informative about point prediction, but they are not intrinsically aligned with distribution-preserving multiple imputation. The later 2025 work makes the same point in a different way: it states that artificially masking observations can be misleading if not done carefully and that it is generally not valid under MAR, and it develops a score whose ranking is geared towards the imputation that best replicates the distribution of the data, allowing to find imputations that are suitable for a range of downstream tasks (Näf et al., 15 Jul 2025).

2. Density-ratio I-Score: formal definition

The original framework introduces a Density-Ratio I-Score for an imputation law $H\in\mathcal H_P$, where $P^*$ is the unknown “true” joint distribution with density $p^*$, $P$ is the actually observed distribution with missing-value mechanism $M\sim P^M$ and marginal density $p$, and $\mathcal H_P$ is the set of all possible imputation distributions $H$ that are consistent with $P$, meaning that they agree with $H\in\mathcal H_P$0 on the observed entries. A collection of projection index-sets $H\in\mathcal H_P$1 is equipped with a distribution $H\in\mathcal H_P$2. For each projection $H\in\mathcal H_P$3, $H\in\mathcal H_P$4 denotes the induced distribution on patterns $H\in\mathcal H_P$5, and the projected densities are written as

$H\in\mathcal H_P$6

The population Density-Ratio I-Score is

$H\in\mathcal H_P$7

An equivalent representation is obtained through

$H\in\mathcal H_P$8

which yields

$H\in\mathcal H_P$9

This representation links the score to a classification problem: one may distinguish projected observations that are fully observed on $P^*$0 from projected observations that arise from imputation under pattern $P^*$1. The score then evaluates the expected log density ratio between these two projected laws. Because the construction is based on projections, it is applicable to discrete and continuous data, and the use of projections permits evaluation even when there are no complete observations on the full vector (Näf et al., 2021).

3. Properness, MCAR, and MAR upon projections

Within the original framework, an I-Score $P^*$2 is called proper if for every $P^*$3,

$P^*$4

so that the “true” imputation $P^*$5 achieves the maximum. The population version of the density-ratio score is shown to be proper under the missing completely at random assumption, and it is also shown to be valid under missing at random with slightly more restrictive assumptions (Näf et al., 2021).

The key condition for the projected MAR result is the MAR-upon-projections assumption. For each projection $P^*$6 and all patterns $P^*$7 on $P^*$8, the paper requires

$P^*$9

Here $p^*$0 and $p^*$1 split the coordinates in $p^*$2 into those observed versus missing under pattern $p^*$3. Under this condition, Proposition 4.1 shows that the density-ratio score is proper (Näf et al., 2021).

Several special cases are identified. MCAR on the full data implies MCAR on every projection, so condition (4.4) holds. If one knows the overall mechanism is MAR on the full vector but restricts $p^*$4, then this single projection also satisfies (4.4). One may also design $p^*$5 so that each $p^*$6 corresponds to a block for which MAR is known to hold (Näf et al., 2021).

The proof strategy is KL-based. For each fixed $p^*$7, one expands the KL-divergence

$p^*$8

using a factorization of $p^*$9 into the conditional law of the missing part given the observed part and the marginal law of the observed part, with an analogous factorization for $P$0 under MAR. The difference splits into an irreducible term depending only on the marginal of the observed parts plus the KL between the two conditional laws of the missing parts. The latter is minimized to zero exactly when $P$1 (Näf et al., 2021). This suggests that propriety is driven by the conditional law of the missing coordinates rather than by agreement on already observed marginals, which are fixed by consistency with $P$2.

4. Estimation algorithm and software implementation

The practical algorithm in the original paper starts from an incomplete sample $P$3 and an imputation method that produces $P$4 completed data sets $P$5. The sample is first partitioned into groups of rows sharing the same missing-pattern $P$6. For each such pattern, the algorithm draws num.proj subsets $P$7 such that $P$8 contains at least one observed index and at least one missing index under $P$9. This ensures that on projection $M\sim P^M$0 one sees both some complete and some imputed points (Näf et al., 2021).

For each projection $M\sim P^M$1 and each imputed data set $M\sim P^M$2, the method extracts $M\sim P^M$3, the subset of rows in the original $M\sim P^M$4 that are complete on $M\sim P^M$5, and $M\sim P^M$6, the subset of rows in $M\sim P^M$7 that arise from imputing pattern $M\sim P^M$8. These two sets are rebalanced by up/down-sampling so that $M\sim P^M$9. A probability-forest, or another probabilistic classifier, is then fit to distinguish $p$0 with label 1 from $p$1 with label 0. If $p$2 denotes the estimated class probability $p$3, then

$p$4

estimates $p$5 (Näf et al., 2021).

To avoid over-fitting, the classifier step uses cross-fitting: within each $p$6, half of the data are used for training and half for evaluation, and then the roles are swapped. The estimated log-ratios are averaged over all rows of all imputed data sets, all random projections $p$7, both cross-fits, and finally across $p$8 imputations, yielding $p$9. No further masking of observed entries is needed, because missingness itself provides both observed and “pseudo-observed” samples under the random projections (Näf et al., 2021).

Finite-sample recommendations are also given. In practice one sets num.proj approximately 50–200, larger for higher $\mathcal H_P$0, and samples $\mathcal H_P$1 by first choosing sizes of the observed-and-missing subsets uniformly. The default classifier is the party probability forest with num.trees.per.proj approximately 5–20 and mtry = |A|. To avoid $\mathcal H_P$2 issues, one clips $\mathcal H_P$3. Sampling variability is assessed by repeating a split-half subsampling of the entire data $\mathcal H_P$4 approximately 30 times and using the generalized jackknife variance estimator

$\mathcal H_P$5

where $\mathcal H_P$6 is the average of the two half-sample scores in the $\mathcal H_P$7th split. This yields approximate normal confidence intervals

$\mathcal H_P$8

All of these steps are wrapped into the CRAN package Iscores (Näf et al., 2021).

$H\in\mathcal H_P$43

5. Empirical behavior and comparison with RMSE

The empirical evaluation in the original paper is organized around several complementary settings (Näf et al., 2021).

Evaluation	Setup	Reported result
Toy spiral example	Two-dimensional spiral with 30% missing under MCAR or MAR	DR I-Score ranks true data highest, then mice-cart, then sample, then loess
UCI benchmarks	15 real regression data sets, 9 imputation methods, 20% miss, MCAR/MAR	No method ever significantly exceeds the true-data score
Coverage-width calibration	$\mathcal H_P$9 points from empirical quantile intervals	Best-scoring methods cluster in high coverage, low width quadrant
CDC 2020 natality data	Mixed discrete and continuous, $H$0, $H$1	DR I-Score ranking matches out-of-bag RF ranking

In the toy “spiral” example, three simple imputers—loess regression, random “sample,” and mice-cart—are compared to the true data under

$H$2

RMSE ranks the conditional-mean imputer, loess, best, even though it clearly distorts the joint scatter. The DR I-Score, without access to the hidden $H$3, ranks the true data highest, then mice-cart, then sample, then loess, matching visual intuition (Näf et al., 2021).

On 15 UCI regression data sets with 9 imputation methods and 20% missingness under MCAR and MAR, a propriety check is performed by testing $H$4 via a one-sided $H$5-test using the jackknife variance. At $H$6, no method ever significantly exceeds the true-data score, providing no empirical contradictions to propriety. In terms of ranking, mice-cart, mice-rf, mice-pmm, and mice-midastouch consistently rank at the top, whereas mean- and missForest-imputations, described as conditional means, rank at the bottom—exactly as distribution-preserving theory would predict (Näf et al., 2021).

The calibration study examines 95% empirical quantile intervals formed from the multiple-imputation draws for each missing entry and computes average marginal coverage and average width. Points are color-coded by DR I-Score rank, and the best-scoring methods cluster in the quadrant of high coverage and low width. The paper interprets this as demonstrating that a high I-Score correlates with good multiple-imputation performance on this classical diagnostic. In the CDC 2020 natality data, DR I-Scores with confidence intervals are compared to an out-of-bag RF error obtained by contrasting each imputed set to an independent sample of complete births. Although the RF comparison uses extra complete data, its misclassification ranking exactly matches the DR I-Score ranking, providing an external sanity check (Näf et al., 2021).

Taken together, these results show that the DR I-Score is proper under MCAR, and under MAR on each projection; can be estimated without ever masking observed values; handles mixed data via generic classifiers; and yields intuitive, reliable rankings that avoid the well-known RMSE pitfall of favoring conditional-mean imputers (Näf et al., 2021).

6. Energy-I-Score and subsequent developments

A later development by Näf et al. follows the concept of Imputation Scores and develops a new, reliable, and easy-to-implement score to rank missing value imputations for a given data set without access to the complete data (Näf et al., 15 Jul 2025). The paper begins from the same general definition: a real-valued function $H$7 is a proper I-Score if for every $H$8 in the class $H$9 of valid imputation laws,

$P$0

with equality only when $P$1. Here

$P$2

so any $P$3 must agree with $P$4 on the observed cells (Näf et al., 15 Jul 2025).

The distinguishing feature of the 2025 construction is a new missingness assumption. For each variable $P$5, the paper defines $P$6, $P$7, and

$P$8

the set of coordinates that are always observed whenever $P$9 is observed. Assumption 1, denoted CIMAR$H\in\mathcal H_P$00, requires that there exists at least one $H\in\mathcal H_P$01 such that $H\in\mathcal H_P$02, $H\in\mathcal H_P$03, $H\in\mathcal H_P$04, and

$H\in\mathcal H_P$05

The motivation given is that under ordinary PMM-MAR, one only knows that $H\in\mathcal H_P$06 for the missing patterns $H\in\mathcal H_P$07, whereas a valid comparison of imputed versus observed values requires that the same conditional law governs $H\in\mathcal H_P$08 when it is observed or missing, at least after conditioning on always-observed coordinates $H\in\mathcal H_P$09 (Näf et al., 15 Jul 2025).

For a fixed variable $H\in\mathcal H_P$10, the paper defines a population subscore based on the energy score: $H\in\mathcal H_P$11 and the full score is

$H\in\mathcal H_P$12

where $H\in\mathcal H_P$13. Under Assumption 1 and MAR so that $H\in\mathcal H_P$14, each subscore is maximized exactly when $H\in\mathcal H_P$15, and summing over $H\in\mathcal H_P$16 shows $H\in\mathcal H_P$17 (Näf et al., 15 Jul 2025).

The estimation algorithm fixes $H\in\mathcal H_P$18, identifies $H\in\mathcal H_P$19, splits the sample into a test set $H\in\mathcal H_P$20 and a train set $H\in\mathcal H_P$21, masks $H\in\mathcal H_P$22 for all $H\in\mathcal H_P$23, and applies the candidate imputation method $H\in\mathcal H_P$24 times, each time re-imputing only column $H\in\mathcal H_P$25 conditional on $H\in\mathcal H_P$26. This yields, for each $H\in\mathcal H_P$27, a cloud of $H\in\mathcal H_P$28 draws from $H\in\mathcal H_P$29. The empirical subscore is then

$H\in\mathcal H_P$30

and $H\in\mathcal H_P$31 is the average over $H\in\mathcal H_P$32 (Näf et al., 15 Jul 2025).

The 2025 paper states explicitly why extra masking under MAR can be misleading. Standard cross-validation-style masking of an observed cell $H\in\mathcal H_P$33 in a MAR setting typically violates MAR, because once $H\in\mathcal H_P$34 is artificially deleted in a pattern $H\in\mathcal H_P$35, one is comparing distributions from two different missingness mechanisms, each of which may have different covariate distributions. As a result, even the true imputation law $H\in\mathcal H_P$36 can appear to fit badly. By contrast, the energy-I-Score only masks $H\in\mathcal H_P$37 in rows where $H\in\mathcal H_P$38 was originally observed, but conditions on the same subvector $H\in\mathcal H_P$39 which was always observed; because of Assumption 1, the conditional law that generated $H\in\mathcal H_P$40 in the original pattern is identical to the conditional law used when it is missing (Näf et al., 15 Jul 2025).

The empirical findings in the later paper are differentiated rather than uniformly favorable to one score. In a uniform-example with three-pattern MAR and pattern-dependent shifts, the energy-I-Score correctly ranks the $H\in\mathcal H_P$41 “draw Uniform(0,1)” imputation highest, matching the full-information RMSE ranking, whereas the DR-I-Score based on KL with random projections fails. In a Gaussian-mixture example with strong covariate shifts but CIMAR, the energy-I-Score again picks the true Gaussian imputer, while DR-I-Score misranks. In a nonlinear mixture example, all methods struggle, but energy-I-Score and the full-information ranking agree on the relative ordering of best methods. In a downstream task linking 401 k eligibility to net worth via Double-ML, imputations with best energy-I-Score also yield estimates closest to the full-data benchmark, and standard errors reflect improved uncertainty quantification (Näf et al., 15 Jul 2025).

The paper also discusses the choice of proper scoring rules. Energy score was chosen because it is strictly proper for multivariate distributions and relatively easy to estimate via U-statistics. Other choices include the Continuous Ranked Probability Score for univariate imputations and proper kernel scores such as Maximum Mean Discrepancy with characteristic kernel. The stated criteria are strict propriety, computational tractability, and sensitivity to full distribution, not just mean. In principle, the energy-score inner loop can be replaced by any proper scoring rule $H\in\mathcal H_P$42, yielding an analogous I-Score (Näf et al., 15 Jul 2025). This suggests that I-Scores are not a single statistic but a broader proper-scoring framework for ranking imputations under explicit missingness assumptions and explicit evaluation targets.