ADIHQ Template: High-Quality Imputation

• ADIHQ Template is an integrated pipeline that chains univariate and multivariate imputers with robust statistical vetting to ensure reliable data imputation.
• It sequentially applies simple methods and complex models while using tests like KS, t-test, Chi-square, and Hotelling’s T2 to validate imputed distributions.
• The template produces detailed diagnostic plots and quantitative metrics that facilitate both manual inspection and automated feature filtering for downstream analysis.

The ADIHQ (Automatic Data Imputation & High-quality) Template provides an integrated, scikit-learn–style pipeline for the robust assessment and management of missing data imputation quality in heterogeneous datasets. Developed as the core architectural framework within the ITI-IQA toolbox, the ADIHQ template offers systematic, statistically-guarded processes for chaining univariate and multivariate imputers, vetting their outputs via rigorous statistical tests, quantitatively scoring the reliability of each (feature,imputer) pair, and filtering out features that fail specific quality, completeness, and stability criteria. Its design explicitly accommodates mixed data types—including continuous, discrete, binary, and categorical variables—and produces both per-feature diagnostics and global graphical summaries to facilitate both manual inspection and automated flagging of imputation artifacts (Pons-Suñer et al., 2024).

1. Pipeline Structure and Conceptual Framework

The ADIHQ template is instantiated as a single pipeline with three sequential phases:

1. Univariate Imputation Stage:

Each feature undergoes parallel imputation using simple methods such as mean, median, mode, and APPRandom. This is implemented using a ColumnTransformer, which fits each imputer independently to the feature in question.

1. Multivariate Imputation Stage: Features are then imputed using complex models—KNN, Iterative-BayesianRidge, Iterative-Random Forest, and XGBoost—each receiving as covariates only those variables connected in the dependency graph. Again, a ColumnTransformer structures the parallel evaluation.
2. Post-processing and Filtering Stage: Outputs are subjected to post-imputation statistical tests to reject methods introducing distributional bias; accuracy and reliability metrics are computed; per-feature optimal imputers are selected; features not meeting predefined quality thresholds are dropped.

Meta-estimator wrappers record all per-feature outputs, enabling cross-validation and reproducibility in selection and filtering steps.

2. Statistical Vetting of Imputed Data

For every feature–imputer pair, the pipeline tests the null hypothesis that the imputed and observed data share the same distribution. The type of statistical test is determined by the data class:

• Continuous Variables:
  • Kolmogorov-Smirnov test:

  $$D = \sup_x |F_{\mathrm{obs}}(x) - F_{\mathrm{imp}}(x)|$$

  Reject $H_0$ if $D > K_\alpha$, with scaling for sample sizes and significance. - Optionally, two-sample Student's t-test:

  $$t = \frac{\overline{x}_{\mathrm{obs}} - \overline{x}_{\mathrm{imp}}}{s_p \sqrt{1/n + 1/m}}$$

  with $s_p^2$ the pooled variance.

• Categorical/Discrete/Binary Variables:

  • Chi-square test of independence:

  $\chi^2 = \sum_{i=1}^k \frac{(O_i - E_i)^2}{E_i}$

  Degrees of freedom $k-1$, thresholded by significance.

• Multivariate Continuous:

  • Hotelling’s $T^2$ test for distributional equivalence of $p$-dim vectors.
  • Mahalanobis distance–based outlier rejection on individual imputed vectors.

Any imputer failing these tests is excluded from further consideration for the feature.

3. Quantitative Imputation Quality Metrics

Once a candidate imputer passes distributional testing, its predictive accuracy on held-out (i.e., actually observed but temporarily withheld) entries is evaluated using metrics tailored to the feature type:

• Normalized RMSE (continuous/discrete):

$$\mathrm{NRMSE}(y, \hat{y}) = \frac{\sqrt{ \tfrac{1}{N} \sum_{i=1}^N (y_i - \hat{y}_i)^2 }} {y_{\max} - y_{\min}}$$

The imputation “score” is $\delta = 1 - \mathrm{NRMSE} \in [0,1]$.

• Balanced Accuracy (binary):

$$\mathrm{BA} = \frac{1}{2}\left(\frac{\mathrm{TP}}{\mathrm{P}} + \frac{\mathrm{TN}}{\mathrm{N}}\right)$$

• Kullback-Leibler Divergence (categorical/nominal):

$$D_{KL}(P \parallel Q) = \sum_{j=1}^k P_j \ln \frac{P_j}{Q_j}, \quad \delta = 1 - \frac{D_{KL}}{\ln k}$$

• Earth Mover’s Distance/p-Wasserstein (continuous histograms).

4. Selection Logic and Feature Filtering

The best imputer for each feature is chosen by maximizing its accuracy score $\delta$ over all imputers passing the statistical test. If no imputer passes, APPRandom is used as fallback. The final per-feature quality score combines completeness and accuracy:

$\omega_x = \mu_x + (1 - \mu_x)\cdot \delta_x$

where $\mu_x$ is the proportion of observed (non-missing) data. Features with $\omega_x < \tau$ (user-defined threshold) are dropped.

Cross-validation is performed at the feature level, recording per-fold $\delta_x$. Stability filters require $\sigma_x$ (stddev of $\delta_x$ across folds) below some maximum, and only features with $\mu_x \geq \mu_{\min}$ are retained.

Pseudocode for the selection:

for each feature x in X: for each candidate imputer I: apply fit+transform on hold-out perform distribution test → pass_I? if pass_I: compute δ_I(x) if any pass_I: I* = argmax_I δ_I(x) else: I* = APPRandom assign δ_x = δ_{I*}(x)

5. Graphical Diagnostics and User Inspection

ADIHQ generates multiple diagnostic plots to aid in quality assurance:

• Bar charts of $\omega_x$ across features, with color-coded composition (blue: $\mu_x$, orange: $(1-\mu_x)\delta_x$, line: $\tau$).
• QQ-plots of observed versus imputed quantiles per feature.
• Overlaid histograms and density plots: $f_{\mathrm{obs}}(x)$ vs $f_{\mathrm{imp}}(x)$, critical for visually detecting distributional mismatch.
• Correlation-matrix heatmaps pre- and post-imputation.
• Mahalanobis distance scatter plots highlighting outlier status of imputed samples.

These allow both investigative and automatic detection of problematic features (e.g., if KS p-value $<$ 0.05).

6. Implementation, Algorithmic Workflow, and Recipe

The standard ADIHQ workflow comprises:

1. Injecting missingness (e.g., MCAR) into a dataframe with mixed-type columns.
2. Initializing ADIHQ with specified univariate/multivariate imputers, statistical tests, quality scorer, $\tau$, and cross-validation folds.
3. Fitting the pipeline, which triggers the imputation selection, filtering, and testing procedure.
4. Transforming the data to produce a clean, high-quality matrix with problematic features removed.
5. Visualizing results to inspect per-feature quality and imputation distributions.

Table: Core Steps and Objects in the ADIHQ Pipeline

Phase	Implementation (scikit-learn style)	Key Outputs per Feature
Univariate	ColumnTransformer (mean, median, mode, APPRandom)	Per-imputer $\delta_x$ scores
Multivariate	ColumnTransformer (KNN, Iterative-BR/RF/XGB)	Imputer acceptance flags
Postprocessing	Meta-estimator + filters	$\omega_x$, $\mu_x$, $\tau$ decisions

After fitting, selected_imputers_ holds the best imputer per column, and feature_scores_ gives all $\mu$, $\delta$, and $\omega$ values.

7. Applications, Limitations, and Interoperability

The ADIHQ template is suitable for any heterogeneous dataset subject to missingness, enables rigorous, automated minimization of imputation-induced bias, and supports downstream integrity in ML pipelines. Its compositional approach—chaining both simple and complex imputers, early-rejecting those that fail statistical tests, and automatically filtering inadequate features—raises confidence in the ultimate data reliability. It is particularly distinguished by its capacity to integrate cross-validation stability, statistical vetting by data type, and direct graphical feedback in a scikit-learn–compliant interface (Pons-Suñer et al., 2024).

Limitations include the potential for over-rejection in ultra-high-dimensional, highly correlated datasets (danger of correlated feature loss), dependence on threshold tuning for $\tau$ and $\sigma_{\max}$, and computational overhead for very large-scale application, especially with intensive multivariate estimation. The design is extensible—enabling customized importers, tests, or scorers—but always prioritizes abstaining from imputation where empirical evidence of distributional fidelity and predictive value is insufficient.