Weighted Alignment Error (WAE) Metric

• Weighted Alignment Error (WAE) metric, or xGEWFI, is a robust measure that weights per-feature KS distances by their Random Forest–derived importances to assess data fidelity.
• It is computed through a pipeline of outlier detection, imputation (using KNN), and augmentation (via SMOTE) followed by weighted aggregation of distributional errors.
• The metric enhances ethical AI reporting by pinpointing high-impact discrepancies, enabling targeted improvements in data processing and model performance.

The Weighted Alignment Error (WAE) metric—formally named "Explainable Global Error Weighted on Feature Importance" (xGEWFI)—is a metric designed to evaluate the fidelity of generated data (via imputation or augmentation) with respect to an original dataset, specifically incorporating feature importance weights derived from Random Forests. Unlike traditional metrics that assign equal weight to each feature, xGEWFI explicitly weights per-feature distributional errors by their relative predictive value, providing a decomposable and explainable global error signal aligned with ethical AI reporting standards (Dessureault et al., 2022).

1. Formal Definition

Let $d$ denote the number of features in a dataset. For each feature $j = 1, \dots, d$:

• $F_o^{(j)}(x)$ is the empirical cumulative distribution function (ecdf) for the original data on feature $j$.
• $F_g^{(j)}(x)$ is the ecdf for the generated (imputed or augmented) data on feature $j$.
• The Kolmogorov-Smirnov (KS) distance for feature $j$ is defined as:

$$D_j = \sup_x \left| F_o^{(j)}(x) - F_g^{(j)}(x) \right|$$

(See Eq. (3) in (Dessureault et al., 2022))

• $I_j$ denotes the importance of feature $j$ as estimated by a Random Forest, normalized so that $\sum_{j=1}^d I_j = 1$.
• The unweighted per-feature error is $E_j = D_j$.
• The weighted per-feature error is $WErr_j = I_j \cdot D_j$.

The global xGEWFI error aggregates the weighted errors:

$$xGEWFI = \sum_{j=1}^d I_j D_j = \sum_{j=1}^d WErr_j$$

(Eq. (8) in (Dessureault et al., 2022))

2. Calculation Pipeline

The xGEWFI metric is integrated within a canonical data preprocessing workflow:

1. Outlier Detection and Null Replacement:
   • For each feature $j$, compute the first and third quartiles $Q1_j$ and $Q3_j$. The interquartile range is $IQR_j = Q3_j - Q1_j$.
   • Define lower and upper bounds: $Q1_j - 1.5 \times IQR_j$, $Q3_j + 1.5 \times IQR_j$.
   • Values outside these bounds are replaced by nulls.
   • This procedure identifies extreme values in a distribution-free manner.
2. Data Imputation (KNNImputer):
   • Apply a $k$-nearest neighbors imputer, computing Euclidean distances over non-null entries:

   $D(x, y) = \sqrt{\sum_f (x_f - y_f)^2}$

• Nulls are imputed as the mean of the $k$ nearest neighbors, preserving local data structure.

1. Data Augmentation (SMOTE):

   • For classification, identify minority-class samples.
   • For each, sample $k$ nearest neighbors and synthesize new points via linear interpolation.
   • Augmentation increases representation of minority classes.
2. Distributional Divergence Calculation:
   • For each feature, compute $D_j$ as the KS distance between original and generated feature distributions.
   • The KS statistic quantifies the maximal difference between the two empirical distributions.
3. Random Forest Feature Importance:
   • Fit a Random Forest (regressor or classifier as context requires) on the cleaned, preprocessed original data.
   • Extract normalized mean-decrease-in-impurity importances $I_j$.
4. Weighted Aggregation:
   • For each feature, calculate $WErr_j = I_j \cdot D_j$.
   • Sum across all features to yield the xGEWFI score.

The methodology is typically implemented following the procedural outline below:

Step	Operation	Tool/Algorithm
1	Outlier detection & null replacement	IQR rule
2	Imputation	KNNImputer
3	Augmentation (if class labels)	SMOTE
4	Error measurement	Kolmogorov-Smirnov
5	Feature importance	Random Forest
6	Weighted error aggregation	xGEWFI formula

3. Interpretability and Explainability

xGEWFI is explicitly designed for interpretability and explainability:

• Each feature’s contribution to the total error is decomposed into three components: the KS error ($D_j$), the feature importance ($I_j$), and the resulting weighted error ($WErr_j$).
• Outputs include tabular and visual summaries—such as bar plots of $D_j$, $I_j$, and $WErr_j$—enabling immediate identification of high-impact features with strong distributional shifts.
• The metric’s structure supports ethical AI reporting by clarifying which features most significantly contribute to overall misalignment and justifying the aggregate error reported via transparent weighting.

4. Comparison with Unweighted Metrics

Traditional global error metrics (e.g., unweighted sum $\sum D_j$) presume all features are equally valuable. In practice, feature predictivities are highly imbalanced:

• xGEWFI rescales per-feature errors by their Random Forest–derived importance. Errors on more informative (higher $I_j$) variables penalize the global metric more heavily, correcting for bias in unweighted schemes.
• When feature importances are uniform (rare in real-world data), xGEWFI and unweighted global error converge. As importances become more imbalanced, the distinction grows—highlighting distributional mismatches on key predictors rather than obscure variables.

5. Sensitivity to Feature Importance Imbalance

xGEWFI is intentionally sensitive to the heterogeneity of Random Forest importances:

• In datasets where a small subset of features holds most of the predictive information, the xGEWFI metric will disproportionately penalize distributional deviations on those features.
• This structure prioritizes the preservation of distributional integrity on features vital to downstream tasks.
• A plausible implication is that practitioners can focus data cleaning, imputation, or augmentation efforts on high-$I_j$ features to most efficiently reduce xGEWFI and, by extension, likely performance degradation.

6. Empirical Illustrations

Empirical case studies in (Dessureault et al., 2022) demonstrate the impact of importance weighting:

• Case 1: Regression, $d=5$, $n=25{,}000$, 30% missing, 5% outliers:
  • Nearly constant $D_j\approx 0.34$ for all $j$; importances highly skewed ($I_1\approx 0.57$, $I_2\approx 0.20$, $I_3\approx 0.18$, $I_4\approx 0.004$, $I_5\approx 0.04$).
  • Weighted errors reflect importances: $WErr_1 \approx 0.19$, $WErr_4 \approx 0.0013$, etc.
  • xGEWFI $\approx 33.76$ (distinguished from the unweighted global error $1.68$).
• Case 2: Classification, same structure:
  • KS errors variable ($D_3$ small, $D_4$ large). $I_3$ largest, $I_4$ near zero.
  • Post-weighting, $WErr_3$ is elevated, $WErr_4$ minimized, altering the ranking of error sources.
  • xGEWFI $=0.54$ vs. unweighted sum $0.60$.

In both, xGEWFI enables more meaningful tracking of errors relevant to real-world tasks—errors on critical features dominate, those on low-importance dimensions become de-emphasized.

7. Applications and Ethical AI Considerations

The xGEWFI metric is applicable wherever generated data requires validation against an observed reference distribution, including:

• Missing data imputation
• Synthetic data generation for class-imbalanced learning
• Preprocessing in any supervised context (regression or classification)

The method’s explainable and decomposable nature directly supports ethical AI mandates: it enables developers and auditors to display not only aggregate errors but also the specific variables contributing most to observed discrepancies, providing transparency in reporting and guidance for iterative model/data improvements (Dessureault et al., 2022).