Directed Prediction Change (DPC) Metric

• Directed Prediction Change (DPC) is a deterministic metric that assesses local feature attribution by measuring the directional alignment between feature importance and prediction changes.
• DPC improves on traditional Prediction Change by incorporating sign alignment and eliminating stochastic sampling, resulting in an order-of-magnitude speedup in evaluation.
• DPC provides robust, interpretable, and reproducible fidelity measures, making it particularly valuable for high-stakes applications such as medical decision support and financial risk assessment.

The Directed Prediction Change (DPC) metric is a deterministic evaluation protocol for assessing the fidelity of local feature attribution (FA) methods in machine learning, specifically within the context of guided perturbation experiments. DPC extends the traditional Prediction Change (PC) metric by incorporating the directional alignment between attribution and perturbation, resulting in a fidelity metric that is trustworthy, reproducible, and efficient. By eliminating reliance on Monte Carlo sampling prevalent in metrics like Infidelity, DPC achieves an order-of-magnitude computational speedup while providing robust fidelity measures for high-stakes domains such as medical decision support (Iselborn et al., 26 Nov 2025).

1. Guided Perturbation Experiment and Prediction Change (PC)

The guided perturbation approach forms the foundational experimental protocol underlying both PC and DPC. It operates as follows:

• Let $f$ denote a black-box model, and $s_f^y : \mathbb{R}^d \rightarrow \mathbb{R}$ return the scalar score for class $y$.
• Given an input $x \in \mathbb{R}^d$ and a local attribution method $\mathcal{A}_f^y(x) \in \mathbb{R}^d$ assigning an importance $a_i(x)$ to each feature $i$, features are ablated (set to a baseline $b$) in a specified order.
• The perturbed input after $t$ steps is $\pi_t(x)$.

In the Most Relevant First (MoRF) setup, features are ablated in descending order of importance, and in Least Relevant First (LeRF), in ascending order. The traditional PC at each step $t$ is:

$$\mathrm{PC}_t^y(x) = s_f^y(\pi_t(x)) - s_f^y(\pi_{t-1}(x))$$

A high-fidelity attribution leads to a significant early decrease in $s_f^y$ during MoRF, captured by a large area between the MoRF and LeRF PC curves (ABPC).

2. Definition and Formula of Directed Prediction Change (DPC)

PC does not differentiate the correctness of attribution direction, as it measures only the magnitude of prediction change per ablation. DPC introduces sign alignment by multiplying PC with the signs of both the attribution and the actual feature perturbation. The explicit DPC formula for each ablation step $t$ and feature $i$ is:

$$\mathrm{DPC}^y_t(x) = \sigma(a_i(x)) \cdot \sigma(\pi_{t-1}(x)_i - \pi_t(x)_i) \cdot \mathrm{PC}_t^y(x)$$

where $\sigma(u) = \mathrm{sign}(u)$.

If multiple features (e.g., image superpixels) are ablated at once, their attributions and perturbation values are summed before applying the sign function. The final DPC score is computed as the weighted area between the MoRF and LeRF DPC curves (ABPC), with higher scores indicating greater directional agreement and fidelity.

3. Determinism and Computational Efficiency

DPC attains determinism by omitting random sampling entirely:

• Unlike Infidelity, which estimates an expectation over random perturbations and thus requires hundreds of model evaluations per input for low-variance estimates, DPC uses only $2T$ guided steps (with $T$ the number of features or superpixels).
• Empirical results show that DPC achieves a median speedup of approximately $9.9\times$ (mean $8.3\times$) over Infidelity (e.g., 40 evaluations vs. 640 for Infidelity).
• The absence of randomness ensures identical results upon repeated runs, enabling reproducibility and reliability, particularly for hyperparameter selection and regulatory workflows.
• When running both PC and DPC in tandem, the shared guided experiment yields an aggregate speedup of approximately $20\times$ compared to running Infidelity alone.

4. Relation to Other Fidelity Metrics

Metric	Perturbation Protocol	Directionality Sensitivity	Stochasticity
PC	Guided (MoRF/LeRF)	No	No
Infidelity	Random (Monte Carlo)	Yes	Yes
DPC	Guided (MoRF/LeRF)	Yes	No

• PC measures the magnitude of prediction change for each ablated feature but does not account for whether the attribution sign aligns with the ablation-induced score change, leading to systematic misranking of local methods.
• Infidelity uses stochastic perturbations to compute the mean squared error between the predicted and observed effects, is applicable to both baseline-oriented and local methods, but suffers from computational cost and variance.
• DPC reuses the guided ablation of PC but evaluates fidelity through directional alignment, providing a metric that is robust, deterministic, and specific to local attributions. DPC penalizes attributions whose signs do not match the observed prediction change, in stark contrast to PC.

5. Algorithmic Implementation

A direct implementation is as follows:

moRF_indices = argsort_descending(a) leRF_indices = argsort_ascending(a) def run_curve(order): x_t = x curve = [] for t in range(1, T+1): i = order[t] x_prev = x_t x_t = replace_feature(x_prev, i, b) delta_score = s_f^y(x_t) - s_f^y(x_prev) sign_attr = sign(a[i]) sign_pert = sign(x_prev[i] - x_t[i]) dpc_t = sign_attr * sign_pert * delta_score curve.append(dpc_t) return curve moRF_curve = run_curve(moRF_indices) leRF_curve = run_curve(leRF_indices) DPC_score = weighted_area_between(leRF_curve, moRF_curve)

Choosing an appropriate baseline $b$, setting $T$ to the number of features (tabular) or a manageable number of superpixels (images), and employing the weighted ABPC aggregation are essential for rigorous assessment.

6. Empirical Evaluation

DPC was evaluated across diverse tasks and methods (Iselborn et al., 26 Nov 2025):

• Datasets: HELOC tabular credit data (20 features), ISIC skin lesion classification (224×224 images).
• Models: Linear/logistic models, MLP with 7 hidden layers (HELOC), InceptionV1 CNN (ISIC).
• Explanation Methods: Seven local FA techniques (Vanilla Gradient, Guided Backprop, SmoothGrad, VarGrad, Integrated Gradients, LIME, DeepLiftSHAP), random baselines, and edge detectors; a total of 4,744 FA configurations.
• Results:
  • DPC correctly elevates local FA methods—such as gradients on linear models—over contrastive, baseline-oriented methods, whereas PC systematically fails to do so.
  • DPC and Infidelity are strongly anti-correlated (Spearman $\rho \approx -0.4$ to $-0.8$), indicating that both capture similar fidelity properties, but DPC is less sensitive to degenerate or trivial explanations, where Infidelity may spuriously reward misleading hyperparameters.
  • Mean runtime per run for Infidelity is approximately 4,900 s versus 594 s for DPC (mean speedup $8.3\times$, median $9.9\times$). Combined with PC, overall workflow speedups reach $20\times$.

7. Practical Advice, Limitations, and Future Research

Practical recommendations for DPC include:

• Selecting a baseline $b$ that genuinely represents feature absence, such as zero, feature mean, or a domain-informed constant.
• Setting the guided ablation sequence length $T$ to the number of features (tabular) or an interpretable number of superpixels (images), with 20–50 steps typically sufficient.
• Using the same runs for both PC and DPC when evaluating baseline-oriented and local methods to streamline hyperparameter search.
• Weighting early ablation steps in ABPC, as these most acutely reveal attribution fidelity.
• Exercising caution with out-of-distribution perturbations, particularly in image domains; smaller, region-based ablations or local Gaussian perturbations may be more appropriate if full baseline replacements are unrealistic.

Noted limitations include potential attenuation of the fidelity signal in high-dimensional settings due to aggregation of directional signs, and persistent vulnerability to out-of-distribution artifacts which afflict all perturbation-based metrics. Future work may include adaptive feature ablation (fractional or smooth perturbations), automated baseline selection to minimize distributional shift, and the generalization of DPC to sequences, graphs, or certification via adversarial approaches.

DPC preserves the conceptual simplicity and diagnostic power of guided perturbation but corrects PC’s directionality blindspot, offering a deterministic, interpretable, and substantially more efficient alternative to stochastic fidelity metrics (Iselborn et al., 26 Nov 2025).