Efficient MPRT (eMPRT) Evaluation

• Efficient Model Parameter Randomisation Test (eMPRT) is an XAI evaluation technique that measures the relative increase in explanation entropy following full parameter randomisation to assess explanation faithfulness.
• It employs full model randomisation and Shannon entropy of attribution maps to overcome biases in layer-wise tests, yielding a single, interpretable reliability score.
• Empirical evaluations on benchmarks like ImageNet and MNIST show that eMPRT delivers enhanced metric stability and discriminative power compared to conventional MPRT methods.

Efficient Model Parameter Randomisation Test (eMPRT) is an evaluation methodology in explainable artificial intelligence (XAI) designed to assess the faithfulness of local explanation methods with respect to neural network model parameters. Unlike the standard Model Parameter Randomisation Test (MPRT), eMPRT eschews repeated layer-wise randomisation and similarity metrics, focusing instead on quantifying the relative rise in explanation complexity, measured as entropy, after a full model parameter randomisation. This paradigm addresses specific methodological caveats found in MPRT approaches and yields substantially improved metric reliability and computational efficiency (Hedström et al., 2024, Hedström et al., 2024).

1. Formal Definition and Theoretical Foundation

Let $f\!:\!\mathbb{R}^d\!\to\!\mathbb{R}^c$ denote a differentiable model with parameters $\theta$. For input $x$, ground truth $y$, and an explanation function $\Phi(x, f, y) \in \mathbb{R}^d$, let $e$ be the explanation for $(x, f, y)$. Let $\hat{f}$ denote a fully randomised copy of $f$, where all parameters are replaced by independent draws from the parameters’ prior or are shuffled.

Define the eMPRT score for a single sample $(x, y)$ as: $\theta$0 where $\theta$1, $\theta$2, and $\theta$3 is a scalar-valued complexity function.

A canonical choice of $\theta$4 is the Shannon entropy computed over $\theta$5-bin histograms of the attribution vector entries: $\theta$6 where $\theta$7 is the count of attribution values in bin $\theta$8. This functional automatically normalizes for sign, magnitude, and scale, and is adaptable via $\theta$9, typically set near $x$0 for stability (Hedström et al., 2024).

2. Motivation and Methodological Advantages

The core principle underlying eMPRT is that a genuinely sensitive explanation method should react to the destruction of a model’s learned structure—i.e., full parameter randomisation. In such untrained networks, attribution maps are expected to become high-entropy or unstructured, as the mapping from input to output degenerates to noise. Thus, a faithful explanation method will exhibit a positive relative increase in attribution complexity.

The eMPRT framework resolves several known deficiencies in conventional MPRT paradigms:

• Mitigation of similarity metric bias: Conventional use of $x$1 (e.g., Spearman-$x$2, SSIM) is vulnerable to artifacts from nonstationary attribution statistics ("gradient shattering").
• Insensitivity to layer-randomisation order: Layer-wise partial randomisation typically leaves substantial portions of the computation intact, distorting faithfulness metrics.
• eMPRT’s full randomisation and entropy-based complexity measure avoids these pitfalls by requiring only two explanation passes per sample (original and randomised) and outputting a single, interpretable reliability score (Hedström et al., 2024).

3. Implementation Procedure

The eMPRT evaluation consists of the following steps for a batch of $x$3 test examples:

1. For each $x$4, compute $x$5, then $x$6 via histogram entropy.
2. Construct $x$7 by fully randomizing all $x$8’s parameters (independently per layer, with replacement from the prior or via shuffling).
3. For each $x$9, compute $y$0 and $y$1.
4. For each sample, evaluate $y$2.
5. Aggregate by averaging: $y$3.

This approach yields an aggregate reliability score per XAI method, requiring only two forward/explanation passes per sample (Hedström et al., 2024, Hedström et al., 2024).

4. Comparison With MPRT and Related Variants

Aspect	MPRT	sMPRT	eMPRT
Randomisation Protocol	Progressive, layer-wise (top-down/bottom-up)	Same as MPRT	Full randomisation in a single step
Evaluation Metric	Similarity metric ($y$4)	Smoothed similarity via noisy input sampling	Relative rise in entropy (complexity)
Computational Passes	$y$5 (layers per sample)	$y$6 (layers $y$7 sample denoising)	2 (original/randomised per sample)
Diagnostic Granularity	Per-layer profile	Smoothed per-layer profile	Single global score
Biases Addressed	Layer-order artifacts, similarity metric bias	Reduced shattering noise, but not all biases	Mitigates similarity and layer-order confounds

eMPRT eliminates dependence on similarity measures and circumvents partial-randomisation artifacts, achieving both higher computational efficiency and more reliable faithfulness assessment (Hedström et al., 2024).

5. Empirical Findings and Reliability

Empirical evaluation by Hedström et al. compared MPRT, sMPRT, and eMPRT on standard benchmarks (ImageNet, MNIST, Fashion-MNIST) and multiple explanation methods (e.g., Saliency, Gradient, GradCAM, IG, LRP), plus random-attribution baselines. Evaluation used the MetaQuantus protocol’s noise-resilience (NR), adversarial-reactivity (AR), and meta-consistency (MC) scores.

Key findings include:

• eMPRT consistently achieved higher MC scores across all datasets and method-sets, often improving mean MC by 5–10 points relative to standard MPRT (Hedström et al., 2024, Hedström et al., 2024).
• eMPRT assigned lower (more discriminative) scores to random-baseline explanations than MPRT.
• Statistic example: On LeNet+FashionMNIST, group $y$8,
  • eMPRT: $y$9
  • sMPRT: $\Phi(x, f, y) \in \mathbb{R}^d$0
  • MPRT: $\Phi(x, f, y) \in \mathbb{R}^d$1

These results indicate both increased discriminative power and enhanced ranking stability, as measured by meta-evaluation under input and parameter perturbations.

6. Limitations and Practical Considerations

While eMPRT provides significant advantages, there are limitations:

• The histogram bin count $\Phi(x, f, y) \in \mathbb{R}^d$2 is a hyperparameter; $\Phi(x, f, y) \in \mathbb{R}^d$3 was found robust but may require tuning for specific domains.
• eMPRT yields only an aggregate score per explanation method, precluding layer-wise diagnostic insights.
• For highly sparse or skewed attribution distributions, the histogram entropy’s adequacy as a complexity proxy should be validated prior to application.
• Domain specificity may affect the entropy rise, especially in settings where randomised networks produce non-uniform attribution patterns, suggesting a need for context-aware calibration (Hedström et al., 2024).

7. Summary and Significance

Efficient Model Parameter Randomisation Test reframes the XAI faithfulness evaluation problem as detecting the relative increase in explanation entropy upon global parameter randomisation, rather than via layer-wise similarity decay. This results in a dramatically more efficient algorithm requiring only two explanation evaluations per sample and eliminates longstanding methodological issues inherent to the MPRT family. Large-scale empirical studies demonstrate that eMPRT offers more reliable and discriminative assessment of explanation sensitivity than prior art, making it a preferred baseline for robust sanity-checking of XAI methods (Hedström et al., 2024, Hedström et al., 2024).