Metamorphic Testing with the Rashomon Set: Explanation Faithfulness in Machine Learning

Abstract: Multiple machine learning models can achieve near-equivalent predictive performance on the same task, yet provide divergent feature-based explanations. This is called the Rashomon effect of (explainable) machine learning, and it raises the question of which explanations, if any, are trustworthy. We propose a framework based on metamorphic testing that assesses explanation faithfulness without requiring ground-truth labels by exploring attributed feature importance from post-hoc explanation methods. Five metamorphic relations formalize expected consistency properties between model behavior and feature attributions. We apply this general framework to two tabular regression datasets and two post-hoc explainers (SHAP and LIME) to demonstrate the approach. The framework offers a practical, model-agnostic tool for selecting accurate models with reliable and trustworthy explanations.

• The paper introduces a metamorphic testing framework that leverages the Rashomon set to evaluate explanation faithfulness in ML without requiring ground-truth attributions.
• It empirically compares popular explainers like SHAP and LIME on tabular regression datasets, revealing significant discrepancies in explanation sensitivity and robustness.
• The work highlights that both cross-model and within-model explanation divergences can mislead users, emphasizing the need for reliable interpretability in ML models.

Metamorphic Testing with the Rashomon Set: Explanation Faithfulness in Machine Learning

Introduction

This paper addresses the Rashomon effect in explainable machine learning—where multiple models yield near-equivalent predictive accuracy but diverge significantly in their feature attributions. The phenomenon complicates the trust and interpretability of model explanations, especially when standard explanation faithfulness metrics require ground-truth attributions, which are typically unavailable in non-synthetic, real-world domains. The authors propose a metamorphic testing framework, leveraging a constructed Rashomon set of high-performing models, to assess the faithfulness and reliability of post-hoc explanation methods without requiring ground-truth explanations. This approach formalizes five metamorphic relations to systematically map the relationship between feature perturbations and attributed importances, introducing new evaluation criteria for explanation quality and robustness at both single-model and cross-model scales.

The Rashomon Effect and Model Multiplicity

The Rashomon effect, as defined by Breiman, postulates that collections of functionally distinct models can score equivalently in predictive performance. Formally, the $\varepsilon$-Rashomon set aggregates all models whose loss is within a relative $\varepsilon$-tolerance of the optimum. This property has substantial implications for variable importance studies and model interpretability: two models with indistinguishable predictive risk may yield substantially divergent post-hoc feature attributions, particularly when utilizing explanation methods like SHAP or LIME. This divergence can induce practical risks in applications requiring defensible, actionable explanations.

Metamorphic Relations for Evaluating Explanation Faithfulness

The proposed framework encodes five metamorphic relations (MRs), each operationalized via a test metric to quantify the level of violation or fidelity:

1. Explanation Faithfulness (MR1): Higher attributed importance between two features must correspond to greater model sensitivity upon perturbation. Assessed using binary faithfulness score (FS), rank-correlation ($\text{FS}_{rank}$), and AOPC metrics.
2. Cross-Model Sensitivity Consistency (MR2): When models agree on a top feature, their outputs should be directionally consistent under perturbation.
3. Explanation Divergence Implies Sensitivity Divergence (MR3): Divergence in top-attributed features between models should correspond to differentiated model sensitivities upon perturbation.
4. Invariance Under Irrelevant Transformations (MR4): Features with low attributed importance should not yield substantial output or attribution changes under perturbation.
5. Proportional Attribution Response (MR5): The magnitude of prediction changes under feature perturbation should be proportional to the attributions assigned to those features.

These relations are tested on the $\varepsilon$-Rashomon sets constructed from two tabular regression datasets (California Housing, Wine Quality) using two popular explainers (SHAP, LIME). Each explainer is applied to all models in the Rashomon set, and the attributions are evaluated for the described MRs via perturbation tests.

Figure 1: Rashomon set size as a function of relative error tolerance $\varepsilon_r$ on California Housing, demonstrating stabilization at 16 models for $\varepsilon_r \ge 0.10$.

Experimental Setup

The study examines candidate pools of 85 models from six diverse families, tuned and evaluated on standardized splits of the California Housing and Wine Quality datasets. $\varepsilon$-Rashomon set thresholds are tuned to 15% above the minimal observed MSE, resulting in constricted sets favoring gradient boosting and tree models. For each model in the Rashomon set, SHAP and LIME are used to generate post-hoc explanations on held-out test inputs. Feature perturbation pipelines are systematically executed to quantify MR violations for each explainer.

Results

The empirical evaluation reveals systematic differences between explainers, datasets, and metamorphic relations.

Figure 2: MR metric comparison across explainers and datasets, illustrating faithfulness (FS$_\text{rank}$), output perturbation (AOPC), cross-model agreement (REA), divergence consistency (DC), fragility (EFI), and proportionality (PS) metrics, with relevant CIs and baseline comparisons.

Explanation Faithfulness (MR1)

MR1 violations remain low (6%) on California Housing but increase significantly (20–26%) on Wine Quality, highlighting the challenge post-hoc explainers face with correlated features. SHAP and LIME achieve high rank-correlation with model sensitivities ($\rho \approx 0.63$ SHAP, $0.71$ LIME) on California Housing but drop on Wine Quality.

Cross-Model Sensitivity Consistency (MR2)

High applicability rates (88–100%) are observed, with violation rates around 14% for both explainers, indicating that when models concur on feature importance, the explainers mostly capture consistent output sensitivities.

Explanation Divergence Implies Sensitivity Divergence (MR3)

Here, SHAP exhibits measurable divergence, while LIME’s explanation diversity is nearly trivial—most Rashomon models are assigned identical top features regardless of family or local structural differences. Applicability rates for DC using LIME are near zero, making this MR non-informative for that explainer/dataset combination. For SHAP, observed divergence rates among models do not always correspond to output sensitivity divergence, indicating possible explainer artifacts.

Invariance Under Irrelevant Transformations (MR4)

EFI values (0.26–0.33) indicate that a non-trivial fraction of features labelled "irrelevant" by explainers still influence outputs on perturbation. This reflects either limitations of the explainer in capturing complex interactions or inherent coupling in the data.

Proportional Attribution Response (MR5)

This MR is violated frequently (33–70%), indicating that while explainers may select important features correctly, their attribution magnitudes are poorly calibrated relative to actual model response.

Implications for Explainable AI

The results demonstrate that metamorphic testing formalizes and quantifies explanation reliability independently of ground truth, enabling robust comparison of explainer behavior and model-explainer interactions in the presence of significant model multiplicity. The cross-model relations (MR2, MR3) are particularly salient for understanding whether explanation diversity genuinely reflects model diversity or is an artifact of the explanation process.

The consistent homogenization observed with LIME, despite structural differences among models, suggests a strong risk that some explanation methods may mask the presence of the Rashomon effect, potentially misleading practitioners about the plurality of valid model rationales within their solution set. Conversely, SHAP exposes more variation aligned with the underlying model behavior but may introduce its own artifacts when model sensitivities are not clearly distinct.

Limitations and Future Directions

Several experimental design choices—perturbation magnitudes, Rashomon set thresholds, and sampling—affect the generalizability of the findings. Moreover, these tests are limited to tabular regression with classical and ensemble models; extension to deep models and high-dimensional modalities (e.g., image, text) is necessary to validate whether the observed phenomena persist. Further development of MRs to capture higher-order interactions and path dependencies in attribution quality is warranted, as is the construction of task-specific or domain-specific relations for specialized applications.

Conclusion

This work introduces a principled, model-agnostic methodology for assessing explanation faithfulness in machine learning models via metamorphic testing mapped onto the Rashomon set. The framework’s independence from ground-truth attributions enables its deployment in authentic, non-synthetic settings. Numerical analysis supports that explanation reliability, divergence, and proportionality are strongly explainer-dependent and dataset-dependent, with notable discrepancies between popular methods. These insights have direct bearing on model selection pipelines driven by explanation diagnostics, emphasizing the necessity of holistic, cross-model explanation evaluation in explainable AI practice, particularly when the Rashomon effect is present or suspected.