---
title: Rashomon Set in Machine Learning
url: https://www.emergentmind.com/topics/rashomon-set
type: topic
---

# Rashomon Set in Machine Learning

Searching arXiv for recent Rashomon-set papers to ground the article and verify the cited works.
arxiv_search.search(query="Rashomon set model multiplicity Rashomon effect", max_results=10)
arxiv_search.search(query="all:\"Rashomon set\" OR all:\"Rashomon effect\" machine learning", max_results=10)
arxiv_search.search(query="ti:\"Rashomon Sets and Model Multiplicity in Federated Learning\" OR ti:\"Beyond the Single-Best Model: Rashomon Partial Dependence Profile for Trustworthy Explanations in AutoML\" OR ti:\"Fairness and Sparsity within Rashomon sets\"", max_results=10)
The Rashomon set is the collection of models that are all nearly as good as the best model under a specified objective, even though they may differ substantially in predictions, decision boundaries, explanations, fairness properties, robustness, or internal structure. Recent work treats it as the operational counterpart of model multiplicity: in empirical ML pipelines, the relevant object is often not a single selected predictor but a family of near-optimal candidates whose coexistence reveals uncertainty that aggregate performance metrics suppress [2412.12115][2602.09520][2511.21799].

## 1. Core definition and the logic of near-optimality

A standard formulation defines a reference model as
\[
f_R = \arg \min_{f \in F} E[L(f)],
\]
and the Rashomon set as
\[
\mathcal{R}_{L,\epsilon}(f_R) = \{f \in F \mid E[L(f)] \leq E[L(f_R)] + \epsilon\},
\]
with tolerance parameter \(\epsilon>0\) [2412.12115]. In empirical settings, the full model space \(F\) is usually infeasible to search exhaustively, so papers often work with an empirical subset \(\hat F \subset F\), or with an explicitly generated candidate pool.

Closely related formulations appear across tasks. In regularized empirical risk minimization, the empirical Rashomon set is
\[
\hat{\mathcal{R}}(\epsilon)=\{f \in \mathcal{F}: \hat{obj}(f)\leq \hat{obj}(\hat{f}) + \epsilon\},
\]
with \(obj_D(f)=L_D(f)+\lambda \Omega(f)\) [2511.21799]. In AutoML for regression, the best model is
\[
M^* = \arg\min_{M_k \in \mathcal{M}} \phi(M_k),
\]
and the Rashomon set is
\[
\mathcal{R_\varepsilon} = \left\{ M_k \in \mathcal{M} \mid \phi(M_k) \leq \phi(M^*) * (1 + \varepsilon) \right\},
\]
with \(\varepsilon = 0.05\) in the reported experiments [2507.14744]. In sparse decision trees, the set is defined multiplicatively around a benchmark tree \(t_{\_}\):
\[
R_{\textrm{set}(\epsilon, t_{\_}, \mathcal{T})\!:=\!\{t \!\in\! \mathcal{T}\!:\! Obj(t,x,y)\!\leq\! (1+\epsilon)\times Obj(t_{\_},x,y) \},
\]
where \(Obj(t,x,y)=\ell(t,x,y)+\lambda H_t\) [2209.08040].

Two implications recur. First, similar aggregate performance does not imply similar behavior. Second, the size of the Rashomon set is itself informative: a larger set is interpreted as a stronger Rashomon effect, because more models are nearly as good as the optimum [2412.12115].

## 2. Domain-specific formalizations

The most basic formulation is prediction-centric, but recent work generalizes the Rashomon set to settings where “same prediction” is not the right equivalence notion. In federated learning, existing centralized definitions are insufficient because clients have heterogeneous local distributions, standard aggregation can hide minority or client-specific behavior, and privacy constraints limit what can be shared. Three notions are introduced: a global Rashomon set defined over aggregated client statistics, a \(t\)-agreement Rashomon set requiring the condition to hold for at least a fraction \(t\) of clients, and individual Rashomon sets defined on each client’s local distribution [2602.09520].

In sequential decision-making, the Rashomon effect is translated from classification to policies in an MDP. Two policies belong to the relevant Rashomon set when they induce identical DTMCs on an environment with respect to a specified property \(\varphi\), yet differ in internal structure under a user-specified metric \(\delta\), such as feature-attribution ranking. Because stochastic transitions make single trajectories inadequate, the criterion is established via probabilistic model checking rather than rollout comparison [2512.17470].

For dimension reduction, the object is a set of embeddings rather than classifiers. A loss-based definition uses
\[
\mathcal{R}_{\mathrm{loss}(X,F,\delta,L_{DR}) = \left\{\theta\in\Theta:\; L_{DR}(X,F_\theta)\le L_{DR}(X,F_{\theta^*})+\delta\right\},
\]
while a graph-based perspective defines a Rashomon set of embeddings with similar weighted neighborhood geometry under a soft Jaccard distance [2604.00485]. In factorial heterogeneity estimation, the analogous object is the Rashomon Partition Set, the set of partitions whose posterior score is within a user-chosen tolerance of the MAP partition, allowing multiple statistically indistinguishable but substantively different pooling structures [2404.02141].

Chaotic forecasting introduces a further modification: the Rashomon set becomes horizon-dependent. With horizon-specific loss
\[
L_k(h) = \mathbb{E}_{t}\left[\|\mathbf{x}_{t+k} - \hat{\mathbf{x}_{t+k}^h}\|^2\right],
\]
the horizon-constrained Rashomon set is
\[
\mathcal{R}_{\epsilon_k}^{(k)} = \{h \in \mathcal{H} : L_k(h) \leq L_k^* + \epsilon_k\},
\]
so membership can change with forecast horizon \(k\) [2605.05218]. This suggests that the Rashomon set is not a single fixed ontology but a task-dependent family of near-optimality objects.

## 3. Multiplicity, instability, and set-level diagnostics

Once a Rashomon set is defined, the next question is how to quantify multiplicity. Several papers treat predictive disagreement among near-optimal models as the central object. In federated and centralized settings, three recurring metrics are ambiguity,
\[
\alpha(D) \triangleq \frac{1}{|D|} \sum\limits_{x_i\inD} \max\limits_{h_w \in R} \mathbbm{1} \left[ \argmax h_w(x_i) \neq \argmax h_{w^*}(x_i) \right],
\]
discrepancy,
\[
\gamma(D) \triangleq \max\limits_{h_w \in R} \frac{1}{|D|} \sum\limits_{x_i\inD}  \mathbbm{1} \left[ \argmax h_w(x_i) \neq \argmax h_{w^*}(x_i) \right],
\]
and disagreement,
\[
\mu(x_i)\triangleq2\text{Pr}\{\mathbbm{1}\left[ h_w(x_i) > \tau\right] \neq \mathbbm{1} \left[h_w'(x_i)>\tau\right]; h_w, h'_w \in \mathcal{R}\},
\]
all of which make explicit that near-optimality can coexist with pointwise inconsistency [2602.09520].

A second line of work studies explanation instability rather than raw prediction disagreement. In educational prediction, variable importance is computed with Permutational Variable Importance, and ranking stability across the Rashomon set is assessed by Kendall’s \(\tau\) and the Variable Importance Order Discrepancy
\[
V_\epsilon(f_R) = \max_{f \in \mathcal{R}_{L,\epsilon}(f_R)} \tau(f_R, f),
\]
with lower values indicating greater instability in importance ordering [2412.12115]. In AutoML explanation analysis, the Rashomon Ratio
\[
\text{RR}_\varepsilon = \frac{|\mathcal{R}_\varepsilon|}{|\mathcal{M}|}
\]
and Rashomon Set Size
\[
\text{RSS}_\varepsilon = |\mathcal{R}_\varepsilon|
\]
measure predictive multiplicity, while explanation variability is summarized through the Rashomon PDP, its bootstrap confidence interval, the Mean Width of Confidence Intervals, and the Coverage Rate against the best model’s PDP [2507.14744].

Medical modeling introduces Rashomon_DETECT, which selects the \(k\) most different models in a Rashomon set using explanation profiles, together with the Profile Disparity Index,
\[
PDI(g_{f_1}^j, g_{f_2}^j) = \frac{1}{\sup \mathcal{D}_j - \inf \mathcal{D}_j} \int_{\mathcal{D}_j}\left[\mathrm{sgn} \left( \frac{\partial}{\partial X^{(j)} g_{f_1}^j(z) \right) \neq \mathrm{sgn} \left( \frac{\partial}{\partial X^{(j)} g_{f_2}^j(z) \right) \right] \,\mathrm{d}z,
\]
which measures the fraction of the domain where two profiles disagree in slope direction [2308.11446]. A related critique argues that explanation evaluation must be model-relative within a Rashomon set: AXE evaluates whether the features named by an explanation can recover the model’s predictions, and is reported to detect adversarial fairwashing with a 100% success rate [2601.08703]. Metamorphic testing extends this logic with five relations—faithfulness, cross-model sensitivity consistency, explanation divergence implies sensitivity divergence, invariance under irrelevant transformations, and proportional attribution response—specifically to test whether post-hoc attributions track behavior across near-equivalent models [2606.06056].

## 4. Construction, enumeration, and sampling

Because the relevant hypothesis spaces are often combinatorial or high-dimensional, Rashomon-set analysis depends on algorithms for exploration rather than on the definition alone. For sparse decision trees, treeFARMS gives the first complete enumeration of the Rashomon set for a highly nonlinear discrete function class, using branch-and-bound pruning and a specialized Model Set representation that supports counting, querying, and uniform sampling [2209.08040]. For rule set models, exact branch-and-bound enumeration is complemented by approximate counting and near-uniform sampling via random parity constraints, with the thresholded feasible region written as
\[
\mathcal{R}_\theta = \{ s \;:\; f(s) \le \theta \},
\]
where \(f(s)=\ell(s)+\lambda |s|\) [2406.03059].

Other work deliberately avoids enumeration. An enumeration-free mathematical programming framework characterizes fairness or sparsity exactly within the Rashomon set for scoring systems and decision diagrams by optimizing fairness metrics over the feasible region
\[
R := \{h \in H : \mathcal{L}_S(h) \le \mathcal{L}_S(h_S) + \epsilon\},
\]
optionally under sparsity constraints \(\text{Sparsity}(h)\le \alpha\) [2502.05286]. In active learning, UNREAL constructs a committee by enumerating the Rashomon set of decision trees, grouping trees by identical classification patterns, and retaining one representative per group so that disagreement reflects diversity among high-performing explanations rather than redundancy or noise [2503.06770].

For neural models, dropout-based exploration replaces repeated retraining with inference-time perturbations around a pretrained empirical risk minimizer. The empirical Rashomon set is approximated by sampled models satisfying a loss constraint, and the paper reports runtime speedup up to \(20\times \sim 5000\times\) compared with baselines while improving multiplicity metric estimation [2402.00728]. In domain-specific AutoML workflows, the Rashomon set can also be operationalized as a finite candidate pool generated by random search and Bayesian optimization, followed by thresholding relative to the best-performing model [2412.12115].

These methods separate two distinct tasks: identifying which models are “good enough,” and then exploring variability within that subset. The distinction matters because exhaustive enumeration may be impossible or unnecessary, whereas representative sampling, certified optimization, or compressed set representations can still expose the structure of multiplicity.

## 5. Uses across explanation, fairness, robustness, and scientific inference

A recurring use of the Rashomon set is to replace single-model interpretation with set-level interpretation. In educational data mining, a Rashomon set of 424 tree-based models built from decision trees, random forests, LightGBM, and XGBoost improves predictive accuracy by roughly 2–6% relative to the broader model space, while showing that variable-importance rankings are more consistent in binary than in multiclass classification. The same study identifies `imd_band` and `highest_education` as consistently important variables, but also shows that their importance varies across courses, especially course DDD, whose multiclass VIOD reaches \(-0.866\) [2412.12115].

In AutoML explanation analysis, the Rashomon PDP aggregates partial dependence profiles across all near-optimal models rather than privileging the single best model. On 35 regression datasets from the OpenML CTR23 benchmark suite, the reported result is that in most cases the Rashomon PDP covers less than 70% of the best model’s PDP, with a moderate negative Spearman correlation between Rashomon ratio and coverage rate of \(\rho=-0.53\), 95% CI \([-0.75,-0.19]\), \(p=0.003\) [2507.14744]. In medical data, the same set-level logic is used to compare near-optimal models for 6-month survival prediction in hemophagocytic lymphohistiocytosis, where equally accurate random forest and gradient boosting models can diverge in clinically important partial dependence profiles for APTT and PLT [2308.11446].

Fairness analysis is another major application. In federated learning, all three Rashomon-set notions—global, \(t\)-agreement, and individual—are reported to yield useful but different views of client heterogeneity; strict \(t\)-agreement thresholds such as \(t=0.9\) may produce an empty set, which is interpreted as strong incompatibility across client distributions [2602.09520]. In scoring systems and decision diagrams, fairness values ranging from highly favorable to significantly unfavorable for a protected group remain attainable while staying within less than 1% of the best possible training accuracy for the hypothesis class, and sparsity constraints can shrink that range or disproportionately harm specific subgroups [2502.05286].

Set-level analysis is also used to improve robustness or downstream selection. In sequential decision-making, ensembles constructed from the Rashomon set are reported to be more robust to distribution shifts than individual policies, and permissive policies derived from the Rashomon set reduce verification cost while preserving optimality [2512.17470]. In trustworthy ML more broadly, a large Rashomon set can support reactive robustness—an attack may break one model while others remain accurate—but releasing more near-optimal models increases information leakage, making the set both a resource and a risk [2511.21799]. In chaotic forecasting, decision-aligned selection over horizon-constrained Rashomon sets improves decision quality by 18–34% on Lorenz-96, Kuramoto–Sivashinsky, wind power, traffic, and weather tasks while maintaining competitive predictive performance [2605.05218].

## 6. Critiques, limits, and selection principles

Recent literature also emphasizes that the existence of a Rashomon set does not by itself guarantee desirable outcomes. A central fairness result is that one should be intentional about selection within the set: random choice among equally accurate models is not equivalent to choosing the fairest model. In a broad combinatorial analysis of binary classifiers, the Rashomon set can grow exponentially with dataset size, individual predictions can flip with probabilities that depend on \(w_i=|2p_i-1|\), and direct optimization can find much fairer models than uniform random sampling from the set [2501.15634].

A sharper critique comes from allocation multiplicity. In scarce-resource settings, the Rashomon set of predictive models may be a poor proxy for the much larger space of equal-utility allocations. In a healthcare case study, the least discriminatory allocation recovered from a sampled Rashomon set remains discriminatory, outcome homogenization persists, and 15% to 30% of qualified individuals are never selected in any recovered allocation, leading to the argument that model multiplicity only matters when it translates into allocation multiplicity [2503.16621]. This suggests that some promised fairness and diversity benefits of the Rashomon set depend on the mapping from model scores to decisions, not on model-space multiplicity alone.

A further limit is that multiplicity can be undesirable when the objective is consistency. Predictive multiplicity may create arbitrariness for individuals because equally accurate models can still disagree on their labels. To address this, outlier correction, local patching, and pairwise reconciliation are proposed as mechanisms for reducing disagreement among the members of a Rashomon set before distilling the reconciled behavior into a single interpretable model for deployment [2601.09071].

Taken together, these results establish a set of selection principles. The Rashomon set is valuable because it reveals where performance-equivalent models agree and where they diverge; because it allows fairness, robustness, sparsity, and explanation quality to be optimized under an explicit near-optimality constraint; and because it exposes when single-model interpretation is misleading. At the same time, its practical meaning depends on the hypothesis class, the tolerance parameter, the evaluation metric, the decision rule that maps predictions to actions, and the computational method used to approximate the set. A plausible implication is that the Rashomon set is best understood not as a universal cure for arbitrariness, but as a structured object for making arbitrariness visible and therefore governable.

Source: https://www.emergentmind.com/topics/rashomon-set