Rashomon Set in Machine Learning

• Rashomon set is the collection of models that perform nearly as well as the best model, revealing inherent uncertainty and diversity in predictions.
• Recent research applies the concept across domains such as federated learning, sequential decision-making, and dimensionality reduction to enhance robustness and fairness.
• Various computational strategies—from exhaustive enumeration to dropout-based sampling—are used to construct and analyze these sets, balancing efficiency with fidelity.

Searching arXiv for recent Rashomon-set papers to ground the article and verify the cited works. arxiv_search.search(query="^^^^^^^^7Rashomon set model multiplicity Rashomon effect7^^^^^^^^", max_results=^^^^7all:\7Rashomon set model multiplicity Rashomon effect7^^^^) arxiv_search.search(query="^^^^^^^^7all:\7 set\" OR ^^^^^^^^7all:\7 effect\"^^^^^^^^7 machine learning7^^^^^^^^", max_results=^^^^7all:\7Rashomon set model multiplicity Rashomon effect7^^^^) arxiv_search.search(query="^^^^^^^^7ti:\7 Sets and Model Multiplicity in Federated Learning\" OR ^^^^^^^^7ti:\7 the Single-Best Model: Rashomon Partial Dependence Profile for Trustworthy Explanations in AutoML\" OR ^^^^^^^^7ti:\7 and Sparsity within Rashomon sets\"", max_results=^^^^7all:\7Rashomon set model multiplicity Rashomon effect7^^^^) The Rashomon set is the collection of models that are all nearly as good as the best model under a specified objective, even though they may differ substantially in predictions, decision boundaries, explanations, fairness properties, robustness, or internal structure. Recent work treats it as the operational counterpart of model multiplicity: in empirical ML pipelines, the relevant object is often not a single selected predictor but a family of near-optimal candidates whose coexistence reveals uncertainty that aggregate performance metrics suppress (&&&^^^^7Rashomon set model multiplicity Rashomon effect7^^^^&&&, &&&^^^^7all:\7^^^^&&&, &&&^^^^7 OR all:\7^^^^&&&).

^^^^7all:\7^^^^. Core definition and the logic of near-optimality

A standard formulation defines a reference model as

PRESERVED_PLACEHOLDER_^^^^7Rashomon set model multiplicity Rashomon effect7^^^^

and the Rashomon set as

PRESERVED_PLACEHOLDER_^^^^7all:\7^^^^

with tolerance parameter PRESERVED_PLACEHOLDER_^^^^7 OR all:\7^^^^ (&&&^^^^7Rashomon set model multiplicity Rashomon effect7^^^^&&&). In empirical settings, the full model space PRESERVED_PLACEHOLDER_^^^^7 machine learning7^^^^ is usually infeasible to search exhaustively, so papers often work with an empirical subset PRESERVED_PLACEHOLDER_^^^^7ti:\7^^^^, or with an explicitly generated candidate pool.

Closely related formulations appear across tasks. In regularized empirical risk minimization, the empirical Rashomon set is

PRESERVED_PLACEHOLDER_^^^^7 OR ti:\7^^^^

with PRESERVED_PLACEHOLDER_^^^^7 OR ti:\7^^^^ (&&&^^^^7 OR all:\7^^^^&&&). In AutoML for regression, the best model is

$M^* = \arg\min_{M_k \in \mathcal{M}} \phi(M_k),$

and the Rashomon set is

$$\mathcal{R_\varepsilon} = \left\{ M_k \in \mathcal{M} \mid \phi(M_k) \leq \phi(M^*) * (1 + \varepsilon) \right\},$$

with $\varepsilon = 0.05$ in the reported experiments (&&&^^^^7 OR ti:\7^^^^&&&). In sparse decision trees, the set is defined multiplicatively around a benchmark tree PRESERVED_PLACEHOLDER_^^^^7all:\7Rashomon set model multiplicity Rashomon effect7^^^^: PRESERVED_PLACEHOLDER_^^^^7all:\7all:\7^^^^ where PRESERVED_PLACEHOLDER_^^^^7all:\7 OR all:\7^^^^ (&&&^^^^7 OR ti:\7^^^^&&&).

Two implications recur. First, similar aggregate performance does not imply similar behavior. Second, the size of the Rashomon set is itself informative: a larger set is interpreted as a stronger Rashomon effect, because more models are nearly as good as the optimum (&&&^^^^7Rashomon set model multiplicity Rashomon effect7^^^^&&&).

^^^^7 OR all:\7^^^^. Domain-specific formalizations

The most basic formulation is prediction-centric, but recent work generalizes the Rashomon set to settings where “same prediction” is not the right equivalence notion. In federated learning, existing centralized definitions are insufficient because clients have heterogeneous local distributions, standard aggregation can hide minority or client-specific behavior, and privacy constraints limit what can be shared. Three notions are introduced: a global Rashomon set defined over aggregated client statistics, a PRESERVED_PLACEHOLDER_^^^^7all:\7 machine learning7^^^^-agreement Rashomon set requiring the condition to hold for at least a fraction PRESERVED_PLACEHOLDER_^^^^7all:\7ti:\7^^^^ of clients, and individual Rashomon sets defined on each client’s local distribution (&&&^^^^7all:\7^^^^&&&).

In sequential decision-making, the Rashomon effect is translated from classification to policies in an MDP. Two policies belong to the relevant Rashomon set when they induce identical DTMCs on an environment with respect to a specified property PRESERVED_PLACEHOLDER_^^^^7all:\7 OR ti:\7^^^^, yet differ in internal structure under a user-specified metric PRESERVED_PLACEHOLDER_^^^^7all:\7 OR ti:\7^^^^, such as feature-attribution ranking. Because stochastic transitions make single trajectories inadequate, the criterion is established via probabilistic model checking rather than rollout comparison (Gross et al., 19 Dec 2025).

For dimension reduction, the object is a set of embeddings rather than classifiers. A loss-based definition uses

PRESERVED_PLACEHOLDER_^^^^7all:\77^^^^

while a graph-based perspective defines a Rashomon set of embeddings with similar weighted neighborhood geometry under a soft Jaccard distance (&&&^^^^7all:\7Rashomon set model multiplicity Rashomon effect7^^^^&&&). In factorial heterogeneity estimation, the analogous object is the Rashomon Partition Set, the set of partitions whose posterior score is within a user-chosen tolerance of the MAP partition, allowing multiple statistically indistinguishable but substantively different pooling structures (&&&^^^^7all:\7all:\7^^^^&&&).

Chaotic forecasting introduces a further modification: the Rashomon set becomes horizon-dependent. With horizon-specific loss

PRESERVED_PLACEHOLDER_^^^^7all:\7^^^^8

the horizon-constrained Rashomon set is

PRESERVED_PLACEHOLDER_^^^^7all:\7^^^^9

so membership can change with forecast horizon PRESERVED_PLACEHOLDER_^^^^7 OR all:\7Rashomon set model multiplicity Rashomon effect7^^^^ (&&&^^^^7all:\7 OR all:\7^^^^&&&). This suggests that the Rashomon set is not a single fixed ontology but a task-dependent family of near-optimality objects.

^^^^7 machine learning7^^^^. Multiplicity, instability, and set-level diagnostics

Once a Rashomon set is defined, the next question is how to quantify multiplicity. Several papers treat predictive disagreement among near-optimal models as the central object. In federated and centralized settings, three recurring metrics are ambiguity,

PRESERVED_PLACEHOLDER_^^^^7 OR all:\7all:\7^^^^

discrepancy,

PRESERVED_PLACEHOLDER_^^^^7 OR all:\7 OR all:\7^^^^

and disagreement,

PRESERVED_PLACEHOLDER_^^^^7 OR all:\7 machine learning7^^^^

all of which make explicit that near-optimality can coexist with pointwise inconsistency (&&&^^^^7all:\7^^^^&&&).

A second line of work studies explanation instability rather than raw prediction disagreement. In educational prediction, variable importance is computed with Permutational Variable Importance, and ranking stability across the Rashomon set is assessed by Kendall’s PRESERVED_PLACEHOLDER_^^^^7 OR all:\7ti:\7^^^^ and the Variable Importance Order Discrepancy

PRESERVED_PLACEHOLDER_^^^^7 OR all:\7 OR ti:\7^^^^

with lower values indicating greater instability in importance ordering (&&&^^^^7Rashomon set model multiplicity Rashomon effect7^^^^&&&). In AutoML explanation analysis, the Rashomon Ratio

PRESERVED_PLACEHOLDER_^^^^7 OR all:\7 OR ti:\7^^^^

and Rashomon Set Size

PRESERVED_PLACEHOLDER_^^^^7 OR all:\77^^^^

measure predictive multiplicity, while explanation variability is summarized through the Rashomon PDP, its bootstrap confidence interval, the Mean Width of Confidence Intervals, and the Coverage Rate against the best model’s PDP (&&&^^^^7 OR ti:\7^^^^&&&).

Medical modeling introduces Rashomon_DETECT, which selects the PRESERVED_PLACEHOLDER_^^^^7 OR all:\7^^^^8 most different models in a Rashomon set using explanation profiles, together with the Profile Disparity Index,

PRESERVED_PLACEHOLDER_^^^^7 OR all:\7^^^^9

which measures the fraction of the domain where two profiles disagree in slope direction (&&&^^^^7all:\7 OR ti:\7^^^^&&&). A related critique argues that explanation evaluation must be model-relative within a Rashomon set: AXE evaluates whether the features named by an explanation can recover the model’s predictions, and is reported to detect adversarial fairwashing with a ^^^^7all:\7Rashomon set model multiplicity Rashomon effect7Rashomon set model multiplicity Rashomon effect7^^^^% success rate (&&&^^^^7all:\77^^^^&&&). Metamorphic testing extends this logic with five relations—faithfulness, cross-model sensitivity consistency, explanation divergence implies sensitivity divergence, invariance under irrelevant transformations, and proportional attribution response—specifically to test whether post-hoc attributions track behavior across near-equivalent models (&&&^^^^7all:\7^^^^8&&&).

^^^^7ti:\7^^^^. Construction, enumeration, and sampling

Because the relevant hypothesis spaces are often combinatorial or high-dimensional, Rashomon-set analysis depends on algorithms for exploration rather than on the definition alone. For sparse decision trees, treeFARMS gives the first complete enumeration of the Rashomon set for a highly nonlinear discrete function class, using branch-and-bound pruning and a specialized Model Set representation that supports counting, querying, and uniform sampling (&&&^^^^7 OR ti:\7^^^^&&&). For rule set models, exact branch-and-bound enumeration is complemented by approximate counting and near-uniform sampling via random parity constraints, with the thresholded feasible region written as

PRESERVED_PLACEHOLDER_^^^^7 machine learning7Rashomon set model multiplicity Rashomon effect7^^^^

where PRESERVED_PLACEHOLDER_^^^^7 machine learning7all:\7^^^^ (&&&^^^^7 OR all:\7Rashomon set model multiplicity Rashomon effect7^^^^&&&).

Other work deliberately avoids enumeration. An enumeration-free mathematical programming framework characterizes fairness or sparsity exactly within the Rashomon set for scoring systems and decision diagrams by optimizing fairness metrics over the feasible region

PRESERVED_PLACEHOLDER_^^^^7 machine learning7 OR all:\7^^^^

optionally under sparsity constraints PRESERVED_PLACEHOLDER_^^^^7 machine learning7 machine learning7^^^^ (&&&^^^^7 OR all:\7all:\7^^^^&&&). In active learning, UNREAL constructs a committee by enumerating the Rashomon set of decision trees, grouping trees by identical classification patterns, and retaining one representative per group so that disagreement reflects diversity among high-performing explanations rather than redundancy or noise (&&&^^^^7 OR all:\7 OR all:\7^^^^&&&).

For neural models, dropout-based exploration replaces repeated retraining with inference-time perturbations around a pretrained empirical risk minimizer. The empirical Rashomon set is approximated by sampled models satisfying a loss constraint, and the paper reports runtime speedup up to PRESERVED_PLACEHOLDER_^^^^7 machine learning7ti:\7^^^^ compared with baselines while improving multiplicity metric estimation (&&&^^^^7 OR all:\7 machine learning7^^^^&&&). In domain-specific AutoML workflows, the Rashomon set can also be operationalized as a finite candidate pool generated by random search and Bayesian optimization, followed by thresholding relative to the best-performing model (&&&^^^^7Rashomon set model multiplicity Rashomon effect7^^^^&&&).

These methods separate two distinct tasks: identifying which models are “good enough,” and then exploring variability within that subset. The distinction matters because exhaustive enumeration may be impossible or unnecessary, whereas representative sampling, certified optimization, or compressed set representations can still expose the structure of multiplicity.

^^^^7 OR ti:\7^^^^. Uses across explanation, fairness, robustness, and scientific inference

A recurring use of the Rashomon set is to replace single-model interpretation with set-level interpretation. In educational data mining, a Rashomon set of ^^^^7ti:\7 OR all:\7ti:\7^^^^ tree-based models built from decision trees, random forests, LightGBM, and XGBoost improves predictive accuracy by roughly ^^^^7 OR all:\7^^^^–^^^^7 OR ti:\7^^^^% relative to the broader model space, while showing that variable-importance rankings are more consistent in binary than in multiclass classification. The same study identifies imd_band and highest_education as consistently important variables, but also shows that their importance varies across courses, especially course DDD, whose multiclass VIOD reaches PRESERVED_PLACEHOLDER_^^^^7 machine learning7 OR ti:\7^^^^ (&&&^^^^7Rashomon set model multiplicity Rashomon effect7^^^^&&&).

In AutoML explanation analysis, the Rashomon PDP aggregates partial dependence profiles across all near-optimal models rather than privileging the single best model. On ^^^^7 machine learning7 OR ti:\7^^^^ regression datasets from the OpenML CTR^^^^7 OR all:\7 machine learning7^^^^ benchmark suite, the reported result is that in most cases the Rashomon PDP covers less than 7^^^^7Rashomon set model multiplicity Rashomon effect7^^^^% of the best model’s PDP, with a moderate negative Spearman correlation between Rashomon ratio and coverage rate of PRESERVED_PLACEHOLDER_^^^^7 machine learning7 OR ti:\7^^^^, 9^^^^7 OR ti:\7^^^^% CI PRESERVED_PLACEHOLDER_^^^^7 machine learning77^^^^, PRESERVED_PLACEHOLDER_^^^^7 machine learning7^^^^8 (&&&^^^^7 OR ti:\7^^^^&&&). In medical data, the same set-level logic is used to compare near-optimal models for ^^^^7 OR ti:\7^^^^-month survival prediction in hemophagocytic lymphohistiocytosis, where equally accurate random forest and gradient boosting models can diverge in clinically important partial dependence profiles for APTT and PLT (&&&^^^^7all:\7 OR ti:\7^^^^&&&).

Fairness analysis is another major application. In federated learning, all three Rashomon-set notions—global, PRESERVED_PLACEHOLDER_^^^^7 machine learning7^^^^9-agreement, and individual—are reported to yield useful but different views of client heterogeneity; strict PRESERVED_PLACEHOLDER_^^^^7ti:\7Rashomon set model multiplicity Rashomon effect7^^^^-agreement thresholds such as PRESERVED_PLACEHOLDER_^^^^7ti:\7all:\7^^^^ may produce an empty set, which is interpreted as strong incompatibility across client distributions (&&&^^^^7all:\7^^^^&&&). In scoring systems and decision diagrams, fairness values ranging from highly favorable to significantly unfavorable for a protected group remain attainable while staying within less than ^^^^7all:\7^^^^% of the best possible training accuracy for the hypothesis class, and sparsity constraints can shrink that range or disproportionately harm specific subgroups (&&&^^^^7 OR all:\7all:\7^^^^&&&).

Set-level analysis is also used to improve robustness or downstream selection. In sequential decision-making, ensembles constructed from the Rashomon set are reported to be more robust to distribution shifts than individual policies, and permissive policies derived from the Rashomon set reduce verification cost while preserving optimality (Gross et al., 19 Dec 2025). In trustworthy ML more broadly, a large Rashomon set can support reactive robustness—an attack may break one model while others remain accurate—but releasing more near-optimal models increases information leakage, making the set both a resource and a risk (&&&^^^^7 OR all:\7^^^^&&&). In chaotic forecasting, decision-aligned selection over horizon-constrained Rashomon sets improves decision quality by ^^^^7all:\7^^^^8–^^^^7 machine learning7ti:\7^^^^% on Lorenz-9^^^^7 OR ti:\7^^^^, Kuramoto–Sivashinsky, wind power, traffic, and weather tasks while maintaining competitive predictive performance (&&&^^^^7all:\7 OR all:\7^^^^&&&).

^^^^7 OR ti:\7^^^^. Critiques, limits, and selection principles

Recent literature also emphasizes that the existence of a Rashomon set does not by itself guarantee desirable outcomes. A central fairness result is that one should be intentional about selection within the set: random choice among equally accurate models is not equivalent to choosing the fairest model. In a broad combinatorial analysis of binary classifiers, the Rashomon set can grow exponentially with dataset size, individual predictions can flip with probabilities that depend on PRESERVED_PLACEHOLDER_^^^^7ti:\7 OR all:\7^^^^, and direct optimization can find much fairer models than uniform random sampling from the set (&&&^^^^7 machine learning7 machine learning7^^^^&&&).

A sharper critique comes from allocation multiplicity. In scarce-resource settings, the Rashomon set of predictive models may be a poor proxy for the much larger space of equal-utility allocations. In a healthcare case study, the least discriminatory allocation recovered from a sampled Rashomon set remains discriminatory, outcome homogenization persists, and ^^^^7all:\7 OR ti:\7^^^^% to ^^^^7 machine learning7Rashomon set model multiplicity Rashomon effect7^^^^% of qualified individuals are never selected in any recovered allocation, leading to the argument that model multiplicity only matters when it translates into allocation multiplicity (&&&^^^^7 machine learning7ti:\7^^^^&&&). This suggests that some promised fairness and diversity benefits of the Rashomon set depend on the mapping from model scores to decisions, not on model-space multiplicity alone.

A further limit is that multiplicity can be undesirable when the objective is consistency. Predictive multiplicity may create arbitrariness for individuals because equally accurate models can still disagree on their labels. To address this, outlier correction, local patching, and pairwise reconciliation are proposed as mechanisms for reducing disagreement among the members of a Rashomon set before distilling the reconciled behavior into a single interpretable model for deployment (&&&^^^^7 machine learning7 OR ti:\7^^^^&&&).

Taken together, these results establish a set of selection principles. The Rashomon set is valuable because it reveals where performance-equivalent models agree and where they diverge; because it allows fairness, robustness, sparsity, and explanation quality to be optimized under an explicit near-optimality constraint; and because it exposes when single-model interpretation is misleading. At the same time, its practical meaning depends on the hypothesis class, the tolerance parameter, the evaluation metric, the decision rule that maps predictions to actions, and the computational method used to approximate the set. A plausible implication is that the Rashomon set is best understood not as a universal cure for arbitrariness, but as a structured object for making arbitrariness visible and therefore governable.