Predictive Multiplicity in Supervised Learning

• Predictive multiplicity is a phenomenon in supervised learning where near-optimal models produce differing instance-level predictions, resulting in heightened epistemic uncertainty.
• The methodology employs rigorous metrics such as the ambiguity measure and analysis of Rashomon sets to quantify the impact of slight data perturbations on model behavior.
• Practical approaches like multiplicity-aware active learning and imputation are used to steer predictive ambiguity, reducing uncertainty without sacrificing test accuracy.

Predictive multiplicity is a phenomenon in supervised learning wherein multiple models, all achieving comparable aggregate performance (e.g., classification error), can yield discrepant predictions for individual instances. This arbitrariness—whereby any one of several near-optimal models might be selected for deployment—induces significant epistemic uncertainty, especially in high-stakes domains. Recent advances have established theoretical foundations, rigorous metrics, and practical interventions for quantifying and managing predictive multiplicity, with a particular focus on the effects of data perturbations, model class, and data-processing choices (Ganesh et al., 24 Oct 2025).

1. Formal Definitions and Metrics

Predictive multiplicity is characterized within the canonical supervised learning framework. Let $\mathcal{X}$ and $\mathcal{Y}$ denote feature and label spaces, and $\mathcal{D}$ the unknown data distribution on $\mathcal{X} \times \mathcal{Y}$. Models are specified by a parameterized class $\mathcal{H} = \{h_\theta : \theta \in \Theta\}$. Given a dataset $$D_{\mathrm{train}} = (X_{\mathrm{tr}}, Y_{\mathrm{tr}}) \sim \mathcal{D}$$ of size $n$, a model $h_\theta$ is evaluated via the empirical loss

$$L(\theta, D) = \frac{1}{|D|} \sum_{(x, y) \in D} \ell(h_\theta(x), y),$$

e.g., with 0–1 loss $\ell(\hat{y}, y) = \mathbb{I}[\hat{y} \neq y]$.

Rashomon set (Definition 1): For a fixed tolerance $\mathcal{Y}$0 ("Rashomon parameter"),

$\mathcal{Y}$1

This is the set of near-optimal models (those with empirical loss within $\mathcal{Y}$2 of the empirical risk minimizer).

Ambiguity (Definition 2): The canonical metric for predictive multiplicity in classification is ambiguity: $\mathcal{Y}$3 This is the fraction of test points on which any pair of Rashomon models disagree.

k-neighbouring datasets (Definition 3): Two datasets $\mathcal{Y}$4 of size $\mathcal{Y}$5 are $\mathcal{Y}$6-neighbouring if they differ in exactly $\mathcal{Y}$7 data points, i.e., $\mathcal{Y}$8.

2. Neighbouring-Datasets Framework

The framework studies how slight perturbations in the training data impact predictive multiplicity by comparing the Rashomon sets resulting from two $\mathcal{Y}$9-neighbouring datasets ($\mathcal{D}$0 and $\mathcal{D}$1) under a common Rashomon tolerance $\mathcal{D}$2: $\mathcal{D}$3

By evaluating $\mathcal{D}$4 and $\mathcal{D}$5, one quantifies the sensitivity of model-level disagreement to data perturbations, particularly under the granular case $\mathcal{D}$6.

3. Theoretical Result: Inter-Class Overlap Controls Multiplicity

A principal contribution is the demonstration that the direction of the relationship between data overlap and multiplicity is reversed under a fixed Rashomon parameter. Define the overlapping coefficient between class-conditional densities $\mathcal{D}$7: $\mathcal{D}$8 Theorem 4.1 (for $\mathcal{D}$9, 0–1 loss): Under regularity conditions (all good models and Bayes optimals agree that the differing point in $\mathcal{X} \times \mathcal{Y}$0 is "harder" than in $\mathcal{X} \times \mathcal{Y}$1),

$\mathcal{X} \times \mathcal{Y}$2

and

$\mathcal{X} \times \mathcal{Y}$3

Thus, increasing the overlap between classes for a fixed $\mathcal{X} \times \mathcal{Y}$4 shrinks the Rashomon set, thereby reducing predictive multiplicity—contrary to the intuition that overlap makes classification "harder" and hence more diverse solutions arise. This trend is conjectured to extend to general $\mathcal{X} \times \mathcal{Y}$5 ((Ganesh et al., 24 Oct 2025), Conjecture 4.1).

4. Proof Sketch and Mechanism

The proof proceeds via:

• Step 1: The Bayes 0–1 risk is $\mathcal{X} \times \mathcal{Y}$6 for equal priors.
• Step 2: For datasets differing at a single point, a point that's "harder" to classify increases Bayes risk and overlap.
• Step 3: Any model with $\mathcal{X} \times \mathcal{Y}$7 maintains $\mathcal{X} \times \mathcal{Y}$8 (since the single-point loss can only decrease in $\mathcal{X} \times \mathcal{Y}$9), so $\mathcal{H} = \{h_\theta : \theta \in \Theta\}$0; ambiguity is monotone under set inclusion.

For $\mathcal{H} = \{h_\theta : \theta \in \Theta\}$1, an inductive, compositional argument applies by sequentially composing $\mathcal{H} = \{h_\theta : \theta \in \Theta\}$2 single-point transitions.

5. Multiplicity-Aware Extensions: Active Learning and Imputation

5.1 Active Learning

Active learning (AL) sequentially builds labeled datasets $\mathcal{H} = \{h_\theta : \theta \in \Theta\}$3 by selecting $\mathcal{H} = \{h_\theta : \theta \in \Theta\}$4 unlabeled points per round. AL strategies induce $\mathcal{H} = \{h_\theta : \theta \in \Theta\}$5-neighbouring outcomes ($\mathcal{H} = \{h_\theta : \theta \in \Theta\}$6 after $\mathcal{H} = \{h_\theta : \theta \in \Theta\}$7 rounds). Committees of $\mathcal{H} = \{h_\theta : \theta \in \Theta\}$8 models are trained on each AL-labeled set, Rashomon sets constructed, and ambiguity evaluated.

Multiplicity-aware acquisition:

• MultLow: Requests the $\mathcal{H} = \{h_\theta : \theta \in \Theta\}$9 unlabeled points with lowest maximal committee confidence, targeting points where models are least committed, empirically shown to minimize ambiguity.
• MultHigh: Requests $$D_{\mathrm{train}} = (X_{\mathrm{tr}}, Y_{\mathrm{tr}}) \sim \mathcal{D}$$0 with highest minimal committee confidence, empirically maximizing ambiguity.

5.2 Data Imputation

Given a dataset with $$D_{\mathrm{train}} = (X_{\mathrm{tr}}, Y_{\mathrm{tr}}) \sim \mathcal{D}$$1 missing entries, each imputation strategy leads to a possible $$D_{\mathrm{train}} = (X_{\mathrm{tr}}, Y_{\mathrm{tr}}) \sim \mathcal{D}$$2-neighbouring full dataset ($$D_{\mathrm{train}} = (X_{\mathrm{tr}}, Y_{\mathrm{tr}}) \sim \mathcal{D}$$3).

Multiplicity-aware imputation:

• MultLow: For each missing value, chooses the candidate imputation with minimal model confidence (most conservative, lowers ambiguity).
• MultHigh: Uses maximal confidence (increases ambiguity).

6. Empirical Validation and Algorithms

Experiments using ACSIncome, ACSEmployment (Folktables), and Bank Churn datasets confirm the theoretical predictions:

• Active Learning: Across all initializations, numbers of AL rounds, and baseline strategies (random, uncertainty, committee disagreement), the Spearman correlation between class overlap $$D_{\mathrm{train}} = (X_{\mathrm{tr}}, Y_{\mathrm{tr}}) \sim \mathcal{D}$$4 and ambiguity is negative (often $$D_{\mathrm{train}} = (X_{\mathrm{tr}}, Y_{\mathrm{tr}}) \sim \mathcal{D}$$5). MultLow achieves the lowest ambiguity without degrading test accuracy; MultHigh the highest. These trends are robust to label set size and number of rounds, including near full-labeled regimes ($$D_{\mathrm{train}} = (X_{\mathrm{tr}}, Y_{\mathrm{tr}}) \sim \mathcal{D}$$6).
• Imputation: Across missingness fractions $$D_{\mathrm{train}} = (X_{\mathrm{tr}}, Y_{\mathrm{tr}}) \sim \mathcal{D}$$7 (1–25%), ambiguity and its negative correlation with $$D_{\mathrm{train}} = (X_{\mathrm{tr}}, Y_{\mathrm{tr}}) \sim \mathcal{D}$$8 persist. With larger $$D_{\mathrm{train}} = (X_{\mathrm{tr}}, Y_{\mathrm{tr}}) \sim \mathcal{D}$$9, the multiplicity-steering effect of MultLow/MultHigh becomes stronger. MultLow can reduce ambiguity by up to 40% relative to other imputers, at negligible cost in accuracy.

Domain	Method	Ambiguity	Effect	Test Accuracy
Active Learning	MultLow	Lowest	Robust over $n$0	Matched Baseline
	MultHigh	Highest	Robust over $n$1	Matched Baseline
Data Imputation	MultLow	Lowest	Stronger as $n$2	Matched Baseline
	MultHigh	Highest	Stronger as $n$3	Matched Baseline

These computational findings demonstrate practical algorithms for controlling predictive multiplicity through data-centric design (Ganesh et al., 24 Oct 2025).

7. Implications and Outlook

The neighbouring-datasets framework establishes data—and, more precisely, the degree of inter-class overlap—as a principal lever for regulating predictive multiplicity at fixed accuracy. Contrary to earlier heuristics, larger overlap under a fixed $n$4 constraint reduces the Rashomon set and thus narrows the range of permissible per-instance predictions. This theoretical reversal translates directly into actionable procedures for active learning and imputation, allowing practitioners to steer the stability (ambiguity) of model predictions up or down as needed, without sacrificing global performance. This approach reveals an operational path to controlling epistemic uncertainty via data design, with direct consequences for robust, contestable, and fair machine learning systems.