Model Valuation in Ensembles

• Model Valuation in Ensembles is a framework that quantifies each base model's contribution using cooperative game theory and methods like Shapley values.
• Utility functions based on accuracy, calibration, or uncertainty guide the allocation of credit, optimizing ensemble performance and supporting model pruning.
• Efficient estimation techniques, such as Monte Carlo sampling and mean-field approximations, address the computational challenges of scaling these valuation methods.

Model valuation in ensembles denotes the principled quantification of each base model’s contribution to the predictive performance or uncertainty decomposition of a model ensemble. Modern approaches treat the ensemble as a cooperative system, with individual models regarded as “players” whose interactions determine the quality of aggregate predictions. This paradigm is grounded in cooperative game theory, notably through the use of Shapley values, energy-based variational indices, and probabilistic weighting schemes. Such methodologies permit fair allocation of credit for predictive impact, guide ensemble pruning and construction, and support rigorous uncertainty attribution. They are central to applications spanning classification, probabilistic forecasting, actuarial science, model markets, and interpretability.

1. Cooperative Frameworks for Model Valuation

The foundational abstraction models the ensemble as a cooperative game: each base model is a player, and subsets (coalitions) are scored by a characteristic function reflecting the performance of the subensemble. The canonical example is the Shapley value allocation (Liu et al., 2023, Rozemberczki et al., 2021, Bimonte et al., 27 Oct 2025):

$$\phi_i(v) = \sum_{S \subseteq N\setminus\{i\}} \frac{|S|! (n - |S| - 1)!}{n!} [v(S \cup \{i\}) - v(S)]$$

where $v: 2^N \to \mathbb{R}$ is a utility function (e.g., validation accuracy, negative loss). Efficiency, symmetry, dummy, and additivity axioms guarantee a unique, fair assignment of value to each model (Liu et al., 2023, Bimonte et al., 27 Oct 2025).

Alternative formulations include energy-based valuation via maximum entropy principles (Bian et al., 2021), yielding a sequence of allocations (the “Variational Index”) generalizing classical values under iterative mean-field updates.

2. Utility Functions and Performance Criteria

The definition of the coalition value function $v$ encodes the target of valuation. In classification or regression, typical utility choices include:

• Classification accuracy: $$v(S) = \frac{1}{|V|} \sum_{(x, y) \in V} \mathbf{1}(y = \hat{y}_S(x))$$
• Calibration or diversity-weighted objectives: $$v(S) = \textrm{Accuracy}(S) + \lambda \sum_{i<j \in S} \textrm{Disagreement}(M_i, M_j)$$
• Full predictive scoring (statistical forecasting): utility defined by log-score or continuous ranked probability score (CRPS) over a validation set, as in insurance reserving (Avanzi et al., 2022).

For uncertainty quantification and decomposition, Bayesian nonparametric ensembles construct $v(S)$ as the mutual information or variance contribution attributable to each model or group (Liu et al., 2019).

3. Algorithmic Estimation: Shapley, Mean-Field, and Weight Selection

Exact Shapley computation is #P-hard in the number of models $n$. The dominant paradigm employs stochastic approximations (Liu et al., 2023, Rozemberczki et al., 2021, Bimonte et al., 27 Oct 2025):

• Monte Carlo Sampling: Randomly permute model orderings, compute marginal utility increments, and average across permutations.
• Truncated Monte Carlo (TMC): Early stopping of utility computations if remaining marginal gain is negligible (Liu et al., 2023).
• Troupe (ensemble games): For voting/classification tasks, utilizes weighted voting game structure and expected marginal contributions for $\mathcal{O}(n^2)$ time efficiency (Rozemberczki et al., 2021).

Energy-based mean-field algorithms supply iterative fixed-point updates, with one-step yielding Banzhaf/Shapley values and multi-step yielding the Variational Index, optimizing KL divergence between mean-field and Boltzmann distributions over coalitions (Bian et al., 2021).

For linear-pool density ensembles and ADLP (Accident/Development Linear Pool) (Avanzi et al., 2022), weights are learned by maximizing (or minimizing) a proper scoring rule (e.g., Log-Score, CRPS) under simplex constraints via minorization-maximization.

4. Empirical Protocols and Illustrative Diagnostics

Empirical workflows universally deploy hold-out validation sets and comparison to established baselines. Standard experiments include (Liu et al., 2023, Rozemberczki et al., 2021, Bimonte et al., 27 Oct 2025):

• Pruning curves: Iteratively remove models by ascending Shapley/valuation; plot ensemble accuracy versus number of models—often exhibits a U-shape.
• Acquisition curves: Assemble ensemble from high-value models, showing that compact subensembles achieve near-optimal or improved accuracy.
• Comparison to leave-one-out or random selection: Assess the marginal value versus random, inverse-error, or heuristic weight assignment.
• Sensitivity and risk diagnostics: Remove models with low valuation or perturb (inject noise); adversarial/low-quality models have near-zero or negative Shapley values (Rozemberczki et al., 2021).

A representative table (as in (Liu et al., 2023)):

Model	Shapley Value	Accuracy if Removed
ResNet-50	0.032	85.2%
DenseNet-121	0.029	85.5%
MobileNet-v2	0.024	85.8%
VGG16	0.011	86.1%
AlexNet	–0.002	86.3%

Negative Shapley values indicate detrimental models whose removal improves ensemble performance.

5. Theoretical Properties, Limitations, and Trade-Offs

Shapley and related values satisfy efficiency, symmetry, dummy, and (under addition) linearity. Mean-field Variational Indices retain null-player, symmetry, and marginalism under uniform initialization (Bian et al., 2021). For probabilistic weights, uncertainty and calibration are captured via CRPS-regularized variational inference (Liu et al., 2018).

The primary computational bottleneck is scaling: exponential in exact Shapley, quadratic or better for voting-game approximations, and linear or sublinear in ensemble forecasting with linear pools (Liu et al., 2023, Avanzi et al., 2022, Rozemberczki et al., 2021). Approximation trade-offs balance bias/variance against cost.

Key limitations include:

• Additivity assumption: Individual marginal contributions may not always capture higher-order interactions or synergy in strongly non-linear ensembles (Liu et al., 2023).
• Collinearity: Highly correlated models can dilute individual valuation, necessitating clustering or pre-processing in practice (Bimonte et al., 27 Oct 2025).
• Sample dependence: Reliable valuation requires sufficiently large, independent validation sets; overfitting of surrogate models can compromise the quality (Bimonte et al., 27 Oct 2025).
• Interpretation: Shapley and related indices measure average-case contributions and can mask worst-case or conditional effects critical in safety or regulated domains (Liu et al., 2023).

6. Generalizations: Uncertainty Decomposition and Adaptive/Probabilistic Weights

Model valuation frameworks extend beyond deterministic credit assignment:

• Uncertainty Decomposition: Bayesian nonparametric ensembles (BNE) partition epistemic and aleatoric uncertainty, and attribute MI-based uncertainty shares to each model (Liu et al., 2019).
• Adaptive weighting: Dependent tail-free process priors and variational objectives with CRPS enforce local adaptivity and calibrated, interpretable weights varying in feature space (Liu et al., 2018).
• Distributional Forecasting: In insurance reserving or mortality forecasting, model weights reflect performance on relevant quantiles and can vary by accident, development, or calendar period for maximum predictive accuracy and risk calibration (Avanzi et al., 2022, Bimonte et al., 27 Oct 2025).

7. Practical Implications and Recommendations

The use of Shapley or advanced mean-field indices enables informed pruning, budgeted selection, interpretability, and adaptive model selection. For resource-constrained or high-stakes settings, such methodologies yield minimal subensembles achieving maximal predictive or calibration performance (Liu et al., 2023, Rozemberczki et al., 2021, Bimonte et al., 27 Oct 2025). Model valuation, underpinned by cooperative game theory or maximum entropy variational analysis, is now central for rigorous, interpretable, and fair ensemble learning.

Limitations include computation for large $n$, interpretability in the presence of model dependencies, and challenges in capturing extreme or synergistic interactions. Monte Carlo approximations and proper experimental design (validation splitting, bootstrap analysis) are essential for robust deployment of these methods (Liu et al., 2023, Bimonte et al., 27 Oct 2025). Use of CRPS or log-score as model selection criteria ensures alignment of valuation with forecasting quality (Liu et al., 2018, Avanzi et al., 2022).

For further detail and empirical examples, see "Prompt Valuation Based on Shapley Values" (Liu et al., 2023), "The Shapley Value of Classifiers in Ensemble Games" (Rozemberczki et al., 2021), and "Mortality Models Ensemble via Shapley Value" (Bimonte et al., 27 Oct 2025).