Ensembling Multiple UE Scores: Methods & Insights

• The paper demonstrates that ensembling UE scores using frameworks like SUEL and UEC improves performance metrics, e.g., up to +0.04 PRR and ROC-AUC of around 0.93.
• It employs structured latent modeling, Bayesian aggregation, and probabilistic embedding to combine heterogeneous uncertainty scores and address calibration challenges.
• The work stresses the importance of preprocessing steps such as z-normalization and isotonic regression along with adaptive weighting to manage variable score distributions.

Ensembling Multiple UE Scores refers to the principled combination of multiple uncertainty estimation (UE) scores—derived from different models or methods—into a single, more robust and reliable uncertainty metric. This operation is increasingly critical in LLM agents, uncertainty-aware retrieval, and scientific applications, where no single uncertainty method is consistently optimal across all data regimes or use cases. Ensemble techniques for UE scores span supervised and unsupervised settings, structured latent modeling, and advanced Bayesian aggregation, with substantial empirical and theoretical support.

1. Foundations and Motivation

The integration of multiple UE scores leverages the complementary failure modes, statistical characteristics, and domain-specific advantages of individual uncertainty estimators. Since UE scores often differ widely in their calibration, scale, and sensitivity, their naïve combination is ineffective. Well-founded ensemble approaches address several scenarios:

• Aggregating heterogeneous UE methods (white-box, black-box, probabilistic, deterministic).
• Mitigating distribution shift and robustness issues.
• Achieving near-oracle reliability with limited or absent labeled data. Motivations include empirical findings in LLM hallucination detection, retrieval, and genomics where ensembling yields up to 0.03–0.08 improvements in evaluation metrics such as Prediction Rejection Ratio (PRR) and ROC-AUC over the best single estimator (Bakman et al., 1 Jun 2025, Afshar et al., 2024, Lim et al., 28 Jul 2025). The theoretical basis increasingly centers on Bayesian model combination, latent variable modeling, and calibration-based aggregation.

2. Structured Ensemble Methods: SUEL and Latent Score Modeling

The Structured Unsupervised Ensemble Learning (SUEL) framework formalizes the unsupervised ensembling of continuous UE scores with unknown and possibly correlated performance (Afshar et al., 2024). Each of $M$ predictors produces a standardized score $s_i^{(j)}$ over $N$ samples. The latent-group model posits that predictors fall into $K$ unknown blocks (or groups), with predictors in the same group being linear perturbations of a shared latent variable. This block structure is reflected in the population covariance $R$ of the scores, which is exploited for both group recovery and weight estimation.

Estimation Methods:

• SUEL.CQO (Constrained Quadratic Optimization): Solves for the log-parameters of each predictor/group using an overdetermined linear system constructed from covariance estimates, subject to nonnegativity constraints.
• SUEL.MF (Matrix Factorization): Directly factorizes the sample correlation matrix $R$ as $R \approx B A B' + D$, where $B$ encodes group loadings, $A$ is the group covariance, and $D$ is diagonal idiosyncratic noise. Parameters are optimized in block-coordinate-descent.

Predictor ensemble weights are theoretically derived to maximize separation between hidden classes:

$$w_j \propto \frac{b_j (\mu_{k,1}-\mu_{k,0})}{b_j^2\sigma_k^2 + \tau_j^2}$$

where $j \in C_k$, and $b_j, \mu_{k,1}, \mu_{k,0}, \sigma_k, \tau_j$ are estimated parameters.

Empirically, SUEL.MF attains ROC-AUCs $\approx$0.92–0.93 in simulation and improves rankings in genomics and risk prediction tasks. The identification of latent groups is robust to dependence, and correct specification of $K$ leads to strictly superior ensemble performance versus naively averaging or using the best single score (Afshar et al., 2024).

3. Calibrated and Supervised Ensembles of LLM UE Methods

Supervised and semi-supervised ensembling is systematically studied in the context of LLM uncertainty estimation by Bakman et al. (Bakman et al., 1 Jun 2025). With $K$ UE scores per sample—often with highly variable distributions—they examine preprocessing steps and aggregation strategies:

Preprocessing Options:

• No transformation (raw scores).
• Z-normalization: $s_i'=(s_i-\mu_i)/\sigma_i$, fitted on a calibration set $D_{cal}$.
• Isotonic regression: nonparametric calibration $f_i(s_i)\to[0,1]$ trained per method.

Aggregation Functions:

1. Max: $U_{ens} = \max_i s'_i$
2. Min: $U_{ens} = \min_i s'_i$
3. Mean: $U_{ens} = \frac{1}{K} \sum_i s'_i$
4. Weighted mean: $U_{ens} = \sum_i w_i s'_i$, with $w_i \propto \mathrm{PRR}_i(D_{cal})$
5. Voting: $U_{ens}(x) = \sum_i 1[s'_i(x) > t]$ for a chosen threshold $t$
6. Linear stacker: $g(s'_1, ..., s'_K)$ fitted by logistic/squared loss on $D_{cal}$
7. Decision tree: shallow tree trained on the $K$-dim ensemble features

Linear and weighted-mean combinations, particularly after Z-normalization or isotonic regression, yield robust PRR improvements (up to +0.04 absolute) over the best individual method. Supervised stacking is further beneficial when calibration data ($D_{cal}$) is not too limited. Voting and min/max rules are less stable. Supervised ensembles are robust under distribution shift and can compensate for method-specific weaknesses (Bakman et al., 1 Jun 2025).

4. Uncertainty-Driven Embedding Convolution (UEC): Bayesian Convolution and Adaptive Weighting

"Uncertainty-driven Embedding Convolution" (UEC) (Lim et al., 28 Jul 2025) introduces a fully probabilistic post-hoc transformation of deterministic embeddings, yielding tractable uncertainty-aware ensembles suitable for semantic retrieval, classification, and similarity scoring.

Key Steps:

1. Laplace Approximation: Each deterministic model $f_k$ yields a Gaussian embedding $z_k(x)\sim \mathcal{N}(\mu_k(x), \Sigma_k(x))$, with the mean and (diagonal) covariance computed from the model’s MAP weights and Hessian.
2. Bayes-optimal Weighting: Combine $K$ Gaussian embeddings by computing mixture weights $\pi_k(x)$ minimizing a data-dependent surrogate loss:

$$\min_{\pi\in\Delta} \sum_{k=1}^K \pi_k(x)\bigl(\operatorname{tr}\Sigma_k(x)+\|\mu_k(x)\|^2\bigr)$$

Closed-form: $$\pi_k^* \propto [\operatorname{tr}\Sigma_k(x)+\|\mu_k(x)\|^2]^{-1}$$. For $\ell_2$-normalized $\mu_k(x)$, this reduces to an inverse-variance rule.

1. Uncertainty-aware Similarity: The dot-product similarity between two ensembled Gaussian embeddings is itself approximated as Gaussian, with a probit correction:

$$\hat s = \frac{\mu_s}{\sqrt{1+\frac{\pi}{8}\sigma_s^2}}$$

where $\mu_s = \mu_q^T\mu_c$, $\sigma_s^2$ includes all means/covariances. This correction penalizes large uncertainty during ranking or retrieval.

2. Algorithms: Efficient (pseudocode-defined) procedures exist for offline calibration and online query-time aggregation.

Ablation reveals that the uncertainty-aware convolution (adaptive $\pi_k$) is crucial for achieving performance and robustness gains. The approach yields $O(Kd)$ overhead and maintains efficiency (Lim et al., 28 Jul 2025).

5. Practical Algorithms and Implementation Procedures

The following table summarizes ensemble workflows, data requirements, and calibration needs:

Technique	Requires Labels	Group Structure	Calibration
SUEL.CQO/MF	No	Latent blocks (auto-est.)	Standardize inputs; select K
LLM UE Weighted/Supervised	Yes ($D_{cal}$ small)	None/implicit	Z-norm or isotonic per method; weight/train on $D_{cal}$
UEC	Yes (few, in-domain)	No	Diagonal Laplace approx.; normalize means; tune $\tau, \beta$

Implementation notes:

• Standardization or calibration (Z-norm, isotonic) must precede any linear or non-trivial aggregation to address scale differences among UE scores (Bakman et al., 1 Jun 2025, Afshar et al., 2024).
• Block/group recovery in unsupervised settings utilizes sample covariance eigen-decomposition and quartet-statistics, followed by hierarchical clustering for group assignment (Afshar et al., 2024).
• In probabilistic embedding ensembling, diagonal covariance estimates and adaptive weighting are critical for robust uncertainty propagation (Lim et al., 28 Jul 2025).
• For supervised stacking, avoid complex models (e.g. deep trees) when calibration data is limited to prevent overfitting.

6. Empirical Effects, Guarantees, and Limitations

Ensembling UE scores is empirically observed to provide consistent and sometimes substantial performance improvements.

• In LLM hallucination detection, z-normalized linear combination or stacking of 19 methods yields up to +0.04 PRR gain, robustly across models and tasks (Bakman et al., 1 Jun 2025).
• SUEL.MF in genomics matches "oracle" ensembles in ROC-AUC and outperforms naïve mean or best-single predictors, including scenarios with highly correlated input scores (Afshar et al., 2024).
• UEC achieves improved robustness and accuracy for semantic retrieval, with principled downweighting of uncertain components (Lim et al., 28 Jul 2025).

Theoretical properties:

• Under mild separability and group structure, SUEL identifies the number of latent groups and approximately recovers optimal weights without labels.
• UEC’s ensemble weights correspond to Bayes-optimal solutions under a contrastive surrogate loss.

Limitations include computational overhead scaling with the number of UE methods (though practical unless individual UE computation is costly), dependence of ensemble accuracy on correct group number (SUEL), and limited gains when input methods are highly correlated (Bakman et al., 1 Jun 2025, Afshar et al., 2024, Lim et al., 28 Jul 2025). Careful calibration, group identification, and hyperparameter tuning are critical.

7. Recommendations and Future Directions

• Always preprocess UE scores for scale alignment (z-norm or isotonic regression) before combining (Bakman et al., 1 Jun 2025).
• When labeled data is extremely limited or unavailable, apply structure-exploiting methods such as SUEL with block detection and covariance modeling (Afshar et al., 2024).
• For probabilistic embedding ensembles, compute uncertainty-aware weights via inverse-variance rules and propagate uncertainty in similarity computations (Lim et al., 28 Jul 2025).
• Prefer simple supervised or weighted mean aggregation under small calibration sets; increase complexity only with sufficient data.
• Under distribution shift, ensemble models or stacking offer lower sensitivity to hyperparameters and better transfer. A plausible implication is that further gains may be possible by designing UE methods with intentionally orthogonal error modes, as highly correlated methods offer diminishing returns from ensembling (Bakman et al., 1 Jun 2025). Extension of these frameworks to dynamic, online, or domain-adaptive UE scoring remains an open frontier.