Uncertainty Baselines in Deep Learning
- Uncertainty baselines are comparative frameworks that standardize the evaluation of uncertainty quantification, calibration, and robustness in deep learning.
- They encompass various methodological families such as deterministic confidence proxies, sampling-free probabilistic heads, and multi-pass ensemble techniques to balance performance and efficiency.
- They integrate task-specific metrics and domain adaptations to guide decision-making and optimize predictive reliability across diverse applications.
Uncertainty baselines are reference methods, benchmark tasks, and evaluation protocols used to compare uncertainty quantification and robustness techniques under shared experimental conditions. In current usage, the term covers both general benchmark infrastructures—most explicitly the open-source Uncertainty Baselines suite, which spans 19 methods, 9 benchmark tasks, and 83 baseline implementations—and domain-specific baseline sets built around standard uncertainty estimators, calibration procedures, and robustness evaluations (Nado et al., 2021, Bui et al., 2022).
1. Scope and historical role
The benchmark-suite interpretation of uncertainty baselines was formalized by “Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning” (Nado et al., 2021). That project defines a benchmark by a base model or architecture, a training dataset, and an evaluation suite spanning predictive performance, uncertainty quality, and robustness. Its released artifacts include source code, model checkpoints, experiment outputs as Python notebooks, leaderboards, benchmark-specific READMEs, and TensorBoard dashboards, with implementations designed to be self-contained, minimal-dependency, and easy to extend (Nado et al., 2021).
A parallel, domain-specific interpretation appears in later benchmark papers. In self-supervised learning under covariate shift, baseline design centers on ERM, auxiliary-task SSL methods such as Context Prediction, Rotation Prediction, Geometric Transformations Prediction (“Affine”), and Jigsaw Puzzles, and language baselines such as BERT and GPT2, all evaluated with in/out accuracy, NLL, and ECE (Bui et al., 2022). This suggests that “uncertainty baselines” is not restricted to one library or one estimator family; it names a comparative regime in which uncertainty quality becomes a first-class evaluation target.
2. Major baseline families
A useful taxonomy, assembled from the benchmark and task-specific papers, is shown below.
| Family | Representative baselines | Typical role |
|---|---|---|
| Deterministic confidence proxies | max-softmax confidence, predictive entropy, $1-$MSP, maximum heatmap activation, sequence log-probability | cheap uncertainty surrogates |
| Sampling-free calibrated or probabilistic heads | temperature scaling, evidential learning, heteroscedastic Gaussian output heads, DUN | calibration or single-pass uncertainty |
| Multi-pass epistemic baselines | deep ensembles, MC-Dropout, MCBN, bagging, boosting, boosted bagging, E-CPV | stronger model-uncertainty references |
| Task-aware or post hoc add-ons | EaUC, conformal ETO box/ellipsoid/PICNN, PUFR, Naive Selection, Majority Voting | uncertainty for downstream selection or aggregation |
Several papers clarify the technical character of these families. MCBN interprets batch-normalized networks as approximate Bayesian models and turns conventional BN architectures into uncertainty-estimation models without modifying the network or the training procedure; uncertainty is obtained by repeated stochastic forward passes using sampled mini-batch statistics (1802.06455). Depth Uncertainty Networks instead place a distribution over network depth and marginalize over shared-weight subnetworks, yielding model uncertainty with a single forward pass (Antorán et al., 2020). In dense prediction, temperature scaling and evidential learning are prominent sampling-free baselines for panoptic segmentation, while in regression and motion prediction, a heteroscedastic Gaussian output head provides mean and variance predictions with minimal architectural change (Sirohi et al., 2022, Sirohi et al., 2022, Ding et al., 2021, Iversen et al., 2023).
The family of add-on baselines is increasingly important. EaUC is explicitly framed as a general uncertainty-training objective for continuous structured prediction and regression and is best evaluated as “base model + EaUC” versus “base model”, not as a standalone estimator (Kose et al., 2022). In that sense, modern uncertainty baselines often measure the marginal value of an uncertainty mechanism layered on top of an already uncertainty-aware model.
3. Benchmarking protocols and evaluation metrics
Uncertainty baselines are defined as much by their metrics as by their estimators. The general benchmark suite emphasizes predictive accuracy, calibration, selective prediction, OOD and distribution-shift robustness, corruption robustness, and inference latency (Nado et al., 2021). The SSL benchmark under covariate shift makes this explicit by pairing Accuracy, Negative Log-Likelihood, and Expected Calibration Error, with the standard calibration quantity
There, confidence is derived from maximum predictive softmax, and robustness is evaluated on datasets such as MNIST-C, CIFAR-10-C, CIFAR-10.1, and MNLI (Bui et al., 2022).
In continuous structured prediction, uncertainty quality is often operationalized differently. The trajectory-prediction benchmark around EaUC uses Pearson correlation coefficient between uncertainty and prediction error, AUROC for classifying accurate versus inaccurate predictions from uncertainty scores, error retention curves with R-AUC, and F1 retention curves, F1-AUC, and F1@95%. The paper explicitly notes that its notion of reliability is mostly error alignment, not full probabilistic calibration (Kose et al., 2022). That distinction is central: some uncertainty baselines target calibrated predictive distributions, while others target useful ranking of high-risk cases.
Task-specific metrics often embed uncertainty directly into the task objective. In uncertainty-aware panoptic segmentation, confidence is defined as , and the key summary is
which combines segmentation quality with panoptic calibration error (Sirohi et al., 2022). In heatmap-based landmark localization, Quantile Binning evaluates whether predicted uncertainty bins recover true error strata via the Jaccard Index
and whether isotonic-regression-derived error bounds are accurate within each bin (Schobs et al., 2022). These cases show that uncertainty baselines frequently require bespoke metrics rather than a single universal calibration score.
4. Domain-specific repertoires
In dense prediction and segmentation, baseline hierarchies are especially clear. In seismic facies segmentation, deep ensembles are treated as the reference standard for epistemic uncertainty, with MC dropout as a canonical stochastic-forward-pass baseline and softmax entropy and $1-$MSP as the cheapest single-pass alternatives. RADMI is introduced as a single-pass proxy that estimates uncertainty from mutual information between consecutive decoder layers and is evaluated by agreement with deep ensemble uncertainty (Stevens et al., 2 May 2026). In image and LiDAR panoptic segmentation, baseline suites typically include entropy from predicted probabilities, temperature scaling, and evidential learning, all chosen partly because sampling-based methods are too expensive for real-time deployment (Sirohi et al., 2022, Sirohi et al., 2022).
Autoregressive structured prediction uses a different baseline vocabulary. For token-level uncertainty, the core ensemble-based quantities are predictive entropy, expected entropy, and mutual information; for sequence-level uncertainty, the paper emphasizes sequence log-probability, length-normalized log-probability, and averages of token uncertainty along the decoded path (Malinin et al., 2020). In landmark localization, the main baselines are Single Maximum Heatmap Activation (S-MHA), Ensemble Coordinate Prediction Variance (E-CPV), and the hybrid Ensemble Maximum Heatmap Activation (E-MHA), with Quantile Binning converting their continuous uncertainty scores into actionable review strata (Schobs et al., 2022). In human motion prediction and generic heteroscedastic regression, the corresponding baseline is a Gaussian output head that predicts both mean and variance, usually trained with Gaussian NLL and sometimes coupled to uncertainty-guided weighting (Ding et al., 2021, Iversen et al., 2023).
Extreme multi-label classification requires yet another repertoire because of enormous label spaces. There the baseline framework combines ensemble construction methods—Bagging, Boosting, and Boosted Bagging—with label-level uncertainty scores Total Uncertainty (TU), Knowledge Uncertainty (KU), and Probability Variance (PV). To make these tractable, the paper proposes a beam-search approximation with a theoretical guarantee under long-tail XMC predictions, reducing uncertainty estimation from linear in the number of labels to a sub-linear tree-based procedure (Jiang et al., 2022).
5. Decision-aware and system-level baselines
A notable recent shift is that uncertainty baselines are increasingly evaluated by downstream decision quality rather than by prediction alone. In optimization under uncertainty, strong reference baselines are estimate-then-optimize (ETO) models with box, ellipsoidal, or PICNN uncertainty sets, plus Predict-then-Calibrate baselines from Sun et al. and the Johnstone & Cox conformal ellipsoid baseline. These methods are conformally calibrated and provide marginal coverage guarantees, but they remain task-agnostic relative to the paper’s end-to-end alternative (Yeh et al., 2024).
In clinical time series, baseline counterfactual estimators—CRN, CT, G-NET, CF-ODE, and BNCDE—are compared both without uncertainty awareness and with uncertainty-aware treatment optimization. For deterministic models, uncertainty is added by MC dropout, ensembling, or geometric ensembling; for stochastic models, repeated forward passes provide predictive mean and variance. The downstream objective adds a variance penalty weighted by , turning uncertainty into a treatment-selection signal rather than a passive diagnostic (Schwarz et al., 2024).
In ranking, PUFR uses predictive score uncertainty from a last-layer Laplace approximation as a fully post hoc reranking signal, while standard fairness baselines include UNFAIR, ADV, CODER, CVXOPT, and FA*IR (Heuss et al., 2023). In multi-VLM systems, the corresponding system-level uncertainty baselines are Naive Selection and Majority Voting, both adapted from single-model semantic-entropy ideas, whereas SCoOP aggregates model distributions with inverse-entropy weights for hallucination detection and abstention (Yu et al., 25 Mar 2026). These papers make uncertainty baselines explicitly decision-centric: the benchmark question is not only whether uncertainty is calibrated, but whether it improves reranking, abstention, or robust optimization.
6. Strengths, limitations, and current interpretation
No baseline family dominates across tasks. In SSL under covariate shift, Jigsaw Puzzle is the strongest and most stable vision baseline on CIFAR-style benchmarks, Rotation is often strongest for calibration, Context Prediction is a weak baseline, and on MNIST-C several SSL baselines worsen both robustness and calibration relative to ERM (Bui et al., 2022). In segmentation, deep ensembles remain the strongest reference standard, while single-pass proxies such as entropy, $1-$MSP, RADMI, or sampling-free evidential heads trade uncertainty richness for deployability (Stevens et al., 2 May 2026). Under strict inference-cost constraints, DUN is presented as a strong cheap single-network baseline, but it does not replace deep ensembles when maximum uncertainty quality is required (Antorán et al., 2020).
A second limitation is that many uncertainty baselines optimize operational usefulness rather than strict probabilistic calibration. EaUC optimizes uncertainty–error alignment, not interval coverage or calibration-error minimization (Kose et al., 2022). Quantile Binning for landmark localization produces useful ranking and empirical error bounds, but its middle bins are consistently weaker than the extremes (Schobs et al., 2022). In decision-focused robust optimization, more expressive uncertainty sets such as PICNNs can improve in-distribution task loss, yet under temporal distribution shift they can underperform simpler box sets in coverage robustness (Yeh et al., 2024).
A third limitation is task dependence. Directionally ambiguous landmarks favor E-CPV over E-MHA because isotropic heatmap peaks do not capture anisotropic ambiguity well (Schobs et al., 2022). MCBN is attractive only when batch normalization is present and batch sizes are not too small (1802.06455). XMC baselines depend on tree structure and long-tail score distributions (Jiang et al., 2022). Taken together, these results suggest that baseline selection should be conditioned on output structure, computational budget, deployment constraints, and the downstream role of uncertainty rather than on a single generic ranking of methods.
