---
title: Uncertainty Baselines in Deep Learning
url: https://www.emergentmind.com/topics/uncertainty-baselines
type: topic
---

# Uncertainty Baselines in Deep Learning

Uncertainty baselines are reference methods, benchmark tasks, and evaluation protocols used to compare uncertainty quantification and robustness techniques under shared experimental conditions. In current usage, the term covers both general benchmark infrastructures—most explicitly the open-source **Uncertainty Baselines** suite, which spans **19 methods**, **9 benchmark tasks**, and **83 baseline implementations**—and domain-specific baseline sets built around standard uncertainty estimators, calibration procedures, and robustness evaluations [2106.04015][2212.12411].

## 1. Scope and historical role

The benchmark-suite interpretation of uncertainty baselines was formalized by **“Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning”** [2106.04015]. That project defines a benchmark by a base model or architecture, a training dataset, and an evaluation suite spanning predictive performance, uncertainty quality, and robustness. Its released artifacts include source code, model checkpoints, experiment outputs as Python notebooks, leaderboards, benchmark-specific READMEs, and TensorBoard dashboards, with implementations designed to be self-contained, minimal-dependency, and easy to extend [2106.04015].

A parallel, domain-specific interpretation appears in later benchmark papers. In self-supervised learning under covariate shift, baseline design centers on **ERM**, auxiliary-task SSL methods such as **Context Prediction**, **Rotation Prediction**, **Geometric Transformations Prediction** (“Affine”), and **Jigsaw Puzzles**, and language baselines such as **BERT** and **GPT2**, all evaluated with in/out accuracy, NLL, and ECE [2212.12411]. This suggests that “uncertainty baselines” is not restricted to one library or one estimator family; it names a comparative regime in which uncertainty quality becomes a first-class evaluation target.

## 2. Major baseline families

A useful taxonomy, assembled from the benchmark and task-specific papers, is shown below.

| Family | Representative baselines | Typical role |
|---|---|---|
| Deterministic confidence proxies | max-softmax confidence, predictive entropy, \(1-\)MSP, maximum heatmap activation, sequence log-probability | cheap uncertainty surrogates |
| Sampling-free calibrated or probabilistic heads | temperature scaling, evidential learning, heteroscedastic Gaussian output heads, DUN | calibration or single-pass uncertainty |
| Multi-pass epistemic baselines | deep ensembles, MC-Dropout, MCBN, bagging, boosting, boosted bagging, E-CPV | stronger model-uncertainty references |
| Task-aware or post hoc add-ons | EaUC, conformal ETO box/ellipsoid/PICNN, PUFR, Naive Selection, Majority Voting | uncertainty for downstream selection or aggregation |

Several papers clarify the technical character of these families. **MCBN** interprets batch-normalized networks as approximate Bayesian models and turns conventional BN architectures into uncertainty-estimation models without modifying the network or the training procedure; uncertainty is obtained by repeated stochastic forward passes using sampled mini-batch statistics [1802.06455]. **Depth Uncertainty Networks** instead place a distribution over network depth and marginalize over shared-weight subnetworks, yielding model uncertainty with a single forward pass [2006.08437]. In dense prediction, **temperature scaling** and **evidential learning** are prominent sampling-free baselines for panoptic segmentation, while in regression and motion prediction, a heteroscedastic Gaussian output head provides mean and variance predictions with minimal architectural change [2206.14554][2210.04472][2107.03575][2312.07252].

The family of add-on baselines is increasingly important. **EaUC** is explicitly framed as a general uncertainty-training objective for continuous structured prediction and regression and is best evaluated as **“base model + EaUC”** versus **“base model”**, not as a standalone estimator [2212.04812]. In that sense, modern uncertainty baselines often measure the marginal value of an uncertainty mechanism layered on top of an already uncertainty-aware model.

## 3. Benchmarking protocols and evaluation metrics

Uncertainty baselines are defined as much by their metrics as by their estimators. The general benchmark suite emphasizes predictive accuracy, calibration, selective prediction, OOD and distribution-shift robustness, corruption robustness, and inference latency [2106.04015]. The SSL benchmark under covariate shift makes this explicit by pairing **Accuracy**, **Negative Log-Likelihood**, and **Expected Calibration Error**, with the standard calibration quantity
$$
\operatorname{ECE} = \sum_{m=1}^{M}\frac{|B_m|}{n}\left|\operatorname{acc}(B_m)-\operatorname{conf}(B_m)\right|.
$$
There, confidence is derived from maximum predictive softmax, and robustness is evaluated on datasets such as **MNIST-C**, **CIFAR-10-C**, **CIFAR-10.1**, and **MNLI** [2212.12411].

In continuous structured prediction, uncertainty quality is often operationalized differently. The trajectory-prediction benchmark around **EaUC** uses **Pearson correlation coefficient \(r\)** between uncertainty and prediction error, **AUROC** for classifying accurate versus inaccurate predictions from uncertainty scores, **error retention curves** with **R-AUC**, and **F1 retention curves**, **F1-AUC**, and **F1@95%**. The paper explicitly notes that its notion of reliability is mostly **error alignment**, not full probabilistic calibration [2212.04812]. That distinction is central: some uncertainty baselines target calibrated predictive distributions, while others target useful ranking of high-risk cases.

Task-specific metrics often embed uncertainty directly into the task objective. In uncertainty-aware panoptic segmentation, confidence is defined as \(1-u_i\), and the key summary is
$$
\mathrm{uPQ} = (1-\mathrm{pECE})\,\mathrm{PQ},
$$
which combines segmentation quality with panoptic calibration error [2206.14554]. In heatmap-based landmark localization, **Quantile Binning** evaluates whether predicted uncertainty bins recover true error strata via the Jaccard Index
$$
J_q(B_q,\widehat B_q)=\frac{|B_q\cap \widehat B_q|}{|B_q\cup \widehat B_q|},
$$
and whether isotonic-regression-derived error bounds are accurate within each bin [2203.02351]. These cases show that uncertainty baselines frequently require bespoke metrics rather than a single universal calibration score.

## 4. Domain-specific repertoires

In dense prediction and segmentation, baseline hierarchies are especially clear. In seismic facies segmentation, **deep ensembles** are treated as the reference standard for epistemic uncertainty, with **MC dropout** as a canonical stochastic-forward-pass baseline and **softmax entropy** and **\(1-\)MSP** as the cheapest single-pass alternatives. **RADMI** is introduced as a single-pass proxy that estimates uncertainty from mutual information between consecutive decoder layers and is evaluated by agreement with deep ensemble uncertainty [2605.01502]. In image and LiDAR panoptic segmentation, baseline suites typically include entropy from predicted probabilities, **temperature scaling**, and **evidential learning**, all chosen partly because sampling-based methods are too expensive for real-time deployment [2206.14554][2210.04472].

Autoregressive structured prediction uses a different baseline vocabulary. For token-level uncertainty, the core ensemble-based quantities are **predictive entropy**, **expected entropy**, and **mutual information**; for sequence-level uncertainty, the paper emphasizes **sequence log-probability**, **length-normalized log-probability**, and averages of token uncertainty along the decoded path [2002.07650]. In landmark localization, the main baselines are **Single Maximum Heatmap Activation (S-MHA)**, **Ensemble Coordinate Prediction Variance (E-CPV)**, and the hybrid **Ensemble Maximum Heatmap Activation (E-MHA)**, with Quantile Binning converting their continuous uncertainty scores into actionable review strata [2203.02351]. In human motion prediction and generic heteroscedastic regression, the corresponding baseline is a Gaussian output head that predicts both mean and variance, usually trained with Gaussian NLL and sometimes coupled to uncertainty-guided weighting [2107.03575][2312.07252].

Extreme multi-label classification requires yet another repertoire because of enormous label spaces. There the baseline framework combines ensemble construction methods—**Bagging**, **Boosting**, and **Boosted Bagging**—with label-level uncertainty scores **Total Uncertainty (TU)**, **Knowledge Uncertainty (KU)**, and **Probability Variance (PV)**. To make these tractable, the paper proposes a beam-search approximation with a theoretical guarantee under long-tail XMC predictions, reducing uncertainty estimation from linear in the number of labels to a sub-linear tree-based procedure [2210.10160].

## 5. Decision-aware and system-level baselines

A notable recent shift is that uncertainty baselines are increasingly evaluated by downstream decision quality rather than by prediction alone. In **optimization under uncertainty**, strong reference baselines are **estimate-then-optimize (ETO)** models with **box**, **ellipsoidal**, or **PICNN** uncertainty sets, plus **Predict-then-Calibrate** baselines from Sun et al. and the **Johnstone & Cox** conformal ellipsoid baseline. These methods are conformally calibrated and provide marginal coverage guarantees, but they remain task-agnostic relative to the paper’s end-to-end alternative [2409.20534].

In **clinical time series**, baseline counterfactual estimators—**CRN**, **CT**, **G-NET**, **CF-ODE**, and **BNCDE**—are compared both without uncertainty awareness and with uncertainty-aware treatment optimization. For deterministic models, uncertainty is added by **MC dropout**, **ensembling**, or **geometric ensembling**; for stochastic models, repeated forward passes provide predictive mean and variance. The downstream objective adds a variance penalty weighted by \(\lambda\), turning uncertainty into a treatment-selection signal rather than a passive diagnostic [2410.08816].

In **ranking**, **PUFR** uses predictive score uncertainty from a **last-layer Laplace approximation** as a fully post hoc reranking signal, while standard fairness baselines include **UNFAIR**, **ADV**, **CODER**, **CVXOPT**, and **FA\*IR** [2309.09833]. In **multi-VLM systems**, the corresponding system-level uncertainty baselines are **Naive Selection** and **Majority Voting**, both adapted from single-model semantic-entropy ideas, whereas **SCoOP** aggregates model distributions with inverse-entropy weights for hallucination detection and abstention [2603.23853]. These papers make uncertainty baselines explicitly decision-centric: the benchmark question is not only whether uncertainty is calibrated, but whether it improves reranking, abstention, or robust optimization.

## 6. Strengths, limitations, and current interpretation

No baseline family dominates across tasks. In SSL under covariate shift, **Jigsaw Puzzle** is the strongest and most stable vision baseline on CIFAR-style benchmarks, **Rotation** is often strongest for calibration, **Context Prediction** is a weak baseline, and on **MNIST-C** several SSL baselines worsen both robustness and calibration relative to ERM [2212.12411]. In segmentation, **deep ensembles** remain the strongest reference standard, while single-pass proxies such as entropy, \(1-\)MSP, RADMI, or sampling-free evidential heads trade uncertainty richness for deployability [2605.01502]. Under strict inference-cost constraints, **DUN** is presented as a strong cheap single-network baseline, but it does not replace deep ensembles when maximum uncertainty quality is required [2006.08437].

A second limitation is that many uncertainty baselines optimize operational usefulness rather than strict probabilistic calibration. **EaUC** optimizes uncertainty–error alignment, not interval coverage or calibration-error minimization [2212.04812]. Quantile Binning for landmark localization produces useful ranking and empirical error bounds, but its middle bins are consistently weaker than the extremes [2203.02351]. In decision-focused robust optimization, more expressive uncertainty sets such as PICNNs can improve in-distribution task loss, yet under temporal distribution shift they can underperform simpler box sets in coverage robustness [2409.20534].

A third limitation is task dependence. Directionally ambiguous landmarks favor **E-CPV** over **E-MHA** because isotropic heatmap peaks do not capture anisotropic ambiguity well [2203.02351]. MCBN is attractive only when batch normalization is present and batch sizes are not too small [1802.06455]. XMC baselines depend on tree structure and long-tail score distributions [2210.10160]. Taken together, these results suggest that baseline selection should be conditioned on output structure, computational budget, deployment constraints, and the downstream role of uncertainty rather than on a single generic ranking of methods.

Source: https://www.emergentmind.com/topics/uncertainty-baselines