Multi-Stage Classifier with Uncertainty Quantification

• Multi-stage classifiers with uncertainty quantification are advanced systems that decompose prediction into sequential stages while explicitly modeling uncertainty to inform decisions.
• They employ methodologies such as Bayesian inference, Monte Carlo dropout, and evidential deep learning to propagate uncertainty through different stages.
• Practical applications include medical imaging, selective classification, and cost-aware routing, demonstrating improvements in prediction accuracy, calibration, and resource efficiency.

A multi-stage classifier with uncertainty quantification is a predictive system in which classification is decomposed into sequential stages and uncertainty estimates are computed, propagated, or acted upon as part of the decision process. In the cited literature, this design appears in several technically distinct forms: a Bayesian image-analysis pipeline that passes predictive mean and predictive standard deviation from segmentation to detection (Ozdemir et al., 2017); a decoupled “last-layer” formulation in which representation learning and uncertainty estimation are separated (Brosse et al., 2020); a cost-aware cascade that uses uncertainty-gated transitions across stages and model tiers (Xu et al., 2022); a two-stage evidential pipeline that first optimizes cross-entropy and then fine-tunes with an evidential loss (Li et al., 2022); a medical decision cascade that requests additional SPECT MPI features only for uncertain cases (Larsen et al., 2023); a tree-based extreme multi-label classifier with ensemble uncertainty and beam-search approximation (Jiang et al., 2022); and a Bayesian multi-stage treatment framework in which posterior uncertainty over later-stage pseudo-outcomes propagates into earlier decisions (Bi et al., 2024).

1. Scope and canonical problem formulations

The literature does not treat “multi-stage classifier with uncertainty quantification” as a single canonical architecture. Instead, it denotes a family of systems in which either the input modalities, the feature abstractions, the label hierarchy, the decision sequence, or the computational budget are partitioned across stages, while uncertainty is explicitly modeled and used to improve prediction, calibration, triage, or cost control.

A useful taxonomy emerges from the cited work. In one class, an upstream model produces both predictions and uncertainty maps that become downstream features; the pulmonary-nodule system stacks the original CT slice with $\mu_{\mathrm{seg}}$ and $\sigma_{\mathrm{seg}}$ and passes the resulting 3-channel volume to a second-stage detector (Ozdemir et al., 2017). In a second class, the network trunk is trained once and frozen, and uncertainty is concentrated in the final classifier; this is the “last-layer” approach based on features $\phi(x)$ and posterior approximations over the final multinomial logistic regressor (Brosse et al., 2020). In a third class, uncertainty is the control signal for dynamic routing; UnfoldML uses gate functions to decide whether to exit, escalate to a more expensive model, or advance to the next stage under a cost budget (Xu et al., 2022). In a fourth class, the stages correspond to training phases rather than inference modules; TEDL first learns a point estimator by cross-entropy and then fine-tunes for evidential uncertainty (Li et al., 2022). In a fifth class, a stage transition can trigger acquisition of new data modalities; the CRT-response model escalates from clinical variables plus ECG to a larger feature set including SPECT MPI only when first-stage uncertainty is high or the probability is too close to 0.5 (Larsen et al., 2023). Additional variants include top-down tree traversal in extreme multi-label classification (Jiang et al., 2022) and posterior inference over optimal decisions in dynamic treatment regimes (Bi et al., 2024).

Archetype	Stage decomposition	Representative papers
Predict-then-refine	Upstream predictions become downstream inputs	(Ozdemir et al., 2017, Larsen et al., 2023)
Decoupled last layer	Representation learning then uncertainty estimation	(Brosse et al., 2020, Li et al., 2022)
Cost-aware cascade	Cheap-to-expensive routing with gates	(Xu et al., 2022)
Hierarchical classifier	Top-down tree traversal with pruning	(Jiang et al., 2022)
Sequential decision regime	Stagewise optimal actions with posterior uncertainty	(Bi et al., 2024)

This diversity matters because the role of uncertainty differs across designs. It can be a propagated feature, a confidence score for abstention, an escalation trigger, a posterior over labels or actions, a calibration device, or a mechanism for variable relevance assessment.

2. Architectural patterns and stage coupling

The most explicit uncertainty-propagation architecture in the cited set is the two-stage Bayesian CNN for pulmonary nodule detection. Stage 1 is a 2D Bayesian U-Net operating on full axial slices, and Stage 2 is a 3D Bayesian CNN for false-positive reduction on candidate volumes. Stage 1 performs Monte Carlo dropout inference and outputs per-pixel predictive mean $\mu_{\mathrm{seg}}$ and predictive standard deviation $\sigma_{\mathrm{seg}}$. These are stacked with the grayscale CT slice to form a 3-channel composite volume. In parallel, a second 1-channel Bayesian 3D CNN is trained on raw volume data alone, and test-time ensembling uses the convex combination $$y_{\mathrm{ensemble}} = 0.5\cdot y_1^* + 0.5\cdot y_2^*$$ (Ozdemir et al., 2017). The defining architectural claim is that uncertainty is not merely reported; it is transmitted to the next stage as part of the representation.

The “last-layer” line of work implements a different coupling principle. An ordinary deep net is first trained end-to-end by standard cross-entropy plus SGD, all parameters are frozen, and the last-hidden-layer representation $\phi(x)\in\mathbb{R}^d$ becomes the substrate for uncertainty estimation. The second stage fits an ensemble or approximate posterior over the final weight matrix $W\in\mathbb{R}^{K\times d}$ and bias $b\in\mathbb{R}^K$, using snapshot-SGD, SGLD, bootstrapped logistic regressions, or Monte Carlo Dropout on the last layer (Brosse et al., 2020). Here the multi-stage aspect lies in decoupling representation formation from posterior approximation.

UnfoldML uses a two-dimensional cascade. For each stage $s$ there is a “model zoo” of binary classifiers $\sigma_{\mathrm{seg}}$0 ordered by cost and accuracy. Query flow proceeds horizontally across stages and vertically across cost tiers within a stage. The decision policy can exit with “NO,” advance vertically to a more expensive classifier if it does not know, or forward horizontally to the next stage if it confidently predicts “YES” (Xu et al., 2022). This is a multi-stage classifier in the strict control-theoretic sense: stage transitions are part of the predictor.

The CRT-response system is a modality-escalation cascade. “Ensemble 1” uses clinical variables and ECG; “Ensemble 2” uses the same features plus SPECT MPI variables. Every patient is first processed by Ensemble 1; only uncertain or boundary-near cases are routed to Ensemble 2 (Larsen et al., 2023). Tree-based XMC offers yet another stage decomposition: each tree depth is a stage, a top-$\sigma_{\mathrm{seg}}$1 beam is retained, and only those branches are expanded further (Jiang et al., 2022). In the Bayesian treatment-regime framework, the stages are clinical decision times, and later-stage pseudo-outcomes directly enter earlier-stage models through augmentation (Bi et al., 2024).

3. Uncertainty representations and estimation mechanisms

Monte Carlo dropout is the central mechanism in the pulmonary-nodule pipeline. With dropout probability $\sigma_{\mathrm{seg}}$2 after every convolutional layer and $\sigma_{\mathrm{seg}}$3 stochastic forward passes, predictive mean and predictive variance are estimated as

$\sigma_{\mathrm{seg}}$4

In the segmentation net these become per-pixel $\sigma_{\mathrm{seg}}$5 and $\sigma_{\mathrm{seg}}$6, while in the detection nets they become a per-candidate probability $\sigma_{\mathrm{seg}}$7 and uncertainty $\sigma_{\mathrm{seg}}$8 (Ozdemir et al., 2017). The construction is operationally simple: stochasticity induced by dropout yields an approximate posterior predictive distribution.

The last-layer framework uses posterior samples over the final linear classifier to define several confidence scores. Given posterior samples $\sigma_{\mathrm{seg}}$9, one may compute softmax-response, predictive variance, and posterior-vote entropy. The paper formalizes selective classification through coverage and risk, with the area under the risk-coverage curve (AURC) as a summary metric; lower AURC is better (Brosse et al., 2020). In this formulation, uncertainty is not propagated between task modules but is estimated in a reduced parameter space, which the paper motivates by the difference between $\phi(x)$0–$\phi(x)$1 full-network parameters and $\phi(x)$2–$\phi(x)$3 last-layer parameters.

UnfoldML permits two uncertainty channels. The first derives directly from predictive probability, using either $\phi(x)$4 or entropy. The second uses a Dirichlet-based model via Prior Networks or Dirichlet Knowledge Distillation, with a small multi-head MLP outputting $\phi(x)$5 and uncertainty computed from Dirichlet entropy or mutual information (Xu et al., 2022). TEDL is also Dirichlet-based, but its uncertainty originates in evidential deep learning. It defines evidence by $\phi(x)$6, Dirichlet parameters by $\phi(x)$7, and predictive probabilities by $\phi(x)$8 with $\phi(x)$9. It further defines belief masses $\mu_{\mathrm{seg}}$0 and uncertainty mass $\mu_{\mathrm{seg}}$1 (Li et al., 2022).

In the CRT cascade, uncertainty is ensemble disagreement. If $\mu_{\mathrm{seg}}$2 is the $\mu_{\mathrm{seg}}$3th base model’s predicted probability, the aggregated probability is

$\mu_{\mathrm{seg}}$4

and the predictive standard deviation is

$\mu_{\mathrm{seg}}$5

A higher $\mu_{\mathrm{seg}}$6 indicates more disagreement among the base learners (Larsen et al., 2023). In tree-based XMC, uncertainty can be label-level total entropy, knowledge uncertainty via mutual information, or probability variance aggregated to the instance level (Jiang et al., 2022). In the Bayesian treatment-regime model, uncertainty is posterior throughout: posterior-inclusion probabilities for variables, credible intervals for coefficients, and posterior treatment-choice probabilities $\mu_{\mathrm{seg}}$7 for each patient and stage (Bi et al., 2024).

4. Decision rules, routing, and uncertainty propagation

A defining feature of multi-stage classifiers is that uncertainty can alter the computation path. In the pulmonary-nodule system, uncertainty propagation is literal feature propagation: the stage-1 uncertainty map $\mu_{\mathrm{seg}}$8 is concatenated with the raw image and the predictive mean to form $\mu_{\mathrm{seg}}$9. Candidate generation thresholds $\sigma_{\mathrm{seg}}$0 at 0.5, groups connected components into 3D candidate boxes, and then evaluates both the fused and raw-image detectors. Ensembling produces

$\sigma_{\mathrm{seg}}$1

with $\sigma_{\mathrm{seg}}$2 (Ozdemir et al., 2017). The downstream model therefore receives both epistemic information from the upstream stage and complementary information from a raw-image branch.

UnfoldML formalizes routing as constrained optimization. It seeks to minimize end-to-end loss subject to a total cost budget $\sigma_{\mathrm{seg}}$3, with stage-specific transitions governed by confidence score $\sigma_{\mathrm{seg}}$4 and gate functions for “IDK” and “ICK$\sigma_{\mathrm{seg}}$5.” Hard-gating allocates a budget $\sigma_{\mathrm{seg}}$6 per stage and searches thresholds by grid; soft-gating uses $\sigma_{\mathrm{seg}}$7 and optimizes a negative-log-likelihood plus cost-penalty objective (Xu et al., 2022). The architecture turns uncertainty into a control variable rather than a passive statistic.

The CRT model uses simpler but highly interpretable rules. After computing $\sigma_{\mathrm{seg}}$8 and $\sigma_{\mathrm{seg}}$9 from Ensemble 1, it escalates to Stage 2 if either $$y_{\mathrm{ensemble}} = 0.5\cdot y_1^* + 0.5\cdot y_2^*$$0 or $$y_{\mathrm{ensemble}} = 0.5\cdot y_1^* + 0.5\cdot y_2^*$$1 (Larsen et al., 2023). This combines disagreement-based uncertainty and boundary proximity. The threshold pair $$y_{\mathrm{ensemble}} = 0.5\cdot y_1^* + 0.5\cdot y_2^*$$2 is tuned by nested cross-validation using a “scaled-weighted AUC” objective that trades off the fraction of patients kept in Stage 1 against the ROC-AUC of the cascade.

Selective classification in the last-layer framework is another routing mechanism. A threshold $$y_{\mathrm{ensemble}} = 0.5\cdot y_1^* + 0.5\cdot y_2^*$$3 on the confidence score $$y_{\mathrm{ensemble}} = 0.5\cdot y_1^* + 0.5\cdot y_2^*$$4 determines whether a prediction is accepted; sweeping $$y_{\mathrm{ensemble}} = 0.5\cdot y_1^* + 0.5\cdot y_2^*$$5 yields a risk-coverage curve (Brosse et al., 2020). In tree-based XMC, the beam itself is the routing policy: only top-$$y_{\mathrm{ensemble}} = 0.5\cdot y_1^* + 0.5\cdot y_2^*$$6 clusters are retained at each depth, and uncertainty over omitted labels is approximated under a long-tail assumption (Jiang et al., 2022). In the Bayesian treatment-regime setting, propagation occurs through the posterior itself: stage-2 pseudo-outcomes feed into stage 1, so posterior uncertainty in $$y_{\mathrm{ensemble}} = 0.5\cdot y_1^* + 0.5\cdot y_2^*$$7 and $$y_{\mathrm{ensemble}} = 0.5\cdot y_1^* + 0.5\cdot y_2^*$$8 propagates into the marginal posterior of $$y_{\mathrm{ensemble}} = 0.5\cdot y_1^* + 0.5\cdot y_2^*$$9 and $\phi(x)\in\mathbb{R}^d$0 (Bi et al., 2024).

5. Empirical behavior across application domains

The pulmonary-nodule study provides a compact demonstration of the advantages of uncertainty propagation in a heavily imbalanced detection task. On LUNA16 low-dose CT scans with 888 patients and 1186 nodules in 601 patients, stage 1 alone achieved approximately 92% recall and 0.9% precision. On the final test set, the non-Bayesian 1-channel 3D CNN obtained ROC AUC $\phi(x)\in\mathbb{R}^d$1, PR AUC $\phi(x)\in\mathbb{R}^d$2, and Brier Score 0.1045. The Bayesian 1-channel model obtained ROC AUC $\phi(x)\in\mathbb{R}^d$3, PR AUC $\phi(x)\in\mathbb{R}^d$4, Avg. $\phi(x)\in\mathbb{R}^d$5 0.31, and Brier Score 0.1214. The Bayesian fusion model obtained ROC AUC $\phi(x)\in\mathbb{R}^d$6, PR AUC $\phi(x)\in\mathbb{R}^d$7, Avg. $\phi(x)\in\mathbb{R}^d$8 0.27, and Brier Score 0.1010. The $\phi(x)\in\mathbb{R}^d$9 ensemble obtained ROC AUC $W\in\mathbb{R}^{K\times d}$0, PR AUC $W\in\mathbb{R}^{K\times d}$1, Avg. $W\in\mathbb{R}^{K\times d}$2 0.28, and Brier Score 0.0948. The paper also reports Spearman rank correlation $W\in\mathbb{R}^{K\times d}$3 with $W\in\mathbb{R}^{K\times d}$4 between the two Bayesian CNNs, indicating complementary errors (Ozdemir et al., 2017).

The last-layer study evaluates selective classification and OOD detection on MNIST, CIFAR-10, CIFAR-100, and ImageNet. On CIFAR-100 and ImageNet, last-layer ensembles substantially outperform a point-estimate SGD baseline, and on ImageNet the best last-layer method reduces AURC by approximately 10–12% relative to SGD-PE. For held-out-class OOD detection on CIFAR-100, AUROC increases from approximately 0.72 for SGD-PE to approximately 0.73–0.75 for last-layer methods (Brosse et al., 2020). The paper’s practical conclusion is that simple last-layer uncertainty methods can capture much of the benefit of more expensive full-network Bayesian approximations.

UnfoldML reports a cost-sensitive accuracy trade-off in both clinical prediction and hierarchical image classification. On the Sepsis $W\in\mathbb{R}^{K\times d}$5 Septic Shock task from MIMIC-III, the multi-class LSTM-A baseline has AUC 88.9%, Cost 269, and EarlyShock 24.0 hr, whereas UnfoldML-S (Rec.) has AUC 88.8%, Cost 13.7, and EarlyShock 26.1 hr. In the single-stage analysis, UnfoldML-S achieves Sepsis AUC 91.1% at Cost 8.3 and Shock AUC 96.2% at Cost 10.0, compared with 92.8% and 96.9% respectively for the best LSTM-A at Cost 268. On CIFAR-100 hierarchical image classification, UnfoldML-S obtains People 97.1% at $W\in\mathbb{R}^{K\times d}$6 MACs and Vehicles 98.9% at $W\in\mathbb{R}^{K\times d}$7 MACs, compared with baseline values 98.1% at $W\in\mathbb{R}^{K\times d}$8 MACs and 99.3% at $W\in\mathbb{R}^{K\times d}$9 MACs (Xu et al., 2022).

The CRT-response cascade shows the clinical value of uncertainty-triggered modality acquisition. Across 10-fold stratified cross-validation, Ensemble 1 achieved AUC $b\in\mathbb{R}^K$0, Accuracy $b\in\mathbb{R}^K$1, Sensitivity $b\in\mathbb{R}^K$2, and Specificity $b\in\mathbb{R}^K$3; Ensemble 2 achieved AUC $b\in\mathbb{R}^K$4, Accuracy $b\in\mathbb{R}^K$5, Sensitivity $b\in\mathbb{R}^K$6, and Specificity $b\in\mathbb{R}^K$7; the multi-stage model achieved AUC $b\in\mathbb{R}^K$8, Accuracy $b\in\mathbb{R}^K$9, Sensitivity $s$0, and Specificity $s$1 while requiring SPECT MPI data for only 52.7% of patients (Larsen et al., 2023).

TEDL reports that single-stage EDL has AUC below cross-entropy by 0.5–1.5% across data sizes, whereas TEDL recovers nearly full cross-entropy AUC and outperforms EDL by up to 1.2%. The XMC study reports predictive gains of approximately 0.5–1.5 points over single models, 2–5 AUROC-point gains in misclassification detection, up to 3 AUROC-point gains in OOD detection, and 30–110$s$2 or 40–60$s$3 speedups from beam-search approximation, depending on the task (Li et al., 2022, Jiang et al., 2022).

6. Limitations, controversies, and research directions

A recurrent issue is that not every additional uncertainty layer improves decision quality. The last-layer study explicitly states that adding 2–3 dropout or bootstrap layers on top of a pretrained network yields at most approximately 5% marginal AURC improvement, often none, and argues that once $s$4 is a rich, high-level representation, most uncertainty lives in the last logits (Brosse et al., 2020). This directly qualifies a common assumption that deeper uncertainty modeling is automatically better.

Another controversy concerns evidential training stability. The TEDL paper reports that EDL tends to yield inferior AUC compared with models learned by cross-entropy loss and is highly sensitive in training. Its diagnosis is the Dying ReLU issue in evidential outputs: with ReLU, negative logits can produce zero evidence, leading to $s$5 for all $s$6, constant uniform belief, and no learning signal. TEDL mitigates this by replacing ReLU with ELU and by performing stage-1 cross-entropy pre-training before stage-2 evidential fine-tuning (Li et al., 2022).

The pulmonary-nodule work highlights a subtler trade-off: the fusion network lowers average model uncertainty and improves PR AUC relative to a 1-channel Bayesian detector, but the ensemble trades slightly higher uncertainty for substantially higher overall accuracy and the lowest Brier score (Ozdemir et al., 2017). This suggests that the relation between “more confidence” and “better performance” is not monotone; calibration, complementarity, and error correlation matter. The XMC study similarly makes approximation explicit: beam-search uncertainty estimation incurs only 1–3 points of AUROC loss, but this is the price of 30–110$s$7 or 40–60$s$8 speedups (Jiang et al., 2022).

Several forward directions are already explicit in the cited papers. The pulmonary-nodule study proposes integrating ensembling inside one network, using adaptive ensembling based on per-candidate $s$9, and comparing MC dropout with variational dropout or deep ensembles (Ozdemir et al., 2017). UnfoldML proposes adaptive gating and budget-aware stage design under real deployment costs (Xu et al., 2022). The Bayesian treatment-regime paper suggests a broader extension in which uncertainty quantification is tied not only to class probabilities but also to posterior-inclusion probabilities and patient-specific treatment-choice probabilities, thereby unifying prediction, explanation, and sequential decision support (Bi et al., 2024). A plausible implication is that future multi-stage classifiers will increasingly combine stagewise routing, posterior uncertainty, and selective feature or modality acquisition within a single decision-theoretic framework.