Uncertainty-aware Dynamic Threshold Selection (UDTS)

• UDTS is a framework that dynamically selects decision thresholds by accounting for aleatoric and epistemic uncertainty to improve inference and decision-making.
• It adapts thresholds in scenarios such as extreme value statistics, deep learning, online conformal prediction, and combinatorial optimization, ensuring robust model performance.
• Its methodologies, including Bayesian resampling and online adjustment, provide practical gains in prediction accuracy, coverage, and sample efficiency across varied applications.

Uncertainty-aware Dynamic Threshold Selection (UDTS) is a general mathematical and algorithmic framework that adaptively selects thresholds for decision-making or statistical inference in the presence of aleatoric and epistemic uncertainty. Broadly, UDTS encompasses methods in statistical inference, machine learning, combinatorial optimization, and sequential decision processes where the threshold is not predetermined but is dynamically selected based on estimated uncertainty, aiming to optimize prediction quality, coverage guarantees, robustness, or sample efficiency. Applications span from extreme value analysis and robust classification to semi-supervised learning and real-time autonomous systems.

1. Theoretical Foundations and Core Principles

Central to UDTS is the interdependence between threshold selection and uncertainty quantification. In traditional threshold-based methodologies—such as generalized Pareto (GP) modeling, conformal prediction, or combinatorial set selection—the threshold defines the subset of data or solution space that controls the trade-off between bias and variance, prediction region volume, or inference stability. UDTS extends this by incorporating:

• Formal uncertainty estimation for the predictive modeling or the decision variable (entropy, variance, nonconformity scores, evidence mass, or optimality intervals);
• Dynamic or sequential adjustment of the threshold as a function of observed uncertainty;
• Explicit propagation of threshold uncertainty into downstream predictions.

These principles address the classical dilemma where static thresholds either fail under distributional shift, adversarial perturbations, or class imbalance, or require extensive pre-tuning.

2. Methodologies and Algorithmic Realizations

Multiple methodological instantiations of UDTS exist:

2.1. Bayesian and Bootstrap-Based UDTS for Extremes

In extreme value statistics, UDTS is realized via empirical loss or risk minimization over candidate thresholds, using discrepancy measures that quantify the fit between model and observed quantiles. For instance, the method of "Automated threshold selection and associated inference uncertainty for univariate extremes" (Murphy et al., 2023) minimizes an integrated absolute error metric over grid-searches of possible thresholds and propagates threshold uncertainty via bootstrap resampling to confidence intervals for return levels. Bayesian cross-validation and model-averaging further refine the approach by averaging predictions over thresholds weighted by their posterior predictive risk, as in (Northrop et al., 2015).

2.2. Uncertainty-Aware Thresholds in Deep Learning

In supervised and semi-supervised learning, UDTS forms the basis for selection of pseudo-labels or ensemble members. Examples include:

• Epistemic uncertainty estimation via Monte Carlo dropout, followed by per-class, exponentially-smoothed thresholds for pseudo-label selection in long-tailed datasets (Yang et al., 2024). Selection criteria combine both model confidence and uncertainty, with thresholds dynamically adapted to learning stage and class frequency.
• Dynamic model selection in deep neural network ensembles for adversarial robustness, where sub-models output Dirichlet concentration parameters, uncertainty is quantified using Dirichlet entropy, and the sub-model with minimum uncertainty is greedily selected for prediction. No aggregation or voting is used—selection is hard and uncertainty-driven (Qin et al., 2023).

2.3. Online and Sequential UDTS in Conformal Prediction

Real-time prediction under non-exchangeable or drifting distributions (e.g., in robotics) leverages UDTS via online adaptation of both the nonconformity score function and the quantile threshold. "AdaptNC" (Tumu et al., 2 Feb 2026) combines stochastic approximation of quantile thresholds (DtACI) with a replay buffer for coverage stability and periodic adaptation of score functions to maintain target coverage and shrinkage of prediction region volume.

2.4. Combinatorial Optimization under Unstable Weights

In combinatorial optimization, UDTS formalizes the process of maintaining inclusion and exclusion thresholds for each element under uncertainty intervals. For each element, two thresholds (t_e^–, t_e^+) demarcate where it is always in or always out of all optima, allowing for efficient minimum admissible querying and dynamic adaptation as weights are revealed sequentially (Dürr et al., 2024).

3. Representative Algorithms and Mathematical Statements

UDTS is instantiated with task-specific but structurally similar selection rules. The following table summarizes representative algorithmic patterns:

Task/Domain	Uncertainty Metric	Dynamic Selection Rule
Extreme Value Analysis	Empirical quantile discrepancy (boot.)	Select threshold u* = argmin_u estimated risk
Ensemble DNN	Dirichlet entropy	Select m* = argmin_m H^{(m)}(x); output model m*
Semi-SL	MC Dropout entropy	Select x with u(x) ≤ T_c^t, p_c(x) ≥ T_c^t
Conformal Prediction	Nonconformity score	Tune τ_t online for target coverage, adapt score shape θ_t
Set Selection	Inclusion/exclusion thresholds	Update (t_e^–, t_e^+) as intervals contract/revealings occur

All realizations rely on updating thresholds in response to uncertainty as measured either by explicit predictive entropy, nonconformity statistics, or changes in the value function under parametric uncertainty.

4. Empirical Performance and Application Case Studies

Application-specific empirical findings include:

• In multi-omics decision systems, UDTS enables over 60% of patients to be classified confidently using a single data modality, matching the accuracy of full-modal integration while drastically reducing data acquisition costs (Mu et al., 20 Jun 2025).
• In long-tailed semi-supervised image classification, UDTS yields consistent improvements in accuracy relative to static-threshold FixMatch baselines, with gains of 1–10% absolute depending on dataset and experimental settings (Yang et al., 2024).
• In online conformal prediction for robotic systems, AdaptNC (a UDTS instance) achieves target coverage with region volumes reduced by 30–90% compared to fixed-threshold/score methods, and avoids instability in coverage under sudden environment shifts (Tumu et al., 2 Feb 2026).
• In threshold selection for ocean storm extremes, Bayesian model-averaged UDTS reduces sensitivity to threshold choice and achieves robustness with respect to return-level estimation uncertainty (Northrop et al., 2015).

5. Sensitivity, Uncertainty Propagation, and Limitations

A defining characteristic of UDTS is explicit propagation of threshold uncertainty to downstream statistical inference or decisions. In statistical extremes, this is realized via double bootstrap variance estimation for return-level confidence intervals; in combinatorial optimization, ambiguity about membership in any optimum induces further querying/adaptation. All current frameworks assume access to either an accurate uncertainty measure or efficient computation of inclusion/exclusion thresholds. Challenges exist in highly non-stationary, high-dimensional, or non-convex domains:

• For prediction under dependency or non-iid settings, threshold weights may need continual adaptation and buffer-based recalibration.
• Some optimality threshold problems (e.g., shortest s–t path edge inclusion/exclusion under interval uncertainty) are NP-hard, limiting UDTS tractability in these classes (Dürr et al., 2024).
• Empirical threshold selection remains sensitive to choice of discrepancy measure, smoothing hyperparameters, and window sizes in dynamic regimes.

6. Relations to Broader Research and Extensions

UDTS unifies a class of adaptive selection strategies motivated by uncertainty quantification, providing a rigorous mathematical scaffold applicable across statistics, learning, decision theory, and combinatorial structures. It is closely related to dynamic risk minimization, online learning with partial information, and robust inference under model misspecification. Ongoing research includes extension to:

• Multi-view and multi-modal data with complex fusion rules (e.g., Dempster–Shafer based evidence accumulation (Mu et al., 20 Jun 2025));
• Non-iid and adversarial domains using replay or buffer stabilization (Tumu et al., 2 Feb 2026);
• Integration with structured covariate models and nonparametric density estimation for time-varying thresholding in extremes (Murphy et al., 2023).

A plausible implication is that as predictive modeling transitions to high-stakes applications with distributional uncertainty and heterogeneity, UDTS-like procedures will form the backbone of robust, adaptive, and cost-efficient decision-making frameworks.