Conformal Decision Theory (CDT)

• CDT is a unified framework that combines conformal prediction with decision theory to provide distribution-free risk control under uncertainty.
• It calibrates decision rules using nonconformity scores and directly adjusts decision parameters for applications in robotics, clinical diagnosis, and multi-agent systems.
• CDT guarantees statistical bounds on decision risk and coverage without strict model assumptions, even in adversarial scenarios.

Conformal Decision Theory (CDT) is a unified mathematical and algorithmic framework that leverages the distribution-free calibration properties of conformal prediction to control downstream decision risk in the presence of imperfect, potentially adversarial predictions. CDT generalizes classical set-valued conformal prediction by focusing explicitly on autonomous or human-in-the-loop decisions, providing statistical guarantees on coverage or risk without strong model or data assumptions. The approach has been instantiated in settings ranging from multi-agent deliberation, optimization under uncertainty, and robotics, to human–AI collaboration and utility-aware clinical decision-making.

1. Problem Formulation and Scope

CDT addresses decision-making problems where predictions from a learned or expert system are subject to uncertainty, misspecification, or adversarial perturbation. The core setup is as follows:

• Let $X$ denote observed features, $Y$ possible outcomes or labels, $x\in X$, $y\in Y$.
• A predictive model produces probabilistic or point predictions, often in the form $\hat p(y|x)$ or a distribution $\pi(y|x)$.
• A decision $a\in \mathcal{A}$ (action or control) is selected based on $x$ and predictive signals, with incurred loss $L(a,y)$ or cost $c(a,y)$ after observing the true outcome $Y$0.
• CDT seeks to control the frequency of high-risk or miscovered decisions, i.e., to guarantee $Y$1 for user-specified $Y$2, with minimal conservativeness.

Statistical guarantees are distribution-free: CDT requires only exchangeability in held-out calibration data in the classical batch setting, and makes no distributional or calibration assumptions on the predictor or the data in online adversarial settings (Lekeufack et al., 2023).

2. Core Methodological Components

Central to CDT are procedures for constructing sets, thresholds, or adjusted decisions based on nonconformity scores, enabling the mapping of predictive uncertainty into actionable decisions or escalation policies.

2.1. Split Conformal Calibration

Given a calibration set $Y$3, conformal methods compute nonconformity scores $Y$4 (e.g., $Y$5 for probability-based classifiers) and calibrate a threshold $Y$6 such that, for any test $Y$7,

$Y$8

Prediction sets or decision rules are then formed by $Y$9 for each $x\in X$0 (Cortes-Gomez et al., 2024, Wang et al., 9 Apr 2026).

2.2. Direct Risk Calibration

CDT extends beyond set prediction by calibrating decision parameters directly. In an online setting, one tunes a control variable $x\in X$1 (e.g., penalty for safety margin violation in robotics), updating it as

$x\in X$2

where $x\in X$3 is the incurred loss and $x\in X$4 is the user-specified risk level. The update provably controls average loss to be at most $x\in X$5, even under adversarial sequences (Lekeufack et al., 2023).

2.3. Utility and Cost Incorporation

CDT incorporates explicit utility (or cost) functions $x\in X$6 on prediction sets (or actions), constructing utility-directed conformal sets that minimize expected loss subject to marginal coverage:

$x\in X$7

Specific algorithms exist for separable and non-separable losses, including penalized ratio methods and greedy set construction (Cortes-Gomez et al., 2024).

3. Decision-Theoretic Guarantees

The principal theoretical claim of CDT is the distribution-free guarantee on calibrated risk or coverage:

• For any decision policy $x\in X$8 constructed via the above conformal procedure,

$x\in X$9

where $y\in Y$0 and $y\in Y$1 are user-specified risk and miscoverage levels, respectively (Lekeufack et al., 2023, Zhou et al., 19 May 2025).

• In optimization settings, risk certificates of the form

$y\in Y$2

are produced for any candidate $y\in Y$3 (Zhou et al., 19 May 2025).

• In multi-agent deliberation, the conformal post-processing layer intercepts high-risk consensus and guarantees that marginal coverage of the true answer among prediction sets remains at least $y\in Y$4, regardless of agent calibration (Wang et al., 9 Apr 2026).

4. Algorithmic Instantiations

CDT has been instantiated in a range of algorithmic pipelines; common characteristics include:

Setting	Nonconformity Score	Action Rule / Output
Multi-agent debate (Wang et al., 9 Apr 2026)	$y\in Y$5	Singleton: act; Larger set: escalate
Optimization & CREDO (Zhou et al., 19 May 2025)	Suboptimality $y\in Y$6	Certify $y\in Y$7
Robotics & manufacturing (Lekeufack et al., 2023)	Realized loss $y\in Y$8	Control $y\in Y$9 to cap average loss
Utility-aware sets (Cortes-Gomez et al., 2024)	Penalized ratio $\hat p(y|x)$0	Set selection minimizing $\hat p(y|x)$1

Algorithmic complexity ranges from $\hat p(y|x)$2 in basic split-conformal and ratio methods to $\hat p(y|x)$3 for CREDO with convex inverse geometry, and up to $\hat p(y|x)$4 for greedy submodular set construction (Cortes-Gomez et al., 2024, Zhou et al., 19 May 2025).

5. Decision Rules and Human–AI Collaboration

Conformal set-valued signals do not specify a unique decision; CDT situates these signals within statistical decision theory:

• If calibrated probabilities are reliably available and there is no unmodeled private expert information, a rational Bayesian can act optimally using posterior $\hat p(y|x)$5.
• For conformal sets, strategies include randomization, uniform selection over set members, risk-averse maximin policies (minimize maximal loss over $\hat p(y|x)$6), or Bayesian updating with miscoverage-weighted priors. CDT provides a taxonomy and recommends empirical elicitation of actual human use (Hullman et al., 12 Mar 2025).

A key observation is that human decision makers with private signals $\hat p(y|x)$7 or bounded rationality may benefit more from conformal set signals, which can support risk-averse or simplified decision policies—especially in high-stakes or uncertain domains.

6. Applications and Empirical Results

Notable instantiations and results include:

• Conformal Social Choice for LLM-based debate intercepts 81.9% of wrong-consensus cases (i.e., cases where consensus is confidently wrong but conformal prediction prevents an erroneous autonomous action) at $\hat p(y|x)$8. Conformal singletons achieve 90.0–96.8% accuracy, with coverage tightly tracking the target ($\hat p(y|x)$9 points) (Wang et al., 9 Apr 2026).
• CREDO Risk Certificates provide coverage-valid risk bounds on suboptimality in high-stakes optimization, outperforming standard predict-then-optimize and robust optimization on both synthetic and real-world tasks. Key metrics include conservativeness (true coverage), true-positive (flagging suboptimality), and relative accuracy of risk estimates (Zhou et al., 19 May 2025).
• Utility-Directed Conformal Prediction delivers 60–75% reductions in empirical decision loss in hierarchical clinical diagnosis tasks while maintaining desired coverage, demonstrating that decision-aware conformal sets yield sharply more actionable and semantically coherent recommendations (Cortes-Gomez et al., 2024).
• Robot Planning and Control experiments show that direct conformal decision calibration enables risk-sensitive but less conservative real-time planning, outperforming classical set-inflation methods on both safety and utility (Lekeufack et al., 2023).

7. Limitations, Open Problems, and Extensions

Several structural challenges and research directions for CDT are identified:

• Marginal vs. Conditional Guarantees: All formal coverage/risk controls are marginal over the data-generating distribution; coverage at each $\pi(y|x)$0 is not guaranteed. Users may misinterpret guarantees as conditional when they are not (a known source of confusion) (Hullman et al., 12 Mar 2025).
• Decision Rule Specification: Conformal prediction outputs are under-specified as decision signals. Their utility depends crucially on the adopted decision rule and the presence of human expertise or extra signals.
• Computational Burden: Richer decision families, non-exchangeable data, or high-dimensional output spaces introduce computational and statistical complexity, calling for scalable approximate algorithms, dimensionality reduction, and policy-specific conformalization (Lekeufack et al., 2023, Zhou et al., 19 May 2025).
• Extensions: Contextual calibration, multistage/sequential decisions, direct incorporation of user utilities, and alignment with expert strategies in human-AI teams are all open areas (Cortes-Gomez et al., 2024, Hullman et al., 12 Mar 2025).
• Integration with Generative Models and Inverse Optimization: When outcome distributions are complex, fitting a conditional generative model and estimating risk via generative sampling and inverse feasible-region geometry is essential for extending CDT to real-world, multimodal settings (Zhou et al., 19 May 2025).

The cumulative effect is that CDT provides a general, theoretically principled, and practically adaptable approach to risk-sensitive decision-making under predictive uncertainty, enabling statistically valid, less conservative, and utility-aware interventions across automated, human-in-the-loop, and hybrid systems.