Prediction Sets for Counterfactual Decisions: Coverage, Optimality, and Conformal Prediction

Abstract: Predictions are increasingly used to guide high-stakes decisions, from treatment selection to policy making. To ensure reliability with imperfect predictions, uncertainty quantification methods such as conformal prediction build prediction sets with coverage guarantees. However, statistical validity alone does not immediately determine the decisions to take, nor the optimality thereof. This gap is especially delicate in counterfactual settings where the outcome that materializes depends on the action taken, so uncertainty cannot be specified independently of the decision rule. We develop a decision-theoretic framework for uncertainty-informed counterfactual decisions. We identify a novel notion of \emph{policy-coupled coverage} -- namely, coverage of the realized outcome under the action induced by the prediction sets themselves -- as the optimal and lossless interface between uncertainty and action. It plays three roles. First, it justifies acting via a natural max-min rule as minimax-optimal under distributional ambiguity. Second, optimizing prediction sets under policy-coupled coverage is equivalent both to a stronger universal-coverage formulation and to the direct risk-averse optimization over policies and utility certificates; this equivalence yields the explicit form of the population-optimal prediction sets. Third, it admits a two-stage procedure, Policy-Coupled Risk-Averse Conformal Prediction (PC-RACP), that approximates these optimal sets with rigorous finite-sample coverage. Simulations and a real email-marketing experiment confirm that PC-RACP delivers higher utility than existing approaches while maintaining valid coverage, and that ignoring the counterfactual structure of the decision problem is suboptimal for both validity and utility.

• The paper introduces policy-coupled coverage (PCC) to ensure prediction sets align with the action-induced outcomes, securing reliable coverage for decision-making.
• It presents a methodology that couples conformal prediction with minimax utility optimization using a Lagrangian calibration procedure to balance coverage and risk.
• Empirical results from the PC-RACP algorithm demonstrate superior performance with tight coverage and higher utility compared to traditional baselines in both synthetic and real-world tests.

Prediction Sets for Counterfactual Decisions: Technical Foundations and Empirical Performance

Introduction and Motivation

The paper "Prediction Sets for Counterfactual Decisions: Coverage, Optimality, and Conformal Prediction" (2607.02206) develops a comprehensive framework for uncertainty quantification in counterfactual decision-making scenarios. Classical conformal prediction provides coverage guarantees for predictive inference, but does not directly address the challenges presented when the outcome of interest depends on the action taken—a setting intrinsic to causal inference, contextual bandits, and individualized decision-making. The main technical contribution is the introduction and characterization of policy-coupled coverage (PCC): a notion of statistical validity that is coupled to the policy induced by the prediction sets themselves, ensuring that the realized outcome under the selected action is reliably covered.

Policy-Coupled Coverage: The Decision-Theoretic Interface

Standard conformal prediction guarantees (e.g., marginal or per-action coverage) are insufficient for counterfactual decisions, because the prediction-set action selection rule modifies which potential outcome is realized. The paper rigorously establishes that PCC is the optimal interface between uncertainty quantification and downstream counterfactual action selection.

Under PCC, prediction sets $\{C(x,a)\}_{a\in\mathcal{A}}$ are constructed so that, when the agent selects action $a^*(x)$ according to a max--min (risk-averse) utility rule

$$a^*(x) = \arg\max_{a\in\mathcal{A}} \inf_{y\in C(x,a)}u(a,y),$$

the coverage guarantee holds for the realized outcome $Y(a^*(x))$. The upshot, formalized in Theorem 1, is that the set-induced action rule is minimax-optimal over the ambiguity class of distributions consistent with the PCC constraint.

Figure 1: The deployment pipeline for PC-RACP, where action-indexed prediction sets are constructed with policy-coupled coverage, supporting risk-averse counterfactual maximization.

This result establishes that valid uncertainty quantification in counterfactuals cannot be decoupled from decision-making; the sets must be constructed to cover the realized outcome under the action chosen as a function of those very sets.

Equivalence, Optimality, and Explicit Characterization

The paper further demonstrates that optimizing prediction sets under PCC is equivalent—both in statistical power and attainable utility—to directly optimizing risk-averse policies and their associated utility certificates, and to universal coverage over all policies. This corresponds to the lossless alignment of uncertainty quantification and utility optimization: prediction sets that satisfy PCC are not only sufficient for risk-averse policy optimization but also necessary for achieving minimax guarantees.

Figure 2: Theoretical organization of the framework; user-facing sets induce the max--min policy, which is justified as worst-case optimal over distributions compatible with PCC.

Population-optimal sets are explicitly characterized by a Lagrangian-based procedure. For each $(x,a)$, the set $C^*(x,a)$ includes all $y$ such that $u(a,y)$ exceeds a policy- and coverage-dependent threshold derived from conditional quantiles. The explicit form follows by solving a convex problem that balances marginal coverage constraints with risk-averse utility maximization.

Algorithm: Policy-Coupled Risk-Averse Conformal Prediction (PC-RACP)

The practical algorithmic innovation is PC-RACP, a two-stage, weighted conformal prediction pipeline that achieves finite-sample PCC under logged (counterfactual) data. The key is a separation between learning the policy structure (which action is selected under which scenarios) and calibrating the prediction sets to ensure valid coverage along the induced action pathway.

• Stage 1: Model training and optimal policy learning. Predictive models estimate potential outcome distributions and behavior-policy probabilities. A policy–coverage assignment rule is selected via Lagrangian calibration.
• Stage 2: Conformal calibration. Weighted conformal scores are assigned using importance weighting to account for covariate shift, and sets are calibrated to guarantee coverage coupled to the learned policy.

This construction ensures that (even in finite samples and under covariate shift) the coverage guarantee applies to the realized outcome under the action that is itself a function of the prediction sets.

Simulation Experiments

Simulation studies compare PC-RACP to two baselines: (i) RAC, which applies marginal conformal prediction methods that ignore the counterfactual structure, and (ii) Plug-in, which uses an outcome model without conformal calibration. The experiments encompass well-specified and model-misspecified conditions using multinomial and random forest classifiers.

Figure 3: Empirical coverage (multinomial model) versus target miscoverage $\alpha$; PC-RACP matches the nominal $1-\alpha$ line, outperforming baselines.

Figure 4: Utility certificate as a function of $a^*(x)$0; PC-RACP produces sharper, higher-certainty lower bounds on utility compared to RAC and Plug-in.

PC-RACP consistently achieves:

• Nominal coverage: Empirical coverage tightly matches the $a^*(x)$1 nominal level across all $a^*(x)$2, while RAC and Plug-in under-cover.
• Maximized utility: The lower bounds on realized utility attained by PC-RACP are higher, reflecting less conservativeness without loss of validity.

Real-World Data: Hillstrom E-Mail Experiment

PC-RACP is further validated on the Hillstrom email marketing dataset configured as a randomized experiment. Here, the utility function encodes business objectives: sending emails to users who convert, avoiding both missed opportunities and wasted interventions.

Figure 5: Composition of prediction sets among test points with selected action $a^*(x)$3; PC-RACP produces sharper (singleton) sets more frequently than RAC.

Figure 6: For small $a^*(x)$4, PC-RACP leads to a significantly lower fraction of ambiguous sets $a^*(x)$5, improving policy sharpness and utility.

In these experiments:

• Empirical coverage is tight for PC-RACP and overly conservative for RAC.
• The average utility certificate from PC-RACP is uniformly higher across all $a^*(x)$6.
• The improved performance is traced to (i) more aggressive selection of singleton prediction sets when "no email" is optimal, and (ii) more frequent optimal targeting in higher-risk regimes.

Implications and Outlook

The results unambiguously establish that ignoring counterfactual structure leads to both excess conservativeness (wasted utility) and statistical invalidity (under-coverage or over-coverage on realized outcomes). The developed framework generalizes conformal prediction to the counterfactual domain by coupling statistical guarantees to downstream action selection mechanisms.

The theoretical implications are profound: policy-coupled coverage is strictly weaker than universal policy coverage (i.e., it's only required along the induced action pathway), but this is exactly sufficient for minimax rationality—strengthening is superfluous, and weakening forfeits optimality.

Practically, the pipeline is well-suited for any setting where decisions must be justified by valid uncertainty quantification under distributional ambiguity, including medical personalized treatment selection, online interventions (A/B testing), and automated policy deployment under compliance, auditing, or safety constraints.

Conclusion

This work establishes policy-coupled coverage as the essential notion of validity for uncertainty quantification in counterfactual decision-making. The corresponding population-optimal conformal sets not only maintain finite-sample guarantees for the realized outcome but also constitute the lossless interface to risk-averse decision-making, achieving minimax utility optimization. The proposed PC-RACP algorithm is empirically validated, demonstrating consistently higher realized utility and calibrated error control across synthetic and real-world domains. The introduced paradigm will inform the development of future conformal, decision-aligned uncertainty quantification methods across a range of causal and policy-driven machine learning settings.