Triage Score: A Counterfactual Risk Assessment Instrument

Abstract: Risk assessment instruments, also known as "risk scores," are widely used in high-stakes decision-making settings such as medicine and the criminal justice system. A risk score predicts the likelihood of an undesired outcome if no intervention is made. Thus, a sufficiently high score is often interpreted as a recommendation to intervene. However, risk scores fail to account for what would happen if a decision-maker does intervene. This failure is problematic because effective decision making requires consideration of both or multiple potential outcomes. We propose "triage scores," which are based on additive counterfactual utilities and include risk scores as a special case. Unlike risk scores, triage scores can incorporate counterfactual outcomes under alternative decisions, enabling decision makers to incorporate a wide range of ethical and practical factors. We illustrate the use of triage scores with an application to our own randomized controlled trial evaluating a pretrial risk score. Our analysis demonstrates that triage scores are able to capture rich utility structures and yield substantively distinct results regarding policy evaluation and learning.

• The paper presents a triage score framework that integrates both realized and counterfactual utilities to better guide intervention policies.
• It utilizes semiparametric estimation methods, including augmented inverse probability weighting and deep neural architecture analyses, validated through a pretrial RCT.
• The framework discriminates among preventive, unnecessary, and counterproductive interventions, facilitating fair and welfare-optimized decision-making.

Formal Essay on "Triage Score: A Counterfactual Risk Assessment Instrument" (2606.21050)

Overview and Motivation

The paper introduces the concept of "triage scores," a generalization of conventional risk scores for high-stakes decision-making scenarios, with a specific focus on the pretrial criminal justice context. Traditional risk scores produce a predictive estimate of an adverse outcome (e.g., rearrest, illness) under a fixed baseline action—typically, the null or no-intervention condition. These scores have become central in domains such as criminal justice and medicine, where a high risk score often triggers intervention. However, their structural one-sidedness precludes any direct reasoning about counterfactual outcomes: what would transpire under different alternative actions? This limitation results in policy misalignment when the decision-maker's objective is to optimize welfare across available interventions.

The triage score framework is grounded in statistical decision theory and principal stratification, constructed from additive counterfactual utilities rather than solely baseline potential outcomes. This approach allows for explicit modeling of both the realized outcome and the counterfactual outcomes under alternative decisions, thereby supporting richer utility structures that capture ethical and practical considerations such as regret, fairness, and costs of unnecessary interventions.

Conceptual and Mathematical Foundation

The triage score is defined in settings where joint potential outcomes $(Y(0), \ldots, Y(K_D-1))$ under all possible decision alternatives are considered, creating principal strata that index individuals according to the outcome vector under different policies. Instead of assigning utility exclusively to observed outcomes under the chosen action, as in conventional risk scores, the triage score assigns an additive utility composed of both realized and counterfactual utilities across all strata.

Under the standard unconfoundedness assumption and additive counterfactual utility specification, the framework achieves identification of the expected utility for any system of decision rules (human-alone, human+AI recommendation, AI-alone), using semiparametric estimation strategies, specifically augmented inverse probability weighting (AIPW). This extends existing confusion-matrix-based causal evaluation approaches by incorporating joint potential outcomes and regret terms.

Notably, the triage score reduces to the standard risk score when utilities depend only on the baseline potential outcome. This key theoretical property ensures compatibility with legacy systems while enabling substantial generalization.

Empirical Application and Numerical Results

The primary empirical study is centered on a randomized controlled trial (RCT) in Utah, where judges deciding pretrial release and bail conditions were either provided the Public Safety Assessment (PSA) or operated without it. The dataset encompasses 9,855 first-arrest cases, with randomization ensuring credible identification. The empirical evaluation leverages structured covariates, three PSA risk scores, and high-dimensional probable-cause affidavits, the latter processed using GenAI-powered inference (GPI) via deep neural architectures for causal adjustment.

The results consistently demonstrate that expected utility estimates and policy evaluations under triage score frameworks diverge substantively from those under conventional risk scores. For example, tuning parameters related to counterfactual regret or cost of cash bail influences the welfare ranking of decision-making systems (human-alone vs. human+PSA vs. optimal learned policies). When regret from unnecessary detention is penalized in the utility, triage scores recommend more lenient actions, as expected. The estimated change in the proportion of cash bail decisions under optimal triage-score-based policies shows strong sensitivity to utility specification, often deviating sharply from judge or PSA-based decisions.

The empirical figures robustly highlight that using triage scores allows decision makers to discriminate between individuals for whom an intervention would be preventive, unnecessary, or counterproductive—a discrimination unavailable in standard risk score analytics. The numerical estimates clarify that even with conservative parameter settings, triage scores yield substantively distinct intervention policies and welfare estimates.

Implications for Policy Evaluation and Learning

The triage score framework formalizes welfare optimization in algorithm-assisted human decision-making by allowing utilities to encode regret, fairness, and costs across all principal strata. Identification and inference are possible under standard assumptions, even when outcomes are only observed for selected actions, via semiparametric estimators and sufficient adjustment for high-dimensional confounding features.

From a practical standpoint, triage scores fundamentally alter the policy evaluation landscape. Decision-makers can specify utility functions to align with ethical priorities or societal costs, explicitly penalizing unnecessary interventions or missed prevention opportunities. Empirical policy learning then proceeds via utility maximization over feasible decision classes, reducible to weighted classification in the binary case.

This instrument robustly addresses limits of the selective labels problem and supports nuanced evaluation across various regimes (human-alone, human+AI, AI-alone, and learned decision trees).

Theoretical Contributions and Future Directions

The triage score framework generalizes prior counterfactual risk assessment methods by parameterizing utilities on the joint potential outcome vector, not solely the baseline outcome. Additivity is shown to be both necessary and sufficient for identification, extending statistical decision theory with counterfactual loss. The inference procedures are built on modern semiparametric machine learning techniques, including doubly robust estimation and causal representation learning, with empirical validation against field RCT data.

Future work should address elicitation and specification of utility parameters for real-world deployment, and extend the triage score to dynamic decision settings, where policy updating and sequential carryover effects require temporally dependent modeling and learning.

Conclusion

The triage score paradigm constitutes an advance in counterfactual risk assessment, allowing policy evaluation and learning to proceed via joint potential outcomes, additive counterfactual utilities, and explicit encoding of welfare-driven factors such as regret, fairness, and practical costs. Empirical results in criminal justice show that triage scores yield quantitatively and qualitatively distinct results, thereby facilitating more aligned, ethical, and ultimately efficient decision-making. These developments have direct implications for fair, utility-maximizing AI-assisted decision support across domains where outcome counterfactuals are central.