Learning An Interpretable Risk Scoring System for Maximizing Decision Net Benefit

Abstract: Risk scoring systems are widely used in high-stakes domains to assist decision-making. However, existing approaches often focus on optimizing predictive accuracy or likelihood-based criteria, which may not align with the main goal of maximizing utility. In this paper, we propose a novel risk scoring system that directly optimizes net benefit over a range of decision thresholds. The model is formulated as a sparse integer linear programming problem which enables the construction of a transparent scoring system with integer coefficients, and hence, facilitates interpretation and practical application. We also establish fundamental relationships among net benefit, discrimination, and calibration. Our analysis proves that optimizing net benefit also guarantees conventional performance measures. We thoroughly evaluated our method on multiple public datasets as well as on a real-world clinical dataset. This computational study demonstrated that our interpretable method can effectively achieve high net benefit while maintaining competitive discrimination and calibration performance.

• The paper introduces RSS-DNB, a novel risk scoring framework that uses an MILP formulation to directly optimize decision net benefit rather than traditional accuracy measures.
• It employs sparse, integer-valued coefficients with multiple intercepts to ensure both high interpretability and clinical actionability.
• Empirical evaluations demonstrate competitive discrimination, perfect training calibration, and superior sparsity across various benchmark and clinical datasets.

Learning an Interpretable Risk Scoring System for Maximizing Decision Net Benefit

Introduction

The paper "Learning An Interpretable Risk Scoring System for Maximizing Decision Net Benefit" (2604.04241) presents a novel approach for constructing interpretable risk scoring systems directly optimized for clinical or operational utility, as measured by decision net benefit. The proposed framework, labeled RSS-DNB, addresses the limitations of conventional risk scoring methods that primarily optimize predictive accuracy or likelihood-based criteria, which are misaligned with maximizing actionable utility. The core contribution is an integer programming-based methodology for sparse linear scoring systems whose training objective is the weighted area under the net benefit curve (AUNBC) across user-specified thresholds.

Problem Formulation and Methodology

The RSS-DNB model is a sparse linear scoring system with integer-valued coefficients and multiple intercepts, designed to maximize net benefit across a range of decision thresholds. Formally, the training objective is a mixed-integer linear program (MILP) that directly optimizes a weighted sum of net benefits:

$$\min_{\bm{\lambda},\bm{T}} -\frac{1}{N}\sum_{i=0}^M \omega_i\left(\mathrm{TP}_i - \mathrm{FP}_i\cdot\frac{p_i}{1-p_i} \right) + C_0\|\bm{\lambda}\|_0$$

where $\bm{\lambda}$ are feature coefficients, $\bm{T}$ are threshold-specific intercepts, and $\omega_i$ weights net benefit at threshold $p_i$. This integer programming approach ensures transparency, strict interpretability, and suitability for high-stakes domains.

Key distinctions with related interpretable scoring systems, notably SLIM and RISKSLIM, are:

• The optimization target is decision utility (AUNBC), not 0-1 classification (SLIM) or logistic loss (RISKSLIM).
• Multiple intercepts are introduced, enabling piecewise-constant risk estimates aligned with the selected decision thresholds and relevant for clinical actionability.
• The framework accommodates monotonicity and domain constraints, essential for robust deployment in fields like healthcare and criminal justice.

To address the computational complexity inherent in MILP for large-scale datasets, the authors also provide a simulated annealing-based heuristic (RSS-DNB-SA) to efficiently approximate solutions.

Theoretical Results

The paper delivers several theoretical advances relevant to both statistical learning and decision analysis:

• Net Benefit and Discrimination: For any set of thresholds, the AUNBC is shown to be lower-bounded by a monotonic function of AUROC, establishing that maximizing net benefit implies non-trivial discrimination performance. The relationship is formally characterized, with tight bounds elucidated (Theorem 1).
• Net Benefit and Calibration: It is proven that models maximizing net benefit (AUNBC) can always be post-processed to achieve moderate calibration across the specified risk strata, with explicit construction for adjusting predicted risks within intervals to match observed event frequencies (Corollary 1).
• Learning Capacity: Under sufficiently large coefficient bounds, the RSS-DNB integer program can recover or closely approximate the AUNBC of any real-valued linear classifier, guaranteeing no loss of utility due to integer restrictions. Approximate quantization bounds are provided for finite integer sets.
• Generalization Guarantees: The paper supplies finite-sample generalization bounds controlling the empirical-to-population net benefit discrepancy, leveraging the finiteness of parameter sets.

These results collectively demonstrate that direct optimization of utility within the interpretable integer scoring framework does not sacrifice discrimination or calibration and maintains strong generalization behavior.

Empirical Results

Comprehensive empirical evaluation is conducted on eight UCI benchmark datasets and a real-world clinical dataset for preoperative assessment of lung adenocarcinoma invasiveness. The following outcomes are noteworthy:

• Net Benefit Performance: RSS-DNB either matches or outperforms SLIM, RISKSLIM, logistic regression, Lasso, and decision trees in AUNBC across most datasets. The structured clinical case study shows an AUNBC of 0.694 (std=0.038), outperforming logistic regression and Lasso.
• Discrimination: AUROC for RSS-DNB is competitive with logistic regression and Lasso, and strictly better than SLIM in high-complexity datasets.
• Calibration: RSS-DNB achieves perfect calibration (expected calibration error ECE = 0) on training data by construction, and outperforms baselines on test calibration.
• Sparsity and Interpretability: RSS-DNB models are highly sparse (frequently selecting <5 predictors in clinical settings), with all coefficients constrained to small integers. This enforces reliability in manual calculation and interpretability for domain experts.

Notably, combining strong discrimination, calibration, and decision utility with strict sparsity is rare among scoring systems and forms a main empirical argument for the framework.

Practical and Theoretical Implications

The implications of the proposed approach are significant:

• Clinical and High-Stakes Decision Making: Models built with RSS-DNB are naturally interpretable, tractable for manual use, and directly optimize the utility that matters to practitioners. Risk thresholds are linked to explicit decision policies, enhancing transparency.
• Statistical Learning Theory: The theoretical results establish that optimizing utility can subsume performance on standard predictive criteria, a nontrivial result for the design of learning objectives in deployment-critical domains.
• Scalability: While MILP remains computationally challenging, heuristic optimization (simulated annealing) allows practical application on medium-to-large datasets without substantial utility loss.

Limitations and Future Directions

Limitations include the inability of linear integer models to express non-additive variable interactions or complex nonlinearities, and scalability issues for massive datasets under exact optimization. Future developments include:

• Extending the framework to integer-valued nonlinear models or additive scoring systems with monotonic transformation support.
• Algorithmic improvements for even larger learning problems via branch-and-bound enhancements, Lagrangian relaxations, or distributed optimization.
• Adapting utility weighting schemes in AUNBC to reflect practitioner or population-level preferences rather than fixed threshold spacing.

Conclusion

The RSS-DNB framework demonstrates that interpretable risk scoring systems directly optimizing decision net benefit can robustly satisfy clinical and operational requirements for utility, discrimination, calibration, and sparsity. The integration of utility maximization into model construction advances both theoretical and applied practice for risk scoring in high-stakes domains. This work suggests that post-hoc evaluation of decision utility should be replaced by training objectives that explicitly encode actionable benefit, without abdicating the gains of interpretability and statistical generalizability.