Area Under the Performance-Cost Curve (AUC-PCC)

• AUC-PCC is an area-under-curve metric that summarizes expected classifier performance as a function of cost parameters and threshold-choice methods.
• It reveals a linear relationship with ROC AUC, allowing transformations between observed classifier discrimination and expected loss, especially in balanced settings.
• In cost-constrained scenarios (e.g., Partial VOROS), AUC-PCC incorporates precision and capacity limits to provide a normalized performance summary over feasible operating regions.

Searching arXiv for the cited papers to ground the article and verify identifiers. Area Under the Performance-Cost Curve (AUC-PCC) denotes an area-under-curve functional in cost space in which classifier performance is evaluated as a function of an operating-condition or cost parameter rather than solely in ROC space. In "Technical Note: Towards ROC Curves in Cost Space" (Hernández-Orallo et al., 2011), AUC-PCC is the area under the ROC Cost Curve generated by a rate-driven threshold choice method under a uniform distribution of cost proportions, and it is linearly related to ROC AUC. In "Partial VOROS: A Cost-aware Performance Metric for Binary Classifiers with Precision and Capacity Constraints" (Ratigan et al., 21 Oct 2025), AUC-PCC is the area under a performance-cost curve whose ordinate is the normalized partial area of lesser classifiers over a feasible ROC region, and this area is exactly Partial VOROS. This suggests that AUC-PCC is best understood as a family of expected-performance or expected-loss summaries indexed by the choice of cost parameterization, threshold-selection rule, and feasible operating set.

1. Cost-space formulation and operating conditions

For binary classification with class-dependent costs only, the cost-space formulation in (Hernández-Orallo et al., 2011) uses misclassification costs $c_0 \ge 0$ and $c_1 \ge 0$, class proportions $\pi_0$ and $\pi_1$ with $\pi_0+\pi_1=1$, and a threshold $t$. With total cost normalized by $b=c_0+c_1$ and cost proportion $c=c_0/b$, the loss at threshold $t$ and cost proportion $c$ is

$c_1 \ge 0$0

where $c_1 \ge 0$1 is sensitivity and $c_1 \ge 0$2 is false positive rate for class $c_1 \ge 0$3. The same note also uses skew,

$c_1 \ge 0$4

and defines the normalized loss in skew parameterization as

$c_1 \ge 0$5

Balanced classes, $c_1 \ge 0$6, imply $c_1 \ge 0$7, and after setting $c_1 \ge 0$8, $c_1 \ge 0$9 and $\pi_0$0 coincide up to the normalization (Hernández-Orallo et al., 2011).

This formulation places ROC analysis and cost analysis in a common expected-loss framework. Following Adams and Hand (1999), the note defines expected loss over a distribution of operating conditions by

$\pi_0$1

so any AUC-PCC interpretation is necessarily tied to a threshold-choice method $\pi_0$2 and a weighting over operating conditions. A plausible implication is that the phrase “area under the performance-cost curve” is not meaningful without specifying both ingredients.

2. ROC Cost Curves and threshold-choice methodology

Traditional cost curves in (Hernández-Orallo et al., 2011) are defined in cost space by linear “cost lines” for each fixed threshold. In skew space,

$\pi_0$3

with intercept $\pi_0$4 at $\pi_0$5 and slope $\pi_0$6. The lower envelope of these lines gives the optimal cost curve, obtained by the cost-minimizing selection criterion

$\pi_0$7

and equivalently

$\pi_0$8

The distinctive construction introduced in (Hernández-Orallo et al., 2011) is the ROC Cost Curve. The note first defines a rate-fixed threshold choice method,

$\pi_0$9

with

$\pi_1$0

and then specializes it to the rate-driven threshold choice method

$\pi_1$1

Under this rule, the threshold is chosen so that the proportion of positive predictions equals the operating condition. The ROC Cost Curve in cost space is then

$\pi_1$2

For a scored classifier with $\pi_1$3 examples, construction proceeds by sorting examples by score ascending, considering the $\pi_1$4 cutpoints between successive scores and at the extremes, forming the $\pi_1$5 cost lines $\pi_1$6, and then following each line over an interval of width $\pi_1$7 on the $\pi_1$8-axis. The note states: “From left to right, draw the curve following each cost line (from $\pi_1$9 to $\pi_0+\pi_1=1$0) with a width on the x-axis of $\pi_0+\pi_1=1$1.” Each segment corresponds to a point on the ROC curve; segments that coincide with the optimal cost curve correspond to ROCCH points, and other segments correspond to suboptimal thresholds (Hernández-Orallo et al., 2011).

3. Area, expected loss, and the linear relation to ROC AUC

With the ROC Cost Curve threshold rule, the expected loss over uniform cost proportion is

$\pi_0+\pi_1=1$2

When $\pi_0+\pi_1=1$3, this quantity is the area under the ROC Cost Curve. The main result in (Hernández-Orallo et al., 2011) is the exact linear transformation

$\pi_0+\pi_1=1$4

If AUC-PCC is defined as the area under the ROC Cost Curve generated by the rate-driven threshold choice method under uniformly distributed cost proportions, then

$\pi_0+\pi_1=1$5

For balanced classes, $\pi_0+\pi_1=1$6, the skew version becomes

$\pi_0+\pi_1=1$7

This relation makes AUC-PCC an expected-loss summary that is linearly coupled to AUC only under the specific operating-condition distribution and threshold rule above. The inverse mapping is also given: $\pi_0+\pi_1=1$8 and for balanced classes,

$\pi_0+\pi_1=1$9

Several limiting values are reported for balanced classes. For a random classifier with $t$0, expected loss is $t$1. For a perfect classifier with $t$2, expected loss is $t$3. For a worst classifier with $t$4, expected loss is $t$5. The note also states that, in the limit $t$6, the ROC Cost Curve for a random classifier is $t$7, identified as Gini impurity. In the example of Figure 1, $t$8, the empirical area under the ROC Cost Curve is $t$9 for cost proportions, and the theoretical transformation gives $b=c_0+c_1$0 (Hernández-Orallo et al., 2011).

4. Distinction from optimal cost curves and relation to the Brier score

A recurrent point in (Hernández-Orallo et al., 2011) is that the area under the optimal cost curve and the area under the ROC Cost Curve are different quantities. The area under the optimal cost curve is

$b=c_0+c_1$1

and it is computed by assuming perfect knowledge of the operating condition and choosing the cost-minimizing threshold at each $b=c_0+c_1$2. By contrast, AUC-PCC in the ROC Cost Curve sense uses the rate-driven rule $b=c_0+c_1$3, not the cost-minimizing rule. The note states explicitly that the area under the optimal cost curve is not linearly coupled to AUC in general; it depends on the ROC convex hull and reflects cost-minimizing thresholds.

The same note establishes a second connection through the probabilistic threshold choice method

$b=c_0+c_1$4

Under uniform $b=c_0+c_1$5, expected loss becomes

$b=c_0+c_1$6

and this is proved to equal the Brier score,

$b=c_0+c_1$7

If scores are evenly spaced, the probabilistic threshold choice method reduces to the rate-driven method, ROC Cost Curves equal Brier Curves, and the corollary is

$b=c_0+c_1$8

Thus, under evenly-spaced scores, AUC-PCC equals the Brier score exactly. Without the evenly-spaced assumption, the area under the ROC Cost Curve remains linearly related to AUC, but it will not generally equal the Brier score. The note further states that, for perfectly calibrated classifiers, the optimal cost curve may coincide with the Brier curve, but the ROC Cost Curve may still make suboptimal choices at some $b=c_0+c_1$9 (Hernández-Orallo et al., 2011).

5. Partial VOROS as a constrained AUC-PCC

The framework of (Ratigan et al., 21 Oct 2025) introduces a different performance-cost construction for binary classifiers subject to a minimum precision constraint and a capacity constraint. With data $c=c_0/b$0, positives $c=c_0/b$1, negatives $c=c_0/b$2, prevalence $c=c_0/b$3, and a thresholded score function yielding ROC point $c=c_0/b$4, the feasible ROC region is determined by

$c=c_0/b$5

and by the capacity bound

$c=c_0/b$6

The feasible region is therefore the intersection of the unit square with the half-planes $c=c_0/b$7 and $c=c_0/b$8. Under the paper’s practical assumptions, this feasible polygon falls into one of three non-degenerate cases.

Costs are parameterized by unit costs $c=c_0/b$9 and $t$0, cost ratio $t$1, and the fractional cost parameter

$t$2

The normalized cost in ROC space is

$t$3

and iso-cost lines through $t$4 satisfy

$t$5

A feasible classifier $t$6 is lesser than $t$7 at cost parameter $t$8 if $t$9. For a feasible classifier $c$0, $c$1 is the area of the subset of the feasible region consisting of feasible lesser classifiers; geometrically, it is the polygonal area under the iso-cost line within the feasible polygon. The paper proves that $c$2 is cost monotone and is a rational linear function of $c$3 (Ratigan et al., 21 Oct 2025).

After normalization by the feasible-region area $c$4, performance at cost $c$5 is

$c$6

Partial VOROS is then

$c$7

The paper states that AUC-PCC over a cost range $c$8 with uniform weighting $c$9 is

$c_1 \ge 0$00

and therefore Partial VOROS equals the area under the performance-cost curve when performance is defined as the normalized partial area function $c_1 \ge 0$01. If stakeholders specify a distribution over $c_1 \ge 0$02 rather than $c_1 \ge 0$03, the generalized form is

$c_1 \ge 0$04

6. Computation, assumptions, and interpretation

For ROC Cost Curves in (Hernández-Orallo et al., 2011), AUC-PCC can be computed directly from scores and labels by sorting examples by score ascending, evaluating $c_1 \ge 0$05 and $c_1 \ge 0$06 at the $c_1 \ge 0$07 cutpoints, forming the cost lines

$c_1 \ge 0$08

assembling the piecewise linear curve with segment width $c_1 \ge 0$09, and numerically integrating over $c_1 \ge 0$10. Because each segment is linear in $c_1 \ge 0$11, exact segment area is straightforward. The note also gives a more robust route: compute empirical AUC from the ROC curve and then apply

$c_1 \ge 0$12

The exact linear form depends on the uniform operating-condition distribution; different distributions change the coefficients in expected loss and break the exact linear relation. The formula also depends explicitly on class proportions $c_1 \ge 0$13, so interpretation is tied to the evaluation dataset. Ties may lead to fewer unique thresholds, small discrepancies in implementations, and exact agreement only in the continuous limit as $c_1 \ge 0$14 (Hernández-Orallo et al., 2011).

For the constrained framework of (Ratigan et al., 21 Oct 2025), computation starts by restricting ROC points to those feasible under both bounds, computing the convex hull of feasible ROC points, and then, for each $c_1 \ge 0$15, selecting the feasible hull point with minimum $c_1 \ge 0$16. The corresponding polygon area $c_1 \ge 0$17 is obtained from the relevant canonical polygon by the shoelace formula, normalized by $c_1 \ge 0$18, and integrated over $c_1 \ge 0$19 or under a distribution $c_1 \ge 0$20. The paper reports experiments on a MIMIC-IV cohort of ICU patient-stays with train/val/test sizes $c_1 \ge 0$21 and $c_1 \ge 0$22, using logistic regression, MLP, and random forest with 250+ ROC curves. In Scenario 1, $c_1 \ge 0$23, $c_1 \ge 0$24, and $c_1 \ge 0$25, max PV achieved the lowest average test cost $c_1 \ge 0$26, while alternatives were $c_1 \ge 0$27–$c_1 \ge 0$28. In Scenario 2, $c_1 \ge 0$29, $c_1 \ge 0$30, and $c_1 \ge 0$31, most strategies were similar at $c_1 \ge 0$32, while VOROS was worse at $c_1 \ge 0$33. The paper’s interpretation is that PV better ranks and selects operating points under realistic precision and capacity constraints and cost ranges (Ratigan et al., 21 Oct 2025).

Across both constructions, AUC-PCC is not the minimum achievable cost unless the thresholding rule itself is cost minimizing. In the ROC Cost Curve setting it is the expected loss of the rate-driven threshold rule under uniformly distributed operating conditions. In the Partial VOROS setting it is the expected normalized partial area over a feasible ROC region. Common misunderstandings arise when these are conflated with the area under the optimal cost curve, with unconstrained ROC AUC, or with the Brier score outside the evenly-spaced-score condition. The sources instead support a narrower conclusion: AUC-PCC is a cost-space summary whose semantics are determined by the chosen threshold rule, the weighting over cost parameters, and, where applicable, the precision and capacity constraints that define the feasible operating region.