Balanced DRPS for Ordinal Prediction Tasks

• Balanced DRPS is an evaluation metric for ordinal, probabilistic predictions that integrates class imbalance correction and distance sensitivity.
• It measures the divergence between predicted cumulative distributions and observed outcomes using weighted squared differences over ordered classes.
• Its applications span educational testing, medical risk staging, and consumer feedback, ensuring fair and well-calibrated model assessments.

The Balanced Discrete Ranked Probability Score (DRPS) is an evaluation metric designed for probabilistic prediction tasks on ordered discrete outcomes, particularly addressing the requirements of ordinality and class imbalance. It generalizes core ideas from the classical Ranked Probability Score (RPS) and the Continuous Ranked Probability Score (CRPS), adapting them to discrete, imbalanced, multi-class settings prevalent in real-world applications such as question difficulty estimation, medical risk staging, and educational assessment.

1. Formal Definition and Motivation

The Balanced DRPS quantifies the divergence between a predicted cumulative distribution function over ordered categories and the observed outcome, incorporating explicit class imbalance correction. For $N$ samples and $K$ ordered classes, let $F_k(\hat{y}_i)$ denote the predicted cumulative probability up to class $k$ for observation $i$, and $y_i$ the ground-truth label. The class weight for sample $i$ is $w_i = 1 / \left(\sum_{j=1}^{N} \mathds{1}\{y_j = y_i\}\right)$ (inversely proportional to the label frequency). The metric is computed as: $\text{Balanced DRPS}(F, y) = \frac{1}{N}\sum_{i=1}^{N} \sum_{k=1}^{K-1} w_i \left(F_k(\hat{y}_i) - \mathds{1}\{k \geq y_i\}\right)^2$ This formulation extends the vanilla Discrete RPS (where all $w_i = 1$), integrating an importance correction for rare classes, thereby ensuring all levels contribute proportionally to the score.

The Balanced DRPS is designed to:

• Reward models that concentrate probability mass on outcomes near the observed class (distance-sensitivity).
• Respect the intrinsic ordering (“ordinality”) of categories.
• Counteract distorting effects from class imbalance, a critical issue in educational and medical datasets.
• Support full probabilistic as well as deterministic predictions, enabling fair benchmarking across modeling paradigms.

2. Historical and Theoretical Foundations

The DRPS draws its lineage from the Continuous Ranked Probability Score (CRPS), a metric widely used in probabilistic forecasting for continuous outcomes (1902.10173). The discrete version, DRPS, adapts the squared difference between predictive CDFs and realization step functions to ordinal categories, as seen in meteorology and related fields (2106.14345). The “balanced” formulation, introduced for tasks such as question difficulty estimation (2507.00736), explicitly addresses weaknesses of classical metrics in the presence of class frequency skew.

Theoretical results establish DRPS as a proper scoring rule for discrete ordinal outcomes, ensuring that the metric incentivizes truthful probabilistic forecasting. The same mixability and regret analysis techniques developed for CRPS in online learning contexts can be leveraged for DRPS, as mixability arguments extend naturally to discrete scores (1902.10173).

3. Comparison with Alternative Metrics

Balanced DRPS addresses limitations and conceptual gaps of existing metrics for ordinal prediction:

Metric	Ordinality	Class Imbalance	Probabilistic	Limitation
Accuracy	No	No	No	Considers only exact class matches
RMSE (regression)	Partial	No	No	Assumes uniform level spacing
Adjacent Accuracy	Partial	No	No	Threshold-based, arbitrary margin
Brier Score	No	No	Yes	Ignores class ordering
RPS	Yes	No	Yes	Sensitive to distance, not imbalance
Balanced DRPS	Yes	Yes	Yes	Principled for order + imbalance

Balanced DRPS uniquely provides:

• Distance-awareness: Misclassifications “far” from ground truth are penalized more heavily.
• Imbalance-resistance: Scores are not dominated by frequent classes; rare but important cases influence assessment.
• Probabilistic evaluation: Full probability distributions, not just point or mode, are assessed, rewarding well-calibrated uncertainty quantification.
• Cross-paradigm comparability: Enables direct, unbiased comparison of models producing different output types.

Simulation studies in related literature (1908.08980) highlight that non-local, distance-sensitive metrics (like RPS) can sometimes underperform strictly local proper scores (like the Ignorance Score), especially when the practical value of “near-miss” predictions is questionable. This suggests the importance of empirically validating Balanced DRPS relative to alternative proper scoring rules for given application domains.

4. Methodological Implementation

Implementation of Balanced DRPS requires:

• Generation of probabilistic predictions, often as class-wise softmax outputs from neural networks or cumulative probability models such as OrderedLogitNN (2507.00736).
• Calculation of cumulative probabilities up to each class threshold.
• Construction of the outcome indicator as a step function at the observed label.
• Computation of the sample-wise squared distance between predicted and observed CDFs, averaged over relevant thresholds.
• Application of class imbalance weights to each sample, with normalization by sample count.

Balanced DRPS is agnostic to training objectives; it can be computed post hoc for any model, supporting transparency in model selection and reporting. Deterministic predictions can be trivially handled (reducing to mean absolute error for point predictions).

In model evaluation, DRPS supports both calibration and discrimination analysis. Score decomposition techniques for reliability and discrimination, standard in CRPS and RPS literature (2106.14345), extend to DRPS, facilitating diagnostic assessment of both calibration (forecast-observation alignment) and sharpness (ability to resolve different difficulty/outcome levels).

5. Empirical Performance and Findings

Experimental results across question difficulty estimation benchmarks (RACE++, ARC) demonstrate that Balanced DRPS is more diagnostic and fair than conventional metrics, particularly in multi-class, imbalanced settings (2507.00736). Key findings include:

• OrderedLogitNN and other advanced ordinal regression models outperform regression- and classification-oriented baselines when evaluated using Balanced DRPS.
• The metric incentivizes probabilistic models that retain well-calibrated uncertainty, with scores worsening if degenerate (delta-mass) predictions are enforced.
• Baselines such as majority-class or random predictors perform poorly under Balanced DRPS, whereas accuracy-based metrics may disguise this poor performance.
• Models evaluated on Balanced DRPS are not unfairly penalized or favored by overrepresented classes, addressing a principal concern in education, medicine, and other domains with data scarcity at the extremes.

A plausible implication is that widespread adoption of Balanced DRPS can standardize evaluation in ordinal machine learning tasks, aligning metric properties with practical requirements for fairness, granularity, and probabilistic reasoning.

6. Applications and Extensions

Balanced DRPS has immediate applications in any ordinal regression scenario involving rare classes or requiring probabilistic predictions:

• Educational systems: question difficulty calibration, essay grading, placement tests.
• Medical diagnosis: risk or severity scoring across ordinal stages.
• Consumer feedback: star ratings, satisfaction levels, ordinal sentiment analysis.
• Industrial and engineering: defect severity, hazard level prediction.

Potential future directions include adapting Balanced DRPS to complex settings (multi-label ordinal outcomes, hierarchical categories), developing model training procedures that directly optimize DRPS or its decomposed components (e.g., reliability or discrimination), and integrating DRPS-based calibration diagnostics into production system monitoring.

7. Limitations and Ongoing Debates

Literature critiquing the non-locality and distance-sensitivity of RPS (1908.08980) raises important considerations for Balanced DRPS. These critiques emphasize that in some forecasting situations, rewarding proximity in ordinal space may not lead to better identification of high-quality forecasting systems, and strictly local proper rules (such as the Ignorance Score) may outperform RPS and DRPS variants in discriminative efficiency. This suggests the need for empirical validation of Balanced DRPS in each target context, particularly when outcomes are non-repeatable, label distributions are extreme, or the practical value of “close” predictions is ambiguous.

Furthermore, recent advances advocate for corrections to RPS that address artifacts such as linear penalty with distance and preference for symmetry (e.g., squared-absolute RPS) (2309.08701). These findings indicate ongoing evolution of scoring rules for ordinal prediction and motivate continued scrutiny and refinement of Balanced DRPS formulations.

---

Balanced DRPS provides a principled, order-sensitive, and imbalance-robust paradigm for evaluating ordinal, probabilistic predictions in discrete spaces. Its adoption supports fair benchmarking and advances interpretability in machine learning for structured categorical tasks. However, continued theoretical and empirical analysis is warranted to ensure optimality and appropriateness across diverse domains.