Ordinal Classification Systems

• Ordinal classification systems are machine learning methods that handle ordered labels by incorporating model structures and loss functions to penalize errors based on their ordinal distance.
• They use diverse frameworks such as order-aware decision trees, proportional odds models, and deep architectures to better capture the intrinsic ranking of classes.
• Evaluated with metrics like MAE, QWK, and RPS, these systems drive improved performance in applications like clinical grading, customer satisfaction, and credit risk assessment.

Ordinal classification systems are a class of machine learning methodologies specifically designed for tasks where the target labels exhibit a natural order but lack a well-defined numerical distance. This paradigm is fundamentally distinct from nominal classification, which treats all labels as unordered categories, and from regression, which assumes interval or ratio scales. Ordinal classification systems aim both to improve predictive accuracy and to enable more semantically meaningful error analysis in domains where the “magnitude” of misclassification is intrinsically meaningful—such as clinical grading, customer satisfaction, credit risk assessment, and many other real-world applications.

1. Conceptual Foundations and Motivation

Ordinal classification (OC) addresses supervised learning problems with finite, ordered sets of discrete classes—$C_1 \prec C_2 \prec \cdots \prec C_Q$—where neither the true inter-class distances nor a linear mapping can be assumed. OC models are required to encode the ordinal structure so that, for a misclassification, the penalty or cost increases with the "distance" between predicted and true classes. In contrast, nominal approaches such as one-hot classification ignore the ordered context, thereby failing to distinguish between near-miss and distant errors, which is undesirable in tasks such as diagnostic staging or rating prediction (Ayllón-Gavilán et al., 2024).

Decision trees and binary classifier pooling (e.g., SVM with ordered partitions, proportional odds models) are canonical examples of general-purpose methods, but specialized frameworks also exist for handling multicriteria decision aid (MCDA), interval data, high-dimensional genomics, time series, and deep learning scenarios. Each framework constrains the model structure, loss function, or inference process to leverage the label ordering—typically by modifying splitting criteria, introducing monotonic constraints, threshold decompositions, or order-sensitive loss functions (Ayllón-Gavilán et al., 2024, Fernandez et al., 2021, Wang et al., 2023, Bérchez-Moreno et al., 2024, Qiao, 2015, Heredia-Gómez et al., 2018).

2. Ordinal Decision Tree Splitting Criteria

Traditional decision trees use splitting criteria such as the Gini index or Shannon entropy, which are agnostic to class ordering. Ordinal tree-based approaches introduce order-aware impurity measures which penalize non-adjacent splits, leading to superior performance on OC tasks.

• Nominal Gini index:

$G(D_m) = 1 - \sum_{q=1}^Q \hat{p}_{q|m}^2$

with splitting criterion $$\varphi_{Gini}(D_m, \theta) = G(D_m) - [p_\ell G(\ell) + p_r G(r)]$$, where $\hat{p}_{q|m}$ is empirical frequency, $p_\ell, p_r$ denote left/right child proportions.

• Ordinal Gini (OGini):

Uses cumulative class probabilities $\hat{c}_q(D_m) = \sum_{j=1}^q \hat{p}_{j|m}$,

$OG(D_m) = 1 - \sum_{q=1}^Q \hat{c}_q(D_m)$

This penalizes splits that produce non-contiguous class groupings.

• Weighted Information Gain (WIG):

Assigns weight to each class relative to the mode, upweighting distant errors,

$$H_w(D_m) = -\sum_{q=1}^Q w_q(D_m) \hat{p}_{q|m} \log(\hat{p}_{q|m})$$

The splitting criterion $\varphi_{WIG}$ is analogously defined.

• Ranking Impurity (RI):

Quantifies impurity as the sum over all pairs, weighted by their ordinal distance $\beta(C_q, C_j)$.

Experimental evidence over 45 public datasets and standard ordinal metrics (MAE, QWK, RPS) confirms OGini's superiority, particularly for larger numbers of classes ($G(D_m) = 1 - \sum_{q=1}^Q \hat{p}_{q|m}^2$0, with substantial reductions in distant-class misclassifications compared to nominal Gini and information gain (Ayllón-Gavilán et al., 2024).

3. Model Architectures and Pooling Frameworks

Ordinal classifiers are instantiated in several forms:

• Binary-threshold pooling: Decompose an $G(D_m) = 1 - \sum_{q=1}^Q \hat{p}_{q|m}^2$1-class OC task into $G(D_m) = 1 - \sum_{q=1}^Q \hat{p}_{q|m}^2$2 binary subproblems. Each $G(D_m) = 1 - \sum_{q=1}^Q \hat{p}_{q|m}^2$3 is trained to estimate $G(D_m) = 1 - \sum_{q=1}^Q \hat{p}_{q|m}^2$4. Aggregation schemes include:
  • Difference-based: Reconstruct probabilities $G(D_m) = 1 - \sum_{q=1}^Q \hat{p}_{q|m}^2$5 via differences of consecutive $G(D_m) = 1 - \sum_{q=1}^Q \hat{p}_{q|m}^2$6. Enforce monotonicity via pool-adjacent-violators adjustment to maintain $G(D_m) = 1 - \sum_{q=1}^Q \hat{p}_{q|m}^2$7.
  • Tree-based: A hierarchical chain-rule aggregation incorporating conditional dependencies between splits (Rotenberg et al., 18 Mar 2026).
  • Noncrossing SVM pooling: Impose noncrossing constraints on boundary functions to avoid ambiguous predictions (empty or multiply-valued intersections) (Qiao, 2015).
• Proportional Odds Model (POM): Classical cumulative link model with class cutpoints:

$G(D_m) = 1 - \sum_{q=1}^Q \hat{p}_{q|m}^2$8

trained via likelihood maximization, under the proportional odds assumption (Heredia-Gómez et al., 2018).

• Kernel-based and nonparametric OC: Kernel discriminant learning for ordinal regression, ordinal random forests with kernel-induced features, and weighted $G(D_m) = 1 - \sum_{q=1}^Q \hat{p}_{q|m}^2$9-nearest neighbor methods using ordinal-aware aggregations are effective for non-linear or functional/interval-valued data (Alcacer et al., 2023).

4. Deep Ordinal Classification and Loss Architectures

Recent progress in OC has been driven by custom deep learning output heads, order-sensitive loss functions, and regularizers:

• Architectural Heads: Cumulative Link (CLM), stick-breaking, and ordinal binary decomposition (OBD) output layers enforce, respectively, monotonic CDFs, unimodal PMFs, or ordered binary decomposition, compatible with OC semantics (Bérchez-Moreno et al., 2024).
• Loss Functions:
  • Soft label cross-entropy via unimodal target distributions (Poisson, triangular, binomial, or exponential kernels) over classes around the ground truth class.
  • Weighted Kappa Loss reflects squared ordinal distance; OBD-MSE penalizes each binary subproblem as an MSE to the thresholded target.
• Feature space control: Constrained proxies learning (CPL) aligns class proxies along predefined curves (lines, semicircles) or enforces unimodality in proxy similarities, guaranteeing that embedding distances reflect ordinal order (Wang et al., 2023).
• Dropout and interpretability: Hybrid ordinal dropout preserves units highly informative about ordering.

Extensive benchmarking in image, text, and tabular settings demonstrates significant improvements in both mean absolute error and task-specific ordinal metrics when exploiting ordering information in the model pipeline (Bérchez-Moreno et al., 2024).

5. Evaluation Metrics and Benchmarking Practices

Proper evaluation of OC models requires metrics that penalize misclassifications according to ordinal distance and (optionally) class imbalance:

• Mean Absolute Error (MAE): $$\varphi_{Gini}(D_m, \theta) = G(D_m) - [p_\ell G(\ell) + p_r G(r)]$$0
• Averaged MAE (AMAE): Class-wise averaged MAE to account for imbalance (Ayllón-Gavilán et al., 23 Jul 2025).
• Quadratic Weighted Kappa (QWK): Weights errors quadratically by label distance.
• Ranked Probability Score (RPS): Measures squared error in cumulative distributions.
• Closeness Evaluation Measure (CEM): Information-theoretic distance using class and prediction frequencies, robust to ordinal invariance, imbalance, and monotonicity (Amigó et al., 2020).

The TOC-UCO repository embodies current benchmarking practice, providing 46 extensively preprocessed tabular OC datasets spanning a range of class numbers and domains, with 30 stratified splits per dataset and standardized metric protocols for reproducibility and rigorous ranking (Ayllón-Gavilán et al., 23 Jul 2025).

6. Uncertainty Quantification, Conformal Prediction, and Risk Group Calibration

Ordinal risk quantification and uncertainty estimation is central in high-stakes tasks:

• Uncertainty decomposition: Aleatoric and epistemic uncertainty quantification adapts binary variance and entropy-based measures through order-consistent splits, outperforming nominal analogs for detecting errors and out-of-distribution inputs (Haas et al., 1 Jul 2025).
• Distribution-free prediction intervals: Conformal prediction methods construct prediction sets, using marginal and class-conditional conformal p-values and multiple-testing procedures to guarantee finite-sample marginal and per-class coverage. Optimal interval construction is provided by minimum-length sliding-window algorithms, yielding provably minimal expected prediction set lengths (Chakraborty et al., 2024, Zhang et al., 20 Nov 2025).
• Ordinal risk-group classification (ORGC): Explicitly enforces a pre-specified vector of group-level risks in the optimization—subject to interval risk deviation and penalties to avoid degenerate intervals—resulting in calibrated, non-overlapping risk group assignments that are optimal for real-world decision-making thresholds (Toren, 2010).

7. Special Problem Structures: Reference Sets, Boundaries, and MCDA

Ordinal classification in multicriteria settings (MCDA) employs relational assignment based on reference sets of characteristic actions ($$\varphi_{Gini}(D_m, \theta) = G(D_m) - [p_\ell G(\ell) + p_r G(r)]$$1) or limiting boundaries ($$\varphi_{Gini}(D_m, \theta) = G(D_m) - [p_\ell G(\ell) + p_r G(r)]$$2) between ordered classes. Relational systems $$\varphi_{Gini}(D_m, \theta) = G(D_m) - [p_\ell G(\ell) + p_r G(r)]$$3, with $$\varphi_{Gini}(D_m, \theta) = G(D_m) - [p_\ell G(\ell) + p_r G(r)]$$4 reflexive and compatible and $$\varphi_{Gini}(D_m, \theta) = G(D_m) - [p_\ell G(\ell) + p_r G(r)]$$5 transitive, allow for robust and symmetric rules (pseudo-conjunctive/disjunctive) and satisfaction of key properties: conformity, monotonicity, stability, and uniqueness. Methods such as ELECTRE TRI-nC, INTERCLASS-nC, and their hierarchical extensions arise as special cases. Two-layer, dual assignment rules resolve separation, transpositional, and conformity issues pervasive in traditional outranking frameworks (Fernandez et al., 2021, Fernandez et al., 2021).

Table: Representative Ordinal Criteria in Decision Trees

Splitting Criterion	Formula / Concept	Key Property
Gini index	$$\varphi_{Gini}(D_m, \theta) = G(D_m) - [p_\ell G(\ell) + p_r G(r)]$$6	Nominal (order-agnostic)
Information Gain	$$\varphi_{Gini}(D_m, \theta) = G(D_m) - [p_\ell G(\ell) + p_r G(r)]$$7	Nominal (order-agnostic)
OGini	$$\varphi_{Gini}(D_m, \theta) = G(D_m) - [p_\ell G(\ell) + p_r G(r)]$$8	Order-aware (penalizes non-adjacent)
WIG	$$\varphi_{Gini}(D_m, \theta) = G(D_m) - [p_\ell G(\ell) + p_r G(r)]$$9	Order-aware (weighted by distance)
Ranking Impurity	$\hat{p}_{q|m}$0	Order-aware (ranking errors weighted)

These criteria serve as the engine for modern, practical ordinal decision trees, leading to empirically and statistically validated gains over traditional nominal tree approaches for OC problems (Ayllón-Gavilán et al., 2024).

---

In summary, ordinal classification systems comprise a diverse and technically sophisticated toolkit, including tree-based, ensemble, MCDA, interval/functional data, time series, and deep learning methods, each with bespoke loss functions, optimization constraints, and uncertainty quantification. Rigorous evaluation on standardized repositories with order-sensitive metrics is standard. Active research targets further integration with probabilistic calibration, uncertainty quantification, score-free splitting, and extension to problematic or high-dimensional label structures. The field continues to evolve toward ever more robust, transparent, and theoretically justified exploitation of label order information.