Weighted Ordinal Cross-Entropy Loss

• Weighted ordinal cross-entropy loss is a family of loss functions that integrates ordinal distance or cost matrices into standard cross-entropy for ordered label prediction.
• It adjusts penalty strengths based on the closeness of misclassified labels, thereby reducing the impact of severe errors in domains like medical diagnosis and forecasting.
• Different formulations, such as CDW-CE, OCE, and weighted cumulative likelihoods, have demonstrated practical improvements in calibration, gradient behavior, and overall performance.

Weighted ordinal cross-entropy loss denotes a family of loss functions for ordered labels in which cross-entropy terms are modulated by ordinal distance, threshold position, or a task-specific cost matrix. The common objective is to make training sensitive to the fact that some mistakes are ordinally or operationally closer than others: confusing adjacent ranks should not generally incur the same penalty as confusing distant ranks, and errors near a clinically or physically meaningful boundary may deserve special emphasis. In the recent literature, this idea appears in several mathematically distinct forms, including distance-weighted softmax losses, complement-log penalties on non-true classes, weighted cumulative-threshold likelihoods, and weighted binary cross-entropy for binary tasks with latent ordinal sub-classes (Polat et al., 2022, Jenkinson et al., 2021, Dvora et al., 24 Jun 2026, Pandey et al., 5 Oct 2025, Wang et al., 13 Nov 2025).

1. Ordinal motivation and the limitation of nominal cross-entropy

In ordinal prediction, labels are ordered, but standard categorical cross-entropy treats them as nominal. This means that a model is penalized through the mass assigned to the true class, without direct regard to how the remaining probability is distributed across nearby or distant incorrect classes. In the neural ordinal regression literature, this is criticized on two fronts. First, categorical cross-entropy does not account for ordinal distances. Second, it does not enforce structural properties such as monotone cumulative probabilities or unimodal predictive distributions (Jenkinson et al., 2021, Kim et al., 2024).

For ordered classes $r_1 < r_2 < \dots < r_K$, this mismatch can be operationally significant. A model that confuses class $2$ with class $3$ is often making a much less severe error than a model that confuses class $2$ with class $K$, yet nominal cross-entropy does not express that asymmetry. In medical severity estimation, this distinction corresponds to non-uniform clinical risk; in thresholded binary tasks, it appears as heightened ambiguity around the boundary class. In calibration work, one-hot cross-entropy is also associated with overconfident predictions and a failure to encourage unimodal probability profiles across ordered classes (Kim et al., 2024).

The term “weighted ordinal cross-entropy” therefore does not refer to one universally adopted formula. Rather, it refers to a design principle: preserve the optimization advantages of cross-entropy while injecting ordinal structure through weights, soft ordinal targets, cumulative decompositions, or complementary penalties. This suggests that the decisive modeling choice is not whether cross-entropy is used, but where ordinal information enters the loss.

2. Principal formulations

Several formulations are now used under the broader weighted ordinal cross-entropy umbrella.

Distance-weighted complementary softmax losses penalize probability assigned to incorrect classes according to their ordinal distance from the true label. In Class Distance Weighted Cross-Entropy (CDW-CE), for ordered classes indexed by $i \in \{0,\dots,K-1\}$, true class $y$, softmax probabilities $p_i$, and $d(y,i)=|i-y|$, the per-sample loss is

$$L_{\mathrm{CDW\text{-}CE}}(z, y) = -\sum_{i=0}^{K-1} w_{y,i}\,\log(1-p_i), \qquad w_{y,i}=|i-y|^\alpha.$$

Because $2$0, the true class contributes indirectly through softmax coupling, while far-away wrong classes incur larger penalties (Polat et al., 2022, Polat et al., 2024).

Cost-matrix ordinal cross-entropy generalizes this idea by replacing distance powers with an explicit ordinal cost matrix. In the OCE framework, a nonnegative matrix $2$1 encodes the cost of predicting $2$2 when the true class is $2$3, potentially with asymmetric underestimation and overestimation penalties. After transforming $2$4 into a normalized penalty–reward matrix $2$5, the loss for one example is

$2$6

Standard multi-class cross-entropy is recovered when $2$7 and $2$8 for $2$9 (Dvora et al., 24 Jun 2026).

Weighted cumulative-threshold likelihoods arise in ordinal regression models that decompose the task into $3$0 binary threshold subtasks. In the CONDOR formulation, the extended-binary indicators are $3$1, and the conditional transition probabilities $3$2 form a Markov chain. The exact maximum-likelihood loss is

$3$3

and the weighted extension is

$3$4

Here $3$5 is a sample weight, $3$6 a threshold weight, and $3$7 a cost-sensitive factor that can encode ordinal distance (Jenkinson et al., 2021).

Boundary-aware weighted binary cross-entropy appears when the observed task is binary but the labels contain hidden ordered sub-classes. In solar flare prediction with threshold $3$8, the ordered flare subclasses $3$9 induce distances to the decision boundary between $2$0 and $2$1. The proposed BCE-PP loss uses weights $2$2, with larger values for classes closer to the threshold: $2$3 Equivalently, with distance $2$4, $2$5 (Pandey et al., 5 Oct 2025).

Weighted cumulative ordinal cross-entropy for discretized forecasting uses cumulative probabilities rather than class probabilities directly. In OCE-TS, continuous targets are discretized into ordered bins, yielding ground-truth cumulative probabilities $2$6 and predicted cumulative probabilities $2$7. The weighted loss is

$2$8

with $2$9 acting as per-threshold importance weights (Wang et al., 13 Nov 2025).

These formulations share a cross-entropy core but differ in where the weighting operates: on wrong-class complements, on true-class and wrong-class terms via a cost matrix, on conditional threshold likelihoods, on hidden ordinal subclasses in a binary problem, or on cumulative probabilities after discretization.

3. Gradient behavior and optimization

The main optimization effect of weighted ordinal cross-entropy is gradient reshaping. The loss does not merely alter the scalar objective; it reallocates gradient magnitude toward ordinally consequential mistakes.

For BCE-PP, standard BCE yields the per-sample derivative

$K$0

Under ordinal weighting,

$K$1

Samples at the boundary subclasses $K$2 or $K$3, where $K$4, therefore receive larger gradient magnitudes than distant subclasses such as $K$5, where $K$6. The paper explicitly interprets this as intensified learning pressure on borderline cases (Pandey et al., 5 Oct 2025).

In CONDOR, the weighted maximum-likelihood construction preserves the conditional-threshold semantics. If $K$7 is the logit for $K$8, then

$K$9

The gating term $i \in \{0,\dots,K-1\}$0 is structurally important: subtask $i \in \{0,\dots,K-1\}$1 is only active when the example has already “passed” threshold $i \in \{0,\dots,K-1\}$2. Because the monotonicity condition $i \in \{0,\dots,K-1\}$3 is architectural, nonnegative weighting changes gradient allocation without breaking universal rank consistency (Jenkinson et al., 2021).

In OCE, the gradient with respect to logits depends on both the true-class reward and the complement penalties on all wrong classes. With $i \in \{0,\dots,K-1\}$4, $i \in \{0,\dots,K-1\}$5 for $i \in \{0,\dots,K-1\}$6, and $i \in \{0,\dots,K-1\}$7,

$i \in \{0,\dots,K-1\}$8

and for $i \in \{0,\dots,K-1\}$9,

$y$0

This gives far and clinically severe errors larger corrective gradients when both $y$1 and $y$2 are large (Dvora et al., 24 Jun 2026).

For cumulative forecasting WOCE, the threshold-level derivative is

$y$3

so weighting is applied directly at ordinal thresholds rather than at discrete classes. This is especially natural when the primary concern is crossing critical ranges rather than misclassifying a single categorical state (Wang et al., 13 Nov 2025).

4. Structural relations: cumulative modeling, calibration, and unimodality

Weighted ordinal cross-entropy intersects with several neighboring approaches in ordinal learning. One major distinction is between ordinal weighting and ordinal architecture. Weighting changes the training signal; architectural methods change the output parameterization.

Cumulative-threshold models such as cumulative link methods, CORAL, CORN, and CONDOR represent ordinal structure through $y$4 binary comparisons. In CONDOR, the subtask probabilities form a Markov chain, and the cumulative probabilities satisfy $y$5 for any parameters. Weighted ordinal cross-entropy in this setting is therefore a weighted conditional likelihood rather than a weighted softmax loss (Jenkinson et al., 2021).

A second distinction is between hard one-hot targets with weights and soft ordinal targets. Calibration work argues that standard cross-entropy with one-hot labels pushes the model toward overconfidence and does not discourage multimodal distributions across ordered classes. The ORCU loss addresses this by combining soft ordinal encoding,

$y$6

with an ordinal-aware regularizer,

$y$7

The paper explicitly contrasts this with WOCE/CDW-CE-style baselines: weighted CE introduces distance sensitivity, but soft ordinal encoding plus ordinal-aware regularization additionally targets calibration and unimodality (Kim et al., 2024).

A third distinction is between penalizing class probabilities and constraining probability shape. The unimodal Poisson/binomial approach for deep ordinal classification constrains the output distribution itself to be unimodal by parameterizing class probabilities through a PMF over the ordered support. That work does not explicitly propose a weighted ordinal cross-entropy, although the accompanying synthesis describes a compatible distance-aware weighted extension. This makes clear that weighting and unimodality are complementary rather than interchangeable design choices (Beckham et al., 2017).

Finally, weighted ordinal cross-entropy is often compared with Earth Mover’s Distance, label-distribution learning, focal loss, and class-weighted cross-entropy. The recurrent pattern in these comparisons is that weighted ordinal cross-entropy is simpler than full distribution-transport objectives, more ordinally faithful than nominal CE, and more directly cost-sensitive than focal or class-frequency weighting, but it does not automatically guarantee calibration, unimodality, or strict cumulative monotonicity unless those properties are imposed elsewhere in the model (Polat et al., 2022, Kim et al., 2024, Pandey et al., 5 Oct 2025).

5. Reported applications and empirical behavior

In ulcerative colitis severity estimation on LIMUC, CDW-CE was evaluated on Mayo Endoscopic Score classes $y$8–$y$9. For the 4-class task, Inception-v3 trained with CDW-CE achieved $p_i$0, $p_i$1, $p_i$2, and $p_i$3, compared with $p_i$4, $p_i$5, $p_i$6, and $p_i$7 for CE. The paper also reports that optimal $p_i$8 differed by architecture, with approximately $p_i$9 for ResNet18, $d(y,i)=|i-y|$0 for Inception-v3, and $d(y,i)=|i-y|$1 for MobileNet-v3-large, and that training became unstable when $d(y,i)=|i-y|$2 was too large (Polat et al., 2022).

A later disease-severity study using the same CDW-CE formulation reported similar qualitative findings beyond classification metrics. On LIMUC, a ResNet18 trained with CE had Silhouette score $d(y,i)=|i-y|$3, whereas CDW-CE yielded $d(y,i)=|i-y|$4, and clinician assessment of CAMs indicated that CDW-CE highlighted broader and more clinically relevant disease areas. The same study reports slight QWK gains from adding a margin term, for example $d(y,i)=|i-y|$5 on Inception-v3 with $d(y,i)=|i-y|$6 (Polat et al., 2024).

In solar flare prediction with binary threshold $d(y,i)=|i-y|$7, the BCE-PP loss was evaluated on SHARP AR patches from SDO/HMI LOS magnetograms. On the test set with a MobileNet backbone, BCE gave $d(y,i)=|i-y|$8, $d(y,i)=|i-y|$9, $$L_{\mathrm{CDW\text{-}CE}}(z, y) = -\sum_{i=0}^{K-1} w_{y,i}\,\log(1-p_i), \qquad w_{y,i}=|i-y|^\alpha.$$0, $$L_{\mathrm{CDW\text{-}CE}}(z, y) = -\sum_{i=0}^{K-1} w_{y,i}\,\log(1-p_i), \qquad w_{y,i}=|i-y|^\alpha.$$1, $$L_{\mathrm{CDW\text{-}CE}}(z, y) = -\sum_{i=0}^{K-1} w_{y,i}\,\log(1-p_i), \qquad w_{y,i}=|i-y|^\alpha.$$2, $$L_{\mathrm{CDW\text{-}CE}}(z, y) = -\sum_{i=0}^{K-1} w_{y,i}\,\log(1-p_i), \qquad w_{y,i}=|i-y|^\alpha.$$3, and $$L_{\mathrm{CDW\text{-}CE}}(z, y) = -\sum_{i=0}^{K-1} w_{y,i}\,\log(1-p_i), \qquad w_{y,i}=|i-y|^\alpha.$$4. BCE-PP gave $$L_{\mathrm{CDW\text{-}CE}}(z, y) = -\sum_{i=0}^{K-1} w_{y,i}\,\log(1-p_i), \qquad w_{y,i}=|i-y|^\alpha.$$5, $$L_{\mathrm{CDW\text{-}CE}}(z, y) = -\sum_{i=0}^{K-1} w_{y,i}\,\log(1-p_i), \qquad w_{y,i}=|i-y|^\alpha.$$6, $$L_{\mathrm{CDW\text{-}CE}}(z, y) = -\sum_{i=0}^{K-1} w_{y,i}\,\log(1-p_i), \qquad w_{y,i}=|i-y|^\alpha.$$7, $$L_{\mathrm{CDW\text{-}CE}}(z, y) = -\sum_{i=0}^{K-1} w_{y,i}\,\log(1-p_i), \qquad w_{y,i}=|i-y|^\alpha.$$8, $$L_{\mathrm{CDW\text{-}CE}}(z, y) = -\sum_{i=0}^{K-1} w_{y,i}\,\log(1-p_i), \qquad w_{y,i}=|i-y|^\alpha.$$9, $2$00, and $2$01. The reported interpretation is a notable reduction in false positives, a slight increase in false negatives, and a modest overall improvement in skill (Pandey et al., 5 Oct 2025).

In calibration-focused ordinal regression, ORCU was evaluated against CE, SORD, and CDW-CE. On Adience, CDW-CE obtained $2$02, $2$03, $2$04, and $2$05, whereas ORCU obtained $2$06, $2$07, $2$08, and $2$09. This comparison is often read as evidence that distance-aware weighting improves ordinal awareness but does not, by itself, solve calibration or unimodality (Kim et al., 2024).

In probabilistic time-series forecasting, OCE-TS replaces MSE with cumulative ordinal cross-entropy after discretizing continuous targets into ordered bins and constructing soft supervision distributions. The paper compares OCE-TS with Autoformer, DLinear, iTransformer, TimeXer, and TimeBridge on seven public datasets and reports that OCE-TS consistently outperforms the benchmark models under MSE and MAE evaluation (Wang et al., 13 Nov 2025).

6. Weight design, limitations, and open problems

The central design variable in weighted ordinal cross-entropy is the weighting scheme itself. The literature uses several mechanisms. Distance-based schemes include linear, quadratic, Huber, and exponential growth with $2$10 or $2$11; threshold-based schemes upweight certain cumulative boundaries; sample weights handle class imbalance, label reliability, or domain importance; and cost-matrix formulations allow asymmetric underestimation and overestimation costs. In OCE, asymmetry is encoded directly through

$2$12

while in weighted CONDOR the factors $2$13, $2$14, and $2$15 separate sample-level, threshold-level, and cost-sensitive contributions (Dvora et al., 24 Jun 2026, Jenkinson et al., 2021).

The main limitations are also consistent across formulations. If ordinal mappings or subclass labels are noisy, the weighting can misguide optimization. If weights are too large, training can become unstable, gradients can saturate, or probability calibration can deteriorate. Fixed threshold assumptions can become suboptimal when the operational boundary drifts. For very large $2$16, cumulative-product formulations may underflow, and conditional gating can leave rare extreme thresholds under-trained. These failure modes are documented in the solar-flare, CONDOR, OCE, and forecasting settings (Pandey et al., 5 Oct 2025, Jenkinson et al., 2021, Dvora et al., 24 Jun 2026, Wang et al., 13 Nov 2025).

Several remedies recur in practice. Weight normalization to mean $2$17, annealing of extreme weights, numerical stabilization through BCE-with-logits or clipped complement logs, per-class imbalance weights, temperature scaling, and post-hoc calibration are all proposed. In the forecasting setting, clipping $2$18 to $2$19 is recommended; in CDW-CE, $2$20 is stabilized via clamping; in CONDOR, the maximum-likelihood conditional loss is preferred over heuristic weighted BCE on marginals for stability, especially when $2$21 is large (Polat et al., 2024, Wang et al., 13 Nov 2025, Jenkinson et al., 2021).

A persistent open question is whether weighted ordinal cross-entropy should be viewed primarily as a surrogate for ordinal metrics, a calibrated probabilistic model, or a cost-sensitive decision objective. The recent weighted-metric framework formalizes weighted cross-entropy as one instance of a broader mapping from weighted evaluation metrics to differentiable losses, suggesting that the most principled weighting scheme is task-dependent and should be derived from the metric or utility function of interest rather than chosen ad hoc (Marchetti et al., 2023).