Probability Scale Loss: Foundations & Extensions

• Probability scale loss is a class of proper scoring rules that evaluates predictions on the probability scale (scalar, vector, or distribution) rather than point estimates.
• Canonical losses like the Brier score, log-loss, and CRPS impose distinct geometric structures—Euclidean or information-theoretic—on the prediction space, guiding model calibration.
• Extensions include multi-scale decompositions, adaptive loss scaling, and online aggregation methods, which optimize calibration, reduce grouping losses, and ensure robust predictive performance.

Probability scale loss denotes a loss or scoring rule whose primary argument is a probabilistic prediction—typically a probability $p\in[0,1]$, a probability vector in the simplex $\Delta(\mathcal Y)$, or an entire predictive distribution represented by a cumulative distribution function (CDF)—rather than a single point estimate. In the cited literature, the term covers binary and multiclass proper losses such as the Brier score and log-loss, continuous-outcome scores such as the continuous ranked probability score (CRPS), and related scale-aware constructions that operate on expected probability mass or on multi-scale decompositions of probabilistic fields (Charpentier et al., 16 Mar 2026, V'yugin et al., 2019, Lang et al., 12 Jun 2025, Li et al., 11 Apr 2026).

1. Formal definitions and scope

For finite-label prediction with $\mathcal Y=\{1,\dots,K\}$, a probabilistic predictor is a measurable map $s:\mathcal X\to\Delta(\mathcal Y)$, and the induced random prediction is $S=s(X)$. A proper probabilistic loss is written as

$$\ell:\Delta(\mathcal Y)\times \mathcal Y\to\mathbb R,$$

with conditional risk

$L(p,q):=\mathbb E_{Y\sim q}[\ell(p,Y)].$

Properness means that for every $q\in\Delta(\mathcal Y)$,

$$L(q,q)\le L(p,q)\quad\text{for all }p\in\Delta(\mathcal Y),$$

and strict propriety requires equality only when $p=q$. The associated generalized entropy and proper divergence are

$\Delta(\mathcal Y)$0

These objects provide the standard decision-theoretic formalization of losses on the probability scale (Charpentier et al., 16 Mar 2026).

A binary formulation used in probabilistic prediction writes the loss as

$\Delta(\mathcal Y)$1

where the predictor outputs a probability $\Delta(\mathcal Y)$2 for label $\Delta(\mathcal Y)$3. In the statistical evaluation literature, the related quantity of interest is often a loss $\Delta(\mathcal Y)$4 that compares the forecast probability $\Delta(\mathcal Y)$5 with the true event probability $\Delta(\mathcal Y)$6, rather than only with the realized Bernoulli outcome $\Delta(\mathcal Y)$7; this distinction underlies martingale-based inference for forecast evaluation (Vovk, 2015, Lai et al., 2012).

This suggests that “probability scale loss” is best understood as a family of losses defined on probabilistic objects. The common structure is that the prediction itself lies on a probability scale—scalar, vector, or distributional—and the loss evaluates the quality of that probabilistic object directly.

2. Canonical proper losses on probabilities

The canonical probability-scale losses in binary and multiclass prediction are the Brier score, log-loss, and spherical loss. For multiclass prediction, the Brier score is

$\Delta(\mathcal Y)$8

and for binary prediction it reduces to

$\Delta(\mathcal Y)$9

Log-loss is

$\mathcal Y=\{1,\dots,K\}$0

In binary form, the paper on the fundamental nature of log loss writes

$\mathcal Y=\{1,\dots,K\}$1

and

$\mathcal Y=\{1,\dots,K\}$2

Spherical loss is also strictly proper and mixable in that framework (Charpentier et al., 16 Mar 2026, Vovk, 2015).

These losses induce different geometries on the probability scale. For the Brier score,

$\mathcal Y=\{1,\dots,K\}$3

so the divergence is Euclidean and quadratic. For log-loss,

$\mathcal Y=\{1,\dots,K\}$4

so the divergence is information-theoretic. The decomposition framework therefore interprets Brier as variance-based and log-loss as entropy-based (Charpentier et al., 16 Mar 2026).

A central theoretical distinction concerns selectivity. In the algorithmic theory of randomness, any prediction algorithm that is optimal for a data sequence under log loss is optimal under any computable proper mixable loss; the converse fails for Brier and spherical loss. For CPMS loss functions, the fundamental losses are exactly those of degree $\mathcal Y=\{1,\dots,K\}$5, which includes log loss but not Brier or spherical loss (Vovk, 2015). A common misconception is therefore that all proper losses are interchangeable. They are not: they agree on truthful reporting in expectation, but they do not induce the same notion of optimality, especially near extreme probabilities.

3. CRPS as a probability-scale loss for continuous outcomes

For scalar continuous outcomes, the standard probability-scale loss is CRPS. On a finite interval $\mathcal Y=\{1,\dots,K\}$6, for outcome $\mathcal Y=\{1,\dots,K\}$7 and forecast CDF $\mathcal Y=\{1,\dots,K\}$8,

$\mathcal Y=\{1,\dots,K\}$9

where $s:\mathcal X\to\Delta(\mathcal Y)$0 is the Heaviside step function. The same paper also cites a weighted version on the real line,

$s:\mathcal X\to\Delta(\mathcal Y)$1

with nonnegative density $s:\mathcal X\to\Delta(\mathcal Y)$2 (V'yugin et al., 2019).

CRPS is a probability-scale loss because its argument is the whole predictive distribution, via its CDF, and the loss is obtained by integrating over thresholds. For a realized outcome $s:\mathcal X\to\Delta(\mathcal Y)$3, the “perfect” CDF is $s:\mathcal X\to\Delta(\mathcal Y)$4, which places all mass at $s:\mathcal X\to\Delta(\mathcal Y)$5. CRPS measures the integrated squared discrepancy between the forecast CDF and this ideal CDF. Equivalently, for each threshold $s:\mathcal X\to\Delta(\mathcal Y)$6, the event $s:\mathcal X\to\Delta(\mathcal Y)$7 has predicted probability $s:\mathcal X\to\Delta(\mathcal Y)$8 and realized indicator $s:\mathcal X\to\Delta(\mathcal Y)$9, so CRPS is an integrated Brier score across thresholds. It is therefore both an $S=s(X)$0 distance on CDF space and a threshold-integrated proper score (V'yugin et al., 2019).

This formulation distinguishes CRPS from pointwise losses. It is defined directly on the distribution rather than on a mean, median, or quantile; it is proper and sensitive to both calibration and sharpness; and, unlike log-loss, it remains finite even when densities are not absolutely continuous or have bounded support. In the continuous-outcome setting, CRPS occupies the same role that the Brier score occupies for binary events.

4. Reliability, information loss, and uncertainty

A major development in the theory of probability-scale losses is the decomposition of expected proper loss into calibration and information components. For any sub-$S=s(X)$1-algebra $S=s(X)$2, define the conditional law

$S=s(X)$3

If $S=s(X)$4 is an $S=s(X)$5-measurable predictor, then the one-level decomposition is

$S=s(X)$6

For nested information levels $S=s(X)$7,

$S=s(X)$8

The first term is proper-regret at information level $S=s(X)$9, the second is information gain from $$\ell:\Delta(\mathcal Y)\times \mathcal Y\to\mathbb R,$$0 to $$\ell:\Delta(\mathcal Y)\times \mathcal Y\to\mathbb R,$$1, and the third is residual uncertainty at the richer information level (Charpentier et al., 16 Mar 2026).

For classification with features $$\ell:\Delta(\mathcal Y)\times \mathcal Y\to\mathbb R,$$2, score $$\ell:\Delta(\mathcal Y)\times \mathcal Y\to\mathbb R,$$3, and label $$\ell:\Delta(\mathcal Y)\times \mathcal Y\to\mathbb R,$$4, let

$$\ell:\Delta(\mathcal Y)\times \mathcal Y\to\mathbb R,$$5

Then

$$\ell:\Delta(\mathcal Y)\times \mathcal Y\to\mathbb R,$$6

This three-term identity separates miscalibration, grouping, and irreducible uncertainty. The miscalibration term $$\ell:\Delta(\mathcal Y)\times \mathcal Y\to\mathbb R,$$7 vanishes iff $$\ell:\Delta(\mathcal Y)\times \mathcal Y\to\mathbb R,$$8 is perfectly calibrated; the grouping term $$\ell:\Delta(\mathcal Y)\times \mathcal Y\to\mathbb R,$$9 measures information loss from compressing $L(p,q):=\mathbb E_{Y\sim q}[\ell(p,Y)].$0 to $L(p,q):=\mathbb E_{Y\sim q}[\ell(p,Y)].$1; and the final term is the Bayes risk at the feature level (Charpentier et al., 16 Mar 2026).

The specializations clarify the geometry of different losses. For binary Brier loss,

$L(p,q):=\mathbb E_{Y\sim q}[\ell(p,Y)].$2

For log-loss,

$L(p,q):=\mathbb E_{Y\sim q}[\ell(p,Y)].$3

These identities show that calibration is only one component of probabilistic performance. A perfectly calibrated but coarse score can still incur large grouping loss. They also delimit the effect of post-hoc recalibration: the population-optimal recalibrated score is $L(p,q):=\mathbb E_{Y\sim q}[\ell(p,Y)].$4, which can reduce the miscalibration term but cannot change grouping or irreducible uncertainty. Even aggregation of calibrated models need not preserve calibration (Charpentier et al., 16 Mar 2026).

5. Mixability, regret, and online aggregation

Probability-scale losses are especially important in online learning when forecasts from several experts must be aggregated sequentially. In Vovk’s framework, a loss $L(p,q):=\mathbb E_{Y\sim q}[\ell(p,Y)].$5 is $L(p,q):=\mathbb E_{Y\sim q}[\ell(p,Y)].$6-mixable if there exists a substitution function such that

$L(p,q):=\mathbb E_{Y\sim q}[\ell(p,Y)].$7

for all outcomes $L(p,q):=\mathbb E_{Y\sim q}[\ell(p,Y)].$8, given expert forecasts $L(p,q):=\mathbb E_{Y\sim q}[\ell(p,Y)].$9 and mixture weights $q\in\Delta(\mathcal Y)$0. Mixability yields time-independent regret bounds of order $q\in\Delta(\mathcal Y)$1 (V'yugin et al., 2019).

For continuous outcomes on $q\in\Delta(\mathcal Y)$2, the main theoretical result is that CRPS is $q\in\Delta(\mathcal Y)$3-mixable. In the prediction-with-expert-advice setting, expert $q\in\Delta(\mathcal Y)$4 outputs a CDF $q\in\Delta(\mathcal Y)$5, the learner outputs an aggregated CDF $q\in\Delta(\mathcal Y)$6, and the cumulative regret is

$q\in\Delta(\mathcal Y)$7

The resulting bound is

$q\in\Delta(\mathcal Y)$8

for every expert $q\in\Delta(\mathcal Y)$9 and every horizon $$L(q,q)\le L(p,q)\quad\text{for all }p\in\Delta(\mathcal Y),$$0. The regret is therefore time-independent and logarithmic in the number of experts (V'yugin et al., 2019).

The proof approximates each CDF by a piecewise-constant function on a grid, turns CRPS into a scaled sum of squared component losses, uses that square loss on $$L(q,q)\le L(p,q)\quad\text{for all }p\in\Delta(\mathcal Y),$$1 is $$L(q,q)\le L(p,q)\quad\text{for all }p\in\Delta(\mathcal Y),$$2-mixable, and then lets the grid step go to zero. This places CRPS in the class of mixable proper scoring rules on distributions, closely paralleling the role of the Brier score on binary probabilities (V'yugin et al., 2019).

The same paper also gives practical aggregation algorithms. Vovk’s Aggregating Algorithm uses the CRPS substitution function pointwise in the threshold variable $$L(q,q)\le L(p,q)\quad\text{for all }p\in\Delta(\mathcal Y),$$3 and updates expert weights with learning rate $$L(q,q)\le L(p,q)\quad\text{for all }p\in\Delta(\mathcal Y),$$4. A simpler exponential-concave weighted average forecaster uses convex combinations of CDFs and attains the weaker bound

$$L(q,q)\le L(p,q)\quad\text{for all }p\in\Delta(\mathcal Y),$$5

For specialized experts with confidence values $$L(q,q)\le L(p,q)\quad\text{for all }p\in\Delta(\mathcal Y),$$6, the discounted regret guarantee becomes

$$L(q,q)\le L(p,q)\quad\text{for all }p\in\Delta(\mathcal Y),$$7

These results show that probability-scale losses are not only evaluative devices but also algorithmic primitives for online distributional learning (V'yugin et al., 2019).

6. Scale-aware and adaptive extensions

A related line of work extends probability-scale losses to explicitly encode spatial scale or learnable loss scale. In machine-learned weather forecasting at ECMWF, AIFS-CRPS is trained by directly optimizing the almost fair CRPS,

$$L(q,q)\le L(p,q)\quad\text{for all }p\in\Delta(\mathcal Y),$$8

and a multi-scale loss is formed by decomposing each field into scale components through smoothing operators $$L(q,q)\le L(p,q)\quad\text{for all }p\in\Delta(\mathcal Y),$$9 and summing afCRPS across scales:

$p=q$0

In the reported experiments, the model uses two scales with equal weights, a Gaussian filter with standard deviation equal to eight grid spacings, and exhibits nearly identical fCRPS skill to the scale-unaware baseline while better constraining small-scale variability, especially in spectra for smooth fields such as $p=q$1 geopotential height (Lang et al., 12 Jun 2025).

In dense segmentation, the network output after sigmoid is a per-pixel probability map $p=q$2, and the global predicted area

$p=q$3

is interpreted as expected foreground area. The diff-based scale factor is

$p=q$4

and the final loss after warm-up is

$p=q$5

Its defining property is strictly monotonic decay as the area mismatch increases, in contrast to the non-monotonic Var-based alternative. On IRSTD-1k, the reported L1-GP-Rotated variant reaches $p=q$6 mIoU, $p=q$7 $p=q$8, and $p=q$9 $\Delta(\mathcal Y)$00, compared with $\Delta(\mathcal Y)$01, $\Delta(\mathcal Y)$02, and $\Delta(\mathcal Y)$03 for the Var-based counterpart (Li et al., 11 Apr 2026).

Another extension treats the loss itself as a negative log-likelihood with learnable scale or shape parameters. For Gaussian regression,

$\Delta(\mathcal Y)$04

and for softmax classification with temperature $\Delta(\mathcal Y)$05,

$\Delta(\mathcal Y)$06

Jointly optimizing model parameters with likelihood parameters such as variance, temperature, and robust-loss shape produces adaptive loss scaling for robust modeling, outlier-detection, and re-calibration; the same logic extends to $\Delta(\mathcal Y)$07 and $\Delta(\mathcal Y)$08 regularization through Gaussian and Laplace priors with learnable scales (Hamilton et al., 2020).

Taken together, these developments indicate that probability-scale loss is not a single standardized object but a coherent class of constructions. In its strictest sense, it denotes proper losses on probabilities or predictive distributions; in adjacent literatures, it also denotes scale-conditioned probabilistic objectives that preserve the central idea that the prediction is itself a probabilistic quantity, and that the loss should act on that quantity at the appropriate statistical or spatial scale.