Penalized Brier Score (PBS)

• Penalized Brier Score is a proper scoring rule that adds a class-based penalty to ensure correct predictions are always prioritized over misclassifications.
• It employs a fixed penalty of (c-1)/c, which eliminates the need for extra tuning and aligns the score with the underlying classification objective.
• Empirical studies on spatio-temporal tasks demonstrate that PBS improves the correlation with F1 scores and enhances model selection by up to 7.1 points.

The Penalized Brier Score (PBS) is a strictly proper scoring rule designed for the evaluation of probabilistic predictions in single-label multi-class classification tasks. Introduced to address the limitations of the traditional Brier Score (BS), PBS augments the evaluation criterion by incorporating an explicit, class-dependent penalty on misclassifications. This ensures that all correct predictions, regardless of their probability calibration, are always preferred over incorrect predictions, thereby aligning evaluation with the classification objective and improving the reliability of model selection and early stopping in practical settings (Ahmadian et al., 2024).

1. Mathematical Definition and Theoretical Properties

Let $c$ denote the number of classes. For a one-hot ground truth vector $y\in\{0,1\}^c$ and a predicted probability distribution $q\in[0,1]^c$ with $\sum_j q_j=1$, define

• $\psi = \{ q \mid \arg\max q = \arg\max y \}$: correct decisions,
• $\xi = \{ q \mid \arg\max q \neq \arg\max y \}$: incorrect decisions.

The standard Brier Score is

$S_{\text{BS}}(q, y) = \sum_{j=1}^c (q_j - y_j)^2.$

The Penalized Brier Score is then defined as

$$S_{\text{PBS}}(q, y) = \sum_{j=1}^c (q_j - y_j)^2 + \begin{cases} \displaystyle{\frac{c-1}{c}}, & q \in \xi \ 0, & q \in \psi \end{cases}$$

The penalty value $(c-1)/c$ serves as the smallest constant sufficient to ensure that any misclassification scores strictly worse than any correct classification. The classical notion of a (negatively oriented) strictly proper scoring rule is preserved: for any true distribution $Q$ and any prediction $P \neq Q$, the expected PBS satisfies $S_{\text{PBS}}(P, Q) > S_{\text{PBS}}(Q, Q)$, due to the non-negative penalty term that is strictly positive when $P$ yields misclassifications with nonzero probability (see Theorem 4.5 in (Ahmadian et al., 2024)).

2. Penalty Setting and Hyperparameterization

PBS contains a single penalty parameter, fixed at $(c-1)/c$ for $c$ classes. This value is justified by Theorems 4.3 and 4.4, which show that it is exactly the maximum Brier Score possible for any correct prediction (i.e., when $q$ places all mass except for a $1/c$ portion on the correct label). The penalty enforces that every incorrect decision obtains a higher (worse) PBS than the "worst-case" correct decision. No further calibration or tuning of this penalty is required; the setting is uniquely determined by the class count.

3. Experimental Methodology

The evaluation of PBS was conducted across nine real-world spatio-temporal multi-class tasks, including activity recognition, driver identification, power-consumption zoning, air-quality site classification, indoor localization (via RSSI), and motor-failure time prediction. Class cardinality ranged from 3 to 13 across these datasets.

The learning architecture employed a small convolutional neural network (CNN) that operated on sliding windowed sensor or time-series input segments. Optimization was conducted using Nadam with default settings. Temporal dependency was addressed by $h$-block cross-validation (Algorithm 4 in (Ahmadian et al., 2024)): each dataset was segmented into $h$ contiguous blocks, with 20% reserved for validation and 30% for testing per fold, and training on the remainder. Grid search determined window length and overlap.

Checkpointing (CP) and early stopping (ES) mechanisms were driven separately by conventional (BS/LL) and penalized (PBS/PLL) criteria. Cross-validation was repeated 100 times, and the best checkpoint by each metric was tested on the associated fold's test partition.

4. Quantitative Results and Comparative Analysis

Key comparative results are as follows:

Metric	Correlation with F1 (Validation)	Macro-F1 Improvement (Test)
Brier Score	0.68–0.99	Baseline
Penalized Brier	0.01–0.26 higher (on avg.)	+0.0 to +7.1 points

The average Pearson correlation between F1 and BS ranged 0.68–0.99 depending on the dataset and fold. PBS exhibited uniformly higher correlation with F1 by 0.01–0.26, attributed to the penalty aligning PBS curves with the discrete F1 peaks. In held-out macro-F1, models selected by PBS outperformed those selected by BS/LL, yielding improvements up to 7.1 points (absolute). For example, on the driver identification (Casale2012), early stopping with BS resulted in F1 ≈ 45.00%, whereas with PBS, F1 ≈ 51.65% (+6.65%). These gains were consistent across nearly all datasets, demonstrating that PBS is aligned more closely with the practical objective of accuracy maximization than the standard Brier Score (Ahmadian et al., 2024).

5. Implementation, Usage, and Integration

A Python-like pseudocode for PBS is provided:

import numpy as np def penalizing(q, y, penalty): # q, y: (n_samples, c) # y is one-hot; penalty is scalar (e.g. (c-1)/c) wrong = (q.argmax(axis=1) != y.argmax(axis=1)) return penalty * wrong.astype(float) def PBS(q, y): # q, y shape (n, c) n, c = q.shape bs_sample = np.sum((q - y)**2, axis=1) penalty = (c - 1) / c payoffs = penalizing(q, y, penalty) return np.mean(bs_sample + payoffs)

Integration practice:

• For validation, compute $PBS$ at the end of each epoch.
• Use $PBS$ as the optimization criterion for checkpointing (save model if validation $PBS$ improves) and for early stopping (halt if no improvement in $PBS$ for a set patience).
• Log $PBS$ concurrently with $BS$, $F_1$, and cross-entropy during evaluation to monitor convergence and calibration with discrete accuracy-oriented metrics.
• While $PBS$ can serve as a training objective, empirical focus has been on evaluation and model selection (Ahmadian et al., 2024).

6. Interpretations, Strengths, and Practical Implications

The penalized nature of PBS ("superiority" property) ensures that model selection systematically favors correctly classified samples over incorrectly classified ones—addressing a documented failure mode of classical proper scoring rules, which can, in certain miscalibrated scenarios, rank an incorrect prediction as better than a correct one. The strict propriety and class-calibrated penalty jointly guarantee that probability forecasts are both honest (well-calibrated) and accuracy-oriented (aligned with F1 improvements).

Empirically, PBS provides more reliable model selection for challenging, temporally dependent multi-class tasks common in sensor-based recognition, time-series classification, and related domains. A plausible implication is that PBS may be advantageous in any setting where the cost of incorrect dominant-label prediction materially exceeds the penalty for suboptimal probability mass allocation among non-true classes. The method obviates the need for penalty hyperparameter tuning, and is easily integrated into standard deep learning workflows for evaluation, model checkpointing, and early stopping.

For further formalism, experimental analysis, and derivation details, see Ahmadian et al., "Superior Scoring Rules for Probabilistic Evaluation of Single-Label Multi-Class Classification Tasks" (Ahmadian et al., 2024).