--- title: Bounded Accuracy Scoring Metric url: https://www.emergentmind.com/topics/bounded-accuracy-based-scoring-metric type: topic --- # Bounded Accuracy Scoring Metric A Bounded Accuracy-Based Scoring Metric, exemplified by the "Balance score," is a single-instance, strictly proper scoring rule for probability estimation models. Developed in response to limitations of existing metrics in probabilistic classification and calibration assessment, this metric is notable for boundedness, known optimum, calibration orientation, and evaluation granularity. Its primary application addresses win probability estimation models in esports, but it is positioned as suitable for general probability estimation contexts due to its robust theoretical and empirical properties [2309.06248]. ## 1. Formal Definition and Functional Form Let $n$ be the number of test instances, $\hat p_i\in[0,1]$ the predicted probability for instance $i$, $y_i\in\{0,1\}$ the observed label, and $\hat y_i = \mathbf{1}\{\hat p_i \geq 0.5\}$ the thresholded class decision. The Balance score averages a pointwise gain-and-loss function $f_{ba}(\hat p,y)$: \[ \text{Balance score} = \frac{1}{n} \sum_{i=1}^n f_{ba}(\hat p_i, y_i), \] where \[ f_{ba}(\hat p, y) = \begin{cases} 1 - \hat p, & \hat p \geq 0.5,\; y = 1 \\ \hat p, & \hat p < 0.5,\; y = 0 \\ -\hat p, & \hat p \geq 0.5,\; y = 0 \\ -1 + \hat p, & \hat p < 0.5,\; y = 1 \end{cases} \] Alternatively, using indicator notation: \[ f_{ba}(\hat p, y) = (2\,\mathbf{1}\{y = \hat y\} - 1)\left[\hat y (1 - \hat p) + (1 - \hat y)\hat p\right] \] This construction provides a geometric interpretation, where the score is linear in $\hat p$ on each region defined by the high/low-confidence regime and correctness. ## 2. Boundedness and Score Interpretation For all $\hat p, y$, $f_{ba}(\hat p, y)$ satisfies \[ -1 \leq f_{ba}(\hat p, y) \leq 1 \] ensuring the global boundedness $-1 \leq \text{Balance score} \leq 1$. The extremal values correspond to high-confidence, incorrect predictions ($f_{ba} = -1$ at $\hat p = 0$ or $1$ with a wrong label), and the best-case gain of $0.5$ occurs at $\hat p = 0.5$ when the decision is correct. The ideal value for a perfectly calibrated model is 0. ## 3. Desired Metric Properties A robust probability estimation metric is expected to satisfy criteria enumerated in Table 1 of [2309.06248]: | Property | Requirement | Satisfied by Balance | |-----------------------------|-----------------------------------------------------------------|----------------------| | Properness | Unique maximizer at true $p$ | Yes | | Single-instance evaluation | No grouping/binning required | Yes | | Known optimum | The perfectly calibrated value is specified | Yes | | Absolute scoring | Comparable across datasets | Yes | | No hyperparameters | Does not require e.g. number of bins | Yes | | Calibration measurement | Truly reflects calibration, not just discrimination | Yes | Properness is proven via the expected score $g(q; p) = p f_{ba}(q, 1) + (1-p) f_{ba}(q, 0)$, which has a unique zero at $q = p$ and deviation $|g(q; p)| = |q - p|$ otherwise. Calibration orientation comes from the fact that $f_{ba}$ yields a zero value if and only if the predicted and actual probabilities coincide. ## 4. Comparative Analysis: Brier Score and Expected Calibration Error ### Brier Score \[ \mathrm{Brier} = \frac{1}{n} \sum_i (\hat p_i - y_i)^2 \] The Brier score is strictly proper, free of hyperparameters, but its optimum depends on the underlying true probability distribution, precluding direct comparability across datasets. ### Expected Calibration Error (ECE) \[ \mathrm{ECE} = \sum_{m=1}^M \frac{|B_m|}{n} \Bigl| \overline{y}(B_m) - \overline{\hat p}(B_m) \Bigr| \] where $M$ is the number of bins and $B_m$ are bin assignments. ECE has a known optimum (0 for perfect calibration) and is absolute, but it requires selection of $M$ and does not operate at the single-instance level. ### Relationship to True ECE Given $\Pr(\hat Y=1|\hat P=p) = q_p$, the true expected calibration error is $\mathbb{E}[|q_p - p|]$. The Balance score approximates this without binning: for overconfident models, $\mathrm{True\, ECE} \approx -\text{Balance score}$, and for underconfident models, $\mathrm{True\, ECE} \approx +\text{Balance score}$. This bin-free property underpins its bias advantage over ECE. ## 5. Proof Sketches of Calibration and Discrimination The properness of the Balance score is established by: \[ g(q; p) = p f_{ba}(q, 1) + (1-p) f_{ba}(q, 0), \] yielding $g(p; p) = 0$ and $g(q; p) = q - p$ or $p - q$ for $q \neq p$, so the zero point is unique. The magnitude $|g(q; p)| = |q - p|$ is a direct measure of calibration error. For discrimination, $f_{ba}$ changes only at second order for small $\hat p$ perturbations that do not affect class decision, so ranking by Balance closely aligns with that by accuracy, but with penalization for systematic miscalibration. ## 6. Empirical Evaluation: Simulation and Real Data ### Synthetic Data When $p_i \sim \mathrm{Beta}(\alpha, \alpha)$ and $y_i \sim \mathrm{Bernoulli}(p_i)$, with $\hat p_i = p_i$ (optimal calibration), metrics behave as follows: - Accuracy fluctuates depending on the $p_i$ distribution (69–82%); - Brier score corresponds to spread but is not distribution-invariant (0.124–0.200); - ECE$(M=10)$ and Balance score both concentrate near 0, but Balance exhibits lower bias due to the absence of binning. ### Esports Data On 100,000 “League of Legends” matches (logistic regression model using 14 features): - Accuracy increases (65.6% → 73.6% → 79.8%) as the game proceeds, - Brier decreases (0.216 → 0.176 → 0.139), - ECE and Balance are consistently near zero for perfectly calibrated models: ECE ≈ 0.0068, Balance ≈ −0.0016 at 10 minutes. Only ECE and Balance are centered at zero for perfectly calibrated models, providing meaningful baseline anchoring, while accuracy and Brier drift with the underlying class difficulty distribution. ### Binning Sensitivity and Sample Size Synthetic overconfident models (bias 0.10 vs. 0.11) reveal that ECE's ranking can flip under different $M$, demonstrating sensitivity to binning specification. In contrast, Balance converges rapidly to the true ECE value with smaller $n$. ## 7. Practical Computation and Implementation Given prediction and label arrays, the Balance score can be computed in linear time without binning or sorting. The pseudocode is: ```python def compute_balance_score(p_hat, y): n = len(y) total = 0.0 for i in range(n): p = p_hat[i] yi = y[i] if p >= 0.5: if yi == 1: s = 1 - p else: s = -p else: if yi == 0: s = p else: s = -1 + p total += s return total / n ``` The Balance score returns an absolute, calibration-oriented evaluation with ideal value 0, strictly properness, and no hyperparameter sensitivity. All claims and evaluations are supported by simulation and real-world esports datasets as detailed in [2309.06248]. Source: https://www.emergentmind.com/topics/bounded-accuracy-based-scoring-metric