---
title: Predictive E-measures
url: https://www.emergentmind.com/topics/predictive-e-measures
type: topic
---

# Predictive E-measures

Searching arXiv for the cited papers and closely related uses of “predictive E-measures.”
[arXiv search] query: "1904.04896 Performance Monitoring for End-to-End Speech Recognition 2605.01796 Beyond ECE 2301.01335 The E-Posterior 2502.04294 Prediction-Powered E-Values 2604.20788 The E-measure"
Predictive E-measures are quantitative constructs used to assess prediction quality, predictive uncertainty, calibration, discriminative value, or evidential strength from model outputs, often without relying exclusively on conventional held-out error summaries. In the papers considered here, the term is attached to several different technical constructions: entropy-based uncertainty metrics for end-to-end speech recognition [1904.04896], risk-aware calibration and confidence-weighted evaluation beyond Expected Calibration Error [2605.01796], e-value and e-measure frameworks for prediction and decision under optional stopping [2301.01335; 2502.04294; 2604.20788], interpretable meta-scores for model predictive power [2006.02293; 1908.09213], and domain-specific predictive indicators in software engineering, smart grids, macroeconomics, and distributional concordance analysis [1207.2237; 1406.0223; 2503.09839; 2606.14382].

## 1. Terminological scope

The literature surveyed here indicates that “predictive E-measures” is not a single standardized object. Instead, it names several families of measures that share a predictive role but differ in mathematical form, inferential target, and operational semantics. In some settings the “E” refers to entropy; in others it refers to evidential quantities such as e-values and e-measures; elsewhere it denotes evaluation or meta-measurement of predictive performance.

| Usage | Core quantity | Representative papers |
|---|---|---|
| End-to-end ASR performance monitoring | Entropy-based uncertainty score over decoder or attention distributions | [1904.04896] |
| Calibration and discrimination evaluation | CSR, $P_{\mathrm{risk}}$, cwA, cwAUC, Murphy decomposition terms | [2605.01796], [2605.12679] |
| Evidential prediction and decision | e-values, e-processes, e-posterior risk bounds, predictive E-measure kernels | [2301.01335], [2502.04294], [2604.20788] |
| Benchmark meta-scores | Elo-based Predictive Power from pairwise win probabilities | [2006.02293], [1908.09213] |
| Application-sensitive predictive evaluation | Holistic measures of scale independence, reliability, volatility, and cost | [1406.0223] |
| Domain-specific early predictors | Z-specification predictors, recession indicators, distributional concordance | [1207.2237], [2503.09839], [2606.14382] |

A recurrent misconception is that all predictive E-measures belong to a unified formalism. The papers do not support that view. They instead document a family resemblance: each construction converts predictive behavior into a compact quantitative object that is intended to be interpretable for monitoring, comparison, or sequential decision making.

## 2. Entropy-based predictive E-measures in end-to-end speech recognition

In end-to-end ASR, predictive E-measures are entropy-based uncertainty metrics computed from token-level probability outputs and used to predict recognition quality without ground truth [1904.04896]. The motivating problem is performance monitoring under domain shift, where recognition quality can be highly inconsistent but reference transcripts are unavailable.

The paper adapts classic performance monitoring ideas to a joint CTC/attention architecture. At decoder step $l$, the decoder emits a posterior vector $p_l = p(c_l \mid c_1,\dots,c_{l-1},X)$ over an output alphabet of size $K=52$, and the attention mechanism emits a frame-distribution $a_l$ over $T$ encoder time steps. The per-step decoder entropy is
$$
H_l = - \sum_{k=1}^{K} p_l^{(k)} \log p_l^{(k)},
$$
and the utterance-level decoder E-score is
$$
E_{\mathrm{score}} = \frac{1}{L}\sum_{l=1}^{L} H_l.
$$
For attention, the paper uses
$$
H_l^{\mathrm{att}} = - \sum_{t=1}^{T} a_{l,t}\log a_{l,t},
\qquad
E_{\mathrm{score}}^{\mathrm{att}} = \frac{1}{L}\sum_{l=1}^{L}\frac{H_l^{\mathrm{att}}}{\log T},
$$
where normalization by $\log T$ places entropy in $[0,1]$ across utterances with different $T$ [1904.04896].

This entropy-based E-measure is evaluated together with Mean Character Distance (MCD), autoencoder reconstruction error, and a learned BLSTM regressor. MCD is the end-to-end adaptation of the hybrid-ASR M-measure: it replaces frame-level temporal distances by symmetric KL divergences between decoder posteriors across small output-step offsets $\Delta l \in \{1,2,3,4,5\}$. The resulting score is
$$
\mathcal{M}_{\mathrm{score}} = \frac{1}{N_{\mathrm{pairs}}}\sum_{\Delta l=1}^{5}\sum_{l=\Delta l}^{L}\mathcal{D}(p_{l-\Delta l},p_l),
$$
with $\mathcal{D}$ the symmetric KL divergence and $N_{\mathrm{pairs}}=\sum_{\Delta l=1}^{5}(L-\Delta l)$ [1904.04896].

All measures are mapped to predicted CER through a linear regressor
$$
\hat y = \alpha + \beta s,
$$
trained by least squares on development data from WSJ, Aurora4, and CHiME4-Sim. The ASR system itself uses an ESPnet/PyTorch joint CTC/attention model with 4 BLSTM encoder layers, content-based attention, a 1-layer unidirectional LSTM decoder, 52 output labels, beam width 30, unigram label smoothing, and no external language model [1904.04896].

The central empirical result is that decoder-level features consistently outperform attention-level probabilities for predictive monitoring. On all test utterances, the reported MSEs $\times 10^{-2}$ are 1.02 for decoder entropy, 3.83 for attention entropy, 0.79 for decoder MCD, 1.67 for attention MCD, 2.49 for the autoencoder, and 1.02 for the RNN predictor. The best overall average predictive accuracy is therefore achieved by decoder-based MCD, with average prediction error about $8.8\%$ RMSE, while decoder entropy and the RNN predictor are both about $10.1\%$ RMSE [1904.04896].

The interpretation offered in the paper is structurally specific. Decoder posteriors directly reflect uncertainty over output symbols while integrating acoustic evidence and the implicit language model; attention weights are alignment distributions that can remain sharp even when symbol choices are wrong. The paper also notes an important limitation: entropy and MCD assume a monotonic relationship with error, so an overconfident but wrong decoder can exhibit low entropy despite high CER [1904.04896].

## 3. Calibration, miscalibration, and discrimination as predictive evaluation

A second lineage treats predictive E-measures as diagnostics for confidence calibration and discriminative usefulness. In classification, the baseline object is Expected Calibration Error,
$$
\mathrm{ECE}=\sum_{m=1}^{M}\frac{|B_m|}{N}\,\big|\mathrm{acc}(B_m)-\mathrm{conf}(B_m)\big|,
$$
which is a bin-based average of absolute calibration gaps [2605.01796]. The paper argues that ECE is “linear” in the sense that it counts a gap $\delta$ identically regardless of the confidence level where it occurs, and proves an orthogonality result: low ECE can coexist with arbitrarily large overconfidence risk [2605.01796].

The proposed replacement for tail-risk assessment is the Calibrated Size Ratio,
$$
\mathrm{CSR}=\frac{1}{N}\sum_{i:\,\hat y_i\neq y_i}\frac{1}{1-\mathrm{conf}_i},
$$
with $\mathrm{CSR}=1$ under perfect pointwise calibration, $\mathrm{CSR}>1$ indicating overconfidence on errors, and $\mathrm{CSR}<1$ indicating underconfidence on errors [2605.01796]. The associated overconfidence risk probability $P_{\mathrm{risk}}$ is obtained from a Gaussian approximation based on the deviation of CSR from 1. The paper recommends treating $z\ge 1$ as moderate evidence and $z\ge 3$ as strong evidence that the profile is overconfident [2605.01796].

Calibration alone is not treated as sufficient. The same paper argues that risk assessment must be complemented by a measure of discriminative value. Confidence-weighted accuracy is
$$
\mathrm{cwA}=\frac{\sum_{i=1}^{N}\mathrm{conf}_i\,\mathbf{1}[\hat y_i=y_i]}{\sum_{i=1}^{N}\mathrm{conf}_i},
$$
and confidence weighting extends to the full confusion matrix and hence to precision, recall, specificity, F1, balanced accuracy, MCC, and AUC-like quantities [2605.01796]. Classical AUC is invariant to strictly monotone score recalibration and therefore cannot encode calibration information; cwAUC is designed to incorporate confidence magnitudes through pair weights, so the gap cwAUC$-$AUC reflects whether high-confidence pairs are also correctly ranked [2605.01796].

The empirical comparison across fifteen real datasets is explicitly risk-sensitive. Isotonic calibration attained the highest average cwA, but it also produced catastrophic CSR values on several datasets; Platt scaling was the safest regime, with fewer one-sigma and three-sigma CSR exceedances while maintaining strong cwA improvements [2605.01796]. This directly supports the claim that minimizing ECE alone is insufficient for evaluating predictive confidence profiles.

For real-valued point prediction, an analogous decomposition is given in terms of mean-consistent Bregman losses. The expected loss satisfies Murphy’s decomposition
$$
S_L(Y,X)=\mathrm{UNC}_L(Y)-\mathrm{DSC}_L(Y,X)+\mathrm{MCB}_L(Y,X),
$$
with predictor-only forms
$$
\mathrm{DSC}_L(Y,X)=E[L(E[Y\mid X],E[Y])],\qquad
\mathrm{MCB}_L(Y,X)=E[L(E[Y\mid X],X)].
$$
Here discrimination is the distance from the flat predictor $E[Y]$, and miscalibration is the distance from the mean-calibrated predictor $E[Y\mid X]$ [2605.12679]. The paper then shows that Lorenz-curve-based measures such as ABC, ABC$^2$, and the Gini score depend on predictor-dependent weights and can therefore lead to dishonest evaluation when used for model selection [2605.12679].

Taken together, these papers recast predictive evaluation as a two-part problem. One part concerns whether reported confidences or point predictions are statistically consistent with outcomes; the other concerns whether those confidences or predictions are operationally useful for distinguishing correct from incorrect or informative from uninformative decisions.

## 4. E-values, e-posteriors, and predictive evidence

A third lineage uses “predictive E-measures” in the evidential sense of e-values and e-measures. Under a null hypothesis $H_0$, an e-value is a nonnegative random variable $E$ satisfying
$$
\mathbb{E}_{H_0}[E]\le 1.
$$
If $(E_t)$ is a nonnegative supermartingale or, more generally, an e-process, Ville’s inequality yields anytime-valid testing:
$$
\mathbb{P}_{H_0}\!\left(\sup_{t\ge 0}E_t\ge \frac{1}{\alpha}\right)\le \alpha
$$
[2502.04294; 2209.00991]. This is the key difference from classical fixed-time p-value logic: optional stopping, post-hoc thresholding, and sequential accumulation are built into the validity notion.

Within the e-posterior framework, the primitive object is an e-collection $\{S_\theta:\theta\in\Theta\}$ with $\mathbb{E}_{P_\theta}[S_\theta]\le 1$. The e-posterior is defined by reciprocal inversion,
$$
\bar P(\theta\mid y)=\frac{1}{S_\theta(y)},
$$
and a decision rule is assessed through the e-posterior risk envelope
$$
\bar R(y,a)=\sup_{\theta\in\Theta}\bar P(\theta\mid y)\,L(\theta,a).
$$
The defining guarantee is
$$
\mathbb{E}_{Y\sim P_\theta}\!\left[\frac{L(\theta,\delta(Y))}{\bar R(\delta)}\right]\le 1,
$$
so prior misspecification loosens the bound rather than invalidating it [2301.01335]. In this sense, predictive e-measures are loss-specific stochastic upper bounds with frequentist validity.

Prediction-Powered E-Values extend the same logic to scarce-label settings. If a base procedure factorizes into predictable one-step e-factors $e_i(Y_i)$, and if labels are queried according to $\xi_i\sim\mathrm{Bern}(\pi_i(X_i))$, the prediction-powered factor is
$$
e_i^{\mathrm{ppi}}
=
e_i(\mu_i(X_i))
+
\bigl(e_i(Y_i)-e_i(\mu_i(X_i))\bigr)\frac{\xi_i}{\pi_i(X_i)}.
$$
The cumulative predictive e-value is
$$
E_n^{\mathrm{ppi}}=\prod_{i=1}^{n}e_i^{\mathrm{ppi}}.
$$
The paper shows that if the base process is a valid e-value, test supermartingale, or e-process, the prediction-powered version inherits the same validity; power degrades gracefully with prediction error through a Wasserstein-type bound on expected log-growth [2502.04294]. The framework is applied to bounded-mean testing, online risk monitoring, change-point detection, and causal discovery [2502.04294].

“The E-measure” then generalizes the single-hypothesis e-value to a hypothesis class. An E-measure $\e$ satisfies the closure rule
$$
\e\!\left(\bigcup_{H\in\mathcal S}H\right)=\inf_{H\in\mathcal S}\e(H),
$$
so evidence is closed under infima rather than additive unions [2604.20788]. On intersection-closed classes, the paper shows that E-measures are the only non-dominated valid evidence assignments, and that familywise evidence and false evidence rate can be controlled without multiplicity correction [2604.20788]. In the predictive setting, validity reduces to a least-hypothesis object $H_x$ with
$$
\sup_{H\ni x}\e(H\mid x)=\e(H_x\mid x),
$$
so a single valid E-variable suffices to construct a full predictive E-measure kernel [2604.20788].

The same sequential-evidence architecture underlies e-backtesting of financial risk forecasts. For VaR and ES, per-time backtest e-statistics are derived from identification functions and multiplied into e-processes, producing standard and comparative backtests with anytime-valid type-I control under optional stopping; later work extends this to mean, variance, VaR, ES, and expectiles [2209.00991; 2511.05840].

## 5. Meta-scores and application-sensitive predictive evaluation

Another use of predictive E-measures is meta-evaluation: instead of scoring single predictions, the measure scores models or benchmarking procedures themselves. Elo-based Predictive Power (EPP) is built from pairwise comparisons of models across rounds such as CV splits or dataset-tasks. If $y_{ij,k}=1$ indicates that model $i$ beats model $j$ in round $k$, then EPP assigns ratings $\beta_i$ through a Bradley–Terry-type logit model
$$
\logit P(y_{ij,k}=1)=\beta_i-\beta_j.
$$
Differences $\Delta_{ij}=\beta_i-\beta_j$ have immediate probabilistic meaning:
$$
\hat p_{i>j}=\mathrm{invlogit}(\Delta_{ij}),
$$
and the mean-zero identification constraint yields an “average Player” with probability-to-beat anchor
$$
\hat p_{i>\mathrm{avg}}=\mathrm{invlogit}(\beta_i).
$$
Deviance relative to a saturated pairwise-outcome model provides a goodness-of-fit diagnostic for the leaderboard [2006.02293; 1908.09213].

The substantive claim of EPP is not that it supersedes base metrics such as AUC or RMSE, but that it places model comparisons on an interval log-odds scale that is comparable across datasets and captures performance stability across folds. This is why the papers emphasize navigated hyperparameter tuning, probabilistic interpretation of score differences, and dataset embeddings derived from EPP vectors [2006.02293; 1908.09213].

In smart-grid forecasting, predictive E-measures are instead organized as an application-sensitive suite spanning scale independence, reliability, volatility, and cost. The application-independent measures include MAPE, CVRMSE, Relative Improvement Measure (RIM), Volatility-Adjusted Benefit (VAB), Computation Cost (CC), and Data-collection Cost (CD); the application-dependent measures are Domain Bias Percentage Error (DBPE), Reliability Threshold Estimate (REL), Total Compute Cost (TCC), and Cost–Benefit Measure (CBM) [1406.0223]. DBPE uses an asymmetric linlin loss with parameters $\alpha$ and $\beta$, REL measures how often the relative error falls within a user-specified threshold $e_t$, and CBM combines accuracy with compute expenditure through
$$
\mathrm{CBM}=\frac{1-\mathrm{DBPE}/100}{\mathrm{TCC}}
$$
[1406.0223].

The general lesson is that predictive evaluation depends on downstream use. Frequency of beating a baseline, stability of that advantage, asymmetry of over- versus under-prediction, and data or compute requirements may all be decisive even when conventional error summaries are similar.

## 6. Domain-specific predictive indicators, concordance measures, and recurring limitations

Several papers apply predictive E-measures to concrete prediction problems outside generic model evaluation. In software engineering, structural and semantic metrics extracted from Z specifications are used as early predictors of Ada implementation size and complexity. Conceptual Complexity $CC(\psi)$, logical complexity bounds $v'_l(\psi)$ and $v'_u(\psi)$, Definition-Use Count, Use Count, Definition Count, And-Count, Or-Count, slicing-based Overlap $O(\psi)$, and schema coupling $\chi(\Psi,\psi)$ are correlated with code measures such as CL, CLCE, CYC, KNOTS, and FOUT. The resulting regression models achieve adjusted $R^2$ values from $0.62$ to $0.84$, indicating that specification-level measures can act as early predictors of downstream effort and quality proxies [1207.2237].

In macroeconomic recession forecasting, the E-Rule is a composite of the 10-year minus 2-year Treasury spread
$$
S_t=y_{10y,t}-y_{2y,t}
$$
and the Sahm statistic
$$
SR_t=\bar u_t^{(3)}-\min_{s\in\{t-11,\dots,t\}}\bar u_s^{(3)}.
$$
The paper evaluates recession signaling by near-zero bands for the undisclosed composite $E_t$, with $[-0.3,0.3]$ preferred over $[-0.2,0.2]$. In the reported threshold simulations, the wider band achieves $92.7\%$ accuracy over 1,000 scenarios with 927 true positives, 73 false negatives, and 0 false positives. A central limitation is explicit in the paper: the exact functional form $E_t=f(S_t,SR_t)$ is not disclosed [2503.09839].

For distributional concordance, predictive E-measures take yet another form. The expected misclassification rate between a reference distribution $p_0$ and a candidate $p_1$ under priors $(\pi_0,\pi_1)$ is
$$
\mathrm{EMR}(p_0,p_1;\pi_0,\pi_1)=\int \min\{\pi_0p_0(x),\pi_1p_1(x)\}\,dx.
$$
With equal priors, $\mathrm{EMR}=\frac{1}{2}(1-\mathrm{TV}(p_0,p_1))$. The paper then works operationally with a closely related bounded concordance
$$
\mathrm{EMR}^{\star}(p_0,p_1)=\int \frac{p_0(x)p_1(x)}{p_0(x)+p_1(x)}\,dx,
$$
which is symmetric, lies in $[0,1/2]$, and has strict concavity properties for finite mixtures [2606.14382]. In mixture synthesis, these properties yield exact gradients and Hessians, strict concavity in mixture weights, and a unique maximizer, while a synthetic Dirichlet prior induces a log-barrier regularizer for stabilizing weights [2606.14382].

Across these disparate usages, several limitations recur. Some measures require calibration data and degrade under distribution shift, as with linear CER mappings in ASR and learned RNN predictors in unseen reverberant conditions [1904.04896]. Others can be numerically unstable near confidence one and therefore require clipping, as with CSR and related variance terms [2605.01796]. Some descriptive measures are informative but unsuitable for model selection because they depend on predictor-specific weights, as shown for ABC, ABC$^2$, and Gini [2605.12679]. And some proposed domain indicators remain only partially reproducible because the decisive transformation is omitted from the published description, as in the E-Rule [2503.09839].

The broad significance of predictive E-measures therefore lies less in a single formal definition than in a shared methodological role. Each construction attempts to compress predictive behavior into an interpretable scalar, curve, or process that is aligned with a particular inferential task: runtime monitoring without ground truth, calibration and tail-risk auditing, anytime-valid sequential evidence, cross-dataset model comparison, application-sensitive deployment evaluation, or early-warning detection in domain-specific forecasting.

Source: https://www.emergentmind.com/topics/predictive-e-measures