Predictive E-measures

• Predictive E-measures are quantitative constructs that aggregate model prediction behavior into interpretable metrics for performance monitoring, risk assessment, and evidential decision making.
• They encompass various techniques such as entropy-based uncertainty in end-to-end ASR, advanced calibration methods surpassing ECE, and e-value frameworks for anytime-valid testing.
• While facilitating cross-dataset model evaluation and application-specific analysis, these measures face challenges like dependency on calibration data, sensitivity to distribution shift, and numerical instability near extreme confidences.

Searching arXiv for the cited papers and closely related uses of “predictive E-measures.” [arXiv search] query: "(Li et al., 2019) Performance Monitoring for End-to-End Speech Recognition (Martin-Maroto et al., 3 May 2026) Beyond ECE (Grünwald, 2023) The E-Posterior (Csillag et al., 6 Feb 2025) Prediction-Powered E-Values (Koning, 22 Apr 2026) The E-measure" Predictive E-measures are quantitative constructs used to assess prediction quality, predictive uncertainty, calibration, discriminative value, or evidential strength from model outputs, often without relying exclusively on conventional held-out error summaries. In the papers considered here, the term is attached to several different technical constructions: entropy-based uncertainty metrics for end-to-end speech recognition (Li et al., 2019), risk-aware calibration and confidence-weighted evaluation beyond Expected Calibration Error (Martin-Maroto et al., 3 May 2026), e-value and e-measure frameworks for prediction and decision under optional stopping (Grünwald, 2023, Csillag et al., 6 Feb 2025, Koning, 22 Apr 2026), interpretable meta-scores for model predictive power (Gosiewska et al., 2020, Gosiewska et al., 2019), and domain-specific predictive indicators in software engineering, smart grids, macroeconomics, and distributional concordance analysis [(Bollin et al., 2012); (Aman et al., 2014); (Ebadi, 12 Mar 2025); (Adrian et al., 12 Jun 2026)].

1. Terminological scope

The literature surveyed here indicates that “predictive E-measures” is not a single standardized object. Instead, it names several families of measures that share a predictive role but differ in mathematical form, inferential target, and operational semantics. In some settings the “E” refers to entropy; in others it refers to evidential quantities such as e-values and e-measures; elsewhere it denotes evaluation or meta-measurement of predictive performance.

Usage	Core quantity	Representative papers
End-to-end ASR performance monitoring	Entropy-based uncertainty score over decoder or attention distributions	(Li et al., 2019)
Calibration and discrimination evaluation	CSR, $P_{\mathrm{risk}}$, cwA, cwAUC, Murphy decomposition terms	(Martin-Maroto et al., 3 May 2026, Delong et al., 12 May 2026)
Evidential prediction and decision	e-values, e-processes, e-posterior risk bounds, predictive E-measure kernels	(Grünwald, 2023, Csillag et al., 6 Feb 2025, Koning, 22 Apr 2026)
Benchmark meta-scores	Elo-based Predictive Power from pairwise win probabilities	(Gosiewska et al., 2020, Gosiewska et al., 2019)
Application-sensitive predictive evaluation	Holistic measures of scale independence, reliability, volatility, and cost	(Aman et al., 2014)
Domain-specific early predictors	Z-specification predictors, recession indicators, distributional concordance	(Bollin et al., 2012, Ebadi, 12 Mar 2025, Adrian et al., 12 Jun 2026)

A recurrent misconception is that all predictive E-measures belong to a unified formalism. The papers do not support that view. They instead document a family resemblance: each construction converts predictive behavior into a compact quantitative object that is intended to be interpretable for monitoring, comparison, or sequential decision making.

2. Entropy-based predictive E-measures in end-to-end speech recognition

In end-to-end ASR, predictive E-measures are entropy-based uncertainty metrics computed from token-level probability outputs and used to predict recognition quality without ground truth (Li et al., 2019). The motivating problem is performance monitoring under domain shift, where recognition quality can be highly inconsistent but reference transcripts are unavailable.

The paper adapts classic performance monitoring ideas to a joint CTC/attention architecture. At decoder step $l$, the decoder emits a posterior vector $p_l = p(c_l \mid c_1,\dots,c_{l-1},X)$ over an output alphabet of size $K=52$, and the attention mechanism emits a frame-distribution $a_l$ over $T$ encoder time steps. The per-step decoder entropy is

$H_l = - \sum_{k=1}^{K} p_l^{(k)} \log p_l^{(k)},$

and the utterance-level decoder E-score is

$$E_{\mathrm{score}} = \frac{1}{L}\sum_{l=1}^{L} H_l.$$

For attention, the paper uses

$$H_l^{\mathrm{att}} = - \sum_{t=1}^{T} a_{l,t}\log a_{l,t}, \qquad E_{\mathrm{score}}^{\mathrm{att}} = \frac{1}{L}\sum_{l=1}^{L}\frac{H_l^{\mathrm{att}}}{\log T},$$

where normalization by $\log T$ places entropy in $l$0 across utterances with different $l$1 (Li et al., 2019).

This entropy-based E-measure is evaluated together with Mean Character Distance (MCD), autoencoder reconstruction error, and a learned BLSTM regressor. MCD is the end-to-end adaptation of the hybrid-ASR M-measure: it replaces frame-level temporal distances by symmetric KL divergences between decoder posteriors across small output-step offsets $l$2. The resulting score is

$l$3

with $l$4 the symmetric KL divergence and $l$5 (Li et al., 2019).

All measures are mapped to predicted CER through a linear regressor

$l$6

trained by least squares on development data from WSJ, Aurora4, and CHiME4-Sim. The ASR system itself uses an ESPnet/PyTorch joint CTC/attention model with 4 BLSTM encoder layers, content-based attention, a 1-layer unidirectional LSTM decoder, 52 output labels, beam width 30, unigram label smoothing, and no external LLM (Li et al., 2019).

The central empirical result is that decoder-level features consistently outperform attention-level probabilities for predictive monitoring. On all test utterances, the reported MSEs $l$7 are 1.02 for decoder entropy, 3.83 for attention entropy, 0.79 for decoder MCD, 1.67 for attention MCD, 2.49 for the autoencoder, and 1.02 for the RNN predictor. The best overall average predictive accuracy is therefore achieved by decoder-based MCD, with average prediction error about $l$8 RMSE, while decoder entropy and the RNN predictor are both about $l$9 RMSE (Li et al., 2019).

The interpretation offered in the paper is structurally specific. Decoder posteriors directly reflect uncertainty over output symbols while integrating acoustic evidence and the implicit LLM; attention weights are alignment distributions that can remain sharp even when symbol choices are wrong. The paper also notes an important limitation: entropy and MCD assume a monotonic relationship with error, so an overconfident but wrong decoder can exhibit low entropy despite high CER (Li et al., 2019).

3. Calibration, miscalibration, and discrimination as predictive evaluation

A second lineage treats predictive E-measures as diagnostics for confidence calibration and discriminative usefulness. In classification, the baseline object is Expected Calibration Error,

$p_l = p(c_l \mid c_1,\dots,c_{l-1},X)$0

which is a bin-based average of absolute calibration gaps (Martin-Maroto et al., 3 May 2026). The paper argues that ECE is “linear” in the sense that it counts a gap $p_l = p(c_l \mid c_1,\dots,c_{l-1},X)$1 identically regardless of the confidence level where it occurs, and proves an orthogonality result: low ECE can coexist with arbitrarily large overconfidence risk (Martin-Maroto et al., 3 May 2026).

The proposed replacement for tail-risk assessment is the Calibrated Size Ratio,

$p_l = p(c_l \mid c_1,\dots,c_{l-1},X)$2

with $p_l = p(c_l \mid c_1,\dots,c_{l-1},X)$3 under perfect pointwise calibration, $p_l = p(c_l \mid c_1,\dots,c_{l-1},X)$4 indicating overconfidence on errors, and $p_l = p(c_l \mid c_1,\dots,c_{l-1},X)$5 indicating underconfidence on errors (Martin-Maroto et al., 3 May 2026). The associated overconfidence risk probability $p_l = p(c_l \mid c_1,\dots,c_{l-1},X)$6 is obtained from a Gaussian approximation based on the deviation of CSR from 1. The paper recommends treating $p_l = p(c_l \mid c_1,\dots,c_{l-1},X)$7 as moderate evidence and $p_l = p(c_l \mid c_1,\dots,c_{l-1},X)$8 as strong evidence that the profile is overconfident (Martin-Maroto et al., 3 May 2026).

Calibration alone is not treated as sufficient. The same paper argues that risk assessment must be complemented by a measure of discriminative value. Confidence-weighted accuracy is

$p_l = p(c_l \mid c_1,\dots,c_{l-1},X)$9

and confidence weighting extends to the full confusion matrix and hence to precision, recall, specificity, F1, balanced accuracy, MCC, and AUC-like quantities (Martin-Maroto et al., 3 May 2026). Classical AUC is invariant to strictly monotone score recalibration and therefore cannot encode calibration information; cwAUC is designed to incorporate confidence magnitudes through pair weights, so the gap cwAUC$K=52$0AUC reflects whether high-confidence pairs are also correctly ranked (Martin-Maroto et al., 3 May 2026).

The empirical comparison across fifteen real datasets is explicitly risk-sensitive. Isotonic calibration attained the highest average cwA, but it also produced catastrophic CSR values on several datasets; Platt scaling was the safest regime, with fewer one-sigma and three-sigma CSR exceedances while maintaining strong cwA improvements (Martin-Maroto et al., 3 May 2026). This directly supports the claim that minimizing ECE alone is insufficient for evaluating predictive confidence profiles.

For real-valued point prediction, an analogous decomposition is given in terms of mean-consistent Bregman losses. The expected loss satisfies Murphy’s decomposition

$K=52$1

with predictor-only forms

$K=52$2

Here discrimination is the distance from the flat predictor $K=52$3, and miscalibration is the distance from the mean-calibrated predictor $K=52$4 (Delong et al., 12 May 2026). The paper then shows that Lorenz-curve-based measures such as ABC, ABC$K=52$5, and the Gini score depend on predictor-dependent weights and can therefore lead to dishonest evaluation when used for model selection (Delong et al., 12 May 2026).

Taken together, these papers recast predictive evaluation as a two-part problem. One part concerns whether reported confidences or point predictions are statistically consistent with outcomes; the other concerns whether those confidences or predictions are operationally useful for distinguishing correct from incorrect or informative from uninformative decisions.

4. E-values, e-posteriors, and predictive evidence

A third lineage uses “predictive E-measures” in the evidential sense of e-values and e-measures. Under a null hypothesis $K=52$6, an e-value is a nonnegative random variable $K=52$7 satisfying

$K=52$8

If $K=52$9 is a nonnegative supermartingale or, more generally, an e-process, Ville’s inequality yields anytime-valid testing:

$a_l$0

(Csillag et al., 6 Feb 2025, Wang et al., 2022). This is the key difference from classical fixed-time p-value logic: optional stopping, post-hoc thresholding, and sequential accumulation are built into the validity notion.

Within the e-posterior framework, the primitive object is an e-collection $a_l$1 with $a_l$2. The e-posterior is defined by reciprocal inversion,

$a_l$3

and a decision rule is assessed through the e-posterior risk envelope

$a_l$4

The defining guarantee is

$a_l$5

so prior misspecification loosens the bound rather than invalidating it (Grünwald, 2023). In this sense, predictive e-measures are loss-specific stochastic upper bounds with frequentist validity.

Prediction-Powered E-Values extend the same logic to scarce-label settings. If a base procedure factorizes into predictable one-step e-factors $a_l$6, and if labels are queried according to $a_l$7, the prediction-powered factor is

$a_l$8

The cumulative predictive e-value is

$a_l$9

The paper shows that if the base process is a valid e-value, test supermartingale, or e-process, the prediction-powered version inherits the same validity; power degrades gracefully with prediction error through a Wasserstein-type bound on expected log-growth (Csillag et al., 6 Feb 2025). The framework is applied to bounded-mean testing, online risk monitoring, change-point detection, and causal discovery (Csillag et al., 6 Feb 2025).

“The E-measure” then generalizes the single-hypothesis e-value to a hypothesis class. An E-measure $T$0 satisfies the closure rule

$T$1

so evidence is closed under infima rather than additive unions (Koning, 22 Apr 2026). On intersection-closed classes, the paper shows that E-measures are the only non-dominated valid evidence assignments, and that familywise evidence and false evidence rate can be controlled without multiplicity correction (Koning, 22 Apr 2026). In the predictive setting, validity reduces to a least-hypothesis object $T$2 with

$T$3

so a single valid E-variable suffices to construct a full predictive E-measure kernel (Koning, 22 Apr 2026).

The same sequential-evidence architecture underlies e-backtesting of financial risk forecasts. For VaR and ES, per-time backtest e-statistics are derived from identification functions and multiplied into e-processes, producing standard and comparative backtests with anytime-valid type-I control under optional stopping; later work extends this to mean, variance, VaR, ES, and expectiles (Wang et al., 2022, Jiao et al., 8 Nov 2025).

5. Meta-scores and application-sensitive predictive evaluation

Another use of predictive E-measures is meta-evaluation: instead of scoring single predictions, the measure scores models or benchmarking procedures themselves. Elo-based Predictive Power (EPP) is built from pairwise comparisons of models across rounds such as CV splits or dataset-tasks. If $T$4 indicates that model $T$5 beats model $T$6 in round $T$7, then EPP assigns ratings $T$8 through a Bradley–Terry-type logit model

$T$9

Differences $H_l = - \sum_{k=1}^{K} p_l^{(k)} \log p_l^{(k)},$0 have immediate probabilistic meaning:

$H_l = - \sum_{k=1}^{K} p_l^{(k)} \log p_l^{(k)},$1

and the mean-zero identification constraint yields an “average Player” with probability-to-beat anchor

$H_l = - \sum_{k=1}^{K} p_l^{(k)} \log p_l^{(k)},$2

Deviance relative to a saturated pairwise-outcome model provides a goodness-of-fit diagnostic for the leaderboard (Gosiewska et al., 2020, Gosiewska et al., 2019).

The substantive claim of EPP is not that it supersedes base metrics such as AUC or RMSE, but that it places model comparisons on an interval log-odds scale that is comparable across datasets and captures performance stability across folds. This is why the papers emphasize navigated hyperparameter tuning, probabilistic interpretation of score differences, and dataset embeddings derived from EPP vectors (Gosiewska et al., 2020, Gosiewska et al., 2019).

In smart-grid forecasting, predictive E-measures are instead organized as an application-sensitive suite spanning scale independence, reliability, volatility, and cost. The application-independent measures include MAPE, CVRMSE, Relative Improvement Measure (RIM), Volatility-Adjusted Benefit (VAB), Computation Cost (CC), and Data-collection Cost (CD); the application-dependent measures are Domain Bias Percentage Error (DBPE), Reliability Threshold Estimate (REL), Total Compute Cost (TCC), and Cost–Benefit Measure (CBM) (Aman et al., 2014). DBPE uses an asymmetric linlin loss with parameters $H_l = - \sum_{k=1}^{K} p_l^{(k)} \log p_l^{(k)},$3 and $H_l = - \sum_{k=1}^{K} p_l^{(k)} \log p_l^{(k)},$4, REL measures how often the relative error falls within a user-specified threshold $H_l = - \sum_{k=1}^{K} p_l^{(k)} \log p_l^{(k)},$5, and CBM combines accuracy with compute expenditure through

$H_l = - \sum_{k=1}^{K} p_l^{(k)} \log p_l^{(k)},$6

(Aman et al., 2014).

The general lesson is that predictive evaluation depends on downstream use. Frequency of beating a baseline, stability of that advantage, asymmetry of over- versus under-prediction, and data or compute requirements may all be decisive even when conventional error summaries are similar.

6. Domain-specific predictive indicators, concordance measures, and recurring limitations

Several papers apply predictive E-measures to concrete prediction problems outside generic model evaluation. In software engineering, structural and semantic metrics extracted from Z specifications are used as early predictors of Ada implementation size and complexity. Conceptual Complexity $H_l = - \sum_{k=1}^{K} p_l^{(k)} \log p_l^{(k)},$7, logical complexity bounds $H_l = - \sum_{k=1}^{K} p_l^{(k)} \log p_l^{(k)},$8 and $H_l = - \sum_{k=1}^{K} p_l^{(k)} \log p_l^{(k)},$9, Definition-Use Count, Use Count, Definition Count, And-Count, Or-Count, slicing-based Overlap $$E_{\mathrm{score}} = \frac{1}{L}\sum_{l=1}^{L} H_l.$$0, and schema coupling $$E_{\mathrm{score}} = \frac{1}{L}\sum_{l=1}^{L} H_l.$$1 are correlated with code measures such as CL, CLCE, CYC, KNOTS, and FOUT. The resulting regression models achieve adjusted $$E_{\mathrm{score}} = \frac{1}{L}\sum_{l=1}^{L} H_l.$$2 values from $$E_{\mathrm{score}} = \frac{1}{L}\sum_{l=1}^{L} H_l.$$3 to $$E_{\mathrm{score}} = \frac{1}{L}\sum_{l=1}^{L} H_l.$$4, indicating that specification-level measures can act as early predictors of downstream effort and quality proxies (Bollin et al., 2012).

In macroeconomic recession forecasting, the E-Rule is a composite of the 10-year minus 2-year Treasury spread

$$E_{\mathrm{score}} = \frac{1}{L}\sum_{l=1}^{L} H_l.$$5

and the Sahm statistic

$$E_{\mathrm{score}} = \frac{1}{L}\sum_{l=1}^{L} H_l.$$6

The paper evaluates recession signaling by near-zero bands for the undisclosed composite $$E_{\mathrm{score}} = \frac{1}{L}\sum_{l=1}^{L} H_l.$$7, with $$E_{\mathrm{score}} = \frac{1}{L}\sum_{l=1}^{L} H_l.$$8 preferred over $$E_{\mathrm{score}} = \frac{1}{L}\sum_{l=1}^{L} H_l.$$9. In the reported threshold simulations, the wider band achieves $$H_l^{\mathrm{att}} = - \sum_{t=1}^{T} a_{l,t}\log a_{l,t}, \qquad E_{\mathrm{score}}^{\mathrm{att}} = \frac{1}{L}\sum_{l=1}^{L}\frac{H_l^{\mathrm{att}}}{\log T},$$0 accuracy over 1,000 scenarios with 927 true positives, 73 false negatives, and 0 false positives. A central limitation is explicit in the paper: the exact functional form $$H_l^{\mathrm{att}} = - \sum_{t=1}^{T} a_{l,t}\log a_{l,t}, \qquad E_{\mathrm{score}}^{\mathrm{att}} = \frac{1}{L}\sum_{l=1}^{L}\frac{H_l^{\mathrm{att}}}{\log T},$$1 is not disclosed (Ebadi, 12 Mar 2025).

For distributional concordance, predictive E-measures take yet another form. The expected misclassification rate between a reference distribution $$H_l^{\mathrm{att}} = - \sum_{t=1}^{T} a_{l,t}\log a_{l,t}, \qquad E_{\mathrm{score}}^{\mathrm{att}} = \frac{1}{L}\sum_{l=1}^{L}\frac{H_l^{\mathrm{att}}}{\log T},$$2 and a candidate $$H_l^{\mathrm{att}} = - \sum_{t=1}^{T} a_{l,t}\log a_{l,t}, \qquad E_{\mathrm{score}}^{\mathrm{att}} = \frac{1}{L}\sum_{l=1}^{L}\frac{H_l^{\mathrm{att}}}{\log T},$$3 under priors $$H_l^{\mathrm{att}} = - \sum_{t=1}^{T} a_{l,t}\log a_{l,t}, \qquad E_{\mathrm{score}}^{\mathrm{att}} = \frac{1}{L}\sum_{l=1}^{L}\frac{H_l^{\mathrm{att}}}{\log T},$$4 is

$$H_l^{\mathrm{att}} = - \sum_{t=1}^{T} a_{l,t}\log a_{l,t}, \qquad E_{\mathrm{score}}^{\mathrm{att}} = \frac{1}{L}\sum_{l=1}^{L}\frac{H_l^{\mathrm{att}}}{\log T},$$5

With equal priors, $$H_l^{\mathrm{att}} = - \sum_{t=1}^{T} a_{l,t}\log a_{l,t}, \qquad E_{\mathrm{score}}^{\mathrm{att}} = \frac{1}{L}\sum_{l=1}^{L}\frac{H_l^{\mathrm{att}}}{\log T},$$6. The paper then works operationally with a closely related bounded concordance

$$H_l^{\mathrm{att}} = - \sum_{t=1}^{T} a_{l,t}\log a_{l,t}, \qquad E_{\mathrm{score}}^{\mathrm{att}} = \frac{1}{L}\sum_{l=1}^{L}\frac{H_l^{\mathrm{att}}}{\log T},$$7

which is symmetric, lies in $$H_l^{\mathrm{att}} = - \sum_{t=1}^{T} a_{l,t}\log a_{l,t}, \qquad E_{\mathrm{score}}^{\mathrm{att}} = \frac{1}{L}\sum_{l=1}^{L}\frac{H_l^{\mathrm{att}}}{\log T},$$8, and has strict concavity properties for finite mixtures (Adrian et al., 12 Jun 2026). In mixture synthesis, these properties yield exact gradients and Hessians, strict concavity in mixture weights, and a unique maximizer, while a synthetic Dirichlet prior induces a log-barrier regularizer for stabilizing weights (Adrian et al., 12 Jun 2026).

Across these disparate usages, several limitations recur. Some measures require calibration data and degrade under distribution shift, as with linear CER mappings in ASR and learned RNN predictors in unseen reverberant conditions (Li et al., 2019). Others can be numerically unstable near confidence one and therefore require clipping, as with CSR and related variance terms (Martin-Maroto et al., 3 May 2026). Some descriptive measures are informative but unsuitable for model selection because they depend on predictor-specific weights, as shown for ABC, ABC$$H_l^{\mathrm{att}} = - \sum_{t=1}^{T} a_{l,t}\log a_{l,t}, \qquad E_{\mathrm{score}}^{\mathrm{att}} = \frac{1}{L}\sum_{l=1}^{L}\frac{H_l^{\mathrm{att}}}{\log T},$$9, and Gini (Delong et al., 12 May 2026). And some proposed domain indicators remain only partially reproducible because the decisive transformation is omitted from the published description, as in the E-Rule (Ebadi, 12 Mar 2025).

The broad significance of predictive E-measures therefore lies less in a single formal definition than in a shared methodological role. Each construction attempts to compress predictive behavior into an interpretable scalar, curve, or process that is aligned with a particular inferential task: runtime monitoring without ground truth, calibration and tail-risk auditing, anytime-valid sequential evidence, cross-dataset model comparison, application-sensitive deployment evaluation, or early-warning detection in domain-specific forecasting.