Prior-Calibrated Scoring

• Prior-Calibrated Scoring is a family of methods that adjust raw scores to align with an explicitly defined prior, ensuring numerical meaning for downstream decisions.
• It employs techniques such as Platt scaling, isotonic regression, and Bayesian intercept adjustments across varied settings like LLM selection, semisupervised estimation, and classifier calibration.
• The approach leads to significant performance improvements and efficiency gains in tasks ranging from code and math problem solving to precision-based evaluations under deployment shifts.

to=arxiv_search.search ＿奇米影视 天天中彩票在 】!【json {"^^^^^^^^4query4 Scoring^^^^^^^^4\4 OR SCATR OR ^^^^^^^^4\4 Prediction-Powered Inference^^^^^^^^4\4 OR ^^^^^^^^4\4 Square Calibration for Peer Review^^^^^^^^4\4 to=arxiv_search.search 微信里的天天中彩票 天天中彩票提现? 】【。】【”】【json {"^^^^^^^^4query4 OR (&&&^^^^4\4^^^^&&&) OR (&&&^^^^4 OR SCATR OR \4^^^^&&&) OR (&&&^^^^4 OR \4^^^^&&&) OR (Ferrer et al., 2024) OR (Siblini et al., 2019) OR (Brummer et al., 2013) OR (Giummolè et al., 2017) OR (Seo et al., 29 May 2026) OR (Brümmer et al., 2013) OR (&&&^^^^4\4query4^^^^&&&)","max_results":^^^^4\4 Prior-calibrated scoring is a family of procedures that modifies, learns, or evaluates a score so that it is aligned with an explicitly specified prior structure relevant to deployment. In recent work, that structure may be a base model’s own generative prior and the event that a sampled candidate is correct, the marginal outcome scale in semisupervised estimation, class prevalences under label shift, or a reference prevalence for precision-based metrics. The calibrated object therefore differs by setting—posterior correctness probability, PRESERVED_PLACEHOLDER_^^^^4query4^^^^, posterior class probability, calibrated log-likelihood ratio, or prior-invariant precision—but the shared objective is to make scores numerically meaningful for downstream selection, inference, or thresholded decision-making (&&&^^^^4query4^^^^&&&, &&&^^^^4\4^^^^&&&, &&&^^^^4 OR \4^^^^&&&, Siblini et al., 2019).

^^^^4\4^^^^. Core formulations

Across the literature, prior-calibrated scoring is defined by the target quantity to which a raw score is aligned. In Best-of-PRESERVED_PLACEHOLDER_^^^^4\4^^^^ selection for LLMs, the target is a correctness probability conditioned on the base model and a completed candidate. In semisupervised mean estimation, the target is the conditional mean PRESERVED_PLACEHOLDER_^^^^4 OR SCATR OR \4^^^^ of an externally learned score. In classifier calibration, the target is a posterior probability or a calibrated log-likelihood ratio under an explicit class prior. In prior-invariant metric design, the target is not the score itself but the evaluation functional, such as precision or PRESERVED_PLACEHOLDER_^^^^4 OR \4^^^^, re-expressed at a chosen reference prior (&&&^^^^4query4^^^^&&&, &&&^^^^4\4^^^^&&&, &&&^^^^4 OR \4^^^^&&&, Siblini et al., 2019).

Setting	Calibrated object	Representative expression
Best-of-$N$ LLM selection	$P(\text{correct} \mid x,y,M)$	$y^*=\arg\max_i s(x,y_i)$
Semisupervised estimation	$m(X)=c(s(X))$ targeting $E[Y\mid s(X)]$	$E[Y \mid m(X)] = m(X)$
Prior-shifted classification	Deployment posterior	PRESERVED_PLACEHOLDER_^^^^4\4query4^^^^
Precision-based evaluation	Precision at reference prior PRESERVED_PLACEHOLDER_^^^^4\4\4^^^^	PRESERVED_PLACEHOLDER_^^^^4\4 OR SCATR OR \4^^^^

A central distinction in this literature is between calibrating a score and calibrating the metric used to assess it. Some methods post-hoc transform a score so that its numerical value matches an outcome probability or mean. Others leave the score unchanged but evaluate it under deployment priors, costs, or prevalence ranges. This distinction is explicit in work on calibrated prediction-powered inference, classifier calibration under prior probability shift, and prior-invariant PR-based metrics (&&&^^^^4\4^^^^&&&, &&&^^^^4 OR \4^^^^&&&, Siblini et al., 2019).

^^^^4 OR SCATR OR \4^^^^. Best-of-PRESERVED_PLACEHOLDER_^^^^4\4 OR \4^^^^ selection and correctness scoring in LLMs

In the BoN setting, prior-calibrated scoring is formulated as a ranking problem over sampled candidates. Given prompt PRESERVED_PLACEHOLDER_^^^^4\44^^^^, the base model PRESERVED_PLACEHOLDER_^^^^4\4^^^^5 generates PRESERVED_PLACEHOLDER_^^^^4\4^^^^6, and selection uses

PRESERVED_PLACEHOLDER_^^^^4\4^^^^7

SCATR instantiates this by learning a lightweight scorer on the base model’s hidden representations so that PRESERVED_PLACEHOLDER_^^^^4\4^^^^8 approximates the probability that a candidate is correct, using the embedding of the last non-padding token at the penultimate layer and a shallow MLP with sigmoid output. The training labels come from objective evaluators: exact final answer match for math and pass/fail unit tests for code. The loss is weighted binary cross-entropy with class weighting, described as a Platt-style logistic calibration of hidden features to correctness probability (&&&^^^^4query4^^^^&&&).

This formulation is explicitly model-specific and domain-specific. The hidden state is treated as a distilled summary of the base model’s internal prior over the completed sequence, and the learned map PRESERVED_PLACEHOLDER_^^^^4\4^^^^9 is described as “prior-calibrated” to the particular model, domain, and sampling policy. SCATR also supports weighted majority voting for tasks with canonical final answers, using

PRESERVED_PLACEHOLDER_^^^^4 OR SCATR OR \4query4^^^^

so that agreement and calibrated confidence are aggregated jointly (&&&^^^^4query4^^^^&&&).

The method is positioned against two alternatives: token-logit confidence heuristics and learned process reward models. The paper reports that token-probability heuristics are often overconfident on incorrect responses and underconfident on correct ones, and in experiments these metrics frequently perform near random selection. Against such baselines, SCATR improves by up to PRESERVED_PLACEHOLDER_^^^^4 OR SCATR OR \4\4^^^^ percentage points on math and PRESERVED_PLACEHOLDER_^^^^4 OR SCATR OR \4 OR SCATR OR \4^^^^ points on coding. Relative to LoRA fine-tuning on the same calibration data, it achieves comparable accuracy with up to PRESERVED_PLACEHOLDER_^^^^4 OR SCATR OR \4 OR \4^^^^ fewer trainable parameters, with training latency reductions up to PRESERVED_PLACEHOLDER_^^^^4 OR SCATR OR \44^^^^ and inference latency reductions up to PRESERVED_PLACEHOLDER_^^^^4 OR SCATR OR \4^^^^5. Against PRM baselines, it is reported to improve accuracy by up to PRESERVED_PLACEHOLDER_^^^^4 OR SCATR OR \4^^^^6 points on math and PRESERVED_PLACEHOLDER_^^^^4 OR SCATR OR \4^^^^7 on coding while enabling up to PRESERVED_PLACEHOLDER_^^^^4 OR SCATR OR \4^^^^8 faster inference (&&&^^^^4query4^^^^&&&).

A notable feature is that no extra forward pass through the base transformer is required for feature extraction; the hidden embedding is collected during generation, and the additional scoring cost is an PRESERVED_PLACEHOLDER_^^^^4 OR SCATR OR \4^^^^9 pass through a tiny MLP. This makes prior-calibrated scoring in SCATR an inference-time ranking correction rather than a replacement for the base model’s generative procedure (&&&^^^^4query4^^^^&&&).

^^^^4 OR \4^^^^. Post-hoc calibration in semisupervised mean estimation

In semisupervised mean estimation, prior-calibrated scoring means post-processing a black-box score PRESERVED_PLACEHOLDER_^^^^4 OR \4query4^^^^ so that it aligns with the outcome scale before entering an augmented estimator. The calibration map PRESERVED_PLACEHOLDER_^^^^4 OR \4\4^^^^ is learned on a labeled sample, yielding PRESERVED_PLACEHOLDER_^^^^4 OR \4 OR SCATR OR \4^^^^, with the ideal property

PRESERVED_PLACEHOLDER_^^^^4 OR \4 OR \4^^^^

The paper studies linear calibration, PRESERVED_PLACEHOLDER_^^^^4 OR \44^^^^, and isotonic calibration, where PRESERVED_PLACEHOLDER_^^^^4 OR \4^^^^5 is constrained to be nondecreasing and is fitted by isotonic regression using the pool-adjacent-violators algorithm (&&&^^^^4\4^^^^&&&).

The downstream estimator is a pooled AIPW form,

PRESERVED_PLACEHOLDER_^^^^4 OR \4^^^^6

with known sampling fraction PRESERVED_PLACEHOLDER_^^^^4 OR \4^^^^7. Because the calibrator is fitted by least squares on the labeled data and the calibration class contains constants, the normal equations imply mean calibration, PRESERVED_PLACEHOLDER_^^^^4 OR \4^^^^8, so the calibrated plug-in estimator has an exact AIPW representation. For isotonic calibration, the paper proves a stronger full empirical calibration condition,

PRESERVED_PLACEHOLDER_^^^^4 OR \4^^^^9

for all measurable $N$^^^^4query4^^^^ (&&&^^^^4\4^^^^&&&).

The theoretical contribution is first-order optimality. Linear calibration is shown to be first-order equivalent to PPI++, which the paper identifies as AIPW with empirical efficiency maximization. Isotonic calibration yields an $N$^^^^4\4^^^^ rate of $N$^^^^4 OR SCATR OR \4^^^^, asymptotic linearity, and a variance no larger than that of AIPW based on any monotone transformation of the original score. The fitted isotonic score also admits a “calibeating” result: no further post-processing of the fitted isotonic score yields additional first-order efficiency gains for mean estimation (&&&^^^^4\4^^^^&&&).

Empirically, calibrated estimators often outperform raw PPI and are competitive with, or outperform, AIPW and PPI++. The paper reports reductions in RMSE in synthetic large-unlabeled regimes, competitive or superior performance on reproduced real-data benchmarks, and label savings of $N$^^^^4 OR \4^^^^–NN4 on the Human track and $N$5–$N$6 on the Correctness track in LLM evaluation at $N$7–$N$8 when calibration is applied to evaluator proxies (&&&^^^^4\4^^^^&&&).

4. Posterior probabilities, log-likelihood ratios, and prior shift

In binary classification, prior-calibrated scoring is the conversion of an arbitrary score $N$9 into a posterior probability or log-likelihood ratio that can be adjusted when class priors change. The basic Bayesian identity is

$P(\text{correct} \mid x,y,M)$^^^^4query4^^^^

where $P(\text{correct} \mid x,y,M)$^^^^4\4^^^^ is the log-likelihood ratio and $P(\text{correct} \mid x,y,M)$^^^^4 OR SCATR OR \4^^^^ is the class prior. Under prior probability shift, the deployment posterior is obtained by an intercept adjustment,

$P(\text{correct} \mid x,y,M)$^^^^4 OR \4^^^^

provided the class-conditional score distributions remain invariant (&&&^^^^4 OR \4^^^^&&&).

The classifier-calibration literature in this corpus compares Platt scaling, isotonic regression, histogram binning, and multi-score logistic calibration. Platt’s calibrator is described as having “very stable and acceptable performance,” especially at small sample sizes or exceptionally high AUC, while isotonic regression is robust and tuning-free but piecewise constant. The paper also introduces multi-score calibration, treating multiple classifier outputs as features for logistic calibration, and reports that multi-score calibration improves performance in the majority of experiments, including cybersecurity datasets (&&&^^^^4 OR \4^^^^&&&).

A separate theoretical strand concerns isotonic calibration by PAV. For binary pattern classifiers, PAV is shown to optimize calibration for all regular binary proper scoring rules, not only convex ones. The same piecewise-constant monotone mapping is optimal for all such rules, and for log-likelihood-ratio calibration the solution is prior independent: once the LLR mapping is calibrated, changing the deployment prior requires only a log-odds shift in the posterior, not recalibration of the mapping itself (Brummer et al., 2013).

In speaker recognition, prior weighting is pushed further. Prior-weighted logistic regression is generalized via a parametric family of proper scoring rules that emphasize different threshold regions. The paper shows how the scoring-rule parameters and application prior interact to weight operating points, and reports that for low false-alarm-rate regimes on NIST SRE’^^^^4\4 OR SCATR OR \4^^^^, scoring rules tailored to emphasize higher thresholds can yield better accuracy than logistic regression. This reframes prior-calibrated scoring as calibration to an application region, not merely to a global posterior scale (Brümmer et al., 2013).

A recurrent caution in these papers is that prior adjustment is valid only under prior probability shift. If the score generator changes, or if $P(\text{correct} \mid x,y,M)$4 and $P(\text{correct} \mid x,y,M)$5 move under domain shift, intercept-only correction is no longer sufficient and full recalibration is required (&&&^^^^4 OR \4^^^^&&&, Brummer et al., 2013).

5. Evaluation under deployment priors and proper scoring rules

Several papers argue that prior-calibrated scoring is as much an evaluation problem as a calibration problem. In the decision-theoretic treatment of posterior probabilities, expected proper scoring rules are presented as the principled measure of posterior quality. The paper argues that calibration metrics such as ECE assess only one aspect of posterior quality and should not be used as overall performance measures. Instead it recommends expected proper scoring rules, optionally normalized as NCE, NBS, or NRisk, and introduces calibration loss as the improvement in expected proper score after post-hoc calibration (Ferrer et al., 2024).

This perspective explicitly supports evaluation under deployment priors. The paper gives cross-entropy and Brier formulas parameterized by chosen priors, so that systems can be assessed under the target environment rather than the empirical test-set prevalence. It also notes that the standard odds or logit prior correction is the natural deployment-time update when priors differ between training and test (Ferrer et al., 2024).

A distinct but related line of work calibrates precision-based metrics themselves. Precision at threshold $P(\text{correct} \mid x,y,M)$6 can be written as

$P(\text{correct} \mid x,y,M)$7

Replacing $P(\text{correct} \mid x,y,M)$8 by a reference prior $P(\text{correct} \mid x,y,M)$9 yields calibrated precision,

$y^*=\arg\max_i s(x,y_i)$^^^^4query4^^^^

from which calibrated PR curves, calibrated AUC-PR, and calibrated $y^*=\arg\max_i s(x,y_i)$^^^^4\4^^^^ scores follow. This procedure leaves recall unchanged and is explicitly described as metric calibration rather than probability calibration (Siblini et al., 2019).

The clinical decision-support framework extends this idea to prevalence ranges and asymmetric costs. It proposes an adjusted variant of cross-entropy that averages cost-weighted performance over clinically relevant prevalence ranges in log-odds space. The work also proves that AUROC equals an average of thresholded accuracy over a model-induced label-shift distribution, which is used to argue that AUROC does not reflect clinician-chosen prevalences or error costs. In this formulation, prior-calibrated scoring means evaluating calibrated thresholded classifiers under explicit uncertainty about prevalence and domain-specific cost asymmetries (&&&^^^^4\4query4^^^^&&&).

A common misconception addressed in this group of papers is that good ranking is enough. The cited results dispute that view from multiple directions: AUROC is invariant to monotone transformations and thus blind to calibration, ECE does not measure overall posterior quality, and precision-based metrics vary mechanically with prevalence unless re-expressed at a reference prior (Ferrer et al., 2024, Siblini et al., 2019, &&&^^^^4\4query4^^^^&&&).

6. Structured and generative variants: peer review and streaming TTS

Prior-calibrated scoring also appears in settings where the “prior” is structural rather than probabilistic. In peer review calibration, least square calibration models reviewer-specific scoring functions $y^*=\arg\max_i s(x,y_i)$^^^^4 OR SCATR OR \4^^^^ applied to noisy perceptions of latent paper utilities:

$y^*=\arg\max_i s(x,y_i)$^^^^4 OR \4^^^^

Prior knowledge enters through the hypothesis class for $y^*=\arg\max_i s(x,y_i)$4—linear, monotone, convex, concave, or reviewer-specific mixtures—and through assignment-graph structure. In the noiseless linear case, the paper proves that perfect recovery up to a global affine transform is achieved if and only if there exists a doubly-connected component covering all items. This makes prior-calibrated scoring an unsupervised constrained-inference problem rather than a labeled post-hoc regression problem (&&&^^^^4 OR SCATR OR \4^^^^&&&).

The same paper emphasizes that the inside-noise formulation is crucial for tractability. With noise placed inside the scoring function, the linear and monotone variants become convex programs; if noise is placed outside, the problem becomes a matrix seriation variant that is NP-hard in general. Under accurate linear priors, the method attains perfect calibration in noiseless synthetic experiments, and in mixed reviewer classes knowledge of which reviewers are linear versus monotone produces large gains (&&&^^^^4 OR SCATR OR \4^^^^&&&).

In block-diffusion decoding for discrete speech tokens, prior-calibrated scoring is implemented as prior subtraction in log space. For masked position $y^*=\arg\max_i s(x,y_i)$5 at denoising step $y^*=\arg\max_i s(x,y_i)$6, the score is

$y^*=\arg\max_i s(x,y_i)$7

where $y^*=\arg\max_i s(x,y_i)$8 is a cached unconditional block prior computed from an all-masked block with conditioning zeroed. This is described as a PMI-style score intended to suppress the long-tail token frequency bias that otherwise makes parallel unmasking overcommit high-marginal codec tokens (Seo et al., 29 May 2026).

In standard read-speech benchmarks, PMI scoring and top-confidence ranking are nearly tied in WER under saturated compute, but the calibrated score enables early decoding with fewer denoising steps per block. For $y^*=\arg\max_i s(x,y_i)$9, average steps per block fall from $m(X)=c(s(X))$^^^^4query4^^^^ to $m(X)=c(s(X))$^^^^4\4^^^^ on LibriSpeech-PC and to $m(X)=c(s(X))$^^^^4 OR SCATR OR \4^^^^ on Seed-TTS, with near-constant quality. On EmergentTTS-Eval hard samples, WER drops from $m(X)=c(s(X))$^^^^4 OR \4^^^^ to $m(X)=c(s(X))$4, including a large gain in the Pronunciation category. Here prior-calibrated scoring is purely inference-time, requires no architectural modification, and is compatible with classifier-free guidance (Seo et al., 29 May 2026).

7. Theoretical extensions, design principles, and recurrent limitations

Two additional strands broaden the concept. In objective Bayesian inference with proper scoring rules, the posterior is built from a scoring-rule loss rather than a full likelihood,

$m(X)=c(s(X))$5

with curvature correction chosen to match the Godambe information. The associated reference prior is a Jeffreys-type prior proportional to $m(X)=c(s(X))$6, where $m(X)=c(s(X))$7 is the Godambe information matrix. In this setting, prior-calibrated scoring means aligning posterior curvature and prior choice to the geometry induced by the scoring rule rather than by the Fisher information of a full likelihood (Giummolè et al., 2017).

In the optimization-of-scoring-rules literature, prior calibration is framed as rewarding the value of refinement over a prior baseline. For mean elicitation, the expected gain is

$m(X)=c(s(X))$8

The paper characterizes the optimal one-dimensional rule as V-shaped at the prior mean $m(X)=c(s(X))$9, derives efficient linear programs for finite-support settings, and shows that in multidimensional rectangular spaces a max-over-separate rule achieves at least $E[Y\mid s(X)]$^^^^4query4^^^^ of optimal. It also shows that averaging separate per-task scores can be $E[Y\mid s(X)]$^^^^4\4^^^^-suboptimal (Hartline et al., 2020).

Taken together, these formulations suggest several recurring design principles. First, calibration is usually local to a model, domain, or application region rather than universal. SCATR is model-specific and domain-specific; semisupervised calibration targets the labeled outcome scale; prior-shift corrections assume stable class-conditional score distributions; and clinical adjusted log scores require clinically chosen prevalence ranges and costs (&&&^^^^4query4^^^^&&&, &&&^^^^4\4^^^^&&&, &&&^^^^4 OR \4^^^^&&&, &&&^^^^4\4query4^^^^&&&). Second, the best transformation class depends on structure: affine calibration is stable and first-order equivalent to PPI++, isotonic calibration has stronger monotone optimality guarantees, PAV gives rule-independent optimal isotonic solutions in binary settings, and structural constraints such as linearity, convexity, or double connectivity can be decisive in peer review (&&&^^^^4\4^^^^&&&, Brummer et al., 2013, &&&^^^^4 OR SCATR OR \4^^^^&&&).

The cited work also converges on a common set of failure modes. These include lack of access to intermediate hidden states in open-weight-only methods, domain shift beyond within-domain generalization, very small labeled sets, sparse overlap structures in peer review, overly flexible calibration classes, large block sizes in block diffusion, and deployment shifts that violate prior-probability-shift assumptions (&&&^^^^4query4^^^^&&&, &&&^^^^4\4^^^^&&&, &&&^^^^4 OR SCATR OR \4^^^^&&&, Seo et al., 29 May 2026). A plausible implication is that prior-calibrated scoring is most effective when the source of mismatch is identifiable—misaligned outcome scale, prior shift, generative overconfidence, or frequency bias—rather than when the scoring problem is dominated by uncontrolled distributional change.