Elicitation-Then-Calibration (EliCal)

• Elicitation-Then-Calibration (EliCal) is a two-stage design pattern that first elicits latent uncertainty signals (e.g., confidence scores, self-assessments) and then calibrates them to align with empirical or judge-based targets.
• The approach improves prediction reliability by combining direct elicitation with sample-based methods, effectively reducing overconfidence and improving metrics like ECE and AUROC in LLM and Bayesian applications.
• EliCal spans diverse domains such as misinformation mitigation, QA honesty alignment, and ecological modeling, offering a flexible framework to refine both neural network outputs and Bayesian prior constructions.

Searching arXiv for papers referring to “Elicitation-Then-Calibration” / “EliCal” to ground the article in the current literature. Elicitation-Then-Calibration (EliCal) denotes a recurring two-stage design pattern in which a latent belief, confidence signal, prior judgment, or self-assessment is first made explicit and is then calibrated against a more reliable target such as empirical correctness, judge scores, coverage, or prior-predictive constraints. The literature suggests that the label does not refer to a single canonical algorithm, but to a family of structurally related procedures spanning LLM confidence estimation, honesty alignment, judge-aligned self-evaluation, Bayesian prior construction, ecological modeling, and multicalibration theory (Rivera et al., 2024, Ni et al., 20 Oct 2025, Zhang et al., 3 Jun 2026, Zhang et al., 2024, Huang et al., 2024, Bousquet et al., 2017, Kaurila et al., 2022, Bousquet, 2010).

1. Common architecture and domain-level scope

Across its uses, EliCal separates two operations that are often conflated. In the elicitation stage, a system is induced to produce an uncertainty-bearing object: a numeric confidence score, a self-consistency proxy, a multi-attribute self-evaluation block, a prior-predictive percentile, a map of subjective probabilities, or a low-dimensional property report. In the calibration stage, that elicited object is adjusted, supervised, or constrained so that it better matches a target notion of correctness or belief quality. The target varies by domain: binary truthfulness, multi-attribute judge scores, true sampling accuracy, empirical interval coverage, survey-validated ecological occurrence, or prior-predictive compatibility.

Domain	Elicited object	Calibration target
Misinformation mitigation	0–100 verbalized certainty and stochastic answer samples	Empirical correctness and calibration error
Honesty alignment for QA	Self-consistency confidence from sampled responses	True probability of producing a correct answer
Judge-aligned self-evaluation	[SELF_EVAL]...[/SELF_EVAL] JSON scores	External judge’s multi-attribute scores
Long-form generation	Confidence distribution $P_C$	Correctness distribution $P_G$
Bayesian prior elicitation	Expert prior-predictive quantiles or pseudo-data summaries	Prior-predictive matching and posterior coherence

This shared structure is clearest when contrasted with one-stage baselines. In several LLM settings, direct verbalized confidence is treated as informative but over-confident, while sample-based methods are treated as informative but imperfectly calibrated; EliCal therefore combines them rather than choosing one source exclusively. In Bayesian settings, expert knowledge is not inserted directly as a prior parameter value, but is first elicited in an observable space such as quantiles or maps and only then calibrated through a prior-predictive or hierarchical model (Rivera et al., 2024, Bousquet et al., 2017, Kaurila et al., 2022).

2. Hybrid confidence estimation in LLM classification and misinformation tasks

In misinformation mitigation, EliCal was formulated as a hybrid uncertainty-quantification framework that combines direct confidence elicitation with sample-based consistency methods (Rivera et al., 2024). The direct elicitation component uses an “Explain-Score” prompt: the model rates the truthfulness of a statement on a 0–100 scale, gives analysis first, then outputs a score after a vertical bar. The raw elicited confidence is

$P_e(x)=\frac{c(x)}{100}\in[0,1].$

The paper distinguishes single-step and two-step prompting. In the two-step version, the first prompt elicits the truthfulness score and explanation, and the second prompt asks separately for uncertainty on 0–100. The reported rationale is that verbalized scores capture intrinsic uncertainty but are systematically over-confident and bunched toward high values.

The sample-based component generates $k$ stochastic samples at temperature $T>0$ and computes a consistency-derived uncertainty. The best performer is SampleAvgDev,

$$U_s(x)=\frac1k\sum_{i=1}^k \frac{|a_i-50|}{50}, \qquad P_s(x)=1-U_s(x).$$

The hybrid confidence is then

$P_c(x)=\alpha P_s(x)+(1-\alpha)P_e(x),$

with $\alpha$ chosen by cross-validation and reported as $\alpha\approx0.9$ for SampleAvgDev. On the LIAR dataset with a binary split, sample-based confidence alone at $k=10$, $P_G$0, and SampleAvgDev yielded $P_G$1 and Brier $P_G$2; two-step elicitation alone yielded $P_G$3, compared with single-step $P_G$4; and the hybrid method with $P_G$5 yielded $P_G$6 and Brier $P_G$7 (Rivera et al., 2024). The reported interpretation is that two-step elicitation reduces distributional shift, while hybridization combines intrinsic and extrinsic uncertainty.

A related post-hoc EliCal formulation for multiple-choice QA decomposes confidence into uncertainty about the question and fidelity to the generated answer (Zhang et al., 2024). There, uncertainty is defined as normalized entropy over sampled answer frequencies,

$P_G$8

fidelity is derived from “fidelity chains” generated by repeatedly replacing an option with “All other options are wrong.”, and overall confidence is

$P_G$9

This variant is explicitly “plug-and-play” and requires no held-out split. It was evaluated with Expected Calibration Error (ECE), Inverse Pair Ratio (IPR), and Confidence Evenness (CE). Relative to the best baseline on each model and dataset, EliCal “consistently lowers ECE_10 (e.g. on average from ≈0.18 → 0.08), zeroes or near-zeroes IPR_10, boosts CE_10 (often >0.85), without harming raw accuracy” (Zhang et al., 2024). The same work argues that low ECE alone can be trivial, because always predicting the dataset’s average accuracy can yield $P_e(x)=\frac{c(x)}{100}\in[0,1].$0 while collapsing CE.

3. Training-based honesty alignment and judge calibration

In free-form QA honesty alignment, EliCal has been used for annotation-efficient training rather than purely post-hoc scoring (Ni et al., 20 Oct 2025). The problem setup defines the model’s true capability on a question $P_e(x)=\frac{c(x)}{100}\in[0,1].$1 as the probability of sampling a correct response under a decoding policy, and the goal is to learn a confidence function $P_e(x)=\frac{c(x)}{100}\in[0,1].$2 such that

$P_e(x)=\frac{c(x)}{100}\in[0,1].$3

Stage 1 is confidence elicitation via self-consistency. With $P_e(x)=\frac{c(x)}{100}\in[0,1].$4 and a binary semantic-consistency indicator $P_e(x)=\frac{c(x)}{100}\in[0,1].$5, the self-consistency score is

$P_e(x)=\frac{c(x)}{100}\in[0,1].$6

A frozen backbone receives LoRA adapters and a linear head $P_e(x)=\frac{c(x)}{100}\in[0,1].$7, trained to minimize

$P_e(x)=\frac{c(x)}{100}\in[0,1].$8

Stage 2 calibrates this elicited signal with a small labeled set by minimizing

$P_e(x)=\frac{c(x)}{100}\in[0,1].$9

The supporting benchmark, HonestyBench, covers ten free-form QA datasets, with 567,647 training pairs, 37,904 in-domain evaluation pairs, and 32,805 out-of-domain evaluation pairs (Ni et al., 20 Oct 2025). Full supervision uses all 560k correctness annotations, whereas 1k labels corresponds to “0.18% of full supervision.” At 1k labels, Cal-Only achieves in-domain AUROC $k$0 and OOD $k$1, while EliCal achieves in-domain AUROC $k$2 and OOD $k$3, described as “approximately 98% of the full-supervision upper bound.” On MMLU, EliCal also outperforms Cal-Only, with the interpretation that self-consistency yields better cross-format transfer than calibration-only fine-tuning (Ni et al., 20 Oct 2025).

A distinct but related line of work uses the name EliCal for judge-aligned self-evaluation elicitation, under the alternate name Self-Evaluation Elicitation (SEE) (Zhang et al., 3 Jun 2026). Here the model is prompted to answer a user request and then append a fixed [SELF_EVAL]…[/SELF_EVAL] JSON block containing five integers 0–9 for quality attributes. The elicitation is read directly from the model’s token distribution over those digits. Without any dedicated training, Qwen3-4B-Base in SEE format reportedly achieves calibration scores of 0.63 on HelpSteer2 validation and 0.50–0.70 across open-ended benchmarks, with the true judge score within its top-5 predicted tokens over 77% of the time (Zhang et al., 3 Jun 2026).

SEE then adds a calibration-coupled reinforcement learning phase, with reward

$k$4

followed by a masked distillation phase restricted to the self-evaluation tokens. The distillation loss is

$k$5

With 160 unique examples, described as roughly 31x fewer than an RL-only baseline, SEE improves HelpSteer2 validation calibration from 0.632 to 0.731 and quality from 0.644 to 0.704, while preserving transfer to unseen judges Claude Sonnet 4.6 and Gemini 3.1 Flash-Lite (Zhang et al., 3 Jun 2026). The explicit interpretation is that the relevant ability is already latent in the base model and is being elicited rather than acquired.

4. Distributional, interval, and embodied extensions

EliCal has also been generalized beyond binary correctness. For long-form generation, correctness and confidence are both modeled as distributions over $k$6 rather than point probabilities (Huang et al., 2024). If $k$7 is the answer-correctness random variable and $k$8 is the model’s confidence random variable, the framework defines ground-truth correctness distribution $k$9 and confidence distribution $T>0$0, then evaluates their alignment. Ground-truth correctness can be elicited by repeated GPT-4 evaluation,

$T>0$1

and confidence can be elicited either by self-evaluation or by self-consistency over sampled answers. The framework uses dataset-level Pearson correlation between mean correctness and mean confidence, Wasserstein similarity, and selective F1. Reported findings include that larger models are not always better calibrated, that performance is metric-dependent, that self-consistency excels on factoid datasets, and that calibration can be improved by temperature scaling, fine-tuning, document grounding, and hybridizing self-evaluation with self-consistency through

$T>0$2

The same work reports that GPT-3.5-turbo achieves the highest selective F1 on all tasks, but not the highest correlation or Wasserstein similarity (Huang et al., 2024).

A separate interval-estimation formulation treats elicitation as asking an LLM for a point estimate and a nominal 95% credible interval $T>0$3, followed by split-conformal recalibration (Hobor et al., 2 Apr 2026). Raw intervals were found to be severely overconfident: empirical coverage ranged from 9% to 44% across model-effort combinations. With normalized conformal calibration, coverage rose to approximately 94%–96% for groups with at least 15 calibration points, and calibration gaps moved from raw $T>0$4 to calibrated $T>0$5 (Hobor et al., 2 Apr 2026). The same study reports that larger models produce more accurate estimates, but increasing reasoning effort provides no consistent benefit.

In embodied agents, EliCal becomes a sequential confidence-control loop over perception and action (Yu et al., 13 Mar 2025). The elicitation side comprises five “Elicitation Policies”: Vanilla, Self-Intervention, Chain-of-Thought (inductive), Plan-and-Solve (deductive), and Top-K (abductive). The calibration side comprises three “Execution Policies”: Action Sampling, Scenario Reinterpretation, and Hypothetical Reasoning. Calibration is evaluated mainly with ECE and AUROC. On Minecraft tasks, GPT-4V with Vanilla elicitation has $T>0$6 and AUROC $T>0$7; Self-Intervention reduces ECE to 0.21 and raises AUROC to 0.76; Chain-of-Thought reaches ECE $T>0$8 and AUROC $T>0$9; and Plan-and-Solve reaches ECE $$U_s(x)=\frac1k\sum_{i=1}^k \frac{|a_i-50|}{50}, \qquad P_s(x)=1-U_s(x).$$0 and AUROC $$U_s(x)=\frac1k\sum_{i=1}^k \frac{|a_i-50|}{50}, \qquad P_s(x)=1-U_s(x).$$1 (Yu et al., 13 Mar 2025). Pairing elicitation with execution policies amplifies gains: GPT-4V+CoT+Action Sampling reaches ECE $$U_s(x)=\frac1k\sum_{i=1}^k \frac{|a_i-50|}{50}, \qquad P_s(x)=1-U_s(x).$$2 and AUROC $$U_s(x)=\frac1k\sum_{i=1}^k \frac{|a_i-50|}{50}, \qquad P_s(x)=1-U_s(x).$$3. The same experiments report degradation with task difficulty, especially under abductive settings.

5. Bayesian and applied-statistical antecedents

Long before its recent LLM usage, the elicitation-then-calibration pattern appeared in Bayesian prior construction. In extreme-value analysis, expert knowledge is turned into informative priors by treating a prior as the posterior of a noninformative reference prior and a virtual sample (Bousquet et al., 2017). With parameter $$U_s(x)=\frac1k\sum_{i=1}^k \frac{|a_i-50|}{50}, \qquad P_s(x)=1-U_s(x).$$4, reference prior $$U_s(x)=\frac1k\sum_{i=1}^k \frac{|a_i-50|}{50}, \qquad P_s(x)=1-U_s(x).$$5, and virtual data $$U_s(x)=\frac1k\sum_{i=1}^k \frac{|a_i-50|}{50}, \qquad P_s(x)=1-U_s(x).$$6 of size $$U_s(x)=\frac1k\sum_{i=1}^k \frac{|a_i-50|}{50}, \qquad P_s(x)=1-U_s(x).$$7,

$$U_s(x)=\frac1k\sum_{i=1}^k \frac{|a_i-50|}{50}, \qquad P_s(x)=1-U_s(x).$$8

Hyperparameters are then calibrated by matching the prior-predictive distribution to expert-specified quantiles, either through direct quantile matching or through Cooke’s criterion,

$$U_s(x)=\frac1k\sum_{i=1}^k \frac{|a_i-50|}{50}, \qquad P_s(x)=1-U_s(x).$$9

In the Corsican pluviometry example, the expert specifies annual-maxima quartiles $P_c(x)=\alpha P_s(x)+(1-\alpha)P_e(x),$0 mm, $P_c(x)=\alpha P_s(x)+(1-\alpha)P_e(x),$1 mm, and $P_c(x)=\alpha P_s(x)+(1-\alpha)P_e(x),$2 mm; calibrated virtual-sample summaries include $P_c(x)=\alpha P_s(x)+(1-\alpha)P_e(x),$3, $P_c(x)=\alpha P_s(x)+(1-\alpha)P_e(x),$4 for Fréchet with $P_c(x)=\alpha P_s(x)+(1-\alpha)P_e(x),$5, $P_c(x)=\alpha P_s(x)+(1-\alpha)P_e(x),$6, $P_c(x)=\alpha P_s(x)+(1-\alpha)P_e(x),$7 for Weibull with $P_c(x)=\alpha P_s(x)+(1-\alpha)P_e(x),$8, and pseudo-data $P_c(x)=\alpha P_s(x)+(1-\alpha)P_e(x),$9 for Gumbel with $\alpha$0 (Bousquet et al., 2017).

A closely related Weibull-lifetime construction also treats the prior as a “reference posterior” generated by a virtual sample of size $\alpha$1 (Bousquet, 2010). The elicited quantity is a prior-predictive percentile,

$\alpha$2

and the calibration step replaces the unknown virtual sample by

$\alpha$3

This yields tractable conditional and marginal priors for $\alpha$4 and supports calibration of $\alpha$5 by minimizing an incoherency risk $\alpha$6. In the real-data example, expert-specific solutions include $\alpha$7 and $\alpha$8, with aggregated $\alpha$9 and $\alpha\approx0.9$0 (Bousquet, 2010).

In applied ecology, the same pattern appears in species distribution modeling with expert maps and survey-based calibration (Kaurila et al., 2022). Ten local fishermen each delineated an assessment region and color-coded cells using four probability bands: “Known important spawning area” for $\alpha\approx0.9$1, “Likely spawning area” for $\alpha\approx0.9$2, “Possible spawning area” for $\alpha\approx0.9$3, and “No spawning” for $\alpha\approx0.9$4. A hierarchical Bayesian model then combines survey counts or presence/absence data with expert-specific intercepts, skill parameters $\alpha\approx0.9$5, and spatial bias fields $\alpha\approx0.9$6. In the abundance model, adding expert information reduced LOO-lpd from $\alpha\approx0.9$7 to $\alpha\approx0.9$8; in the occurrence model, it improved ACC from 0.718 to 0.737 and CRPS from 0.538 to 0.436, although LOO-lpd became marginally worse, from $\alpha\approx0.9$9 to $k=10$0 (Kaurila et al., 2022). The reported interpretation is that joint inference can both exploit informative experts and down-weight those with $k=10$1.

6. Theoretical interpretation, evaluation criteria, and recurrent limitations

In the theoretical literature on multicalibration, the elicitation–calibration connection is formalized at the level of statistical properties rather than particular prompts or architectures (Noarov et al., 2023). Under mild technical assumptions, the main equivalence states that a continuous scalar distributional property $k=10$2 is multicalibratable if and only if it is elicitable; the same paper also shows that for non-elicitable continuous properties, even the true distributional predictor can fail to be calibrated on simple two-point distributions. Conditionally elicitable pairs such as quantile and CVaR can, however, be jointly multicalibrated. This yields a precise sense in which elicitation is not merely an engineering trick, but a structural prerequisite for some forms of calibration.

For discrete properties, approximate calibration has been extended through Lipschitz continuous surrogate properties (Finocchiaro et al., 21 May 2026). If a discrete property is strongly orderable, its Lipschitz elicitation complexity is 1, and explicit surrogate constructions can be used to derive approximate $k=10$3-calibration guarantees. The paper states that calibration sample complexity scales polynomially in the number of bins, approximately $k=10$4, so lower-dimensional elicitable surrogates can reduce complexity substantially (Finocchiaro et al., 21 May 2026). This suggests that the choice of elicited report is itself a complexity-control decision.

Several recurrent limitations appear across the empirical EliCal literature. First, calibration is metric-dependent: the long-form generation framework reports that different models can rank differently under correlation, Wasserstein similarity, and selective F1 (Huang et al., 2024). Second, low ECE alone can be misleading, because a near-constant predictor can obtain low ECE while failing to separate confident from unconfident cases; the MCQA fidelity work therefore insists on jointly considering ECE, IPR, and CE (Zhang et al., 2024). Third, formal guarantees may be narrow: conformal recalibration yields marginal coverage under exchangeability, but subpopulation coverage may still fail (Hobor et al., 2 Apr 2026). Fourth, some systems depend on expensive auxiliary machinery, such as a 32B model for semantic consistency judgments in HonestyBench (Ni et al., 20 Oct 2025). Fifth, judge-aligned self-evaluation remains mostly validated against LLM judges rather than human annotators, and the SEE paper explicitly notes that “human-annotator alignment remains to be tested” (Zhang et al., 3 Jun 2026).

Taken together, these results indicate that “elicitation” and “calibration” should not be treated as interchangeable. In EliCal-style systems, elicitation determines which latent signal becomes observable, while calibration determines whether that signal can be trusted operationally. The literature suggests that this separation is valuable precisely because raw self-reports, raw logits, raw sample variability, raw prior judgments, and raw interval widths are each informative yet systematically misaligned in different ways.