--- title: Elicitation-Then-Calibration (EliCal) url: https://www.emergentmind.com/topics/elicitation-then-calibration-elical type: topic --- # Elicitation-Then-Calibration (EliCal) Searching arXiv for papers referring to “Elicitation-Then-Calibration” / “EliCal” to ground the article in the current literature. Elicitation-Then-Calibration (EliCal) denotes a recurring two-stage design pattern in which a latent belief, confidence signal, prior judgment, or self-assessment is first made explicit and is then calibrated against a more reliable target such as empirical correctness, judge scores, coverage, or prior-predictive constraints. The literature suggests that the label does not refer to a single canonical algorithm, but to a family of structurally related procedures spanning large language model (LLM) confidence estimation, honesty alignment, judge-aligned self-evaluation, Bayesian prior construction, ecological modeling, and multicalibration theory [2401.08694], [2510.17509], [2606.05122], [2404.02655], [2402.06544], [1712.00685], [2206.08817], [1007.4740]. ## 1. Common architecture and domain-level scope Across its uses, EliCal separates two operations that are often conflated. In the elicitation stage, a system is induced to produce an uncertainty-bearing object: a numeric confidence score, a self-consistency proxy, a multi-attribute self-evaluation block, a prior-predictive percentile, a map of subjective probabilities, or a low-dimensional property report. In the calibration stage, that elicited object is adjusted, supervised, or constrained so that it better matches a target notion of correctness or belief quality. The target varies by domain: binary truthfulness, multi-attribute judge scores, true sampling accuracy, empirical interval coverage, survey-validated ecological occurrence, or prior-predictive compatibility. | Domain | Elicited object | Calibration target | |---|---|---| | Misinformation mitigation | 0–100 verbalized certainty and stochastic answer samples | Empirical correctness and calibration error | | Honesty alignment for QA | Self-consistency confidence from sampled responses | True probability of producing a correct answer | | Judge-aligned self-evaluation | `[SELF_EVAL]...[/SELF_EVAL]` JSON scores | External judge’s multi-attribute scores | | Long-form generation | Confidence distribution $P_C$ | Correctness distribution $P_G$ | | Bayesian prior elicitation | Expert prior-predictive quantiles or pseudo-data summaries | Prior-predictive matching and posterior coherence | This shared structure is clearest when contrasted with one-stage baselines. In several LLM settings, direct verbalized confidence is treated as informative but over-confident, while sample-based methods are treated as informative but imperfectly calibrated; EliCal therefore combines them rather than choosing one source exclusively. In Bayesian settings, expert knowledge is not inserted directly as a prior parameter value, but is first elicited in an observable space such as quantiles or maps and only then calibrated through a prior-predictive or hierarchical model [2401.08694], [1712.00685], [2206.08817]. ## 2. Hybrid confidence estimation in LLM classification and misinformation tasks In misinformation mitigation, EliCal was formulated as a hybrid uncertainty-quantification framework that combines direct confidence elicitation with sample-based consistency methods [2401.08694]. The direct elicitation component uses an “Explain-Score” prompt: the model rates the truthfulness of a statement on a 0–100 scale, gives analysis first, then outputs a score after a vertical bar. The raw elicited confidence is $$ P_e(x)=\frac{c(x)}{100}\in[0,1]. $$ The paper distinguishes single-step and two-step prompting. In the two-step version, the first prompt elicits the truthfulness score and explanation, and the second prompt asks separately for uncertainty on 0–100. The reported rationale is that verbalized scores capture intrinsic uncertainty but are systematically over-confident and bunched toward high values. The sample-based component generates $k$ stochastic samples at temperature $T>0$ and computes a consistency-derived uncertainty. The best performer is SampleAvgDev, $$ U_s(x)=\frac1k\sum_{i=1}^k \frac{|a_i-50|}{50}, \qquad P_s(x)=1-U_s(x). $$ The hybrid confidence is then $$ P_c(x)=\alpha P_s(x)+(1-\alpha)P_e(x), $$ with $\alpha$ chosen by cross-validation and reported as $\alpha\approx0.9$ for SampleAvgDev. On the LIAR dataset with a binary split, sample-based confidence alone at $k=10$, $T=1.0$, and SampleAvgDev yielded $\mathrm{ECE}=0.139$ and Brier $=0.291$; two-step elicitation alone yielded $\mathrm{ECE}=0.260$, compared with single-step $\mathrm{ECE}=0.313$; and the hybrid method with $\alpha=0.9$ yielded $\mathrm{ECE}=0.076$ and Brier $=0.334$ [2401.08694]. The reported interpretation is that two-step elicitation reduces distributional shift, while hybridization combines intrinsic and extrinsic uncertainty. A related post-hoc EliCal formulation for multiple-choice QA decomposes confidence into uncertainty about the question and fidelity to the generated answer [2404.02655]. There, uncertainty is defined as normalized entropy over sampled answer frequencies, $$ U(x)=-\frac{\sum_{i=1}^M p_i\log p_i}{\log M}, $$ fidelity is derived from “fidelity chains” generated by repeatedly replacing an option with “All other options are wrong.”, and overall confidence is $$ C(x,y)=\bigl[1-U(x)\bigr]\times F(x,y). $$ This variant is explicitly “plug-and-play” and requires no held-out split. It was evaluated with Expected Calibration Error (ECE), Inverse Pair Ratio (IPR), and Confidence Evenness (CE). Relative to the best baseline on each model and dataset, EliCal “consistently lowers ECE_10 (e.g. on average from ≈0.18 → 0.08), zeroes or near-zeroes IPR_10, boosts CE_10 (often >0.85), without harming raw accuracy” [2404.02655]. The same work argues that low ECE alone can be trivial, because always predicting the dataset’s average accuracy can yield $\mathrm{ECE}=0$ while collapsing CE. ## 3. Training-based honesty alignment and judge calibration In free-form QA honesty alignment, EliCal has been used for annotation-efficient training rather than purely post-hoc scoring [2510.17509]. The problem setup defines the model’s true capability on a question $q$ as the probability of sampling a correct response under a decoding policy, and the goal is to learn a confidence function $\widehat C(q)\in[0,1]$ such that $$ \widehat C(q)\approx \mathrm{Accuracy}_\theta(q). $$ Stage 1 is confidence elicitation via self-consistency. With $\tilde r=\arg\max_r p_\theta^\pi(r\mid q)$ and a binary semantic-consistency indicator $s(r,\tilde r)$, the self-consistency score is $$ C_{\mathrm{SC}}(q)\approx \frac1k\sum_{r\in\hat{\mathcal R}} s(r,\tilde r). $$ A frozen backbone receives LoRA adapters and a linear head $f_\phi$, trained to minimize $$ \mathcal L_{\mathrm{elic}} = \frac{1}{|\mathcal Q|} \sum_{q\in\mathcal Q} \Bigl(f_\phi(\mathbf h_T^{(L)})-C_{\mathrm{SC}}(q)\Bigr)^2. $$ Stage 2 calibrates this elicited signal with a small labeled set by minimizing $$ \mathcal L_{\mathrm{cal}} = \frac{1}{|\mathcal Q_{\mathrm{small}}|} \sum_{q\in\mathcal Q_{\mathrm{small}}} \Bigl(f_\phi(\mathbf h_T^{(L)})-\mathrm{Accuracy}_\theta(q)\Bigr)^2. $$ The supporting benchmark, HonestyBench, covers ten free-form QA datasets, with 567,647 training pairs, 37,904 in-domain evaluation pairs, and 32,805 out-of-domain evaluation pairs [2510.17509]. Full supervision uses all 560k correctness annotations, whereas 1k labels corresponds to “0.18% of full supervision.” At 1k labels, Cal-Only achieves in-domain AUROC $\approx 73.4\%$ and OOD $\approx 77.3\%$, while EliCal achieves in-domain AUROC $\approx 84.4\%$ and OOD $\approx 84.5\%$, described as “approximately 98% of the full-supervision upper bound.” On MMLU, EliCal also outperforms Cal-Only, with the interpretation that self-consistency yields better cross-format transfer than calibration-only fine-tuning [2510.17509]. A distinct but related line of work uses the name EliCal for judge-aligned self-evaluation elicitation, under the alternate name Self-Evaluation Elicitation (SEE) [2606.05122]. Here the model is prompted to answer a user request and then append a fixed `[SELF_EVAL]…[/SELF_EVAL]` JSON block containing five integers 0–9 for quality attributes. The elicitation is read directly from the model’s token distribution over those digits. Without any dedicated training, Qwen3-4B-Base in SEE format reportedly achieves calibration scores of 0.63 on HelpSteer2 validation and 0.50–0.70 across open-ended benchmarks, with the true judge score within its top-5 predicted tokens over 77% of the time [2606.05122]. SEE then adds a calibration-coupled reinforcement learning phase, with reward $$ r(x;\theta)=w_q\cdot q(j)+w_c\cdot c(s,j), $$ followed by a masked distillation phase restricted to the self-evaluation tokens. The distillation loss is $$ L_{\mathrm{distill}} = \mathbb E_x\bigl[D_{KL}(f_{\mathrm{prompt}}(x;\theta_{RL})\|f_{\mathrm{prompt}}(x;\theta))\bigr]. $$ With 160 unique examples, described as roughly 31x fewer than an RL-only baseline, SEE improves HelpSteer2 validation calibration from 0.632 to 0.731 and quality from 0.644 to 0.704, while preserving transfer to unseen judges Claude Sonnet 4.6 and Gemini 3.1 Flash-Lite [2606.05122]. The explicit interpretation is that the relevant ability is already latent in the base model and is being elicited rather than acquired. ## 4. Distributional, interval, and embodied extensions EliCal has also been generalized beyond binary correctness. For long-form generation, correctness and confidence are both modeled as distributions over $[0,1]$ rather than point probabilities [2402.06544]. If $G$ is the answer-correctness random variable and $C$ is the model’s confidence random variable, the framework defines ground-truth correctness distribution $P_G$ and confidence distribution $P_C$, then evaluates their alignment. Ground-truth correctness can be elicited by repeated GPT-4 evaluation, $$ P_G(x)=\frac1N\sum_{j=1}^N \mathbf 1[\mathrm{Eval}_j(A)=x], $$ and confidence can be elicited either by self-evaluation or by self-consistency over sampled answers. The framework uses dataset-level Pearson correlation between mean correctness and mean confidence, Wasserstein similarity, and selective F1. Reported findings include that larger models are not always better calibrated, that performance is metric-dependent, that self-consistency excels on factoid datasets, and that calibration can be improved by temperature scaling, fine-tuning, document grounding, and hybridizing self-evaluation with self-consistency through $$ P_{\mathrm{hybrid}}=\alpha P_{\mathrm{eval}}+(1-\alpha)P_{\mathrm{consist}}. $$ The same work reports that GPT-3.5-turbo achieves the highest selective F1 on all tasks, but not the highest correlation or Wasserstein similarity [2402.06544]. A separate interval-estimation formulation treats elicitation as asking an LLM for a point estimate and a nominal 95% credible interval $(\hat y,\ell,u)$, followed by split-conformal recalibration [2604.01896]. Raw intervals were found to be severely overconfident: empirical coverage ranged from 9% to 44% across model-effort combinations. With normalized conformal calibration, coverage rose to approximately 94%–96% for groups with at least 15 calibration points, and calibration gaps moved from raw $\Delta\approx0.50$ to calibrated $\Delta\approx0.01$ [2604.01896]. The same study reports that larger models produce more accurate estimates, but increasing reasoning effort provides no consistent benefit. In embodied agents, EliCal becomes a sequential confidence-control loop over perception and action [2503.10628]. The elicitation side comprises five “Elicitation Policies”: Vanilla, Self-Intervention, Chain-of-Thought (inductive), Plan-and-Solve (deductive), and Top-K (abductive). The calibration side comprises three “Execution Policies”: Action Sampling, Scenario Reinterpretation, and Hypothetical Reasoning. Calibration is evaluated mainly with ECE and AUROC. On Minecraft tasks, GPT-4V with Vanilla elicitation has $\mathrm{ECE}=0.27$ and AUROC $=0.69$; Self-Intervention reduces ECE to 0.21 and raises AUROC to 0.76; Chain-of-Thought reaches ECE $=0.16$ and AUROC $=0.83$; and Plan-and-Solve reaches ECE $=0.15$ and AUROC $=0.82$ [2503.10628]. Pairing elicitation with execution policies amplifies gains: GPT-4V+CoT+Action Sampling reaches ECE $=0.11$ and AUROC $>0.85$. The same experiments report degradation with task difficulty, especially under abductive settings. ## 5. Bayesian and applied-statistical antecedents Long before its recent LLM usage, the elicitation-then-calibration pattern appeared in Bayesian prior construction. In extreme-value analysis, expert knowledge is turned into informative priors by treating a prior as the posterior of a noninformative reference prior and a virtual sample [1712.00685]. With parameter $\theta$, reference prior $p_0(\theta)$, and virtual data $x^\*$ of size $m$, $$ p(\theta\mid x^\*)\propto p_0(\theta)L(\theta;x^\*). $$ Hyperparameters are then calibrated by matching the prior-predictive distribution to expert-specified quantiles, either through direct quantile matching or through Cooke’s criterion, $$ \omega^\*=\arg\min_\omega \sum_{i=0}^M (\alpha_{i+1}-\alpha_i) \log \frac{\alpha_{i+1}-\alpha_i} {\tilde\alpha_{i+1}(\omega)-\tilde\alpha_i(\omega)}. $$ In the Corsican pluviometry example, the expert specifies annual-maxima quartiles $x_{0.25}=75$ mm, $x_{0.50}=100$ mm, and $x_{0.75}=150$ mm; calibrated virtual-sample summaries include $x_{e1}=87.7$, $x_{e2}=133.9$ for Fréchet with $m=5$, $x_{e3}=92.7$, $x_{e4}=128.4$ for Weibull with $m=5$, and pseudo-data $\tilde x=(81,93,101)$ for Gumbel with $m=3$ [1712.00685]. A closely related Weibull-lifetime construction also treats the prior as a “reference posterior” generated by a virtual sample of size $m$ [1007.4740]. The elicited quantity is a prior-predictive percentile, $$ P_\pi(T0.90$, “Likely spawning area” for $0.50