Confidence-Guided Decoding (CGD)

• Confidence-Guided Decoding (CGD) is a method that uses internal activation-derived confidence to steer token selection and improve output factuality.
• It combines traditional language model probabilities with calibrated confidence estimates to rerank candidate tokens during inference.
• CGD extends to diverse applications including refined token reranking, adaptive self-refinement, and controlled decoding in diffusion models.

Searching arXiv for papers on confidence-guided decoding and related methods. Confidence-Guided Decoding (CGD) denotes a class of inference-time generation methods that use an explicit confidence signal to influence decoding decisions. In the formulation introduced in "Enhancing LLM Factuality via Activation-Based Confidence Calibration and Guided Decoding" (Liu et al., 2024), CGD—called CoDec in that paper—reranks next-token candidates by combining the LLM’s token probability with a learned confidence estimate derived from internal activations, with the aim of eliciting answers that are not merely likely under the model but also predicted to be correct. Subsequent work applied the same general idea more broadly: confidence can guide token selection, document arbitration, refinement actions, position unmasking in diffusion LLMs, or even logical consistency projections, rather than serving only as a post hoc reliability score (Jin et al., 9 Feb 2026).

1. Origin in activation-based calibration

The 2024 CoDec framework is built on ActCab (Activation-based Calibration), a calibration model designed to estimate answer correctness from the LLM’s last-layer representations rather than from logits alone (Liu et al., 2024). For a query $x$ and generated response $y$ with hidden activations

$$\mathbf{h}_{[1:|y|]} = [\mathbf{h}_1, \mathbf{h}_2, \ldots, \mathbf{h}_{|y|}],$$

ActCab uses the hidden states after the feed-forward network in the last layer and trains a lightweight linear classifier on the average last-layer activation: $$p_{\theta}(y|x)=\sigma\!\left(\mathbf{W} \cdot \frac{1}{|y|}\sum_{i=1}^{|y|}\mathbf{h}_i + \mathbf{B}\right).$$ The scalar output in $[0,1]$ is interpreted as a confidence score intended to align with actual answer correctness.

ActCab is trained with two related objectives. The first is an MSE regression objective over binary correctness labels: $$\mathcal{L}_{MSE}= \frac{1}{|\mathcal{D}|} \sum_{(x,y)\in\mathcal{D}} \big(\mathbf{1}(y \text{ is correct})-p_{\theta}(y|x)\big)^2.$$ The paper states that regression is preferred over classification because it yields smoother outputs and better calibration. The second objective introduces ECE loss with soft labels. The training data are split into $K$ folds; $K$ classifiers are trained using each fold as validation; out-of-fold confidence predictions are collected; instances are binned by predicted confidence into 10 equal-width bins; and each instance is assigned the bin accuracy as its soft target. The resulting objective is

$$\mathcal{L}_{ECE}= \frac{1}{|\mathcal{D}|} \sum_{(x,y)\in\mathcal{D}} \big(\operatorname{acc}(B_{(x,y)}) - p_{\theta}(y|x)\big)^2.$$

This soft-label formulation is the version reported to yield the best calibration performance.

The methodological premise is that last-layer activations can encode richer signals about whether the model “knows” an answer than token logits alone. This directly motivates using confidence during decoding rather than only after generation.

2. CoDec as a decoding-time intervention

CoDec is described as a greedy procedure whose novelty lies in confidence-aware token choice rather than ordinary maximum-probability decoding (Liu et al., 2024). At each decoding step $t$, the LLM head provides a next-token distribution, but CoDec restricts attention to the top 7 tokens by language-model probability. For each candidate token $y$0, it computes the language-model probability $y$1, obtains the candidate activation $y$2, and evaluates that activation with ActCab. The candidate score is

$y$3

with $y$4 tuned on the development set; the experiments use $y$5. The selected token is the candidate with the highest $y$6, after which generation continues greedily.

This procedure differs from two common alternatives. It is not ordinary greedy decoding, because token probability alone does not determine the next token. It is also not selective generation or filtering, because it does not simply reject low-confidence completed answers after the fact. Instead, confidence becomes an active decoding signal that reranks local continuations.

The method includes a second, response-level confidence comparison. After producing a full CoDec response, ActCab is applied again to the average activations over the full sequence to estimate overall confidence. The CoDec-generated response is kept only if its overall confidence is higher than that of the standard greedy-decoding response. This means the procedure combines token-level reranking with a final sequence-level safeguard.

The paper explicitly contrasts this design with inference-time intervention methods such as ITI and RepE. Its claim is not that CoDec edits the model’s internal computation; rather, it leaves the internal computation unchanged and adjusts output ranking. It is likewise contrasted with selective generation, which can improve trustworthiness at the cost of coverage and helpfulness when low-confidence answers are in fact correct.

3. Calibration performance and factuality gains

The evaluation in (Liu et al., 2024) separates calibration quality from downstream factuality. For calibration, the paper uses the CaT benchmark tasks: Natural Questions (NQ), SciQ, TriviaQA, TruthfulQA, and WikiQA. On these benchmarks, ActCab is reported to achieve superior calibration performance relative to all competitive baselines, including an average expected calibration error (ECE) reduction of up to 39%.

For factuality, CoDec is tested on the same five tasks with Llama2-7b, Llama2-13b, and Llama3-8b. The paper reports consistent improvements on challenging QA datasets, with the clearest gains on TruthfulQA.

Model / dataset	Greedy decoding	CoDec
Llama2-7b / TruthfulQA	True.*Info = 27.50	True.*Info = 41.10
Llama2-13b / TruthfulQA	28.62	39.91
Llama3-8b / TruthfulQA	25.03	37.84

For Llama2-7b on TruthfulQA, the change from 27.50 to 41.10 is described as about a 50% increase over greedy decoding (Liu et al., 2024). The method also improves or matches performance on NQ, TriviaQA, and WikiQA, although the gains are stated to be non-uniform.

A notable limitation is performance on SciQ, where CoDec is reported to be less effective for some models, including Llama2-7b and Llama3-8b. The paper’s explanation is that the model may lack sufficient underlying knowledge, and a linear confidence layer cannot invent missing facts. In that sense, CoDec can steer the model toward knowledge it already internalized, but it cannot reliably recover facts that were never represented.

4. From calibrated reranking to confidence-controlled inference

Later work broadened the meaning of CGD from token reranking to a more general inference principle: confidence can decide what the model should do next, not merely which token should be emitted (Jin et al., 9 Feb 2026). "CoRefine: Confidence-Guided Self-Refinement for Adaptive Test-Time Compute" introduces a frozen-LLM-compatible refinement loop in which a 211k-parameter Conv1D controller reads a full token-level confidence trace and outputs one of three actions: HALT, RETHINK, or ALTERNATIVE. The controller averages about 2.7 iterations per problem and uses roughly a 190-fold token reduction relative to a 512-sample baseline, while achieving 92.6 percent precision when it confidently halts. In this formulation, confidence is a control statistic rather than a correctness guarantee.

A related generalization appears in multi-document retrieval-augmented generation. "Dual-Confidence Contrastive Decoding for Retrieval-Augmented Generation" proposes DCCD, which combines document-level confidence $y$7 with token-level confidence $y$8, defines a dual-confidence score $y$9, and uses the highest- and lowest-scoring document-conditioned streams as positive and negative contrastive branches (Li et al., 1 Jul 2026). The method is evaluated on DRQA, a benchmark designed so that answers are synthetic enterprise facts not recoverable from parametric memory; zero-shot accuracy is reported as 0% across models, forcing retrieval-grounded evidence selection. DCCD achieves the best average performance among full-context and contrastive baselines, with the largest gains on DRQA.

This expansion of CGD also includes domain-specific formulations in which confidence governs copying, external grounding, or localized intervention.

Method	Confidence signal	Decoding use
CoRefine (Jin et al., 9 Feb 2026)	Full-trace confidence from logprobs	HALT / RETHINK / ALTERNATIVE
DCCD (Li et al., 1 Jul 2026)	Document-level $$\mathbf{h}_{[1:|y|]} = [\mathbf{h}_1, \mathbf{h}_2, \ldots, \mathbf{h}_{|y|}],$$0 and token-level $$\mathbf{h}_{[1:|y|]} = [\mathbf{h}_1, \mathbf{h}_2, \ldots, \mathbf{h}_{|y|}],$$1	Positive/negative document stream selection
CoCoLex (S et al., 7 Aug 2025)	Entropy-derived $$\mathbf{h}_{[1:|y|]} = [\mathbf{h}_1, \mathbf{h}_2, \ldots, \mathbf{h}_{|y|}],$$2	Interpolate generation with context copying
CCD (Tang et al., 20 Feb 2026)	Token-level confidence $$\mathbf{h}_{[1:|y|]} = [\mathbf{h}_1, \mathbf{h}_2, \ldots, \mathbf{h}_{|y|}],$$3	Trigger contrastive subtraction at low-confidence tokens
CLIP-Guided Decoding (Deng et al., 2024)	CLIP image-text similarity	Rerank sentence candidates for visual grounding
CGD-PD (Huang et al., 12 Mar 2026)	Consistency and proof support	Resolve True/False/Unknown outcomes

In "CoCoLex: Confidence-guided Copy-based Decoding for Grounded Legal Text Generation," confidence is computed from normalized entropy and used to mix the model distribution with a copy distribution derived from context hidden states; low confidence increases copying, with the goal of preserving exact legal phrasing and improving faithfulness in long-form legal generation (S et al., 7 Aug 2025). In "Seeing is Believing: Mitigating Hallucination in Large Vision-LLMs via CLIP-Guided Decoding," the guidance signal is external rather than internal: sentence-level CLIP similarity is combined with length-normalized sentence likelihood to reduce object hallucination in LVLMs (Deng et al., 2024). In "Thinking by Subtraction: Confidence-Driven Contrastive Decoding for LLM Reasoning," low-confidence tokens trigger selective contrastive decoding, while high-confidence tokens remain untouched (Tang et al., 20 Feb 2026). In "Consistency-Guided Decoding with Proof-Driven Disambiguation for Three-Way Logical Question Answering," confidence is operationalized through negation consistency, decisive versus Unknown outputs, and binary entailment probes rather than through calibrated scalar scores (Huang et al., 12 Mar 2026).

5. Confidence-guided decoding in diffusion LLMs

Diffusion LLMs (DLMs) made CGD a central question of decoding policy because generation order is not fixed. "Confidence-Based Decoding is Provably Efficient for Diffusion LLMs" develops the first theoretical analysis framework for confidence-based decoding in DLMs and studies an entropy sum-based strategy that keeps unmasking tokens in a random order until the cumulative entropy of the current batch exceeds a threshold (Cai et al., 23 Mar 2026). Under the assumption of an optimal mask predictor, the paper shows that this strategy achieves $$\mathbf{h}_{[1:|y|]} = [\mathbf{h}_1, \mathbf{h}_2, \ldots, \mathbf{h}_{|y|}],$$4-accurate sampling in KL divergence with expected iteration complexity

$$\mathbf{h}_{[1:|y|]} = [\mathbf{h}_1, \mathbf{h}_2, \ldots, \mathbf{h}_{|y|}],$$5

more precisely

$$\mathbf{h}_{[1:|y|]} = [\mathbf{h}_1, \mathbf{h}_2, \ldots, \mathbf{h}_{|y|}],$$6

The stated significance is adaptivity: when the target data distribution has low entropy relative to sequence length, decoding can be substantially faster without requiring prior knowledge of $$\mathbf{h}_{[1:|y|]} = [\mathbf{h}_1, \mathbf{h}_2, \ldots, \mathbf{h}_{|y|}],$$7.

At the same time, later DLM papers argue that local confidence can be systematically misleading. "When Confidence Misleads: Suffix Anchoring and Anchor-Proximity Confidence Modulation for Diffusion LLMs" identifies two failure modes in fully non-autoregressive decoding: EOT overconfidence, which causes incomplete generation, and local overconfidence near a suffix anchor, which causes anchor-adjacent tokens to be decoded too early (Park et al., 27 May 2026). The proposed remedy is Suffix-Anchored Confidence Modulation, which inserts a short suffix anchor and rescales confidence near that anchor according to decoding progress: $$\mathbf{h}_{[1:|y|]} = [\mathbf{h}_1, \mathbf{h}_2, \ldots, \mathbf{h}_{|y|}],$$8 Across reasoning, vision-language reasoning, and code generation, the method is reported to outperform both naïve confidence-based decoding and explicit EOT suppression while preserving the parallel decoding advantage of fully non-autoregressive generation.

A deeper critique appears in "The Confidence Shortcut: A Reasoning Failure Mode of Masked Diffusion Models" (Kim et al., 27 May 2026). That paper argues that confidence-based decoding can be a shortcut policy: it reveals tokens that look locally easy before the long-range dependencies that logically determine them have been resolved. The controlled example is 32-digit addition, where the correct logical-flow order is least-significant-digit first. Confidence-based decoding is reported to commit prematurely to the chain-MSB cell in long carry chains; for the 406 confidence-decoding failures in addition, 99.8% occur at the chain-MSB cell, with the wrong token receiving top-1 probability about 0.997 on average. The same paper argues that confidence-aligned training can amplify this failure on tasks whose logical-flow trajectory diverges from confidence order, although it can help on tasks such as Sudoku where high confidence more closely tracks logical readiness.

6. Beyond local confidence: coherence, consistency, and misconceptions

One recurrent misconception is that CGD is simply confidence thresholding. The literature summarized here does not support that reduction. CoDec uses confidence as a reranking signal during generation rather than as a post hoc filter (Liu et al., 2024). CoRefine uses confidence to allocate test-time compute and choose among refinement actions (Jin et al., 9 Feb 2026). DCCD uses confidence to arbitrate among conflicting retrieved sources (Li et al., 1 Jul 2026). Diffusion-language-model work treats confidence as a rule for selecting positions or budgets, not just as an uncertainty report (Cai et al., 23 Mar 2026).

Another misconception is that any strong confidence signal is necessarily beneficial. Several papers explicitly reject that conclusion. The diffusion-language-model literature shows that high confidence may correspond to premature EOT, anchor-adjacent overconfidence, or shortcut reasoning states rather than genuine readiness (Park et al., 27 May 2026). The masked-diffusion critique goes further by arguing that confidence-based decoding is only reliable when the confidence trajectory is aligned with the task’s logical-flow trajectory (Kim et al., 27 May 2026).

This motivates methods that move beyond purely local confidence. "Beyond Confidence: Adaptive and Coherent Decoding for Diffusion LLMs" proposes Coherent Contextual Decoding (CCD), which averages predictive distributions over historical decoding states and interprets trajectory consistency through conditional mutual information (Chen et al., 26 Nov 2025). The method uses historical agreement of high-confidence positions and marginal entropy of the averaged distribution to reject locally confident but globally unstable paths. Empirically, it reports up to 3.48x speedup alongside 3.91% performance improvement across benchmarks on Dream and LLaDA.

Taken together, these results indicate that CGD is best understood not as a single algorithm but as an inference-time design pattern. In its narrow form, it refers to the activation-calibrated token reranking strategy of CoDec (Liu et al., 2024). In the broader contemporary literature, it designates a family of decoding policies that use confidence to guide generation, source selection, refinement, or unmasking order. The central unresolved issue is not whether confidence matters, but which confidence signal is informative for a given task and how that signal should be coupled to the decoding policy.