---
title: Confidence-Guided Decoding (CGD)
url: https://www.emergentmind.com/topics/confidence-guided-decoding-cgd
type: topic
---

# Confidence-Guided Decoding (CGD)

Searching arXiv for recent papers on confidence-guided decoding and related methods.
Confidence-Guided Decoding (CGD) denotes a class of inference-time generation methods that use an explicit confidence signal to influence decoding decisions. In the formulation introduced in "Enhancing Language Model Factuality via Activation-Based Confidence Calibration and Guided Decoding" [2406.13230], CGD—called **CoDec** in that paper—reranks next-token candidates by combining the language model’s token probability with a learned confidence estimate derived from internal activations, with the aim of eliciting answers that are not merely likely under the model but also predicted to be correct. Subsequent work applied the same general idea more broadly: confidence can guide token selection, document arbitration, refinement actions, position unmasking in diffusion language models, or even logical consistency projections, rather than serving only as a post hoc reliability score [2602.08948].

## 1. Origin in activation-based calibration

The 2024 CoDec framework is built on **ActCab** (Activation-based Calibration), a calibration model designed to estimate answer correctness from the language model’s last-layer representations rather than from logits alone [2406.13230]. For a query \(x\) and generated response \(y\) with hidden activations
\[
\mathbf{h}_{[1:|y|]} = [\mathbf{h}_1, \mathbf{h}_2, \ldots, \mathbf{h}_{|y|}],
\]
ActCab uses the hidden states **after the feed-forward network in the last layer** and trains a lightweight linear classifier on the average last-layer activation:
\[
p_{\theta}(y|x)=\sigma\!\left(\mathbf{W} \cdot \frac{1}{|y|}\sum_{i=1}^{|y|}\mathbf{h}_i + \mathbf{B}\right).
\]
The scalar output in \([0,1]\) is interpreted as a confidence score intended to align with actual answer correctness.

ActCab is trained with two related objectives. The first is an MSE regression objective over binary correctness labels:
\[
\mathcal{L}_{MSE}= \frac{1}{|\mathcal{D}|} \sum_{(x,y)\in\mathcal{D}} \big(\mathbf{1}(y \text{ is correct})-p_{\theta}(y|x)\big)^2.
\]
The paper states that regression is preferred over classification because it yields smoother outputs and better calibration. The second objective introduces **ECE loss** with soft labels. The training data are split into \(K\) folds; \(K\) classifiers are trained using each fold as validation; out-of-fold confidence predictions are collected; instances are binned by predicted confidence into 10 equal-width bins; and each instance is assigned the bin accuracy as its soft target. The resulting objective is
\[
\mathcal{L}_{ECE}= \frac{1}{|\mathcal{D}|} \sum_{(x,y)\in\mathcal{D}} \big(\operatorname{acc}(B_{(x,y)}) - p_{\theta}(y|x)\big)^2.
\]
This soft-label formulation is the version reported to yield the best calibration performance.

The methodological premise is that last-layer activations can encode richer signals about whether the model “knows” an answer than token logits alone. This directly motivates using confidence during decoding rather than only after generation.

## 2. CoDec as a decoding-time intervention

CoDec is described as a **greedy procedure** whose novelty lies in confidence-aware token choice rather than ordinary maximum-probability decoding [2406.13230]. At each decoding step \(t\), the language model head provides a next-token distribution, but CoDec restricts attention to the **top 7 tokens** by language-model probability. For each candidate token \(y_t^*\), it computes the language-model probability \(LM(y_t^*)\), obtains the candidate activation \(\mathbf{h}_t^*\), and evaluates that activation with ActCab. The candidate score is
\[
s(y_t^*) = \lambda \cdot LM(y_t^*) + (1-\lambda)\cdot \sigma(\mathbf{W}\cdot \mathbf{h}_t^*+\mathbf{B}),
\]
with \(\lambda\) tuned on the development set; the experiments use \(\lambda = 0.3\). The selected token is the candidate with the highest \(s(\cdot)\), after which generation continues greedily.

This procedure differs from two common alternatives. It is **not** ordinary greedy decoding, because token probability alone does not determine the next token. It is also **not** selective generation or filtering, because it does not simply reject low-confidence completed answers after the fact. Instead, confidence becomes an active decoding signal that reranks local continuations.

The method includes a second, response-level confidence comparison. After producing a full CoDec response, ActCab is applied again to the **average activations over the full sequence** to estimate overall confidence. The CoDec-generated response is kept only if its overall confidence is higher than that of the standard greedy-decoding response. This means the procedure combines token-level reranking with a final sequence-level safeguard.

The paper explicitly contrasts this design with inference-time intervention methods such as ITI and RepE. Its claim is not that CoDec edits the model’s internal computation; rather, it leaves the internal computation unchanged and adjusts output ranking. It is likewise contrasted with selective generation, which can improve trustworthiness at the cost of coverage and helpfulness when low-confidence answers are in fact correct.

## 3. Calibration performance and factuality gains

The evaluation in [2406.13230] separates calibration quality from downstream factuality. For calibration, the paper uses the **CaT benchmark tasks**: **Natural Questions (NQ)**, **SciQ**, **TriviaQA**, **TruthfulQA**, and **WikiQA**. On these benchmarks, ActCab is reported to achieve superior calibration performance relative to all competitive baselines, including an **average expected calibration error (ECE) reduction of up to 39%**.

For factuality, CoDec is tested on the same five tasks with **Llama2-7b**, **Llama2-13b**, and **Llama3-8b**. The paper reports consistent improvements on challenging QA datasets, with the clearest gains on **TruthfulQA**.

| Model / dataset | Greedy decoding | CoDec |
|---|---:|---:|
| Llama2-7b / TruthfulQA | True.*Info = 27.50 | True.*Info = 41.10 |
| Llama2-13b / TruthfulQA | 28.62 | 39.91 |
| Llama3-8b / TruthfulQA | 25.03 | 37.84 |

For **Llama2-7b on TruthfulQA**, the change from **27.50** to **41.10** is described as about a **50% increase** over greedy decoding [2406.13230]. The method also improves or matches performance on **NQ**, **TriviaQA**, and **WikiQA**, although the gains are stated to be non-uniform.

A notable limitation is performance on **SciQ**, where CoDec is reported to be less effective for some models, including **Llama2-7b** and **Llama3-8b**. The paper’s explanation is that the model may lack sufficient underlying knowledge, and a linear confidence layer cannot invent missing facts. In that sense, CoDec can steer the model toward knowledge it already internalized, but it cannot reliably recover facts that were never represented.

## 4. From calibrated reranking to confidence-controlled inference

Later work broadened the meaning of CGD from token reranking to a more general inference principle: confidence can decide **what the model should do next**, not merely which token should be emitted [2602.08948]. "CoRefine: Confidence-Guided Self-Refinement for Adaptive Test-Time Compute" introduces a frozen-LLM-compatible refinement loop in which a **211k-parameter Conv1D controller** reads a full token-level confidence trace and outputs one of three actions: **HALT**, **RETHINK**, or **ALTERNATIVE**. The controller averages **about 2.7 iterations per problem** and uses roughly a **190-fold token reduction** relative to a 512-sample baseline, while achieving **92.6 percent precision** when it confidently halts. In this formulation, confidence is a control statistic rather than a correctness guarantee.

A related generalization appears in multi-document retrieval-augmented generation. "Dual-Confidence Contrastive Decoding for Retrieval-Augmented Generation" proposes **DCCD**, which combines **document-level confidence** \(q_i\) with **token-level confidence** \(c_{(i,t)}\), defines a dual-confidence score \(s_{(i,t)} = q_i + c_{(i,t)}\), and uses the highest- and lowest-scoring document-conditioned streams as positive and negative contrastive branches [2607.00570]. The method is evaluated on **DRQA**, a benchmark designed so that answers are synthetic enterprise facts not recoverable from parametric memory; zero-shot accuracy is reported as **0% across models**, forcing retrieval-grounded evidence selection. DCCD achieves the best average performance among full-context and contrastive baselines, with the largest gains on DRQA.

This expansion of CGD also includes domain-specific formulations in which confidence governs copying, external grounding, or localized intervention.

| Method | Confidence signal | Decoding use |
|---|---|---|
| CoRefine [2602.08948] | Full-trace confidence from logprobs | HALT / RETHINK / ALTERNATIVE |
| DCCD [2607.00570] | Document-level \(q_i\) and token-level \(c_{(i,t)}\) | Positive/negative document stream selection |
| CoCoLex [2508.05534] | Entropy-derived \(\lambda_t\) | Interpolate generation with context copying |
| CCD [2602.18232] | Token-level confidence \(C_i\) | Trigger contrastive subtraction at low-confidence tokens |
| CLIP-Guided Decoding [2402.15300] | CLIP image-text similarity | Rerank sentence candidates for visual grounding |
| CGD-PD [2604.06196] | Consistency and proof support | Resolve True/False/Unknown outcomes |

In "CoCoLex: Confidence-guided Copy-based Decoding for Grounded Legal Text Generation," confidence is computed from normalized entropy and used to mix the model distribution with a copy distribution derived from context hidden states; low confidence increases copying, with the goal of preserving exact legal phrasing and improving faithfulness in long-form legal generation [2508.05534]. In "Seeing is Believing: Mitigating Hallucination in Large Vision-Language Models via CLIP-Guided Decoding," the guidance signal is external rather than internal: sentence-level CLIP similarity is combined with length-normalized sentence likelihood to reduce object hallucination in LVLMs [2402.15300]. In "Thinking by Subtraction: Confidence-Driven Contrastive Decoding for LLM Reasoning," low-confidence tokens trigger selective contrastive decoding, while high-confidence tokens remain untouched [2602.18232]. In "Consistency-Guided Decoding with Proof-Driven Disambiguation for Three-Way Logical Question Answering," confidence is operationalized through negation consistency, decisive versus Unknown outputs, and binary entailment probes rather than through calibrated scalar scores [2604.06196].

## 5. Confidence-guided decoding in diffusion language models

Diffusion language models (DLMs) made CGD a central question of decoding policy because generation order is not fixed. "Confidence-Based Decoding is Provably Efficient for Diffusion Language Models" develops the first theoretical analysis framework for confidence-based decoding in DLMs and studies an **entropy sum-based strategy** that keeps unmasking tokens in a random order until the cumulative entropy of the current batch exceeds a threshold [2603.22248]. Under the assumption of an optimal mask predictor, the paper shows that this strategy achieves \(\varepsilon\)-accurate sampling in KL divergence with expected iteration complexity
\[
\widetilde O\!\left(\frac{H(X_0)}{\varepsilon}\right),
\]
more precisely
\[
\mathbb E[T] \le \frac{4H(X_0)}{\varepsilon}(\log_2 L+1)+1.
\]
The stated significance is adaptivity: when the target data distribution has low entropy relative to sequence length, decoding can be substantially faster without requiring prior knowledge of \(H(X_0)\).

At the same time, later DLM papers argue that local confidence can be systematically misleading. "When Confidence Misleads: Suffix Anchoring and Anchor-Proximity Confidence Modulation for Diffusion Language Models" identifies two failure modes in fully non-autoregressive decoding: **EOT overconfidence**, which causes incomplete generation, and **local overconfidence near a suffix anchor**, which causes anchor-adjacent tokens to be decoded too early [2605.28181]. The proposed remedy is **Suffix-Anchored Confidence Modulation**, which inserts a short suffix anchor and rescales confidence near that anchor according to decoding progress:
\[
\tilde{c}_i^{(t)}= c_i^{(t)} \left( 1 - w_i (1-p^{(t)})^{\gamma} \right), \qquad
p^{(t)} = 1 - \frac{|\mathcal{M}^{(t)}|}{L}.
\]
Across reasoning, vision-language reasoning, and code generation, the method is reported to outperform both naïve confidence-based decoding and explicit EOT suppression while preserving the parallel decoding advantage of fully non-autoregressive generation.

A deeper critique appears in "The Confidence Shortcut: A Reasoning Failure Mode of Masked Diffusion Models" [2605.29123]. That paper argues that confidence-based decoding can be a **shortcut policy**: it reveals tokens that look locally easy before the long-range dependencies that logically determine them have been resolved. The controlled example is **32-digit addition**, where the correct logical-flow order is least-significant-digit first. Confidence-based decoding is reported to commit prematurely to the chain-MSB cell in long carry chains; for the **406 confidence-decoding failures in addition**, **99.8% occur at the chain-MSB cell**, with the wrong token receiving top-1 probability about **0.997 on average**. The same paper argues that confidence-aligned training can amplify this failure on tasks whose logical-flow trajectory diverges from confidence order, although it can help on tasks such as Sudoku where high confidence more closely tracks logical readiness.

## 6. Beyond local confidence: coherence, consistency, and misconceptions

One recurrent misconception is that CGD is simply confidence thresholding. The literature summarized here does not support that reduction. CoDec uses confidence as a reranking signal during generation rather than as a post hoc filter [2406.13230]. CoRefine uses confidence to allocate test-time compute and choose among refinement actions [2602.08948]. DCCD uses confidence to arbitrate among conflicting retrieved sources [2607.00570]. Diffusion-language-model work treats confidence as a rule for selecting positions or budgets, not just as an uncertainty report [2603.22248].

Another misconception is that any strong confidence signal is necessarily beneficial. Several papers explicitly reject that conclusion. The diffusion-language-model literature shows that high confidence may correspond to premature EOT, anchor-adjacent overconfidence, or shortcut reasoning states rather than genuine readiness [2605.28181]. The masked-diffusion critique goes further by arguing that confidence-based decoding is only reliable when the confidence trajectory is aligned with the task’s logical-flow trajectory [2605.29123].

This motivates methods that move beyond purely local confidence. "Beyond Confidence: Adaptive and Coherent Decoding for Diffusion Language Models" proposes **Coherent Contextual Decoding (CCD)**, which averages predictive distributions over historical decoding states and interprets trajectory consistency through conditional mutual information [2512.02044]. The method uses historical agreement of high-confidence positions and marginal entropy of the averaged distribution to reject locally confident but globally unstable paths. Empirically, it reports up to **3.48x speedup** alongside **3.91% performance improvement** across benchmarks on Dream and LLaDA.

Taken together, these results indicate that CGD is best understood not as a single algorithm but as an inference-time design pattern. In its narrow form, it refers to the activation-calibrated token reranking strategy of CoDec [2406.13230]. In the broader contemporary literature, it designates a family of decoding policies that use confidence to guide generation, source selection, refinement, or unmasking order. The central unresolved issue is not whether confidence matters, but **which confidence signal** is informative for a given task and **how** that signal should be coupled to the decoding policy.

Source: https://www.emergentmind.com/topics/confidence-guided-decoding-cgd