Energy-Based Decoding: Methods & Variants

• Energy-Based Decoding (EBD) is a family of inference methods that use explicit energy functions to guide the generation process, balancing prior plausibility and high-reward outcomes.
• It spans diverse formulations including reward-tilted sampling for LLMs, constrained generation via Langevin dynamics, and energy-aware stopping for efficient code generation.
• Empirical evaluations show significant improvements in accuracy, latency reduction, and output calibration across text and multimodal applications.

Searching arXiv for the cited papers to ground the article in current literature. Energy-Based Decoding (EBD) denotes a family of inference-time procedures in which decoding is guided by an explicit energy over candidate outputs, latent variables, or internal representations, rather than by direct next-token selection from a final-layer distribution alone. In recent arXiv literature, the term spans several distinct but related constructions: reward-tilted sampling from frozen pre-trained LLMs, differentiable constrained text generation with Langevin dynamics, layer selection by minimum energy in large vision-LLMs, and energy-aware stopping policies motivated by the measured energy cost of inference in software-development workloads (Wang et al., 27 May 2026, Qin et al., 2022, Liu et al., 10 Jul 2025, Solovyeva et al., 5 Feb 2026).

1. Scope and conceptual variants

A common misconception is that Energy-Based Decoding refers to a single standardized decoder. The literature instead uses the term for several inference mechanisms that share an energy-centered decision rule but differ in what the energy represents and how it is optimized.

Formulation	Energy object	Representative paper
Reward-guided LLM decoding	Reward-tilted full-response distribution	(Wang et al., 27 May 2026)
Constrained text generation	Sequence-level differentiable energy over soft tokens	(Qin et al., 2022)
LVLM hallucination mitigation	Per-layer token-level energy from logits	(Liu et al., 10 Jul 2025)
Energy-aware stopping in code LLMs	Inference energy cost across prefill and decoding phases	(Solovyeva et al., 5 Feb 2026)

In the reward-guided formulation, a frozen base LLM defines a prior over full responses, and decoding targets a reward-tilted posterior that remains anchored to that prior. In COLD, the energy is a weighted sum of differentiable constraint functions defined on continuous token logits. In LVLM hallucination mitigation, energy is computed from intermediate-layer token logits, and the decoder dynamically selects the layer with minimal energy. In software-development workloads, the paper does not use the label “Energy-Based Decoding,” but its phase-level and token-level measurements are explicitly synthesized as a basis for designing EBD strategies (Solovyeva et al., 5 Feb 2026).

2. Formal principles

The most explicit probabilistic formulation appears in reward-guided EBD for frozen pre-trained LLMs. For a prompt $\mathbf{x}$ and full response $\mathbf{y}$, the base model induces a prior

$p_\theta(\mathbf{y}\mid\mathbf{x}),$

and a reward model provides a score that is standardized per prompt. EBD then defines the KL-regularized objective

$$\max_{q(\cdot \mid \mathbf{x})} \;\; \mathbb{E}_{\mathbf{y}\sim q(\cdot\mid\mathbf{x})} \left[ S_{\mathbf{x}}(\mathbf{y}) \right] - \frac{1}{\beta} \mathcal{D}_{\mathrm{KL}}\!\left( q(\cdot\mid\mathbf{x}) \;\big\| \; p_\theta(\cdot\mid\mathbf{x}) \right),$$

whose unconstrained optimum is the reward-tilted target distribution

$$\pi_\beta^*(\mathbf{y} \mid \mathbf{x}) = \frac{1}{Z_\beta(\mathbf{x})} \; p_\theta(\mathbf{y} \mid \mathbf{x}) \exp\!\left( \beta S_{\mathbf{x}}(\mathbf{y}) \right).$$

This can be written as an energy-based model with

$$E_{\mathbf{x}}(\mathbf{y}) = -\log p_\theta(\mathbf{y} \mid \mathbf{x}) -\beta S_{\mathbf{x}}(\mathbf{y}) .$$

The paper’s central point is that low-energy responses are both prior-plausible and high-reward (Wang et al., 27 May 2026).

COLD uses a different construction. It defines an energy directly over a soft sequence $\tilde{\mathbf{y}}$,

$$p(\tilde{\mathbf{y}}) \propto \exp\{-E(\tilde{\mathbf{y}})\}, \qquad E(\tilde{\mathbf{y}}) = -\sum_i \lambda_i f_i(\tilde{\mathbf{y}}),$$

where the $f_i$ are differentiable constraint functions. The paper instantiates these constraints with LM-based fluency terms, future-token prediction, and differentiable n-gram similarity, then performs approximate sampling by Langevin dynamics in logit space before projecting back to discrete tokens (Qin et al., 2022).

In LVLM hallucination mitigation, energy is neither a full-sequence prior nor a differentiable sequence-level constraint. Instead, for a layer-$k$ hidden state $\mathbf{y}$0 at decoding step $\mathbf{y}$1, the vocabulary head $\mathbf{y}$2 yields logits $\mathbf{y}$3, and energy is defined as

$\mathbf{y}$4

The decoder selects

$\mathbf{y}$5

and uses that layer’s logits for token selection. This makes energy a layer-selection criterion rather than a target distribution over full responses (Liu et al., 10 Jul 2025).

3. Reward-guided decoding of frozen pre-trained LLMs

The 2026 paper “The Missing Piece in Pre-trained Model Evaluation: Reward-Guided Decoding Unlocks Task-Oriented Behavior Without Parameter Updates” presents EBD as a training-free, reward-guided framework for activating task-oriented behaviors from frozen pre-trained LLMs across both open-ended and objective tasks. The method addresses a specific evaluation problem: base pre-trained models are optimized for next-token prediction and often fail to follow instructions or produce well-formed answers under standard prompting and direct decoding, so benchmark performance can conflate underlying capability with decoding-induced failure (Wang et al., 27 May 2026).

Algorithmically, EBD has two stages. Stage I samples a small initialization pool from the prior, computes prompt-local reward statistics, standardizes the reward into an advantage score, and initializes the chain at the best sample. Stage II performs block-wise Metropolis–Hastings refinement: the current response is partitioned into contiguous blocks, a cut position is sampled, the suffix is regenerated from the base model’s conditional prior, and the proposal is accepted with a Metropolis–Hastings ratio that simplifies to a pure reward difference because of proposal-prior cancellation. The implementation defaults are $\mathbf{y}$6, $\mathbf{y}$7, $\mathbf{y}$8, $\mathbf{y}$9, and $p_\theta(\mathbf{y}\mid\mathbf{x}),$0 (Wang et al., 27 May 2026).

Empirically, the paper reports that EBD outperforms baselines across five models and six benchmarks. On AlpacaEval2.0, Qwen3-8B-Base improves from 8.811 under Direct decoding to 44.519 under EBD. On objective tasks, Qwen2.5-7B improves from 0.398 to 0.557 averaged over GPQA, Math500, and HumanEval, and Olmo-3-7B improves from 0.329 to 0.538. The method also changes behavior in a way the authors quantify directly: Pearson correlation of binary correctness across 1,000 questions between a base model and its instruct-tuned counterpart rises on average from 0.256 to 0.385, and for Meta-Llama-3-8B from 0.112 to 0.307. Relative to Power Sampling, EBD is also substantially cheaper: for Mistral-7B-v0.3 on Math500, the table reports 1166.87 s/question for Power Sampling and 66.17 s/question for EBD, while the abstract summarizes this as an 18.9x latency reduction (Wang et al., 27 May 2026).

These results establish one influential sense of EBD: approximate sampling from a KL-regularized, reward-tilted posterior over full responses, without parameter updates.

4. Constrained text generation and continuous-sequence energies

COLD, introduced in 2022 as “COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics,” is an explicit instantiation of energy-based decoding for text. Its key move is to replace left-to-right local token decisions with sampling from a global energy landscape defined over an entire soft sequence of token logits. The method operates directly on off-the-shelf left-to-right LLMs and does not require task-specific fine-tuning (Qin et al., 2022).

The soft sequence $p_\theta(\mathbf{y}\mid\mathbf{x}),$1 carries a logit vector at each position. Constraint functions are differentiable with respect to these logits, so COLD can backpropagate through the LM and update the whole sequence jointly. The paper instantiates four classes of constraint: a soft forward-LM fluency term, an optional reverse-LM fluency term, a future-token prediction term, and a differentiable n-gram similarity term. These are combined into task-specific energies for abductive reasoning, counterfactual story rewriting, and lexically constrained generation. Sampling is performed by discretized Langevin dynamics,

$p_\theta(\mathbf{y}\mid\mathbf{x}),$2

with annealed noise and final LM-guided top-$p_\theta(\mathbf{y}\mid\mathbf{x}),$3 discretization (Qin et al., 2022).

The empirical profile is mixed in a characteristic way. On abductive reasoning, COLD improves over Left-only and DeLorean on BLEU$p_\theta(\mathbf{y}\mid\mathbf{x}),$4, ROUGE-L, CIDEr, and BERTScore, and human evaluation shows the best right-context coherence and overall coherence. On CommonGen canonical lexical constraints, COLD achieves the best coverage, with 4.24 covered words and 94.50% coverage, compared with NeuroLogic’s 3.30 and 91.00%, but NeuroLogic retains lower perplexity, 28.61 versus 54.98, and higher human fluency, 2.53 versus 2.07. The method therefore exemplifies a recurring trade-off in EBD: stronger global constraint satisfaction may come with some loss of local fluency (Qin et al., 2022).

COLD is important historically because it makes “energy-based decoding” explicit at the algorithmic level: energy functions are specified first, and decoding is then recast as continuous-space sampling under those energies.

5. Energy-aware decoding in software-development inference

The paper “Towards Green AI: Decoding the Energy of LLM Inference in Software Development” studies energy in a different sense: literal GPU energy consumption during inference. It decomposes inference into a prefill phase and a decoding phase, measures GPU power every 0.1 s with NVML, timestamps each generated token, and analyzes energy at both phase level and token level. The paper does not itself define an algorithm named Energy-Based Decoding, but it explicitly synthesizes its results as useful for designing EBD strategies (Solovyeva et al., 5 Feb 2026).

Its central empirical claim is that higher prefill cost amplifies the per-token energy cost in decoding. Comparing short-input and long-input workloads with the same output ceiling, the paper reports amplification factors ranging from 1.3% to 51.8%. For CodeLlama-7B, the initial decoding cost increases from approximately 5.2 J to approximately 7.74 J, a 48.8% increase; for Phi4-4B, the corresponding increase is from 4.54 J to 4.60 J, only 1.3%. The study also shows that prefill contribution depends strongly on workload geometry: on short-prompt code generation, prefill contributes less than 1–1.4% of total energy, while on LongBench code understanding with 4000–8000 input tokens and at most 10 output tokens, prefill dominates with 67.3–84.4% of total energy (Solovyeva et al., 5 Feb 2026).

The paper’s most concrete EBD-relevant mechanism is babbling suppression. It defines babbling behavior as excessive, unnecessary tokens beyond what is needed to solve the task, including redundant solution variants, extra white space, comments, alternative implementations, and extended reasoning that does not affect pass@1. Three models—CodeLlama-7B, Deepseek-Coder-6.7B, and Qwen3-4B—are identified as clear babblers. The authors implement test-driven early stopping: after each end-of-line token, generated code is checked for compilation, then against HumanEval test cases; if all tests pass, decoding stops immediately. Under this policy, token reductions range from 44% to 93% and energy reductions from 44% to 89%, while pass@1 remains essentially unchanged. For example, at a 1000-token budget, Deepseek-6.7B drops from 981 to 72 output tokens and from 5338 J to 597 J, a reduction of 89% (Solovyeva et al., 5 Feb 2026).

This work reframes EBD as a utility–energy trade-off. The paper explicitly proposes the interpretation

$p_\theta(\mathbf{y}\mid\mathbf{x}),$5

or, in budget form,

$p_\theta(\mathbf{y}\mid\mathbf{x}),$6

A plausible implication is that EBD can refer not only to semantic control of outputs but also to decoding policies that internalize the physical cost of generation itself.

6. Energy-guided decoding in large vision-LLMs

In multimodal generation, the 2025 paper “Energy-Guided Decoding for Object Hallucination Mitigation” uses energy as a calibration signal over decoder layers. The motivation is object hallucination in LVLMs, especially yes/no bias in visual question answering. The paper defines yes ratio as the fraction of “Yes” answers among total questions and yes-ratio gap as

$p_\theta(\mathbf{y}\mid\mathbf{x}),$7

Across POPE, MME, and MMVP, standard greedy decoding from the last layer produces a substantial imbalance in this ratio, and the paper shows that the yes ratio is already high even without images, indicating a strong language prior (Liu et al., 10 Jul 2025).

The proposed method is simple: compute token logits from every hidden layer, convert each layer’s last-token logits into an energy score with $p_\theta(\mathbf{y}\mid\mathbf{x}),$8, and decode from the layer with minimal energy. The paper emphasizes that the token selection rule itself is not changed; greedy decoding remains greedy, and other decoding rules can also be used. What changes is the source layer of the logits. Empirically, the penultimate layer is often selected, which aligns with the paper’s observation that late layers may over-amplify language priors (Liu et al., 10 Jul 2025).

The gains are consistent. The abstract reports an average accuracy improvement of 4.82% compared to greedy decoding and an average yes-ratio gap reduction of 8.81%. On GQA-Adversarial with LLaVA-1.5, accuracy rises from 69.43 to 79.63, F1 from 75.85 to 81.16, and $p_\theta(\mathbf{y}\mid\mathbf{x}),$9 falls from 0.2657 to 0.0809. On MME with LLaVA-1.5, the total score rises from 531.67 under Greedy+HALC to 641.67 under EBD. On MMVP with LLaVA-1.5, accuracy rises from 57.38 to 64.75 while the yes-ratio gap drops from 0.3279 to 0.0902. The paper also reports improved calibration: the gap between empirical accuracy and mean confidence is reduced for both “Yes” and “No” answers (Liu et al., 10 Jul 2025).

This line of work broadens the meaning of EBD. Energy need not score full candidate sequences; it can also score intermediate representations and thereby govern which internal state is trusted at each decoding step.

7. Adjacent usages, image decoding, and terminological ambiguity

The broader literature contains adjacent constructions that are not identical to the reward-guided or layer-selection formulations but are plainly energy-based decoders in spirit. In “How to Construct Energy for Images? Denoising Autoencoder Can Be Energy Based Model,” Denoising-EBM decomposes image energy into “semantic energy” in latent space and “texture energy” in pixel space. Sampling is explicitly two-stage: fast MCMC in latent space synthesizes semantic images, then Langevin refinement in pixel space adds detail. The texture energy is

$$\max_{q(\cdot \mid \mathbf{x})} \;\; \mathbb{E}_{\mathbf{y}\sim q(\cdot\mid\mathbf{x})} \left[ S_{\mathbf{x}}(\mathbf{y}) \right] - \frac{1}{\beta} \mathcal{D}_{\mathrm{KL}}\!\left( q(\cdot\mid\mathbf{x}) \;\big\| \; p_\theta(\cdot\mid\mathbf{x}) \right),$$0

and the semantic energy is

$$\max_{q(\cdot \mid \mathbf{x})} \;\; \mathbb{E}_{\mathbf{y}\sim q(\cdot\mid\mathbf{x})} \left[ S_{\mathbf{x}}(\mathbf{y}) \right] - \frac{1}{\beta} \mathcal{D}_{\mathrm{KL}}\!\left( q(\cdot\mid\mathbf{x}) \;\big\| \; p_\theta(\cdot\mid\mathbf{x}) \right),$$1

On CIFAR-10, the model reports FID 21.24 and IS 7.86, and the paper explicitly frames the procedure as decoding from latent noise to final images by following negative gradients of learned energies (Zeng, 2023).

A related but explicitly “energy-inspired” construction appears in self-supervised vision pretraining. There, a backbone plus a scalar head defines $$\max_{q(\cdot \mid \mathbf{x})} \;\; \mathbb{E}_{\mathbf{y}\sim q(\cdot\mid\mathbf{x})} \left[ S_{\mathbf{x}}(\mathbf{y}) \right] - \frac{1}{\beta} \mathcal{D}_{\mathrm{KL}}\!\left( q(\cdot\mid\mathbf{x}) \;\big\| \; p_\theta(\cdot\mid\mathbf{x}) \right),$$2, and corrupted inputs are restored by iterative input-space updates

$$\max_{q(\cdot \mid \mathbf{x})} \;\; \mathbb{E}_{\mathbf{y}\sim q(\cdot\mid\mathbf{x})} \left[ S_{\mathbf{x}}(\mathbf{y}) \right] - \frac{1}{\beta} \mathcal{D}_{\mathrm{KL}}\!\left( q(\cdot\mid\mathbf{x}) \;\big\| \; p_\theta(\cdot\mid\mathbf{x}) \right),$$3

The decoder is therefore the backward pass itself. The paper supports masked image modeling, patch sorting, super-resolution, denoising, and colorization within this single mechanism, and reports that a ViT-B pretrained for 800 epochs and finetuned for 100 epochs reaches 83.5% on ImageNet-1K (Wang et al., 2023). This suggests that EBD can be understood more generally as inference by energy minimization, even when the primary goal is representation learning rather than generation.

The acronym is also ambiguous across fields. In coding theory, “Error-Building Decoding of Linear Block Codes” uses EBD to denote error-building decoding, a maximum-likelihood soft-decision decoding framework for linear block codes that searches for a globally minimum-energy error pattern using only the parity-check matrix. For extended Hamming codes of lengths 64, 128, and 256, the fully optimized method requires approximately an order of magnitude fewer floating-point operations on average than minimum-edge trellis Viterbi decoding at a frame error rate of $$\max_{q(\cdot \mid \mathbf{x})} \;\; \mathbb{E}_{\mathbf{y}\sim q(\cdot\mid\mathbf{x})} \left[ S_{\mathbf{x}}(\mathbf{y}) \right] - \frac{1}{\beta} \mathcal{D}_{\mathrm{KL}}\!\left( q(\cdot\mid\mathbf{x}) \;\big\| \; p_\theta(\cdot\mid\mathbf{x}) \right),$$4 (Qiu et al., 5 Jan 2026). This is an energy-based decoder in the literal optimization sense, but it is terminologically separate from the recent generative-model literature.

Taken together, these usages show that Energy-Based Decoding is best understood as a methodological family rather than a single algorithm. The recurring invariant is the replacement of direct decoding by an explicit energy criterion—over responses, soft token trajectories, decoder layers, latent variables, or computational cost—followed by sampling, refinement, selection, or early stopping according to that criterion.