---
title: Energy-Based Decoding
url: https://www.emergentmind.com/topics/energy-based-decoding
type: topic
---

# Energy-Based Decoding

Searching arXiv for recent and foundational papers on energy-based decoding across text generation, LVLMs, and codec/system interpretations.
arXiv_search.query({"search_query":"all:\"Energy-based Decoding\" OR ti:\"COLD Decoding\" OR ti:\"Energy-Guided Decoding\" OR ti:\"Decoding-Energy-Rate-Distortion\"","max_results":10,"sort_by":"relevance","sort_order":"descending"})
Energy-based decoding is a family of decoding-time methods in which an energy quantity influences candidate generation, selection, or refinement. In contemporary research, the term is used in several technically distinct ways. In neural text generation, it often denotes a scalar defined over complete sequences, so decoding becomes approximate inference in an energy-based model rather than left-to-right token choice; representative formulations include \(E(\mathbf{y})=-\sum_i \lambda_i f_i(\mathbf{y})\) and \(E_{\mathbf{x}}(\mathbf{y})=-\log p_\theta(\mathbf{y}\mid \mathbf{x})-\beta S_{\mathbf{x}}(\mathbf{y})\) [2202.11705] [2605.28020]. In large vision-language models, it can denote a per-layer logit score \(Energy(h_t^k)=-\logsumexp[\mathcal H(h_t^k)]\) used to choose the layer from which the next token is decoded [2507.07731]. In video coding and systems work, the same phrase instead refers to minimizing the physical energy required by the decoder, for example through objectives such as \(J=D+\lambda_R R+\lambda_E E\) or by comparing the GPU energy of alternative decoding policies [2203.01099] [2502.11723]. In communication-theoretic and VLSI analyses, it refers even more literally to the physical energy of iterative decoders and decoder circuits [1309.4374] [1412.4130].

## 1. Terminological scope and recurrent structures

Across the cited literature, “energy-based decoding” does not identify a single formalism. It names several practices in which a scalar energy, or a measured/estimated physical energy, becomes part of the decoding rule. The commonality is not the mathematical form alone, but the relocation of decoding away from pure local likelihood maximization toward a broader optimization or inference problem.

| Usage in the literature | Energy quantity | Representative papers |
|---|---|---|
| Sequence-level neural decoding | Global sequence energy or reward-tilted energy | [2202.11705], [2605.28020] |
| LVLM layer selection | Per-layer logit energy \(-\logsumexp\) | [2507.07731] |
| Video codec optimization | Predicted decoder processing energy | [2203.01099], [2209.10266], [2204.10151], [2206.13483] |
| Systems and hardware studies | Measured GPU energy, iterative decoding energy, or VLSI area-time energy | [2502.11723], [1309.4374], [1412.4130] |

This terminological breadth matters because superficially similar language can refer to very different decoded objects. In COLD and EBD, the decoded object is a full text response or a relaxed soft-token sequence; in the LVLM setting, the decoded object remains the next token, but the decoding layer changes dynamically; in codec papers, the immediate decision variable is often an encoder-side mode or tool choice chosen to reduce downstream decoder energy; and in hardware theory, the object of study is the physical energy required to perform decoding at all [2202.11705] [2203.01099] [1412.4130].

## 2. Sequence-level energy models in text generation

A canonical sequence-level formulation appears in COLD decoding, which casts constrained text generation as inference in an energy-based model over whole sequences rather than as left-to-right token selection. Constraint scores \(f_i(\mathbf{y})\in\mathbb{R}\) are combined into
\[
p(\mathbf{y})=\exp\left\{\sum_i \lambda_i f_i(\mathbf{y})\right\}/Z,
\qquad
E(\mathbf{y})=-\sum_i \lambda_i f_i(\mathbf{y}),
\]
with nonnegative weights \(\lambda_i\ge 0\). Because text is discrete, COLD relaxes each token position into a logit vector \(\tilde{\mathbf{y}}_t\in\mathbb{R}^V\), forms soft tokens with temperature-scaled softmax, and performs Langevin dynamics over the full relaxed sequence:
\[
\tilde{\mathbf{y}}^{(n+1)}\leftarrow \tilde{\mathbf{y}}^{(n)}-\eta \nabla_{\tilde{\mathbf{y}}}E(\tilde{\mathbf{y}}^{(n)})+\epsilon^{(n)},
\qquad
\epsilon^{(n)}\sim \mathcal N(0,\sigma^{(n)}).
\]
The framework uses a left-to-right LM fluency term, can optionally add a reverse-LM term, and instantiates additional differentiable energies such as future-token prediction and differentiable \(n\)-gram similarity. It is applied without task-specific fine-tuning to lexically constrained generation, abductive reasoning, and counterfactual reasoning. The reported procedure initializes from greedy LM decoding logits, typically uses \(N=2000\) and \(\eta=0.1\), anneals \(\sigma^{(n)}\) through \(\{1,0.5,0.1,0.05,0.01\}\) at iterations \(\{0,50,500,1000,1500\}\), and finally discretizes with an LM-guided top-\(k\) “guardian” rule rather than naive argmax. Empirically, the method outperforms DeLorean and left-only LM baselines on abductive reasoning, improves coherence over DeLorean on counterfactual rewriting, significantly outperforms Mix-and-Match, and achieves higher keyword coverage than NeuroLogic, while exhibiting a recurring fluency–constraint tradeoff; runtime is reported as about \(33.6\) seconds per sample with GPT2-XL and \(22.7\) seconds with GPT2-M on counterfactual rewriting [2202.11705].

A later sequence-level variant, explicitly titled Energy-Based Decoding, replaces soft-token Langevin refinement with reward-guided posterior sampling over complete responses from frozen base LLMs. EBD begins from the KL-regularized objective
\[
\max_{q} \;
\mathbb E_{\mathbf y\sim q}[S_{\mathbf x}(\mathbf y)]
-\frac{1}{\beta}\mathcal D_{\mathrm{KL}}(q\|p_\theta),
\]
whose optimizer is the reward-tilted target
\[
\pi_\beta^*(\mathbf y\mid \mathbf x)
\propto
p_\theta(\mathbf y\mid \mathbf x)\exp(\beta S_{\mathbf x}(\mathbf y)).
\]
In energy form,
\[
E_{\mathbf x}(\mathbf y)
=
-\log p_\theta(\mathbf y\mid \mathbf x)-\beta S_{\mathbf x}(\mathbf y).
\]
The implementation uses a prompt-normalized standardized advantage
\[
\mathcal A_{\mathbf x}(\mathbf y)=\frac{R(\mathbf y,\mathbf x)-\mu_R(\mathbf x)}{\sigma_R(\mathbf x)},
\]
initializes from a small pool of prior samples, and refines responses with a short block-wise Metropolis–Hastings chain that preserves a prefix, regenerates a suffix from the matched conditional prior, and accepts proposals with
\[
\alpha=\min\left(1,\exp\left(\beta\left[\mathcal A_{\mathbf x}(\mathbf y')-\mathcal A_{\mathbf x}(\mathbf y)\right]\right)\right).
\]
Because the proposal matches the conditional prior, proposal and prior terms cancel in the MH ratio. Default settings are \(\beta=3.5\), \(\tau=1\), \(K=12\), \(n_{\mathrm{init}}=4\), \(M=12\), and \(L_{\max}=3072\). On five base models and six benchmarks, EBD improves both objective and subjective evaluations relative to direct decoding and Power Sampling; the abstract highlights Qwen3-8B-Base on AlpacaEval2.0 from \(8.8\) to \(44.5\), an \(18.9\times\) Math500 latency reduction for Mistral-7B relative to prior decoding work, and robustness to reward-model size from \(0.6\)B to \(8\)B [2605.28020].

Taken together, these two frameworks show two distinct sequence-level regimes. COLD performs approximate sampling in a continuous relaxation of text with Langevin dynamics. EBD keeps text discrete and instead samples from a reward-tilted posterior with matched block-wise MH proposals. Both reject the assumption that decoding should be identical to ancestral next-token selection, but they do so with markedly different inference mechanisms [2202.11705] [2605.28020].

## 3. Energy-guided layer selection in vision-language decoding

In large vision-language models, energy-guided decoding is neither an EBM over full responses nor an optimization over physical power. The method introduced for object hallucination mitigation inspects hidden states from all decoder layers at each generation step, projects each layer’s last-position hidden state through the existing LM head, and computes the energy score
\[
Energy(h_t^k)=-\logsumexp[\mathcal H(h_t^k)].
\]
The chosen layer is
\[
k^*=\arg\min_k Energy(h_t^k),
\]
and decoding then proceeds from \(\mathcal H(h_t^{k^*})\). The method is hyperparameter-free in the energy definition, requires no retraining, no finetuning, no extra model, no visual perturbation, no prompt tuning, and no contrastive pair of distributions. It is evaluated on LLaVA-1.5, InstructBLIP, and mPLUG-Owl2 across POPE, MME, MMVP, and CHAIR, with a maximum of 16 new tokens for yes/no evaluations and temperature \(1\) [2507.07731].

The motivating observation is a strong “Yes”-ratio imbalance in balanced yes/no VQA datasets. The paper argues that ordinary final-layer decoding often over-predicts “Yes,” partly due to language-prior transfer, and that the final layer is not always the most reliable source for the next-token distribution. Selecting the minimum-energy layer is presented as a way to reduce that bias and improve calibration. Empirically, the abstract reports an average accuracy improvement of \(4.82\%\) over greedy decoding and an average yes-ratio gap reduction of \(8.81\%\). The detailed tables show especially large gains on harder POPE settings such as LLaVA-1.5 on GQA adversarial, where accuracy rises from \(69.43\) to \(79.63\), F1 from \(75.85\) to \(81.16\), and \(\Delta_{\text{gap}}\) drops from \(26.57\) to \(8.09\). On MME, LLaVA-1.5 improves from \(531.67\) to \(641.67\), and on MMVP the method substantially reduces yes-ratio bias, although F1 changes are sometimes small. The layer-wise analysis indicates that the selected hidden states mostly come from the second-last layer, and the penultimate layer often has the lowest energy [2507.07731].

The method is not uniformly dominant in every condition. Some easier MSCOCO POPE settings do not improve on every metric, and in open-ended captioning the method gives the best CHAIR\(_S\) but worse CHAIR\(_I\); for LLaVA-1.5, CHAIR\(_S\) changes from \(17.8\) to \(12.2\) while CHAIR\(_I\) changes from \(5.4\) to \(9.4\). This establishes a narrower but precise meaning of energy-guided decoding: a layer-selection rule defined by log-sum-exp over vocabulary logits, used chiefly as a reliability and calibration signal during autoregressive decoding [2507.07731].

## 4. Decoder-energy-rate-distortion optimization in video coding

In video coding, energy-based decoding refers primarily to decoder-energy-aware encoding. The foundational DERDO formulation extends classical rate-distortion optimization by treating decoder processing energy as a first-class objective. Over feasible encoding solutions \(s\in S\), the constrained problem is
\[
\min_{s\in S} D(s)
\quad\text{s.t.}\quad
R(s)\le R_{\max},\;
E(s)\le E_{\max},
\]
with Lagrangian relaxation
\[
\min J(s)=D(s)+\lambda_R R(s)+\lambda_E E(s).
\]
A central practical contribution is the encoder-side parameter \(\tau\in[0,1]\), which continuously shifts operation from pure RDO \((\tau=0)\) toward pure decoding-energy optimization \((\tau=1)\). The paper integrates this objective into a modified HM-14.0 HEVC encoder using a 27-feature bitstream model trained on FFmpeg measurements, and validates the two-multiplier QP relation
\[
\mathrm{QP}=4+3\cdot \mathrm{ld}(\rho\lambda_R+\varepsilon\lambda_E).
\]
Measured on a Pandaboard with FFmpeg 2.8, libde265 0.7, and HM-13.0, the method yields up to about \(30\%\) measured decoder energy reduction at equal objective quality, with bitrate increase of similar order; the paper reports, for example, class-averaged FFmpeg results at \(\tau=0.2\) of about \(10\)–\(21\%\) BDDE with only \(0.4\)–\(5.6\%\) BDR increase, and at \(\tau=0.5\) about \(20\)–\(30\%\) BDDE with \(15\)–\(52\%\) BDR penalties [2203.01099].

Such optimization depends on accurate energy estimation. A closely related line of work models video decoding energy directly from bitstream features using
\[
\hat E=\sum_{f=1}^{F} n_f e_f.
\]
For HEVC, H.264, H.263, and VP9, fewer than 20 features suffice: \(F=11\) for H.263, \(14\) for H.264, and \(19\) for HEVC and VP9. Using 10-fold cross-validation and external measurements with a ZES Zimmer LMG95 on a Pandaboard, mean relative estimation errors on FFmpeg are \(5.27\%\) for HEVC, \(6.41\%\) for H.264, \(2.51\%\) for H.263, and \(5.11\%\) for VP9; on alternative implementations the method reaches \(3.18\%\) for HEVC on libde265, \(0.77\%\) for H.263 on TMN-2.0, and \(7.50\%\) for a hardware-accelerated H.264 decoder. The paper uses these results to support rate-distortion-energy analysis before playback rather than to define a decoder-side probabilistic energy [2204.10151].

For VVC, the same modeling philosophy is extended with richer feature sets. The paper on VVC decoding-energy modeling emphasizes that VVC random-access decoding energy increases by over \(80\%\) on average relative to HEVC and proposes two feature-based models, FVS with 67 features and FV with 230 features, both learned by trust-region-reflective least squares from VTM-7.0 measurements on an Intel i7-8700 using RAPL. With 10-fold cross-validation, FV achieves \(1.85\%\) mean relative estimation error on the merged dataset, which the paper frames as sufficiently accurate for subsequent DERDO-style optimization [2209.10266].

A more deployment-oriented VVC study moves from modeling to practical coding-tool profiles in VVenC 1.3.1 and VVdeC 1.3.0. Starting from an aggressive Energy Efficient profile and then selectively restoring tools such as restricted ALF/CCALF, deeper partitioning, affine/PROF, and LMChroma, it defines two recommended profiles. The bitrate-efficient profile v2568 achieves JVET CTC averages of \(\mathrm{BDR}=4.84\%\) and \(\mathrm{BDDE}=-25.84\%\), matching the abstract’s statement of over \(25\%\) energy-efficiency improvement with bitrate increase below \(5\%\). The more aggressive profile v58 achieves \(\mathrm{BDR}=14.39\%\) and \(\mathrm{BDDE}=-34.07\%\), corresponding to \(34\%\) energy savings at bitrate increases below \(15\%\). Here again, the “energy-based” component is chiefly encoder-side control of decoder-expensive syntax and tools rather than a decoder-side inference rule [2206.13483].

## 5. Physical energy of decoding strategies and decoder circuits

A systems-oriented interpretation appears in work on GPU energy consumption during LLM inference. This literature is explicit that it is not about energy-based models, but about the physical energy implications of decoding choices. A comparative study on Qwen2.5-7B-Instruct evaluates 12 strategies—greedy, beam search, diverse beam search, contrastive search, DoLa, assisted decoding, temperature sampling, top-\(k\), top-\(p\), epsilon, typical, and min-\(p\)—over WMT16 translation, CodeXGLUE code summarization, and GSM8K, measuring GPU energy with \(E=\sum_i P_i\Delta t_i\) from \(1\) s nvidia-smi samples. It reports that decoding choice often changes energy much more than it changes task quality, that beam and DBS typically improve metrics at substantial energy cost, that contrastive search can be quality-optimal but very energy-expensive, and that assisted decoding often yields the best efficiency ratio \(ER=\text{Quality}/\text{Energy}\). On German\(\rightarrow\)English, for example, beam search reaches BLEU \(42.71\) at \(11.45\) Wh, while assisted decoding reaches BLEU \(41.76\) at \(8.33\) Wh and the best ER \(5.01\); on GSM8K, beam reaches \(0.90\) exact match at \(20.32\) Wh, whereas assisted decoding reaches \(0.80\) at \(10.63\) Wh and the best ER \(0.075\) [2502.11723].

Communication and coding theory use the term even more literally. In Wireless Body Area Sensor Networks, iterative LDPC decoding is analyzed from the viewpoint that decoder energy can exceed transmit energy on short links. The paper models decoder energy as increasing linearly with the number of computational nodes and iterations, lower bounded by
\[
E_{\text{dec}}\ge E_{\text{node}}\,m\,l,
\]
and proposes Adaptive Iterative Decoding, an early-stopping rule that halts once a BER target of \(10^{-4}\) is reached. At \(\mathrm{SNR}=1.5\) dB, the required iteration counts are \(16\) for \(R_c=0.75\) and \(12\) for \(R_c=0.5\); compared with a 50-iteration baseline, the paper reports total energy reduction of \(20\)–\(25\%\) and gives an example in which energy drops from about \(1.3\) J to about \(1.05\) J [1309.4374].

A different communication-theoretic usage appears in bit-interleaved coded energy-based modulation with iterative decoding. There, “energy-based” refers to modulation symbols whose information is carried in energy levels under non-coherent reception, and the decoding problem is solved with an iterative BICM-ID-style receiver. The central non-coherent likelihood is
\[
f([\mathbf y_1]_v,\ldots,[\mathbf y_R]_v\mid [\mathbf s]_v=s)
=
\frac{\exp\!\left(-\frac{\sum_{a=1}^{R}|[\mathbf y_a]_v|^2}{s^2+N_0}\right)}
{\pi^R(s^2+N_0)^R},
\]
and the paper derives FF and EFF pairwise-error bounds, shows that nearest neighbors differ from the coherent BICM-ID case, and proves that the mapping from bits to energy levels influences diversity order and coding gain [2211.12655].

At the most abstract end, VLSI theory studies the physical energy of decoder circuits through Thompson’s model, where
\[
E=\xi A\tau.
\]
For families of circuits decoding over a binary erasure channel, the paper proves that, as blocklength grows, either block error probability becomes asymptotically lower bounded by \(1/2\) or total decoding energy scales at least as \(\Omega(n(\log n)^{1/2})\), implying energy per decoded bit \(\Omega((\log n)^{1/2})\). For serial computation with a constant number of output pins, the lower bound strengthens to \(\Omega(n\log n)\); in a more general setting with varying output pins, it becomes \(\Omega(n(\log n)^{1/5})\). A further implication is that the average energy per decoded bit must approach infinity for any sequence of codes that approaches capacity [1412.4130].

## 6. Trade-offs, misconceptions, and comparative interpretation

The surveyed literature establishes that “energy” in decoding is not a single object. It may be a global score over whole text sequences, a reward-tilted posterior energy, a per-layer logit reliability score, a predicted decoder processing cost extracted from bitstream features, the actual GPU energy used by an inference policy, the iteration-dependent energy of a message-passing decoder, or the area-time energy of a VLSI circuit [2202.11705] [2507.07731] [2203.01099] [2502.11723] [1412.4130]. A persistent misconception is therefore to equate all energy-based decoding with energy-based models; the codec and systems literature uses the same phrase for physically measured or estimated energy without introducing an EBM over outputs.

A second recurring pattern is that energy-aware decoding nearly always exposes a trade-off rather than a uniformly dominant operating point. In COLD, stronger task constraints can lower grammaticality, and top-\(k\) discretization mediates grammar against right/overall coherence [2202.11705]. In reward-guided EBD, the prior–reward balance is controlled by \(\beta\), and the method remains only an approximate sampler because it runs a short MH chain [2605.28020]. In LVLMs, energy-guided layer selection reduces yes-ratio bias and improves calibration, but CHAIR\(_I\) can worsen even when CHAIR\(_S\) improves [2507.07731]. In HEVC and VVC, lower decoder energy is exchanged against bitrate and sometimes encoding time [2203.01099] [2206.13483]. In energy-conscious LLM serving, beam-like methods often buy modest quality gains at disproportionate energy cost, while assisted decoding and simple stochastic truncation rules offer better quality-per-energy trade-offs [2502.11723]. In short-range communication, aggressive transmit-power minimization can backfire because it raises decoding iterations and hence decoding energy [1309.4374]. In circuit theory, approaching capacity forces diverging energy per decoded bit [1412.4130].

A plausible synthesis is that the literature has converged on a shared design principle rather than a single formalism: decoding should be treated as an optimization or inference stage whose objective need not be identical to local next-token likelihood or local rate-distortion cost. What differs across domains is the status of the energy term—probabilistic, heuristic, predictive, or physical. That distinction determines both the mathematical tools used, such as Langevin dynamics, Metropolis–Hastings, log-sum-exp layer scoring, or Lagrangian rate-energy-distortion optimization, and the failure modes that dominate in practice, such as discretization error, reward dependence, calibration drift, bitrate penalty, or hardware energy overhead [2202.11705] [2605.28020] [2507.07731] [2203.01099] [2502.11723].

Source: https://www.emergentmind.com/topics/energy-based-decoding