The Missing Piece in Pre-trained Model Evaluation: Reward-Guided Decoding Unlocks Task-Oriented Behavior Without Parameter Updates

Abstract: With the rapid progress of LLMs, reliably evaluating the capabilities of pre-trained LLMs has become increasingly important. The challenge is that base pre-trained models are optimized for next-token prediction and often fail to follow instructions or produce well-formed answers under standard prompting and direct decoding. As a result, benchmark performance can conflate model capability with decoding-induced failures to produce task-oriented outputs, while exposing such behavior often relies on costly post-training. Recent decodingonly approaches attempt to reshape output distributions, but such methods can be inefficient and brittle across open-ended tasks. To address these limitations, we propose Energy-Based Decoding (EBD), a training-free, reward-guided framework for activating task-oriented behaviors from frozen pre-trained LLMs across both open-ended and objective tasks. EBD augments decoding with an external lightweight reward model, steering generations toward high-utility responses while anchoring them to the pre-trained model prior through a reward-tilted target distribution. We show that EBD shifts base-model outputs toward more instructionfollowing behavior, increasing behavioral similarity to post-trained counterparts and enabling a fairer inference-time evaluation of accessible pre-trained-model behavior. Empirically, EBD outperforms baselines across five models and six benchmarks, improving Qwen3-8B-Base on AlpacaEval2.0 from 8.8 to 44.5, reducing Mistral-7B Math500 latency by 18.9x relative to prior decoding work, and remaining robust to reward-model size.

• The paper presents a reward-guided inference method that tilts the frozen model prior using a lightweight reward model.
• The methodology employs a block-wise Metropolis–Hastings sampler, achieving notable improvements such as a +405% score boost on AlpacaEval2.0 and significant latency reduction on Math500.
• The approach efficiently bridges the gap between frozen and post-trained models, enhancing task performance and providing a robust diagnostic tool for model evaluation.

Energy-Based Decoding for Frozen Pre-trained LLMs: Reward-Guided Inference Without Parameter Updates

Background and Motivation

Evaluating the true capabilities of pre-trained LLMs is confounded by issues intrinsic to direct decoding. Standard evaluation benchmarks assume models will recognize prompts as explicit tasks, but pre-trained LLMs, optimized for next-token prediction, frequently generate continuation-style outputs rather than task-completion answers. This mismatch leads to unreliable score attribution: low performance may stem from limitations in the decoding procedure rather than absence of task-relevant knowledge in the model. Likelihood-only decoding interventions—such as sharpening via temperature reduction or search-based sampling—are hyperparameter-sensitive, amplify spurious modes, and remain computationally inefficient. These approaches lack an explicit external utility signal and are fundamentally limited for eliciting task-aligned outputs from the frozen distribution.

Figure 1: Task prompts lead to behavioral mismatch; EBD augments decoding with reward guidance, activating task-oriented responses from the frozen prior.

Energy-Based Decoding: Formalism and Algorithm

EBD reformulates inference-time decoding as sampling from a reward-tilted posterior. The target distribution $\pi_\beta^*(\mathbf{y}|\mathbf{x})$ is proportional to the pre-trained prior $p_\theta(\mathbf{y}|\mathbf{x})$ exponentially tilted by a reward signal:

$$\pi_\beta^*(\mathbf{y}|\mathbf{x}) \propto p_\theta(\mathbf{y}|\mathbf{x}) \exp(\beta S_{\mathbf{x}(\mathbf{y})})$$

where $S_{\mathbf{x}(\mathbf{y})}$ is a locally standardized reward given by a lightweight reward model and $\beta$ controls the trade-off between prior fidelity and reward alignment.

EBD instantiates an efficient block-wise Metropolis–Hastings (MH) sampler: given a sampled cut in the response, suffixes are regenerated from the matched conditional prior. Acceptance occurs with probability depending only on the reward difference between the current and proposed response, due to proposal-prior cancellation. This algorithm guarantees distributional anchoring to the frozen prior with minimal computational overhead.

Figure 2: EBD operates in two stages: initialization pools for warm start, followed by block-wise MH refinement targeting the reward-tilted posterior.

Empirical Evaluation

Aggregate Quality, Latency, and Behavioral Shifts

EBD is tested across five open-source LLMs (Llama3-8B, Mistral-7B-v0.3, Qwen2.5-7B, Qwen3-8B-Base, Olmo-7B) and six benchmarks spanning both objective tasks (GPQA, Math500, HumanEval) and subjective evaluation (AlpacaEval2.0, MT-Bench, WritingBench). EBD demonstrates substantial improvements over direct decoding and Power Sampling across all settings. Notable claims supported by strong numerical results include:

• Qwen3-8B-Base on AlpacaEval2.0: Direct decoding scores 8.8, EBD achieves 44.5, a +405% improvement.
• Mistral-7B Math500 latency: EBD delivers a $18.9\times$ reduction relative to prior decoding work.
• Behavioral alignment: EBD increases similarity to post-trained models across five base/instruct pairs, raising average Pearson correlation from 0.256 to 0.385 (+50.4%).

  Figure 3: EBD’s quality/efficiency frontier: highest task scores and substantially lower inference time versus Power Sampling.

  

  Figure 4: EBD consistently shifts behavioral patterns of pre-trained models toward post-trained signatures.

  

  Figure 5: EBD closes the performance gap between pre-trained and post-trained models at inference time, often matching or surpassing post-trained baselines.

Format Adherence and Robustness to SFT

On Math500, EBD enhances valid response rates (VRR) and accuracy across both base and instruction-tuned models, regardless of SFT initialization recipe. Performance gains confirm EBD’s robustness to underlying SFT trajectories; improvement persists even as SFT progresses and when cold-started from diverse datasets.

Figure 6: Case studies show EBD transforming incoherent continuations into structured, task-formatted solutions.

Reward Model Scale and Hyperparameter Stability

EBD's downstream performance and efficiency are insensitive to reward-model scale. Lightweight reward models (e.g., 0.6B) nearly match larger variants (e.g., 8B) in both accuracy and inference latency, reducing deployment costs without impacting task utility. Hyperparameter studies indicate forgiving ranges for $\beta$, block count, and MCMC steps.

Figure 7: EBD retains high performance and low latency across reward model sizes.

Mechanistic Insights and Theoretical Implications

EBD steers outputs by distributional control—tilting the prior, not maximizing reward outright—thus avoiding reward artifacts and excessive mode collapse. Divergence from the prior is regularized by $\beta$, enabling controlled exploration. Unlike SFT, EBD does not suppress continuation modes; it operates on the full frozen distribution, offering complementary activation mechanisms to post-training. This separation exposes accessible behavioral capacity in frozen checkpoints previously masked by direct decoding limitations.

Inference-time steering via EBD provides a diagnostic for pre-trained model sensitivity and alignment. It enables a more faithful benchmark evaluation of model knowledge, decoupled from parameter updates or retraining. EBD further supports dual-use: boosting benchmark scores for deployment and revealing latent task capability for model audits.

Implications for Future Research and AI Development

EBD displaces the assumption that post-training is necessary for task-aligned generation. By demonstrating that reward-guided inference can robustly activate instruction-following, reasoning, and format-adherent behaviors at scale, this research reframes model evaluation. Practical implications include:

• Immediate improvement of frozen pre-trained checkpoints without retraining.
• Efficient large-scale evaluation and deployment in resource-constrained settings.
• Deeper mechanistic understanding of emergence and distributional control in LLMs.
• Complementarity to SFT, RLHF, and direct preference optimization, supporting mixed post-training strategies—including reward-guided inference in deployable agents.

Theoretically, EBD’s efficiency and anchoring guarantee open pathways for integrating richer reward models, developing adaptive tilting strategies, and investigating cross-modal or multi-task distributional control. As model sizes increase and fine-tuning resources plateau, inference-time reward guidance will be a foundational tool for bridging the utilization gap in LLM deployment.

Conclusion

Energy-Based Decoding (EBD) constitutes a training-free, efficient reward-guided inference framework. It consistently outperforms direct decoding in both objective and subjective benchmarks, maintains robustness across reward model families and hyperparameter settings, and narrows behavioral gaps to post-trained models without modifying parameters. EBD establishes that decoding is a critical axis for model evaluation, and that reward-guided inference is key to unlocking task-aligned behaviors from frozen pre-trained LLMs.