Temper and Tilt Lead to SLOP: Reward Hacking Mitigation with Inference-Time Alignment

Abstract: Inference-time alignment techniques offer a lightweight alternative or complement to costly reinforcement learning, while enabling continual adaptation as alignment objectives and reward targets evolve. Existing theoretical analyses justify these methods as approximations to sampling from distributions optimally tilted toward a given reward model. We extend these techniques by introducing reference-model temperature adjustment, which leads to further generalization of inference-time alignment to ensembles of generative reward models combined as a sharpened logarithmic opinion pool (SLOP). To mitigate reward hacking, we propose an algorithm for calibrating SLOP weight parameters and experimentally demonstrate that it improves robustness while preserving alignment performance.

• The paper introduces SLOP, a framework that generalizes inference-time alignment with tempered and tilted distributions to effectively mitigate reward hacking.
• The paper demonstrates that calibrating expert weights on limited data can balance reference and proxy models, yielding robust performance on VQA and GSM8K benchmarks.
• The paper validates through empirical evaluations and ablation studies that SLOP adjusts for proxy inaccuracies, outperforming traditional methods in alignment tasks.

SLOP: Inference-Time Alignment and Reward Hacking Mitigation via Sharpened Logarithmic Opinion Pooling

Overview

The paper "Temper and Tilt Lead to SLOP: Reward Hacking Mitigation with Inference-Time Alignment" (2605.13537) presents a unified framework for inference-time alignment via tempered and tilted distributions that generalize common RL approaches and sampling-based alignment (BoN, SBoN, RAD). The central contribution is the introduction and analysis of SLOP (Sharpened Logarithmic Opinion Pool), which generalizes logarithmic opinion pools of generative and reward models through arbitrary, optimizable weights. The authors demonstrate that optimizing these weights on a limited set of calibration data can mitigate reward hacking, preserving or even enhancing alignment and task performance in both VQA and math reasoning benchmarks.

Generalized Formulation of Inference-Time Alignment

Modern alignment methods often optimize (via RL) a policy $\pi$ for increased expected reward, regularized by KL divergence from a reference LLM. This is formalized as maximizing $\mathbb{E}_\pi[r] - \lambda \mathrm{KL}(\pi \| p)$, with the optimal policy being a "tilted" distribution $\pi^* \propto p \exp(r/\lambda)$. This framework unifies RL-fine-tuning and post-hoc inference-time techniques like BoN, as previously explored in recent literature.

The authors generalize this canonical setting by introducing a reference model temperature parameter $\alpha$ and a reward tilting strength $\beta$, leading to a family of "tempered and tilted" distributions:

$$\pi^*_{\alpha, \beta}(y \mid x) \propto p(y \mid x)^\alpha \exp(\beta r(x, y))$$

This formulation not only subsumes standard regularized reward maximization but also enables more complex inference-time alignments and ensemble strategies involving multiple reward/generative models with different contributions.

SLOP: Sharpened Logarithmic Opinion Pools

Formal Definition

The SLOP framework interprets $\pi^*_{\alpha, \beta}$ as a two-expert logarithmic opinion pool. More generally, given $m$ experts (where each expert may be a generative model or a log-proxy reward), the SLOP distribution is

$$\pi^*_\omega(y \mid x) \propto \prod_{i=1}^m p_i(y \mid x)^{\omega_i}$$

where $\omega \in \mathbb{R}^m$ are the expert weights. Unlike traditional opinion pools constrained by $\mathbb{E}_\pi[r] - \lambda \mathrm{KL}(\pi \| p)$0 and $\mathbb{E}_\pi[r] - \lambda \mathrm{KL}(\pi \| p)$1, SLOP employs free weights, enabling sharpening (large, potentially negative or diverging weights) and exploitation of anti-aligned experts.

This view unifies tempered reward maximization, product-of-experts, Mixture-of-Experts, and aligns with recent directions in regularized inference-time RL.

Practical Sampling and Approximate Inference

Direct sampling from SLOP is intractable for large output spaces. The authors leverage and extend SBoN: first, several candidates $\mathbb{E}_\pi[r] - \lambda \mathrm{KL}(\pi \| p)$2 are sampled from a reference model $\mathbb{E}_\pi[r] - \lambda \mathrm{KL}(\pi \| p)$3. Each candidate is scored via a weighted sum over all expert log-likelihoods or proxy scores. Selection is via softmax over the pseudo-rewards. This mechanism is shown to tightly approximate the target SLOP distribution as the number of samples grows.

Figure 1: Accuracy achieved by SLOP in VQA tasks as a function of proxy reward accuracy, demonstrating robust performance across a range of reward model reliabilities.

Calibration for Reward Hacking Mitigation

The principal theoretical and practical contribution is weight calibration for SLOP using a small, verifiable calibration set. Algorithmically, SLOP weights are adjusted via gradient ascent to maximize expected gold reward (or "oracle" reward) over the SLOP-aligned distribution sampled using the above method. The framework incorporates regularization (weight decay) to mitigate overfitting and avoid degenerate solutions where weights diverge.

Empirically, this approach is shown to restrict the model's capacity for reward hacking—a pathology where imperfect proxy rewards misguide the policy—by balancing the influence of the reference and proxy models, or between diverse experts in an ensemble.

Figure 2: Evaluation of SLOP across four LLMs on GSM8K with each LLM as the reference, illustrating performance dynamics under different reference models.

Empirical Evaluation

Visual Question Answering (SQA)

Experiments on ScienceQA (SQA) leverage VLMs (e.g., LLaVA-1.5-7B, Gemma-3-4B, Qwen3-VL-4B) with synthetically degraded proxy reward models to stress-test alignment robustness. Closed-form SLOP sampling is feasible on small multi-choice sets. The results show:

• SLOP with calibrated weights matches or exceeds either the reference model or proxy at all reward accuracies.
• With highly unreliable proxies, SLOP automatically down-weights their contribution, reverting toward the reference model.
• For anti-aligned proxies (reward accuracy < random guessing), SLOP exploits this with negative weights, yielding robust accuracy across the proxy quality spectrum.

  Figure 3: SLOP performance with Gemma-3-4B and variable proxy reward accuracy on SQA, reflecting the adaptability of ensemble weights.

  

  Figure 4: SLOP performance with Qwen3-VL-4B under varying proxy reward accuracy, further validating SLOP’s resilience to degraded proxies.

  

Figure 5: Optimized two-expert SLOP weights as a function of proxy model accuracy, showing adaptive rebalancing between reference and proxy.

Math Reasoning (GSM8K)

A four-expert SLOP is constructed using Gemma-3-1B, Qwen2-1.5B, Qwen3-0.6B, and Phi-3.5-mini. The reward is the correctness on GSM8K. SLOP with calibration consistently outperforms the reference, the best single-expert, and naive BoN across most settings, with especially pronounced gains when the reference is a weaker LLM.

• SLOP sampling scales monotonically with the number of candidates, highlighting the utility of diverse experts.
• Hard selection (max-score candidate) improves further; top accuracy is achieved with SLOP or top BoN depending on the reference.

  Figure 6: SLOP, reference: Gemma-3-1B on GSM8K.

  

  Figure 7: SLOP, reference: Qwen2-1.5B on GSM8K.

  

  Figure 8: SLOP, reference: Qwen3-0.6B on GSM8K.

  

  Figure 9: SLOP, reference: Phi-3.5-mini on GSM8K.

Ablations and Covariance-Based Weighting

The paper also investigates model calibration using just means and covariances of expert scores, motivated by the Gaussian detection perspective. "Inverse covariance" weighting approaches the performance of direct calibration, especially when expert errors are uncorrelated. Ablation results confirm model diversity as critical: strong correlation limits ensemble gains, consistent with theoretical predictions.

Figure 10: SLOP ablations with Gemma-3-1B as reference, comparing fixed and covariance-based weights.

Figure 11: SLOP ablations with Qwen2-1.5B as reference.

Figure 12: SLOP ablations with Qwen3-0.6B as reference.

Figure 13: SLOP ablations with Phi-3.5-mini as reference.

Theoretical Implications

The SLOP framework clarifies several key theoretical points:

• SLOP generalizes log-opinion pools and product-of-experts inference to arbitrary (including negative) weighting, handling anti-aligned or adversarial reward models.
• Weight calibration (with small, high-quality calibration sets) yields robust ensemble alignment, limiting reward hacking and bias amplification.
• Gaussian analysis of LLRs provides practical weight-selection heuristics and motivates the pursuit of diverse, decorrelated experts to maximize ensemble benefits.
• SLOP encompasses and unifies a range of prior practical methods (BoN, SBoN, RAD, HedgeTune) under a tractable sampling-based perspective.

Practical Implications and Limitations

SLOP enables fast, flexible inference-time adaptation to changing reward specifications without retraining or computationally expensive RL. This lowers overhead and democratizes continual alignment. Its hybridization capability lets practitioners mix base models, reward signals, and calibrate for new objectives—potentially even across modalities (language, vision, reasoning). However, the dependence on a high-quality, representative calibration set and the risk of overfitting or misalignment if proxies are adversarial remain open practical challenges. Scaling SLOP to very high-dimensional or fully open-ended generative settings (e.g., long-form text, dialogue) is an area for future exploration.

Conclusion

The paper delivers a systematic extension and synthesis of inference-time alignment and ensembling strategies, culminating in the SLOP formalism. SLOP provides both a conceptual framework and a practical algorithm for mitigating reward hacking and maximizing alignment utility, with demonstrated gains in VQA and math reasoning LLMs. Theoretical results underline the necessity of expert diversity and careful calibration. Future research is likely to focus on more scalable ensemble construction, adversarial and safety-sensitive reward signals, and extensions to embodied or multimodal generative agents.