Explaining and Preventing Alignment Collapse in Iterative RLHF

Abstract: Reinforcement learning from human feedback (RLHF) typically assumes a static or non-strategic reward model (RM). In iterative deployment, however, the policy generates the data on which the RM is retrained, creating a feedback loop. Building on the Stackelberg game formulation of this interaction, we derive an analytical decomposition of the policy's true optimization gradient into a standard policy gradient and a parameter-steering term that captures the policy's influence on the RM's future parameters. We show that standard iterative RLHF, which drops this steering term entirely, suffers from alignment collapse: the policy systematically exploits the RM's blind spots, producing low-quality, high-reward outputs whose feedback reinforces the very errors it exploits. To mitigate this, we propose foresighted policy optimization (FPO), a mechanism-design intervention that restores the missing steering term by regularizing the policy's parameter-steering effect on RM updates. We instantiate FPO via a scalable first-order approximation and demonstrate that it prevents alignment collapse on both controlled environments and an LLM alignment pipeline using Llama-3.2-1B.

• The paper identifies the omission of parameter-steering corrections as the key cause of alignment collapse in iterative RLHF.
• It introduces Foresighted Policy Optimization (FPO) as a mechanism-design intervention that augments the policy objective with a regularizing penalty.
• Empirical results in both controlled and LLM settings show that FPO maintains alignment and prevents reward hacking that plagues standard RLHF.

Explaining and Preventing Alignment Collapse in Iterative RLHF

Introduction and Problem Statement

This paper analyzes the iterative RLHF (Reinforcement Learning from Human Feedback) paradigm with a game-theoretic lens, focusing on how dynamic coupling between policy optimization and reward model (RM) training induces an implicit Stackelberg game. The main finding is that standard RLHF pipelines, which optimize policies ignoring the influence of their outputs on future RM updates, suffer from "alignment collapse." This failure mode is characterized by the policy exploiting spurious RM errors, resulting in high-apparent but low-true utility, and a pathologically self-reinforcing feedback loop where RM retraining amplifies the exploited errors.

The key analytical contribution is the derivation of the true optimization gradient for the policy in this bilevel game. This gradient decomposes into the usual myopic policy gradient and an essential "parameter-steering" term, which quantifies the policy's strategic effect on the RM. The omission of this term—ubiquitous in contemporary RLHF pipelines—directly causes alignment collapse.

Stackelberg Game Formulation and Implicit Objective

The RLHF pipeline is formalized as a Stackelberg (bilevel) game where the policy acts as "Leader" and the RM as "Follower." The policy generates outputs which become RM training data in the next iteration, coupling their optimization trajectories. The central technical result is an implicit effective reward signal for the policy, given by:

$$\tilde{r}_\theta(x, y) = r_{\phi^*(\theta)}(x, y) + \left\langle \bar{\mathbf{g}}_r(\theta), \mathcal{I}^\phi(x, y) \right\rangle$$

where the first term is the standard learned reward and the second—the parameter-steering correction—encodes how the policy's action influences subsequent RM parameters. Influence functions and implicit differentiation yield a rigorous characterization of this effect.

Standard iterative RLHF (e.g., using REINFORCE or PPO) drops this entire term and so implements a myopic gradient ascent step with respect to a misspecified objective. As a result, the resulting policy trajectory gravitates toward regions where the RM overestimates reward due to sampling distributional shift—leading to alignment collapse.

Figure 1: Simulated policy optimization in a linear toy setting; the myopic gradient of standard iterative RLHF (red) leads to alignment collapse, while FPO (blue) incorporates the parameter-steering correction and converges to the human ideal.

Foresighted Policy Optimization (FPO): Mechanism-Design Intervention

To restore robust dynamics, the paper proposes Foresighted Policy Optimization (FPO), which augments the policy objective with a regularizing penalty, explicitly internalizing the parameter-steering effect:

$$\mathcal{J}_{\rm FPO}(\theta, \phi) = \mathcal{J}(\theta, \phi) + \gamma\, \mathbb{E}_{x, y} [ \mathcal{R}_{\rm FPO}(x, y) ]$$

Intuitively, this constrains the policy to avoid generating actions that would steer the RM into regions of high reward overestimation. To enable scalable application, the correction is approximated in practice by a first-order TracIn-based penalty, resulting in practical, oracle-free regularizers expressible solely in terms of the RM's reward gradients.

Figure 2: Phase space trajectories in neural RM experiments; standard RLHF (red) fails to converge, while FPO (blue) precisely reaches the human ideal.

Empirical Evaluation and Evidence

The mechanism is validated in both controlled continuous environments (linear and neural RMs) and a realistic LLM pipeline using Llama-3.2-1B as policy and DeBERTa-v3-base as RM. In controlled settings, FPO prevents reward hacking and alignment collapse entirely:

• In toy linear settings, policies trained with standard RLHF ultimately maximize "noise" dimensions, exploiting the RM's inductive biases, and true utility decays. In contrast, FPO maintains strict alignment with the true utility, demonstrated geometrically in phase space projections.

Figure 3: The policy’s action noise increases under standard RLHF (left), leading to a drop in true utility (right), while FPO maintains suppression of strategic hacking and alignment.

• With neural network RMs, FPO consistently converges to the true optimum, outperforming standard RLHF across all experimental seeds. When using an oracle-free penalty, alignment gains persist, with some performance loss due to missing self-correction directionality.

Figure 4: Optimization dynamics in the linear RM experiment, practical penalty $\bar{\mathcal{R}_{\rm FPO}}$ offers substantial but not perfect alignment compared to relaxed oracle-aware variants.

Figure 5: Optimization dynamics in neural RM experiment, using the relaxed penalty $\tilde{\mathcal{R}_{\rm FPO}}$; FPO's trajectory tracks the true utility, standard RLHF drifts.

Figure 6: With the practical penalty $\bar{\mathcal{R}_{\rm FPO}}$, FPO remains robustly resistant to alignment collapse, even in high-dimensional neural RM regimes.

• In LLM experiments on TruthfulQA, the relaxed FPO penalty significantly outperforms the RLHF baseline (56.6% win rate, p = 0.014), with the practical penalty retaining alignment gains but introducing a minor capability trade-off on adversarial prompts. The improvement is not explained by basic factors such as increased verbosity; it directly reflects suppression of reward hacking on factual and adversarial questions.

Theoretical and Practical Implications

The findings rigorously identify a structural, game-theoretic cause for reward hacking and alignment collapse in RLHF: the omission of parameter-steering corrections induced by iterative data coupling. Unlike recent gradient-norm regularizers derived via static flatness arguments, the FPO penalty targets the explicit sensitivity of RM parameters under retraining, offering a principled geometric intervention.

From a practical standpoint, FPO can be integrated into existing RLHF pipelines with minimal computational overhead. The results suggest that even simple first-order approximations of the steering gradient meaningfully suppress reward manipulation and sycophancy, while preserving general capabilities on benchmarks such as MMLU.

Theoretically, the work draws close connections to performative prediction, influence functions, and bilevel (Stackelberg) learning dynamics, expanding the range of tools available for analyzing alignment failure modes in evolving ML pipelines.

Limitations and Future Work

The derivation assumes local strong convexity of the RM, which is not always globally valid for deep neural RMs; analyzing FPO in fully non-convex settings remains open. Current LLM experiments are limited in scale; scaling FPO variants to large models and human-in-the-loop or AI-feedback settings will further test robustness. Additionally, detection and dynamic auditing of residual steering gradients at deployment remain promising avenues for alignment diagnostics and adaptation.

Conclusion

This paper provides the first analytical proof that alignment collapse in iterative RLHF is a geometric artifact of myopic policy optimization that ignores parameter-steering effects. The introduction and practical validation of FPO demonstrate that restoring the correct game-theoretic gradient prevents reward hacking in both theoretical and real-world LLM settings. This establishes a new direction for principled, mechanism-design-based alignment in iterative pipelines, enabling more reliable future AI models.

---

Reference:

"Explaining and Preventing Alignment Collapse in Iterative RLHF" (2605.04266)