REFUEL: Regressing the Relative Future in RLHF

• Regressing the Relative Future (REFUEL) defines a multi-turn RLHF framework that models dialogues as Markov Decision Processes to address covariate shift.
• It employs on-policy data generation and regression over relative Q-differences to stabilize policy updates without relying on an explicit critic.
• The algorithm integrates a KL-constrained soft policy update method, achieving state-of-the-art results in extended dialogue and multi-turn tasks.

Regressing the Relative Future (REFUEL) is an on-policy, regression-based policy optimization algorithm designed for multi-turn Reinforcement Learning from Human Feedback (RLHF) in LLMs. REFUEL addresses fundamental challenges in long-horizon LLM tasks, where agent responses influence not only immediate reward but also future conversation dynamics, and where naïve extensions of single-turn RLHF induce severe covariate shift between training and deployment distributions. REFUEL incorporates a theoretical framework, efficient empirical protocol, and practical implementation with state-of-the-art results in multi-turn dialogue domains (Gao et al., 2024).

1. Multi-turn RLHF as an MDP Framework

In conventional RLHF for single-turn tasks, each model response is treated as a contextual bandit problem: given context $x$ (e.g., a prompt or dialogue history), the model produces a reply $y$, receives a scalar reward, and updates policy parameters. However, for genuine multi-turn tasks such as extended dialogue, the assistant’s response at turn $h$ conditions all subsequent user turns and assistant replies up to a horizon $H$. The conversation is thus modeled as an $H$-step Markov Decision Process (MDP):

• State at turn $h$: $s_h = (x_1, y_1, ..., x_{h-1}, y_{h-1}, x_h)$
• Policy: $\pi(y_h | s_h)$ generates the assistant’s $h$th reply
• User transition: $x_{h+1} \sim T(\cdot | s_h, y_h)$ is stochastic
• Reward: Only the full trajectory receives a scalar reward $r(s_{H+1})$ at turn $H$

Naïve "last-turn RLHF" treats the full dialogue as a context and applies single-turn algorithms (e.g., DPO, REBEL) to optimize only the final reply. This approach introduces a covariate shift: training uses contexts drawn from an offline reference policy but, at test time, the model samples its own outputs into its future contexts, leading to compounding errors as described by Ross & Bagnell (2011) in imitation learning.

2. Covariate Shift and Prior Methodological Limitations

A core obstacle in multi-turn RLHF is the train/test mismatch due to covariate shift. Traditional methods—for example, treating the previous dialogue as a static context—lead to models that perform poorly when deployed in-the-loop, since the state distribution drifts as the agent interacts with itself. Baseline single-turn offline methods (such as DPO and REBEL variants) and even last-turn on-policy approaches fail to maintain performance over multiple dialogue turns. Empirical results show a collapse in winrate for these approaches as the conversation progresses, confirming the severity of covariate shift in this regime.

3. Algorithmic Principles of REFUEL

REFUEL is structured as an iterative, on-policy regression method with the following key mechanisms:

1. On-policy Data Generation: All dialogue prefixes for training are generated by rolling in with the current policy $\pi_t$. This ensures that training data matches the agent’s own deployment distribution, directly mitigating covariate shift.
2. Regression over Relative Q-differences: There is no explicit critic. Instead, the policy itself is used as an implicit critic via a reparameterization trick: the algorithm regresses the difference of log-policy ratios against observed differences in trajectory-level rewards.
3. KL-constrained Soft Policy Update: The optimal soft update under a KL constraint has a closed-form:

$$\pi_{t+1}(y|s) \propto \pi_t(y|s) \exp(\eta Q^{\pi_t}(s, y))$$

The Q-value differences (relative Qs) can be recovered as:

$$Q^{\pi_t}(s, y) - Q^{\pi_t}(s, y') = \frac{1}{\eta} \left[ \ln\frac{\pi_{t+1}(y|s)}{\pi_t(y|s)} - \ln\frac{\pi_{t+1}(y'|s)}{\pi_t(y'|s)} \right]$$

Critically, the additive baseline cancels in differences, and $Q$-value differences can be unbiasedly estimated with reward differences from full rollouts.

1. Optimization Objective: For sampled $(s_h, y, y', \Delta r)$, the algorithm minimizes:

$$L(\pi) = \mathbb{E}_D \left[ \left( \frac{1}{\eta}\left[ \ln\frac{\pi(y|s_h)}{\pi_t(y|s_h)} - \ln\frac{\pi(y'|s_h)}{\pi_t(y'|s_h)} \right] - \Delta r \right)^2 \right]$$

2. Iteration Protocol: The learning process proceeds by alternating data collection with policy optimization over this regression objective, starting from a pretrained supervised policy.

REFUEL Iteration Summary

Step	Operation	Purpose
Roll-in (on-policy)	Simulate prefixes $s_h$ under $\pi_t$	Avoids covariate shift
Dual continuation	Sample two replies $y, y'$ under $\pi_t$	Enables Q-difference estimation
Rollout	Complete trajectories to observe $r$	Get unbiased reward differences
Regression	Minimize $L(\pi)$ over collected tuples	Policy improvement (no critic