Reward-Enhanced Policy Optimization (REPO)

• REPO is a reinforcement learning framework that leverages a primary dense reward model enhanced by auxiliary judge and programmatic signals.
• It employs a sign-sensitive reward enhancement rule to balance soft constraints like persuasive style with hard constraints such as numerical validity.
• Empirical results demonstrate significant gains in dialogue ratings, excellent-response rates, and robust repair rates in multi-turn hotel price negotiations.

Reward-Enhanced Policy Optimization (REPO) is a reinforcement learning post-training framework for LLMs in which policy optimization is aligned with heterogeneous rewards rather than a single reward source. In the formulation introduced for persuasive online travel agency price negotiation, REPO uses a preference-trained reward model as the primary dense reward and enhances it with an LLM-based reward judge and programmatic reward functions so that persuasive style, Standard Operating Procedure (SOP) compliance, and deterministic business constraints are optimized jointly (Zhuang et al., 5 Oct 2025).

1. Domain, task structure, and motivation

REPO was introduced for proactive multi-turn price negotiation in an online travel agency setting, where a business-development agent interacts with hotel managers to negotiate room prices while balancing traveler affordability, hotel profitability, and partner relationship preservation. The task is not ordinary customer-service dialogue. The model must follow a multi-stage SOP, infer the current negotiation stage from dialogue history, interpret colloquial inputs, reason about room/date/price consistency, remain persuasive and empathetic, and obey guardrails such as no over-promising and no hallucinated business relationships or unsupported offers (Zhuang et al., 5 Oct 2025).

The framework is motivated by the claim that conventional post-training methods fail for complementary reasons. Supervised fine-tuning can imitate annotated scripts but tends to overfit observed dialogue patterns. Direct Preference Optimization improves alignment to preference data but remains bounded by annotation coverage and does not naturally enforce structured business constraints such as numeric validity or SOP progress tracking. Reinforcement learning with a single reward source is described as unstable and susceptible to reward hacking, because the policy may exploit one reward channel while neglecting other task dimensions. The paper therefore treats the negotiation problem as one governed simultaneously by soft constraints, such as persuasion style and emotional value, and hard constraints, such as workflow compliance, formatting, and guardrails (Zhuang et al., 5 Oct 2025).

Within that framing, “heterogeneous rewards” refers to three distinct signal families with different semantics and reliability properties: a preference-trained reward model (RM), an LLM-based reward judge (RJ), and deterministic reward functions (RF). The RM captures dense human preference alignment, the RJ captures high-level rubric-based behavior, and the RF captures exact verifiable constraints. This division of labor is central to REPO’s definition.

2. Reward architecture and the enhancement rule

REPO is best understood as a reward aggregation and reward enhancement strategy layered on standard RL optimization, rather than as a wholly new policy-gradient optimizer. The paper treats the RM as the primary reward and uses RJ and RF as auxiliary “helping scores” that modulate, rather than replace, the RM. The reported enhancement equations are

$$E_{\mathrm{enh}}=\operatorname{clip}(E_{\mathrm{judge}}+E_{\mathrm{func}},-n,n),$$

$$R_{\mathrm{total}}=R_{\mathrm{model}}\left(1+\frac{E_{\mathrm{enh}}}{n}\right),$$

with $n=100$ in the experiments (Zhuang et al., 5 Oct 2025).

This rule is explicitly sign-sensitive. When $R_{\mathrm{model}}>0$, positive $E_{\mathrm{enh}}$ amplifies reward and negative $E_{\mathrm{enh}}$ dampens it. When $R_{\mathrm{model}}<0$, positive $E_{\mathrm{enh}}$ reduces the penalty and negative $E_{\mathrm{enh}}$ amplifies it. If either $R_{\mathrm{model}}=0$ or $$R_{\mathrm{total}}=R_{\mathrm{model}}\left(1+\frac{E_{\mathrm{enh}}}{n}\right),$$0, the total reward remains unchanged. The paper presents this as a stability-preserving alternative to naive multi-reward addition, because auxiliary signals are clipped and only scale the primary dense reward rather than dominating it outright (Zhuang et al., 5 Oct 2025).

Reward source	Primary role	Examples explicitly associated with it
RM	Dense human preference alignment	persuasive style, negotiation tactics
RJ	Rubric-based high-level judgment	SOP compliance, emotional value, progress
RF	Deterministic verifiable checks	numerics, formatting, guardrails, length

The RM is trained on curated pairwise data and is described as teaching “human BD persuasion style and tactics.” The RJ is an LLM-as-a-judge with a task-specific rubric targeting style, emotional value, SOP compliance, and progress. The RF are deterministic checks covering business numerics, formatting, guardrails, and length; the appendix also connects them to chain-of-thought leakage, malformed response format, mixed languages, repeated historical scripts, and overly long outputs. Taken together, these sources operationalize the claim that no single reward source is adequate for realistic enterprise negotiation.

3. Training pipeline and implementation

The training pipeline begins with preference-oriented negotiation data collected from online production, language experts, task experts, and SFT data later enriched into pairwise preferences. The preference dataset is reported as 6,632 samples, built iteratively from 252 production cases, 3,178 samples annotated by language experts, 3,178 samples annotated by task experts, and 1,991 samples originally labeled for SFT and later enriched into preference data. REPO then trains the RM, constructs the RJ rubric, defines the RF checks, generates rollouts from the current policy, computes $$R_{\mathrm{total}}=R_{\mathrm{model}}\left(1+\frac{E_{\mathrm{enh}}}{n}\right),$$1, $$R_{\mathrm{total}}=R_{\mathrm{model}}\left(1+\frac{E_{\mathrm{enh}}}{n}\right),$$2, and $$R_{\mathrm{total}}=R_{\mathrm{model}}\left(1+\frac{E_{\mathrm{enh}}}{n}\right),$$3, aggregates them through the enhancement rule, and uses the resulting $$R_{\mathrm{total}}=R_{\mathrm{model}}\left(1+\frac{E_{\mathrm{enh}}}{n}\right),$$4 with Generalized Advantage Estimation and a value model to produce the policy update signal (Zhuang et al., 5 Oct 2025).

The paper states several concrete implementation choices. The backbone model is Qwen3-32B-Instruct. Maximum response length is 512 tokens, batch size is 128, LoRA rank is $$R_{\mathrm{total}}=R_{\mathrm{model}}\left(1+\frac{E_{\mathrm{enh}}}{n}\right),$$5, LoRA alpha is $$R_{\mathrm{total}}=R_{\mathrm{model}}\left(1+\frac{E_{\mathrm{enh}}}{n}\right),$$6, learning rate is $$R_{\mathrm{total}}=R_{\mathrm{model}}\left(1+\frac{E_{\mathrm{enh}}}{n}\right),$$7, and warmup steps are 2. SFT and DPO are trained for 10 epochs, while PPO, GRPO, and REPO are trained for 2 epochs. For PPO, epochs per update are 10; for GRPO, group size is $$R_{\mathrm{total}}=R_{\mathrm{model}}\left(1+\frac{E_{\mathrm{enh}}}{n}\right),$$8. The paper reports the best-performing checkpoint.

REPO also adopts several cost-saving design choices. It applies LoRA adapters to both policy and value networks, uses no reference model or KL penalty, and computes rewards per trajectory rather than through group computation. The paper explicitly attributes some of this stability to LoRA’s low-rank constraint. A common misconception is therefore that REPO is primarily a new RL backbone. In the paper’s own presentation, its mathematically explicit novelty is the reward enhancement rule, while the downstream optimization retains standard RL machinery.

4. Empirical performance and observed capabilities

Evaluation is split between 30 complete production conversations, about 150 turns, intended to reflect the real distribution of hotel intents, and 45 bad-case conversations, about 225 turns, curated by business experts from deployment failure modes observed with Qwen2.5-32B. Baselines are SFT, DPO, PPO, and GRPO. Online evaluation uses an overall 1–5 dialogue rating and an excellent-response rate, defined as the share of conversations with at least one outstanding negotiation turn. Bad-case evaluation distinguishes solved, solved with minor problem, solved with major problem, unsolved, and unsolved with major problem, from which bad-case fix rate and clean-fix rate are computed (Zhuang et al., 5 Oct 2025).

Metric	REPO	Comparator highlights
Average dialogue rating	4.63	base 3.43, DPO 3.80, GRPO 4.30
Excellent-response rate	66.67%	base 13.33%, DPO 33.33%, GRPO 43.33%
Bad-case fix rate	93.33%	tied with SFT 93.33% and DPO 93.33%; PPO 86.66%, GRPO 71.10%
Clean-fix rate	75.56%	GRPO 44.44%, DPO 40.00%, PPO 33.33%, SFT 31.11%

The online rating of 4.63 corresponds to gains of $$R_{\mathrm{total}}=R_{\mathrm{model}}\left(1+\frac{E_{\mathrm{enh}}}{n}\right),$$9 over base, $n=100$0 over DPO, and $n=100$1 over GRPO. The excellent-response rate of 66.67% is $n=100$2 percentage points over base, $n=100$3 over DPO, and $n=100$4 over GRPO. On the bad-case set, REPO ties SFT and DPO on raw fix coverage at 93.33% but is reported as decisively better on quality of repair: clean-fix rate is 75.56%, and “fixed but with major problems” falls to 4.44%, compared with 13.33% for GRPO, 31.11% for DPO, 31.11% for PPO, and 42.22% for SFT (Zhuang et al., 5 Oct 2025).

The paper also reports a GPT-based fine-grained skill evaluation over dialogue fluency, workflow compliance, negotiation effectiveness, and scope understanding. On online samples, REPO’s main advantage is negotiation effectiveness. On the bad-case set, it remains strongest on negotiation effectiveness, leading the next best method by 16 percentage points. The text further claims emergent capabilities—proactive empathy, localized reasoning, and calibrated tactics—that surpass gold annotations. Training-curve analysis is described as showing an early unstable phase, a mid-training “learning” phase in which tactics emerge, and a convergence phase; late checkpoints reportedly outperform early ones by roughly $n=100$5 points on the persuasion score, about 30% (Zhuang et al., 5 Oct 2025).

5. Position within the broader policy-optimization literature

REPO belongs to a wider post-training trend in which policy optimization quality depends not only on the backbone optimizer but also on reward construction, normalization, filtering, and robustness. In scalar-feedback alignment, “Value-Free Policy Optimization via Reward Partitioning” introduces Reward Partitioning Optimization (RPO), which replaces value-function learning with prompt-wise empirical reward normalization over $n=100$6 data and directly regresses policy log-ratios to centered reward targets (Faye et al., 16 Jun 2025). In process-supervised reasoning, “PRPO: Aligning Process Reward with Outcome Reward in Policy Optimization” constructs tokenwise advantages

$n=100$7

thereby aligning dense process rewards with sparse outcome rewards in a critic-free framework (Ding et al., 12 Jan 2026).

The multi-reward case yields additional design problems. “GD$n=100$8PO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimization” filters rollouts with severe reward-wise disagreement and reweights queries by retained consensus, rather than simply summing reward dimensions (Liu et al., 15 Jun 2026). “Multi-Objective and Mixed-Reward Reinforcement Learning via Reward-Decorrelated Policy Optimization” processes heterogeneous rewards with Magnitude-Aware Quantile normalization and Mahalanobis whitening before scalarization, targeting both scale mismatch and correlation redundancy (Bai et al., 13 May 2026). “Reward Dithering for Improved LLM Policy Optimization” modifies discrete rewards by zero-mean perturbations to increase reward variance and smooth optimization under GRPO-style training (Wei et al., 23 Jun 2025). “Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimation” instead emphasizes pessimistic optimization under reward uncertainty, replacing point-estimate reward maximization with a distributionally robust objective (Zhang et al., 2024).

This suggests that REPO is one member of a broader family of reward-centric policy optimization methods. Its distinguishing choice is not value-free normalization, process-level credit assignment, multi-reward decorrelation, or uncertainty-aware pessimism, but an RM-anchored multiplicative enhancement rule in which judge and programmatic signals modulate the dense preference reward. That places it squarely in the literature on heterogeneous reward composition for RL alignment.

6. Terminological ambiguity, scope, and limitations

The acronym is not standardized across recent RL and LLM post-training work. “RePO: Replay-Enhanced Policy Optimization” denotes a replay-buffer extension of GRPO that augments on-policy groups with off-policy samples for the same prompt; its “enhanced” component is replay, not heterogeneous reward shaping (Li et al., 11 Jun 2025). “Reference-guided Policy Optimization for Molecular Optimization via LLM Reasoning” uses the same acronym for a GRPO-based RLVR method with a masked answer-level imitation term conditioned on sampled reasoning trajectories (Li et al., 6 Mar 2026). “$n=100$9REPO: implicit Reward Pairwise Difference based Empirical Preference Optimization” uses yet another related acronym for a preference-optimization method based on implicit reward pairwise difference regression (Le et al., 2024). A plausible implication is that “REPO” should be read contextually rather than assumed to denote a unique canonical algorithm family.

The specific REPO framework for OTA negotiation also has a narrow empirical scope. The paper evaluates only one task, persuasive hotel-price negotiation; judge prompts are desensitized for confidentiality; no full numerical ablation table isolates RM-only, RM+RJ, or RM+RF variants; and explicit optimization equations beyond the reward enhancement rule are not provided in the same detail as the reward-design discussion. The main evaluation is expert-curated but relatively small in absolute size—30 online conversations and 45 bad-case conversations. The paper itself notes that bad-case persuasion ceilings are inherently lower because those cases are deliberately adversarial and relationship-sensitive (Zhuang et al., 5 Oct 2025).

Within those limits, REPO’s encyclopedic significance lies in formalizing a particular answer to a recurrent alignment problem: when task success depends simultaneously on nuanced human preference, rubric-based strategic judgment, and exact verifiable constraints, policy optimization can be organized around a primary dense reward whose sign-aware magnitude is modulated by clipped auxiliary reward channels.