RLVP: Penalize the Path, Reward the Outcome

Abstract: Agents acting on our behalf in the real world (e.g. placing phone calls) must learn online from costly, often irreversible interactions rather than cheap simulator steps. Two things follow. First, deployability depends on the path, not only the outcome. An agent must respect outcome-neutral constraints such as not repeatedly calling an unresponsive user, respecting business hours, or completing required authentication constraints that outcome-based rewards cannot express, since violating them frequently improves apparent success. Second, because each interaction is expensive, the agent must learn efficiently from very few examples. Reinforcement learning from verifiable rewards (RLVR) is blind to both challenges: it optimizes solely on the outcome and wastes expensive rollouts on all-fail groups where group-relative advantage collapses to zero. Attempts to densify supervision by rewarding progress target the hard-to-verify direction. In contrast, real agentic environments can cheaply detect bad moves. Since group-relative advantage is equivalent to within-group variance, a dense signal helps only when it supplies variance the outcome lacks. A verifiable penalty on the path meets this condition reliably, while a progress potential helps only where partial progress is reachable. The resulting recipe "penalize the path, reward the outcome" achieves high task success with near-zero violations, where outcome-only training violates constraints on nearly every episode. We provide four design rules for effective penalties, including avoidance of the inaction trap that arises when a penalty is used in isolation.

• The paper introduces RLVP, which combines outcome rewards with separately normalized penalties for verifiable bad actions and credits for fulfilled obligations, reducing TerminalBench violations from 3.71 to 0.66 per episode while preserving statistically equal success.
• The method restores learning signal in all-fail groups where GRPO-style advantages collapse, cutting dead updates from 65% to 8% on chained file-operation tasks and accelerating miniF2F training when verifiable progress is reachable.
• The results show that penalties must target commission rather than omission, remain paired with outcome rewards and fulfillment credits, and be seeded with demonstrations, while generic or unreachable signals can cause inaction, reward gaming, or no benefit.

Overview

"RLVP: Penalize the Path, Reward the Outcome" (2607.07435) addresses two structural deficiencies of reinforcement learning from verifiable rewards (RLVR) when applied to agents that learn online from costly, irreversible real-world interactions. First, RLVR optimizes only terminal outcomes and is therefore blind to outcome-neutral path constraints—such as not repeatedly calling an unresponsive user or bypassing required authentication—whose violation often improves apparent success. Second, group-relative methods such as GRPO waste expensive rollouts on all-fail groups, where within-group advantage variance collapses to zero and no gradient is produced. The paper's central claim is that real agentic environments are asymmetric verifiers: they can cheaply and reliably detect bad moves but cannot certify meaningful progress. Consequently, the reliable dense signal is a verifiable penalty on the path rather than a reward for progress, and a verifiable progress potential helps only where partial progress is reachable.

The unifying theoretical device is an observation about group-relative advantage: since the baseline is the group mean, advantage is equivalent to within-group variance. For a shaped reward $R = O + \beta\Phi$, the group variance decomposes as

$$\operatorname{Var}_G(R) = \operatorname{Var}_G(O) + \beta^2\operatorname{Var}_G(\Phi) + 2\beta\,\operatorname{Cov}_G(O,\Phi),$$

so on all-fail or all-success groups ($\operatorname{Var}_G(O)=0$) all usable gradient must come from the process term's own within-group variance. A dense signal is useful exactly when it supplies reachable variance the outcome lacks. A verifiable penalty satisfies this by construction—bad actions are always detectable and typically differ across rollouts—while a progress potential is reachability-gated.

Penalizing the path for deployability

The method keeps the outcome reward and adds a second, per-action channel driven by a deterministic rule engine: a pure predicate over pre-action state and action that attaches a penalty $-\lambda$ to verified bad actions (e.g., destructive commands, calls before preconditions) and a credit $+\beta$ to fulfillment of pending obligations. Channels are normalized separately so the sparse path signal is not diluted by the outcome reward. Crucially, the penalized constraints are outcome-neutral—the type of signal no outcome-based reward can supply.

Empirically, on system-administration and customer-service proxies, outcome-only training solves tasks but violates rules on nearly every episode; adding the penalty channel drives violation rates to near zero at preserved task success across five seeds and model scales from 1.7B to 8B. On TerminalBench with Qwen3-4B, where task success is near the floor, the verifiable harm penalty reduces violations per episode from $3.71 \pm 0.52$ to $\mathbf{0.66 \pm 0.63}$—roughly sixfold—at statistically equal success, while productive actions per episode rise from ~4 to ~13. This is a strong result, though the authors concede it was obtained in a near-floor success regime; validation at higher capability remains open.

Four design rules govern effective penalties:

1. Penalize commission, never omission — target concrete machine-checkable bad actions, not absence of progress.
2. Never optimize a penalty in isolation — a pure penalty collapses to zero success on every seed (the inaction trap); the outcome reward must remain the task driver.
3. Pair each penalty with a fulfillment credit — ablations show removal slows learning and destabilizes compliance acquisition.
4. Ensure reachability and un-gameability — scripted compliant demonstrations seed exploration; learned judges of "compliance" merely relocate the gaming problem into the judge.

Ablations confirm that only the full recipe—penalty plus fulfillment credit plus seeding plus annealing—consistently attains high success with near-zero violations and low seed variance.

Rewarding verified progress where reachable

The same $+\beta$ credit, paid for verifiable progress (falling proof obligations, rising test-pass fraction, satisfied preconditions), becomes a dense potential addressing sample efficiency. Its benefit is strictly gated by reachability, which can be diagnosed before training by measuring $\operatorname{Var}_G(\Phi)$ on base-policy rollouts. On SWE-bench software repair this diagnostic is vacuous: two-thirds of instances have a single failing test, and across 156 rollouts of a 30B policy every episode scores $\Phi=0$, yielding zero gradient. Where the potential is reachable, it eliminates dead all-fail updates (0 vs. ~16 of 40 iterations per seed) and converts them into useful learning signal.

On miniF2F algebra theorem proving, a matched five-seed matrix at 4B and 30B scales shows the aligned potential crossing the 0.9 success threshold in $$\operatorname{Var}_G(R) = \operatorname{Var}_G(O) + \beta^2\operatorname{Var}_G(\Phi) + 2\beta\,\operatorname{Cov}_G(O,\Phi),$$0 iterations versus $$\operatorname{Var}_G(R) = \operatorname{Var}_G(O) + \beta^2\operatorname{Var}_G(\Phi) + 2\beta\,\operatorname{Cov}_G(O,\Phi),$$1 for outcome-only at 4B, with zero divergence aborts across both scales. At 30B, outcome-only faces a stark trade-off—diverging on three of five seeds under Muon, or training roughly 3.6× slower ($$\operatorname{Var}_G(R) = \operatorname{Var}_G(O) + \beta^2\operatorname{Var}_G(\Phi) + 2\beta\,\operatorname{Cov}_G(O,\Phi),$$2 vs. $$\operatorname{Var}_G(R) = \operatorname{Var}_G(O) + \beta^2\operatorname{Var}_G(\Phi) + 2\beta\,\operatorname{Cov}_G(O,\Phi),$$3 iterations) under AdamW—while the aligned potential is both fast and stable everywhere. Notably, the authors report these gains are modest and seed-consistent, explicitly stating that dramatic single-run speed-ups observed in their own pilots disappeared under re-seeding—a candid contrast with speed-up claims common in the literature. Annealing the potential, often considered mandatory, proves unnecessary because the potential is outcome-instrumental; generic structural proxies, by contrast, impose artificial performance ceilings once over-optimized.

Appendix results reinforce the account. On chained file-operation tasks, outcome-only GRPO is dead on 65% of iterations and DAPO on 54% at a 5.6× generation tax, while the aligned potential cuts dead updates to 8% at no extra sampling. On a silent-precondition gate task, every reward-only method fails to zero because the pivotal action is never sampled; a single synthesized demonstration through the process channel breaks the wall to perfect held-out success—an explicit demonstration that imitation must seed what reward then grows. On SWE-smith bug-fixing at 8B (essentially 0% solve rate), the process channel drives four-to-tenfold increases in productive actions and test-runs where outcome-only trajectories remain flat, though the authors honestly note part of this rise is by construction and that improved discipline did not convert to task success within budget.

Boundaries and limitations

The paper is unusually explicit about where its machinery fails. On τ-bench airline customer service, where difficulty turns on semantic policy adherence (intent) rather than procedural ordering, no verifiable rule is finer than the outcome: generic structural rules actively harm and collapse into the inaction trap, policy-derived procedural and verifiable semantic rules remove the harm but never beat outcome-only on average. A same-model self-critic fares poorly as a training reward in every tested quadrant—even with perfect recall against the rule oracle, its ~6% false-positive rate renders it inert relative to a deterministic rule—though it serves as a useful offline intent detector (F1 of 0.63 vs. 0.23 for semantic rules). An un-gameability sweep on Lean confirms the survival pattern: penalty-free signals survive on every seed, a pure penalty reliably collapses, and penalty-plus-fulfillment is bimodal with the largest variance of any configuration. Additional concessions include manually identified penalties (automated constraint discovery remains open), high sweep variance at larger scale (only qualitative patterns reported), and the mechanistic-rather-than-numerical nature of the claimed transfer to live deployment, since controlled multi-seed experimentation is impractical on real phone traffic.

Conclusion

This paper's contribution is less a new optimizer than a reallocation of a scarce resource—the verifier. Group-relative agentic RL is blind wherever within-group outcome variance vanishes; a verifiable penalty supplies the missing variance reliably and teaches outcome-neutral constraints essential for deployability, provided it accompanies (never replaces) the outcome reward, is paired with fulfillment credits, and targets reachable, un-gameable actions. A verifiable potential accelerates learning precisely where partial progress is reachable and is otherwise vacuous. The open questions the work leaves are concrete: validating harm reduction at high task success rates, automating discovery of penalizable constraints from environment affordances, and demonstrating the sample-efficiency gains directly on deployed agents learning from live interactions.