Reward-Variance Policy Optimization (RVPO)

• Reward-Variance Policy Optimization (RVPO) comprises a suite of techniques that manipulate reward variance to balance expected returns and risk.
• It leverages methods such as risk-sensitive objectives, augmented MDP formulations, and reward dithering to address volatility and constraint adherence.
• RVPO adjusts the reward signal—either penalizing or amplifying variance—to improve policy stability and performance across varied reinforcement learning settings.

Searching arXiv for the cited RVPO-related papers and terminology to ground the article in current literature. Reward-Variance Policy Optimization (RVPO) is a term that has acquired multiple, partly overlapping meanings in reinforcement learning and RLHF. In one line of work, RVPO denotes a risk-averse objective that maximizes expected reward while penalizing the variance of rewards under a policy-induced occupancy measure, yielding formulations such as reward-volatility reduction and mean–variance control (Bisi et al., 2019). In more recent RLHF literature, the same label refers to methods that use reward variance as an optimization resource: either by explicitly penalizing inter-reward variance across multiple alignment objectives via a SoftMin aggregator to prevent “constraint neglect” (Montero et al., 7 May 2026), or by increasing within-group reward variance to accelerate GRPO-style training while preserving reward expectation and relative preferences (Yang et al., 29 May 2025). A related estimator-level perspective appears in reward dithering, where zero-mean noise is added to discrete rewards so that variance is injected without changing the expected reward, thereby mitigating gradient collapse in group-relative policy optimization (Wei et al., 23 Jun 2025). The term therefore names a family of variance-aware policy optimization ideas rather than a single universally standardized algorithm.

1. Terminological scope and historical usage

The literature uses “Reward-Variance Policy Optimization” in at least three distinct senses. The earliest explicit formulation in the provided corpus is risk-averse control based on the variance of step-wise rewards under the discounted state-occupancy measure, instantiated as VOLA-PG and TRVO (Bisi et al., 2019). In that setting, RVPO optimizes

$J_\lambda(\pi) = J_\pi - \lambda \nu_\pi^2,$

where $J_\pi$ is the normalized expected discounted return and $\nu_\pi^2$ is reward volatility, defined as the variance of one-step rewards under the discounted occupancy measure (Bisi et al., 2019).

A second usage appears in mean–variance policy iteration. Although the paper does not use the phrase RVPO explicitly, it directly optimizes a mean-minus-variance objective over the per-step reward random variable and presents an augmented-MDP construction that makes risk-averse control compatible with standard risk-neutral algorithms (Zhang et al., 2020). The paper states that this is a concrete instantiation of RVPO in practice (Zhang et al., 2020).

A third, more recent usage emerges in RLHF and critic-less LLM post-training. Here, RVPO no longer means classical risk aversion over temporal reward volatility. Instead, it refers to variance-aware manipulation or aggregation of reward signals to improve optimization. One instance is GRPO with reward variance increase, which adjusts rewards to provably increase within-group variance while preserving expected reward and relative preferences (Yang et al., 29 May 2025). Another is the 2026 “RVPO: Risk-Sensitive Alignment via Variance Regularization,” where multi-objective rewards are aggregated with a SoftMin that penalizes inter-reward variance across concurrent objectives (Montero et al., 7 May 2026). ReDit is explicitly described as tightly connected to the idea of RVPO because it injects reward variance at the estimator level without changing the expected reward (Wei et al., 23 Jun 2025).

This terminological plurality is important. By contrast, “Ratio-Variance Regularized Policy Optimization” concerns the variance of the policy ratio, not the variance of rewards, and the paper explicitly states that the term “Reward-Variance Policy Optimization” does not appear there (Luo et al., 26 May 2026). That distinction prevents conflating trust-region regularization in ratio space with reward-variance manipulation.

2. Classical RVPO as risk-averse reward-volatility optimization

In the risk-averse formulation, RVPO is defined on an infinite-horizon discounted MDP with discounted state-occupancy measure $d_{\mu,\pi}(s,a)$ (Bisi et al., 2019). The normalized expected discounted return is

$$J_\pi = \mathbb{E}_{(s,a)\sim d_{\mu,\pi}}[r(s,a)],$$

and reward volatility is

$$\nu_\pi^2 = \mathrm{Var}_{(s,a)\sim d_{\mu,\pi}}[r(s,a)] = \mathbb{E}[r(s,a)^2] - J_\pi^2.$$

The resulting RVPO objective is

$\eta_\pi = J_\pi - \lambda \nu_\pi^2,$

with $\lambda \ge 0$ controlling the return–volatility trade-off (Bisi et al., 2019).

A central theoretical result is that reward volatility upper-bounds return variance:

$\sigma_\pi^2 \le \nu_\pi^2/(1-\gamma)^2,$

where $\sigma_\pi^2$ is the variance of the discounted return (Bisi et al., 2019). This makes volatility reduction a tractable surrogate for return-risk reduction. The paper emphasizes that volatility is defined over single-step rewards under the occupancy distribution, which leads to a linear Bellman equation for the volatility value function

$J_\pi$0

and enables a policy gradient theorem analogous to the standard one (Bisi et al., 2019).

The corresponding gradient takes the form

$J_\pi$1

or equivalently with a baseline,

$J_\pi$2

where $J_\pi$3 (Bisi et al., 2019). The paper instantiates this objective in both an actor-only Monte Carlo algorithm, VOLA-PG, and a trust-region method, TRVO, with a TRPO-style monotonic-improvement bound adapted to the risk-averse objective (Bisi et al., 2019).

The practical significance of this version of RVPO is that it is a genuine objective-level reformulation: the policy is optimized toward lower reward volatility, not merely trained with a variance-shaped estimator. This differs materially from later RLHF variants that preserve the population objective while changing the optimization signal.

3. Mean–variance policy iteration and augmented-MDP formulations

Mean–Variance Policy Iteration (MVPI) recasts variance-aware control through an augmented reward derived from the per-step reward random variable under the policy’s discounted occupancy distribution (Zhang et al., 2020). The objective is

$J_\pi$4

where $J_\pi$5 is the per-step reward random variable induced by the occupancy measure (Zhang et al., 2020).

The difficulty is the $J_\pi$6 term, which makes the reward depend on the policy. MVPI removes this dependence using Fenchel duality:

$J_\pi$7

For fixed $J_\pi$8, one obtains an augmented MDP with reward

$J_\pi$9

which no longer depends on $\nu_\pi^2$0 (Zhang et al., 2020). This yields a block cyclic coordinate ascent procedure: first estimate $\nu_\pi^2$1, then optimize the risk-neutral control problem on the augmented MDP $\nu_\pi^2$2 (Zhang et al., 2020).

The practical implication is that any risk-neutral control method can serve as the policy-improvement step. The paper explicitly positions this as a plug-and-play bridge from risk-neutral to risk-averse control, including both on-policy and off-policy settings (Zhang et al., 2020). Its exemplar is risk-averse TD3, which replaces the original reward with

$\nu_\pi^2$3

and then uses standard TD3 critics and actor updates on that modified target (Zhang et al., 2020).

This formulation broadens the RVPO family beyond actor-only policy gradients. It suggests that, in the classical risk-averse sense, RVPO is best understood as an objective template—mean reward minus a variance term—rather than as a single optimizer. The choice of optimizer can be inherited from risk-neutral RL once the augmented reward is defined.

4. RVPO in RLHF: reward variance as an optimization resource

Recent RLHF work shifts the emphasis from risk aversion to optimization geometry. “What Makes a Reward Model a Good Teacher? An Optimization Perspective” proves that low reward variance can make the RLHF objective flat even when the reward model is perfectly accurate (Razin et al., 19 Mar 2025). For autoregressive softmax policies, the gradient norm is upper-bounded by a quantity proportional to the cube root of the KL-regularized reward variance, and escape-time lower bounds scale inversely with initial on-policy reward variance (Razin et al., 19 Mar 2025). The paper further shows that a less accurate reward model can train faster than a perfectly accurate one if it induces higher on-policy reward variance (Razin et al., 19 Mar 2025).

This optimization perspective directly motivates RVPO-style mechanisms in RLHF. “Accelerating RLHF Training with Reward Variance Increase” proposes a reward adjustment model that increases within-group reward variance while preserving expected reward and relative preferences (Yang et al., 29 May 2025). For a GRPO group with rewards $\nu_\pi^2$4 and initial-policy probabilities $\nu_\pi^2$5, the adjusted rewards $\nu_\pi^2$6 are obtained by solving

$\nu_\pi^2$7

subject to bounds, expectation preservation, and monotonicity constraints that preserve relative preferences (Yang et al., 29 May 2025). Because the mean is fixed, maximizing $\nu_\pi^2$8 is equivalent to maximizing variance (Yang et al., 29 May 2025).

The paper proves that any maximizer lies at an extreme point of the feasible polyhedron and that extreme points have at most three distinct plateau values: a top block at $\nu_\pi^2$9, a middle plateau at $d_{\mu,\pi}(s,a)$0, and a bottom block at $d_{\mu,\pi}(s,a)$1 (Yang et al., 29 May 2025). It then gives an $d_{\mu,\pi}(s,a)$2 algorithm for the globally optimal adjustment after sorting by rewards (Yang et al., 29 May 2025). When integrated into GRPO as GRPOVI, the adjusted rewards replace the original group rewards before standardization, strengthening the contrastive advantage signal (Yang et al., 29 May 2025).

A closely related estimator-level mechanism is ReDit. In GRPO-style training with discrete rewards, flat mini-batches cause near-zero group-relative advantages, and reward discontinuities can create occasional gradient explosions (Wei et al., 23 Jun 2025). ReDit perturbs each reward as

$d_{\mu,\pi}(s,a)$3

with $d_{\mu,\pi}(s,a)$4 and either Gaussian or Uniform noise (Wei et al., 23 Jun 2025). The paper states that this preserves the expected gradient direction for the original objective at the REINFORCE level while increasing gradient variance by an explicit controllable term (Wei et al., 23 Jun 2025). Within GRPO, the perturbed rewards are standardized to form

$d_{\mu,\pi}(s,a)$5

which prevents advantage collapse when the base rewards are identical (Wei et al., 23 Jun 2025).

These RLHF formulations share a common principle: variance can be deliberately engineered to improve optimization. However, they differ in where the intervention occurs. GRPOVI alters rewards through a constrained optimization problem (Yang et al., 29 May 2025), whereas ReDit alters the estimator statistics via zero-mean perturbations and explicitly does not change the expected reward (Wei et al., 23 Jun 2025). A plausible implication is that “RVPO” in RLHF increasingly denotes variance-aware signal design rather than classical risk-averse objective design.

5. Multi-objective RVPO and variance regularization across reward channels

The 2026 paper “RVPO: Risk-Sensitive Alignment via Variance Regularization” introduces a distinct but formally precise use of RVPO for critic-less multi-objective RLHF (Montero et al., 7 May 2026). The motivating problem is “constraint neglect”: when $d_{\mu,\pi}(s,a)$6 concurrent reward signals are aggregated by arithmetic mean, high performance on easy objectives can numerically offset failures on strict bottleneck constraints such as safety, structure, or completeness (Montero et al., 7 May 2026).

RVPO replaces arithmetic averaging with a SoftMin, applied to per-objective standardized scores:

$d_{\mu,\pi}(s,a)$7

and then

$d_{\mu,\pi}(s,a)$8

This score is whitened across the batch and inserted into an otherwise standard PPO-style clipped objective with KL regularization (Montero et al., 7 May 2026).

The paper’s key analytic result is the cumulant expansion

$d_{\mu,\pi}(s,a)$9

showing that the SoftMin acts as a smooth variance penalty on inter-objective disagreement (Montero et al., 7 May 2026). As $$J_\pi = \mathbb{E}_{(s,a)\sim d_{\mu,\pi}}[r(s,a)],$$0, the aggregator approaches the mean; as $$J_\pi = \mathbb{E}_{(s,a)\sim d_{\mu,\pi}}[r(s,a)],$$1, it approaches the hard minimum (Montero et al., 7 May 2026). The gradient weights are larger for worse-performing objectives, so bottlenecks are emphasized rather than averaged away (Montero et al., 7 May 2026).

This sense of RVPO is neither classical mean–variance control over time nor within-group variance injection for faster RLHF. It is a risk-sensitive aggregator across reward channels. The variance being controlled is inter-objective variance at a fixed sample, not temporal reward volatility and not policy-ratio variance (Montero et al., 7 May 2026). That distinction matters conceptually. Here, the method explicitly changes the scalar training objective from “maximize sum” to “maximize consistency” across objectives (Montero et al., 7 May 2026).

Empirically, the paper reports that RVPO improves HealthBench scores over GDPO at 14B scale, reaching 0.261 versus 0.215 with $$J_\pi = \mathbb{E}_{(s,a)\sim d_{\mu,\pi}}[r(s,a)],$$2, and avoids the late-stage degradation observed in other multi-reward methods (Montero et al., 7 May 2026). It also maintains competitive accuracy on GPQA-Diamond and enforces tool-calling format constraints earlier and more stably while retaining execution correctness (Montero et al., 7 May 2026). These findings suggest that, in multi-objective alignment, variance regularization is not merely an optimization-speed device but a mechanism for preserving hard constraints that mean aggregation obscures.

6. Algorithms, guarantees, and empirical patterns

The major RVPO-related variants differ in their objective, the location of the variance term, and the type of guarantee they provide.

Variant	Variance object	Main mechanism
Classical RVPO / TRVO	Step-wise reward variance under occupancy	Optimize $$J_\pi = \mathbb{E}_{(s,a)\sim d_{\mu,\pi}}[r(s,a)],$$3 (Bisi et al., 2019)
MVPI	Per-step reward variance	Augmented reward via Fenchel duality (Zhang et al., 2020)
GRPOVI-style RVPO	Within-group reward variance	Constrained reward adjustment preserving mean and order (Yang et al., 29 May 2025)
ReDit connection to RVPO	Estimator-visible reward variance	Zero-mean dithering of discrete rewards (Wei et al., 23 Jun 2025)
Multi-objective RVPO	Inter-objective variance across reward channels	SoftMin aggregation of per-objective Z-scores (Montero et al., 7 May 2026)

In the classical risk-averse setting, the guarantee is structural: reward volatility bounds return variance, the volatility Bellman equation is linear, and TRVO inherits a TRPO-style monotonic-improvement bound under a KL trust region (Bisi et al., 2019). In MVPI, monotonic policy improvement follows from alternating optimization over the auxiliary variable $$J_\pi = \mathbb{E}_{(s,a)\sim d_{\mu,\pi}}[r(s,a)],$$4 and the policy on the augmented MDP, with convergence to stationary points under boundedness and smoothness assumptions (Zhang et al., 2020).

In RLHF variance-increase methods, the guarantees are optimization-centric. Theoretical analysis in the reward-model paper states that insufficient on-policy reward variance implies flat landscapes and slow improvement, with explicit escape-time lower bounds (Razin et al., 19 Mar 2025). GRPOVI proves that the reward adjustment preserves expected reward and relative preferences while increasing variance over the response space under the initial policy (Yang et al., 29 May 2025). ReDit proves unbiasedness at the REINFORCE level and gives an explicit variance formula for the added noise term, such as

$$J_\pi = \mathbb{E}_{(s,a)\sim d_{\mu,\pi}}[r(s,a)],$$5

for Gaussian noise (Wei et al., 23 Jun 2025).

The empirical patterns are correspondingly different. Classical RVPO and MVPI are validated in finance and MuJoCo-style continuous control, where lower volatility or better risk-aware metrics are the target (Bisi et al., 2019, Zhang et al., 2020). GRPOVI and ReDit are validated in LLM post-training, where the relevant outcomes are convergence rate, sample efficiency, and stability of group-relative optimization (Yang et al., 29 May 2025, Wei et al., 23 Jun 2025). Multi-objective RVPO is validated in rubric-based health and science reasoning and tool-calling, where the operative desideratum is bottleneck adherence under many concurrent rewards (Montero et al., 7 May 2026).

A recurrent theme across these settings is that variance is never treated as uniformly good or uniformly bad. Classical RVPO penalizes it (Bisi et al., 2019, Zhang et al., 2020). GRPOVI and ReDit inject or amplify it to escape flat regions (Yang et al., 29 May 2025, Wei et al., 23 Jun 2025). Multi-objective RVPO penalizes disagreement across objectives while still preserving within-channel group normalization (Montero et al., 7 May 2026). This suggests that the meaning of “variance” in RVPO is inseparable from the statistical object on which it is defined.

7. Limitations, ambiguities, and related misconceptions

A common misconception is that any policy optimization method with “variance” in its name belongs to RVPO. The ratio-variance method R2VPO is an explicit counterexample: it constrains the variance of the importance ratio $$J_\pi = \mathbb{E}_{(s,a)\sim d_{\mu,\pi}}[r(s,a)],$$6, not the variance of rewards, and the paper states that there is no reward-variance penalty in the method (Luo et al., 26 May 2026). Its role is local trust-region approximation and off-policy reuse, not reward shaping or risk-sensitive reward aggregation (Luo et al., 26 May 2026).

Another source of ambiguity is that reward-variance optimization can mean either variance penalization or variance amplification. In the 2019 and 2020 risk-averse literature, the objective is explicitly mean minus variance (Bisi et al., 2019, Zhang et al., 2020). In the 2025 RLHF literature, higher reward variance can accelerate training, and perfectly accurate but low-variance reward models can be poor teachers (Razin et al., 19 Mar 2025). ReDit goes further by stating that additive zero-mean noise can preserve the expected objective while improving optimization dynamics through variance shaping (Wei et al., 23 Jun 2025). These are not contradictory claims; they concern different tasks, different variance objects, and different desiderata.

The methods also have domain-specific failure modes. In classical RVPO, large risk-aversion coefficients can reduce return and slow learning, and on-policy estimation of variance can be data-intensive (Bisi et al., 2019). In MVPI, reward scaling affects the stability of critics because the augmented reward contains $$J_\pi = \mathbb{E}_{(s,a)\sim d_{\mu,\pi}}[r(s,a)],$$7 (Zhang et al., 2020). In GRPOVI, aggressive bounds can destabilize gradients, and incorrect reward-model rankings can be amplified because the method preserves the weak order of the original rewards (Yang et al., 29 May 2025). In ReDit, too much noise can overwhelm the reward signal, and the paper reports that large noise magnitudes such as $$J_\pi = \mathbb{E}_{(s,a)\sim d_{\mu,\pi}}[r(s,a)],$$8 degrade performance (Wei et al., 23 Jun 2025). In multi-objective RVPO, overly strong risk aversion early in training can destabilize learning, and SoftMin can amplify noisy reward channels if calibration is poor (Montero et al., 7 May 2026).

The broader lesson is that RVPO is best viewed as a design space of variance-aware policy optimization. Depending on the formulation, variance may be a quantity to minimize for risk control, a signal to amplify for better exploration and faster learning, or a disagreement pattern to penalize so that bottleneck objectives are not ignored. The unifying thread across the literature is not a single algorithmic template, but the recognition that the statistical structure of reward signals materially shapes policy optimization dynamics (Bisi et al., 2019, Zhang et al., 2020, Razin et al., 19 Mar 2025, Yang et al., 29 May 2025, Wei et al., 23 Jun 2025, Montero et al., 7 May 2026).