---
title: Reward-Variance Policy Optimization (RVPO)
url: https://www.emergentmind.com/topics/reward-variance-policy-optimization-rvpo
type: topic
---

# Reward-Variance Policy Optimization (RVPO)

Searching arXiv for the cited RVPO-related papers and terminology to ground the article in current literature.
Reward-Variance Policy Optimization (RVPO) is a term that has acquired multiple, partly overlapping meanings in reinforcement learning and RLHF. In one line of work, RVPO denotes a risk-averse objective that maximizes expected reward while penalizing the variance of rewards under a policy-induced occupancy measure, yielding formulations such as reward-volatility reduction and mean–variance control [1912.03193]. In more recent RLHF literature, the same label refers to methods that use reward variance as an optimization resource: either by explicitly penalizing inter-reward variance across multiple alignment objectives via a SoftMin aggregator to prevent “constraint neglect” [2605.05750], or by increasing within-group reward variance to accelerate GRPO-style training while preserving reward expectation and relative preferences [2505.23247]. A related estimator-level perspective appears in reward dithering, where zero-mean noise is added to discrete rewards so that variance is injected without changing the expected reward, thereby mitigating gradient collapse in group-relative policy optimization [2506.18631]. The term therefore names a family of variance-aware policy optimization ideas rather than a single universally standardized algorithm.

## 1. Terminological scope and historical usage

The literature uses “Reward-Variance Policy Optimization” in at least three distinct senses. The earliest explicit formulation in the provided corpus is risk-averse control based on the variance of step-wise rewards under the discounted state-occupancy measure, instantiated as VOLA-PG and TRVO [1912.03193]. In that setting, RVPO optimizes
$$
J_\lambda(\pi) = J_\pi - \lambda \nu_\pi^2,
$$
where $J_\pi$ is the normalized expected discounted return and $\nu_\pi^2$ is reward volatility, defined as the variance of one-step rewards under the discounted occupancy measure [1912.03193].

A second usage appears in mean–variance policy iteration. Although the paper does not use the phrase RVPO explicitly, it directly optimizes a mean-minus-variance objective over the per-step reward random variable and presents an augmented-MDP construction that makes risk-averse control compatible with standard risk-neutral algorithms [2004.10888]. The paper states that this is a concrete instantiation of RVPO in practice [2004.10888].

A third, more recent usage emerges in RLHF and critic-less LLM post-training. Here, RVPO no longer means classical risk aversion over temporal reward volatility. Instead, it refers to variance-aware manipulation or aggregation of reward signals to improve optimization. One instance is GRPO with reward variance increase, which adjusts rewards to provably increase within-group variance while preserving expected reward and relative preferences [2505.23247]. Another is the 2026 “RVPO: Risk-Sensitive Alignment via Variance Regularization,” where multi-objective rewards are aggregated with a SoftMin that penalizes inter-reward variance across concurrent objectives [2605.05750]. ReDit is explicitly described as tightly connected to the idea of RVPO because it injects reward variance at the estimator level without changing the expected reward [2506.18631].

This terminological plurality is important. By contrast, “Ratio-Variance Regularized Policy Optimization” concerns the variance of the policy ratio, not the variance of rewards, and the paper explicitly states that the term “Reward-Variance Policy Optimization” does not appear there [2605.26784]. That distinction prevents conflating trust-region regularization in ratio space with reward-variance manipulation.

## 2. Classical RVPO as risk-averse reward-volatility optimization

In the risk-averse formulation, RVPO is defined on an infinite-horizon discounted MDP with discounted state-occupancy measure $d_{\mu,\pi}(s,a)$ [1912.03193]. The normalized expected discounted return is
$$
J_\pi = \mathbb{E}_{(s,a)\sim d_{\mu,\pi}}[r(s,a)],
$$
and reward volatility is
$$
\nu_\pi^2 = \mathrm{Var}_{(s,a)\sim d_{\mu,\pi}}[r(s,a)]
= \mathbb{E}[r(s,a)^2] - J_\pi^2.
$$
The resulting RVPO objective is
$$
\eta_\pi = J_\pi - \lambda \nu_\pi^2,
$$
with $\lambda \ge 0$ controlling the return–volatility trade-off [1912.03193].

A central theoretical result is that reward volatility upper-bounds return variance:
$$
\sigma_\pi^2 \le \nu_\pi^2/(1-\gamma)^2,
$$
where $\sigma_\pi^2$ is the variance of the discounted return [1912.03193]. This makes volatility reduction a tractable surrogate for return-risk reduction. The paper emphasizes that volatility is defined over single-step rewards under the occupancy distribution, which leads to a linear Bellman equation for the volatility value function
$$
X_\pi(s,a) = (r(s,a)-J_\pi)^2 + \gamma \mathbb{E}[X_\pi(s',a')],
$$
and enables a policy gradient theorem analogous to the standard one [1912.03193].

The corresponding gradient takes the form
$$
\nabla_\theta \eta_\pi
=
\mathbb{E}_{(s,a)\sim d_{\mu,\pi}}
\big[\nabla_\theta \log \pi_\theta(a|s)\,Q_\pi^\lambda(s,a)\big],
$$
or equivalently with a baseline,
$$
\nabla_\theta \eta_\pi
=
\mathbb{E}_{(s,a)\sim d_{\mu,\pi}}
\big[\nabla_\theta \log \pi_\theta(a|s)\,A_\pi^\lambda(s,a)\big],
$$
where $Q_\pi^\lambda = Q_\pi - \lambda X_\pi$ [1912.03193]. The paper instantiates this objective in both an actor-only Monte Carlo algorithm, VOLA-PG, and a trust-region method, TRVO, with a TRPO-style monotonic-improvement bound adapted to the risk-averse objective [1912.03193].

The practical significance of this version of RVPO is that it is a genuine objective-level reformulation: the policy is optimized toward lower reward volatility, not merely trained with a variance-shaped estimator. This differs materially from later RLHF variants that preserve the population objective while changing the optimization signal.

## 3. Mean–variance policy iteration and augmented-MDP formulations

Mean–Variance Policy Iteration (MVPI) recasts variance-aware control through an augmented reward derived from the per-step reward random variable under the policy’s discounted occupancy distribution [2004.10888]. The objective is
$$
J_\lambda(\pi) = \mathbb{E}[R] - \lambda \mathrm{Var}(R)
= \mathbb{E}[R] - \lambda \mathbb{E}[R^2] + \lambda(\mathbb{E}[R])^2,
$$
where $R$ is the per-step reward random variable induced by the occupancy measure [2004.10888].

The difficulty is the $(\mathbb{E}[R])^2$ term, which makes the reward depend on the policy. MVPI removes this dependence using Fenchel duality:
$$
(\mathbb{E}[R])^2 = \max_y \big(2\mathbb{E}[R]y - y^2\big).
$$
For fixed $y$, one obtains an augmented MDP with reward
$$
\hat r(s,a;y) = r(s,a) - \lambda r(s,a)^2 + 2\lambda y r(s,a),
$$
which no longer depends on $\pi$ [2004.10888]. This yields a block cyclic coordinate ascent procedure: first estimate $y_{k+1} = (1-\gamma)J(\pi_k)$, then optimize the risk-neutral control problem on the augmented MDP $M_{y_{k+1}}$ [2004.10888].

The practical implication is that any risk-neutral control method can serve as the policy-improvement step. The paper explicitly positions this as a plug-and-play bridge from risk-neutral to risk-averse control, including both on-policy and off-policy settings [2004.10888]. Its exemplar is risk-averse TD3, which replaces the original reward with
$$
\hat r_i = r_i - \lambda r_i^2 + 2\lambda y r_i
$$
and then uses standard TD3 critics and actor updates on that modified target [2004.10888].

This formulation broadens the RVPO family beyond actor-only policy gradients. It suggests that, in the classical risk-averse sense, RVPO is best understood as an objective template—mean reward minus a variance term—rather than as a single optimizer. The choice of optimizer can be inherited from risk-neutral RL once the augmented reward is defined.

## 4. RVPO in RLHF: reward variance as an optimization resource

Recent RLHF work shifts the emphasis from risk aversion to optimization geometry. “What Makes a Reward Model a Good Teacher? An Optimization Perspective” proves that low reward variance can make the RLHF objective flat even when the reward model is perfectly accurate [2503.15477]. For autoregressive softmax policies, the gradient norm is upper-bounded by a quantity proportional to the cube root of the KL-regularized reward variance, and escape-time lower bounds scale inversely with initial on-policy reward variance [2503.15477]. The paper further shows that a less accurate reward model can train faster than a perfectly accurate one if it induces higher on-policy reward variance [2503.15477].

This optimization perspective directly motivates RVPO-style mechanisms in RLHF. “Accelerating RLHF Training with Reward Variance Increase” proposes a reward adjustment model that increases within-group reward variance while preserving expected reward and relative preferences [2505.23247]. For a GRPO group with rewards $\{r_i\}$ and initial-policy probabilities $\{p_i\}$, the adjusted rewards $\mathbf z$ are obtained by solving
$$
\max_{\mathbf z} \sum_{i=1}^n p_i z_i^2
$$
subject to bounds, expectation preservation, and monotonicity constraints that preserve relative preferences [2505.23247]. Because the mean is fixed, maximizing $\sum_i p_i z_i^2$ is equivalent to maximizing variance [2505.23247].

The paper proves that any maximizer lies at an extreme point of the feasible polyhedron and that extreme points have at most three distinct plateau values: a top block at $M$, a middle plateau at $\alpha$, and a bottom block at $m$ [2505.23247]. It then gives an $O(n\log n)$ algorithm for the globally optimal adjustment after sorting by rewards [2505.23247]. When integrated into GRPO as GRPOVI, the adjusted rewards replace the original group rewards before standardization, strengthening the contrastive advantage signal [2505.23247].

A closely related estimator-level mechanism is ReDit. In GRPO-style training with discrete rewards, flat mini-batches cause near-zero group-relative advantages, and reward discontinuities can create occasional gradient explosions [2506.18631]. ReDit perturbs each reward as
$$
r'_i = r_i + \epsilon_i,
$$
with $\mathbb{E}[\epsilon_i]=0$ and either Gaussian or Uniform noise [2506.18631]. The paper states that this preserves the expected gradient direction for the original objective at the REINFORCE level while increasing gradient variance by an explicit controllable term [2506.18631]. Within GRPO, the perturbed rewards are standardized to form
$$
A'_i \propto \frac{r'_i - \mathrm{mean}(\{r'_k\})}{\mathrm{std}(\{r'_k\})},
$$
which prevents advantage collapse when the base rewards are identical [2506.18631].

These RLHF formulations share a common principle: variance can be deliberately engineered to improve optimization. However, they differ in where the intervention occurs. GRPOVI alters rewards through a constrained optimization problem [2505.23247], whereas ReDit alters the estimator statistics via zero-mean perturbations and explicitly does not change the expected reward [2506.18631]. A plausible implication is that “RVPO” in RLHF increasingly denotes variance-aware signal design rather than classical risk-averse objective design.

## 5. Multi-objective RVPO and variance regularization across reward channels

The 2026 paper “RVPO: Risk-Sensitive Alignment via Variance Regularization” introduces a distinct but formally precise use of RVPO for critic-less multi-objective RLHF [2605.05750]. The motivating problem is “constraint neglect”: when $K$ concurrent reward signals are aggregated by arithmetic mean, high performance on easy objectives can numerically offset failures on strict bottleneck constraints such as safety, structure, or completeness [2605.05750].

RVPO replaces arithmetic averaging with a SoftMin, applied to per-objective standardized scores:
$$
Z_k^{(g)} = \frac{R_k^{(g)} - \mu_k}{\sigma_k + \epsilon},
$$
and then
$$
A_{\text{RVPO}}^{(g)}
=
-\beta \log
\Big(
\tfrac{1}{K}
\sum_{k=1}^K e^{-Z_k^{(g)}/\beta}
\Big).
$$
This score is whitened across the batch and inserted into an otherwise standard PPO-style clipped objective with KL regularization [2605.05750].

The paper’s key analytic result is the cumulant expansion
$$
-\beta \log \Big( \tfrac{1}{K} \sum_{k=1}^K e^{-x_k/\beta} \Big)
=
\mu - \frac{\sigma^2}{2\beta} + \frac{\kappa_3}{6\beta^2} + O\!\left(\tfrac{1}{\beta^3}\right),
$$
showing that the SoftMin acts as a smooth variance penalty on inter-objective disagreement [2605.05750]. As $\beta \to \infty$, the aggregator approaches the mean; as $\beta \to 0$, it approaches the hard minimum [2605.05750]. The gradient weights are larger for worse-performing objectives, so bottlenecks are emphasized rather than averaged away [2605.05750].

This sense of RVPO is neither classical mean–variance control over time nor within-group variance injection for faster RLHF. It is a risk-sensitive aggregator across reward channels. The variance being controlled is inter-objective variance at a fixed sample, not temporal reward volatility and not policy-ratio variance [2605.05750]. That distinction matters conceptually. Here, the method explicitly changes the scalar training objective from “maximize sum” to “maximize consistency” across objectives [2605.05750].

Empirically, the paper reports that RVPO improves HealthBench scores over GDPO at 14B scale, reaching 0.261 versus 0.215 with $p < 0.001$, and avoids the late-stage degradation observed in other multi-reward methods [2605.05750]. It also maintains competitive accuracy on GPQA-Diamond and enforces tool-calling format constraints earlier and more stably while retaining execution correctness [2605.05750]. These findings suggest that, in multi-objective alignment, variance regularization is not merely an optimization-speed device but a mechanism for preserving hard constraints that mean aggregation obscures.

## 6. Algorithms, guarantees, and empirical patterns

The major RVPO-related variants differ in their objective, the location of the variance term, and the type of guarantee they provide.

| Variant | Variance object | Main mechanism |
|---|---|---|
| Classical RVPO / TRVO | Step-wise reward variance under occupancy | Optimize $J_\pi - \lambda \nu_\pi^2$ [1912.03193] |
| MVPI | Per-step reward variance | Augmented reward via Fenchel duality [2004.10888] |
| GRPOVI-style RVPO | Within-group reward variance | Constrained reward adjustment preserving mean and order [2505.23247] |
| ReDit connection to RVPO | Estimator-visible reward variance | Zero-mean dithering of discrete rewards [2506.18631] |
| Multi-objective RVPO | Inter-objective variance across reward channels | SoftMin aggregation of per-objective Z-scores [2605.05750] |

In the classical risk-averse setting, the guarantee is structural: reward volatility bounds return variance, the volatility Bellman equation is linear, and TRVO inherits a TRPO-style monotonic-improvement bound under a KL trust region [1912.03193]. In MVPI, monotonic policy improvement follows from alternating optimization over the auxiliary variable $y$ and the policy on the augmented MDP, with convergence to stationary points under boundedness and smoothness assumptions [2004.10888].

In RLHF variance-increase methods, the guarantees are optimization-centric. Theoretical analysis in the reward-model paper states that insufficient on-policy reward variance implies flat landscapes and slow improvement, with explicit escape-time lower bounds [2503.15477]. GRPOVI proves that the reward adjustment preserves expected reward and relative preferences while increasing variance over the response space under the initial policy [2505.23247]. ReDit proves unbiasedness at the REINFORCE level and gives an explicit variance formula for the added noise term, such as
$$
\mathrm{Var}\big[\epsilon \nabla \log \pi_\theta(o|q)\big]
=
\sigma^2 \mathbb{E}\big[\|\nabla \log \pi_\theta(o|q)\|^2\big]
$$
for Gaussian noise [2506.18631].

The empirical patterns are correspondingly different. Classical RVPO and MVPI are validated in finance and MuJoCo-style continuous control, where lower volatility or better risk-aware metrics are the target [1912.03193, 2004.10888]. GRPOVI and ReDit are validated in LLM post-training, where the relevant outcomes are convergence rate, sample efficiency, and stability of group-relative optimization [2505.23247, 2506.18631]. Multi-objective RVPO is validated in rubric-based health and science reasoning and tool-calling, where the operative desideratum is bottleneck adherence under many concurrent rewards [2605.05750].

A recurrent theme across these settings is that variance is never treated as uniformly good or uniformly bad. Classical RVPO penalizes it [1912.03193, 2004.10888]. GRPOVI and ReDit inject or amplify it to escape flat regions [2505.23247, 2506.18631]. Multi-objective RVPO penalizes disagreement across objectives while still preserving within-channel group normalization [2605.05750]. This suggests that the meaning of “variance” in RVPO is inseparable from the statistical object on which it is defined.

## 7. Limitations, ambiguities, and related misconceptions

A common misconception is that any policy optimization method with “variance” in its name belongs to RVPO. The ratio-variance method R2VPO is an explicit counterexample: it constrains the variance of the importance ratio $p_t(\theta)=\pi_\theta/\pi_{\text{off}}$, not the variance of rewards, and the paper states that there is no reward-variance penalty in the method [2605.26784]. Its role is local trust-region approximation and off-policy reuse, not reward shaping or risk-sensitive reward aggregation [2605.26784].

Another source of ambiguity is that reward-variance optimization can mean either variance penalization or variance amplification. In the 2019 and 2020 risk-averse literature, the objective is explicitly mean minus variance [1912.03193, 2004.10888]. In the 2025 RLHF literature, higher reward variance can accelerate training, and perfectly accurate but low-variance reward models can be poor teachers [2503.15477]. ReDit goes further by stating that additive zero-mean noise can preserve the expected objective while improving optimization dynamics through variance shaping [2506.18631]. These are not contradictory claims; they concern different tasks, different variance objects, and different desiderata.

The methods also have domain-specific failure modes. In classical RVPO, large risk-aversion coefficients can reduce return and slow learning, and on-policy estimation of variance can be data-intensive [1912.03193]. In MVPI, reward scaling affects the stability of critics because the augmented reward contains $-\lambda r^2$ [2004.10888]. In GRPOVI, aggressive bounds can destabilize gradients, and incorrect reward-model rankings can be amplified because the method preserves the weak order of the original rewards [2505.23247]. In ReDit, too much noise can overwhelm the reward signal, and the paper reports that large noise magnitudes such as $a=0.5$ degrade performance [2506.18631]. In multi-objective RVPO, overly strong risk aversion early in training can destabilize learning, and SoftMin can amplify noisy reward channels if calibration is poor [2605.05750].

The broader lesson is that RVPO is best viewed as a design space of variance-aware policy optimization. Depending on the formulation, variance may be a quantity to minimize for risk control, a signal to amplify for better exploration and faster learning, or a disagreement pattern to penalize so that bottleneck objectives are not ignored. The unifying thread across the literature is not a single algorithmic template, but the recognition that the statistical structure of reward signals materially shapes policy optimization dynamics [1912.03193, 2004.10888, 2503.15477, 2505.23247, 2506.18631, 2605.05750].

Source: https://www.emergentmind.com/topics/reward-variance-policy-optimization-rvpo