DVAO: Variance-adaptive Advantage Optimization

• The paper presents DVAO, a dynamic method that reweights per-objective normalized advantages using empirical reward variance to enhance multi-reward reinforcement learning.
• DVAO addresses static scalarization failures by adaptively balancing learning signals and cross-objective correlations, ensuring bounded advantage magnitudes.
• Empirical benchmarks in math reasoning and tool use demonstrate that DVAO achieves superior accuracy, compliance, and training stability compared to baseline methods.

Dynamic Variance-adaptive Advantage Optimization (DVAO) is a multi-reward reinforcement learning method for GRPO-style LLM alignment that dynamically adjusts combination weights based on the empirical reward variance of each objective within a rollout group, effectively up-weighting objectives with a stronger learning signal while suppressing noisy ones (Jiang et al., 25 May 2026). In its formal 2026 formulation, DVAO addresses a specific failure mode of multi-objective scalarization in value-model-free policy optimization: Reward Combination can generate advantages with excessively large squared magnitudes, while Advantage Combination relies on static hyperparameters and ignores cross-objective correlations (Jiang et al., 25 May 2026). More broadly, the term also points to a wider research direction in which advantage-like signals are conditioned, reweighted, or selected using online variance information; related instances include adaptive advantage modulation in PPO, difficulty-adaptive group advantages in RLVR, and explained-variance-based critic switching (Sane, 21 May 2025, Gao et al., 1 Mar 2026, Pan et al., 21 Apr 2026).

1. Definition and conceptual placement

DVAO, in the strict sense, denotes the method introduced in "DVAO: Dynamic Variance-adaptive Advantage Optimization for Multi-reward Reinforcement Learning" (Jiang et al., 25 May 2026). Its immediate target is GRPO-style alignment in settings with multiple rewards, such as mathematical reasoning with correctness and length constraints, or tool use with semantic correctness and format compliance. The method remains value-model-free in the GRPO sense, but replaces static scalarization with a groupwise, variance-adaptive combination of normalized per-objective advantages (Jiang et al., 25 May 2026).

The method occupies a specific position within variance-aware policy optimization. It is not a generic reward-weighting heuristic; it is a construction on top of per-objective group-normalized advantages. This places it closer to advantage-space signal conditioning than to reward normalization. That distinction matters because the method operates after each objective has already been standardized within the rollout group, so the adaptive step is applied to signals that are comparable in scale but still retain objective-specific structure (Jiang et al., 25 May 2026).

This suggests a broader methodological family. AM-PPO applies a dynamic, non-linear modulation layer to GAE advantages using batch norm, standard deviation, and a target saturation level (Sane, 21 May 2025). DIVA-GRPO reshapes local and global advantages using difficulty-adaptive variants and reward-range-based scaling (Gao et al., 1 Mar 2026). EVPO monitors explained variance and switches between critic-based and batch-mean advantage estimation at each batch (Pan et al., 21 Apr 2026). DVAO differs from all three in that its variance signal is not the variance of a single advantage stream or critic residual, but the empirical reward variance of each objective within a multi-reward rollout group (Jiang et al., 25 May 2026).

2. Multi-reward GRPO and the scalarization problem

The motivating setting assumes $n$ objectives with reward functions

$$r_k : (x,y) \mapsto r_k(x,y) \in [0,1], \quad k=1,\dots,n.$$

For each query $x_i$, GRPO samples a group of $G$ rollouts $y_1,\dots,y_G$, computes relative advantages, and optimizes a PPO-like clipped surrogate. In the single-reward case, the group-normalized advantage is

$$A^{(i,j)} = \frac{r^{(i,j)} - \text{mean}\{r^{(i,j)}\}_{j=1}^G}{\text{std}\{r^{(i,j)}\}_{j=1}^G}.$$

Multi-reward training requires an additional combination step, and the DVAO formulation isolates two standard baselines: Reward Combination and Advantage Combination (Jiang et al., 25 May 2026).

Reward Combination first merges raw rewards,

$$r_\text{sum}^{(i,j)} = \sum_{k=1}^n w_k r_k^{(i,j)}, \quad \sum_k w_k = 1,\ w_k\in[0,1],$$

and only then normalizes: $$A_\text{sum}^{(i,j)} = \frac{ r_\text{sum}^{(i,j)} - \text{mean}\{r_\text{sum}^{(i,j)}\}_{j=1}^G }{ \text{std}\{r_\text{sum}^{(i,j)}\}_{j=1}^G }.$$ Advantage Combination instead normalizes each reward separately,

$$A_k^{(i,j)} = \frac{ r_k^{(i,j)} - \text{mean}\{r_k^{(i,j)}\}_{j=1}^G }{ \text{std}\{r_k^{(i,j)}\}_{j=1}^G },$$

and then combines them with fixed coefficients: $A^{(i,j)} = \sum_k w_k A_k^{(i,j)}.$

The two constructions fail for different reasons. Reward Combination can inflate the magnitude of the advantage because normalization occurs only after raw rewards have already been mixed. Advantage Combination keeps each component unit-scaled within the group, but the weights $$r_k : (x,y) \mapsto r_k(x,y) \in [0,1], \quad k=1,\dots,n.$$0 are static hyperparameters and the combination ignores cross-objective correlations (Jiang et al., 25 May 2026).

Scheme	Construction	Reported drawback
Reward Combination	Combine rewards, then normalize	Excessively large squared magnitudes
Advantage Combination	Normalize each reward, then linearly combine	Static hyperparameters; ignores cross-objective correlations
DVAO	Normalize each reward, then combine with variance-adaptive weights	Designed to bound magnitudes and adapt to signal strength

A central proposition formalizes the first problem. For a fixed query $$r_k : (x,y) \mapsto r_k(x,y) \in [0,1], \quad k=1,\dots,n.$$1, let $$r_k : (x,y) \mapsto r_k(x,y) \in [0,1], \quad k=1,\dots,n.$$2 be the sample correlation between per-objective advantages $$r_k : (x,y) \mapsto r_k(x,y) \in [0,1], \quad k=1,\dots,n.$$3 and $$r_k : (x,y) \mapsto r_k(x,y) \in [0,1], \quad k=1,\dots,n.$$4 in the group. Then the average squared magnitude of the Reward Combination advantage is at least as large as that of Advantage Combination, with equality only if all pairwise correlations are exactly $$r_k : (x,y) \mapsto r_k(x,y) \in [0,1], \quad k=1,\dots,n.$$5 (Jiang et al., 25 May 2026). The instability issue in multi-reward GRPO is therefore not merely heuristic; it follows from how scalarization interacts with group normalization.

3. Variance-adaptive construction

DVAO preserves per-objective normalization and makes the combination weights depend on the empirical standard deviation of each reward component inside the rollout group. For each query $$r_k : (x,y) \mapsto r_k(x,y) \in [0,1], \quad k=1,\dots,n.$$6, define

$$r_k : (x,y) \mapsto r_k(x,y) \in [0,1], \quad k=1,\dots,n.$$7

The adaptive weight is

$$r_k : (x,y) \mapsto r_k(x,y) \in [0,1], \quad k=1,\dots,n.$$8

and the DVAO advantage is

$$r_k : (x,y) \mapsto r_k(x,y) \in [0,1], \quad k=1,\dots,n.$$9

Objectives with larger $x_i$0 are therefore emphasized more strongly within that group, while low-variance objectives are attenuated (Jiang et al., 25 May 2026).

The interpretation given in the method description is operational rather than philosophical. High within-group variance means the current policy is differentiating sampled responses under that objective, so the signal is informative for policy improvement. Low variance means the reward is nearly flat across the group, so its learning signal is weak or noisy. DVAO turns that observation into a per-query reweighting rule (Jiang et al., 25 May 2026).

The procedure introduces no new modulation hyperparameters beyond the base convex weights $x_i$1. In the reported experiments, those base weights are equal, so the dynamic behavior is driven entirely by the observed $x_i$2 values (Jiang et al., 25 May 2026). This is a notable design choice: the adaptation is group-local and data-dependent, rather than controlled by an external schedule.

4. Theoretical properties

A key identity connects DVAO to Reward Combination: $x_i$3 where

$x_i$4

Using

$x_i$5

and the Cauchy-Schwarz bound

$x_i$6

the method derives

$x_i$7

From this follows the pointwise magnitude bound

$x_i$8

with equality iff all reward pairs are perfectly positively correlated (Jiang et al., 25 May 2026). This is the main stability guarantee: DVAO cannot produce larger absolute advantages than Reward Combination on the same rollout group.

The second theoretical result concerns sensitivity. For static Advantage Combination,

$x_i$9

For DVAO,

$G$0

The cross-term $G$1 is the defining structural difference (Jiang et al., 25 May 2026).

This term is the basis for the claim that DVAO introduces a self-adaptive cross-objective regularization mechanism. In static Advantage Combination, the gradient sensitivity of objective $G$2 depends only on its own normalized advantage. In DVAO, it depends on how that objective aligns with the combined multi-objective advantage on the same rollout. This suggests that DVAO discourages updates that improve one objective in isolation while conflicting with the aggregate signal, and conversely amplifies updates that are jointly supported across objectives (Jiang et al., 25 May 2026).

5. Empirical behavior and benchmark evidence

The reported evaluation covers two domains. In mathematical reasoning, the objectives are correctness and a binary length constraint, with the length reward equal to $G$3 if the response length is at most $G$4 tokens and $G$5 otherwise. Benchmarks are AIME-2024, AIME-2025, MATH500, OlympiadBench, and AMC23. Models are Qwen3-4B-Base and Qwen3-8B-Base (Jiang et al., 25 May 2026).

In tool use, the objectives are semantic correctness and binary format compliance on BFCL-v4. Models are Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct (Jiang et al., 25 May 2026).

Training uses the verl framework, AdamW with learning rate $G$6, group size $G$7, prompt batch size $G$8, $G$9 training steps, maximum generation length $y_1,\dots,y_G$0 tokens, and evaluation with temperature $y_1,\dots,y_G$1 and top-p $y_1,\dots,y_G$2; math evaluation uses avg@16 (Jiang et al., 25 May 2026).

The method is reported to significantly outperform baseline methods, achieving a superior multi-objective Pareto frontier and robust training stability (Jiang et al., 25 May 2026). On math reasoning, DVAO is described as the only method that achieves both the highest average accuracy across AIME-2024/25, MATH500, OlympiadBench, and AMC23 and near-perfect length compliance. On tool use, it is likewise described as the only method to simultaneously achieve top accuracy and top format compliance (Jiang et al., 25 May 2026).

Training dynamics are consistent with the theoretical claims. DVAO maintains the highest mean accuracy reward throughout training, reduces the standard deviation of the accuracy reward more than RC, AC, and GDPO, drives the length reward toward its maximum more quickly, and collapses the length reward standard deviation toward near-zero (Jiang et al., 25 May 2026). Response length increases fastest and reaches the highest final value under DVAO, indicating stronger exploration of longer reasoning chains while retaining bounded-advantage stability (Jiang et al., 25 May 2026).

The Pareto analysis varies the accuracy weight $y_1,\dots,y_G$3 over $y_1,\dots,y_G$4. Across both math and tool-use settings, DVAO dominates the frontier: for a given accuracy level it achieves higher compliance, and for a given compliance level it achieves higher accuracy (Jiang et al., 25 May 2026). This is precisely the regime where static scalarization is expected to struggle, because no single fixed weighting is optimal across training stages or local objective conflicts.

6. Relation to adjacent variance-adaptive advantage methods

DVAO is the named multi-reward formulation, but several nearby methods instantiate related principles. AM-PPO modifies PPO by inserting a dynamic, non-linear modulation layer on top of raw GAE advantages, with an adaptive controller driven by $y_1,\dots,y_G$5 norm, standard deviation, and a target saturation probability; the paper presenting AM-PPO does not use the term DVAO, but the mechanism was explicitly described as a concrete instance of that idea (Sane, 21 May 2025). EVPO uses batch-level explained variance to switch between critic-based and batch-mean baselines, guaranteeing no greater variance than the better of PPO and GRPO at each step (Pan et al., 21 Apr 2026). DIVA-GRPO maintains reward variance by difficulty-adaptive variant generation and combines local and global advantages with difficulty-weighted scaling (Gao et al., 1 Mar 2026). DynaMO combines variance-minimizing rollout allocation with token-level entropy-aware advantage modulation (Fang et al., 22 Feb 2026). ADORA dynamically reweights sample-level advantages according to online rollout statistics such as success rate and length, distinguishing temporarily advantageous and disadvantageous samples (Ren et al., 10 Feb 2026).

Earlier theoretical work supplies deeper antecedents. "Direct Advantage Estimation" shows that the advantage function can be learned directly by minimizing the variance of a shaped return, rather than derived only from value estimation (Pan et al., 2021). "Variance Adjusted Actor Critic Algorithms" optimizes a variance-adjusted expected return and constructs a variance-adjusted advantage-like signal from value and second-moment critics (Tamar et al., 2013). These approaches are not multi-reward GRPO methods, but they establish that variance-aware manipulation of advantage-like quantities is not incidental; it is a principled design axis in reinforcement learning (Pan et al., 2021, Tamar et al., 2013).

This suggests that DVAO is best understood in two layers. At the narrow layer, it is a specific multi-reward GRPO extension using groupwise reward variance to adapt objective weights (Jiang et al., 25 May 2026). At the broader layer, it belongs to a family of methods that treat the advantage stream as a dynamic object whose variance, scale, and informativeness should be explicitly controlled rather than passively inherited from rewards or critics (Sane, 21 May 2025, Pan et al., 21 Apr 2026).

7. Limitations and open directions

The main reported limitation is the quality of variance estimation when the rollout group is small. DVAO relies on empirical within-group standard deviations $y_1,\dots,y_G$6; with very small $y_1,\dots,y_G$7, these estimates can be noisy (Jiang et al., 25 May 2026). This is especially relevant in memory-constrained large-model training, where rollout group size may be reduced.

A second limitation is empirical scope. Most experiments use two objectives, and behavior in higher-dimensional reward spaces remains untested. The formulation supports general $y_1,\dots,y_G$8, but settings with five or more objectives were identified as future work rather than established practice (Jiang et al., 25 May 2026).

A third limitation concerns reward design. Because DVAO up-weights objectives with higher empirical variance, poorly designed rewards that are noisy rather than informative may receive excessive influence. The method therefore does not replace reward engineering; it assumes that high within-group variance tends to indicate useful discrimination among rollouts (Jiang et al., 25 May 2026).

Several extensions follow naturally from the surrounding literature. The DVAO paper itself points to temporal smoothing of $y_1,\dots,y_G$9, scaling to more objectives, and integration with PPO plus a value model (Jiang et al., 25 May 2026). Related work suggests further possibilities: modulation of returns or TD errors rather than only normalized advantages (Sane, 21 May 2025), hybrid critic-selection rules based on explained variance (Pan et al., 21 Apr 2026), and token-level or rollout-allocation variants that combine variance-aware weighting with entropy or difficulty signals (Fang et al., 22 Feb 2026, Gao et al., 1 Mar 2026). These are plausible continuations, but they remain distinct from the formal DVAO mechanism unless they preserve its defining structure: per-objective normalization followed by groupwise variance-adaptive reweighting (Jiang et al., 25 May 2026).