DVAO: Dynamic Variance-adaptive Advantage Optimization for Multi-reward Reinforcement Learning

Abstract: Reinforcement Learning has become a standard paradigm for aligning LLMs with human intent and task requirements. While Group Relative Policy Optimization offers an efficient, value-model-free alternative to Proximal Policy Optimization, adapting it to real-world multi-reward settings remains challenging. Standard scalarization practices, such as Reward Combination and Advantage Combination, suffer from significant drawbacks: Reward Combination frequently generates advantages with excessively large squared magnitudes that lead to training instability, while Advantage Combination relies on static hyperparameters and ignores cross-objective correlations. To address these limitations, we propose Dynamic Variance-adaptive Advantage Optimization (DVAO), which dynamically adjusts combination weights based on the empirical reward variance of each objective within a rollout group, effectively up-weighting objectives with a stronger learning signal while suppressing noisy ones. We mathematically prove that DVAO maintains bounded advantage magnitudes for stable training and introduces a self-adaptive cross-objective regularization mechanism. Extensive experiments on mathematical reasoning and tool-use benchmarks using Qwen3 and Qwen2.5 models demonstrate that DVAO significantly outperforms baseline methods, achieving a superior multi-objective Pareto frontier and robust training stability.

• The paper introduces DVAO, enhancing RL alignment in LLMs by adaptively optimizing advantage across multiple objectives.
• Through dynamic weighting based on empirical reward variance, DVAO achieves stable and convergent training dynamics, outperforming traditional methods.
• Empirical evaluations reveal DVAO's superior accuracy and compliance, consistently outperforming RC, AC, and GDPO in multi-objective scenarios.

Dynamic Variance-adaptive Advantage Optimization for Multi-reward RL in LLMs

Motivation and Background

LLMs increasingly rely on Reinforcement Learning (RL) for aligning their output with complex human preferences and multifaceted task requirements. Contemporary RL post-training pipelines, such as Group Relative Policy Optimization (GRPO), have supplanted traditional value-based approaches like Proximal Policy Optimization (PPO) by leveraging relative advantage estimation within sampled rollout groups, eliminating the need for a separate value model and offering substantial memory and computational efficiency.

In realistic LLM deployments, alignment objectives are seldom singular. Tasks often require simultaneous maximization across disparate and sometimes antagonistic criteria: solution accuracy, length compliance, tool-use correctness, code bug minimization, and hallucination suppression. Existing multi-reward scalarization schemes—Reward Combination (RC) and Advantage Combination (AC)—exhibit pronounced deficiencies: RC induces unstable training via advantage magnitude explosion, while AC normalizes magnitudes but neglects cross-objective synergy, relying instead on static hyperparameter weighting.

DVAO: Algorithmic Formulation and Theoretical Analysis

Dynamic Variance-adaptive Advantage Optimization (DVAO) systematically rectifies these limitations by dynamically weighting individual advantage contributions according to the empirical reward variance within a rollout group. For rollout group $\{y_j\}_{j=1}^G$ and reward functions $\{r_k\}_{k=1}^n$, DVAO computes adaptive weights

$$\tilde{w}_k = \frac{w_k \sigma_k^i}{\sum_l w_l \sigma_l^i}$$

where $\sigma_k^i$ is the standard deviation for objective $k$ in group $i$. The variance-adaptive advantage is then given by

$$A_\text{DVAO}^{(i,j)} = \sum_k \tilde{w}_k A_k^{(i,j)}$$

with $A_k^{(i,j)}$ independently normalized per objective.

Proposition 1 proves that RC always produces a larger mean squared advantage compared to AC unless all objectives are perfectly correlated, leading to erratic and overly aggressive policy gradient updates. Proposition 2 establishes that DVAO further reduces advantage magnitude, proving $$\left|A_\text{DVAO}^{(i,j)}\right| \leq \left|A_\text{sum}^{(i,j)}\right|$$ pointwise for all rollouts, thus bounding policy gradients and promoting stable convergence. Proposition 3 elucidates that DVAO enables adaptive cross-objective regularization: the sensitivity of the combined advantage to an individual reward is modulated by the global multi-objective performance, ensuring synergistic learning and preventing over-optimization of trivial objectives.

Empirical Evaluation

Mathematical Reasoning and Tool-use Benchmarks

DVAO's empirical efficacy is rigorously assessed on mathematical reasoning datasets (AIME-2024, AIME-2025, MATH500, OlympiadBench, AMC23) and tool-use benchmarks (Berkeley Function Call Leaderboard). The models evaluated include Qwen3-4B-Base, Qwen3-8B-Base for math tasks and Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct for tool-use. Metrics focus on task accuracy and auxiliary compliance (length/format).

DVAO consistently achieves the highest average accuracy and near-perfect auxiliary compliance across both tasks and scales. Unlike RC, AC, and GDPO, which exhibit strict trade-offs or severe instability, DVAO maintains superior Pareto frontiers, simultaneously optimizing conflicting objectives.

Training Dynamics

In-depth analysis of training curves demonstrates DVAO's impact on gradient stability and convergence rate.

Figure 1: Training dynamics on Qwen3-4B-Base illustrating DVAO's superior stability in accuracy reward, length reward, and response length.

DVAO yields the highest accuracy rewards with sharply reduced variance relative to RC and AC. Length reward reaches target values rapidly without excessive standard deviation, confirming the cross-objective regularization effect. Response length grows fastest under DVAO, attaining higher plateaus with bounded oscillation.

Figure 2: Training dynamics on Qwen3-8B-Base highlighting DVAO’s variance reduction and response length optimization.

Variance suppression is even more pronounced in larger models, preventing either reward channel from dominating gradient updates and ensuring robust trade-off navigation.

Pareto Frontier Analysis

DVAO’s dominance is evident in frontier plots, which show accuracy versus auxiliary compliance under a sweep of scalarization weights.

Figure 3: Pareto frontier in mathematical reasoning tasks (Qwen3-4B-Base) showing DVAO’s upper-right coverage; analogous superiority holds in tool-use tasks.

DVAO achieves consistently superior trade-offs—maintaining high compliance across accuracy ranges—while baseline methods (RC, AC, GDPO) are constrained by unstable gradients or fail to generalize across objectives.

Practical and Theoretical Implications

Practically, DVAO eliminates the need for labor-intensive hyperparameter tuning and manual reward balancing, offering a principled framework for robust multi-objective alignment in LLM post-training. Its variance-adaptive mechanism is inherently extensible to scenarios with a greater number of objectives, provided that reward calibration is reasonable.

Theoretically, DVAO establishes rigorous bounds on advantage magnitude and introduces a novel cross-objective regularization, ensuring optimization is both stable and cognizant of global multi-objective performance. DVAO bridges the gap between reward normalization and synergistic learning, propelling alignment research toward more holistic RL post-training paradigms.

Speculation on Future Directions

Future extensions may focus on scaling DVAO to high-dimensional reward spaces, further stabilizing variance estimation under extreme group size constraints, and integrating adaptive weighting into broader RL paradigms, including hierarchical, meta-reward, or curriculum-based frameworks. Ensuring reward calibration remains paramount, as variance-driven weighting may inadvertently amplify noisy objectives if auxiliary rewards are poorly defined.

Conclusion

This work critically analyzes and rectifies the shortcomings of conventional scalarization techniques for multi-reward RL in LLMs. Dynamic Variance-adaptive Advantage Optimization offers bounded advantage magnitudes, self-adaptive cross-objective regularization, and empirically achieves superior multi-objective alignment without hyperparameter dependence. DVAO constitutes a robust foundation for future large-scale RL-based LLM alignment and multi-objective optimization (2605.25604).