Dynamic Reward Weighting in Multi-Objective RL

• Dynamic reward weighting is a framework that updates reward coefficients online, adapting to varying optimization dynamics in multi-objective settings.
• It employs mechanisms like bandit controllers, gradient signals, and batch statistics to adjust weights based on training phase, signal variance, and evaluation feedback.
• Empirical results show improved task success, fluency, and alignment in domains such as RL, language generation, and sparse-reward exploration.

Dynamic reward weighting denotes a family of optimization strategies in which the coefficients attached to rewards, reward components, examples, prompts, rollouts, or evaluation criteria are updated online rather than fixed a priori. Across multi-objective reinforcement learning, RLHF and RLVR, controllable text generation, process reward modeling, and sparse-reward exploration, these methods replace static scalarization with weights that depend on training phase, model capability, state, prompt difficulty, empirical reward statistics, or held-out evaluation signals (Min et al., 2024, Langis et al., 2024, Nguyen et al., 27 Feb 2026, Lu et al., 14 Sep 2025).

1. Formal scope and problem statement

A standard starting point is linear scalarization. In the dialogue-policy setting studied by Ultes et al., the reward is a vector $r_t=(r_t^{success}, r_t^{length})$, scalarized as $f(r_t,w)=w_1 r_t^1+w_2 r_t^2$, with per-dialogue reward written as $r=1_{TS}\,w_s\,r_s + T\,w_l\,r_l$ (Ultes et al., 2017). In later multi-reward language-generation work, the same pattern appears in explicitly time-varying form:

$$R_t = \sum_{i=1}^N w_i(t)\,r_i(t), \qquad \sum_{i=1}^N w_i(t)=1,\; w_i(t)\ge 0,$$

where $w_i(t)$ is determined during training rather than fixed beforehand (Min et al., 2024).

The motivation for making $w_i$ dynamic is consistent across domains. Static mixtures are described as suboptimal because different rewards exhibit different learning dynamics; a fixed mix cannot adapt to changing signal scales or stage-wise learning needs, and manual tuning becomes combinatorially expensive as the number of rewards increases (Min et al., 2024). In online multi-objective alignment, fixed-weight linear scalarization is further limited by Pareto geometry: it can recover Pareto-optimal points on convex portions of the front but provably fails on non-convex regions (Lu et al., 14 Sep 2025). In RLVR, DARO argues that static or overly simplistic weighting schemes tied to sample difficulty create a “loss scale issue,” causing training to focus disproportionately on some difficulty levels while neglecting others (Zhou et al., 10 Oct 2025).

These formulations show that dynamic reward weighting is not a single algorithm but a design space. What varies is the object being weighted, the signal used to update the weight, and the optimization layer at which the weight enters the learning rule.

2. Principal adaptation mechanisms

One major family uses external controllers. DynaOpt and C-DynaOpt employ non-contextual and contextual multi-arm bandits to adjust reward weights during training. In DynaOpt, arms correspond to “increase weight of reward $i$” or “do nothing,” with arm probabilities updated by Exp3; in C-DynaOpt, the context vector concatenates current reward weights and recent dev-set average rewards, and arm selection uses LinUCB or VW’s Online Cover (Min et al., 2024).

A second family uses gradient-based signals. In multi-style controllable generation, the weight for style $i$ at iteration $t$ is proportional to the $\ell_2$ norm of the policy gradient of the expected log-discriminator score,

$f(r_t,w)=w_1 r_t^1+w_2 r_t^2$0

so styles with larger current gradient magnitude receive more reward budget (Langis et al., 2024). In dynamic reward weighting for online alignment, weights are treated as trainable variables and updated by mirror descent: $f(r_t,w)=w_1 r_t^1+w_2 r_t^2$1 where $f(r_t,w)=w_1 r_t^1+w_2 r_t^2$2 measures each objective’s influence on overall improvement (Lu et al., 14 Sep 2025).

A third family uses batch statistics. DVAO computes a dynamic weight for each objective from empirical reward variances within a rollout group,

$f(r_t,w)=w_1 r_t^1+w_2 r_t^2$3

and combines individually normalized advantages rather than raw rewards (Jiang et al., 25 May 2026). In distributed RL, R-Weighted gradient merger scales each actor’s gradient according to how high its episodic reward is compared to the other actors, after subtracting the minimum reward and adding a baseline $f(r_t,w)=w_1 r_t^1+w_2 r_t^2$4 term (Holen et al., 2023).

A fourth family is constraint-based. In constrained RLHF, composite reward models are not hand-weighted by fixed coefficients; instead, dynamic weights are expressed by Lagrange multipliers. When a component reward exceeds its threshold of usefulness, the corresponding multiplier rises and penalizes further optimization of that component; when it falls below threshold, the multiplier shrinks (Moskovitz et al., 2023).

These mechanisms differ operationally, but they share a common premise: the scalarized training signal should react to the current optimization regime rather than encode a static compromise.

3. Granularity: what is being weighted

Dynamic reward weighting is often introduced as weighting reward components, but the literature applies the same principle at several granularities. At the most direct level, weights mix multiple objectives inside the scalar reward. The hierarchical UAV framework uses an episode-level module to produce a global preference $f(r_t,w)=w_1 r_t^1+w_2 r_t^2$5, a step-level module to produce instantaneous softmax weights, and a fusion coefficient $f(r_t,w)=w_1 r_t^1+w_2 r_t^2$6 to interpolate between them before scalarizing the two Q-values for action selection (Wang et al., 9 May 2026).

At the advantage level, weighting may occur after each reward component has been normalized. DVAO forms per-objective advantages $f(r_t,w)=w_1 r_t^1+w_2 r_t^2$7 and then combines them with dynamic $f(r_t,w)=w_1 r_t^1+w_2 r_t^2$8, explicitly separating normalization from scalarization (Jiang et al., 25 May 2026). Constrained RLHF similarly defines a mixed advantage $f(r_t,w)=w_1 r_t^1+w_2 r_t^2$9 that blends negative-KL and component-RM advantages using adaptive multiplier-derived coefficients (Moskovitz et al., 2023).

At the prompt, rollout, or query level, the weights do not combine reward components but regulate update intensity. In RLVR, common algorithms such as GRPO, DAPO, and RLOO effectively weight gradients by prompt difficulty and focus on ambiguous prompts, downgrading very easy and very hard prompts; asymmetric prompt weighting instead assigns higher weights to prompts with low, or even zero, empirical success probability, especially benefiting from-scratch RL (Heckel et al., 11 Feb 2026). GD$r=1_{TS}\,w_s\,r_s + T\,w_l\,r_l$0PO performs conflict-aware filtering that masks rollouts suffering from severe reward-wise disagreement, then applies query-level reweighting through the retained fraction $r=1_{TS}\,w_s\,r_s + T\,w_l\,r_l$1, so queries with stronger reward consensus receive larger updates (Liu et al., 15 Jun 2026).

At the data-example level, weighting is learned as a reward parameterization. In data weighting for supervised learning, each training example $r=1_{TS}\,w_s\,r_s + T\,w_l\,r_l$2 has a reward parameter $r=1_{TS}\,w_s\,r_s + T\,w_l\,r_l$3, and the minibatch sampling weight is

$r=1_{TS}\,w_s\,r_s + T\,w_l\,r_l$4

The $r=1_{TS}\,w_s\,r_s + T\,w_l\,r_l$5 values are updated by back-propagating validation log-likelihood through the model update, so example importance changes dynamically over training (Hu et al., 2019).

At the criterion level, dynamic reward weighting becomes part of the reward model itself. DG-PRM stores a hierarchical reward tree, selects contextually relevant criteria at each generation step, and computes criterion weights from Pareto-dominance statistics before aggregating them into a scalar reward (Yin et al., 23 Jul 2025). P-Check generates a dynamic checklist for each user-query pair and assigns criterion saliency scores according to how much each item separates a preferred response from augmented negatives; normalized saliency then determines the final criterion weights (Seo et al., 6 Jan 2026). ACWI applies the same idea to intrinsic motivation: the coefficient on intrinsic reward is a learned state-dependent $r=1_{TS}\,w_s\,r_s + T\,w_l\,r_l$6, not a fixed scalar (Nguyen et al., 27 Feb 2026).

This spread across granularities is one of the topic’s defining characteristics. “Reward weighting” may govern scalarization, data selection, prompt prioritization, criterion aggregation, or exploration bonuses.

4. Theoretical rationales

Several strands of theory explain why dynamic weighting can outperform fixed scalarization. In multi-objective alignment, the basic geometric argument is that fixed linear weights recover only convex portions of the Pareto front, whereas dynamic reward weighting can move through non-convex regions by continuously changing the scalarization during training (Lu et al., 14 Sep 2025).

In RLVR, asymmetric prompt weighting is supported by a time-to-target analysis. The theory characterizes prompt weights that minimize the time needed to raise success probability from an initial level to a target accuracy under a fixed update budget. In low-success regimes, where informative responses are rare and response cost dominates, the optimal weights become asymmetric and upweight low success probabilities (Heckel et al., 11 Feb 2026). This is a direct challenge to the common practice of emphasizing only ambiguous prompts.

In multi-reward policy optimization, DVAO provides two guarantees. First, it states that the combined dynamic advantage remains bounded, avoiding the magnitude explosion associated with raw reward combination. Second, differentiation of the combined advantage produces a cross-term $r=1_{TS}\,w_s\,r_s + T\,w_l\,r_l$7, which is interpreted as a self-adaptive cross-objective regularizer that couples each objective to overall multi-objective performance (Jiang et al., 25 May 2026).

In constrained RLHF, the theoretical concern is overoptimization of proxy reward models. Composite reward models can become anti-correlated with human judgment beyond a threshold, and the location of these proxy points depends on correlation between component reward models. Dynamic weights expressed as Lagrange multipliers are then used to keep each component within the range where it remains a useful proxy (Moskovitz et al., 2023).

An adjacent but informative case appears in reward-weighted sampling for masked diffusion LLMs. There, a reward model scales logits at each diffusion step, and the paper proves rank-reversal conditions under which multiplicative scaling can cause lower-confidence tokens to overtake higher-confidence ones, together with a reward-monotonic guidance theorem stating that the expected reward under the reward-scaled distribution is at least as high as under the unscaled distribution (Gwak et al., 31 Aug 2025). This suggests that dynamic weighting can shape not only parameter updates but also decoding trajectories.

5. Empirical manifestations across domains

Empirical work spans dialogue systems, language generation, LLM alignment, reward modeling, robotics, and sparse-reward exploration. In spoken dialogue, Ultes et al. used MO-GPSARSA to search reward balances over six domains and then retrained single-objective policies with the selected weight. They report statistically significant improvements in task success rate in every domain except CamRestaurants; the Laptops domain rises from $r=1_{TS}\,w_s\,r_s + T\,w_l\,r_l$8 to $r=1_{TS}\,w_s\,r_s + T\,w_l\,r_l$9 TSR, and TV rises from $$R_t = \sum_{i=1}^N w_i(t)\,r_i(t), \qquad \sum_{i=1}^N w_i(t)=1,\; w_i(t)\ge 0,$$0 to $$R_t = \sum_{i=1}^N w_i(t)\,r_i(t), \qquad \sum_{i=1}^N w_i(t)=1,\; w_i(t)\ge 0,$$1 (Ultes et al., 2017). In counselor reflection generation, DynaOpt and C-DynaOpt outperform naive and bandit baselines; relative to the cross-entropy baseline, DynaOpt gives $$R_t = \sum_{i=1}^N w_i(t)\,r_i(t), \qquad \sum_{i=1}^N w_i(t)=1,\; w_i(t)\ge 0,$$2 reflection quality, while C-DynaOpt gives $$R_t = \sum_{i=1}^N w_i(t)\,r_i(t), \qquad \sum_{i=1}^N w_i(t)=1,\; w_i(t)\ge 0,$$3 fluency, and human evaluation reports DynaOpt reflection $$R_t = \sum_{i=1}^N w_i(t)\,r_i(t), \qquad \sum_{i=1}^N w_i(t)=1,\; w_i(t)\ge 0,$$4 versus uniform $$R_t = \sum_{i=1}^N w_i(t)\,r_i(t), \qquad \sum_{i=1}^N w_i(t)=1,\; w_i(t)\ge 0,$$5 and DORB $$R_t = \sum_{i=1}^N w_i(t)\,r_i(t), \qquad \sum_{i=1}^N w_i(t)=1,\; w_i(t)\ge 0,$$6 (Min et al., 2024).

In controllable generation, dynamic weighting improves multi-style adherence while maintaining linguistic quality. For Negative + Informal control, dynamic weighting reaches $$R_t = \sum_{i=1}^N w_i(t)\,r_i(t), \qquad \sum_{i=1}^N w_i(t)=1,\; w_i(t)\ge 0,$$7 both negative and informal, compared with $$R_t = \sum_{i=1}^N w_i(t)\,r_i(t), \qquad \sum_{i=1}^N w_i(t)=1,\; w_i(t)\ge 0,$$8 for static binary reward and $$R_t = \sum_{i=1}^N w_i(t)\,r_i(t), \qquad \sum_{i=1}^N w_i(t)=1,\; w_i(t)\ge 0,$$9 for static logits reward; in three-style Positive + Formal + Irony control, the dynamic-RL model reaches $w_i(t)$0, $w_i(t)$1, and $w_i(t)$2, whereas the base LLaMA2 model gives $w_i(t)$3, $w_i(t)$4, and $w_i(t)$5 (Langis et al., 2024). In masked diffusion decoding, reward-weighted sampling increases Generation Order Deviation from about $w_i(t)$6 to about $w_i(t)$7, raises automatic-model win rate from about $w_i(t)$8 to about $w_i(t)$9, raises LLM-judge win rate from about $w_i$0 to about $w_i$1, and reduces keyword-task perplexity from about $w_i$2 to about $w_i$3 (Gwak et al., 31 Aug 2025).

In LLM post-training and alignment, dynamic weighting is used to stabilize multi-reward optimization and improve Pareto trade-offs. DARO reports the highest final average accuracy across three base models, including $w_i$4 for Llama-3.1-8B versus $w_i$5 for the best baseline and $w_i$6 for Qwen2.5-7B versus $w_i$7, while reaching a $w_i$8 pass-rate on Llama-3.1-8B in half the steps of the next-best method (Zhou et al., 10 Oct 2025). DVAO reports that on mathematical reasoning benchmarks it simultaneously achieves the highest accuracy and near-perfect length compliance, and on BFCL-v4 it yields substantially better function-calling accuracy and format compliance than reward- or advantage-combination baselines (Jiang et al., 25 May 2026). GD$w_i$9PO is reported to consistently and significantly outperform existing baselines in multi-reward scenarios including tool calling and human preference alignment (Liu et al., 15 Jun 2026). For online multi-objective alignment, both hypervolume-guided adaptation and gradient-based weight optimization dominate fixed-weight baselines on Math500 and reduce steps to full Pareto front by about six on average (Lu et al., 14 Sep 2025).

Dynamic weighting also appears inside reward models. DG-PRM improves overall PRM-Score from $i$0 to $i$1, raises missing-condition error identification from $i$2 to $i$3, and in cross-domain transfer shows only about a one-point drop versus five- to eight-point drops for static or critic models (Yin et al., 23 Jul 2025). P-Check reports $i$4 to $i$5 percentage points in user-macro reward accuracy over a default LLM-judge and improves downstream personalized generation under Best-of-$i$6 and DPO (Seo et al., 6 Jan 2026). In sparse-reward MiniGrid tasks, ACWI reaches a high-reward policy in roughly half the steps needed by the best fixed-$i$7 baseline on DoorKey-8x8 and RedBlueDoors, reduces variance across seeds, and matches the best fixed-$i$8 performance on Empty-16x16 without degrading further (Nguyen et al., 27 Feb 2026).

6. Limitations, misconceptions, and open directions

The literature repeatedly notes that dynamic weighting is not universally beneficial in the same way across regimes. Asymmetric prompt weighting particularly benefits from-scratch RL, where training traverses a wide accuracy range, and less so in post-SFT RL, where the model already starts at high accuracy (Heckel et al., 11 Feb 2026). In online multi-objective alignment, dynamic reward weighting presumes that the base model can improve on all objectives simultaneously; when objectives are inherently conflicting, gains are limited, a point explicitly observed on Ministral-8B-Instruct and Llama-3.1-8B (Lu et al., 14 Sep 2025). In ACWI, when extrinsic feedback is extremely sparse, the correlation signal vanishes and the learned $i$9 stays near its prior rather than providing strong adaptation (Nguyen et al., 27 Feb 2026).

A second limitation is computational overhead. Reward-weighted sampling increases inference time by about $i$0 to $i$1 and GPU memory by $i$2 to $i$3 GiB depending on reward-model size (Gwak et al., 31 Aug 2025). Hypervolume-guided dynamic weighting adds hypervolume-computation overhead that rises with Pareto-set size, while gradient-based weight optimization doubles gradient computations because each objective’s gradient must be estimated separately, even though the same trajectories are reused (Lu et al., 14 Sep 2025). Bandit-based methods incur periodic development-set evaluation, with contextual variants adding $i$4 bandit-update cost (Min et al., 2024).

A third limitation concerns reward quality. Dynamic weighting does not repair a biased or misaligned reward by itself. Reward-weighted sampling explicitly notes that biased or misaligned reward functions can degrade output (Gwak et al., 31 Aug 2025). Constrained RLHF addresses this indirectly by trying to keep each reward model inside its range of usefulness rather than maximizing it without bound (Moskovitz et al., 2023). This suggests that dynamic weighting should be understood as a control mechanism for optimization pressure, not as a substitute for reward-model validity.

The main open directions in the cited work extend this control mechanism to richer settings. Proposed directions include adaptive per-step or per-position $i$5, lightweight reward approximators, joint fine-tuning of the base model with reward-weighted feedback, and multimodal reward signals in diffusion decoding (Gwak et al., 31 Aug 2025); fully multi-objective policies that adapt $i$6 on-the-fly to user preference in dialogue (Ultes et al., 2017); and automatic detection of irreconcilable objective conflicts together with dynamic deactivation in online alignment (Lu et al., 14 Sep 2025). Taken together, these directions indicate that dynamic reward weighting is evolving from a scalarization heuristic into a broader framework for regulating learning pressure across objectives, data, prompts, and reward-model criteria.