DualReward: Multi-Objective RL

Updated 3 July 2026

DualReward is a reinforcement learning paradigm that integrates two heterogeneous reward signals to guide multi-objective policy optimization.
It employs techniques like reward decomposition, filtering, and consensus-based aggregation to balance conflicting signals and prevent reward dominance.
DualReward frameworks demonstrate improved sample efficiency and stability across applications such as distractor generation, language model alignment, and tool usage.

DualReward denotes a family of reinforcement learning (RL) methodologies and algorithms that leverage two distinct reward sources, channels, or signals to guide agent policy optimization. The dual-reward paradigm has emerged as a key approach for multi-objective RL, model training under feedback uncertainty, and practical applications such as distractor generation, LLM alignment, and hybrid preference-reward optimization.

1. Conceptual Foundations of DualReward

DualReward frameworks formalize RL problems where an agent receives not just a single scalar reward, but two heterogeneous, complementary, or even conflicting reward signals. Typical motivations for adopting a dual-reward structure include:

Trade-off between objectives: e.g., balancing accuracy vs. length, helpfulness vs. harmlessness, or aligning correctness with diversity in language generation.
Mitigating reward misspecification: Utilizing multiple reward sources to hedge against the brittleness or bias of any single channel.
Bridging supervised and unsupervised RL: Integrating human-verified and model-internal rewards for improved scalability and autonomy.

The canonical dual-reward problem can be stated as maximizing the expected sum of weighted returns from two reward functions $r^1, r^2$ :

$J(\pi_\theta) = \mathbb{E}_{q,\,y\sim\pi_\theta(\cdot|q)}\big[w_1 r^1(q, y) + w_2 r^2(q, y)\big]$

Here, $w_1, w_2$ are importance weights and $q$ indexes queries or tasks (Liu et al., 15 Jun 2026).

2. Methodologies for DualReward Learning

Reward Decomposition and Filtering

Modern dual-reward frameworks decompose reward signals at various granularities—such as answer-level vs. completion-level in language modeling (Joarder et al., 21 May 2026), or human-generated vs. model-generated distractors in cloze test generation (Huang et al., 16 Jul 2025). Each signal is normalized to mitigate scale imbalances, often by per-group $z$ -scoring (GDPO principle) or advantage standardization:

$A_n^i = \frac{r_n^i - \mu^i}{\sigma^i}, \quad i\in\{1,2\}$

where $n$ indexes rollouts and $\mu^i, \sigma^i$ are group mean and standard deviation for reward $i$ .

Group-dynamic filtering (Liu et al., 15 Jun 2026) prunes rollouts with opposing or weakly correlated advantages, e.g., by requiring $\mathrm{sign}(A_n^1) = \mathrm{sign}(A_n^2)$ (hard filtering), or by imposing a signal-to-noise ratio threshold.

Conflict Mitigation and Advantage Aggregation

A central challenge is preventing conflicting reward signals from canceling out or destabilizing training. Group-Dynamic reward-Decoupled Policy Optimization (GD $J(\pi_\theta) = \mathbb{E}_{q,\,y\sim\pi_\theta(\cdot|q)}\big[w_1 r^1(q, y) + w_2 r^2(q, y)\big]$ 0PO) resolves this by masking rollouts with severe reward-wise disagreement and reweighting updates by the fraction of retained (consensus) rollouts per query (Liu et al., 15 Jun 2026):

$J(\pi_\theta) = \mathbb{E}_{q,\,y\sim\pi_\theta(\cdot|q)}\big[w_1 r^1(q, y) + w_2 r^2(q, y)\big]$ 1

Filtered advantages are aggregated only for consensus rollouts, and updates are downweighted for queries with little agreement.

KL-Regularization and Collapse Prevention

In RL from internal feedback (RLIF), dual-reward schemes can suffer from entropy collapse when one reward signal dominates (Joarder et al., 21 May 2026). KL-Covariance (KL-Cov) regularization is introduced to penalize overconfident token-level shifts on high-covariance tokens, preserving exploration dynamics while maintaining multi-channel reward diversity.

3. Algorithmic Developments

A spectrum of algorithmic frameworks address dual-reward scenarios:

Framework	Reward Channels	Key Techniques
DualReward (Huang et al., 16 Jul 2025)	Gold-vs-generated distractors	Adaptive scaling, confidence weighting, policy gradient
GD $J(\pi_\theta) = \mathbb{E}_{q,\,y\sim\pi_\theta(\cdot\|q)}\big[w_1 r^1(q, y) + w_2 r^2(q, y)\big]$ 2PO (Liu et al., 15 Jun 2026)	Task-dependent (e.g. correctness+length)	Filtering, consensus-weighted updates
Collapse-free Multi-Reward RLIF (Joarder et al., 21 May 2026)	Cluster voting, self-certainty	GDPO normalization, KL-Cov regularization
DFA (Khorasani et al., 15 Aug 2025)	Scalar reward & preferences	Log-likelihood fusion, synthetic pair generation
MIRD (Krasheninnikov et al., 2021)	Reward from multiple sources	IRL on mixtures, convex-hull support
MD3QN (Zhang et al., 2021)	Vector-valued rewards	Joint distributional RL, MMD loss

DualReward for cloze distractor generation defines two reward levels: $J(\pi_\theta) = \mathbb{E}_{q,\,y\sim\pi_\theta(\cdot|q)}\big[w_1 r^1(q, y) + w_2 r^2(q, y)\big]$ 3 for human distractors and $J(\pi_\theta) = \mathbb{E}_{q,\,y\sim\pi_\theta(\cdot|q)}\big[w_1 r^1(q, y) + w_2 r^2(q, y)\big]$ 4 for model samples, with an adaptively scaled $J(\pi_\theta) = \mathbb{E}_{q,\,y\sim\pi_\theta(\cdot|q)}\big[w_1 r^1(q, y) + w_2 r^2(q, y)\big]$ 5 (Huang et al., 16 Jul 2025). This ensures robust learning across heterogeneous and cross-domain datasets.

GD $J(\pi_\theta) = \mathbb{E}_{q,\,y\sim\pi_\theta(\cdot|q)}\big[w_1 r^1(q, y) + w_2 r^2(q, y)\big]$ 6PO, when applied with two reward groups, filters rollouts whose standardized advantages disagree in sign or do not pass a signal-to-noise screening, improving reliability and sample efficiency. Empirical studies on tool-calling and helpfulness-safety benchmarks confirm consistent improvements versus non-filtered baselines (Liu et al., 15 Jun 2026).

Collapse-free dual-reward RLIF for language modeling combines cluster voting (answer-level agreement) and token-level certainty, independently normalized, and applies covariance-aware KL regularization to specific tokens, sustaining long-horizon exploration and preventing degenerate learning trajectories (Joarder et al., 21 May 2026).

4. Theoretical Properties and Analysis

DualReward learning introduces key desiderata and theoretical constructs (Krasheninnikov et al., 2021):

Support on S $J(\pi_\theta) = \mathbb{E}_{q,\,y\sim\pi_\theta(\cdot|q)}\big[w_1 r^1(q, y) + w_2 r^2(q, y)\big]$ 7: The agent’s posterior should cover all feature-wise combinations of reward sources, ensuring robustness to misspecification.
Intermediate tradeoff support: The agent’s hypothesis class should contain all convex combinations of relevant feature tradeoffs between sources.
Informativeness and balance: When reward sources agree on optimal behaviors, the agent’s belief/posterior should not introduce extraneous solutions, and mass should be balanced between distinct behaviors when reward functions diverge.

MIRD and its variants (e.g., MIRD-IF) construct a distribution over reward functions consistent with Pareto-optimal convex combinations or full independent-feature hypotheses, ensuring regret bounds linked to worst-case source performance.

DFA demonstrates, under Bradley-Terry preference assumptions, that the preference loss alone recovers the entropy-regularized SAC optimum, proving that hybrid reward-preference feedback leads to theoretically sound policy updates (Khorasani et al., 15 Aug 2025).

Distributional and multi-dimensional dual-reward approaches (e.g., MD3QN (Zhang et al., 2021)) guarantee Bellman contraction and convergence in the generalized setting where return is vector-valued, and model the full joint distribution for richer downstream control.

5. Practical Implementations and Applications

DualReward algorithms have been deployed in diverse domains:

Distractor generation for cloze tests: DualReward’s adaptive dual-reward structure yields superior P@1 on cross-domain MCQ data, outperforming state-of-the-art baselines with contextually robust distractor selection (Huang et al., 16 Jul 2025).
LLM post-training: Collapse-free dual-reward RLIF with cluster voting and self-certainty rectifies collapse phenomena in answer- and token-space, achieving performance near supervised RLVR approaches on code and mathematics benchmarks (Joarder et al., 21 May 2026).
Tool usage and multi-objective language tasks: GD $J(\pi_\theta) = \mathbb{E}_{q,\,y\sim\pi_\theta(\cdot|q)}\big[w_1 r^1(q, y) + w_2 r^2(q, y)\big]$ 8PO harmonizes multi-objective signals such as correctness and length, benefiting from enhanced learning stability and reliability compared to naïve aggregation (Liu et al., 15 Jun 2026).
Control with scalar-plus-preference feedback: Dual-Feedback Actor enables seamless integration of human preferences and rewards, with sample efficiency and stability improvements across MuJoCo and gridworld domains (Khorasani et al., 15 Aug 2025).

6. Empirical Results and Comparative Performance

Selected empirical findings, across frameworks and domains, include:

Setting	DualReward Approach	Main Outcome
CLOTH-F, MCQ (distractor gen)	(Huang et al., 16 Jul 2025)	Adaptive dual reward yields +3.5–6.2 pp in P@1 vs. baselines, especially on diverse MCQ.
Math/code generation (LLMs)	(Joarder et al., 21 May 2026)	Multi-reward + KL-Cov prevents collapse, retains 89–99% of peak accuracy over 340 steps.
Tool-calling, alignment tasks	(Liu et al., 15 Jun 2026)	GD $J(\pi_\theta) = \mathbb{E}_{q,\,y\sim\pi_\theta(\cdot\|q)}\big[w_1 r^1(q, y) + w_2 r^2(q, y)\big]$ 9PO improves consensus, accelerates learning, and increases final correctness/safety metrics.
Continuous control	(Khorasani et al., 15 Aug 2025)	DFA matches or exceeds SAC, with smoother, more stable training dynamics.

DualReward strategies consistently realize gains in sample efficiency, robustness to reward misspecification, and cross-domain generalization, while maintaining or exceeding the performance of established baselines.

7. Open Challenges and Future Directions

Despite substantial progress, several challenges remain in dual-reward RL research:

Reward scaling and dominance: Preventing pathological collapse due to variance or scale mismatch between reward channels remains nontrivial; adaptive normalization strategies and targeted regularization are active areas.
Scalability beyond two rewards: Extending principled dual-reward filtering, advantage decomposition, and conflict mitigation to higher-dimensional, highly-heterogeneous reward settings.
Generalization and transfer under misspecification: Robustness to unseen environments and reward channel failures motivates future work on more general inverse reward design and uncertainty quantification (Krasheninnikov et al., 2021).
Preference-reward fusion under nonstationarity: Ensuring consistent sample efficiency and policy optimality when feedback and reward distributions shift over time.

DualReward frameworks continue to play a critical role in aligning complex models with multi-objective signals, serving as an archetype for multi-reward and hybrid-feedback RL in both research and real-world applications.