Multi-Reward Reinforcement Learning

• Multi-Reward Reinforcement Learning is a framework where agents optimize multiple distinct reward signals to navigate trade-offs and achieve joint optimality.
• It employs techniques like reward shaping, distributional modeling, and meta-learning to address non-linear, constrained, and multimodal objectives.
• Applications range from natural language generation and robotics to cooperative multi-agent systems and drug discovery, enhancing adaptability in complex tasks.

Multi-Reward Reinforcement Learning (RL) extends classical RL by explicitly modeling, aggregating, and optimizing with respect to multiple, often heterogeneous, reward signals. These reward streams may originate from distinct objectives, modalities, constraints, or external signals (including human feedback), and the learning agent must discover and deploy strategies that navigate trade-offs, leverage synergies, or achieve joint optimality across these multiple rewards. Research in multi-reward RL spans theoretical foundations, algorithmic advances, practical frameworks for reward decomposition and shaping, robust integration of human or multimodal input, and application of distributional or multi-objective paradigms.

1. Theoretical Principles and Problem Formulations

Multi-reward RL encompasses scenarios where the agent’s objective is to optimize a function of several long-term reward streams, rather than a scalar accumulation. The general formulation is as follows:

$$\pi^* = \operatorname{argmax}_\pi\ f(\lambda^\pi_1, \ldots, \lambda^\pi_K)$$

where $f : \mathbb{R}^K \to \mathbb{R}$ is typically nonlinear (concave, Lipschitz, or otherwise structured), and $\lambda^\pi_k$ denotes the expected (possibly discounted) return for the $k$-th reward function under policy $\pi$ (Agarwal et al., 2019). The presence of a nonlinear $f$ generally breaks the standard dynamic programming BeLLMan structure, complicating both analysis and algorithm design.

Several formulations arise:

• Multi-objective optimization (fairness, quality of experience) where $f$ may be, e.g., a utility or fairness metric.
• Constrained settings, where some reward channels are objectives and others are interpreted as costs (hard or soft constraints) (Kim et al., 24 Sep 2024).
• Distributional paradigms, where the agent estimates the joint or marginal distribution of multi-dimensional returns (Zhang et al., 2021).

Policy optimality, convergence properties, and regret bounds are established through occupancy measure analysis and contraction mappings in appropriate metrics (e.g., supremum Wasserstein).

2. Algorithmic Frameworks and Reward Integration

Approaches in multi-reward RL include reward decomposition and aggregation, distributional modeling, and meta-learning over reward combinations:

Reward-Shaping and Potential-Based Augmentation

Potential-based reward shaping ensures policy invariance while injecting auxiliary reward signals (e.g., from language or logic-based sources), integrating multiple streams as $R_{total} = R_{ext} + \lambda R_{aux}$ (Goyal et al., 2019, ElSayed-Aly et al., 2022). The LEARN framework, for instance, maps free-form natural language instructions into potential functions, yielding intermediate shaping rewards that are combined with environmental rewards.

Direct Optimization of Nonlinear Objectives

Algorithms address nonlinearly aggregated long-term rewards through:

• Model-based policies optimizing steady-state occupancy distributions under nonlinear reward objectives ($f$-optimization over $d(s,a)$, subject to stochasticity and occupancy constraints) (Agarwal et al., 2019).
• Policy-gradient approaches employing chain rule decompositions of the objective $f$, resulting in gradient signals that are nontrivial mixtures of the gradients for each component reward (Agarwal et al., 2019).

Distributional and Modular Decompositions

Distributional Reward Decomposition for RL (DRDRL) and MD3QN optimize the full return distribution across multiple channels:

• DRDRL learns separate latent sub-distributions per reward channel, convolves these for the overall return, and regularizes for disentanglement (Lin et al., 2019).
• MD3QN models the joint return distribution to explicitly capture inter-reward dependencies, updating using maximum mean discrepancy between predicted and BeLLMan target distributions (Zhang et al., 2021).

Meta-RL and Bandit-Based Weighting

Dynamic adjustment of reward weights via bandit algorithms, both contextual and non-contextual, allows real-time adaptation of the mixture when rewards are incommensurable or dynamically shifting in importance (Min et al., 20 Mar 2024). DynaOpt and C-DynaOpt exemplify this, leveraging multi-arm bandits to adjust reward mixing and rapidly steer policy learning toward underperforming dimensions in multilingual or multi-quality text generation.

Human-in-the-Loop and Reward Learning

Multi-task reward learning integrates autonomous learning from both regression and classification interpretations of human ratings using learnable uncertainty-based weights, producing a reward function that is both smooth and categorically informative (Wu et al., 10 Jun 2025). Bayesian reward modeling with targeted queries (IDRL) concentrates supervision where it most impacts policy discrimination, supporting efficient reward function discovery for multi-reward or multi-objective settings (Lindner et al., 2021).

3. Reward Machines and Structured Decomposition

Reward machines, formalized as finite-state automata or Mealy machines, expose the structure of temporally extended or multi-faceted reward functions. Their explicit representation enables:

• Automated reward shaping: Potential functions over automaton states smooth reward signals while ensuring policy invariance (Icarte et al., 2020).
• Task decomposition via options: Internal RM transitions define hierarchies of subtasks, naturally decomposing multi-reward or complex objective scenarios (Icarte et al., 2020, Icarte et al., 2021).
• Counterfactual reasoning and augmentation: Experience from one RM state is reused across others through relabeling, massively increasing data efficiency and robustness in sparse- or multi-reward settings (Icarte et al., 2020, Xu et al., 2019).
• Joint inference: Algorithms iteratively optimize policies and infer RM structure, exploiting equivalence of RM states to transfer value functions, and support rapid convergence on tasks with high-level reward structure (Xu et al., 2019).

Empirical studies show that leveraging RM structure yields faster and more robust learning, especially in combinatorially complex or partially observable domains (Icarte et al., 2021).

4. Practical Applications and Empirical Advances

Multi-reward RL has seen diverse application across domains where objectives are inherently multifactorial:

• Natural Language Generation: Multiple text quality objectives (fluency, coherence, stylistic reflection) are jointly optimized with dynamic reward weighting (bandit-based techniques) to mirror human expert criteria (Min et al., 20 Mar 2024).
• Robotics and Control: Stage-wise reward and cost segmentation in acrobatic robots enables tractable and robust policy learning for complex, sequential tasks. The CoMOPPO algorithm aggregates normalized multi-reward and cost advantages, enforcing constraints and achieving Pareto-optimal policies in high-dimensional tasks (e.g., back-flip, side-flip, two-hand walk) (Kim et al., 24 Sep 2024).
• Human Preferences and Reward-Free Settings: Multi-task reward learning from ratings (regression and classification) matches and sometimes exceeds the performance of traditional RL and previous rating-based RL, reflecting the need to balance subjective, noisy, or context-dependent objectives (Wu et al., 10 Jun 2025).
• Cooperative Multi-Agent Systems: Differentiated reward mechanisms utilizing transition gradients accelerate learning, enhance safety, and foster rational collective behavior in complex traffic and multi-vehicle control scenarios, scaling robustly with system size (Han et al., 1 Feb 2025).
• Drug Discovery and Goal-Focused Generative Tasks: Maximum-reward RL and variants introduce new BeLLMan operators and recursive formulations to locate and propagate “hit” states—rare events of high value—boosting sample efficiency and alignment of RL objectives with practical reward structures (Gottipati et al., 2020, Veviurko et al., 2 Feb 2024).
• Exploration and Off-Line Planning: Reward-free RL decouples exploration from objective specification, enabling sample-optimal collection of environment data that supports near-optimal planning for any later reward function, thus extending trivially to multi-reward settings (Jin et al., 2020).

5. Representation Learning and Reward Sensitivity

Recent developments in RL representation learning address the shortcomings of reward-agnostic features:

• Reward-Aware Proto-Representations (Default Representation, DR): By integrating exponential reward dynamics into representations, DR captures both transition and reward landscape structure, leading to exploratory and transfer behaviors that intrinsically avoid low-reward regions and better align with multi-reward objectives (2505.16217). Theoretical characterizations reveal correspondences and distinctiveness compared to the (reward-agnostic) successor representation, while empirical results demonstrate improvements in reward shaping, intrinsic motivation, and rapid adaptation to new reward configurations.
• Distributional and Multimodal Reward Modeling: In multimodal LLMs (MRMs), destabilizing variance during RL-based reward learning is addressed by strategies such as StableReinforce, which uses pre-clipping, outlier filtering, and auxiliary reward components (e.g., consistency rewards evaluated by referees) (Zhang et al., 5 May 2025). This stabilizes training and improves alignment with complex, multimodal objectives.

6. Challenges, Limitations, and Future Directions

Fundamental and practical challenges persist:

• Non-commutativity and Non-Markovianity: Nonlinear objective formulations undermine classic dynamic programming guarantees, forcing advances in stochastic occupancy modeling, variational optimization, and function approximation (Agarwal et al., 2019).
• Reward Design and Shaping Complexity: The process of designing intermediate or shaping rewards, especially for multi-modal or highly context-dependent tasks, remains both labor-intensive and domain-specific. Automated approaches using language, logic, or automata mitigate but do not fully obviate this burden (Goyal et al., 2019, ElSayed-Aly et al., 2022).
• Scalability and Expressivity: Approaches relying on explicit automata, joint distributions, or product state spaces can face prohibitive scaling challenges (curse of dimensionality), requiring innovations in decentralization, modular design, or approximate compression (Icarte et al., 2020, Zhang et al., 2021).
• Uncertainty, Interpretability, and Human Alignment: Integrating human signals—especially when ratings are noisy, inconsistent, or contextually variable—necessitates task formulations and loss functions that balance multiple feedback streams under uncertainty (Wu et al., 10 Jun 2025).
• Transfer, Generalization, and Adaptivity: Future (and ongoing) work focuses on learning reward functions and policies that generalize across tasks, adapt to changing objectives, and transfer flexibly among related domains, leveraging structured representations, active reward learning, and dynamic multi-reward integration.

Advances in multi-reward RL continue to be foundational to complex real-world deployment, aligning agents with sophisticated, evolving objectives that characterize modern artificial intelligence tasks.