---
title: Multi-Reward Reinforcement Learning (MRRL)
url: https://www.emergentmind.com/topics/multi-reward-reinforcement-learning-mrrl
type: topic
---

# Multi-Reward Reinforcement Learning (MRRL)

Multi-Reward Reinforcement Learning (MRRL) is a branch of reinforcement learning (RL) focusing on optimization, modeling, and control when the agent’s objective incorporates multiple, often conflicting, reward signals. Unlike standard RL, which targets a single scalar reward, MRRL explicitly formulates tasks with vector-valued, decomposable, or joint reward structures. This class encompasses linearly-weighted multi-objective RL, distributional methods for multi-dimensional returns, non-linear reward aggregation, and dynamic reward-weight adaptation.

## 1. Formal Problem Definitions and Core MRRL Frameworks

In MRRL, the Markov Decision Process is extended so that the reward function returns a vector $r(s,a) \in \mathbb{R}^m$ of $m$ distinct components rather than a scalar. Each component typically represents a separate criterion (e.g., task success, time cost, safety constraint, fairness):

- **Vector reward**: $r(s,a) = [r_1(s,a), ..., r_m(s,a)]$
- **Policy**: $\pi(s)$ (or, in parameterized MRRL, $\pi(s, w)$ for a weight vector $w$)
- **Long-term return vector**: $R_\pi = \sum_{t=0}^\infty \gamma^t r(s_t, a_t)$

Common MRRL approaches transform this vector reward into a scalar via a scalarization function—most often a linear combination:
$$
r_w(s,a) = \sum_{i=1}^m w_i r_i(s,a)
$$
where $w \in \Delta^{m-1}$ (the probability simplex), allowing dynamic trade-offs between objectives.

Alternatively, the objective may be a non-linear function $f: \mathbb{R}^m \to \mathbb{R}$ of the expected long-term returns, or one may seek to model the joint distribution of returns across multiple reward channels.

## 2. Algorithmic Approaches and Scalarization Strategies

### Linearly-Scalarized Policy Learning

Methods such as the extended HER-based MRRL [1809.06364] and MO-GPSARSA [1707.06299] employ a linearly-weighted combination of vector rewards. The key innovations are:

- Treating reward weights $w$ as inputs to the policy and/or value function.
- Data augmentation via sampling auxiliary weights per transition and storing multiple versions of experiences in the replay buffer.
- Off-policy learning (e.g., DDPG-style) with modified Bellman updates:
  $$
  Q^*(s, a, w) = \mathbb{E}_{s'} [r_w(s,a) + \gamma \max_{a'} Q^*(s', a', w)]
  $$
- Networks are conditioned on both the state and the weight vector $w$.

This paradigm enables a single policy $\pi(s, w)$ to generalize over all possible trade-offs between reward components, allowing for real-time adjustment at deployment without retraining [1809.06364].

### Non-linear and Arbitrary Aggregation

MRRL also covers cases where the objective is an arbitrary (often concave) function of the long-term average reward vector, $f(R(\pi))$. In such settings, classical dynamic programming is not directly applicable due to non-additivity. [1909.02940] formulates the problem as a convex program over steady-state occupancy measures and presents both model-based (posterior sampling + convex optimization) and model-free (joint policy-gradient) algorithms with theoretical regret bounds, specifically for optimizing arbitrary concave, $L$-Lipschitz functions.

### Dynamic Reward Mixture Adjustment

For settings where the optimal mixing weights between reward components are unknown and may need to change during training, bandit-driven adaptive mixtures have been proposed. Methods such as DynaOpt and C-DynaOpt [2403.13578] employ non-contextual (Exp3) and contextual (Online Cover) bandits to adapt reward weights $W$ online:

- The combined reward for a sample is $R_{comb}(c, y; W) = \sum_{i=1}^N W_i r_i(c, y)$.
- Weight updates are guided by bandit feedback, modulating the inner RL policy's updates.
- Bandit selection may either be stateless (Exp3) or context-sensitive (context and recent reward statistics via Online Cover).

This dynamic adjustment systematically outperforms static uniform or round-robin weighting in language generation tasks with multiple objectives [2403.13578].

## 3. Distributional Methods and Multi-Reward Decomposition

Distributional MRRL approaches retain the return distribution structure of each reward signal, enabling richer modeling of joint reward statistics and credit assignment.

- **Multi-Dimensional Distributional DQN (MD3QN)** [2110.13578] models the joint distribution $\mu_\pi(s, a) \in \mathcal{P}(\mathbb{R}^d)$ over long-term returns for all $d$ reward sources. Empirical loss uses Maximum Mean Discrepancy (MMD) between sampled return vectors and Bellman targets. Proven contraction of the joint Bellman operator in Wasserstein distance is established, and learning matches or outperforms scalar and per-head baselines when rewards exhibit complex dependencies.

- **Distributional Reward Decomposition for RL (DRDRL)** [1911.02166] learns $N$ parallel return distributions (one per hypothesized reward channel) and reconstructs the full return via convolution. A KL-based disentanglement penalty encourages networks to specialize per sub-channel while avoiding trivial decompositions.

The following table summarizes several MRRL algorithmic paradigms:

| Approach                            | Reward Handling            | Policy Input        |
|--------------------------------------|---------------------------|--------------------|
| Scalarized DDPG (MRRL) [1809.06364]  | Linear weighted sum       | $(s, w)$           |
| MO-GPSARSA [1707.06299]              | Linear weighted sum       | $(b, a, w)$        |
| MD3QN [2110.13578]                   | Joint distribution        | $s, a$             |
| DRDRL [1911.02166]                   | Parallel distributional   | $x, a$ (by channel)|
| DynaOpt/C-DynaOpt [2403.13578]       | Dynamic linear mixture    | $c, y$ (language)  |
| Nonlinear joint optimization [1909.02940] | General $f(R)$     | $s$                |

## 4. Experimental Domains, Empirical Findings, and Metrics

- **Continuous Control**: Double-integrator problems (1D/2D) with time and fuel rewards. The MRRL policy $\pi(s,w)$ smoothly tracks analytic Pareto fronts, interpolating between time- and fuel-optimal behavior [1809.06364].
- **Spoken Dialogue Systems**: Balancing dialogue success and length in statistical SDS across six domains. MO-GPSARSA achieves policy efficiency, matching SO-RL with 3x fewer samples, and demonstrates optimized performance curves as a function of weight [1707.06299].
- **Fairness and Resource Allocation**: Scheduling and queueing environments with up to 8 reward channels. Model-based and policy-gradient MRRL are superior to classical DQN and vanilla policy gradients for proportional-fairness and $\alpha$-fairness [1909.02940].
- **Atari and Pixel-based Tasks**: MD3QN and DRDRL improve both representation of reward dependencies and control performance over scalar and independent RL baselines, particularly when joint statistics or reward decomposition informs strategy [2110.13578, 1911.02166].
- **Language Generation**: DynaOpt/C-DynaOpt increase target metrics (e.g., reflection quality) while maintaining other qualities (fluency, coherence) relative to uniform or alternate reward training [2403.13578].

Metrics vary by domain: task-specific returns, Pareto distance to analytic solutions, joint distribution quality (MMD error), dialogue success rate, task length, sample efficiency, and human evaluation scores.

## 5. Practical Considerations, Insights, and Limitations

Several implementation insights and limitations have emerged:

- Proper data augmentation rate ($k$) is crucial; too little harms generalization, too much slows learning [1809.06364].
- Uniform sampling of weights in the simplex achieves broad coverage; for scalarization-based methods, post-training test-time adaptation is possible with a universal policy [1809.06364, 1707.06299].
- Bandit-based dynamic mixtures shift the reward emphasis as training progresses, avoiding manual tuning [2403.13578].
- Factored distributional models (DRDRL) can outperform joint models for practical sample complexity reasons if reward channels are only weakly coupled [1911.02166].
- For arbitrary non-linear reward aggregation, convex optimization over steady-state occupancy enables tractable solutions and theoretical regret analysis, but scaling to large state-action spaces may require function approximation and loses regret guarantees [1909.02940].
- There is no single best MRRL paradigm; the structure of reward correlations, the functional form of the objective, and application constraints determine the most suitable approach.

Open problems include robust generalization in high reward dimensions, restriction of computational overhead of sample-based or kernel methods, principled exploration in multi-reward regimes, and formally linking dynamic bandit-based mixture policies to theoretical optimality guarantees.

## 6. Broader Interpretations and Future Directions

MRRL subsumes multi-objective RL, reward decomposition, joint distributional modeling, and dynamic reward fusion strategies. The principal advantages are:

- Enabling real-time adaptation to user preferences without retraining [1809.06364].
- Efficient exploitation of structure in multi-channel environments (credit assignment, exploration, constraint satisfaction) [2110.13578, 1911.02166].
- Sample-efficient optimization of complex, possibly non-linear, metrics (fairness, diversity, safety) [1909.02940].
- Algorithm-independent mechanisms (e.g., bandit-guided reward mixing) for adaptive control over learning trade-offs in generative tasks [2403.13578].

Promising directions include improved adaptive reward mixing via meta-bandits or Bayesian optimization, scalable joint distributional estimation, function approximation for occupancy-measure planning, and generalization to broader classes of non-Markovian or context-dependent objectives.

## 7. Key References

- "Generalizing Across Multi-Objective Reward Functions in Deep Reinforcement Learning" [1809.06364]
- "Reward-Balancing for Statistical Spoken Dialogue Systems using Multi-objective Reinforcement Learning" [1707.06299]
- "Distributional Reinforcement Learning for Multi-Dimensional Reward Functions" [2110.13578]
- "Distributional Reward Decomposition for Reinforcement Learning" [1911.02166]
- "Reinforcement Learning for Joint Optimization of Multiple Rewards" [1909.02940]
- "Dynamic Reward Adjustment in Multi-Reward Reinforcement Learning for Counselor Reflection Generation" [2403.13578]

Source: https://www.emergentmind.com/topics/multi-reward-reinforcement-learning-mrrl