---
title: Multi-Objective Reward Function
url: https://www.emergentmind.com/topics/multi-objective-reward-function
type: topic
---

# Multi-Objective Reward Function

A multi-objective reward function is a vector-valued mapping from environment states and actions to ℝ^m, where m is the number of objectives of interest. Instead of collapsing multiple competing desiderata—such as task success, efficiency, safety, or user preferences—into a single scalar, the multi-objective reward maintains explicit structure across objectives. In reinforcement learning and sequential decision-making, multi-objective reward functions are foundational for expressing tasks with inherent trade-offs, supporting user-customizable behaviors, and enabling robust learning in environments where objectives are in tension or evolve dynamically. Modern approaches leverage these vector-valued rewards to enable explicit preference weighting, generalization across trade-off specifications, and direct exploration of the Pareto front over possible returns.

## 1. Mathematical Formulation and Scalarization

Let $r_t \in \mathbb{R}^m$ denote the vector reward observed at time $t$, with $m$ objectives. Preferences among objectives are encoded by a weight vector $w = (w_1, \dots, w_m)^\top$, where $w_i \geq 0$ and $\sum_i w_i = 1$. The scalarized reward is given by
$$
R_w(s_t, a_t) = w^\top r_t.
$$
This linear scalarization reduces the multi-objective problem to a conventional scalar reward MDP for fixed $w$, allowing standard RL techniques to be employed [1809.06364, 1909.05004].

The Pareto front of achievable vector returns arises by varying $w$ over the unit simplex. However, non-linear scalarizations—such as lexicographic ordering, thresholded approaches [2204.04988, 2502.10476], Chebyshev/Tchebycheff max-weighted objectives [2407.17466], or Nash Social Welfare [2212.01382]—provide additional expressivity, enabling non-convex Pareto regions and direct incorporation of fairness or context.

## 2. Deep RL Architectures and Policy Conditioning

Deep multi-objective RL architectures condition both actor and critic networks on the current preference vector $w$. For example, in MO-HER (Multi-Objective Hindsight Experience Replay), actors $\pi(s, w)$ and critics $Q(s, a, w)$ concatenate state and weight inputs, supporting interpolation over arbitrary trade-offs during inference [1809.06364].

This weight-conditioning enables a single network to generalize across the entire weight simplex, offering off-policy efficiency and continuous-action compatibility. Data augmentation by sampling multiple $w$ per transition and relabeling corresponding experience enables shared gradient signals over diverse objectives without additional environment interaction.

For tasks where only a subset of objectives is relevant at a given time, stage-wise or contextual approaches segment policies by regime (e.g., in acrobatic robots, per-maneuver stages with stage-dependent rewards and cost vectors), each with its own reward/cost definitions and potentially context-driven lexicographic or prioritization orderings [2409.15755, 2502.10476].

## 3. Sample-Efficient Generalization and Interpolation

The structure of $V^*(s; w)$, the optimal value function under a weighted-sum multi-objective reward, is Lipschitz and differentiable in $w$ under standard MDP regularity [1909.05004]. This facilitates the use of generalization techniques: once a policy or value function is trained under multiple weightings, interpolation (e.g., via Gaussian processes) enables prediction of optimal value functions for novel preferences, circumventing costly retraining per $w$.

Algorithms such as multi-objective GPSARSA [1707.06299], MO-DQN with reward and discount-factor generalization [2203.04579], and similar deep RL variants exploit these principles—relabelling data with randomly sampled $\mathbf w$ and fitting a weight-conditioned $Q$ or value network.

In the LLM alignment domain, preference-parameterized reward models (e.g., ArmoRM+MoE [2406.12845]) and complementary BT/multi-objective regression frameworks [2507.07375] produce interpretable multi-attribute scores, support gating or mixture mechanisms across objectives, and explicitly mitigate reward hacking by decomposing feedback into fine-grained, human-relevant axes.

## 4. Nonlinear Scalarization and Pareto Optimality

Linear scalarization cannot traverse non-convex Pareto regions. Nonlinear approaches such as generalized Thresholded Lexicographic Ordering (gTLO) [2204.04988], contextual lexicographic RL [2502.10476], and Chebyshev/Tchebycheff scalarization [2407.17466] are required.

For lexicographically ordered specifications, the agent enforces minimum thresholds on prioritized objectives, subsequently optimizing lower-priority rewards only within the admissible action set. This structure accommodates context-dependent priority orderings and can be extended via Bayesian inference of context mappings from demonstrations.

Tchebycheff scalarization,
$$
\mathrm{TCH}_\lambda(\pi) = \max_{i} \lambda_i (V_i^* - V_i^\pi),
$$
with strictly positive $\lambda$, traverses the full set of weakly Pareto-optimal policies and avoids the degeneracies of linear scalarization, yielding provably efficient algorithms for finding all Pareto solutions with $\tilde{\mathcal{O}}(\epsilon^{-2})$ sample complexity [2407.17466].

Welfare-based scalarizations (Nash social welfare, proportional fairness) provide fairness and scale invariance, but optimization is computationally harder and generally requires non-stationary or bi-level policy selection [2212.01382].

## 5. Preference Learning, Human Feedback, and Robustness

Direct specification of objective weights $w$ is often impractical; instead, preferences are inferred from human feedback using pairwise comparisons, demonstrations, or absolute ratings. Reward modeling frameworks (e.g., ArmoRM, Pb-MORL) build vector-valued reward models aligned to underlying human or task-driven trade-offs via regression and Bradley-Terry likelihoods [2406.12845, 2507.07375, 2507.14066].

Active preference elicitation (e.g., MORAL [2201.00012]) adaptively queries for the most informative pairwise comparisons, maintaining Bayesian posteriors over $w$ and updating the policy using the expected scalarized reward. Unified BT and regression heads in reward models are complementary: attribute-vector heads encourage fine-grained improvement and block reward hacking, while pairwise heads provide robustness and scalability when attribute annotation is scarce [2507.07375].

Contextual, stage-wise, or lexicographically varying policies further incorporate shifts in reward structure during operation or dynamically according to environment labels [2409.15755, 2502.10476]. Mixture-of-experts gating, as in ArmoRM+MoE, enables context-sensitive weighting of objectives and decomposable, interpretable output.

## 6. Optimization and Reward Balancing: Algorithms and Practical Insights

Optimization of multi-objective reward functions is performed using off-the-shelf RL algorithms with scalarized rewards or specialized multi-objective extensions. Weight-conditioned policy/value function networks can be efficiently trained by relabelling transitions with randomly sampled weights and using data augmentation from the vector reward [1809.06364, 1707.06299, 2203.04579].

Advanced RLHF and multi-objective policy algorithms employ explicit normalization of reward scales (e.g., MO-GRPO variance normalization [2509.22047]) to prevent objectives with large variance from dominating gradients and to mitigate reward hacking. Stochastic exploration in the reward-parameter space, meta-gradient or bi-level optimization of reward-component weights (e.g., MORSE [2512.15120]), and preference-free exploration for reward-free planning [2011.13034, 2407.17466] further enhance robustness and efficiency.

Table: Representative Multi-Objective Reward Function Approaches

| Method                      | Scalarization Type         | Policy Generalization           |
|-----------------------------|---------------------------|---------------------------------|
| Weighted linear sum         | Linear                    | Weight-conditioned network      |
| Chebyshev/Tchebycheff       | Nonlinear (max)           | Min-max RL, dual optimization   |
| Lexicographic ordering      | Nonlinear (priority)      | Threshold-conditioned network   |
| Nash SW, log-sum            | Nonlinear (concave)       | Welfare Q-learning/Stationary   |
| Contextual/Stage-wise       | Piecewise/Contextual      | Stage-conditional multi-policy  |
| Preference learning         | Adaptive                  | RM learned from pairwise data   |

## 7. Limitations and Research Directions

Multi-objective reward function approaches face inherent scalability challenges as the number of objectives increases, requiring smarter sampling or dimensionality reduction in the simplex [1809.06364]. Capacity to generalize across the weight space is also fundamentally constrained by the expressivity of the underlying policy/value approximator.

Linear scalarization fails to recover non-convex Pareto regions and may concentrate learning on a subset of objectives. Nonlinear or lexicographic scalarizations demand specialized backup operators and careful policy extraction, and may be sensitive to misspecification or to the presence of sharp trade-off regions.

Interpretability and robustness remain major focuses, particularly in RLHF contexts where reward hacking is a concern [2406.12845, 2507.07375, 2509.22047]. Empirical ablations demonstrate the critical role of decomposable proxies, context-dependent weighting, and explicit normalization, while theoretical work has established both lower and upper bounds for regret, exploration complexity, and convergence to the Pareto front [2011.13034, 2407.17466].

Continued advances in efficient preference elicitation, scalable representation, dynamic context adaptation, and theory-guided optimization are central to making multi-objective reward functions effective tools for complex, real-world RL tasks.

Source: https://www.emergentmind.com/topics/multi-objective-reward-function