Multi-Objective Reinforcement Learning (MORL)

• Multi-Objective Reinforcement Learning is a framework that extends standard RL to optimize vector-valued rewards by identifying Pareto-optimal policies across conflicting objectives.
• MORL employs both linear and non-linear scalarization techniques to transform multi-dimensional reward problems into tractable single-objective subproblems for effective trade-off exploration.
• Applications span high-dimensional control to resource management, with challenges in scalability, robustness, and human preference integration driving ongoing research.

Multi-Objective Reinforcement Learning (MORL) generalizes standard reinforcement learning by addressing sequential decision processes where the agent must optimize a vector of possibly conflicting objectives rather than a single scalar reward. In MORL, there does not exist a single policy that simultaneously optimizes all objectives; instead, one seeks to identify a set of policies (typically corresponding to different trade-offs or user preferences) that reliably span the Pareto-optimal frontier of achievable outcome vectors.

1. Problem Formulation and Theoretical Foundations

The multi-objective RL problem is typically modeled as a Multi-Objective Markov Decision Process (MOMDP), a tuple $\langle S, A, T, \gamma, \mu, \mathbf{R} \rangle$, where $$\mathbf{R}: S \times A \times S \rightarrow \mathbb{R}^m$$ specifies an $m$-dimensional reward vector. The value associated with a policy $\pi$ is then a vector $\mathbf{V}^\pi \in \mathbb{R}^m$ given by

$$\mathbf{V}^\pi = \mathbb{E}_{\pi}\Big[\sum_{t=0}^\infty \gamma^t \mathbf{R}(s_t, a_t, s_{t+1}) \Big| s_0 \sim \mu \Big].$$

Since vector-valued outcomes are only partially ordered under standard Pareto dominance:

$$\mathbf{v} \succ_P \mathbf{v}' \iff \forall i,\, v_i \geq v'_i \text{ and } \exists j \text{ s.t. } v_j > v'_j,$$

classical optimality notions (as for single-objective RL) are insufficient; policies must be evaluated in terms of their dominance relations. The goal is to compute or approximate the Pareto front $$\mathcal{Y} = \{\mathbf{V}^\pi \mid \pi \text{ is Pareto-optimal}\}$$.

A policy can be made "optimal" with respect to a utility function $u:\mathbb{R}^m \to \mathbb{R}$ applied to vector rewards, but frequently the utility is partially or wholly unknown, non-linear, or dependent on changing user/stakeholder preferences (Vamplew et al., 15 Oct 2024). When $u$ is not prespecified, MORL methods typically aim to either optimize policies for a broad class of utility functions or construct a representative set of Pareto-optimal policies.

2. Scalarization, Optimization Targets, and the Role of User Preferences

Given the difficulty of working directly in the multi-objective space, most MORL algorithms transform the problem into a sequence of single-objective subproblems via a scalarization function $g(\cdot;\lambda)$, parameterized by a weight or preference vector $\lambda$. The two most prevalent families of scalarization functions are:

• Linear scalarization: $$g^{ws}(\mathbf{r}, \lambda) = \sum_{i=1}^m \lambda_i r_i$$ While efficient and compatible with classical RL methods, linear scalarization can fail to recover strictly Pareto-optimal policies in the presence of front non-convexity, especially in deterministic cases (Qiu et al., 24 Jul 2024).
• Non-linear scalarization: Chebyshev (Tchebycheff) scalarizations, e.g.,

$$\mathrm{TCH}_{\lambda}(\pi) = \max_{i} \lambda_i (V_i^* + \iota - V_i^\pi)$$

where $V_i^*$ is the maximal achievable value for objective $i$, and $\iota>0$ is a small constant, improve controllability and front coverage (Qiu et al., 24 Jul 2024).

The critical distinction between the two principal optimization targets is Scalarized Expected Return (SER) and Expected Scalarized Return (ESR):

• SER: $f(\mathbb{E}[\sum_t \gamma^t \mathbf{r}_t])$
• ESR: $\mathbb{E}[f(\sum_t \gamma^t \mathbf{r}_t)]$

Non-linear scalarization and the SER/ESR ordering may yield markedly different optimal policies (Felten et al., 2023, Ding, 2022). Control over exact preference–policy mapping is further affected by the stochasticity and non-convexity of the optimization surface.

Recent theoretical work demonstrates that reformulating non-smooth objective functions (e.g., Tchebycheff) into min-max-max problems facilitates provably efficient learning with sample complexity $\tilde{\mathcal{O}}(\varepsilon^{-2})$ (Qiu et al., 24 Jul 2024).

3. Algorithmic Approaches

A diverse set of learning paradigms have been developed within MORL:

Approach	Policy Representation	Preference Handling	Theoretical Guarantees
Meta-Learning (Chen et al., 2018)	Meta-policy; rapid adaptation	Distribution over preference vectors ($\omega$)	Adaptation optimality, empirical efficiency
Envelope Q-Learning (Yang et al., 2019, Zhou et al., 2020)	Q-network $Q(s,a,\omega)$, parametric on preference vector	Linear scalarization, envelope operator	Contraction property, Coverage Ratio
Decomposition-based (Felten et al., 2023, Liu et al., 12 Jan 2025)	Ensemble of scalarized policies / hypernetworks	Weight vector decomposition, scalarization selection	BeLLMan contraction, Rademacher complexity
Preference-driven / Conditioned (Basaklar et al., 2022, Mu et al., 18 Jul 2025)	Single universal net; input includes preference	Explicit conditioning, preference-driven loss	Contraction proofs, sample efficiency
Demonstration-guided (Lu et al., 5 Apr 2024)	Mixed policy; self-evolving demonstration set	Alignment via corner weight support	Sample complexity bounds
Policy Gradient / Variance-Reduced (Guidobene et al., 14 Aug 2025)	Single or parameterized stochastic policy	Nonlinear scalarization, batched/adaptive updates	Sample complexity ($\tilde{\mathcal{O}}(M^2)$), convergence
Logical specification (Nottingham et al., 2019)	Specification-encoded (GRU), action-value	Formal logic grammar, token embedding	Generalization, semantic interpolation

Meta-learning MORL: Approaches such as meta-learning reformulate the problem as one of "learning to learn"—they sample preference vectors from a distribution $p(\omega)$ across objectives and train a meta-policy that can be rapidly fine-tuned for any $\omega$. Distinct policy optimization is realized by scalarizing vector rewards with user-specified $f_\omega$ (e.g., weighted sum, Chebyshev), employing a PPO-like objective, and updating both via adaptation (inner loop for each $\omega$) and meta-objective aggregation (outer loop). Meta-policies show increased data efficiency and higher hypervolumes, particularly in high-dimensional control (Chen et al., 2018).

Envelope approaches and policy adaptation: Instead of maintaining a catalog of policies, envelope-based methods train a single Q-network $Q(s, a, \omega)$ covering the spectrum of trade-offs, with a BeLLMan operator that envelops over action and possible preference vectors. At test time, adaptation to new or hidden preferences is achieved by searching for the policy best aligned with observed returns, using techniques reminiscent of few-shot or inverse RL (Yang et al., 2019, Zhou et al., 2020).

Decomposition and modular frameworks: Following multi-objective optimization by decomposition (MOO/D), MORL can subdivide the problem into single-objective RL subproblems (through scalarization), train a (possibly cooperating) ensemble, and dynamically adapt weight vectors to densify coverage on the Pareto set. Taxonomies such as that in (Felten et al., 2023) clarify the design space and modular composition of MORL frameworks, distinguishing aspects like scalarization, weight adaptation, cooperation, selection, and buffer management.

Preference-driven, universal, and conditioned networks: Modern approaches train a single universal policy $Q(s, a, \omega)$ or actor-critic pair that is directly conditioned on (possibly continuous) user preferences (Basaklar et al., 2022). Cosine similarity and directional alignment can be incorporated into BeLLMan updates to ensure robust mapping between preferences and policy responses, with parallelization and HER variants used for sample efficiency.

Policy gradient and variance-reduced methods: MORL with policy gradients presents sample efficiency challenges due to variance scaling with the number of objectives, especially under nonlinear scalarization. Variance reduction via dual batching, projection steps, and direct cumulative reward estimation—as in MO-TSIVR-PG—improves convergence and scalability for large continuous state-action spaces (Guidobene et al., 14 Aug 2025).

Other notable strategies: Demonstration-guided training (for jump-starting the search with suboptimal policies and then self-evolving the guidance set (Lu et al., 5 Apr 2024)), interpretability via parametric-performance mappings (Xia et al., 4 Jun 2025), logical specification of objectives (Nottingham et al., 2019), and Lorenz/fairness-based dominance for high-dimensional societal objectives (Michailidis et al., 27 Nov 2024) continue to expand the MORL landscape.

4. Performance Evaluation, Scalability, and Practical Limits

Performance of MORL methods is assessed by:

• Hypervolume indicator (HV): Measures the volume in objective space dominated by the Pareto front relative to a reference (larger HV indicates better front convergence and diversity).
• Sparsity: Average separation between adjacent solutions on the Pareto front.
• Expected utility (EU): Averaged scalarized return across a sampled or application-specific preference distribution.
• Coverage ratio, adaptation error, and Sen Welfare: For specific domains (e.g., coverage of convex coverage set, fairness).

Recent work demonstrates that meta-learning, envelope, and decomposition-based MORL algorithms can consistently surpass prior approaches in hypervolume and expected utility metrics, particularly in high-dimensional continuous control (e.g., MO-HalfCheetah, MO-Ant) (Chen et al., 2018, Basaklar et al., 2022, Liu et al., 12 Jan 2025).

Nevertheless, even advanced algorithms exhibit scalability limits when applied to high-dimensional, real-world domains. In water management scenarios (Nile Basin, four reservoirs, four objectives), specialized direct policy search methods substantially outperform generic MORL techniques on both hypervolume and solution diversity metrics (Osika et al., 2 May 2025). In transport planning with many objectives (~10 for socioeconomic groups), fairness-constrained algorithms such as Lorenz Conditioned Networks scale more robustly and yield manageable solution set sizes (Michailidis et al., 27 Nov 2024).

5. Incorporation of Human Preferences and Expressive Objective Specifications

Much research transitioning from rigid reward engineering toward user- or stakeholder-driven specification:

• Human preference-based MORL: Here, a reward model is learned from pairwise preferences between trajectory segments under different weightings, then optimized to derive policies across the Pareto front; this framework is shown to match or surpass direct ground-truth optimization in complex tasks (Mu et al., 18 Jul 2025).
• Logical and formal specification: Logical languages (with quantitative semantics) afford greater expressiveness than linear scalarization, allowing conjunctions, disjunctions, and constraints directly mapping to interpretable policies (Nottingham et al., 2019).
• Aggregation and alignment frameworks: Social welfare functions, e.g., Generalized Gini or Nash Social Welfare, facilitate pluralistically-aligned policy sets—essential for AI systems mediating between values and stakeholders (Vamplew et al., 15 Oct 2024).

6. Key Challenges and Active Research Directions

Despite substantial progress, persistent challenges remain:

• Scalability and solution diversity: Large policy sets are difficult to maintain and deploy as the number and dimension of objectives increase. Compression through fairness subset selection (e.g., Lorenz dominance) and hypernetwork approaches ameliorates but do not fully solve this.
• Robustness in stochastic environments: Standard value-based methods may fail to find SER-optimal policies in environments with stochastic transitions, as local learning does not capture global trade-off statistics; options and global statistic-based methods are partially successful but non-scalable for large domains (Ding, 2022).
• Sample efficiency: While variance-reduction in policy gradients provides improved theoretical and practical sample complexities, further reductions—particularly for exploration in high-dimensional spaces—are still sought (Guidobene et al., 14 Aug 2025).
• Interpretability: Approaches that maintain an explicit mapping between parameter and performance spaces facilitate actionable insights but require additional structure and may complicate the optimization landscape (Xia et al., 4 Jun 2025).
• Preference and specification generalization: Bridging logical, natural language, or high-level specification methods with parametrized policy learning remains a highly active line of enquiry.

Future prospects emphasize automatic algorithm configuration (e.g., AutoMORL), preference elicitation, scalable front compression, more advanced cooperation across policies, and generalization to settings with many objectives or complex value structures.

7. Representative Mathematical Formulations

Throughout these methodologies, several crucial formulations recur:

• Multi-objective BeLLMan update:

$$q(s, a, \lambda) \leftarrow q(s, a, \lambda) + \alpha \left( \mathbf{r} + \gamma ( \arg_{q} \max_{a', \lambda'} g^{ws} (q(s', a', \lambda'), \lambda) ) - q(s, a, \lambda) \right).$$

• Linear and non-linear scalarization functions:

$$g^{ws}(\mathbf{r}, \lambda) = \sum_{i=1}^m \lambda_i r_i,\quad \mathrm{TCH}_\lambda(\pi) = \max_i [\lambda_i (V_i^* + \iota - V_i^\pi)].$$

• Expected utility metric:

$$J(\theta) = \mathbb{E}_{\alpha \sim D(\alpha)} [ \alpha^\top J(\theta, \alpha) ].$$

• Policy gradient for scalarized return under non-linear $f$:

$$\nabla_\theta f(\mathbf{J}(\theta)) = \mathbb{E}[ \nabla_\theta \log \pi_\theta(a_t|s_t) \sum_m \frac{\partial f}{\partial J_m} r_m(s_t, a_t) ].$$

The mathematical and algorithmic diversity evident in MORL reflects fundamental trade-offs in expressiveness, scalability, controllability, and alignment. Current research continues to advance theoretically principled methods that are practical for large-scale, real-world settings, specifically targeting efficient front construction, dynamic user preference response, and explicit consideration of fairness, interpretability, and sample efficiency.