Multi-Policy Multi-Objective Q-Learning

• Multi-Policy Multi-Objective Q-Learning (MOQL) is a reinforcement learning approach that learns a set of policies to manage conflicting, vector-valued objectives.
• It employs methods such as vectorized Q-updates, preference-conditioned Bellman operators, and linear scalarization to approximate Pareto and convex coverage sets.
• Recent extensions combine fairness-aware metrics, hierarchical skill reuse, and robust policy bootstrapping to handle non-stationary environments and ensure equitable trade-offs.

Searching arXiv for recent and foundational papers related to multi-policy multi-objective Q-learning and adjacent MORL methods. Search query: "multi-policy multi-objective q-learning MORL coverage set Pareto Q-learning preference-conditioned Bellman" Multi-Policy Multi-Objective Q-Learning (MOQL) denotes the part of multi-objective reinforcement learning concerned with learning not one compromise controller but a set of policies for a multi-objective Markov decision process (MOMDP) whose rewards are vector-valued and whose objectives are typically conflicting. In this setting, the standard target is not a single optimal policy but a family of non-dominated or preference-optimal policies, often organized as a Pareto set, a Pareto coverage set, or, under linear scalarization, a convex coverage set (CCS). In classical multi-policy MOQL, one typically tries to learn a set of Pareto-optimal policies, a convex coverage set / Pareto coverage set, or a collection of non-dominated value vectors associated with distinct policies; later work also includes preference-conditioned Q-functions, parallel scalarized deep agents, and adjacent bootstrapping or fairness-aware extensions (Abdelfattah et al., 2023, Yang et al., 2019, Siddique et al., 16 Jun 2026).

1. Formal setting and optimality notions

A common formulation starts from a multi-objective optimization problem

$$\max \; \left(R^1(\pi), R^2(\pi), \ldots, R^M(\pi)\right)$$

subject to

$g^j(\pi) \le 0, \quad j=1,2,\ldots,J,$

where $\pi \in \Pi$ denotes a policy parameterization and $R^m(\pi)$ is the performance of policy $\pi$ on objective $m$ (Abdelfattah et al., 2023). In MOMDP form, one description used in the literature is

$$\langle S, A, \mathbb{P}_{ss'}, \vec{R}, \mu, \gamma \rangle,$$

with state space $S$, action space $A$, transition distribution $\mathbb{P}_{ss'}$, vector reward $g^j(\pi) \le 0, \quad j=1,2,\ldots,J,$0, initial-state distribution $g^j(\pi) \le 0, \quad j=1,2,\ldots,J,$1, and discount factor $g^j(\pi) \le 0, \quad j=1,2,\ldots,J,$2 (Abdelfattah et al., 2023). A preference is commonly written as a weight vector

$g^j(\pi) \le 0, \quad j=1,2,\ldots,J,$3

and under linear scalarization the immediate scalar reward becomes

$g^j(\pi) \le 0, \quad j=1,2,\ldots,J,$4

Under this assumption, the target is a CCS rather than the full Pareto coverage set (Abdelfattah et al., 2023).

Pareto reasoning remains central even when optimization is carried out through scalarization. For $g^j(\pi) \le 0, \quad j=1,2,\ldots,J,$5, $g^j(\pi) \le 0, \quad j=1,2,\ldots,J,$6 weakly Pareto-dominates $g^j(\pi) \le 0, \quad j=1,2,\ldots,J,$7 if $g^j(\pi) \le 0, \quad j=1,2,\ldots,J,$8, and strictly Pareto-dominates if $g^j(\pi) \le 0, \quad j=1,2,\ldots,J,$9 for all $\pi \in \Pi$0 and $\pi \in \Pi$1 (Siddique et al., 16 Jun 2026). The CCS is then the subset of Pareto-front solutions that are optimal for some linear preference vector: $\pi \in \Pi$2 This distinction is decisive: when scalarization is linear, MOQL methods that claim coverage are generally learning the supported portion of the frontier rather than all non-convex Pareto-optimal trade-offs (Yang et al., 2019, Siddique et al., 16 Jun 2026).

2. Value representations and Bellman-style updates

Classical multi-policy MOQL often maintains either separate Q-functions for a set of fixed preferences, or a vector-valued Q-function and derives policies from it (Abdelfattah et al., 2023). In the scalarized tabular case, the Bellman-style update used inside several broader multi-policy frameworks is standard Q-learning applied after linear scalarization: $\pi \in \Pi$3 This update appears, for example, as the optimizer within policy-coverage methods that store and transfer multiple policies but do not themselves introduce a new vector-Q backup rule (Abdelfattah et al., 2023).

A different line uses vector-valued outputs directly. One deep multi-objective formulation defines per-objective returns

$\pi \in \Pi$4

and groups them into

$\pi \in \Pi$5

In DQN form, the loss is written per objective as

$\pi \in \Pi$6

which yields a vector of per-objective action values rather than a single scalar critic (Tajmajer, 2017).

Not all multi-objective Q-learning variants are multi-policy. A neighboring formulation for constrained and multi-objective MDPs rewrites the problem as a zero-sum Markov-Bandit game and learns a three-index object $\pi \in \Pi$7, where $\pi \in \Pi$8 is an objective or constraint index. Its goal is one stationary, potentially randomized, max-min optimal policy rather than a policy set, and it should therefore be classified as adjacent to MOQL rather than as a coverage-set method (Gattami et al., 2019).

3. Coverage sets, preference conditioning, and frontier recovery

Coverage construction is the defining problem of multi-policy MOQL. In stationary settings, methods such as Optimistic Linear Support (OLS) and Threshold Lexicographic Ordering (TLO) typically assume a stationary environment and often perform an offline or exhaustive search over preference space to construct a coverage set (Abdelfattah et al., 2023). This works for fixed dynamics, but it is limited to the geometry induced by the scalarization. With linear scalarization, only the CCS is recoverable; non-supported Pareto-optimal points remain outside the target set (Abdelfattah et al., 2023, Yang et al., 2019).

Preference-conditioned value learning attempts to compress the policy set into one parametric object. A generalized MORL algorithm for linear preferences defines a preference-conditioned vector-valued Q-function $\pi \in \Pi$9 and a generalized optimality filter

$R^m(\pi)$0

The corresponding Bellman operator is

$R^m(\pi)$1

This operator is designed to learn a single parametric representation for optimal policies over the space of all possible preferences and to recover CCS-optimal behavior under linear scalarization (Yang et al., 2019).

A more recent line targets the full deterministic Pareto frontier rather than only supported solutions. One formulation defines a preference-conditioned Bellman operator motivated by Chebyshev scalarization, with preference space

$R^m(\pi)$2

and scalarization

$R^m(\pi)$3

The resulting operator satisfies an enveloping property, upper-bounds the true Pareto frontier, converges monotonically to a coverage set, and supports extraction of deterministic approximately Pareto-optimal policies for any target preference (Joshi et al., 24 Jun 2026). This should not be confused with standard tabular Q-learning: it is model-based, assumes access to the transition model $R^m(\pi)$4, and is closer to preference-conditioned value iteration than to Watkins-style asynchronous Q-learning (Joshi et al., 24 Jun 2026).

4. Deep, modular, and lexicographic variants

Deep MORL introduced several architectures that are Q-based yet not canonical multi-policy MOQL. One framework based on deep Q-networks supports both single-policy and multi-policy strategies, as well as both linear and non-linear approaches to action selection, by training multiple scalarized agents sequentially or in parallel and approximating the Pareto front from the resulting solutions (Nguyen et al., 2018). In this framework, the network outputs one group of action values per objective, and nonlinear Thresholded Lexicographic Ordering is implemented through

$R^m(\pi)$5

Its multi-policy mechanism is therefore “multi-policy-by-parallel-scalarized-training,” not a set-valued Pareto backup rule (Nguyen et al., 2018).

A frequent misconception is that any deep multi-objective Q architecture with multiple value heads is a multi-policy MOQL method. A modular architecture with one DQN per objective and learned decision values is explicitly not a true multi-policy method: it learns decomposed objective-conditioned value functions and uses a state-dependent scalarization mechanism

$R^m(\pi)$6

so the runtime behavior is a single policy produced by modular recombination, not a stored set of trade-off policies (Tajmajer, 2017).

Thresholded lexicographic deep Q-learning for urban driving is similarly adjacent rather than canonical. It learns objective-specific Q-functions for a fixed priority order and filters admissible actions recursively: $R^m(\pi)$7 This produces one policy architecture induced by one lexicographic preference structure; it does not approximate a Pareto front or coverage set (Li et al., 2018).

5. Non-stationarity, transfer, and policy bootstrapping

A major limitation of classical multi-policy MOQL is the assumption of stationarity. In non-stationary environments, precomputed support policies can become stale or misleading, and changes in dynamics may force costly re-optimization of the policy set (Abdelfattah et al., 2023). A prominent response is Robust Policy Bootstrapping (RPB), which should be understood not as a new direct multi-policy Q-learning variant in the classic sense, but as a framework for online CCS evolution and transfer with scalarized Q-learning as the underlying learner (Abdelfattah et al., 2023).

RPB stores “steppingstone” policies represented as

$R^m(\pi)$8

where $R^m(\pi)$9 is the policy, $\pi$0 its associated preference, and $\pi$1 a robustness metric (Abdelfattah et al., 2023). When preferences change sufficiently, the current policy becomes a candidate for storage; when a new preference arrives, the algorithm retrieves the nearest stored policy

$\pi$2

copies its parameters, and continues scalarized Q-learning from that initialization (Abdelfattah et al., 2023). In the reported experiments, RPB significantly outperformed OLS, TLO, and scalarized Q-learning with random reinitialization in non-stationary environments while remaining comparable in stationary ones (Abdelfattah et al., 2023). Its significance for MOQL lies in policy selection, storage, replacement, and transfer under drift, not in a new vector Bellman update.

A hierarchical extension replaces primitive-action relearning by a reusable skill library. A dual-phase intrinsically motivated method first learns generic skills and then bootstraps a policy coverage set under environmental shifts using robust fuzzy policy bootstrapping and hierarchical DDPG. This again targets multi-policy MORL under dynamic change, but not through set-valued Q-learning; it learns a coverage set of hierarchical policies built from a shared skill library (Abdelfattah et al., 2023).

Sample efficiency in continuous-control MORL has motivated another adjacent development: $\pi$3-Pensieve. This method stores a collection of Q-snapshots from past iterations in a Q replay buffer and reuses them during policy improvement. The core actor objective is

$\pi$4

In the idealized soft policy-iteration setting, the authors prove convergence to a preference-dependent optimum under linear scalarization, while the practical algorithm is a deep actor-critic method rather than classical tabular MOQL (Hung et al., 2022).

6. Fairness, deterministic synthesis, and current limitations

Recent work extends MOQL beyond efficiency alone. In fair multi-policy MORL, the objective is to learn a set of Pareto-optimal policies that are also fair according to a welfare function such as the generalized Gini welfare function

$\pi$5

A key theoretical result is that for concave, piecewise-linear welfare functions, including GGF, there exists a fair optimal policy in the CCS: $\pi$6 On that basis, fairness-aware variants of Envelope-style multi-policy MOQL have been proposed, including state-augmented methods for non-stationary policies and a stochastic extension motivated by the fact that stochastic policies can improve fairness under concave welfare (Siddique et al., 16 Jun 2026).

The literature also delineates what MOQL should not be expected to solve with local scalarized action selection alone. Under stochastic environments, non-linear utility, and the Scalarised Expected Return criterion, value-based MORL Q-learning can fail because the correct local action may depend on global policy-level trajectory statistics rather than on the information available at the current state (Ding, 2022). In a small Space Traders MOMDP, a variant incorporating global statistics outperformed the baseline but still remained below 100 percent effectiveness in finding the desired SER-optimal policy at the end of training, whereas option learning was guaranteed to converge to the desired SER-optimal policy but did not scale (Ding, 2022). This suggests that the limitations of MOQL are not only computational; in some regimes they are representational.

A further boundary case is online discovery of multiple objectives. An adaptive clustering method can discover goal-like objectives online and learn a separate action-value function for each objective in parallel using off-policy Q-learning, but this is not standard MOQL because the objectives are discovered goals rather than conflicting reward dimensions, and there is no scalarization or Pareto-front approximation (Karimpanal et al., 2017). The distinction is important: multi-Q learning over many tasks is not automatically multi-policy multi-objective Q-learning in the Pareto or coverage-set sense.

Taken together, these developments show that MOQL now spans several distinct but connected ideas: explicit policy-set learning for Pareto or convex coverage, preference-conditioned Q-functions for implicit coverage, online maintenance and transfer under non-stationarity, fairness-aware coverage within the CCS, and model-based operators that aim at deterministic Pareto-optimal policy synthesis beyond the convex frontier (Yang et al., 2019, Abdelfattah et al., 2023, Joshi et al., 24 Jun 2026, Siddique et al., 16 Jun 2026).