---
title: Multi-Policy Multi-Objective Q-Learning
url: https://www.emergentmind.com/topics/multi-policy-multi-objective-q-learning-moql
type: topic
---

# Multi-Policy Multi-Objective Q-Learning

Searching arXiv for recent and foundational papers related to multi-policy multi-objective Q-learning and adjacent MORL methods.
Search query: "multi-policy multi-objective q-learning MORL coverage set Pareto Q-learning preference-conditioned Bellman"
Multi-Policy Multi-Objective Q-Learning (MOQL) denotes the part of multi-objective reinforcement learning concerned with learning not one compromise controller but a set of policies for a multi-objective Markov decision process (MOMDP) whose rewards are vector-valued and whose objectives are typically conflicting. In this setting, the standard target is not a single optimal policy but a family of non-dominated or preference-optimal policies, often organized as a Pareto set, a Pareto coverage set, or, under linear scalarization, a convex coverage set (CCS). In classical multi-policy MOQL, one typically tries to learn a set of Pareto-optimal policies, a convex coverage set / Pareto coverage set, or a collection of non-dominated value vectors associated with distinct policies; later work also includes preference-conditioned Q-functions, parallel scalarized deep agents, and adjacent bootstrapping or fairness-aware extensions [2308.09734][1908.08342][2606.18111].

## 1. Formal setting and optimality notions

A common formulation starts from a multi-objective optimization problem
\[
\max \; \left(R^1(\pi), R^2(\pi), \ldots, R^M(\pi)\right)
\]
subject to
\[
g^j(\pi) \le 0, \quad j=1,2,\ldots,J,
\]
where \(\pi \in \Pi\) denotes a policy parameterization and \(R^m(\pi)\) is the performance of policy \(\pi\) on objective \(m\) [2308.09734]. In MOMDP form, one description used in the literature is
\[
\langle S, A, \mathbb{P}_{ss'}, \vec{R}, \mu, \gamma \rangle,
\]
with state space \(S\), action space \(A\), transition distribution \(\mathbb{P}_{ss'}\), vector reward \(\vec{R}\), initial-state distribution \(\mu\), and discount factor \(\gamma \in [0,1)\) [2308.09734]. A preference is commonly written as a weight vector
\[
\vec w = [w^1, w^2, \ldots, w^M]^T,\quad \sum_{m=1}^M w^m = 1,
\]
and under linear scalarization the immediate scalar reward becomes
\[
\rho = \vec w \cdot \vec r_{t+1}.
\]
Under this assumption, the target is a CCS rather than the full Pareto coverage set [2308.09734].

Pareto reasoning remains central even when optimization is carried out through scalarization. For \(\mathbf{V}, \mathbf{V}' \in \mathbb{R}^N\), \(\mathbf{V}\) weakly Pareto-dominates \(\mathbf{V}'\) if \(\forall i,\; V_i \ge V'_i\), and strictly Pareto-dominates if \(V_i \ge V'_i\) for all \(i\) and \(\exists j,\; V_j > V'_j\) [2606.18111]. The CCS is then the subset of Pareto-front solutions that are optimal for some linear preference vector:
\[
\mathbf{V} \in \mathrm{CCS} \iff \exists\ \mathbf{w} \in \Omega \text{ s.t. } \mathbf{w}^T \mathbf{V} \geq \mathbf{w}^T \mathbf{V}',\ \forall\ \mathbf{V}' \in \mathcal{F}.
\]
This distinction is decisive: when scalarization is linear, MOQL methods that claim coverage are generally learning the supported portion of the frontier rather than all non-convex Pareto-optimal trade-offs [1908.08342][2606.18111].

## 2. Value representations and Bellman-style updates

Classical multi-policy MOQL often maintains either separate Q-functions for a set of fixed preferences, or a vector-valued Q-function and derives policies from it [2308.09734]. In the scalarized tabular case, the Bellman-style update used inside several broader multi-policy frameworks is standard Q-learning applied after linear scalarization:
\[
Q(s_t,a_t) \leftarrow Q(s_t,a_t) + \alpha \left[\rho + \gamma \max_{a'} Q(s_{t+1},a') - Q(s_t,a_t)\right].
\]
This update appears, for example, as the optimizer within policy-coverage methods that store and transfer multiple policies but do not themselves introduce a new vector-Q backup rule [2308.09734].

A different line uses vector-valued outputs directly. One deep multi-objective formulation defines per-objective returns
\[
R_{i,t} = \sum_{k=0}^{\infty} \gamma^k r_{i,t+k},
\qquad
Q_i(s,a) = \mathbb{E}[R_{i,t}\mid s_t=s, a_t=a],
\]
and groups them into
\[
\vec{Q}(s,a) = [Q_1(s,a), Q_2(s,a), \dots, Q_n(s,a)].
\]
In DQN form, the loss is written per objective as
\[
L^{Q}_{i,j}(\theta_{i,j}) =
\mathbb{E}_{(s,a,r_i,s')\sim U(M_i)}
\left[
\left(
r_i + \gamma \max_{a'} Q_i(s',a';\theta^{-}_{i,j}) - Q_i(s,a;\theta_{i,j})
\right)^2
\right],
\]
which yields a vector of per-objective action values rather than a single scalar critic [1704.06676].

Not all multi-objective Q-learning variants are multi-policy. A neighboring formulation for constrained and multi-objective MDPs rewrites the problem as a zero-sum Markov-Bandit game and learns a three-index object \(Q(s,a,o)\), where \(o\) is an objective or constraint index. Its goal is one stationary, potentially randomized, max-min optimal policy rather than a policy set, and it should therefore be classified as adjacent to MOQL rather than as a coverage-set method [1901.08978].

## 3. Coverage sets, preference conditioning, and frontier recovery

Coverage construction is the defining problem of multi-policy MOQL. In stationary settings, methods such as Optimistic Linear Support (OLS) and Threshold Lexicographic Ordering (TLO) typically assume a stationary environment and often perform an offline or exhaustive search over preference space to construct a coverage set [2308.09734]. This works for fixed dynamics, but it is limited to the geometry induced by the scalarization. With linear scalarization, only the CCS is recoverable; non-supported Pareto-optimal points remain outside the target set [2308.09734][1908.08342].

Preference-conditioned value learning attempts to compress the policy set into one parametric object. A generalized MORL algorithm for linear preferences defines a preference-conditioned vector-valued Q-function \(\mathbf Q(s,a,\boldsymbol\omega)\) and a generalized optimality filter
\[
(\mathcal H\mathbf Q)(s,\boldsymbol\omega)
=
\arg_Q \sup_{a\in\mathcal A,\ \boldsymbol\omega'\in\Omega}
\boldsymbol\omega^\top \mathbf Q(s,a,\boldsymbol\omega').
\]
The corresponding Bellman operator is
\[
(\mathcal T \mathbf Q)(s,a,\boldsymbol\omega)
=
\mathbf r(s,a)
+
\gamma
\mathbb E_{s'\sim\mathcal P(\cdot\mid s,a)}
(\mathcal H \mathbf Q)(s',\boldsymbol\omega).
\]
This operator is designed to learn a single parametric representation for optimal policies over the space of all possible preferences and to recover CCS-optimal behavior under linear scalarization [1908.08342].

A more recent line targets the full deterministic Pareto frontier rather than only supported solutions. One formulation defines a preference-conditioned Bellman operator motivated by Chebyshev scalarization, with preference space
\[
W = \{w \in \mathbb{R}_{\ge 0}^d \mid \|w\|_2 = 1,\ w_i \ge 0\},
\]
and scalarization
\[
O(w, V) := \min_{i:w_i>0} \frac{V_i}{w_i}.
\]
The resulting operator satisfies an enveloping property, upper-bounds the true Pareto frontier, converges monotonically to a coverage set, and supports extraction of deterministic approximately Pareto-optimal policies for any target preference [2606.26397]. This should not be confused with standard tabular Q-learning: it is model-based, assumes access to the transition model \(p(\cdot|s,a)\), and is closer to preference-conditioned value iteration than to Watkins-style asynchronous Q-learning [2606.26397].

## 4. Deep, modular, and lexicographic variants

Deep MORL introduced several architectures that are Q-based yet not canonical multi-policy MOQL. One framework based on deep Q-networks supports both single-policy and multi-policy strategies, as well as both linear and non-linear approaches to action selection, by training multiple scalarized agents sequentially or in parallel and approximating the Pareto front from the resulting solutions [1803.02965]. In this framework, the network outputs one group of action values per objective, and nonlinear Thresholded Lexicographic Ordering is implemented through
\[
TQ_{s,a,j} = \min(Q_{s,a,j}, T_j).
\]
Its multi-policy mechanism is therefore “multi-policy-by-parallel-scalarized-training,” not a set-valued Pareto backup rule [1803.02965].

A frequent misconception is that any deep multi-objective Q architecture with multiple value heads is a multi-policy MOQL method. A modular architecture with one DQN per objective and learned decision values is explicitly not a true multi-policy method: it learns decomposed objective-conditioned value functions and uses a state-dependent scalarization mechanism
\[
\vec{q}_{\sigma} = \vec{\mu} + \sum_{i=1}^{N} d_i p_i \, scale(\vec{q}_i),
\qquad
a_t = \arg\max_{a \in A} \vec{q}_{\sigma}(a),
\]
so the runtime behavior is a single policy produced by modular recombination, not a stored set of trade-off policies [1704.06676].

Thresholded lexicographic deep Q-learning for urban driving is similarly adjacent rather than canonical. It learns objective-specific Q-functions for a fixed priority order and filters admissible actions recursively:
\[
A_i(\mathbf{s}) :=
\left\{
a \in A_{i-1}(\mathbf{s})
\;\middle|\;
Q_i^*(\mathbf{s}, a)
\ge
\max_{a' \in A_{i-1}} Q_i^*(\mathbf{s}, a') + \tau_i
\ \text{or}\
a = \arg\max_{a' \in A_{i-1}} Q_i^*(\mathbf{s}, a')
\right\}.
\]
This produces one policy architecture induced by one lexicographic preference structure; it does not approximate a Pareto front or coverage set [1811.08586].

## 5. Non-stationarity, transfer, and policy bootstrapping

A major limitation of classical multi-policy MOQL is the assumption of stationarity. In non-stationary environments, precomputed support policies can become stale or misleading, and changes in dynamics may force costly re-optimization of the policy set [2308.09734]. A prominent response is Robust Policy Bootstrapping (RPB), which should be understood not as a new direct multi-policy Q-learning variant in the classic sense, but as a framework for online CCS evolution and transfer with scalarized Q-learning as the underlying learner [2308.09734].

RPB stores “steppingstone” policies represented as
\[
\langle \pi^k, \vec w^k, \beta^k \rangle,
\]
where \(\pi^k\) is the policy, \(\vec w^k\) its associated preference, and \(\beta^k\) a robustness metric [2308.09734]. When preferences change sufficiently, the current policy becomes a candidate for storage; when a new preference arrives, the algorithm retrieves the nearest stored policy
\[
p' = \arg\min_{p^k \in CCS} d(\vec w_t, \vec w^k),
\]
copies its parameters, and continues scalarized Q-learning from that initialization [2308.09734]. In the reported experiments, RPB significantly outperformed OLS, TLO, and scalarized Q-learning with random reinitialization in non-stationary environments while remaining comparable in stationary ones [2308.09734]. Its significance for MOQL lies in policy selection, storage, replacement, and transfer under drift, not in a new vector Bellman update.

A hierarchical extension replaces primitive-action relearning by a reusable skill library. A dual-phase intrinsically motivated method first learns generic skills and then bootstraps a policy coverage set under environmental shifts using robust fuzzy policy bootstrapping and hierarchical DDPG. This again targets multi-policy MORL under dynamic change, but not through set-valued Q-learning; it learns a coverage set of hierarchical policies built from a shared skill library [2308.09733].

Sample efficiency in continuous-control MORL has motivated another adjacent development: \(Q\)-Pensieve. This method stores a collection of Q-snapshots from past iterations in a Q replay buffer and reuses them during policy improvement. The core actor objective is
\[
\mathcal{L}_{\pi}(\theta;\omega)=
\mathbb{E}_{ s \sim \mu}
\bigg[
\mathbb{E}_{ a \sim \pi_{\theta}}
\Big[
\inf_{\omega' \in W(\omega), Q' \in \mathcal{Q}}
\big\{
\alpha \log \pi_{\theta}( a\mid s;\omega)-\omega^{\intercal} Q'( s, a;\omega')
\big\}
\Big]
\bigg].
\]
In the idealized soft policy-iteration setting, the authors prove convergence to a preference-dependent optimum under linear scalarization, while the practical algorithm is a deep actor-critic method rather than classical tabular MOQL [2212.03117].

## 6. Fairness, deterministic synthesis, and current limitations

Recent work extends MOQL beyond efficiency alone. In fair multi-policy MORL, the objective is to learn a set of Pareto-optimal policies that are also fair according to a welfare function such as the generalized Gini welfare function
\[
\mathrm{GGF}(\mathbf{u}) = \sum_{i \in N} \omega_i u_i^\uparrow.
\]
A key theoretical result is that for concave, piecewise-linear welfare functions, including GGF, there exists a fair optimal policy in the CCS:
\[
\exists \pi^* \in \mathrm{CCS}
\text{ such that }
W(V^{\pi^*}(s)) \geq W(V^\pi(s)),
\quad \forall \pi \in \Pi.
\]
On that basis, fairness-aware variants of Envelope-style multi-policy MOQL have been proposed, including state-augmented methods for non-stationary policies and a stochastic extension motivated by the fact that stochastic policies can improve fairness under concave welfare [2606.18111].

The literature also delineates what MOQL should not be expected to solve with local scalarized action selection alone. Under stochastic environments, non-linear utility, and the Scalarised Expected Return criterion, value-based MORL Q-learning can fail because the correct local action may depend on global policy-level trajectory statistics rather than on the information available at the current state [2211.08669]. In a small Space Traders MOMDP, a variant incorporating global statistics outperformed the baseline but still remained below 100 percent effectiveness in finding the desired SER-optimal policy at the end of training, whereas option learning was guaranteed to converge to the desired SER-optimal policy but did not scale [2211.08669]. This suggests that the limitations of MOQL are not only computational; in some regimes they are representational.

A further boundary case is online discovery of multiple objectives. An adaptive clustering method can discover goal-like objectives online and learn a separate action-value function for each objective in parallel using off-policy Q-learning, but this is not standard MOQL because the objectives are discovered goals rather than conflicting reward dimensions, and there is no scalarization or Pareto-front approximation [1705.06342]. The distinction is important: multi-Q learning over many tasks is not automatically multi-policy multi-objective Q-learning in the Pareto or coverage-set sense.

Taken together, these developments show that MOQL now spans several distinct but connected ideas: explicit policy-set learning for Pareto or convex coverage, preference-conditioned Q-functions for implicit coverage, online maintenance and transfer under non-stationarity, fairness-aware coverage within the CCS, and model-based operators that aim at deterministic Pareto-optimal policy synthesis beyond the convex frontier [1908.08342][2308.09734][2606.26397][2606.18111].

Source: https://www.emergentmind.com/topics/multi-policy-multi-objective-q-learning-moql