Convex Coverage Set in Multi-Objective RL

• Convex Coverage Set (CCS) is the subset of Pareto-optimal return vectors that are optimal under at least one linear scalarization, defining the efficient frontier in multi-objective RL.
• It reinterprets domain randomization by treating each domain as a distinct objective, enabling the formulation of uncertainty-aware policies within a Pseudo-MOMDP framework.
• Algorithmic approximations like cMDSAC, eMDSAC, and uMDSAC leverage CCS geometry to train universal policies that enhance sim-to-real transfer across varying domain uncertainties.

Searching arXiv for the main CCS paper and foundational MORL papers. arXiv search query: "Domains as Objectives: Domain-Uncertainty-Aware Policy Optimization through Explicit Multi-Domain Convex Coverage Set Learning" A Convex Coverage Set (CCS) is, in multi-objective reinforcement learning, the subset of the Pareto Coverage Set consisting of all return vectors that are optimal for at least one linear scalarization weight vector. In the formulation used for domain-uncertainty-aware control, CCS is the solution concept that links multi-objective optimization, domain randomization, and uncertainty-aware universal policies: each domain is treated as an objective, and a policy family conditioned on a preference or uncertainty distribution is trained to approximate the CCS over domains (Ilboudo et al., 2024).

1. Formal definition in multi-objective reinforcement learning

The starting point is a multi-objective MDP in which rewards are vectors and policies $\pi$ induce vector returns

$$V^\pi \;=\; \mathbb{E}_\pi\Big[\sum_t \gamma^t\, r(s_t,a_t)\Big] \in \mathbb{R}^n .$$

A value vector $V'$ Pareto-dominates $V$ if

$V'_i > V_i \quad \forall i \in \{1,\dots,n\}.$

The Pareto Coverage Set (PCS) is the set of non-dominated return vectors,

$$\mathrm{PCS} = \left\{ V \;\middle|\; \nexists V' \text{ such that } V'\succ V \right\}.$$

Under a linear utility function

$$f_{\boldsymbol{\omega}}(r) = \boldsymbol{\omega}^\top r = \sum_{i=1}^n \omega_i r_i , \quad \boldsymbol{\omega} \in \Omega ,$$

the Convex Coverage Set (CCS) is defined as

$$\mathrm{CCS} = \bigl\{ V \in \mathrm{PCS} \;\big|\; \exists \boldsymbol{\omega}\in\Omega \;\text{s.t.}\; \boldsymbol{\omega}^{\top}V \geq \boldsymbol{\omega}^{\top}V' \ \forall V' \in \mathrm{PCS} \bigr\}.$$

For each $\boldsymbol{\omega}$, the optimal scalarized value is obtained by some $V \in \mathrm{CCS}$,

$$V^\pi \;=\; \mathbb{E}_\pi\Big[\sum_t \gamma^t\, r(s_t,a_t)\Big] \in \mathbb{R}^n .$$0

The resulting function $$V^\pi \;=\; \mathbb{E}_\pi\Big[\sum_t \gamma^t\, r(s_t,a_t)\Big] \in \mathbb{R}^n .$$1 is piecewise-linear and convex in $$V^\pi \;=\; \mathbb{E}_\pi\Big[\sum_t \gamma^t\, r(s_t,a_t)\Big] \in \mathbb{R}^n .$$2, and the CCS is essentially the set of exposed points of the convex hull of the Pareto front under linear scalarization (Ilboudo et al., 2024).

A recurrent misconception is to read CCS as a requirement for pointwise optimality in every objective. The linear-scalarization definition is weaker and more specific: a point belongs to the CCS if it is optimal for some weight vector. The paper makes this explicit with a two-domain example: $$V^\pi \;=\; \mathbb{E}_\pi\Big[\sum_t \gamma^t\, r(s_t,a_t)\Big] \in \mathbb{R}^n .$$3 Here,

$$V^\pi \;=\; \mathbb{E}_\pi\Big[\sum_t \gamma^t\, r(s_t,a_t)\Big] \in \mathbb{R}^n .$$4

so $$V^\pi \;=\; \mathbb{E}_\pi\Big[\sum_t \gamma^t\, r(s_t,a_t)\Big] \in \mathbb{R}^n .$$5 is CCS-optimal under $$V^\pi \;=\; \mathbb{E}_\pi\Big[\sum_t \gamma^t\, r(s_t,a_t)\Big] \in \mathbb{R}^n .$$6, even though $$V^\pi \;=\; \mathbb{E}_\pi\Big[\sum_t \gamma^t\, r(s_t,a_t)\Big] \in \mathbb{R}^n .$$7 is better in domain $$V^\pi \;=\; \mathbb{E}_\pi\Big[\sum_t \gamma^t\, r(s_t,a_t)\Big] \in \mathbb{R}^n .$$8. This suggests that CCS-based evaluation is inherently distributional over objectives or domains rather than per-objective maximality.

2. Domains as objectives and the PMOMDP construction

The central reinterpretation is to treat domains as objectives. The uncertainty set of domain parameters is

$$V^\pi \;=\; \mathbb{E}_\pi\Big[\sum_t \gamma^t\, r(s_t,a_t)\Big] \in \mathbb{R}^n .$$9

where each $V'$0 defines a distinct MDP with shared state and action spaces $V'$1 but domain-specific dynamics and rewards: $V'$2 For a policy $V'$3 and environment $V'$4, the domain-specific return is

$V'$5

and these are collected into the vector

$V'$6

A preference or uncertainty vector

$V'$7

is interpreted as a discrete distribution over target domains,

$V'$8

The induced utility is then

$V'$9

Under this identification, the domain-randomization objective

$V$0

becomes exactly a linear-scalarization MORL problem. The paper therefore states that the correct solution concept for domain randomization under domain uncertainty is the CCS for linear scalarization (Ilboudo et al., 2024).

To formalize this correspondence, the paper introduces the Pseudo-Multi-Objective MDP (PMOMDP): $V$1 Here, each domain is an objective, $V$2 is the space of preference distributions over domains, and $V$3 is the family of linear utility functions. In policy space, the CCS is correspondingly the set of policies in the PCS that are optimal for some $V$4, and for a specific $V$5 the optimal CCS policy satisfies

$V$6

A plausible implication is that PMOMDP does not introduce a new utility principle; it recasts an existing expectation-over-domains objective into an explicit coverage-set geometry in policy and value space.

3. Algorithmic approximations to the CCS

The paper adapts three MORL CCS algorithms into the multi-domain setting, implemented with Soft Actor-Critic as cMDSAC, eMDSAC, and uMDSAC, with additional SIRSA-based variants (Ilboudo et al., 2024).

Method	Core mechanism	CCS relation
cMDRL / cMDSAC	Universal policy and critic conditioned on $V$7	Implicitly approximates the CCS
eMDRL / eMDSAC	Envelope-style optimality filter over weights	Explicitly enforces CCS consistency
uMDRL / uMDSAC	Utopia-based alignment with domain-specific critics	Implicitly approximates the CCS via utopian targets

In conditioned MDRL, the policy and critic are universal functions,

$V$8

so that one parametric model represents the entire family of policies indexed by uncertainty distributions. For each fixed $V$9, this is equivalent to domain randomization with that uncertainty. The Bellman operator for domain $V'_i > V_i \quad \forall i \in \{1,\dots,n\}.$0 is

$V'_i > V_i \quad \forall i \in \{1,\dots,n\}.$1

This construction implicitly approximates the CCS because each slice $V'_i > V_i \quad \forall i \in \{1,\dots,n\}.$2 is meant to approximate $V'_i > V_i \quad \forall i \in \{1,\dots,n\}.$3.

In envelope MDRL, the key device is an optimality filter. For a target $V'_i > V_i \quad \forall i \in \{1,\dots,n\}.$4 and next state $V'_i > V_i \quad \forall i \in \{1,\dots,n\}.$5, an auxiliary weight $V'_i > V_i \quad \forall i \in \{1,\dots,n\}.$6 is chosen to maximize the scalarized action value: $V'_i > V_i \quad \forall i \in \{1,\dots,n\}.$7 The Bellman backup then uses this filtered CCS element,

$V'_i > V_i \quad \forall i \in \{1,\dots,n\}.$8

The purpose is to correct misalignment between the policy indexed by $V'_i > V_i \quad \forall i \in \{1,\dots,n\}.$9 and the value geometry of the CCS.

In utopia-based MDRL, the alignment target is a utopian point constructed from domain-specialized policies. For each domain $$\mathrm{PCS} = \left\{ V \;\middle|\; \nexists V' \text{ such that } V'\succ V \right\}.$$0, the policy $$\mathrm{PCS} = \left\{ V \;\middle|\; \nexists V' \text{ such that } V'\succ V \right\}.$$1 focuses only on domain $$\mathrm{PCS} = \left\{ V \;\middle|\; \nexists V' \text{ such that } V'\succ V \right\}.$$2, and the utopia point is

$$\mathrm{PCS} = \left\{ V \;\middle|\; \nexists V' \text{ such that } V'\succ V \right\}.$$3

The paper shows that the envelope filter can be viewed as a reachable utopian target, and proposes a simpler utopia-based objective in which the critic depends only on $$\mathrm{PCS} = \left\{ V \;\middle|\; \nexists V' \text{ such that } V'\succ V \right\}.$$4 while the policy remains universal: $$\mathrm{PCS} = \left\{ V \;\middle|\; \nexists V' \text{ such that } V'\succ V \right\}.$$5 The corresponding optimal policy form is

$$\mathrm{PCS} = \left\{ V \;\middle|\; \nexists V' \text{ such that } V'\succ V \right\}.$$6

These three constructions differ primarily in how directly they encode CCS geometry: cMDRL conditions on preferences, eMDRL shares information across preferences through an explicit filter, and uMDRL uses utopian alignment to simplify and stabilize CCS learning.

4. Universal policies, uncertainty conditioning, and online adaptation

A central operational consequence of CCS learning is the use of uncertainty-aware universal policies. These policies take as input both the current state $$\mathrm{PCS} = \left\{ V \;\middle|\; \nexists V' \text{ such that } V'\succ V \right\}.$$7 and a representation of domain uncertainty $$\mathrm{PCS} = \left\{ V \;\middle|\; \nexists V' \text{ such that } V'\succ V \right\}.$$8, such as the mean and standard deviation of a distribution over $$\mathrm{PCS} = \left\{ V \;\middle|\; \nexists V' \text{ such that } V'\succ V \right\}.$$9, and output an action distribution $$f_{\boldsymbol{\omega}}(r) = \boldsymbol{\omega}^\top r = \sum_{i=1}^n \omega_i r_i , \quad \boldsymbol{\omega} \in \Omega ,$$0 (Ilboudo et al., 2024).

In the PMOMDP view, each CCS element corresponds to a policy $$f_{\boldsymbol{\omega}}(r) = \boldsymbol{\omega}^\top r = \sum_{i=1}^n \omega_i r_i , \quad \boldsymbol{\omega} \in \Omega ,$$1 that is optimal for some $$f_{\boldsymbol{\omega}}(r) = \boldsymbol{\omega}^\top r = \sum_{i=1}^n \omega_i r_i , \quad \boldsymbol{\omega} \in \Omega ,$$2. Rather than store a finite set $$f_{\boldsymbol{\omega}}(r) = \boldsymbol{\omega}^\top r = \sum_{i=1}^n \omega_i r_i , \quad \boldsymbol{\omega} \in \Omega ,$$3, the paper learns a single parametric function $$f_{\boldsymbol{\omega}}(r) = \boldsymbol{\omega}^\top r = \sum_{i=1}^n \omega_i r_i , \quad \boldsymbol{\omega} \in \Omega ,$$4 intended to interpolate the CCS over the continuum of $$f_{\boldsymbol{\omega}}(r) = \boldsymbol{\omega}^\top r = \sum_{i=1}^n \omega_i r_i , \quad \boldsymbol{\omega} \in \Omega ,$$5. This is directly analogous to preference-conditioned policies in MORL, but the conditioning variable is interpreted as a distribution over domains rather than a distribution over abstract objectives.

At deployment, the controller can move along the CCS without retraining. The described procedure is:

1. Start with a prior $$f_{\boldsymbol{\omega}}(r) = \boldsymbol{\omega}^\top r = \sum_{i=1}^n \omega_i r_i , \quad \boldsymbol{\omega} \in \Omega ,$$6 over domain parameters.
2. Run an online system identifier (OSI) that updates $$f_{\boldsymbol{\omega}}(r) = \boldsymbol{\omega}^\top r = \sum_{i=1}^n \omega_i r_i , \quad \boldsymbol{\omega} \in \Omega ,$$7 based on observed $$f_{\boldsymbol{\omega}}(r) = \boldsymbol{\omega}^\top r = \sum_{i=1}^n \omega_i r_i , \quad \boldsymbol{\omega} \in \Omega ,$$8 using a variational objective.
3. Feed $$f_{\boldsymbol{\omega}}(r) = \boldsymbol{\omega}^\top r = \sum_{i=1}^n \omega_i r_i , \quad \boldsymbol{\omega} \in \Omega ,$$9 into the universal policy $$\mathrm{CCS} = \bigl\{ V \in \mathrm{PCS} \;\big|\; \exists \boldsymbol{\omega}\in\Omega \;\text{s.t.}\; \boldsymbol{\omega}^{\top}V \geq \boldsymbol{\omega}^{\top}V' \ \forall V' \in \mathrm{PCS} \bigr\}.$$0.

This implements uncertainty-aware adaptation by changing the conditioning variable rather than re-optimizing the policy online. The significance of the CCS perspective is that the policy family is trained to be optimal not for one fixed prior but for a space of admissible uncertainty distributions. This suggests a direct geometric interpretation of adaptation: posterior updates over domains induce motion over the CCS surface.

For continuous domain parameters $$\mathrm{CCS} = \bigl\{ V \in \mathrm{PCS} \;\big|\; \exists \boldsymbol{\omega}\in\Omega \;\text{s.t.}\; \boldsymbol{\omega}^{\top}V \geq \boldsymbol{\omega}^{\top}V' \ \forall V' \in \mathrm{PCS} \bigr\}.$$1, the same linear-utility principle is retained: $$\mathrm{CCS} = \bigl\{ V \in \mathrm{PCS} \;\big|\; \exists \boldsymbol{\omega}\in\Omega \;\text{s.t.}\; \boldsymbol{\omega}^{\top}V \geq \boldsymbol{\omega}^{\top}V' \ \forall V' \in \mathrm{PCS} \bigr\}.$$2 Here $$\mathrm{CCS} = \bigl\{ V \in \mathrm{PCS} \;\big|\; \exists \boldsymbol{\omega}\in\Omega \;\text{s.t.}\; \boldsymbol{\omega}^{\top}V \geq \boldsymbol{\omega}^{\top}V' \ \forall V' \in \mathrm{PCS} \bigr\}.$$3 remains a distribution over the uncertainty set, and in practice is parameterized via a mean and standard deviation for continuous uniform distributions.

5. Theoretical status, evaluation protocol, and empirical behavior

The theoretical claims are primarily structural rather than convergence-theoretic. The paper states three main points (Ilboudo et al., 2024).

First, there is an equivalence between domain randomization and CCS learning under linear utility. The expected-over-domains objective is exactly the scalarization $$\mathrm{CCS} = \bigl\{ V \in \mathrm{PCS} \;\big|\; \exists \boldsymbol{\omega}\in\Omega \;\text{s.t.}\; \boldsymbol{\omega}^{\top}V \geq \boldsymbol{\omega}^{\top}V' \ \forall V' \in \mathrm{PCS} \bigr\}.$$4, so learning an uncertainty-aware policy for any possible posterior over domains is equivalent to learning the CCS in the PMOMDP.

Second, the proposed Bellman operators are CCS-aligned. In cMDRL, policy evaluation is standard for each fixed $$\mathrm{CCS} = \bigl\{ V \in \mathrm{PCS} \;\big|\; \exists \boldsymbol{\omega}\in\Omega \;\text{s.t.}\; \boldsymbol{\omega}^{\top}V \geq \boldsymbol{\omega}^{\top}V' \ \forall V' \in \mathrm{PCS} \bigr\}.$$5 and domain. In eMDRL, the optimality filter explicitly enforces CCS consistency by backing up through the maximizing CCS element. In uMDRL, attraction toward a utopian point is used to align learning with the upper surface of the value set. The paper bases these constructions on SAC, but does not prove a new formal convergence theorem for the CCS algorithms.

Third, evaluation is performed with a CCS score per uncertainty $$\mathrm{CCS} = \bigl\{ V \in \mathrm{PCS} \;\big|\; \exists \boldsymbol{\omega}\in\Omega \;\text{s.t.}\; \boldsymbol{\omega}^{\top}V \geq \boldsymbol{\omega}^{\top}V' \ \forall V' \in \mathrm{PCS} \bigr\}.$$6. The universal policy is evaluated across sampled domain parameters, specifically sigma-points from an unscented transform of $$\mathrm{CCS} = \bigl\{ V \in \mathrm{PCS} \;\big|\; \exists \boldsymbol{\omega}\in\Omega \;\text{s.t.}\; \boldsymbol{\omega}^{\top}V \geq \boldsymbol{\omega}^{\top}V' \ \forall V' \in \mathrm{PCS} \bigr\}.$$7, and the domain returns are aggregated with the linear utility $$\mathrm{CCS} = \bigl\{ V \in \mathrm{PCS} \;\big|\; \exists \boldsymbol{\omega}\in\Omega \;\text{s.t.}\; \boldsymbol{\omega}^{\top}V \geq \boldsymbol{\omega}^{\top}V' \ \forall V' \in \mathrm{PCS} \bigr\}.$$8. Higher CCS scores indicate better approximation of the CCS point corresponding to that uncertainty distribution.

The reported empirical pattern is consistent across several settings. In 2-D domain randomization tasks—Hopper, Walker2D, and D’Claw—uMDSAC-v2 typically achieves the highest CCS scores across most test $$\mathrm{CCS} = \bigl\{ V \in \mathrm{PCS} \;\big|\; \exists \boldsymbol{\omega}\in\Omega \;\text{s.t.}\; \boldsymbol{\omega}^{\top}V \geq \boldsymbol{\omega}^{\top}V' \ \forall V' \in \mathrm{PCS} \bigr\}.$$9. uMDSAC-v1 and uMD-SIRSA are close, while eMDSAC and cMDSAC usually outperform DRSAC and SIRSA. In high-dimensional domain randomization, no single method dominates everywhere, but uMDSAC-v2 is consistently among the top three across tasks; eMDSAC is best on high-dimensional D’Claw, cMDSAC on high-dimensional Hopper, and uMDSAC-v1 on high-dimensional Walker2D.

The sim-to-real D’Claw experiments further separate fixed-uncertainty deployment from adaptive deployment with OSI. In 2-D domain randomization with fixed $\boldsymbol{\omega}$0, DRSAC is best, but uMDSAC-v1 and cMDSAC are very close. In 2-D domain randomization with OSI, cMDSAC becomes best and uMDSAC-v1 is third. In 22-D domain randomization with fixed $\boldsymbol{\omega}$1, cMDSAC is best, followed by eMDSAC and uMDSAC-v1. In 22-D domain randomization with OSI, uMDSAC-v1 is best and uMDSAC-v2 is second. The paper interprets these results as evidence that CCS-aware universal policies improve sim-to-real transfer, especially under high-dimensional uncertainty and when combined with OSI.

A plausible implication is that CCS-based training is most valuable when the deployment-time posterior over domains is expected to vary meaningfully from the training prior, because the learned object is a policy family indexed by uncertainty rather than a single policy optimized for a single distribution.

6. Terminological ambiguity and unrelated uses of the acronym

The acronym CCS is not unique across machine learning. In the paper on domain-uncertainty-aware policy optimization, CCS means Convex Coverage Set in the standard MORL sense. By contrast, in "Coverage-centric Coreset Selection for High Pruning Rates" (Zheng et al., 2022), CCS denotes Coverage-centric Coreset Selection, a one-shot coreset selection method for supervised learning rather than a set-valued solution concept in multi-objective optimization.

The distinction is substantive. In the multi-objective setting, CCS refers to a subset of Pareto-optimal return vectors or policies sufficient for all linear scalarizations: $\boldsymbol{\omega}$2 In the coreset-selection setting, CCS is a sampling algorithm that trades off importance and coverage of a data distribution in order to avoid catastrophic accuracy collapse at high pruning rates. The latter paper explicitly states that its CCS does not mean convex coverage set (Zheng et al., 2022).

This ambiguity matters because both usages involve the word coverage but operate on entirely different objects. Convex Coverage Sets concern the geometry of attainable return vectors under linear scalarization in MORL and multi-domain RL. Coverage-centric Coreset Selection concerns representative subset selection in supervised learning via a distribution-cover perspective. The shared acronym does not indicate shared theory.

Within the multi-domain RL formulation, the CCS notion remains precisely the MORL one: a compact solution set for all linear preferences over objectives, where the objectives have been reinterpreted as domains.