Pareto Conditioned Networks (PCN)

• Pareto Conditioned Networks (PCN) are neural architectures that condition on target return or preference vectors to capture the entire spectrum of Pareto-efficient solutions in multi-objective reinforcement learning.
• PCN fuses state and goal embeddings through efficient architectures, enabling recovery of non-dominated policies without requiring a separate network per Pareto point.
• The approach reframes multi-objective optimization into a supervised learning task, ensuring stable training and scalability in both discrete and continuous action spaces.

Pareto Conditioned Networks (PCN) are a class of neural architectures and associated training techniques designed to recover the full set of Pareto-efficient solutions in multi-objective reinforcement learning (MORL) and control. By leveraging a conditioning mechanism—wherein the policy or value network is conditioned directly on a desired return vector or preference vector—PCN enables a single network to represent the entire continuum of non-dominated solutions, scaling efficiently to high-dimensional, multi-objective problems without requiring one policy per Pareto point or resorting to convexity assumptions. This approach transforms multi-objective optimization into a supervised learning problem, supporting stable training, coverage of complex Pareto fronts (including concave/disconnected shapes), and efficient deployment in both discrete and continuous action settings (Reymond et al., 2022, Reymond et al., 2022, Chen et al., 2 Oct 2025).

1. Multi-Objective Problem Formulation

Consider a multi-objective Markov decision process (MOMDP) defined by a tuple $(S, A, T, r, \gamma)$ where $S$ is the state space, $A$ the action space (possibly continuous), $T$ the transition kernel, and $r: S \times A \times S \to \mathbb{R}^m$ the $m$-dimensional reward function. The discounted return of a trajectory $\tau$ is $$G(\tau) = \sum_{t=0}^T \gamma^t r_t \in \mathbb{R}^m$$. Returns are compared using Pareto dominance: $V \succ_P V'$ iff $\forall i, V_i \geq V'_i$ and $\exists i, V_i > V'_i$.

The solution concept is the Pareto front $\mathcal{F} = \{ V : \nexists V' \succ_P V \}$. Involvement of conflicting objectives and non-convex/complex front shapes renders traditional scalarization-based approaches suboptimal and motivates the use of a flexible, goal-conditioned framework like PCN (Reymond et al., 2022, Chen et al., 2 Oct 2025).

2. Neural Architecture and Conditioning Mechanism

PCN employs neural architectures that condition their outputs on an auxiliary vector encoding a desired target—either a return vector $g \in \mathbb{R}^m$, a preference vector $w \in \Delta^{m-1}$ (the $(m-1)$-simplex), or both a goal and a time horizon $h$.

• Input Processing: The state $s$ is embedded via an MLP or convolutional module (e.g., $64$-dimensional features). The conditioning vector (goal $g$, preference $w$, or $(h, g)$ pair) is embedded via a parallel MLP.
• Fusion: The two embeddings (state and goal/preference) are fused, typically via elementwise (Hadamard) product.
• Output Layer: For discrete actions, the fused embedding is processed through additional MLP layers to output action logits (classification over $A$). For continuous actions, a regression head outputs real-valued action vectors, possibly followed by $\tanh$ and affine rescaling to fulfill action constraints (Reymond et al., 2022, Reymond et al., 2022).

Advanced variants allow separate processing of high-dimensional state components (e.g., epidemiological compartments, previous actions, contextual flags), as well as flexible goal or preference encodings (Chen et al., 2 Oct 2025).

3. Training Procedures and Loss Functions

PCN reframes multi-objective optimization as a supervised learning problem by storing transitions annotated with the total (discounted) return achieved from each state-action pair:

• Replay Buffer Construction: For each transition $(s_t, a_t)$ with observed return $G_t$ and horizon $h_t$, store $(s_t, h_t, G_t, a_t)$.
• Supervised Loss (Discrete Actions): Cross-entropy classification loss over replayed examples:

$$L(\theta) = -\mathbb{E}_{(s, h, G, a) \sim D} \left[ \log \pi_\theta(a \mid s, h, G) \right]$$

• Supervised Loss (Continuous Actions): Mean-squared error regression between predicted and actual action:

$$\mathcal{L}(\theta) = \mathbb{E}_{(s, h, \hat{R}, a) \sim D} \left\| \pi_\theta(s, h, \hat{R}) - a \right\|^2_2$$

• TD Learning Variant (Preference Conditioning): For Q-function based approaches (e.g., multi-objective DQN), use a weighted regression loss on the scalarized Q-values:

$$L(\theta) = \mathbb{E}_{(s, a, \mathbf{r}, s') \sim \mathcal{D}, w} \left[ \left( y - w^\top \mathbf{Q}(s, a \mid w; \theta) \right)^2 \right]$$

where $$y = w^\top \mathbf{r} + \gamma \max_{a'} w^\top \mathbf{Q}(s', a'; \theta^-)$$ and $w$ is sampled from $\mathrm{Uniform}(\Delta^{m-1})$ (Chen et al., 2 Oct 2025).

PCN does not require Bellman backups or explicit scalarization unless using the Q-learning incarnation. Training is stable (avoiding moving targets), efficient for large $m$, and directly leverages shared experience across the Pareto spectrum (Reymond et al., 2022, Reymond et al., 2022).

4. Inference, Policy Extraction, and Pareto Front Approximation

After training, the empirical Pareto front $\hat{\mathcal{F}}$ is estimated by gathering all achieved returns $G(\tau)$ across policies induced by conditioning vectors (either sampled or exhaustively spanning the relevant space). Dominated points are pruned to yield the non-dominated set.

• Policy Selection: At inference, for any target $g \in \hat{\mathcal{F}}$, the network is conditioned on $(s, h, g)$; at each timestep, the goal $g$ is updated as $g \leftarrow g - r_t$ and $h \leftarrow \max(h-1, 1)$ (Reymond et al., 2022, Reymond et al., 2022). For preference-based conditioning, one rolls out the greedy policy for $w^\top \mathbf{Q}$.
• Sweep and Front Construction: By sampling a grid of preference or goal vectors, running policies, and recording the resulting return vectors, PCN sweeps out the Pareto front. The single-network structure enables smooth interpolation between solutions.
• Scalability: A single PCN instance suffices to capture all non-dominated policies, whereas conventional methods require separate networks or combinatorially many policies.

5. Empirical Results and Benchmark Performance

PCN has demonstrated superior or comparable performance to strong MORL baselines (MONES, RA) across standard and challenging domains:

Benchmark	Objectives (m)	PCN-HV (mean±std)	Baseline-HV (MONES/RA)	PCN-$I_\epsilon$ (mean±std)	Baseline-$I_\epsilon$
Deep Sea Treasure	2	22845 ± 19	17385 ± 6521 / 22437 ± 49	0.039 ± 0.087	0.687 / 0.667
Minecart	3	197.6 ± 0.7	123.8 ± 23.0 / 123.9 ± 0.3	0.271 ± 0.087	1.596 / 1.000
Crossroad	2	539.5 ± 6.3	429.1 ± 27.5 / 466.0 ± 31.2	0.247 ± 0.172	0.660 / 0.408
Walkroom	2–9	consistently superior, stable up to $m=9$	RA intractable for $m>5$	low, scales to $m=9$	-

Evaluation metrics include hypervolume (HV, higher is better), $\varepsilon$-indicator ($I_\epsilon$, lower is better; minimal supremal $\ell_\infty$ distance to cover the true Pareto front), and coverage set cardinality (Reymond et al., 2022).

In high-dimensional or application domains (e.g., epidemic response, as in COVID-19), PCN recovers interpretable trade-offs between public health and socioeconomic objectives, efficiently spanning fronts for up to three objectives using a single Q-network or policy network (Chen et al., 2 Oct 2025, Reymond et al., 2022).

6. Extensions: Continuous Actions and High-Dimensional Settings

The PCN methodology readily extends to continuous action spaces via architectural and loss modifications:

• Replace action-classification with regression heads and MSE loss.
• Rescale outputs via $\tanh$ or affine mappings to enforce action bounds.
• Introduce Gaussian action noise for on-policy exploration; no critic/Bellman recursion is needed (Reymond et al., 2022).

Conditional goal or preference embeddings maintain flexibility in complex, high-dimensional domains (e.g., age-structured pandemic control), with empirical stability and scalability (Chen et al., 2 Oct 2025).

7. Significance, Assumptions, and Limitations

PCN relies on several key properties:

• Single-network universal coverage: All Pareto solutions become accessible via goal/preference conditioning, sidestepping policy-per-point redundancy and enabling sample sharing.
• No convexity requirement: Can recover Pareto fronts of arbitrary shape, including concave and disconnected regions, unlike linear-scalarization-based approaches.
• Stable optimization: Fixed supervised labels derived from actual transitions prevent “moving target” pathologies of traditional RL.
• No explicit Bellman dependency: In the supervised formulation, TD learning and its instabilities are circumvented, except in the value-based variant.

Empirical results indicate robust performance even as objectives scale or environments gain complexity. A plausible implication is that PCN’s stability and flexibility recommend it for real-world, high-dimensional, and safety-critical policy optimization contexts.

Table: Core Characteristics of PCN Compared to Key Baselines

Aspect	PCN	RA/MONES Baselines
Network Count	1	$\sim$1 per Pareto point
Front Shape Limitations	None	Convex only (RA)
Scalability ($m \gg 2$)	Stable	Intractable ($m>5$ for RA)
Training Stability	High (supervised)	RL instabilities present
Action Space Support	Discrete, continuous	Varies

(Reymond et al., 2022, Reymond et al., 2022)

PCN enables accelerated, comprehensive Pareto front estimation in MORL and multi-task control, as confirmed by results across synthetic benchmarks and epidemiological simulation environments (Chen et al., 2 Oct 2025, Reymond et al., 2022, Reymond et al., 2022).