---
title: Pareto-Conditioned Network (PCN)
url: https://www.emergentmind.com/topics/pareto-conditioned-network-pcn
type: topic
---

# Pareto-Conditioned Network (PCN)

Searching arXiv for the primary PCN paper and closely related work to ground the article in the relevant literature.
Pareto Conditioned Networks (PCN) are a multi-policy, multi-objective reinforcement-learning method that recasts the problem of learning an entire Pareto front into a single supervised-learning problem over conditioned policy outputs. In the standard multi-objective Markov decision-process setting, the objective is to learn the Pareto-optimal policy set rather than a single scalarized optimum. PCN addresses the cost of recovering all non-dominated policies by using a single neural network conditioned on return and horizon information, so that a concrete policy can be recovered by conditioning the network on a desired Pareto-efficient solution [2204.05036].

## 1. Formal setting and Pareto-optimal policy learning

PCN is formulated in the standard multi-objective Markov decision-process (MOMDP) setting
$M \equiv \langle S, A, T, \gamma, \vec{R}, n \rangle$,
where $S$ is the state space, $A$ the finite action set, $T(s'|s,a)$ the transition kernel, $\gamma \in [0,1)$ a discount factor, and $\vec{R}(s,a,s') \in \mathbb{R}^n$ an $n$-dimensional immediate-reward vector [2204.05036]. A trajectory $\tau = (s_0, a_0, s_1, a_1, \dots, s_T)$ produces a vector return
$\vec{R}(\tau) = \sum_{t=0}^{T} \gamma^t \vec{r}_t$,
with $\vec{r}_t \equiv \vec{R}(s_t,a_t,s_{t+1})$ [2204.05036].

A policy $\pi(a|s)$ induces an expected return vector
$\vec{V}^{\pi} = (V_1^{\pi}, \dots, V_n^{\pi})$,
with
$V_i^{\pi} = E_{\tau \sim \pi}[\sum_{t=0}^{T}\gamma^t r_{t,i}]$ [2204.05036]. Multi-objective optimization uses Pareto dominance:
$\vec{V} \succ_P \vec{V}' \Leftrightarrow (\forall i: V_i \ge V'_i) \land (\exists j: V_j > V'_j)$,
and the Pareto-optimal policy set is
$\Pi^* = \{\pi \mid \neg \exists \pi': \vec{V}^{\pi'} \succ_P \vec{V}^{\pi}\}$,
whose image in objective space is the Pareto front
$F = \{\vec{V}^{\pi} \mid \pi \in \Pi^*\}$ [2204.05036].

Within this formulation, learning $\Pi^*$ means finding a deterministic policy $\pi_z$ for each point $\vec{V}_z \in F$ [2204.05036]. The motivating difficulty is that the set of optimal policies can grow exponentially with the number of objectives, and recovering all solutions requires an exhaustive exploration of the entire state space [2204.05036]. PCN is intended to encompass all non-dominated policies with one network rather than learning one policy per weight vector.

## 2. Reformulation as conditional supervised learning

The defining step in PCN is to avoid temporal-difference targets by collecting full trajectories and extracting labeled data points of the form
$x = (s_t, h_t, \vec{R}_t)$,
where
$h_t = T - t$ and
$\vec{R}_t = \sum_{i=t}^{T}\gamma^{\,i-t}\vec{r}_i$,
with class label $y = a_t$ [2204.05036]. The network is then trained so that, when conditioned on the same $(h,\vec{R})$ actually realized in the trajectory, it reproduces the recorded action:
$\pi_{\theta}(a|s,h,\vec{R}) \approx \Pr[a_t = a \mid s_t = s, h_t = h, \vec{R}_t = \vec{R}]$ [2204.05036].

For discrete action spaces, the resulting objective is a cross-entropy loss over a dataset $D$ of tuples $(s,h,\vec{R},a)$:
$$
L(\theta) = -E_{(s,h,\vec{R},a)\sim D}[\log \pi_{\theta}(a|s,h,\vec{R})].
$$
This transforms the optimization problem into a classification problem [2204.05036]. The paper explicitly characterizes this as stable because training is supervised on fixed targets, thereby avoiding moving target issues [2204.05036].

A related application in epidemic control adopts the same basic idea but extends PCN to a continuous action space $a \in [0,1]^3$ [2204.05027]. There, PCN records tuples
$x = (s_t, \hat{h}_t, \hat{R}_t)$
with label $y = a_t$ into an experience set $D$, but replaces cross-entropy with mean squared error:
$$
\mathrm{MSE} = (1/|A|)\sum_{i=1}^{3}(\hat{y}_i - y_i)^2,
$$
where the network outputs a 3-dimensional vector passed through $\tanh$ and then linearly rescaled to $[0,1]$ [2204.05027]. A small Gaussian noise is added to the target returns during training to guard against overfitting under stochastic transitions, and Gaussian action noise is added during exploration [2204.05027].

This suggests that the core abstraction of PCN is not limited to discrete-action classification, even though the original formulation is discrete-action and uses cross-entropy [2204.05027].

## 3. Conditioning mechanism and network architecture

In the original PCN architecture, the network has three inputs: a state embedding $\phi_s = f_s(s) \in \mathbb{R}^{d_s}$, a conditioning vector $z \equiv [h;\vec{R}] \in \mathbb{R}^{n+1}$ obtained by concatenating horizon and target return, and a fusion mechanism defined by an element-wise Hadamard product [2204.05036]. The conditioning vector is normalized and passed through a second embedding $\phi_z = f_z(z) \in \mathbb{R}^{d_z}$, after which the combined representation is
$\phi(s,h,\vec{R}) = \phi_s(s)\odot \phi_z(h,\vec{R})$ [2204.05036]. This representation is then fed through a final MLP with $|A|$ outputs, producing action logits and, after softmax, $\pi_{\theta}(a|s,h,\vec{R})$ [2204.05036].

Typical hidden-layer sizes used in the reported experiments are $d_s = d_z = 64$, one hidden layer in $\phi_s$ and $\phi_z$, and one hidden layer of size 64 in the final MLP [2204.05036]. The architecture is therefore compact, and its input dimension grows linearly in the number of objectives $n$ [2204.05036].

The COVID-19 application uses a more domain-specific architecture, but preserves the same structural principle: decomposed embeddings for state and conditioning, followed by multiplicative fusion [2204.05027]. In the final “conv1d-big” variant, the compartmental state $s_m \in \mathbb{R}^D$, contact-matrix parameters $p_w,p_s,p_l \in \mathbb{R}^3$, and a Boolean school-holiday flag are embedded separately; these embeddings are multiplied element-wise to yield a 64-dimensional state embedding “s_emb.” The conditioning inputs $(\hat{h},\hat{R})$ are concatenated into a 4-dimensional vector and embedded into “c_emb,” after which fusion is again performed by an elementwise product $z = s\_emb \odot c\_emb \in \mathbb{R}^{64}$ [2204.05027]. The final fully connected layers produce three continuous outputs corresponding to actionable contact-reduction parameters.

A distinct line of work on Pareto front learning describes self-evolutionary Pareto networks (SEPNet) as a Pareto-Conditioned Network in which the conditioning vector is formed from a preference vector $\lambda$ and hyper-parameters $\phi$, and injected via FiLM-style layers [2110.03461]. There, for a feature map $h$, the network computes $\gamma(C), \beta(C) = \mathrm{MLP}(C)$ and applies
$\widetilde{h} = \gamma(C)\odot h + \beta(C)$,
with $C = (\lambda,\phi)$ [2110.03461]. This is architecturally distinct from the original PCN, but it reflects the broader idea of learning a single model that spans a Pareto front through explicit conditioning.

## 4. Training procedure and buffer management

The original PCN training algorithm maintains an empty trajectory buffer $\mathcal{T}$, an empty dataset $D$, and network parameters $\theta$ [2204.05036]. During an initial warm-up phase, exploration uses a random policy uniform over $A$. After warm-up, the algorithm samples a non-dominated $(h,\vec{R}_{nd})$ from $D$, chooses one objective at random, perturbs that objective by a uniform random variable $\delta \sim \mathrm{Uniform}(0,\sigma_o)$ where $\sigma_o$ is the standard deviation of that objective across non-dominated returns, and uses the resulting target return vector $\vec{R}_{target}$ to define the exploration policy
$\pi_{explore}(a|s) \equiv \pi_{\theta}(a|s,h_{nd},\vec{R}_{target})$
with softmax sampling [2204.05036].

After each episode, the algorithm computes for each timestep
$h_t = T - t$ and
$\vec{R}_t = \sum_{i=t}^{T}\gamma^{i-t}\vec{r}_i$,
then appends $(s_t,h_t,\vec{R}_t,a_t)$ to $D$ [2204.05036]. Each full trajectory is stored in the trajectory buffer together with its final return vector. If the buffer exceeds its capacity $N_{traj}$, it is pruned and the dataset is rebuilt from the retained trajectories [2204.05036]. Parameter updates occur every $E_{episodes}$ episodes using minibatches of size $B$ and $K$ gradient steps on the supervised loss [2204.05036].

A specific trajectory-pruning rule is defined through a “dominating score” $I_{ds}$ that combines two terms: the negative $\ell_2$-distance of each return to its closest non-dominated neighbor, and a crowding-distance penalty that discourages clusters [2204.05036]. Trajectories with the lowest $I_{ds}$ are removed until the buffer size is reduced to $N_{traj}$ [2204.05036]. The role of this mechanism is to retain the most informative trajectories while preserving Pareto-front coverage.

In the continuous-action epidemic application, the training protocol is stated in operational terms. Desired returns are sampled uniformly, or by exploring non-dominated returns in the buffer, from the box spanned by current best episodic payoffs; the agent runs episodes conditioned on $(\hat{R},\hat{h})$; stores transitions in a rolling buffer of the last $E$ episodes; retrains for $U$ mini-batches sampled uniformly from the buffer; and may prune the buffer to keep only episodes whose final return is non-dominated [2204.05027]. The reported hyperparameters include Adam with learning rate $1e{-3}$, total environment interactions of 300 000 time steps, batch size 256, 10 episodes per network update, 50 gradient-update steps per batch, replay buffer size 1 000 episodes, 200 initial purely random exploration episodes, exploration noise $\eta = 0.1$, desired-return noise $\sigma_R = 0.2$, and reward normalization by factors $[10\,000,100]$ before conditioning [2204.05027].

## 5. Policy recovery, inference, and front geometry

Once trained, PCN recovers a deterministic policy for any desired Pareto point $\vec{R}^*$ by setting $h^*$ equal to the episode length, or a large constant, and conditioning on $\vec{R}^*$ [2204.05036]. At decision time $t$, with current state $s_t$, the action is selected by
$$
a_t = \arg\max_a \pi_{\theta}(a|s_t,h_t,\vec{R}^*),
$$
with $h_t$ interpreted as the remaining timesteps and decremented at each step [2204.05036]. Because the network is trained to reproduce actions known to reach varying $\vec{R}_t$ in exactly $h_t$ steps, the method is described as generalizing to targets on or inside the convex hull of collected returns [2204.05036].

A central property claimed for PCN is that it makes no convexity assumption on the Pareto front $F$ [2204.05036]. Because conditioning is performed on exact observed returns $\vec{R}_t$ rather than convex combinations of value estimates, PCN is reported to naturally learn trajectories leading to non-convex, concave, or even disconnected Pareto-front geometries [2204.05036]. The only requirement stated is that the target $\vec{R}^*$ lies within the range of previously experienced returns, or not too far outside so that the network generalizes [2204.05036].

This geometric flexibility is important in the comparison to methods based on linear scalarizations. The original paper explicitly contrasts PCN with methods that learn one policy per weight vector or rely on linear scalarizations, noting that such methods recover only convex portions of $F$ [2204.05036]. The empirical discussion further states that PCN consistently recovered all points on concave fronts that RA and linear-scalarization-based methods miss [2204.05036].

In the epidemic-control application, the same inferential logic is used differently: one sweeps $\hat{R}$ over a fine grid, executes the deterministic policy, and plots the achieved return vectors as an approximate Pareto front or coverage set [2204.05027]. This suggests that, operationally, PCN serves both as a conditioned policy and as an implicit generator of Pareto-front approximations.

## 6. Stability, scalability, and empirical performance

The original PCN paper characterizes the method as stable because training is purely supervised on fixed targets, with no moving temporal-difference targets or on-policy distribution shifts [2204.05036]. It also characterizes the method as scalable with respect to the number of objectives: unlike methods that learn one policy per weight vector, PCN uses a single network of size $O(|A|\cdot d_{hidden})$ whose input dimension grows linearly in $n$ [2204.05036]. Experience is shared across all subregions of the Pareto front through the common buffer [2204.05036].

The reported computational complexity is likewise explicit. Each training step is $O(B \cdot |A|)$ for cross-entropy backpropagation, whereas pruning trajectories takes $O(N_{traj}^2 \cdot n)$ once per buffer refresh [2204.05036]. No enumeration of all weight vectors or explicit Pareto-set maintenance is required beyond nondominated filtering of returns in the buffer [2204.05036].

Empirically, PCN was evaluated on Deep-Sea Treasure, Minecart, Crossroad, and a synthetic high-objective “Walkroom” with up to $n=9$ objectives [2204.05036]. Two baselines sharing the same assumption on monotonic utilities were compared: MONES and the Radial Algorithm (RA) [2204.05036]. Metrics were hypervolume, the $\epsilon$-indicator $I_{\epsilon}$, and the $\epsilon$-mean indicator $I_{\epsilon\text{-mean}}$ [2204.05036]. Across all runs and environments, PCN achieved the highest hypervolume, the lowest $I_{\epsilon}$, and the best $I_{\epsilon\text{-mean}}$; on Deep-Sea Treasure, an example hypervolume comparison is $22\,845 \pm 19$ for PCN versus $17\,385 \pm 6\,521$ for MONES, and an example $I_{\epsilon}$ comparison is $0.039 \pm 0.087$ versus $0.687 \pm 0.222$ [2204.05036]. In high-objective Walkroom with $n \ge 5$, RA was reported as infeasible, MONES degraded severely, while PCN maintained high coverage and low $\epsilon$-error up to $n=9$ [2204.05036].

The COVID-19 mitigation study provides an application-level validation of the same methodology in a complex epidemiological MOMDP [2204.05027]. It considers two objective pairs, $(\mathrm{ARH}, \mathrm{SB})$ and $(\mathrm{ARI}, \mathrm{SB})$, in both deterministic ODE and stochastic binomial chain-binomial models [2204.05027]. Against a baseline of 100 fixed policies uniformly swept in $[0,1]^3$, the learned Pareto front in the ODE setting is reported to dominate almost all baseline points [2204.05027]. Quantitative results include, for PCN\_ARH (ODE), hypervolume $= 0.679 \pm 0.010$, $I_{\epsilon} = 0.056 \pm 0.004$, and $I_{\epsilon\text{-mean}} = 0.010 \pm 0.006$; robustness tests re-executing each conditioned policy 5 times yielded deviations from target returns of at most $I_{\epsilon} \approx 0.046$ (ODE) or $\approx 0.068$ (Binomial), with mean deviation $\sim 0.007$ or $\sim 0.024$ respectively [2204.05027].

A plausible implication is that PCN’s principal significance lies in separating Pareto-front recovery from conventional TD-based value learning, while preserving the ability to deploy a single conditioned model across many trade-off regimes. In the original formulation, this is accomplished through return-and-horizon conditioning with supervised replay [2204.05036]; in later adaptations, the same principle is extended to continuous actions and domain-specific structured state embeddings [2204.05027].

Source: https://www.emergentmind.com/topics/pareto-conditioned-network-pcn