Pareto-Conditioned Network (PCN)

• Pareto-Conditioned Network (PCN) is a reinforcement learning approach that models the entire Pareto front of non-dominated policies using a single conditioned neural network.
• It reformulates multi-objective policy learning as a supervised classification (or regression) task by conditioning on return and horizon data, sidestepping moving target issues.
• Empirical studies show that PCN outperforms baselines on metrics like hypervolume and ε-indicator, especially in high-objective, non-convex problem settings.

Searching arXiv for the primary PCN paper and closely related work to ground the article in the relevant literature. Pareto Conditioned Networks (PCN) are a multi-policy, multi-objective reinforcement-learning method that recasts the problem of learning an entire Pareto front into a single supervised-learning problem over conditioned policy outputs. In the standard multi-objective Markov decision-process setting, the objective is to learn the Pareto-optimal policy set rather than a single scalarized optimum. PCN addresses the cost of recovering all non-dominated policies by using a single neural network conditioned on return and horizon information, so that a concrete policy can be recovered by conditioning the network on a desired Pareto-efficient solution (Reymond et al., 2022).

1. Formal setting and Pareto-optimal policy learning

PCN is formulated in the standard multi-objective Markov decision-process (MOMDP) setting $$M \equiv \langle S, A, T, \gamma, \vec{R}, n \rangle$$, where $S$ is the state space, $A$ the finite action set, $T(s'|s,a)$ the transition kernel, $\gamma \in [0,1)$ a discount factor, and $\vec{R}(s,a,s') \in \mathbb{R}^n$ an $n$-dimensional immediate-reward vector (Reymond et al., 2022). A trajectory $\tau = (s_0, a_0, s_1, a_1, \dots, s_T)$ produces a vector return $\vec{R}(\tau) = \sum_{t=0}^{T} \gamma^t \vec{r}_t$, with $\vec{r}_t \equiv \vec{R}(s_t,a_t,s_{t+1})$ (Reymond et al., 2022).

A policy $S$0 induces an expected return vector $S$1, with $S$2 (Reymond et al., 2022). Multi-objective optimization uses Pareto dominance: $S$3, and the Pareto-optimal policy set is $S$4, whose image in objective space is the Pareto front $S$5 (Reymond et al., 2022).

Within this formulation, learning $S$6 means finding a deterministic policy $S$7 for each point $S$8 (Reymond et al., 2022). The motivating difficulty is that the set of optimal policies can grow exponentially with the number of objectives, and recovering all solutions requires an exhaustive exploration of the entire state space (Reymond et al., 2022). PCN is intended to encompass all non-dominated policies with one network rather than learning one policy per weight vector.

2. Reformulation as conditional supervised learning

The defining step in PCN is to avoid temporal-difference targets by collecting full trajectories and extracting labeled data points of the form $S$9, where $A$0 and $A$1, with class label $A$2 (Reymond et al., 2022). The network is then trained so that, when conditioned on the same $A$3 actually realized in the trajectory, it reproduces the recorded action: $A$4 (Reymond et al., 2022).

For discrete action spaces, the resulting objective is a cross-entropy loss over a dataset $A$5 of tuples $A$6:

$A$7

This transforms the optimization problem into a classification problem (Reymond et al., 2022). The paper explicitly characterizes this as stable because training is supervised on fixed targets, thereby avoiding moving target issues (Reymond et al., 2022).

A related application in epidemic control adopts the same basic idea but extends PCN to a continuous action space $A$8 (Reymond et al., 2022). There, PCN records tuples

$A$9

with label $T(s'|s,a)$0 into an experience set $T(s'|s,a)$1, but replaces cross-entropy with mean squared error:

$T(s'|s,a)$2

where the network outputs a 3-dimensional vector passed through $T(s'|s,a)$3 and then linearly rescaled to $T(s'|s,a)$4 (Reymond et al., 2022). A small Gaussian noise is added to the target returns during training to guard against overfitting under stochastic transitions, and Gaussian action noise is added during exploration (Reymond et al., 2022).

This suggests that the core abstraction of PCN is not limited to discrete-action classification, even though the original formulation is discrete-action and uses cross-entropy (Reymond et al., 2022).

3. Conditioning mechanism and network architecture

In the original PCN architecture, the network has three inputs: a state embedding $T(s'|s,a)$5, a conditioning vector $T(s'|s,a)$6 obtained by concatenating horizon and target return, and a fusion mechanism defined by an element-wise Hadamard product (Reymond et al., 2022). The conditioning vector is normalized and passed through a second embedding $T(s'|s,a)$7, after which the combined representation is $T(s'|s,a)$8 (Reymond et al., 2022). This representation is then fed through a final MLP with $T(s'|s,a)$9 outputs, producing action logits and, after softmax, $\gamma \in [0,1)$0 (Reymond et al., 2022).

Typical hidden-layer sizes used in the reported experiments are $\gamma \in [0,1)$1, one hidden layer in $\gamma \in [0,1)$2 and $\gamma \in [0,1)$3, and one hidden layer of size 64 in the final MLP (Reymond et al., 2022). The architecture is therefore compact, and its input dimension grows linearly in the number of objectives $\gamma \in [0,1)$4 (Reymond et al., 2022).

The COVID-19 application uses a more domain-specific architecture, but preserves the same structural principle: decomposed embeddings for state and conditioning, followed by multiplicative fusion (Reymond et al., 2022). In the final “conv1d-big” variant, the compartmental state $\gamma \in [0,1)$5, contact-matrix parameters $\gamma \in [0,1)$6, and a Boolean school-holiday flag are embedded separately; these embeddings are multiplied element-wise to yield a 64-dimensional state embedding “s_emb.” The conditioning inputs $\gamma \in [0,1)$7 are concatenated into a 4-dimensional vector and embedded into “c_emb,” after which fusion is again performed by an elementwise product $\gamma \in [0,1)$8 (Reymond et al., 2022). The final fully connected layers produce three continuous outputs corresponding to actionable contact-reduction parameters.

A distinct line of work on Pareto front learning describes self-evolutionary Pareto networks (SEPNet) as a Pareto-Conditioned Network in which the conditioning vector is formed from a preference vector $\gamma \in [0,1)$9 and hyper-parameters $\vec{R}(s,a,s') \in \mathbb{R}^n$0, and injected via FiLM-style layers (Chang et al., 2021). There, for a feature map $\vec{R}(s,a,s') \in \mathbb{R}^n$1, the network computes $\vec{R}(s,a,s') \in \mathbb{R}^n$2 and applies $\vec{R}(s,a,s') \in \mathbb{R}^n$3, with $\vec{R}(s,a,s') \in \mathbb{R}^n$4 (Chang et al., 2021). This is architecturally distinct from the original PCN, but it reflects the broader idea of learning a single model that spans a Pareto front through explicit conditioning.

4. Training procedure and buffer management

The original PCN training algorithm maintains an empty trajectory buffer $\vec{R}(s,a,s') \in \mathbb{R}^n$5, an empty dataset $\vec{R}(s,a,s') \in \mathbb{R}^n$6, and network parameters $\vec{R}(s,a,s') \in \mathbb{R}^n$7 (Reymond et al., 2022). During an initial warm-up phase, exploration uses a random policy uniform over $\vec{R}(s,a,s') \in \mathbb{R}^n$8. After warm-up, the algorithm samples a non-dominated $\vec{R}(s,a,s') \in \mathbb{R}^n$9 from $n$0, chooses one objective at random, perturbs that objective by a uniform random variable $n$1 where $n$2 is the standard deviation of that objective across non-dominated returns, and uses the resulting target return vector $n$3 to define the exploration policy

$n$4

with softmax sampling (Reymond et al., 2022).

After each episode, the algorithm computes for each timestep $n$5 and $n$6, then appends $n$7 to $n$8 (Reymond et al., 2022). Each full trajectory is stored in the trajectory buffer together with its final return vector. If the buffer exceeds its capacity $n$9, it is pruned and the dataset is rebuilt from the retained trajectories (Reymond et al., 2022). Parameter updates occur every $\tau = (s_0, a_0, s_1, a_1, \dots, s_T)$0 episodes using minibatches of size $\tau = (s_0, a_0, s_1, a_1, \dots, s_T)$1 and $\tau = (s_0, a_0, s_1, a_1, \dots, s_T)$2 gradient steps on the supervised loss (Reymond et al., 2022).

A specific trajectory-pruning rule is defined through a “dominating score” $\tau = (s_0, a_0, s_1, a_1, \dots, s_T)$3 that combines two terms: the negative $\tau = (s_0, a_0, s_1, a_1, \dots, s_T)$4-distance of each return to its closest non-dominated neighbor, and a crowding-distance penalty that discourages clusters (Reymond et al., 2022). Trajectories with the lowest $\tau = (s_0, a_0, s_1, a_1, \dots, s_T)$5 are removed until the buffer size is reduced to $\tau = (s_0, a_0, s_1, a_1, \dots, s_T)$6 (Reymond et al., 2022). The role of this mechanism is to retain the most informative trajectories while preserving Pareto-front coverage.

In the continuous-action epidemic application, the training protocol is stated in operational terms. Desired returns are sampled uniformly, or by exploring non-dominated returns in the buffer, from the box spanned by current best episodic payoffs; the agent runs episodes conditioned on $\tau = (s_0, a_0, s_1, a_1, \dots, s_T)$7; stores transitions in a rolling buffer of the last $\tau = (s_0, a_0, s_1, a_1, \dots, s_T)$8 episodes; retrains for $\tau = (s_0, a_0, s_1, a_1, \dots, s_T)$9 mini-batches sampled uniformly from the buffer; and may prune the buffer to keep only episodes whose final return is non-dominated (Reymond et al., 2022). The reported hyperparameters include Adam with learning rate $\vec{R}(\tau) = \sum_{t=0}^{T} \gamma^t \vec{r}_t$0, total environment interactions of 300 000 time steps, batch size 256, 10 episodes per network update, 50 gradient-update steps per batch, replay buffer size 1 000 episodes, 200 initial purely random exploration episodes, exploration noise $\vec{R}(\tau) = \sum_{t=0}^{T} \gamma^t \vec{r}_t$1, desired-return noise $\vec{R}(\tau) = \sum_{t=0}^{T} \gamma^t \vec{r}_t$2, and reward normalization by factors $\vec{R}(\tau) = \sum_{t=0}^{T} \gamma^t \vec{r}_t$3 before conditioning (Reymond et al., 2022).

5. Policy recovery, inference, and front geometry

Once trained, PCN recovers a deterministic policy for any desired Pareto point $\vec{R}(\tau) = \sum_{t=0}^{T} \gamma^t \vec{r}_t$4 by setting $\vec{R}(\tau) = \sum_{t=0}^{T} \gamma^t \vec{r}_t$5 equal to the episode length, or a large constant, and conditioning on $\vec{R}(\tau) = \sum_{t=0}^{T} \gamma^t \vec{r}_t$6 (Reymond et al., 2022). At decision time $\vec{R}(\tau) = \sum_{t=0}^{T} \gamma^t \vec{r}_t$7, with current state $\vec{R}(\tau) = \sum_{t=0}^{T} \gamma^t \vec{r}_t$8, the action is selected by

$\vec{R}(\tau) = \sum_{t=0}^{T} \gamma^t \vec{r}_t$9

with $\vec{r}_t \equiv \vec{R}(s_t,a_t,s_{t+1})$0 interpreted as the remaining timesteps and decremented at each step (Reymond et al., 2022). Because the network is trained to reproduce actions known to reach varying $\vec{r}_t \equiv \vec{R}(s_t,a_t,s_{t+1})$1 in exactly $\vec{r}_t \equiv \vec{R}(s_t,a_t,s_{t+1})$2 steps, the method is described as generalizing to targets on or inside the convex hull of collected returns (Reymond et al., 2022).

A central property claimed for PCN is that it makes no convexity assumption on the Pareto front $\vec{r}_t \equiv \vec{R}(s_t,a_t,s_{t+1})$3 (Reymond et al., 2022). Because conditioning is performed on exact observed returns $\vec{r}_t \equiv \vec{R}(s_t,a_t,s_{t+1})$4 rather than convex combinations of value estimates, PCN is reported to naturally learn trajectories leading to non-convex, concave, or even disconnected Pareto-front geometries (Reymond et al., 2022). The only requirement stated is that the target $\vec{r}_t \equiv \vec{R}(s_t,a_t,s_{t+1})$5 lies within the range of previously experienced returns, or not too far outside so that the network generalizes (Reymond et al., 2022).

This geometric flexibility is important in the comparison to methods based on linear scalarizations. The original paper explicitly contrasts PCN with methods that learn one policy per weight vector or rely on linear scalarizations, noting that such methods recover only convex portions of $\vec{r}_t \equiv \vec{R}(s_t,a_t,s_{t+1})$6 (Reymond et al., 2022). The empirical discussion further states that PCN consistently recovered all points on concave fronts that RA and linear-scalarization-based methods miss (Reymond et al., 2022).

In the epidemic-control application, the same inferential logic is used differently: one sweeps $\vec{r}_t \equiv \vec{R}(s_t,a_t,s_{t+1})$7 over a fine grid, executes the deterministic policy, and plots the achieved return vectors as an approximate Pareto front or coverage set (Reymond et al., 2022). This suggests that, operationally, PCN serves both as a conditioned policy and as an implicit generator of Pareto-front approximations.

6. Stability, scalability, and empirical performance

The original PCN paper characterizes the method as stable because training is purely supervised on fixed targets, with no moving temporal-difference targets or on-policy distribution shifts (Reymond et al., 2022). It also characterizes the method as scalable with respect to the number of objectives: unlike methods that learn one policy per weight vector, PCN uses a single network of size $\vec{r}_t \equiv \vec{R}(s_t,a_t,s_{t+1})$8 whose input dimension grows linearly in $\vec{r}_t \equiv \vec{R}(s_t,a_t,s_{t+1})$9 (Reymond et al., 2022). Experience is shared across all subregions of the Pareto front through the common buffer (Reymond et al., 2022).

The reported computational complexity is likewise explicit. Each training step is $S$00 for cross-entropy backpropagation, whereas pruning trajectories takes $S$01 once per buffer refresh (Reymond et al., 2022). No enumeration of all weight vectors or explicit Pareto-set maintenance is required beyond nondominated filtering of returns in the buffer (Reymond et al., 2022).

Empirically, PCN was evaluated on Deep-Sea Treasure, Minecart, Crossroad, and a synthetic high-objective “Walkroom” with up to $S$02 objectives (Reymond et al., 2022). Two baselines sharing the same assumption on monotonic utilities were compared: MONES and the Radial Algorithm (RA) (Reymond et al., 2022). Metrics were hypervolume, the $S$03-indicator $S$04, and the $S$05-mean indicator $S$06 (Reymond et al., 2022). Across all runs and environments, PCN achieved the highest hypervolume, the lowest $S$07, and the best $S$08; on Deep-Sea Treasure, an example hypervolume comparison is $S$09 for PCN versus $S$10 for MONES, and an example $S$11 comparison is $S$12 versus $S$13 (Reymond et al., 2022). In high-objective Walkroom with $S$14, RA was reported as infeasible, MONES degraded severely, while PCN maintained high coverage and low $S$15-error up to $S$16 (Reymond et al., 2022).

The COVID-19 mitigation study provides an application-level validation of the same methodology in a complex epidemiological MOMDP (Reymond et al., 2022). It considers two objective pairs, $S$17 and $S$18, in both deterministic ODE and stochastic binomial chain-binomial models (Reymond et al., 2022). Against a baseline of 100 fixed policies uniformly swept in $S$19, the learned Pareto front in the ODE setting is reported to dominate almost all baseline points (Reymond et al., 2022). Quantitative results include, for PCN_ARH (ODE), hypervolume $S$20, $S$21, and $S$22; robustness tests re-executing each conditioned policy 5 times yielded deviations from target returns of at most $S$23 (ODE) or $S$24 (Binomial), with mean deviation $S$25 or $S$26 respectively (Reymond et al., 2022).

A plausible implication is that PCN’s principal significance lies in separating Pareto-front recovery from conventional TD-based value learning, while preserving the ability to deploy a single conditioned model across many trade-off regimes. In the original formulation, this is accomplished through return-and-horizon conditioning with supervised replay (Reymond et al., 2022); in later adaptations, the same principle is extended to continuous actions and domain-specific structured state embeddings (Reymond et al., 2022).