Pareto-NRPA: Discrete Multi-Objective Search

• Pareto-NRPA is a multi-objective Monte Carlo search algorithm that approximates the Pareto set in discrete and constrained settings without scalarization.
• It employs a set of policies to explore diverse regions and adapts them using crowding-distance-weighted updates from multiple non-dominated sequences.
• Benchmarked on MO-TSPTW and NAS tasks, the method shows robust performance in constrained optimization compared to traditional multi-objective evolutionary algorithms.

Pareto-NRPA is a Monte-Carlo search algorithm for multi-objective optimization over discrete search spaces. It extends Nested Rollout Policy Adaptation (NRPA), which was originally formulated for single-objective combinatorial search, by replacing the single-policy, single-incumbent logic of NRPA with a set of policies, non-dominated front maintenance at every nested level, and policy updates weighted by isolation within the retained Pareto front. The method is designed for settings of the form

$$\min \left(f_1(x), f_2(x), \ldots, f_p(x)\right) \quad \text{subject to } x \in X,$$

where $X$ may be discrete and constrained, and the target is an approximation of the Pareto set rather than a single optimum (Lallouet et al., 25 Jul 2025).

1. Problem setting and conceptual basis

Pareto-NRPA is formulated for multi-objective optimization (MOO) in discrete search spaces, with particular emphasis on constrained settings. The objective is to approximate a set of mutually non-dominated solutions that represent different trade-offs among objectives. The paper uses the standard Pareto dominance relation: for two solutions $x$ and $y$, $x \prec y$ iff

$f_i(x) \le f_i(y), \quad \forall i=1,\ldots,p,$

and there exists at least one objective $j$ such that

$f_j(x) < f_j(y).$

A solution is Pareto-optimal iff it is not dominated by any feasible solution, and the set-valued target over sequence space $S$ is

$$S^* = \{ s \in S \mid \nexists u \in S : u \prec s \}. \tag{2}$$

The paper’s starting point is that standard NRPA had been successful in single-objective combinatorial search, but there had been no direct adaptation of NRPA to the multi-objective setting. That extension is nontrivial because standard NRPA repeatedly improves one policy toward one best sequence, whereas MOO generally has no unique best sequence. The required algorithmic shift is therefore from scalar best-so-far optimization to simultaneous preservation of multiple trade-offs and explicit maintenance of diversity along a Pareto front.

A central design choice is that Pareto-NRPA does not optimize weighted sums or use objective aggregation internally. Search returns sets rather than scalar scores; survival is based on non-dominance; and adaptation weights are derived from diversity or isolation within objective space rather than scalarized fitness. This makes the algorithm Pareto-based in a direct sense rather than decomposition-based (Lallouet et al., 25 Jul 2025).

2. Relation to standard NRPA

Standard NRPA combines nested Monte Carlo search with policy learning. At level $X$0, a stochastic rollout is generated under a softmax policy over move codes. If $X$1 denotes the logit for move $X$2, then the selection rule is

$X$3

A level-$X$4 search repeatedly invokes level-$X$5 searches, retains the best returned sequence, and updates the policy by increasing the probability of the moves in that sequence while decreasing competing legal moves proportionally to their current softmax mass.

Pareto-NRPA preserves the nested recursion, the rollout mechanism, move coding, and replay-based policy adaptation, but modifies exactly those elements that are incompatible with multi-objective search. The transformation can be summarized as follows.

Standard NRPA	Pareto-NRPA	Function
Single policy $X$6	Policy set $X$7	Supports multiple regions
Single best sequence	Non-dominated set $X$8	Represents trade-offs
One-sequence update	Multi-sequence update	Preserves multiple solutions
Best-score selection	Non-dominated front maintenance	Enforces Pareto survival
No explicit diversity term	Crowding-distance-weighted update	Favors spread

This modification is intended to let different policies concentrate on different regions of the trade-off surface without requiring explicit scalarization or reference-vector decomposition. The paper explicitly states that Pareto-NRPA reduces to standard NRPA in the special case of one objective and one policy (Lallouet et al., 25 Jul 2025).

3. Multi-policy nested search and front maintenance

The defining architectural change in Pareto-NRPA is the maintenance of a set of policies

$X$9

with $x$0 treated as a user-defined hyperparameter. Each policy is a separate softmax model over move codes, and each is intended to specialize in a different region of the objective trade-off landscape. Policies are not assigned to predefined scalarizations; specialization is expected to emerge from the non-dominated solutions they generate and retain.

At level $x$1, the algorithm samples one policy $x$2 uniformly,

$x$3

then performs the rollout using the standard softmax move-selection rule. The resulting terminal sequence is evaluated on all objectives, and the algorithm stores not only the sequence and its objective vector, but also the identity of the policy that generated it. The uniform policy sampling rule is an explicit design decision intended to equalize the number of evaluations allotted to each policy.

At nonzero search levels, Pareto-NRPA repeatedly calls the next lower level, merges the returned sets of sequences, and prunes the merged archive to its non-dominated subset. The paper describes this pruning step in terms of non-dominated sorting into fronts $x$4, with retention of $x$5, the non-dominated front. Standard NRPA returns one sequence from a nested call; Pareto-NRPA instead returns a non-dominated set.

The archive logic includes a policy-coverage rule that is important for implementation. Policies are adapted only from the solutions they produced, and only if those solutions are in the retained set. To avoid a policy never being updated, the algorithm ensures that at least one solution per policy is retained: if a policy $x$6 has no representative in $x$7, the algorithm scans $x$8 until it finds the first solution generated by $x$9. This prevents policy inactivity and preserves learning pressure across the whole policy set (Lallouet et al., 25 Jul 2025).

4. Pareto-Adapt, crowding distance, and domain bias

The main technical novelty of Pareto-NRPA lies in its adaptation rule. In standard NRPA, a policy is updated toward a single best sequence. In Pareto-NRPA, each policy is updated toward all relevant retained sequences that it generated, with each contribution weighted by the sequence’s isolation in objective space. The algorithm first computes

$y$0

then replays each retained sequence $y$1 from the root through the policy that generated it.

For the move actually taken in a retained sequence, the update is

$y$2

and for every legal move $y$3,

$y$4

with

$y$5

Relative to standard NRPA, the formal difference is the multiplicative factor $y$6 and the fact that the update aggregates over multiple retained sequences rather than one incumbent.

The weight $y$7 is based on the crowding distance from NSGA-II. In the paper’s description, crowding distance approximates the perimeter of the hyper-rectangle defined by nearest neighbors in objective space. Boundary points receive infinite crowding distance, and each interior point accumulates normalized objective gaps. Because extreme points may acquire $y$8 crowding distance, the implementation clips these values to a maximum of $y$9 during adaptation to avoid unstable updates.

The algorithm is also compatible with GNRPA-style domain bias. If a bias term $x \prec y$0 is used, then the partition function becomes

$x \prec y$1

and the negative update term becomes

$x \prec y$2

This is exactly how the paper incorporates the TSPTW distance bias in experiments. The practical intuition presented in the paper is that non-dominance promotes quality, multiple policies promote specialization, and crowding-distance weighting promotes spread along the front (Lallouet et al., 25 Jul 2025).

5. Benchmarks, baselines, and reported performance

The method is evaluated on two classes of problems: a constrained combinatorial benchmark and a multi-objective neural architecture search benchmark. The constrained benchmark is a new bi-objective variant of the Traveling Salesman Problem with Time Windows (MO-TSPTW). Each Solomon-Potvin-Bengio TSPTW instance is augmented with a second cost matrix derived from new Euclidean coordinates, while retaining the original time windows. The two objectives are

$x \prec y$3

where $x \prec y$4 is the number of violated time-window constraints. The penalty $x \prec y$5 ensures that any valid solution dominates any invalid one. The dataset contains $x \prec y$6 instances, and all algorithms are run for $x \prec y$7 objective evaluations with $x \prec y$8 independent runs. For MO-TSPTW, Pareto-NRPA uses level $x \prec y$9, learning rate $f_i(x) \le f_i(y), \quad \forall i=1,\ldots,p,$0, and $f_i(x) \le f_i(y), \quad \forall i=1,\ldots,p,$1, selected by a concise grid search on instance rc_205.2.

The NAS experiments use NAS-Bench-201 and NAS-Bench-101. NAS-Bench-201 contains $f_i(x) \le f_i(y), \quad \forall i=1,\ldots,p,$2 architectures; NAS-Bench-101 contains $f_i(x) \le f_i(y), \quad \forall i=1,\ldots,p,$3. The objectives are

$f_i(x) \le f_i(y), \quad \forall i=1,\ldots,p,$4

so the task is to trade off validation error on CIFAR-10 against parameter count. Each method receives $f_i(x) \le f_i(y), \quad \forall i=1,\ldots,p,$5 objective evaluations and $f_i(x) \le f_i(y), \quad \forall i=1,\ldots,p,$6 runs on each benchmark.

The comparison set consists of NSGA-II, SMS-EMOA, and Pareto-MCTS. The evaluation metrics are hypervolume,

$f_i(x) \le f_i(y), \quad \forall i=1,\ldots,p,$7

overall spread,

$f_i(x) \le f_i(y), \quad \forall i=1,\ldots,p,$8

and spacing,

$f_i(x) \le f_i(y), \quad \forall i=1,\ldots,p,$9

Constraint violations are also reported on MO-TSPTW.

The central empirical claim is that Pareto-NRPA is especially strong on constrained discrete search spaces. On the easy MO-TSPTW instance rc_204.3, NSGA-II and SMS-EMOA are slightly better in hypervolume. On the hard instance rc_201.3, all baselines report $j$0 and $j$1, with nonzero constraint violations, whereas Pareto-NRPA attains $j$2, $j$3, $j$4, and $j$5. On the larger instance rc_204.1, Pareto-NRPA again is the only method reported as consistently satisfying constraints, with $j$6, $j$7, $j$8, and $j$9. Across all $f_j(x) < f_j(y).$0 MO-TSPTW instances, the paper reports greatly superior hypervolume on $f_j(x) < f_j(y).$1 of $f_j(x) < f_j(y).$2 instances; on the $f_j(x) < f_j(y).$3 easiest instances, evolutionary algorithms match or slightly outperform Pareto-NRPA in many cases, whereas on the remaining $f_j(x) < f_j(y).$4 hardest instances, Pareto-NRPA is superior on $f_j(x) < f_j(y).$5.

On NAS-Bench-201, all methods essentially recover the true Pareto front after $f_j(x) < f_j(y).$6 runs, and Pareto-NRPA is competitive but slightly behind NSGA-II and SMS-EMOA by about $f_j(x) < f_j(y).$7 hypervolume. On NAS-Bench-101, Pareto-NRPA slightly outperforms the baselines, with $f_j(x) < f_j(y).$8 and $f_j(x) < f_j(y).$9. The paper explicitly attributes the smaller advantage on NAS to the fact that these search spaces are unconstrained, making MOEAs naturally well suited (Lallouet et al., 25 Jul 2025).

6. Ablations, sensitivities, strengths, and limitations

The supplementary analyses clarify which design choices are carrying the method. One comparison updates each policy using all relevant retained Pareto sequences versus only the single sequence with maximum crowding distance per policy. On rc_204.3, adapting with one sequence yields $S$0, $S$1, and $S$2, while adapting with all sequences yields $S$3, $S$4, and $S$5. This directly supports the paper’s choice of multi-sequence adaptation.

A second ablation compares the method with and without crowding-distance weighting. The reported effect is that crowding-distance weighting generally improves overall spread. The paper gives examples including rc_204.3, rc_204.2, rc_207.3, and rc_201.3, all showing better $S$6 under the weighted update. Hyperparameter sensitivity is studied over $S$7 and $S$8 on rc_205.2, followed by a comparison of levels $S$9 and $$S^* = \{ s \in S \mid \nexists u \in S : u \prec s \}. \tag{2}$$0. The chosen default is $$S^* = \{ s \in S \mid \nexists u \in S : u \prec s \}. \tag{2}$$1, $$S^* = \{ s \in S \mid \nexists u \in S : u \prec s \}. \tag{2}$$2, level $$S^* = \{ s \in S \mid \nexists u \in S : u \prec s \}. \tag{2}$$3, and level $$S^* = \{ s \in S \mid \nexists u \in S : u \prec s \}. \tag{2}$$4 clearly outperforms level $$S^* = \{ s \in S \mid \nexists u \in S : u \prec s \}. \tag{2}$$5 on rc_205.2.

The strengths claimed in the paper are correspondingly specific: strong performance on constrained discrete sequential optimization, a direct extension of NRPA without scalarization, and the ability to produce diverse Pareto approximations through multiple policies and crowding-distance-weighted adaptation. The reported empirical story is that policy adaptation helps the method learn move subsequences that reach feasible and promising regions more quickly in constrained search spaces.

The limitations are equally explicit. Diversity is not always as good as in MOEAs, because Pareto-NRPA maintains non-dominated fronts rather than large populations and does not explicitly regularize separation between policies. Policy specialization can therefore be uneven, with some runs showing clean separation across objective space and others showing overlap. The method is currently targeted at discrete search spaces and is demonstrated mainly in bi-objective settings. The paper also notes that implementation is more involved than vanilla NRPA because archive maintenance, replay over multiple sequences, and multi-policy bookkeeping are required. For reproducibility, the paper highlights several operational details as decisive: uniform sampling over policies at level $$S^* = \{ s \in S \mid \nexists u \in S : u \prec s \}. \tag{2}$$6, storing the generating policy with each sequence, retaining at least one sequence per policy, adapting each policy using all retained sequences it generated, weighting by crowding distance, clipping crowding distance to $$S^* = \{ s \in S \mid \nexists u \in S : u \prec s \}. \tag{2}$$7, and incorporating domain bias directly into the softmax terms when such bias is available (Lallouet et al., 25 Jul 2025).

7. Related methods and terminological boundaries

Pareto-NRPA is situated within a broader set of Pareto-front maintenance problems. In particular, exact algorithms for computing Pareto sums of two bi-criteria Pareto sets—such as Successive Sweep Search with complexity $$S^* = \{ s \in S \mid \nexists u \in S : u \prec s \}. \tag{2}$$8 and Sort & Compare with complexity $$S^* = \{ s \in S \mid \nexists u \in S : u \prec s \}. \tag{2}$$9, both using $X$00 space—address the subproblem of combining partial non-dominated fronts without materializing the full Cartesian product (Funke et al., 2024). This suggests one possible extension path for Pareto-NRPA variants that would combine partial fronts additively in bi-criteria settings, although such operators are not part of the published Pareto-NRPA algorithm itself.

A separate source of ambiguity is terminology. “Pareto-NRPA” refers to the Monte-Carlo multi-objective generalization of NRPA for discrete search. By contrast, “PARO” denotes Pareto Adaptive Robustly Optimality in two-stage linear adaptive robust optimization, where the concern is scenario-wise dominance among worst-case optimal adaptive policies rather than nested rollout policy adaptation (Bertsimas et al., 2020). The shared Pareto vocabulary therefore conceals two distinct research programs: one in Monte-Carlo search over discrete sequences, the other in adaptive robust optimization.

Within its own domain, Pareto-NRPA is best understood as a direct but nontrivial NRPA generalization whose essential innovations are a policy set $X$01, non-dominated front maintenance at each nested level, adaptation from all retained Pareto sequences, and crowding-distance-weighted policy learning. Its strongest reported performance is on multi-objective, discrete, sequential, and especially constrained optimization problems, where the combination of nested rollouts and policy adaptation appears particularly effective at reaching feasible high-quality Pareto sets (Lallouet et al., 25 Jul 2025).