Pareto GFlowNets for Multi-Objective Optimization

• Pareto GFlowNets are generative models that target multi-objective optimization by sampling diverse, non-dominated solutions along the Pareto front.
• They leverage conditional mechanisms such as scalarization, goal-conditioning, and order-preserving losses to overcome limitations in covering concave or disconnected regions.
• Empirical studies show enhanced hypervolume, uniform front coverage, and faster convergence compared to traditional evolutionary and reinforcement learning approaches in design tasks.

Pareto GFlowNets are a class of generative models specifically designed for multi-objective optimization (MOO), targeting the principled exploration and sampling of diverse, high-performance solutions along the Pareto front. They are built upon the framework of Generative Flow Networks (GFlowNets), leveraging conditional or order-based mechanisms to address fundamental limitations of scalarization-based MOO, especially in domains such as molecular and sequence design where objectives are conflicting and fronts are non-convex.

1. Multi-Objective Optimization and the Pareto Criterion

Multi-objective problems in machine learning often require optimizing several, potentially conflicting, quantitative properties of candidate solutions. Each object $x \in \mathcal X$ is mapped to a vector-valued objective $\mathbf R(x) = (r_1(x), \dots, r_K(x))$, and the challenge is to generate or select objects that are non-dominated: i.e., no $x'$ exists with $r_k(x') \ge r_k(x)$ for all $k$ and $r_{k'}(x') > r_{k'}(x)$ for some $k'$. The Pareto front is the set of such non-dominated objective vectors.

Traditional approaches perform scalarization by collapsing multiple objectives to a single, parameterized reward, typically $R_w(x) = \sum_{k=1}^K w_k r_k(x)$ for preferences $w \in \Delta^{K-1}$. However, the Pareto front may be non-convex; uniform sampling of $w$ can under-cover interior (especially concave) regions, biasing exploration toward extremes (Jain et al., 2022, Bengio et al., 2021, Roy et al., 2023).

2. Generative Flow Networks for Multi-Objective Sampling

GFlowNets learn policies that sample object $\mathbf R(x) = (r_1(x), \dots, r_K(x))$0 with probability proportional to a reward $\mathbf R(x) = (r_1(x), \dots, r_K(x))$1 via a sequence of actions in a DAG. Trajectory-balance and flow-matching losses ensure the marginals $\mathbf R(x) = (r_1(x), \dots, r_K(x))$2. Conditional versions allow inputting preferences (scalarization weights or goals), learning an amortized sampler $\mathbf R(x) = (r_1(x), \dots, r_K(x))$3 for any $\mathbf R(x) = (r_1(x), \dots, r_K(x))$4 (Bengio et al., 2021, Jain et al., 2022, Zhu et al., 2023).

Two major instantiations of Pareto GFlowNets have emerged:

• Preference-Conditional GFlowNets (MOGFN-PC): The model is conditioned on a scalarization, typically a weight vector $\mathbf R(x) = (r_1(x), \dots, r_K(x))$5 on the simplex. Conditioning enables a single policy to explore the front by varying $\mathbf R(x) = (r_1(x), \dots, r_K(x))$6. Training is driven by trajectory-balance or flow-matching losses, with the reward $\mathbf R(x) = (r_1(x), \dots, r_K(x))$7. At test time, sampling $\mathbf R(x) = (r_1(x), \dots, r_K(x))$8 for diverse $\mathbf R(x) = (r_1(x), \dots, r_K(x))$9 yields front coverage (Jain et al., 2022, Bengio et al., 2021, Zhu et al., 2023, Laajil et al., 4 Oct 2025).
• Goal-Conditioned GFlowNets: Rather than weighting objectives, the model is conditioned on regions (cones) in the objective space, specified by direction $x'$0 and angle $x'$1. Rewards are nonzero only if $x'$2’s objectives fall into the specified region. Adaptive goal samplers such as Uniform-GS and Tabular-GS ensure uniform or feasible coverage of the front—even for concave geometry (Roy et al., 2023).

GFlowNets thus provide a flexible framework for generating diverse candidates suited to MOO challenges, with or without explicit scalarization, and are compatible with active learning and Bayesian optimization (Zhu et al., 2023).

3. Advances Beyond Scalarization: Order-Conditioned and Goal-Focused Variants

Scalarization-based conditioning is limited for fronts with concave or disconnected geometry: uniform weight sampling yields poor central coverage. Pareto GFlowNets address this by introducing:

• Goal-Conditioning: Directly specifies a target region in objective space, enforcing that generated samples concentrate within a desired trade-off cone ($x'$3). By sampling $x'$4 rather than $x'$5, one recovers full front coverage, including interiors (Roy et al., 2023).
• Order-Preserving GFlowNets (OP-GFN): Instead of defining a reward, OP-GFNs learn a proxy $x'$6 such that, locally or batch-wise, distributions are uniform on the Pareto set; the order-preserving KL loss ensures $x'$7 when $x'$8 (Chen et al., 2023). This removes manual tuning of reward exponents and automatically adapts the effective sampling sharpness.
• Global-Order GFlowNets: Address contradictions inherent to local (batch-wise) Pareto set constraints by imposing a global total (or weakly total) order consistent with the partial Pareto order, using ranking algorithms (global rank, nearest-neighbor). The reward function $x'$9 is globally defined, avoiding infeasible constraints and enhancing consistency (Pastor-Pérez et al., 3 Apr 2025).

Conditioning/Ordering	Mechanism	Coverage Behavior
Scalarization	Preferences $r_k(x') \ge r_k(x)$0	Good for convex regions
Goal-conditioning	Direction/region $r_k(x') \ge r_k(x)$1	Uniform coverage, incl. concave
Order-preserving (OP)	Batchwise non-dominated sorting	Adaptive, sharpens over time
Global-order	Total/weak order (rank, NN)	Consistent, avoids contradictions

4. Algorithmic and Training Methodologies

Pareto GFlowNets can employ various flow-matching and trajectory-balance losses. In the conditional setting, policies are functions of both the generative state and the conditioning (preference or goal):

• Trajectory Balance (TB) Loss: Ensures sample probabilities match the terminal reward conditioned on input.
• Goal Sampling: Adaptive samplers (Tab-GS) select feasible/fruitful regions, preventing collapses to infeasible goals.
• Replay Buffers and Hindsight Relabeling: Essential for sample efficiency and stable credit assignment, especially under non-stationary conditioning (Roy et al., 2023).
• Hypernetworks: Used for generating policy weights adaptively conditioned on preference vectors, facilitating efficient amortization (Zhu et al., 2023).

In curriculum-augmented settings, learning focus is guided by smoothed learning progress metrics to balance easy and difficult sub-tasks, e.g., via protein length intervals in mRNA sequence design (Laajil et al., 4 Oct 2025).

Typical hyperparameters:

• Batch size: $r_k(x') \ge r_k(x)$264
• Forward learning rate: $r_k(x') \ge r_k(x)$3
• Partition function learning rate: $r_k(x') \ge r_k(x)$4
• Replay buffer size: $r_k(x') \ge r_k(x)$5
• Hindsight fraction: 0.3
• Steps: $r_k(x') \ge r_k(x)$640k
• Curriculum schedules: per-task learning progress control (Laajil et al., 4 Oct 2025)

5. Empirical Evaluation and Performance Landscape

Evaluation of Pareto GFlowNets uses both Pareto and diversity metrics:

• Inverted Generational Distance (IGD): Front coverage quality.
• Pareto-Clusters Entropy (PC-ent): Uniformity of sample distribution along the front.
• Avg-PCC: Control of objective alignment with conditioning.
• Hypervolume (HV), R₂-indicator, Top-K diversity, Generational Distance$r_k(x') \ge r_k(x)$7 ($r_k(x') \ge r_k(x)$8): Standard MOO benchmarks.

Key observed phenomena:

• Concave/Multi-concave Fronts: Scalarization collapses to extremes. Goal-conditioned and OP-GFN variants span the entire front (PC-ent $r_k(x') \ge r_k(x)$9, Avg-PCC $k$0) (Roy et al., 2023, Chen et al., 2023).
• High-dimensional Objective Spaces: Uniform coverage by scalarization is inefficient; adaptive or order-based goal selection mitigates this (Roy et al., 2023, Pastor-Pérez et al., 3 Apr 2025).
• Sample Efficiency: Pareto GFlowNets outperform evolutionary and RL baselines in hypervolume, diversity, and speed of convergence, even on structured tasks (DNA, molecules, neural architectures) (Zhu et al., 2023, Chen et al., 2023, Laajil et al., 4 Oct 2025).

Selected empirical results from (Roy et al., 2023, Jain et al., 2022, Zhu et al., 2023), and (Chen et al., 2023):

• On 4-objective molecular tasks, Tab-GS goal-conditioning reaches IGD $k$1 (vs $k$2 for preference-conditioning), PC-ent $k$3 (vs $k$4), Avg-PCC $k$5 (vs $k$6).
• Curriculum-augmented GFlowNets converge $k$7 (vs random) to $k$8 (vs long-only) faster in out-of-distribution sequence design while maintaining Pareto coverage and diversity (Laajil et al., 4 Oct 2025).

6. Limitations, Open Questions, and Extensions

Despite empirical success, open issues and limitations include:

• Sample Inefficiency in High-dimensional or Infeasible Goals: Sampling uniformly over goals or conditions can waste effort on impossible sub-regions. Hierarchical or adaptive samplers (e.g., GFN-GS) are a proposed remedy (Roy et al., 2023).
• Contradictions in Local Order Enforcement: Local order-preserving objectives may be mutually inconsistent; global-ordering remedies this but trade-offs in exploration require further analysis (Pastor-Pérez et al., 3 Apr 2025).
• Scaling to Large/Continuous Spaces: While global-rank and nearest-neighbor methods alleviate batching pathology, their cost and the optimality of totalizations remain open.
• Diversity-Performance Trade-off: Adapting the reward exponent (or proxy-reward sharpness) is automated in order-based GFlowNets, but hyperparameter-free approaches are under study (Chen et al., 2023).
• Generalization Across Structured Domains: Extending current mechanisms to graphs, programs, or combinatorially constrained domains is ongoing (Pastor-Pérez et al., 3 Apr 2025, Laajil et al., 4 Oct 2025).

Future work includes hierarchical samplers for goal selection, theoretical characterization of orderings for maximal front coverage, and integration with active learning and surrogate-assisted acquisition functions in black-box optimization scenarios.

7. Connections to Broader Methodologies and Applications

Pareto GFlowNets unify concepts from multi-objective reinforcement learning, Bayesian optimization, and generative modeling. Key innovations:

• Unified, Amortized Sampling: A single conditional generative model can efficiently cover diverse Pareto-optimal trade-offs across conditioning parameters (Jain et al., 2022, Bengio et al., 2021).
• Robustness Across Structured Domains: Demonstrated performance in molecules, DNA/protein sequences, and neural architectures (Chen et al., 2023, Pastor-Pérez et al., 3 Apr 2025, Laajil et al., 4 Oct 2025).
• Compatibility with Active Learning and Curriculum Learning: GFlowNet approaches have been successfully combined with Bayesian surrogate-guided acquisition and curriculum progression according to task difficulty (Zhu et al., 2023, Laajil et al., 4 Oct 2025).

This positions Pareto GFlowNets as a robust class of methods for tractable, diversity-promoting multi-objective discovery in discrete and structured design spaces.