Learning Fair Pareto-Optimal Policies in Multi-Objective Reinforcement Learning

Abstract: Fairness is an important aspect of decision-making in multi-objective reinforcement learning (MORL), where policies must ensure both optimality and equity across multiple, potentially conflicting objectives. While single-policy MORL methods can learn fair policies for fixed user preferences using welfare functions such as the generalized Gini welfare function (GGF), they fail to provide the diverse set of policies necessary for dynamic or unknown user preferences. To address this limitation, we formalize the fair optimization problem in multi-policy MORL, where the goal is to learn a set of Pareto-optimal policies that ensure fairness across all possible user preferences. Our key technical contributions are threefold: (1) We show that for concave, piecewise-linear welfare functions (e.g., GGF), fair policies remain in the convex coverage set (CCS), which is an approximated Pareto front for linear scalarization. (2) We demonstrate that non-stationary policies, augmented with accrued reward histories, and stochastic policies improve fairness by dynamically adapting to historical inequities. (3) We propose three novel algorithms, which include integrating GGF with multi-policy multi-objective Q-Learning (MOQL), state-augmented multi-policy MOQL for learning non-statoinary policies, and its novel extension for learning stochastic policies. We evaluate our algorithms across various domains and compare our methods against the state-of-the-art MORL baselines. The empirical results show that our methods learn a set of fair policies that accommodate different user preferences.

• The paper introduces a GGF-based framework that integrates fairness into multi-objective reinforcement learning, ensuring Pareto optimality with fairness constraints.
• It presents three scalable algorithms (F-MDQ, FN-MDQ, FNS-MDQ) evaluated on diverse domains, achieving balanced outcomes and superior GGF scores.
• The study reveals that non-stationary and stochastic policies can outperform deterministic ones in fairness metrics for complex multi-objective tasks.

Learning Fair Pareto-Optimal Policies in Multi-Objective Reinforcement Learning

Introduction and Motivation

Multi-objective reinforcement learning (MORL) is increasingly required for real-world tasks where agents must optimize multiple, often conflicting, objectives simultaneously. Conventional MORL approaches primarily optimize for Pareto optimality via linear scalarization, which fails to guarantee any fairness properties between objectives. The study "Learning Fair Pareto-Optimal Policies in Multi-Objective Reinforcement Learning" (2606.18111) addresses this gap by developing a rigorous framework grounded in social welfare theory, with the goal of learning a diverse set of fair policies suited for varying user preference profiles.

The central thesis is the integration of generalized Gini welfare functions (GGF) into the multi-policy MORL paradigm, in contrast to prior efforts which focus almost exclusively on fixed-preference, single-policy settings. The result is a set of algorithms that can produce policies that are not only Pareto optimal but also fair across objectives as preferences change.

Fairness Formulation in MORL

Traditional efficiency- and Pareto-centric MORL only guarantee that learned solutions are not strictly dominated, but do not account for equity or impartiality. The study formalizes fairness in MORL using three established social choice properties:

• Efficiency (Pareto dominance): No objective can be improved without sacrificing another.
• Equity (Pigou-Dalton principle): Transfers from more to less advantaged improve fairness.
• Impartiality: Solutions are invariant to permutation of equivalent objectives.

GGF is chosen as the fairness criterion, combining objectives with strictly decreasing, positive weights over the sorted outcome vector, ensuring Schur-concavity and adherence to all three fairness properties.

Figure 1: Examples of $2$-objective MOMDP where the generalized Gini welfare function (GGF) steers learned policies toward fairer allocations between objectives.

Theoretical Analysis: Sufficiency and Optimality

A key theoretical contribution is the characterization of the sufficiency of the convex coverage set (CCS) for fair policy selection, exploiting the concavity and piecewise linearity of GGF. The results are as follows:

• Sufficiency of the CCS: For piecewise-linear concave welfare functions such as GGF, the set of optimal policies for any user preference lies within the CCS. This is non-trivial as GGF introduces nonlinearity into the scalarization process.
• Non-stationary Policy Fairness: By augmenting state with accrued rewards, non-stationary policies can yield higher welfare scores than any stationary policy when dealing with inequities accumulated over a trajectory.
• Stochastic Policy Advantage: There exist cases where no deterministic policy achieves maximal GGF; mixing deterministic CCS policies into a stochastic policy can yield strictly fairer solutions.

  Figure 2: Left: Pigou-Dalton transfers and GGF scalarization ensure fairer solutions beyond mere Pareto-dominance. Right: The mixture of deterministic CCS policies (dotted points) achieves GGF-optimal solutions unattainable by any single deterministic policy.

These results confirm that simply enumerating Pareto-optimal deterministic policies is insufficient for fairness; explicit construction of non-stationary and stochastic policies is often necessary.

Algorithmic Framework

The authors introduce three practical algorithms with a single shared neural architecture:

• F-MDQ: An Envelope Q-learning adaptation that replaces linear scalarization with GGF. It learns stationary, deterministic policies over the sampled preference space.
• FN-MDQ: Extends F-MDQ for non-stationary policies, where the state is augmented with trajectory-level accrued rewards to enable adaptive, fairness-aware action selection.
• FNS-MDQ: Further extends FN-MDQ by enabling stochastic action selection, sampling from a probability distribution over actions to resolve fairness ties and leverage mixture policies for superior fairness.

Novel resampling schemes akin to HER are employed for efficiency in handling the preference space. All methods share a single parameterized network, ensuring scalability.

Empirical Analysis

The proposed methods are evaluated on three distinct domains, each characterized by multiple objectives with inherent fairness challenges:

• Species Conservation (SC): Population management for two interacting endangered species.
• Resource Gathering (RG): Agent must collect and balance multiple valuable resources in a stochastic gridworld.
• Multi-Product Web Advertising (MWP): Balancing advertisement exposure and reward over $N=7$ product types.

Key performance metrics include total reward, minimum/maximum objective value, coefficient of variation (CV), and GGF score.

Figure 3: Training metrics in the species conservation domain, highlighting total density, CV, and extremal densities.

The results demonstrate:

• F-MDQ, FN-MDQ, and especially FNS-MDQ consistently produce lower CV (more balanced outcomes), higher minimum per-objective rewards, and superior GGF scores compared to Envelope, GPI-LS, and PCN baselines.
• FNS-MDQ yields the highest observed GGF scores, confirming the theoretical advantage of stochastic policies for fairness.
• The methods generalize to unseen preference weights during evaluation, with Envelope exhibiting a lack of fairness in such scenarios, while the proposed methods maintain equity.

Practical and Theoretical Implications

This work introduces an actionable method for deploying multi-policy MORL systems in domains where fairness is a requirement—such as multi-user resource allocation, public resource management, or personalized recommendation—without prior specification of user preferences. The theoretical guarantees also clarify the necessary role of stochasticity and non-stationarity in MORL systems that aim at fairness rather than merely efficiency.

The algorithms retain scalability via a single network, essential for large-scale domains, and maintain Pareto diversity and hypervolume on par with baselines, ensuring no loss in coverage of solutions.

Future Directions

Several avenues for extension are suggested:

• Adaptation to continuous action spaces, where fairness constraints may interact with optimality in nontrivial ways.
• Approximating the non-convex regions of the Pareto front where CCS-based methods may falter.
• Generalizing beyond individual to group or intersectional fairness metrics, and integrating safety or risk bounds in the fairness-aware MORL paradigm.

Conclusion

"Learning Fair Pareto-Optimal Policies in Multi-Objective Reinforcement Learning" sets a new standard for integrating axiomatically justified fairness criteria into scalable deep RL frameworks for MORL. The deployment of GGF within multi-policy settings, supported both theoretically and empirically, demonstrates that fairness and Pareto efficiency can be simultaneously attained with scalable solutions, provided that non-stationarity and stochasticity are explicitly considered.

Figure 4: Results for total resources, CV, and extremal values in the resource gathering scenario, illustrating increased minimum resource collection and reduced variance for the proposed methods.

Figure 5: Total rewards, minimum, and maximum per-objective outcomes for the multi-product web advertising domain, validating fair allocation across all products.

This work positions fairness as a first-class design parameter in MORL and necessitates further study on fairness-aware, policy-diverse RL for multi-agent and high-stakes applications.