Beyond Compromise: Pareto-Lenient Consensus for Efficient Multi-Preference LLM Alignment

Abstract: Transcending the single-preference paradigm, aligning LLMs with diverse human values is pivotal for robust deployment. Contemporary Multi-Objective Preference Alignment (MPA) approaches predominantly rely on static linear scalarization or rigid gradient projection to navigate these trade-offs. However, by enforcing strict conflict avoidance or simultaneous descent, these paradigms often prematurely converge to local stationary points. While mathematically stable, these points represent a conservative compromise where the model sacrifices potential global Pareto improvements to avoid transient local trade-offs. To break this deadlock, we propose Pareto-Lenient Consensus (PLC), a game-theoretic framework that reimagines alignment as a dynamic negotiation process. Unlike rigid approaches, PLC introduces consensus-driven lenient gradient rectification, which dynamically tolerates local degradation provided there is a sufficient dominant coalition surplus, thereby empowering the optimization trajectory to escape local suboptimal equilibrium and explore the distal Pareto-optimal frontier. Theoretical analysis validates PLC can facilitate stalemate escape and asymptotically converge to a Pareto consensus equilibrium. Moreover, extensive experiments show that PLC surpasses baselines in both fixed-preference alignment and global Pareto frontier quality. This work highlights the potential of negotiation-driven alignment as a promising avenue for MPA. Our codes are available at https://anonymous.4open.science/r/aaa-6BB8.

• The paper presents a dynamic coalition-based framework that uses lenient gradient filtering to optimize multi-preference alignment in large language models.
• The proposed method significantly improves Pareto frontier quality and controllability over static baselines, as demonstrated in bi- and tri-objective experiments.
• Empirical results show fine-grained preference control and robust performance under reward corruption, ensuring scalable and effective LLM alignment.

Pareto-Lenient Consensus: Dynamic Negotiation for Multi-Preference LLM Alignment

Motivation and Problem Formulation

Traditional alignment methods for LLMs have predominantly targeted single reward maximization or, more recently, static approaches for multi-objective preference alignment (MPA). Static linear scalarization and rigid multi-gradient projection methods often converge rapidly to risk-averse equilibria, wherein the optimization plateau arises from strict conflict-avoidance between conflicting preference gradients. Such equilibria are mathematically stationary but typically suboptimal, as traversing local valleys associated with transient individual objective regression could yield substantial coalition gains. The motivating insight is to harness local, consensus-driven lenience, tolerating temporary minority objective setbacks for Pareto frontier advances.

The Pareto-Lenient Consensus (PLC) Framework

PLC conceptualizes multi-objective LLM alignment as a dynamic cooperative game. Each preference is an independent agent contributing its policy gradient. Unlike rigid paradigms, PLC introduces a lenient gradient rectification mechanism based on coalition consensus. At each optimization step, the coherent coalition (objectives with positive advantages) and the conflict set are identified. PLC dynamically computes a lenient mask using a temperature-controlled sigmoid, which temporarily suppresses conflicting (negative advantage) gradients—provided the coalition surplus, i.e., the potential gain for the majority, exceeds a prescribed threshold.

This constructs a lenient update trajectory, allowing exploration across preference valleys while preventing the optimization from stagnating at the boundary of local stationary points. As optimization proceeds and the coalition surplus naturally depletes near true Pareto Consensus Equilibria (PCE), the masking decays, reverting to standard (local) gradient-based optimization.

Figure 1: The PLC framework leverages coalition-based lenient filtering, overcoming early gradient stalemates that trap static baseline methods, and enabling optimization to advance towards the true Pareto frontier.

Theoretical Properties

The paper provides formal guarantees substantiating the two essential properties of PLC:

1. Gradient Recovery: At risk-averse equilibria where rigid aggregation yields vanishing gradients, PLC maintains a nonvanishing update direction by projecting confounding gradients onto the "lenient manifold," proportional to the coalition surplus. This ensures that optimization can destabilize and escape locally stationary points.
2. Asymptotic Convergence: As the latent coalition surplus shrinks with progression towards the PCE, the leniency decay ensures convergence, i.e., the PLC update direction vanishes asymptotically, recovering local optimality.

Empirical Evaluation

Pareto Frontier Quality and Multi-Objective Trade-Offs

Experiments on Anthropic-hh-rlhf and BeaverTails cover both bi-objective (helpful, harmless; helpful, humor) and tri-objective (helpful, harmless, humor) settings. PLC is compared with static RLHF optimizers, model-interpolation strategies (Rewarded Soups), gradient-adaptive methods (GAPO), and context-based alignment (RiC). Metrics include proxy rewards, LLM-as-a-Judge ratings, Hypervolume (HV), IGD, Maximum Spread, and Preference Compliance.

PLC demonstrates consistent and significant improvements in Pareto frontier coverage (HV, spread, and IGD) and controllability (Compliance). Notably, in the tri-objective setting, PLC achieves substantially broader and more convex Pareto frontiers, avoiding collapse to single preference optima prevalent in rigid baselines.

Figure 2: PLC consistently achieves a superior Pareto boundary across multiple datasets and objectives. LLM-judge scores under balanced preferences highlight superior equilibrium quality.

Figure 3: PLC dominates multi-objective metrics, confirming advances in frontier diversity and controllability relative to competing methods.

Figure 4: Pareto frontier and LLM-judge scores in the Humor vs. Helpful setting reinforce PLC's consistent advantage.

Figure 5: PLC’s outperformance in multi-objective metrics generalizes across datasets and objective pairs.

Figure 6: 3D Pareto manifold visualization for tri-objective optimization, evidencing PLC’s ability to maintain simultaneous high-reward solutions across all objectives with controllable trade-off configurations.

Fine-Grained Controllability and Robustness

PLC's dynamic masking enables high-fidelity preference controllability. Reward distribution histograms under varying preference weights demonstrate monotonic alignment responsiveness, unlike static scalarization-based approaches.

Figure 7: Shifting reward distributions with varying weights demonstrate PLC’s fine-grained controllability and effective alignment with user priorities.

PLC is also robust to reward corruption: even under significant stochastic perturbations to reward signals, PLC preserves higher average performance compared to linear scalarization, indicating resilience to unreliable preference proxies.

Figure 8: Under noisy reward conditions, PLC consistently outperforms static linear aggregation, attesting to its practical robustness.

Ablations and Sensitivity

Ablation on lenient filtering reverts PLC to a linear scalarization analogue, which succumbs to the risk-averse trap and achieves inferior rewards. Sensitivity analyses on the mask temperature parameter $\tau$ reveal that sharp consensus boundaries (low $\tau$) are crucial for effective escape from local optima, while excessive smoothness impairs discriminative gradient filtering.

Broader Implications and Future Research

PLC establishes dynamic, negotiation-driven optimization as a more effective, practical alternative for scalable multi-objective alignment in LLMs. It offers precise and broad controllability, avoids the dialectic deadlock of static compromise, and is well-suited for real-world deployment where alignment with heterogeneous human values is critical.

Theoretically, PLC motivates extensions in multi-agent RL and cooperative game-theoretic methods for high-dimensional preference alignment. Future work should pursue standardized, model-agnostic evaluation protocols for preference vector adherence, adaptive leniency decay schedules for highly non-convex landscapes, and integration with richer reward modeling beyond proxy-based surrogates.

Conclusion

Pareto-Lenient Consensus introduces a scalable, mathematically principled framework for MPA, enabling LLMs to efficiently traverse the nonconvex landscape of conflicting human values. By treating preference alignment as consensus-driven negotiation among objectives and incorporating coalition-based lenient filtering, PLC circumvents the limitations of rigid aggregation, reaching broader and more controllable Pareto frontiers, as validated empirically and theoretically. This paradigm will likely inform and inspire successor frameworks in both foundation model alignment and multi-agent optimization research.

Figure 9: PLC achieves dominant scores across tri-objective multi-objective metrics, most notably in hypervolume, demonstrating an expansion of the solution space.

Figure 10: Pairwise 2D projections show PLC’s consistently superior convex hull across all tri-objective subspaces compared with baseline methods.

See original manuscript for comprehensive reference listing.