APPA: Adaptive Preference Pluralistic Alignment for Fair Federated RLHF of LLMs

Abstract: Aligning LLMs with diverse human preferences requires pluralistic alignment, where a single model must respect the values of multiple distinct groups simultaneously. In federated reinforcement learning from human feedback (FedRLHF), these groups align a shared policy without centralizing preference data, which makes fair reward aggregation essential. Existing aggregation methods exhibit clear trade offs: average based aggregation systematically under aligns worst performing groups, while min aggregation prioritizes worst group performance at the cost of overall alignment. We propose APPA, an Adaptive Preference Pluralistic Alignment framework that dynamically reweights group level rewards based on historical alignment rewards. Our approach prioritizes under aligned groups without degrading well aligned ones, while requiring no access to raw preference data. Integrated into a proximal policy optimization (PPO) based FedRLHF pipeline and evaluated on GLOBALQA and OQA across three model families (Gemma 2 2B, Llama 3.2 3B, Qwen3 0.6B), APPA achieves strong fairness alignment trade offs, improving worst group alignment by up to 28% over average aggregation while maintaining higher overall alignment than min aggregation across most configurations.

• The paper introduces APPA, using dynamic group-specific weighting to balance global utility and worst-group fairness in federated RLHF settings.
• It leverages adaptive aggregation with metrics like JS divergence and Wasserstein distance, achieving up to 28% improvement in worst-group alignment.
• The approach maintains privacy by transmitting only group-level rewards, setting a new baseline for scalable, fair LLM alignment protocols.

Adaptive Preference Pluralistic Alignment for Fair Federated RLHF of LLMs

Motivation and Problem Formulation

Alignment of LLMs with human preferences fundamentally requires pluralistic alignment—specifically, policies that encode the values and social norms of heterogeneous user subpopulations. In pluralistic alignment, distinct demographic, cultural, or geographic groups each possess non-identical, and often conflicting, preference structures. Federated RLHF (FedRLHF) enables these groups to jointly train a shared model via local reward generation, avoiding the privacy and centralization concerns of raw data pooling. However, the aggregation of group-level rewards becomes the central challenge: naive averaging enforces majority dominance and neglects worst-group alignment, while minimax strategies collapse group contributions to the worst-served, typically at the expense of overall quality. This work targets robust, privacy-preserving optimization of the fairness–utility Pareto frontier for pluralistic alignment, under the strict constraint that only group-level reward signals (never raw preference data or local model parameters) are transmitted for aggregation.

Method: Adaptive Preference Pluralistic Alignment (PP)

The proposed framework, Adaptive Preference Pluralistic Alignment (PP), inserts an adaptive, group-weighted aggregation module into a PPO-based FedRLHF pipeline. Each federated group $g$ maintains a frozen PluralLLM preference predictor, locally inferring group-specific distributional targets from a few-shot local context. Given global rollout data, each group returns local rewards $r_{g,j}$ for each rollout item, computed using metrics such as Jensen–Shannon (JS) divergence, Wasserstein distance (Was.), and cosine similarity, normalized to $[0,1]$. At the server, PP adaptively aggregates $\{r_{g, j}\}_g$ using history-aware group-specific weights $\alpha^t_g$.

Figure 1: Overview of the PP framework: federated RLHF for pluralistic alignment of group preferences with adaptive dynamic reward aggregation.

The distinguishing algorithmic components are as follows:

• Dynamic group-specific weighting: For each PPO update, a reversed softmax over exponentially moving averaged historical group alignment scores $h_g^t$ amplifies the weight of under-aligned groups while ensuring all groups (including well-aligned) retain non-zero influence. The temperature parameter $T$ governs the sharpness of this ‘focusing’ dynamic.
• Fairness Index (FI) gating: When instantaneous reward dispersion across groups is highly uniform ($FI \geq \tau$), the aggregator collapses to a simple average for stability; otherwise, adaptive log-sum-exp reweighting is in force. This enables seamless transition as group alignment disparities shrink.
• Privacy preservation: Only group-level rewards are transmitted, never raw preferences or local contextual samples, preserving federated privacy guarantees.

The aggregation is formulated as:

$$\mathrm{Agg}_{\boldsymbol{\alpha}^t}(\mathbf{r}_j^t) = \begin{cases} \frac{1}{N}\sum_{g} r_{g,j}^t & \text{if } FI \geq \tau \ \log\left(\frac{1}{N} \sum_{g} \exp(\alpha_g^t r_{g,j}^t)\right) & \text{otherwise} \end{cases}$$

where $\alpha_g^t$ is computed via a reversed softmax of $r_{g,j}$0, ensuring that underperforming groups are adaptively upweighted.

Empirical Results

Datasets, Tasks, and Models

Experiments span GLOBALQA (cross-national, nominal response distributions) and OQA (US demographic, ordinal response sets), each partitioned into federated groups (countries or demographic strata, respectively). Three instruction-tuned LLMs are evaluated: Gemma-2-2B, Llama-3.2-3B, and Qwen3-0.6B, representing a broad range of parameter capacities.

Fairness–Alignment Trade-off

The core evaluation concerns global (average) and worst-group (minimum) alignment scores, using per-question, per-group reward metrics defined above. All results are computed on held-out test sets.

• Quantitative gains: PP improves worst-group (min) alignment by up to 28% over average aggregation, while maintaining higher or equal average alignment compared with minimax across almost all configurations. For instance, on GLOBALQA DPA (JS), PP achieves Min AS/Avg AS of 0.843/0.861 for Gemma-2-2B with $r_{g,j}$1. On OQA, PP achieves 0.842/0.872 for Gemma-2-2B (Wasserstein).
• No alignment collapse: Unlike Min, which often substantially reduces average alignment in order to optimize the worst group, PP avoids the extreme utility–fairness trade-off by continuing to utilize rewards from all groups, but with adaptively-lifted signal from lagging groups.

  Figure 2: Per-group alignment score comparison across major aggregation strategies, showing PP yields both higher and more uniform (circular) group-level performance.

Fairness–Alignment Pareto Visualization

Scatter analysis of the (\text{FI}, Min AS) plane demonstrates that PP consistently achieves near-Pareto-optimal joint fairness and worst-group alignment. Average aggregation strategies are displaced downward in Min AS, and Min aggregation, while occasionally matching FI, typically yields lower Avg AS.

Figure 3: Pareto plot of Fairness Index vs. Minimum Alignment Score (GLOBALQA), showing PP (red stars) occupying the upper-right region for all model and aggregation combinations.

Figure 4: Fairness–Alignment Trade-off on OQA; PP remains near the Pareto frontier across capacity regimes.

Theoretical Properties

• The piecewise adaptive aggregation produces a valid, scalar reward signal that is compatible with the PPO objective and maintains trajectory- or batch-level differentiability as required for gradient ascent.
• The reversed softmax dynamic ensures self-correction: as an under-aligned group’s reward rises, the focus shifts smoothly to any remaining lagging groups. No group’s reward dominates permanently or is permanently suppressed.

Implications and Future Directions

This work demonstrates that strictly reward-level federated RLHF can achieve high-fidelity pluralistic alignment, without requiring sensitive raw data or centralized per-group labels. The adaptive reward aggregation paradigm introduces a new regime for balancing utility/fairness Pareto curves in LLM alignment, with direct applicability to other privacy-critical multi-stakeholder optimization domains. Notably:

• Practical significance: Ready integration with LLM training protocols, preservation of privacy, and computational efficiency render PP suitable for deployment in real-world multi-group dialogue, survey, and preference learning scenarios.
• Theoretical implications: The method establishes that neither hard minimax nor naive averaging are optimal for pluralistic RLHF in the federated setting; adaptive, history-aware weighting achieves superior fairness–utility trade-off.
• Extension opportunities: Future work includes extending adaptive aggregation to tasks with richer action and reward structures (e.g., long-form text generation, code, creative writing), where reward signal design and group-level consensus become harder. Design of aggregation rules for settings with partial group overlap, time-varying demography, or adversarial group dynamics remains open.

Conclusion

Adaptive Preference Pluralistic Alignment operationalizes the pluralistic alignment problem for LLMs in a privacy-sensitive federated RLHF setting. Its adaptive reward aggregation improves both global and worst-group alignment without sacrificing either, establishing a new strong baseline for fairness-aware, scalable LLM alignment protocols. As the space of societal uses for LLMs expands, aggregation mechanisms that are both fair and private will be critical; this work delineates a theoretically sound, empirically validated path forward.

For full details, see "APPA: Adaptive Preference Pluralistic Alignment for Fair Federated RLHF of LLMs" (2604.04261).