To Go Far, Go Together: Diverse Preferences Induce a Curriculum for Reward Optimization

Abstract: Learning a reward model from human feedback and optimizing a policy against it is one approach to aligning AI systems with individual users. From a fairness perspective, existing work improves such alignment by developing data-efficient and accurate reward models that capture minority preferences despite scarce data. We push this line of inquiry one step further and argue that data-efficient and accurate per-user reward models are not sufficient: users whose reward models are difficult to \textit{optimize} at the policy level can become a new underserved group. We start from the observation that one user's reward model can be easy to optimize from the initial policy while another's is not. We argue that, given a sufficiently diverse user population, a curriculum naturally emerges between easy- and hard-to-optimize reward models. Building on this insight, we propose CurriPO, which grows a tree-structured curriculum to accommodate diverse user-specific objectives, covering the population in a single traversal. Specifically, CurriPO automatically constructs a curriculum over diverse user reward models, allowing it to branch from the existing curriculum and reuse reward models previously incorporated into the curriculum. To the best of our knowledge, this is the first work to explicitly exploit multi-user structure to address optimization in AI alignment. Extensive experiments on personalized continuous control in a simulated environment show that CurriPO achieves $1.2$--$2.1\times$ the population satisfaction of the strongest baseline while substantially reducing training time. Additional analysis attributes much of this improvement to the users left underserved by conventional optimization.

• The paper introduces 'CurriPO', a method that leverages diverse user preferences to create a curriculum for reward model optimization, significantly improving population satisfaction and reducing training time compared to baseline methods.
• CurriPO leverages a framework of probing, frontier selection, and reuse to create a personalized curriculum by optimizing the reward models from easier to harder users.
• Evaluations show that CurriPO consistently outperforms baselines in utilitarian and distribution-sensitive metrics, demonstrating a robust ability to serve diverse user preferences, even under noisy feedback.

Motivation and problem statement

This paper addresses a gap in personalized alignment that sits downstream of reward modeling. Prior work on per-user alignment focuses on learning accurate, data-efficient reward models from heterogeneous populations, for instance through distributional preference learning, latent preference spaces, or grouped reward models. The authors argue that even when every user—including preference minorities—has an accurate reward model, the subsequent reinforcement learning (RL) optimization stage can itself create a new underserved group. They ground this claim in known results on reward invariance: policy-preserving reward transformations can drastically alter optimization difficulty, and held-out preference accuracy does not distinguish easy-to-optimize from hard-to-optimize reward models (2608.18770). A reward model may rank every pair correctly yet induce an optimization landscape on which PPO-style training fails to recover the intended behavior.

The paper's central observation is empirical and structural: across a sufficiently diverse user population, reward models vary widely in how easily a policy can be optimized against them, and policies optimized for easier users can serve as warm-start initializations—stepping stones—for harder ones. The population itself therefore induces a curriculum over reward models, without any task-generation machinery.

Method

CurriPO operates on a bank of per-user reward models $\{f_u\}$ fit with the standard Bradley–Terry cross-entropy pipeline from pairwise feedback, each reaching at least 90% held-out preference accuracy before use. Starting from a fixed initial policy $\pi_0$, CurriPO runs a sequence of local RL stages; after stage $k$, all checkpoints produced so far form the wake $\mathcal{W}_k$.

Three mechanisms define the traversal:

• Probing: at each stage, short rollouts from every checkpoint are scored by every user's reward model. Competence is defined as the percentile a checkpoint reaches under a user's reward model relative to the offline behavior pool, and coverage $C_k(u)$ is the best competence any checkpoint attains for user $u$.
• Frontier selection with branching: among admissible users whose coverage exceeds a threshold $c_{\mathrm{lo}}$, CurriPO targets the least-covered user $u_k$ and warm-starts its optimization from the checkpoint serving that user best. Because the starting checkpoint is re-selected at every stage rather than fixed to the most recent one, the traversal forms a tree instead of a chain.
• Reusing: a previously optimized reward model may be selected again (up to $q$ times) whenever a newly added checkpoint probes better than the checkpoint from which it was originally optimized. This exploits the observation that optimizing the same reward model from different initializations yields substantially different outcomes.

Each stage optimizes the standardized reward of the selected user under a KL penalty toward the warm-start checkpoint, keeping transitions between stages local. At termination (coverage threshold $c_{\mathrm{stop}}$ reached or budget exhausted), each user receives the wake checkpoint their own reward model prefers. Notably, the final assignment uses only learned reward models; no oracle information enters training or serving.

Experimental results

Evaluation covers four MuJoCo continuous-control settings (HalfCheetah and Hopper, with preferences over velocity alone or velocity plus energy/height), scripted users ($\pi_0$0 and $\pi_0$1), and both oracle and B-Pref-noisy feedback. Population satisfaction is measured against ground-truth utilities via band-shaped kernels over behavioral descriptors.

CurriPO achieves the highest utilitarian satisfaction in all eight environment–feedback combinations, with the largest margins in the 2-D settings. Representative numbers under oracle feedback with 100 users:

Method	HC-1D	HC-2D	H-1D	H-2D	Time (h)
RM	0.352	0.227	0.309	0.142	22
RM+RND	0.344	0.221	0.318	0.145	28
IPL	0.299	0.195	0.240	0.128	48
CPL	0.173	0.089	0.191	0.078	47
CurriPO	0.512	0.319	0.415	0.280	1.6

Two claims stand out numerically. First, the advantage persists under noisy feedback, indicating robustness to altered reward model geometry. Second, the cost asymmetry is stark: scaling the population from 12 to 100 raises CurriPO's training time from 1.1 h to 1.6 h, while baselines scale from roughly 3–6 h to 22–48 h. Across all eight comparisons, CurriPO significantly outperforms every baseline (Wilcoxon signed-rank $\pi_0$2; Holm-corrected $\pi_0$3).

Distribution-sensitive welfare metrics strengthen the fairness interpretation. Under egalitarian satisfaction, most baselines leave at least one user near zero (often below 0.02), whereas CurriPO raises worst-user satisfaction substantially; it achieves the highest Nash welfare in all 16 settings and the highest egalitarian welfare in 15 of 16. Satisfaction landscape visualizations confirm that gains concentrate on users in regions conventional methods leave unserved.

Why the gains are not explained away

The paper pre-empts three alternative explanations with targeted controls:

• Undertrained baselines: extending RM's per-user budget from 1.2M to 3.2M steps leaves hard targets essentially unserved (satisfaction at target velocity 5 remains 0.001), and an RND exploration bonus does not close the gap.
• Serving-rule artifact: giving baselines the same best-checkpoint-over-pool serving rule improves them only marginally, because their pools contain no policies from unreachable regions. Selection repairs within the pool; the traversal changes what the pool contains.
• Warm-starting alone: randomizing CurriPO's curriculum order while keeping identical KL-anchored stages sharply degrades satisfaction on hard targets, and the gap widens with difficulty. The specific coverage-driven ordering matters, not merely initialization from other users' policies.

Ablations attribute distinct roles to the components: removing branching entirely (Chain) causes large drops; removing reuse closes much but not all of the remaining gap; and removing the KL anchor degrades satisfaction in most settings while inflating variance. The threshold $\pi_0$4 is important—removing it drops mean satisfaction from 0.365 to 0.208—but performance is stable within $\pi_0$5, so careful tuning is unnecessary.

Limitations and open questions

Several constraints bound the scope of these results. All experiments use scripted users with parametric utility functions in low-dimensional preference spaces (one or two dimensions), so it remains open whether the induced curriculum structure exists in higher-dimensional or genuinely human preference distributions. The setting assumes per-user reward models reach high held-out accuracy before optimization begins; how CurriPO behaves when reward models are inaccurate or improve online during traversal is not examined. Evaluation is restricted to relatively simple locomotion tasks; the authors themselves note that extension to LLMs, vision-language-action models, and real human feedback remains future work. Additionally, probing costs scale with the product of checkpoint count and population size, though this did not dominate wall-clock time here; behavior under much larger populations or more expensive rollouts is untested. Finally, the competence metric depends on an offline behavior pool defining percentiles, raising the question of sensitivity to pool composition and coverage.

Conclusion

This paper identifies optimization difficulty—not reward accuracy—as a binding constraint in per-user alignment and demonstrates that multi-user diversity can be exploited structurally at the RL stage. By growing a tree-structured curriculum over the reward model bank, with branching, reuse, and KL-anchored local stages, CurriPO delivers 1.2–2.1× the population satisfaction of strong baselines at a fraction of the training cost, with gains concentrated precisely on users conventional optimization abandons. The work reframes heterogeneous preferences from an obstacle into a resource for optimization, and leaves the transfer of this mechanism beyond simulated locomotion as the principal open question.