Determine whether hybrid preference routing transfers to policy training

Determine whether Hybrid Preference Routing (HyPER), which predicts reward-model performance for mixtures of human and AI annotations, transfers effectively to Direct Preference Optimization or reinforcement-learning policy training.

Background

The paper discusses HyPER as related work on deciding which comparisons should receive human annotations and which can use model-generated annotations. HyPER predicts the performance of reward models trained on candidate mixtures and selects a routing configuration using those predictions.

The paper explicitly identifies transfer to downstream policy-training settings as unresolved, specifically for Direct Preference Optimization (DPO) and reinforcement learning. This question is distinct from WAPA's own reported experiments, which evaluate DPO downstream alignment but do not establish HyPER's transferability.

References

Its experiments evaluate reward modeling and best-of-$N$ response selection; transfer to DPO or reinforcement-learning policy training is left open.

— Who Should We Listen to More? Welfare-Aware Preference Acquisition for Pluralistic Alignment  (2610.19035 - Ding et al., 13 Oct 2026) in Appendix, Section Related Work, subsection Human--AI feedback allocation