Steerable Cultural Preference Optimization of Reward Models

Abstract: It is essential for LLM technology to serve many different cultural sub-communities in a manner that is acceptable to each community. However, research on LLM alignment has so far predominantly focused on predicting a unified response preference of annotators from certain regions. This paper aims to advance the development of alignment models with a more global outlook, that are able to accurately represent the preferences of subcommunities and do not exhibit excessive bias towards any of them. We focus on the development of reward models for this purpose and present a novel reward model training algorithm (SCPO) that can incorporate diverse cultural preferences in a balanced manner. Our method results in performance increases of the minority reward model of up to 7 points over the baseline model across two datasets, PRISM and GlobalOpinionQA, and across 7 countries. SCPO is up to 280% more training data-efficient than full-data finetuning of reward models. In addition, we perform analysis of bias by separately evaluating on the preference of subcommunities and show that excessive bias is mitigated via our weighting method. Our code is available at https://github.com/minsik-ai/Steerable-Cultural-Preference

• The paper introduces SCPO, which explicitly calibrates cultural discrepancies in reward models using filtering and inverse weighting techniques.
• It achieves up to 280% higher sample efficiency and improves minority alignment accuracy by up to 7 points over baseline methods.
• SCPO’s architecture-agnostic design enables seamless integration into RLHF/DPO pipelines for ethical, culturally responsible AI alignment.

Steerable Cultural Preference Optimization of Reward Models: A Technical Analysis

Motivation and Problem Statement

The alignment of LLMs to diverse cultural sub-communities remains underexplored compared to the predominant focus on unified, majority-centric preference aggregation. Existing methodologies often yield reward models (RMs) that reinforce the perspectives of overrepresented and privileged populations, particularly Western developed countries, marginalizing minority cultures and viewpoints. This bias is problematic for global deployment and ethical AI alignment, motivating the development of frameworks that facilitate pluralistic, regionally-targeted alignment while maintaining global consistency.

Methodological Innovation: SCPO

The Steerable Cultural Preference Optimization (SCPO) framework introduces a principled algorithm for training RMs that explicitly differentiate and calibrate cultural deviations in a manner compatible with RLHF and DPO pipelines. The design leverages off-the-shelf global RMs as reference baselines, using their reward outputs on pairwise preference data to operationalize the identification and weighting of culturally distinctive samples.

Filtering

The filtering phase truncates the minority preference data by retaining only those preference pairs that exhibit disagreement between the minority annotators and the global RM. This step eliminates universal, consensus-aligned preferences, ensuring data-centric focus on genuine cultural divergence. The procedure employs a Bradley-Terry model to quantify RM preference probabilities and sets a configurable threshold $\tau$, mediating the aggressiveness of filtering.

Weighted Loss

To mitigate overfitting and excessive bias towards extreme minority preferences—often correlated with harmful or annotation-noisy content—SCPO deploys an inverse weighting scheme parametrized by a temperature $\beta$. Preferences showing greater divergence from the global model are dynamically down-weighted within the binary ranking loss used for RM fine-tuning, suppressing the influence of outlier samples while amplifying subtle but reliable distinctions. The weighting is strictly divergence-based, decoupled from any content-based classification of responses.

Integration and Generalizability

A key merit of SCPO is architecture-agnosticity: it does not require auxiliary preference modules and can be instantiated with any transformer-based RM, supporting seamless adoption in standard RLHF and DPO alignment pipelines.

Experimental Protocol

Datasets

Experiments utilize PRISM—a preference dataset annotated across diverse countries—and GlobalOpinionQA, which encodes globally stratified survey responses. Evaluation focuses on seven countries with sufficiently large minority subpopulations. The global RMs (OpenAssistant DeBERTa-V3-base, Tulu-3-8B RM) serve both as starting checkpoints and reference models for filtering/weighting.

Evaluation Design

Balanced evaluation is achieved by reporting accuracy on: (1) PRISM’s entire test set (representing both global and minority preferences) and (2) "true country-specific" subsets (samples where minority and global preferences strongly disagree). The trade-off between overfitting to minority divergence and generalization is explicitly measured.

Key Results

• Data Efficiency: SCPO achieves up to 280% higher sample efficiency than naive full-data fine-tuning, as evidenced by strong performance after aggressive filtering.
• Minority Alignment: On PRISM and GlobalOpinionQA benchmarks, SCPO elevates minority RM accuracy by up to 7 points over baselines, especially when both filtering and weighting are integrated (e.g., “SCPO (F + W)” means both steps are used).
• Bias Mitigation: Weighted training loss reduces excessive skew towards extreme minority preferences—a documented failure case in unweighted or solely filtered approaches.
• Global Alignment: For most countries, the best average accuracy (across all test preferences) is achieved when both filtering and weighting are applied, rather than naive or data-hungry baselines.
• Superiority over GPO: SCPO outperforms Group Preference Optimization (GPO) on alignment-to-survey distributions, as measured by Jensen-Shannon Distance to human response proportions in GlobalOpinionQA, notably for Chile and Mexico.
• Robustness: Ablation studies establish that performance gains are not attributable merely to data reduction; informative selection by SCPO filtering consistently outperforms random size-matched filtering.

Theoretical and Practical Implications

The SCPO framework directly operationalizes pluralistic alignment at the population-level rather than the user-level, circumventing limitations of methods such as PAL or VPL that are evaluated only on synthetic/user-centric benchmarks. Its filtering and weighting strategy disentangles regionally idiosyncratic preferences from those reflecting annotation error or adversarial/harmful content—a critical consideration in scalable, culturally responsible LLM alignment.

Practically, SCPO enables the construction of RMs that (a) minimize cultural bias across majority and minority populations, (b) support explicit steering to a region- or country-specific value frame, and (c) can be implemented with minimal engineering in existing RLHF/DPO workflows. The approach is compatible with iterative preference optimization and online data prioritization.

Limitations and Future Directions

Several caveats are acknowledged: first, operationalization of “cultural preference” remains dataset- and language-dependent; the PRISM data is English-centric, and results may not generalize to local-language annotation without further investigation. Second, country-level grouping risks occluding local intra-country heterogeneity. Third, preference divergence as measured by RM disagreement is a necessary but not sufficient proxy for genuine cultural distinctiveness; domain ethnography would be required for validation.

Future work should pursue extension to subnational and demographic groups, incorporate multilingual preference data, and explore the interaction of SCPO with novel RLHF strategies or adversarial robustness.

Conclusion

SCPO defines a scalable, data-efficient, and robust methodology for learning steerable, culturally-aligned reward models from large-scale human preference annotation. Through decoupling core consensus knowledge from distinctive minority perspectives, and modulating emphasis via inverse weighting, SCPO advances pluralistic alignment in LLM reward modeling pipelines. This yields practical benefits for developers seeking equitable global deployments and provides an empirical foundation for more nuanced, demographically-aware AI alignment research.