CARB: Cultural Reward Model Benchmark

• The paper introduces CARB, a benchmark that quantitatively evaluates reward models’ ability to capture cultural nuances and diverse global perspectives in LLM alignment.
• CARB leverages 8,576 Best-of-N preference sets across 10 languages and four cultural domains, ensuring robust evaluation in cultural commonsense, values, safety, and linguistics.
• Empirical analyses demonstrate that generative reward models outperform classifier models by mitigating spurious correlations and enhancing cultural sensitivity.

Cultural Awareness Reward modeling Benchmark (CARB) is a multilingual, culture-focused benchmark for evaluating reward models used in LLM alignment. It consists of 8,576 Best-of-N preference sets spanning 10 distinct cultures across four cultural domains—cultural commonsense knowledge, cultural values, cultural safety, and cultural linguistics—and was introduced to address the scarcity of culturally relevant evaluation datasets for reward models, especially in contrast to multilingual evaluations created by translating English prompts rather than using culturally grounded data (Zhang et al., 26 Sep 2025).

1. Scope, motivation, and benchmark target

CARB is motivated by the position of the reward model as the central proxy for preference in RLHF pipelines. If reward models encode mostly English-speaking or Western preferences, then globally deployed LLMs can exhibit cultural dominance, return socially inappropriate or unsafe responses in particular cultural contexts, and require expensive end-to-end alignment experiments before their weaknesses become visible (Zhang et al., 26 Sep 2025). Within this framing, a reward model is culturally aware when it can reliably identify the candidate response that is factually correct and appropriate in the target culture, respects that culture’s norms, values, safety constraints, and linguistic nuance, and does not rely primarily on surface-level cues such as explicit country labels or language identity (Zhang et al., 26 Sep 2025).

CARB structures this target into four domains. These domains are not generic multilingual QA categories; they are intended to probe culture-conditioned preference judgment. The benchmark therefore treats cultural awareness as a property of comparative evaluation rather than only generation.

Domain	Source datasets	Focus
Cultural commonsense knowledge	Cultural Atlas, MANGO	Etiquette, social practices, everyday cultural facts
Cultural values	World Values Survey	Social, economic, and ethical value judgments
Cultural safety	PTP, RTP-LX, ViCTSD, ThaiToxicityTweet	Culturally situated toxicity and norm violations
Cultural linguistics	MAPS, IdiomKB, proverb datasets	Idioms, proverbs, sayings, figurative language

The benchmark’s design addresses a gap explicitly identified in reward-model evaluation. RewardBench, RewardBench2, RMB, RM-Bench, Preference Proxy Evaluations, and M-RewardBench are useful for general preference modeling, but they do not jointly provide multilingual and culture-aware evaluation; M-RewardBench is multilingual yet not culture-aware in this sense (Zhang et al., 26 Sep 2025). CARB is therefore positioned as an evaluation instrument for global alignment rather than a generic multilingual stress test.

2. Intellectual antecedents and benchmark lineage

CARB emerges from a broader research trajectory in which cultural adaptation, culture-aware reward design, and culturally grounded evaluation were developed in partially separate strands. Earlier work on affective computing proposed a composite reward design in which cultural appropriateness, emotional appropriateness, and user feedback are explicitly separated, and suggested that a benchmark should test systems in multi-cultural, emotionally charged scenarios rather than treating culture as static conditioning alone (Pussadeniya et al., 17 Jun 2025). This suggests a benchmark logic in which cultural awareness is not a single scalar attribute but a structured objective.

A second strand formalized conversational style as a function of situation, interpersonal relationship, and cultural context, $\text{Style} = f(S, R, C)$, and operationalized culture-specific style preference as a distribution rather than a single correct answer (Havaldar et al., 13 Oct 2025). That framework introduced accepted stylistic ranges derived from annotator mean and standard deviation, using the interval $[\mu - 0.674\sigma,\ \mu + 0.674\sigma]$, and argued for “subjective correctness” in cross-cultural evaluation (Havaldar et al., 13 Oct 2025). This provides a direct conceptual precursor for preference-set construction in culturally aware evaluation.

Reward-model training research also supplied key ingredients. SCPO proposed steerable cultural preference optimization of reward models by separating global and minority reward models, filtering disagreement pairs relative to a global reward model, and weighting them to avoid excessive bias toward extreme local preferences (Oh et al., 17 Jun 2026). CARE then showed that a modest amount of high-quality native human preferences can improve cultural awareness more effectively than larger generic preference datasets, using multilingual native judgments over culturally specific questions in Chinese and Arabic (Guo et al., 7 Apr 2025). Taken together, these works indicate that culture-aware reward modeling depends not only on multilinguality, but on culturally grounded preference data, subgroup-sensitive evaluation, and robustness against shortcut learning.

A more general theoretical antecedent comes from the argument that the relevant target is not merely cultural knowledge but meta-cultural competence: awareness that cultural norms vary, ability to represent uncertainty, and capacity to explicate or negotiate when the model’s priors may not fit the current context (Saha et al., 9 Feb 2025). Although CARB is a discrete-choice benchmark rather than a dialogue-based negotiation environment, its emphasis on authentic cultural nuance over surface cues is consistent with that broader agenda.

3. Benchmark construction, data model, and cultural coverage

CARB uses 10 languages as proxies for major cultural groupings: English, Spanish, Arabic, Chinese, Thai, German, Russian, Vietnamese, Japanese, and Korean (Zhang et al., 26 Sep 2025). These were selected to cover all continents, multiple cultural clusters identified in cross-national studies, typological diversity of language families and scripts, and variation in resource levels (Zhang et al., 26 Sep 2025). Each language subset contains a balanced number of Best-of-N instances across domains, yielding 8,576 instances in total, with domain counts of 2,080 for cultural commonsense, 2,496 for cultural values, 2,000 for cultural linguistics, and 2,000 for cultural safety (Zhang et al., 26 Sep 2025).

Each CARB item is a Best-of-N preference set with one chosen response and three rejected responses, so $N=4$ and the random baseline is 25% accuracy (Zhang et al., 26 Sep 2025). The chosen response is generated by high-performing LLMs and then validated through three mechanisms: cosine similarity to reference cultural material using Qwen3-Embedding-8B, human annotator agreement above 80% on cultural appropriateness and factuality, and GPT-4o proxy annotation correlated with humans (Zhang et al., 26 Sep 2025). Rejected responses are also high-quality outputs, but they are culturally inappropriate, factually wrong about the target culture, or unsafe within the relevant cultural context (Zhang et al., 26 Sep 2025). This design matters because it prevents the task from collapsing into trivial quality detection.

The benchmark also controls known confounds. Chosen and rejected responses have matched length distributions, explicitly avoiding the length bias observed in M-RewardBench, where rejected answers are systematically longer (Zhang et al., 26 Sep 2025). Candidate completions come from 24 different LLMs, including Qwen, Llama, GPT-4o, Aya expanse, and Mistral families, with balanced contribution across models (Zhang et al., 26 Sep 2025). This broadens the negative pool and reduces the chance that a reward model can solve the task by learning idiosyncratic stylistic markers from a small set of generators.

Prompt construction is domain-specific. Cultural commonsense examples are derived from Cultural Atlas and MANGO and then transformed by GPT-4o into structured culturally grounded questions; cultural values examples are adapted from the World Values Survey following methods from WorldValuesBench and CultureLLM; cultural safety items are drawn from multilingual toxicity datasets; cultural linguistics items focus on idioms and proverbs whose correct treatment requires preserving both literal and culturally specific meaning (Zhang et al., 26 Sep 2025). This implies that CARB is not merely multilingual preference data, but a synthesized evaluation layer over culturally specialized corpora.

4. Evaluation protocol and empirical profile

The core metric is accuracy: the reward model must select the chosen response from the four candidates (Zhang et al., 26 Sep 2025). For classifier reward models, this is done by comparing scalar reward scores. For generative reward models, the benchmark uses LM-as-judge prompting that asks the model to act as an impartial judge and choose the assistant that best follows the user’s instructions while assessing cultural awareness (Zhang et al., 26 Sep 2025).

For generative reward models, the paper formalizes the model as a sequence model over a reasoning trajectory $z = \{z_t\}_{t=1}^{T}$:

$$r_\theta(z \mid q, y_1, y_2) = \prod_{t=1}^T r_\theta(z_t \mid q, y_1, y_2, z_{<t})$$

where $q$ is the prompt, $(y_1, y_2)$ are candidate responses, and the final judgment $\hat{j}$ is extracted from the generated reasoning (Zhang et al., 26 Sep 2025). This formulation becomes central later in Think-as-Locals, but it already clarifies that CARB can evaluate both scalar and reasoning-style reward models under a unified preference-selection task.

Empirically, generative reward models dominate the leaderboard. The highest average CARB score is reported for Qwen3-235B-A22B-Instruct-2507 at about 76.5%, and seven of the top ten systems are generative rather than classifier reward models (Zhang et al., 26 Sep 2025). The best classifier reward model, Skywork-Reward-Gemma-2-27B, ranks fifth at about 73.0% average (Zhang et al., 26 Sep 2025). This is notable because the benchmark is not purely open-ended; even in Best-of-N selection, reasoning-style reward models appear to have an advantage on cultural nuance.

Performance is strongly structured by domain. Safety is the easiest domain across models, which the paper attributes to heavy reward-model training emphasis on safety alignment. Cultural values is the hardest domain, with the lowest scores and the largest cross-cultural variability (Zhang et al., 26 Sep 2025). Commonsense and linguistics sit between these extremes, though classifier reward models sometimes outperform generative ones on English linguistics, where chosen and rejected answers can both be high-quality and differ only in fine-grained cultural fidelity (Zhang et al., 26 Sep 2025).

Performance is also structured by language resource level, language family, and script. Higher-resource languages tend to have higher mean accuracy and lower variance than medium-resource languages; German and Chinese show high peak performance, Arabic remains particularly difficult, and Vietnamese has one of the largest gaps between best and worst reward models (Zhang et al., 26 Sep 2025). These results indicate that culture-aware reward modeling still inherits part of the resource asymmetry already documented for multilingual LLMs.

A central empirical result is that CARB predicts downstream cultural alignment much better than prior reward-model benchmarks. In Best-of-N inference-time scaling on INCLUDE, BLEnD, and OMGEval, Spearman correlations between CARB rankings and downstream alignment rankings range from 0.61 to 0.83, whereas M-RewardBench often yields weak and non-significant correlations (Zhang et al., 26 Sep 2025). In RLHF training via GRPO, CARB accuracy shows a strong positive linear relationship with downstream GPT-4o evaluation scores, with $r^2 > 0.6$ and $p < 0.001$, while M-RewardBench again shows weak, non-significant relationships (Zhang et al., 26 Sep 2025). This makes CARB not only an analysis benchmark but a practical proxy for reward-model selection.

5. Spurious correlations, language bias, and diagnostic analyses

CARB’s diagnostic contribution lies in showing that many reward models achieve culture-aware preference judgments through spurious features rather than substantive cultural reasoning. The benchmark evaluates this through controlled perturbations on commonsense instances (Zhang et al., 26 Sep 2025). One perturbation changes the core cultural concept while preserving explicit cultural labels; others remove explicit cultural labels, translate the response into another language while preserving meaning, or rephrase the content (Zhang et al., 26 Sep 2025). The first type is treated as causal, the others as spurious.

For generative reward models, the relevant diagnostic quantity is the log probability of a response given the prompt:

$[\mu - 0.674\sigma,\ \mu + 0.674\sigma]$0

Lower-performing systems show large score changes under spurious perturbations and comparatively small changes under causal perturbations, indicating over-reliance on explicit labels, script identity, or stylistic markers (Zhang et al., 26 Sep 2025). By contrast, the strongest systems show the reverse pattern: strong sensitivity to substantive cultural changes and relative invariance to label removal, language change, or paraphrase (Zhang et al., 26 Sep 2025).

The benchmark also evaluates cross-lingual consistency by translating responses into multiple languages while holding the prompt language fixed. Consistency is defined as

$[\mu - 0.674\sigma,\ \mu + 0.674\sigma]$1

where $[\mu - 0.674\sigma,\ \mu + 0.674\sigma]$2 is the absolute discrepancy relative to the same-language score (Zhang et al., 26 Sep 2025). Larger, higher-scoring reward models are more cross-lingually consistent, but all models still exhibit language bias. Qwen-based reward models are most consistent for Chinese prompts, while LLaMA- and Gemma-based systems are biased toward English (Zhang et al., 26 Sep 2025). This shows that multilingual reward modeling is not equivalent to language-agnostic cultural judgment.

The paper also reports preliminary evidence of reward hacking during RLHF on multilingual cultural preference data. With Qwen2.5-7B-Instruct as the reward model and Llama-3.1-Tulu-3-8B-SFT as the policy, GRPO training causes training and validation rewards to decrease while policy entropy increases, suggesting that the policy learns to exploit weaknesses in the reward model rather than improving cultural alignment (Zhang et al., 26 Sep 2025). This places CARB within a broader alignment problem: evaluating cultural awareness is inseparable from diagnosing reward fragility.

6. Think-as-Locals, RLVR, and the benchmark’s broader significance

To improve culture-aware reward modeling, the CARB paper introduces Think-as-Locals, a generative reward-model training method based on reinforcement learning from verifiable rewards (RLVR) (Zhang et al., 26 Sep 2025). Instead of asking the reward model to emit only a scalar preference, Think-as-Locals trains it to generate structured cultural evaluation criteria, assess the candidates against those criteria, and then issue a final judgment. The resulting reasoning traces are intended to anchor reward-model decisions in culturally grounded rubrics rather than in surface correlations.

The method uses two reward components. The correctness reward is

$[\mu - 0.674\sigma,\ \mu + 0.674\sigma]$3

and the criteria-appropriateness reward measures how much the generated reasoning sequence increases the log probability of the correct judgment relative to directly decoding the judgment without reasoning (Zhang et al., 26 Sep 2025). These rewards are optimized with GRPO, using group-relative advantages and KL regularization against a reference model (Zhang et al., 26 Sep 2025). The structured prompt requires the model to generate a rubric, justify it, evaluate each candidate, and then output the preferred answer (Zhang et al., 26 Sep 2025).

This intervention produces large gains. Qwen2.5-7B-Instruct improves from 62.6 to 78.8 on CARB after Think-as-Locals, while also improving on M-RewardBench from 77.1 to 80.4 (Zhang et al., 26 Sep 2025). The 32B version reaches 84.3 on CARB, surpassing larger baseline reward models, including Qwen3-235B-A22B-Instruct on this benchmark (Zhang et al., 26 Sep 2025). Gains are especially pronounced on Arabic, Chinese, and Japanese subsets (Zhang et al., 26 Sep 2025). Perturbation analyses further show that Think-as-Locals increases sensitivity to causal cultural features and reduces sensitivity to spurious ones (Zhang et al., 26 Sep 2025).

The broader significance of CARB is that it provides a technically grounded evaluation layer for a rapidly expanding literature on cultural alignment. Native preference resources such as CARE supply culturally specific supervision for multilingual alignment (Guo et al., 7 Apr 2025). WorldValuesBench supplies distributional targets for demographic-conditioned value prediction (Zhao et al., 2024). LLM-GLOBE operationalizes cultural value systems in open-generation form through GLOBE dimensions (Karinshak et al., 2024). ExCAM supplies an explainable metric for identifying and rating cultural errors in instruction-output pairs, including severity-based aggregation (Leiter et al., 28 May 2026). CARB differs from these by focusing specifically on reward models and by showing direct predictive validity for downstream cultural alignment (Zhang et al., 26 Sep 2025).

The benchmark nonetheless retains clear limitations. It covers 10 cultures and four domains rather than an exhaustive global space; it is textual rather than multimodal; it assumes a single chosen response per item rather than modeling intra-cultural disagreement explicitly; and it depends partly on LLM-generated candidate responses, even though these are filtered by similarity and human agreement (Zhang et al., 26 Sep 2025). A plausible implication is that future versions will need broader cultural coverage, finer treatment of contested norms, and stronger anti-reward-hacking protocols. Within the present literature, however, CARB functions as the first benchmark explicitly designed to evaluate cultural awareness in reward models themselves, rather than only in the policies those reward models later shape (Zhang et al., 26 Sep 2025).