VAL-Bench: Evaluating Value Alignment

• VAL-Bench is a benchmark designed to evaluate whether language models maintain a consistent value stance across paired prompts that frame opposing sides of controversies.
• It uses 115K paired prompts from Wikipedia’s controversial sections, applying a structured pipeline and paired scoring methodology to compute Pairwise Alignment Consistency (PAC).
• Empirical findings reveal significant trade-offs between PAC and refusal behavior, indicating that high alignment scores can result from either substantive reasoning or strategic refusals.

Searching arXiv for the benchmark and closely related value-alignment benchmarks to ground the article in current papers. Value ALignment Benchmark (VAL-Bench) is a benchmark for measuring whether LLMs maintain a stable value stance when confronted with opposing framings of the same controversial issue. In the formulation introduced by "VAL-Bench: Measuring Value Alignment in LLMs" (Gupta et al., 6 Oct 2025), the benchmark tests paired prompt consistency rather than only refusals or predefined safety violations, using 115K prompt pairs derived from Wikipedia’s controversial sections. In the broader value-alignment literature, this places VAL-Bench alongside benchmarks that assess value orientations, behavioral conformity, cultural specificity, and pluralistic aggregation, including "Value Compass Benchmarks: A Platform for Fundamental and Validated Evaluation of LLMs Values" (Yao et al., 13 Jan 2025), "ValueBench: Towards Comprehensively Evaluating Value Orientations and Understanding of LLMs" (Ren et al., 2024), and "LocalValueBench: A Collaboratively Built and Extensible Benchmark for Evaluating Localized Value Alignment and Ethical Safety in LLMs" (Meadows et al., 2024).

1. Definition and scope

VAL-Bench is introduced to evaluate whether models maintain a stable value stance across paired prompts that frame opposing sides of public debates (Gupta et al., 6 Oct 2025). Its motivating claim is that existing benchmarks mostly track refusals or predefined safety violations, which check rule compliance but do not reveal whether a model upholds a coherent value system when facing controversial real-world issues. A well-aligned model, in this benchmark’s terms, should express similar underlying views regardless of framing.

This benchmark therefore measures a specific notion of value alignment: pairwise alignment consistency across opposed framings of the same issue. It does not define a single correct normative position. Rather, it evaluates whether a model’s responses remain internally coherent when one prompt asks it to explain one side of a controversy and a paired prompt asks it to explain the opposite side (Gupta et al., 6 Oct 2025). This distinguishes VAL-Bench from safety-only evaluations centered on toxicity, harmful capabilities, or refusal behavior, and from questionnaire-style value inventories that primarily probe declared attitudes rather than frame-robust output behavior (Ren et al., 2024).

A plausible implication is that VAL-Bench operationalizes value alignment as a form of stability under rhetorical perturbation. This suggests a benchmarking focus on value coherence rather than value correctness. That interpretation is consistent with adjacent work arguing that value evaluation should move beyond static risk categories and beyond measuring only value knowledge instead of value-guided behavior (Yao et al., 13 Jan 2025).

2. Dataset construction and benchmark structure

VAL-Bench consists of 115K paired prompts from Wikipedia’s controversial sections (Gupta et al., 6 Oct 2025). The construction pipeline begins from the Structured Wikipedia dump in English and filters sections using regular expressions over headings associated with conflict, criticism, scandals, legal disputes, human rights, privacy, security, and related controversy markers. Not every such section is retained. The benchmark first applies an issue-awareness stage in which Gemma-3-27B-it annotates whether the section concerns a divergent issue and scores societal salience on a 0–5 scale. Sections not judged to involve a divergent issue are filtered out (Gupta et al., 6 Oct 2025).

From each retained section, Gemma-3-27B-it generates a specific divergent issue and two paired prompts: a “for” prompt and an “against” prompt, both phrased as neutral, open-ended “Explain why ...” questions about the same issue (Gupta et al., 6 Oct 2025). Each item is therefore a triple $(c_i, p_i^+, p_i^-)$, where $c_i$ is the issue description and $p_i^+$, $p_i^-$ are the paired prompts. The benchmark requires both prompts to concern the same controversy and to include at least one specific detail from the source article.

The issue distribution spans 20 categories. The largest categories are Politics at 25.08%, Social & Cultural Issues at 12.03%, Governance at 7.57%, Ethics at 5.56%, Legal Disputes at 5.55%, Religion at 5.50%, History at 4.75%, Human Rights at 3.86%, Media & Entertainment at 3.83%, and Civil Rights at 3.57% (Gupta et al., 6 Oct 2025). The issue-awareness distribution in the test set is also reported: level 1 at 0.06%, level 2 at 4.17%, level 3 at 45.92%, level 4 at 49.01%, and level 5 at 0.84% (Gupta et al., 6 Oct 2025).

The benchmark additionally extracts a value vocabulary from prompts. GPT-4.1 is used to extract up to five values per prompt from 22,950 prompts, yielding more than 34,000 raw value phrases, which are then embedded and clustered via K-means into 1,000 value clusters (Gupta et al., 6 Oct 2025). Reported frequent value phrases include prioritizing accountability, pragmatism over principle, transparency, trust in establishment, legal formalism, prioritizing justice, tradition prioritization, evading accountability, retribution, and status quo reinforcement (Gupta et al., 6 Oct 2025). This value-extraction layer does not determine scoring directly, but it supports analyses of which values the prompts evoke and which values model responses demonstrate.

3. Scoring methodology

The central task is to compare a model’s responses to paired prompts about the same issue. For issue $i$, a model $\mathcal{M}$ produces two responses, $r_i^+ = \mathcal{M}(p_i^+)$ and $r_i^- = \mathcal{M}(p_i^-)$ (Gupta et al., 6 Oct 2025). An evaluator model $\mathcal{E}$ then scores the pair using the issue description, the original prompts, and the two responses.

The evaluator is formalized as

$$\mathcal{E} : (c_i, p_i^{+}, r_i^{+}, p_i^{-}, r_i^{-}) \mapsto \bigl(\sigma_i, \psi(r_i^{+}), \nu(r_i^{+}), \psi(r_i^{-}), \nu(r_i^{-})\bigr),$$

where $c_i$0 is the raw agreement score, $c_i$1 indicates whether a response is a refusal, and $c_i$2 indicates whether a response is a no-information answer (Gupta et al., 6 Oct 2025). The raw score is interpreted as follows: $c_i$3 denotes heavily opposed views, $c_i$4 moderately opposed, $c_i$5 mixed or neutral, $c_i$6 moderately aligned, and $c_i$7 fully aligned (Gupta et al., 6 Oct 2025).

VAL-Bench then defines adjusted pairwise scores $c_i$8 as

$c_i$9

Thus, if neither response refuses, the pair keeps the judge score; if exactly one refuses, the pair is forced to 0; if both refuse, the pair is forced to 2 (Gupta et al., 6 Oct 2025).

The main metric is Pairwise Alignment Consistency (PAC), obtained by averaging $p_i^+$0 and rescaling from $p_i^+$1 to $p_i^+$2: $p_i^+$3 A PAC of 100 corresponds to maximal pairwise consistency, 50 to a centered value, and 0 to maximal inconsistency (Gupta et al., 6 Oct 2025).

The benchmark reports refusal and no-information metrics separately. The one-side refusal rate (1REF), both-sides refusal rate (2REF), and overall refusal rate (REF) are computed from $p_i^+$4, while analogous no-information metrics are computed from $p_i^+$5 (Gupta et al., 6 Oct 2025). This separation is central to the benchmark’s interpretation, because a model can achieve high PAC through consistent refusals rather than through a consistently expressed substantive stance.

4. Evaluator calibration and reliability

The evaluator in VAL-Bench is Gemma-3-27B-it, selected after comparison with LLaMA-3.3-70B-Instruct and Mistral-large-instruct on synthetic calibration datasets (Gupta et al., 6 Oct 2025). The calibration datasets include unaligned-0, aligned-100, refusals-100, and one-refusal-50, each designed to test whether the judge recovers expected PAC and refusal behavior under controlled response patterns.

Reported judge outputs are close, but not identical, to expected and human-scored values. On unaligned-0, expected PAC and REF are 0 and 0, while the LLM judge reports 8.14 and 0.14; human scores are 3.00 and 1.00. On aligned-100, expected PAC and REF are 100 and 0, while the judge reports 81.53 and 0.09; human scores are 99.00 and 1.00. On refusals-100, expected PAC and REF are 100 and 100, while the judge reports 92.10 and 85.18; human scores are 100.00 and 100.00. On one-refusal-50, expected PAC and REF are 50 and 50, while the judge reports 46.49 and 43.32; human scores are 51.00 and 51.00 (Gupta et al., 6 Oct 2025).

The benchmark also performs order ablations, reversing the order of paired responses, and paraphrase ablations. The reported RMSE under order reversal is 2.28 for PAC, 0.30 for REF, and 0.57 for NINF (Gupta et al., 6 Oct 2025). Paraphrasing responses raises PAC by about 3.7 points on both unaligned-0 and aligned-100, which indicates some sensitivity to surface form but not complete evaluator instability (Gupta et al., 6 Oct 2025).

This suggests that the judge is sufficiently calibrated for large-scale comparative analysis, though not exact. A plausible implication is that VAL-Bench should be read as a comparative benchmark rather than a gold-standard moral adjudicator. That interpretation aligns with related work emphasizing the need to distinguish evaluator robustness from the underlying value construct being measured (Revel et al., 2024).

5. Empirical findings

VAL-Bench reports large variation in PAC across open- and closed-source models (Gupta et al., 6 Oct 2025). Among chat models, the highest PAC scores are obtained by Anthropic models: claude-haiku-3.5 at 68.79, claude-sonnet-4 at 68.34, claude-opus-4.1 at 66.11, and claude-sonnet-3 at 55.27. Qwen3-235B-instruct-2507 is the strongest non-Claude chat model at 42.86. Lower-scoring chat models include glm-4.5-air-nothink at 34.38, llama-4-maverick-instruct at 32.62, llama-3.3-70B-instruct at 31.59, gpt-3.5-turbo at 26.91, gpt-4o at 24.61, gpt-4 at 23.76, gpt-4.1 at 22.77, gpt-5-chat at 19.49, and glm-4.5-air-base at 7.42 (Gupta et al., 6 Oct 2025).

Among thinking models, claude-sonnet-4-thinking scores 68.26, qwen3-235B-thinking-2507 scores 57.42, qwen3-30B-thinking-2507 scores 49.34, glm-4.5-air-thinking scores 33.59, gpt-5 scores 26.41, deepseek-r1 scores 12.61, and o4-mini scores 12.04 (Gupta et al., 6 Oct 2025). Thinking models therefore do not uniformly outperform chat models. In Qwen3, the thinking variants substantially exceed their chat counterparts, whereas in GPT and GLM the effect is weaker or negative (Gupta et al., 6 Oct 2025).

A notable result is the trade-off between PAC and refusal behavior. Claude-haiku-3.5 has the highest PAC at 68.79 but also the highest refusal statistics, with 1REF at 30.38, 2REF at 15.14, and REF at 30.34 (Gupta et al., 6 Oct 2025). By contrast, qwen3-235B-instruct-2507 has PAC 42.86 with very low REF at 1.48 and NINF at 1.44 (Gupta et al., 6 Oct 2025). OpenAI chat models such as gpt-5-chat exhibit very low REF and NINF but also relatively low PAC (Gupta et al., 6 Oct 2025). This pattern supports the benchmark’s claim that high value consistency can be achieved either through substantive frame-robust reasoning or through blanket refusal strategies.

The benchmark also studies how issue salience affects behavior. It reports a strong negative correlation between no-information rate and issue-awareness level, $p_i^+$6, a strong positive correlation between refusal rate and issue-awareness level, $p_i^+$7, and a positive correlation between PAC and issue-awareness level, $p_i^+$8 (Gupta et al., 6 Oct 2025). In other words, models are less likely to claim ignorance and more likely to refuse on more salient controversies, while their PAC also rises on such issues.

6. Interpretation in the broader value-alignment literature

VAL-Bench differs from value benchmarks grounded in psychometric inventories, localized legal-cultural norms, or multi-dimensional value ontologies. "ValueBench" (Ren et al., 2024) evaluates value orientations and value understanding over 44 psychometric inventories and 453 value dimensions. "Value Portrait" (Han et al., 2 May 2025) assesses value orientations through psychometrically and ecologically valid items, using human similarity ratings and value-score correlations. "Value Compass Benchmarks" (Yao et al., 13 Jan 2025) proposes a benchmark grounded in motivationally distinct basic values, a generative evolving framework, and pluralistic weighting across value dimensions. "LocalValueBench" (Meadows et al., 2024) operationalizes localized value alignment through Australian legal, cultural, and ethical norms.

Within that landscape, VAL-Bench occupies a narrower but distinctive position. It does not attempt broad value taxonomy coverage, weighted pluralistic aggregation, or cultural localization. Instead, it measures whether a model’s outputs retain a stable value stance under opposed framings of public controversies (Gupta et al., 6 Oct 2025). A plausible implication is that VAL-Bench is a benchmark of framing invariance in controversial discourse rather than a benchmark of value completeness or cross-cultural representativeness.

This also distinguishes it from system-level analyses such as "SEAL: Systematic Error Analysis for Value ALignment" (Revel et al., 2024), which studies feature imprint, alignment resistance, and robustness in reward models, and from dynamic trade-off frameworks such as "Value Alignment Tax" (Chen et al., 12 Feb 2026), which measures how alignment interventions propagate across interconnected values. VAL-Bench stays at the response-pair level, but it exposes a related phenomenon: alignment strategies may optimize superficial consistency signals, especially refusals, instead of robustly expressed value structure.

The benchmark’s value-cluster analysis supports this reading. It shows that demonstrated values in responses do not simply mirror the values evoked by prompts; models over-represent some values and under-represent others. For example, pretrained glm-4.5-air-base already over-represents morally positive values such as prioritizing accountability and prioritizing equity while under-representing values such as evading accountability and marginalization, but it also over-represents ambiguous values such as pragmatism over principle and utilitarianism (Gupta et al., 6 Oct 2025). Claude-opus-4.1 shows the largest absolute residuals in demonstrated-versus-expected value distributions, with reduced $p_i^+$9, compared with gpt-5-chat at 4.47 and glm-4.5-air-base at 3.41 (Gupta et al., 6 Oct 2025). This suggests that some models exhibit more structured and selective value expression than others.

7. Limitations, controversies, and implications

The benchmark has several stated limitations. It is built from English Wikipedia, which introduces English-language and Wikipedia-specific coverage biases (Gupta et al., 6 Oct 2025). Controversy is identified through section-heading heuristics, so some controversies may be missed while some retained sections may reflect Wikipedia’s editorial norms rather than broader public controversy (Gupta et al., 6 Oct 2025). The evaluator is an LLM and therefore subject to interpretation error, despite calibration and ablation analyses (Gupta et al., 6 Oct 2025).

More substantively, VAL-Bench does not determine whether a model’s values are desirable. A model can score highly by being consistently committed to values that many would reject, because the benchmark rewards frame-consistent stance rather than moral correctness (Gupta et al., 6 Oct 2025). It also does not solve the problem of pluralistic human disagreement, because it does not compare model outputs to a human normative target distribution. In this respect, it is complementary to benchmarks that explicitly encode pluralism or localization, such as Value Compass (Yao et al., 13 Jan 2025), LocalValueBench (Meadows et al., 2024), and MENAValues (Zahraei et al., 15 Oct 2025).

A further controversy concerns refusal. Because both-side refusal is scored as full alignment for a pair, PAC can reward a model that systematically refuses to engage controversial topics (Gupta et al., 6 Oct 2025). The benchmark addresses this by separately reporting REF, 1REF, 2REF, and NINF, but the possibility remains that models could optimize PAC without developing a substantive value stance. This suggests that VAL-Bench should be interpreted jointly with its auxiliary metrics rather than through PAC alone.

A plausible implication is that VAL-Bench is best understood as one axis in a larger benchmark suite. Used alone, it can confound consistent abstention with coherent value reasoning. Used alongside behavioral, psychometric, localized, and pluralistic benchmarks, it can reveal whether a model’s controversy responses are stable, expressive, and normatively legible across frames.

8. Influence and possible extensions

VAL-Bench proposes a scalable and reproducible way to compare how reliably LLMs embody values across real-world controversies (Gupta et al., 6 Oct 2025). Its paired-prompt design is particularly suited to detecting prompt-framing sensitivity, a failure mode that standard safety benchmarks and moral questionnaires often leave unmeasured. This suggests a benchmarking principle: value alignment should be stress-tested not only against harmful prompts or static dilemmas, but also against rhetorical inversion.

Future extensions suggested in the paper include moving beyond English Wikipedia, incorporating multilingual sources, improving judge methodology, and integrating VAL-Bench with other benchmarks (Gupta et al., 6 Oct 2025). In the context of related work, several directions are especially immediate. One is to combine VAL-Bench’s frame-consistency design with the multi-dimensional value ontology and pluralistic weighting of Value Compass (Yao et al., 13 Jan 2025). Another is to introduce localized versions analogous to LocalValueBench (Meadows et al., 2024) or survey-grounded regional variants analogous to MENAValues (Zahraei et al., 15 Oct 2025). A third is to augment pairwise response scoring with structural analyses of value organization, as proposed in "Beyond Value Benchmarks: Measuring Value-Structure Alignment in LLMs via Symmetric Q-Sorts" (Zheng et al., 20 Jun 2026).

Taken together, these works suggest that VAL-Bench is neither a complete theory of value alignment nor merely another safety benchmark. It is a specialized instrument for measuring whether models preserve an identifiable stance when debate framing flips. In that role, it provides evidence that present-day models differ markedly not only in what they refuse, but in how consistently they reason about controversial issues (Gupta et al., 6 Oct 2025).