Deep Value Benchmark (DVB) Evaluation

• Deep Value Benchmark (DVB) is an evaluation framework that distinguishes deep normative values from shallow, surface-level features in models.
• The benchmark employs a confound-then-deconfound protocol to first correlate and then decouple deep values from superficial cues during training and testing.
• Its primary metric, the Deep Value Generalization Rate (DVGR), quantifies a model’s tendency to generalize on deep values, with empirical findings showing mean DVGR at 0.30.

Searching arXiv for the cited benchmark paper to ground the article in the primary source. Deep Value Benchmark (DVB) is an evaluation framework for testing whether LLMs learn underlying human values or merely surface-level preferences from preference data. It is designed around a distinction between “deep values,” understood as second-order desires or normative principles, and “shallow features,” understood as first-order preferences over morally neutral surface attributes. The benchmark operationalizes this distinction through a controlled training-and-test protocol that first correlates values and superficial features and then breaks that correlation, thereby measuring whether a model generalizes on the basis of the value or the superficial cue. Its central metric is the Deep Value Generalization Rate (DVGR), defined as the probability that a model selects the value-aligned option under deconfounding. In the reported evaluation across 9 models, the mean DVGR is 0.30, with all models below chance, which the benchmark presents as an interpretable measure of a core alignment property (Ashkinaze et al., 3 Nov 2025).

1. Conceptual basis

DVB addresses an alignment problem that arises when systems are trained on human preference data. The benchmark is motivated by the possibility that such systems may learn superficial correlations—“shallow preferences”—rather than the deeper values that actually guide human choices. In this framing, superficial patterns may be adequate in familiar settings yet fail under distributional shift, producing misaligned behavior in novel or high-stakes situations (Ashkinaze et al., 3 Nov 2025).

Within the benchmark, deep values are defined as second-order desires or normative principles. The value set is drawn from W.D. Ross’s prima facie duties—beneficence, fidelity, justice, non-maleficence, reparation, and self-improvement—and Schwartz’s social values—security, conformity, tradition, benevolence, and universalism. An example given is preferring doctors who respect patient autonomy regardless of specialty. Shallow features, by contrast, are first-order preferences over morally neutral surface attributes, such as writing style, frequency of feedback, or avatar accent. An example is preferring AI responses in formal language even when the deeper motivation is clarity or professionalism (Ashkinaze et al., 3 Nov 2025).

This conceptual distinction is central to DVB’s interpretation. A model that captures deep values should, under changed surface conditions, continue to track the underlying normative preference. A model that captures only shallow features should instead follow the superficial attribute when the two are placed in conflict. This suggests that DVB is not a generic preference benchmark, but a targeted probe of whether apparent preference learning reflects value-level abstraction rather than proxy fitting.

2. Benchmark design and experimental protocol

DVB uses what it describes as a “confound-then-deconfound” design. Its components are 6 prima facie duties plus 5 social Schwartz values, 20 human-validated morally neutral dichotomies as shallow preferences, and 8 real-world domains crossed with 10 high-relevance O*NET work activities. The benchmark’s purpose is to create training data in which deep values and shallow features are perfectly correlated, and then to test whether a model follows the value or the shallow feature when the correlation is removed (Ashkinaze et al., 3 Nov 2025).

In the training phase, the protocol samples 50 unique $(v_1, v_2, s_1, s_2)$ tuples per context. For each tuple, it generates $N$ in-context training examples, with $N \in \{5, 20, 40\}$, such that the user consistently prefers $(v_1, s_1)$ over $(v_2, s_2)$. Because these examples confound the deep value and the shallow feature perfectly, they do not by themselves reveal whether a model’s generalization is value-based or surface-based.

In the testing phase, the benchmark generates 10 test prompts per tuple. Each test prompt presents two options that swap the shallow features: Option A is $(v_1, s_2)$ and Option B is $(v_2, s_1)$. Under this construction, only one underlying deep-value advantage remains, namely $v_1$. A model that internalized the deep value should select Option A, whereas a model keyed to the shallow signal should select Option B (Ashkinaze et al., 3 Nov 2025).

The significance of this design lies in its controlled isolability. Because training confounds are deliberate and test-time deconfounding is explicit, the observed choice can be interpreted directly as evidence about whether generalization tracks the underlying value or the surface attribute. A plausible implication is that DVB is intended to measure a specific failure mode of preference learning rather than overall task competence.

3. Deep Value Generalization Rate

The benchmark’s primary metric is the Deep Value Generalization Rate. Let $K$ be the number of test questions. For each test instance $i$, the model predicts either the value-aligned option, $N$0, or the shallow-aligned option, $N$1. DVGR is defined as

$N$2

Under this definition, a DVGR of 1.0 corresponds to perfect deep value generalization, 0.0 corresponds to always generalizing the shallow preference, and 0.5 corresponds to chance (Ashkinaze et al., 3 Nov 2025).

DVB treats DVGR as an interpretable metric because it directly quantifies the tendency to act on deep values rather than shallow features. This directness is unusual relative to broader preference-evaluation settings in which multiple latent factors may contribute to a model’s choice. Here, the metric is tied to a specific intervention—deconfounding after confounded exposure—and therefore to a specific interpretive question.

The paper also notes an important qualification: deep values do not always drive real choices, so raw DVGR should be interpreted relative to baseline expectations. Trends between models or over time may therefore be more informative than absolute scores. This suggests that DVB is especially suited to comparative evaluation and progress tracking rather than to an unqualified binary judgment of alignment (Ashkinaze et al., 3 Nov 2025).

4. Dataset structure and validation

The DVB dataset is organized as 8 domains multiplied by 50 value/preference tuples multiplied by 40 training and 40 test natural-language scenarios, yielding approximately 32K completions. Each instance contains a context, two choices $N$3, a training label, and then a single test question per prompt. The domains are ecologically grounded and paired with realistic tasks via high-relevance O*NET work activities (Ashkinaze et al., 3 Nov 2025).

The benchmark reports three separate human validation experiments. First, for the shallow-versus-deep distinction, 41 crowdworkers rated 38 LLM-generated dichotomies plus deep values on shallowness. Participants reliably distinguished shallow from deep, with $N$4, and this process was used to select the top 20 shallow features. Second, an embodiment check used 210 trials in which crowdworkers matched completions to their intended value/preference pairs, achieving 98% accuracy. Third, a preference-plausibility study used 200 trials in which crowdworkers predicted which option a user with a stated value preference would choose, achieving 91% accuracy (Ashkinaze et al., 3 Nov 2025).

The benchmark summarizes these checks under a validity framework. Construct validity is supported by the fact that humans distinguish shallow from deep and that completions embody the intended attributes. Internal validity is supported through randomized option order, test isolation, and balanced neutrality. External validity is supported through ecologically grounded contexts and realistic tasks.

These validations matter because DVB’s claims depend on the semantic legibility of both the value dimension and the shallow-feature dimension. If the scenarios failed to instantiate the intended pairs, then low DVGR could not be cleanly attributed to shallow-feature generalization. The reported validation outcomes are therefore integral to the benchmark’s interpretability, not ancillary dataset documentation.

5. Empirical findings across models

DVB evaluates 9 models. The closed-source models are gpt-4o and gpt-4o-mini, gpt-4.1 and gpt-4.1-mini and gpt-4.1-nano, and gemini-2.0-flash and gemini-2.0-flash-lite. The open models are llama-3-8b-instruct and llama-3-70b-instruct (Ashkinaze et al., 3 Nov 2025).

The headline result is that the mean DVGR is 0.30, and all models generalize deep values less than chance, with significance reported as $N$5. Individual DVGR scores range from 0.23 for gpt-4.1-mini to 0.40 for gemini-2.0-flash. In 3 of 5 developer-paired comparisons, smaller models attain slightly higher DVGR, with mean $N$6 and $N$7 tests reported at $N$8 (Ashkinaze et al., 3 Nov 2025).

The paper further reports analyses by context and by value. Across contexts, DVGR ranges from 0.26 to 0.35, with commerce and healthcare highest and education and customer service lowest; the effect size is summarized as Cramér’s $N$9. Across values, tradition at 0.51 and universalism at 0.42 are highest, while fidelity and self-improvement are lowest; the effect size is Cramér’s $N \in \{5, 20, 40\}$0. Model-similarity analysis shows pairwise question-level agreement of approximately 74% on average, with same-developer pairs at approximately 77% versus approximately 72% for different-developer pairs; a mixed model reports $N \in \{5, 20, 40\}$1 percentage points and $N \in \{5, 20, 40\}$2 (Ashkinaze et al., 3 Nov 2025).

Follow-up prompt strategies produce mixed results. Chain-of-Thought reduces DVGR from 0.30 to 0.25. An explicit instruction to prioritize deep values increases DVGR to 0.33. The benchmark interprets this as indicating latent capability that requires strong prompting. A plausible implication is that the measured deficit is not necessarily an absolute inability to represent values, but a tendency for default inference behavior to privilege shallow cues over the intended normative abstraction.

6. Interpretation, uses, and limitations

DVB is presented as an interpretable alignment evaluation because DVGR directly measures whether a model acts on deep values or shallow features. The benchmark’s motivating example is a system that recommends only family-medicine doctors because that category was correlated with “explains thoroughly”; this illustrates how a superficial correlate can be mistaken for the deeper criterion that actually matters (Ashkinaze et al., 3 Nov 2025).

The framework is also described as extensible beyond value learning. The confound-then-deconfound paradigm is said to generalize to other intended-versus-proxy signal assessments, such as political ideology versus lexical cues. This suggests that DVB occupies a broader methodological space concerned with whether models internalize latent targets or shortcut via accessible correlates.

The paper is explicit about limitations. The artificial confound is a worst-case construction and may not reflect naturalistic settings, although it yields an unambiguous signal for measuring generalization preferences. The evaluation is inference-only and therefore tests in-context learning bias rather than the effects of fine-tuning or RLHF interventions. The scope of values and preferences is limited to a selected set of moral and social values and to the chosen shallow attributes. The paper identifies possible extensions to a broader value space, cross-cultural differences, and additional shallow features, and it points to downstream work on how improving DVGR affects real-world agent safety and trust (Ashkinaze et al., 3 Nov 2025).

A common misconception would be to read DVB as showing that models lack any capacity for value-sensitive reasoning. The reported increase under explicit instruction to prioritize deep values counsels against that interpretation. Another misconception would be to treat low DVGR as a direct estimate of real-world misalignment frequency. The benchmark instead frames raw DVGR as a controlled measure whose comparative trends may be more informative than its absolute magnitude.