---
title: ValueEval'24 / ValuesML Overview
url: https://www.emergentmind.com/topics/valueeval-24-valuesml
type: topic
---

# ValueEval'24 / ValuesML Overview

ValueEval’24 and ValuesML collectively denote a set of evaluation frameworks, datasets, and modeling paradigms aimed at probing, benchmarking, and advancing the understanding and steerability of large language models (LLMs) and related systems with respect to human values, preferences, and stylistic attributes. These efforts span the automated detection of values in text, the assessment of model value comprehension (“know what”/“know why”), the steerability of reward models towards user value/style profiles, and the development of calibration, transferability, and robustness for value-related tasks. The following sections chronologically and thematically organize the major methodologies, datasets, evaluation metrics, findings, and future recommendations that define the ValueEval’24 / ValuesML landscape.

## 1. Datasets and Value Taxonomies

The foundation of ValueEval’24 and ValuesML rests on large, annotated corpora encoding a variety of value taxonomies, value expressions, and linguistic contexts. Central resources include:

- **ValueEval’23 / Touché’23-ValueEval**: Contains 9,324 premise–conclusion–stance arguments drawn from six culturally and stylistically heterogeneous sources: IBM-ArgQ, CoFE, GDI, Zhihu, Nahj al-Balagha, and NYT. Each argument is annotated on 54 Level-1 values (binary), grouped into 20 Level-2 categories following the Schwartz (1994) framework and its contemporary expansions. Annotation relied on three crowdworkers per argument and MACE for label fusion [2301.13771].
- **ValueEval’24 / ValuesML**: Scales to 74,231 English sentences, primarily for sentence-level value detection with Schwartz values. Label spaces include 19 basic values (e.g., Self-direction: thought, Security: societal) and 8 higher-order (HO) categories derived via logical ORs over the corresponding basic values. Additionally, a binary “Presence” label marks the existence of any value [2602.00913].
- **ValEval Dataset (CLAVE framework)**: Comprises 13,000+ (text, value, label) tuples, supporting three value systems: Social Risk, Schwartz Basic Values (10 classic types), and Moral Foundations (5 dimensions). Extensive three-way annotation (majority vote; IAA 85–88%) and coverage across domains, perturbation, and generalization splits [2407.10725].

These corpora enable robust evaluation across registers, domains, topics, and linguistic/cultural backgrounds by capturing both fine-grained and abstracted value signals.

## 2. Task Formulations and Formal Metrics

Multiple related but distinct task formulations underpin ValueEval’24 / ValuesML efforts:

- **Value detection**: Multi-label classification over value taxonomies, either at argument/stance (Touché’23) or sentence level (ValueEval’24), with precision, recall, F₁, accuracy, and macro-F₁ as central metrics [2301.13771, 2602.00913].
- **Comprehensive value understanding**: Quantified by ValueDCG, measuring both the model’s ability to pick the correct value label ("discriminator"—“know what”) and to generate a rationale aligned with expert annotation ("critique"—“know why”). The absolute mean gap (ValueDCG) between the two scores is a metric of holistic value understanding [2310.00378].
- **Reward model steerability**: Assessed via EVALUESTEER, quantifying whether RMs or LLM-judges rank candidate responses in alignment with a user’s composite value and stylistic profile. Pairwise accuracy over $N$ controlled preference pairs is the main metric, with context- and oracle-conditioned protocols [2510.06370].
- **Reference-free value labeling**: As in CLAVE, assign "adhere", "oppose", or "unrelated" for a system response given a value definition and context. Measured by accuracy, macro-F1, and expected calibration error (ECE) [2407.10725].

Mathematical formalizations dictate clear, reproducible evaluation and benchmarking for each sub-task.

## 3. Model Architectures, Baselines, and Calibration

A diversity of model architectures and strategies are benchmarked in ValueEval’24 / ValuesML:

- **Supervised encoders**: E.g., DeBERTa-Base fine-tuned with binary cross-entropy for value multi-labeling; BERT-based multilabel classifiers with example-level concatenation [2301.13771, 2602.00913].
- **Instruction-tuned LLMs**: Llama-3.1 8B, Gemma 2 9B, and others evaluated zero-shot and few-shot, with outputs parsed into multi-hot vectors [2602.00913].
- **Hierarchical and hard-gated architectures**: Multi-phase pipelines enforcing HO category structure via masking or gating at inference (HO→values, Presence→HO→values). Empirical findings indicate that hard constraints often reduce Macro-F₁ via error compounding and recall suppression [2602.00913].
- **Ensembling**: Soft-voting and cross-family ensemble approaches (transformer + instruction-tuned LLM) yield modest but reliable gains in Macro-F₁, up to +0.02 absolute depending on slice [2602.00913].
- **Calibration**: Label-wise threshold tuning (per-label $\tau_k$ search) outperforms global thresholds and unlocks up to +0.05 Macro-F₁, especially for imbalanced or ambiguous labels [2602.00913].
- **CLAVE dual-model framework**: Large LLMs extract “value concepts” from sparse human annotation, enabling calibration and transfer of a smaller, fine-tuned recognizer that maintains accuracy and robustness on minimal data [2407.10725].

Notably, small LLMs ($\le$10B) trail supervised transformers in single-label settings but contribute complementary error patterns when ensembled.

## 4. Advanced Evaluation Frameworks and Steerability

Two recent innovations have extended the evaluation regime well beyond raw multi-label accuracy:

- **ValueDCG** [2310.00378]: For each model and question, the Discriminator (D) score is the semantic similarity between the model’s answer and the correct baseline value answer, while the Critique (C) score is the similarity between the model’s explanation and the expert rationale. The ValueDCG metric is their average absolute difference:
  $$
  \mathcal{Q}_{\mathrm{dcg}(m)} = \mathbb{E}_{x\sim S}
  \Bigl| \mathcal{Q}_{\mathrm{dis}(m,x,v_c)}
  - \mathcal{Q}_{\mathrm{cri}(m,x,v_c)} \Bigr|.
  $$
  “Know what” scales with model size, but “know why” shows only minimal scaling. Persistent gaps suggest plausible rationalization without deep value alignment.

- **EVALUESTEER** [2510.06370]: Systematically generates 165,888 preference pairs by crossing value dimensions (Inglehart–Welzel map) and stylistic dimensions (verbosity, difficulty, confidence, warmth). Steerability is quantified as the pairwise accuracy of RMs given user profiles U:
  $$
  \textrm{Accuracy} = \frac{1}{N} \sum_{n=1}^N 1[ \hat i_n = i_n^* ]
  $$
  where $\hat i_n$ is the model-chosen candidate and $i_n^*$ is ground-truth aligned with U. Even the best systems (GPT-4.1-Mini, CoT, "prefer values") attain $<75\%$ steerability, sharply below the oracle ($>99\%$) with only relevant info, revealing major limitations in context utilization, value–style disentanglement, and implicit bias.

## 5. Empirical Findings, Biases, and Error Analysis

Core empirical discoveries and systematic trends include:

- **Scaling effects**: Larger models improve “know what” (value discrimination) more than “know why” (rationale quality). ValueDCG shrinks as model size grows but remains non-zero. Context priming with value induction lowers ValueDCG for all models, emphasizing strong sensitivity to explicit cues [2310.00378].
- **Dataset size and diversity**: Enlarging and diversifying (Touché’23 vs. Webis-ArgValues-22) exposes the limits of trivial baselines (1-Baseline F₁ drops from 0.16→0.13) but enables more effective supervised learning (BERT macro-F₁ rises from 0.25→0.26 at Level 1, 0.34→0.44 at Level 2) [2301.13771].
- **Effectiveness of hierarchy**: While HO categories in the Schwartz framework are learnable at the sentence level, enforcing them via hard gating or presence cascades suppresses end-task recall and offers no reliable Macro-F₁ advantage. Calibration and ensembling remain more effective under compute constraints [2602.00913].
- **Stylistic and value biases**: LLM-based RMs show systematic biases towards secular-rational, self-expression, verbose, high-confidence, and formal/cold style options (verbosity bias, style-over-substance). When values and style conflict, systems default to style preference by up to 33 percentage points [2510.06370].
- **Adaptability and overfitting**: CLAVE’s concept abstraction layer allows accurate, robust alignment to new value definitions with $<100$ samples per value, outperforming pure fine-tuned baselines especially OOD or when training data is scarce [2407.10725].

## 6. Practical Recommendations and Future Directions

Synthesis across recent studies suggests:

- **Prefer robust calibration and ensembling over hard constraints**: Label-wise threshold optimization and lightweight model ensembling yield the most consistent gains for value detection at sentence or argument level [2602.00913].
- **Integrate value-grounded reasoning in RM training**: Application of chain-of-thought and explicit value balancing in multi-objective architectures (e.g., Mixture-of-Experts on value/style) is recommended [2510.06370].
- **Broaden cultural-ethical axes and dataset coverage**: Extend beyond Schwartz/Inglehart to include underrepresented ontologies (Indigenous, non-Western) and native language corpora [2301.13771, 2510.06370].
- **Develop transparent, interpretable systems**: Emphasize alignment between predictions and rationale explanations (e.g., via attention rationales or integrated gradients), enabling justification as well as accurate labeling [2301.13771, 2310.00378].
- **Exploit value concept abstraction in evaluation and transfer**: Future ValuesML explorations may harness concept-driven transfer for explainability, domain adaptation, and cross-value-system calibration [2407.10725].
- **Audit and mitigate model biases**: Routine analysis for implicit cultural, value, and style biases is necessary as model deployment widens [2510.06370].

## 7. Limitations, Open Challenges, and Perspective

The current ValueEval’24 / ValuesML toolkits represent a substantial advance in systematic, large-scale, and fine-grained value evaluation for LLMs. However, persistent challenges remain:

- End-task accuracy on steerability and value detection is well below the oracle or inter-annotator ceiling, even for the largest models and best calibrated ensembles.
- Hard-coded hierarchies, while attractive theoretically, risk compounding errors and suppressing minority value expressions in practical settings.
- LLMs continue to exhibit context- and prompt-driven behavior, with value “understanding” largely contingent on explicit cues or training distribution properties rather than internalized semantic knowledge.
- The gap between plausible rationalization and genuine value alignment (as quantified by ValueDCG) signals risk for applications demanding high safety, normative compliance, or pluralistic adaptability.

Ongoing research aims to bridge these gaps through hierarchical soft conditioning, cross-cultural dataset enrichment, mechanistic interpretability, and improved concept-based calibration frameworks. As the ValuesML field matures, ValueEval’24 benchmarks, datasets, and analytic methodologies will constitute the reference standard for quantifying, auditing, and advancing human value alignment in LLMs [2301.13771, 2310.00378, 2407.10725, 2510.06370, 2602.00913].

Source: https://www.emergentmind.com/topics/valueeval-24-valuesml