ValueEval’24: Automated Value Analysis

• ValueEval’24 is an evaluation framework that automates the recognition and analysis of human values in natural language arguments using a multi-label approach.
• It integrates diverse datasets and advanced annotation protocols to address challenges such as class imbalance, multi-label intricacies, and domain heterogeneity.
• The framework sets rigorous baselines to spur research in cross-domain generalization, imbalance handling, and ethical AI alignment.

ValueEval’24 is an academic initiative and associated evaluation framework centered on the automated recognition and analysis of human values as expressed in natural language arguments. The impetus for ValueEval’24 arises from growing interest in computational modeling of social, ethical, and cultural values for information retrieval, argument mining, and AI alignment. ValueEval’24 builds on and substantially extends earlier resources, most notably the Touché23-ValueEval dataset, offering greater cross-domain coverage and increased annotation granularity. The task emphasizes leveraging diverse textual sources, mitigating cultural bias, and addressing technical challenges such as extreme class imbalance and multi-label annotation (Mirzakhmedova et al., 2023).

1. Dataset Foundations and Construction

ValueEval’24’s core resource is the Touché23-ValueEval dataset, comprising 9,324 arguments sourced from heterogeneous domains:

• IBM-ArgQ-Rank-30kArgs: 7,368 quality-controlled, crowd-written pro/con arguments on 71 controversial topics.
• Conference on the Future of Europe (CoFE): 1,098 English proposals and comments, featuring manual extraction of premise–conclusion pairs and stance annotations.
• Group Discussion Ideas (GDI): 399 debate arguments from an Indian educational platform, normalized post-crawling.
• Zhihu: 100 Chinese Q&A arguments, paraphrased and translated for representational diversity.
• Nahj al-Balagha: 279 distilled Islamic aphorisms, premise–conclusion split and stance-balanced (additional material unannotated).
• The New York Times (NYT): 80 editorial arguments on epidemic/vaccine topics, annotated by linguists.

Argument lengths and structures were harmonized to match predecessor datasets (e.g., premise 20–45 tokens, conclusion 6–20 tokens). Data splits ensure that all arguments with the same conclusion reside in the same partition to eliminate train–test leakage. Quality assurance protocols—crowdworker filtering, expert linguistic review, encoding correction, manual restructuring, and systematic translation—were rigorously implemented.

2. Annotation Procedures and Taxonomy

A detailed annotation grid supports granular value detection:

• Value taxonomy: 54 “level 1” human values, each subsumed under one of 20 “level 2” categories. The taxonomy adapts theoretical frameworks from Schwartz (1994) and the Webis-ArgValues-22 dataset.
  • Examples: Self-direction, Security (personal/societal), Benevolence, Universalism (concern, nature, tolerance, objectivity).
  • Each value paired with 2–3 concise application examples.
• Crowdsourcing: 27 pre-validated annotators (13 returned for new data), each decision rendered by 3 workers in a yes/no format: “Does this argument appeal to ⟨value-definition⟩?”
• Aggregation: Gold labels inferred using MACE, with thresholding for confidence; ambiguous cases subject to expert correction.
• Agreement: MACE-inferred judgments reached average >0.85 confidence for 90% of values; <0.2% of cases had no value assigned.

3. Statistical Structure and Label Distribution

The dataset exhibits pronounced multi-label characteristics and skewed distributional properties:

• Label multiplicity: 94% of arguments exhibit ≥2 level 1 values; mean is 3.4 values per argument.
• Category co-occurrence: Strongest in the Universalism cluster; Security: societal and Benevolence: caring frequently co-label.
• Imbalance: Most common value “Be just” appears in 24% of arguments, while rare values (e.g., “Be daring”) constitute <0.1%.
• Class ratio: Maximum to minimum value frequency is approximately 84:1.
• Stance split: 56% pro, 44% con.
• Source-specific category spikes: Universalism: concern dominates IBM/CoFE/GDI; Security: personal prevails in NYT; Achievement and Societal Security frequent in Zhihu; Nahj achieves greatest balance.

4. Baseline Methods and System Evaluation

ValueEval’24 defines standardized baselines and metrics for fairness and reproducibility:

• Baselines:
  • 1-Baseline: Assigns all possible values to each argument (max recall, zero precision).
  • BERT-base: 12-layer Transformer (bert-base-uncased), [CLS] conclusion [SEP] premise [SEP] input; output from the CLS vector via 54 sigmoid units (multi-label); trained (3 epochs, 2×10⁻⁵ learning rate, batch size 16, AdamW, weight decay 0.01).
• Performance metrics:
  • Precision, Recall, F₁ (per-value and macro-averaged).
  • Accuracy as overall per-label fraction.
• Empirical results (test set, level 1): | Model | Macro-P | Macro-R | Macro-F₁ | Acc | |-------------|---------|---------|----------|------| | 1-Baseline | 0.07 | 1.00 | 0.13 | 0.07 | | BERT-base | 0.43 | 0.19 | 0.26 | 0.94 |

Performance changes from Webis-ArgValues-22 to Touché23-ValueEval reflect greater label heterogeneity: BERT macro-F₁ improves (level 1: 0.25→0.26, level 2: 0.34→0.44), while baseline F₁ drops due to more diffused value distribution.

5. Technical Challenges in Value Modeling

Multiple aspects complicate automated value detection:

• Extreme class imbalance: High ratio induces false positives for rare labels, false negatives for frequent ones.
• Multi-label intricacies: Averaging 3.4 positive values per argument complicates thresholding for sigmoids.
• Source/domain heterogeneity: Genre- and culture-specific usages of values.
• No-value occurrence: True “no value” assignments are rare but present (<0.2%).
• Evaluation difficulties: Macro metrics penalize both rare-value overprediction and underprediction, making classifier calibration and per-class metric interpretation non-trivial.

6. Research Directions and Open Problems

ValueEval’24 emerges as an experimental and benchmarking platform for advancing methodological frontiers:

• Domain generalization: Multi-source diversity encourages research in cross-domain and culture-aware classifiers, potentially exploiting domain adversarial training (Mirzakhmedova et al., 2023).
• Imbalance handling: Focal loss, paraphrasing-based augmentation, and hierarchical (coarse-to-fine) classification are advocated.
• Zero-/few-shot learning: Especially germane to tail values; prompt-based LLM fine-tuning with value definitions is suggested.
• Taxonomic extension: New genres (e.g., speech transcripts, social media, additional religious/philosophical corpora) and languages need coverage.
• Annotation methodology: Open questions include value implicitness, premise–conclusion structure integration, and mitigating annotator cultural bias—potentially via cross-cultural annotation rounds.
• Downstream evaluation: Value inference utility is to be tested in tasks such as persuasive argument generation and audience modeling.

7. Significance and Impact

The ValueEval’24 framework, predicated on comprehensive, high-fidelity multi-label value annotation and robust methodological baselines, provides a challenging environment for research in computational value detection and argument understanding. Its dataset, with systematic attention to diversity, annotation reliability, and cross-cultural relevance, enables rigorous evaluation and comparison of advanced natural language processing models. Challenges resolved within ValueEval’24—especially those related to label imbalance, implicit value recognition, and multi-source robustness—will influence both the theoretical and practical advances in automated argument analysis, AI alignment, and computational social science (Mirzakhmedova et al., 2023).