---
title: DailyDilemmas Corpus & LLM Value Audits
url: https://www.emergentmind.com/topics/dailydilemmas-corpus
type: topic
---

# DailyDilemmas Corpus & LLM Value Audits

DailyDilemmas Corpus refers to a class of datasets designed to systematically capture, represent, and quantify everyday moral and value-laden decision-making, whether by humans or by artificial agents such as large language models (LLMs). Two principal instantiations are notable in the literature: the large-scale community-judgment–oriented corpus introduced as Scruples [2008.09094] and the value-theory–aligned DailyDilemmas corpus for LLM value audits [2410.02683]. Both resources provide empirical foundations for evaluating normative reasoning, model alignment, and the diversity of value priorities in practical settings.

## 1. Corpus Construction and Annotation Schema

The construction of DailyDilemmas-style corpora incorporates rigorous collection, annotation, and curation methods to ensure authenticity and theoretical breadth.

### Community Ethical Judgments (Scruples)
- **Source and Scope**: Scruples is constructed from 32,766 real-world anecdotes drawn from Reddit’s r/AmItheAsshole (AITA) forum, using posts archived from November 2018–April 2019. Each post chronicles an individual’s description of a normatively salient event, with a required historical (“AITA”) or hypothetical (“WIBTA”) tag specifying the temporal context [2008.09094].
- **Annotation Mode**: Top-level community votes are captured using standard AITA initialisms (yta, nta, esh, nah, info), which are mapped to five normalized classes: author (wrong), other (wrong), everybody, nobody, info. Each anecdote maintains tallied counts $Y_{ij}$ per label $j$, yielding a label distribution $p_{ij} = Y_{ij}/N_i$. Lexical analysis reveals high diversity (13.5M tokens, 64,476 types).
- **Label Distribution**: In the dev split, the annotation density is a median of 8 comments per anecdote (total: 52,433 annotations on 2,500 anecdotes), with frequencies: other (54.4%), author (29.8%), nobody (8.9%), everybody (4.8%), info (2.1%).

### Explicit Value Quandaries (DailyDilemmas)
- **Design**: DailyDilemmas contains 1,360 binary-choice dilemmas stratified across 17 everyday topics (80 dilemmas per topic) [2410.02683]. Each instance is independent, featuring a scenario setup, two explicit actions $\mathcal A^{do}$ (to do) and $\mathcal A^{not}$, negative-consequence stories for each action, involved parties, and a detailed perspectives annotation.
- **Value Extraction**: For every action, each affected party is assigned at least one (value, reason) pair through chain-of-thought reasoning, followed by aggregation into sets $\{v^{do}\}$ and $\{v^{not}\}$—capturing the core value conflict of the dilemma.

## 2. Human Values Inventory and Operationalization

Central to DailyDilemmas is its comprehensive and systematic cataloging of human values as they are manifested in ordinary ethical choices.

- **Value Set**: 301 unique values with at least 100 instances each were identified using a GPT-4–driven induction process, and subsequently filtered and validated by human annotators. The top values by frequency include: trust, self, honesty, responsibility, respect, empathy, understanding, fairness, integrity, accountability (see Table 1).
- **Operational Definition**: Each value is operationalized in the annotation protocol as a tuple: (party, value phrase, one-sentence justification). There are no closed-form definitions; rather, justification is contingent on scenario-specific reasoning.
- **Validation and Reliability**: Human annotator validation on Reddit posts yields F1 = 85.7%, precision = 81.8%, recall = 90.0%, and Cohen’s κ = 0.526. Word-level value match in top Reddit comments is 60.02% (SD 14.2%).

| Statistic               | Scruples [2008.09094]          | DailyDilemmas [2410.02683]   |
|-------------------------|-------------------------------|------------------------------|
| Scenarios/Instances     | 32,766 anecdotes              | 1,360 dilemmas               |
| Label space             | 5-class distribution          | 2 actions, values per action |
| Values catalogued       | N/A (focus on majority class) | 301 distinct values          |
| Domains/topics          | r/AITA diversity              | 17 stratified topics         |

## 3. Theoretical Frameworks for Value Mapping

A distinguishing feature of the DailyDilemmas resource is its explicit mapping of values to five canonical theories of values, facilitating multidimensional audit and comparison.

- **World Values Survey (WVS)**: Two axes—Traditional–Secular-rational and Survival–Self-expression—serve as macro-scale value dimensions.
- **Moral Foundations Theory (MFT)**: Values are binned into Care, Fairness, Loyalty, Authority, Purity.
- **Maslow’s Hierarchy of Needs**: Values correspond to needs levels—Physiological, Safety, Love/Belonging, Self-esteem, Self-actualization.
- **Aristotle’s Virtues**: Nine classical virtues, including Truthfulness, Courage, Patience, Liberality, Ambition, and Friendliness.
- **Plutchik’s Wheel of Emotions**: Values are associated with core emotions: Trust, Joy, Fear, Sadness, Disgust, Anger, Anticipation, Surprise.

Each value was manually assigned to categories using standard definitions. Distribution plots (e.g., Fig. 3) reveal that Self-expression (WVS), Fairness (MFT), Self-esteem (Maslow), Truthfulness (Virtues), and Trust (Emotions) dominate the corpus.

## 4. Quantitative Statistics, Formulas, and Evaluation

Evaluation of models and values within these corpora requires distribution-sensitive metrics and carefully defined operational formulas.

- **Soft Label Vectors**: For Scruples, a gold standard prediction is the full distribution vector $p_{i:}$ inferred from crowd votes, not a mode label.
- **SCOracle Estimator**: Establishes a Bayes-optimal lower bound for cross-entropy by modeling $p_{i:}$ with a Dirichlet prior, fitting hyperparameters $\alpha$ through empirical Bayes, and estimating expected cross-entropy via
  $$
  s = \sum_i \mathbb{E}_{p_{i:}|Y_{i:},\hat\alpha} \left[ -\sum_j \left(\frac{Y_{ij}}{N_i}\right)\log p_{ij} \right]
  $$
  This estimator’s relative error is less than 1% in simulation [2008.09094].
- **Modeling Likelihoods**: Both BERT-Large and RoBERTa-Large are evaluated using:
    - Standard Softmax/Categorical likelihood,
    - Dirichlet-multinomial output layers—separating intrinsic controversy from model uncertainty.
- **DailyDilemmas Evaluation**: The core metrics are (i) choice frequency for each action, (ii) value-alignment rate, and (iii) preference-difference per value-theory dimension:
  $$
  \Delta p = p_{\text{model},A} - p_{\text{model},B}
  $$
  where $p_{\text{model},A}$ and $p_{\text{model},B}$ are normalized selection frequencies for each side of a theoretical axis.
- **Value Preference Delta**: For a given dilemma,
  $$
  \Delta v = |v^{selected}| - |v^{neglected}|
  $$

## 5. Empirical Findings and Model Performance

Performance analysis highlights both the challenge and potential of distributional and value-theoretic moral judgment modeling.

### Scruples Findings
- **Full Corpus**: RoBERTa with Dirichlet-multinomial achieves F1-macro = 0.302/0.259 (dev/test), cross-entropy ≈ 1.03, versus a human majority-vote oracle at F1 ≤ 0.49, CE ≈ 0.742. Simplified variants yield higher alignment (F1-macro ≈ 0.78) [2008.09094].
- **Ambiguity**: Numerous real dilemmas are fundamentally divisive; single-label evaluation over-penalizes models on these.
- **Modeling Improvements**: Dirichlet-multinomial likelihood outperforms soft labels/counts-based alternatives for cross-entropy, reifying both intrinsic and epistemic uncertainty.

### DailyDilemmas (LLM Value Alignment)
- **Model Choices**: All models (GPT-4-turbo, GPT-3.5-turbo, Llama-2/3, Mixtral-8x7B, Claude-Haiku) favor Self-expression over Survival (WVS), and Care over Loyalty (MFT) [2410.02683].
- **Model Differences**: Notable divergence in value selection (e.g., Truthfulness: GPT-4-turbo +9.4%, Mixtral-8x7B –9.7% relative selection); safety values are universally de-prioritized; ambition, friendliness, and courage show model-specific patterns.
- **Steerability**: Prompt-based steering via principles from OpenAI ModelSpec or Anthropic Constitutional AI has only weak effect on GPT-4-turbo’s value priorities.
- **Human Validation**: Semantic value annotation in DailyDilemmas exhibits robust correspondence (F1 = 85.7%, κ = 0.526) with real Reddit community responses.

## 6. Methodological Implications and Applications

The DailyDilemmas corpus—across both Scruples and value-theory instantiations—introduces key methodological recommendations.

- **Distributional Ground Truth**: Gold labels as empirical distributions are essential for ambiguous dilemmas; single-label metrics obscure true ambiguity.
- **Dirichlet-multinomial Output Layers**: These are recommended to jointly model community disagreement and model confidence, improving calibration and interpretability.
- **SCOracle as Empirical Upper Bound**: Adopting the SCOracle estimator is critical for realistic upper-bound benchmarking under label dissent.
- **Value Operationalization**: Comprehensive value extraction (301 values) and rigorous mapping to value theories enable fine-grained, theory-driven audit and alignment of LLMs.
- **Public Resources**: All data, code, and prompts are available at [HF Datasets](https://hf.co/datasets/kellycyy/daily_dilemmas) and [GitHub](https://github.com/kellycyy/daily_dilemmas).

Potential applications include value-alignment audits of emerging LLMs, targeted RLHF or instruction-tuning along under-represented values, value-aware decision-support tooling, and cross-cultural or theory-based analyses of normative reasoning.

## References

- "Scruples: A Corpus of Community Ethical Judgments on 32,000 Real-Life Anecdotes" [2008.09094]
- "DailyDilemmas: Revealing Value Preferences of LLMs with Quandaries of Daily Life" [2410.02683]

Source: https://www.emergentmind.com/topics/dailydilemmas-corpus