DailyDilemmas Corpus & LLM Value Audits

• DailyDilemmas Corpus is a dataset of ethically charged scenarios annotated through community judgments and LLM value audits.
• It captures diverse human values across 17 topics using binary-choice dilemmas and detailed multi-label distributions.
• The corpus facilitates robust value mapping via established frameworks like WVS, MFT, and Maslow to guide LLM alignment.

DailyDilemmas Corpus refers to a class of datasets designed to systematically capture, represent, and quantify everyday moral and value-laden decision-making, whether by humans or by artificial agents such as LLMs. Two principal instantiations are notable in the literature: the large-scale community-judgment–oriented corpus introduced as Scruples (Lourie et al., 2020) and the value-theory–aligned DailyDilemmas corpus for LLM value audits (Chiu et al., 2024). Both resources provide empirical foundations for evaluating normative reasoning, model alignment, and the diversity of value priorities in practical settings.

1. Corpus Construction and Annotation Schema

The construction of DailyDilemmas-style corpora incorporates rigorous collection, annotation, and curation methods to ensure authenticity and theoretical breadth.

Community Ethical Judgments (Scruples)

• Source and Scope: Scruples is constructed from 32,766 real-world anecdotes drawn from Reddit’s r/AmItheAsshole (AITA) forum, using posts archived from November 2018–April 2019. Each post chronicles an individual’s description of a normatively salient event, with a required historical (“AITA”) or hypothetical (“WIBTA”) tag specifying the temporal context (Lourie et al., 2020).
• Annotation Mode: Top-level community votes are captured using standard AITA initialisms (yta, nta, esh, nah, info), which are mapped to five normalized classes: author (wrong), other (wrong), everybody, nobody, info. Each anecdote maintains tallied counts $Y_{ij}$ per label $j$, yielding a label distribution $p_{ij} = Y_{ij}/N_i$. Lexical analysis reveals high diversity (13.5M tokens, 64,476 types).
• Label Distribution: In the dev split, the annotation density is a median of 8 comments per anecdote (total: 52,433 annotations on 2,500 anecdotes), with frequencies: other (54.4%), author (29.8%), nobody (8.9%), everybody (4.8%), info (2.1%).

Explicit Value Quandaries (DailyDilemmas)

• Design: DailyDilemmas contains 1,360 binary-choice dilemmas stratified across 17 everyday topics (80 dilemmas per topic) (Chiu et al., 2024). Each instance is independent, featuring a scenario setup, two explicit actions $\mathcal A^{do}$ (to do) and $\mathcal A^{not}$, negative-consequence stories for each action, involved parties, and a detailed perspectives annotation.
• Value Extraction: For every action, each affected party is assigned at least one (value, reason) pair through chain-of-thought reasoning, followed by aggregation into sets $\{v^{do}\}$ and $\{v^{not}\}$—capturing the core value conflict of the dilemma.

2. Human Values Inventory and Operationalization

Central to DailyDilemmas is its comprehensive and systematic cataloging of human values as they are manifested in ordinary ethical choices.

• Value Set: 301 unique values with at least 100 instances each were identified using a GPT-4–driven induction process, and subsequently filtered and validated by human annotators. The top values by frequency include: trust, self, honesty, responsibility, respect, empathy, understanding, fairness, integrity, accountability (see Table 1).
• Operational Definition: Each value is operationalized in the annotation protocol as a tuple: (party, value phrase, one-sentence justification). There are no closed-form definitions; rather, justification is contingent on scenario-specific reasoning.
• Validation and Reliability: Human annotator validation on Reddit posts yields F1 = 85.7%, precision = 81.8%, recall = 90.0%, and Cohen’s κ = 0.526. Word-level value match in top Reddit comments is 60.02% (SD 14.2%).

Statistic	Scruples (Lourie et al., 2020)	DailyDilemmas (Chiu et al., 2024)
Scenarios/Instances	32,766 anecdotes	1,360 dilemmas
Label space	5-class distribution	2 actions, values per action
Values catalogued	N/A (focus on majority class)	301 distinct values
Domains/topics	r/AITA diversity	17 stratified topics

3. Theoretical Frameworks for Value Mapping

A distinguishing feature of the DailyDilemmas resource is its explicit mapping of values to five canonical theories of values, facilitating multidimensional audit and comparison.

• World Values Survey (WVS): Two axes—Traditional–Secular-rational and Survival–Self-expression—serve as macro-scale value dimensions.
• Moral Foundations Theory (MFT): Values are binned into Care, Fairness, Loyalty, Authority, Purity.
• Maslow’s Hierarchy of Needs: Values correspond to needs levels—Physiological, Safety, Love/Belonging, Self-esteem, Self-actualization.
• Aristotle’s Virtues: Nine classical virtues, including Truthfulness, Courage, Patience, Liberality, Ambition, and Friendliness.
• Plutchik’s Wheel of Emotions: Values are associated with core emotions: Trust, Joy, Fear, Sadness, Disgust, Anger, Anticipation, Surprise.

Each value was manually assigned to categories using standard definitions. Distribution plots (e.g., Fig. 3) reveal that Self-expression (WVS), Fairness (MFT), Self-esteem (Maslow), Truthfulness (Virtues), and Trust (Emotions) dominate the corpus.

4. Quantitative Statistics, Formulas, and Evaluation

Evaluation of models and values within these corpora requires distribution-sensitive metrics and carefully defined operational formulas.

• Soft Label Vectors: For Scruples, a gold standard prediction is the full distribution vector $p_{i:}$ inferred from crowd votes, not a mode label.
• SCOracle Estimator: Establishes a Bayes-optimal lower bound for cross-entropy by modeling $p_{i:}$ with a Dirichlet prior, fitting hyperparameters $\alpha$ through empirical Bayes, and estimating expected cross-entropy via

$j$0

This estimator’s relative error is less than 1% in simulation (Lourie et al., 2020).

• Modeling Likelihoods: Both BERT-Large and RoBERTa-Large are evaluated using:
  • Standard Softmax/Categorical likelihood,
  • Dirichlet-multinomial output layers—separating intrinsic controversy from model uncertainty.
• DailyDilemmas Evaluation: The core metrics are (i) choice frequency for each action, (ii) value-alignment rate, and (iii) preference-difference per value-theory dimension:

$j$1

where $j$2 and $j$3 are normalized selection frequencies for each side of a theoretical axis.

• Value Preference Delta: For a given dilemma,

$j$4

5. Empirical Findings and Model Performance

Performance analysis highlights both the challenge and potential of distributional and value-theoretic moral judgment modeling.

Scruples Findings

• Full Corpus: RoBERTa with Dirichlet-multinomial achieves F1-macro = 0.302/0.259 (dev/test), cross-entropy ≈ 1.03, versus a human majority-vote oracle at F1 ≤ 0.49, CE ≈ 0.742. Simplified variants yield higher alignment (F1-macro ≈ 0.78) (Lourie et al., 2020).
• Ambiguity: Numerous real dilemmas are fundamentally divisive; single-label evaluation over-penalizes models on these.
• Modeling Improvements: Dirichlet-multinomial likelihood outperforms soft labels/counts-based alternatives for cross-entropy, reifying both intrinsic and epistemic uncertainty.

DailyDilemmas (LLM Value Alignment)

• Model Choices: All models (GPT-4-turbo, GPT-3.5-turbo, Llama-2/3, Mixtral-8x7B, Claude-Haiku) favor Self-expression over Survival (WVS), and Care over Loyalty (MFT) (Chiu et al., 2024).
• Model Differences: Notable divergence in value selection (e.g., Truthfulness: GPT-4-turbo +9.4%, Mixtral-8x7B –9.7% relative selection); safety values are universally de-prioritized; ambition, friendliness, and courage show model-specific patterns.
• Steerability: Prompt-based steering via principles from OpenAI ModelSpec or Anthropic Constitutional AI has only weak effect on GPT-4-turbo’s value priorities.
• Human Validation: Semantic value annotation in DailyDilemmas exhibits robust correspondence (F1 = 85.7%, κ = 0.526) with real Reddit community responses.

6. Methodological Implications and Applications

The DailyDilemmas corpus—across both Scruples and value-theory instantiations—introduces key methodological recommendations.

• Distributional Ground Truth: Gold labels as empirical distributions are essential for ambiguous dilemmas; single-label metrics obscure true ambiguity.
• Dirichlet-multinomial Output Layers: These are recommended to jointly model community disagreement and model confidence, improving calibration and interpretability.
• SCOracle as Empirical Upper Bound: Adopting the SCOracle estimator is critical for realistic upper-bound benchmarking under label dissent.
• Value Operationalization: Comprehensive value extraction (301 values) and rigorous mapping to value theories enable fine-grained, theory-driven audit and alignment of LLMs.
• Public Resources: All data, code, and prompts are available at HF Datasets and GitHub.

Potential applications include value-alignment audits of emerging LLMs, targeted RLHF or instruction-tuning along under-represented values, value-aware decision-support tooling, and cross-cultural or theory-based analyses of normative reasoning.

• "Scruples: A Corpus of Community Ethical Judgments on 32,000 Real-Life Anecdotes" (Lourie et al., 2020)
• "DailyDilemmas: Revealing Value Preferences of LLMs with Quandaries of Daily Life" (Chiu et al., 2024)