---
title: 'IndieValueCatalog: Individualistic Value Dataset'
url: https://www.emergentmind.com/topics/indievaluecatalog
type: topic
---

# IndieValueCatalog: Individualistic Value Dataset

IndieValueCatalog is a dataset derived from the World Values Survey (WVS) for studying **individualistic value reasoning**: the task of inferring a specific person’s value system from that person’s prior value-expressing statements and then predicting how the same person will answer novel value questions. Its central motivation is a critique of pluralistic alignment methods that rely on predefined groups such as demographics, cultures, or personas. Instead of treating diversity through fixed buckets, IndieValueCatalog frames values as a person-specific latent structure and evaluates whether language models can reconstruct and use that structure at the individual level [2410.03868].

## 1. Conceptual basis: individualistic alignment

IndieValueCatalog is designed around the distinction between **group-based** and **individualistic** alignment. In the group-based setting, systems commonly sort people into categories such as country, gender, age, class, or personality type, and then model value distributions at that level. The paper argues that this approach is useful but coarse, because people within the same group can differ substantially, group labels can induce stereotyping, and group membership can obscure within-group variation [2410.03868].

The alternative proposed is **individualistic alignment**. In this formulation, the objective is not to predict what a demographic group tends to believe, but to infer what a particular person believes from that person’s own statements. IndieValueCatalog operationalizes this idea by asking whether a model can condition on known statements from one individual and correctly choose that individual’s response in unseen cases. This shifts the unit of analysis from population aggregates to person-specific value profiles.

A common misconception is that individualistic alignment is simply demographic personalization with finer granularity. The reported results argue against that interpretation. The paper finds that a precise description of individualistic values cannot be approximated only with demographic information, and that demographic-only inference risks stereotypical group-based interpretation rather than genuine person-level reasoning [2410.03868].

## 2. Construction from the World Values Survey

The dataset is transformed from **World Values Survey Round Seven**. The source survey contains heterogeneous response formats, including multiple-choice items, Likert scales, coded responses, and demographic questions. IndieValueCatalog converts these into standardized natural-language statements, so that survey responses become textual propositions suitable for LM prompting and supervised training. One example given is the WVS question “Could you tell me how secure you feel these days?”, which is converted into statements such as “I feel very secure these days.” and “I feel not at all secure these days.” [2410.03868].

The paper defines two statement granularities. The **polar** version is coarser and often collapses response options by polarity. The **refined** version is more fine-grained and preserves more response distinctions.

| Quantity | Value |
|---|---:|
| Original questions | 253 |
| Polar statements | 567 |
| Refined statements | 929 |
| People | 93,279 |
| Valid question-answer instances | 22.6M |
| Average valid questions per person | 242.03 |
| Standard deviation | 17.31 |
| People with full question set | 15,819 |

The dataset also includes **31 demographic questions** converted into identity-declaring natural-language statements, such as “I am currently in Andorra.” and “I am a citizen of this country.” These demographic statements can be provided optionally as model input [2410.03868].

The scale of the answer space is emphasized as a substantive property of the dataset rather than a decorative statistic. The paper reports an approximate refined answer space of **$1.65 \times 10^{139}$** and a polar answer space of **$3.94 \times 10^{86}$**. This suggests that the benchmark should not be interpreted as a small-label classification problem; it is instead a large structured prediction problem over many interdependent value judgments.

## 3. Task formulation and benchmark protocol

The benchmark asks: given a subset of a person’s known value statements, can a model infer the person’s underlying values well enough to predict responses to unseen questions? Operationally, each person’s responses are partitioned into **demonstration statements** and **probing statements**. The number of demonstration statements ranges from **50 to 200**, while the probing set contains **39 held-out statements** drawn across **13 WVS categories** [2410.03868].

At inference time, the model receives three types of information: a list of known value statements from the same person, optionally that person’s demographic statements, and for each unseen question a set of candidate statements corresponding to possible answers. The model must select the candidate statement most likely to match the person’s values. The appendix prompt format includes a rationale field and a choice field, but the underlying decision problem is multiple-choice selection conditioned on person-specific prior evidence.

The benchmark uses **three probing splits**. Each split has **200 demonstration questions** and **39 probing questions**, and results are averaged across the three splits to avoid overfitting to one probing partition. The evaluation subset contains **800 held-out individuals**, selected to preserve balanced demographic representation [2410.03868].

The probe questions span 13 categories: Social Values / Attitudes / Stereotypes; Happiness and Well-Being; Social Capital / Trust / Organizational Membership; Economic Values; Corruption; Migration; Security; Postmaterialist Index; Science / Technology; Religious Values; Ethical Values and Norms; Political Interest / Participation; and Political Culture / Regimes. Consequently, the benchmark covers personal beliefs, moral judgments, trust attitudes, religious practice, political and economic orientations, and subjective well-being in a unified format.

## 4. Empirical performance of frontier language models

The main empirical result is that frontier LMs achieve only **about 55% to 65% accuracy** on this task under zero-shot or demonstration-based evaluation, indicating that individualistic value reasoning remains difficult [2410.03868].

The paper reports the following overall **polar-setting accuracies**:

| Model | Accuracy |
|---|---:|
| Claude-3.5 Sonnet | 64.66 |
| GPT-4-turbo (0409) | 63.74 |
| GPT-4o (0513) | 63.71 |
| Llama-3.1-70B | 63.71 |
| Qwen2-72B | 63.61 |
| GPT-4o (0806) | 63.46 |
| Mixtral-8x22B | 61.16 |
| GPT-4o-mini (0718) | 60.80 |
| Llama-3.1-8B | 58.20 |
| Mixtral-8x7B | 56.26 |

Performance also varies across the three probing splits. For example, GPT-4o (0806) obtains **65.21 / 64.77 / 60.39**, GPT-4-turbo (0409) obtains **65.08 / 65.73 / 60.41**, Claude-3.5 obtains **65.74 / 66.48 / 61.76**, and Llama-3.1-8B obtains **58.72 / 62.09 / 53.80**. Probe 3 is generally harder than probes 1 and 2.

The benchmark includes **Random** and **GPT-4o Rand** baselines. All models substantially outperform Random, and GPT-4o without demonstrations performs better than Random, suggesting some default preference structure. However, demonstrations materially matter: the text reports **63.5 vs. 54.8** for GPT-4o with and without demonstrations. This indicates that person-specific evidence improves performance, but does not resolve the task.

The paper also compares **polar** and **refined** statement formulations. One table reports **63.61** average accuracy for refined and **63.46** for polar. The authors further note that refined expressions are more informative and nuanced for reconstructing values. This suggests that representation granularity affects recoverability of individual value structure.

## 5. Demographics, inequity, and partiality

A major finding is that **demographics alone are only marginally informative** for predicting individual value judgments. For GPT-4o, the paper reports **60.31** when using only demographics and **60.59** when using **50 value demonstrations**. Adding demographics to value statements yields only marginal improvement and is not statistically significant for GPT-4o [2410.03868].

This result is conceptually important because it separates **identity descriptors** from **value evidence**. With sufficiently many demonstrations, value statements are more informative than demographics, while heavy reliance on demographic attributes risks reinforcing stereotypes. The paper notes that weaker models such as GPT-4o-mini benefit more from demographics because they struggle more with nuanced value statements. A plausible implication is that demographic cues function partly as a shortcut when value inference capacity is limited.

To quantify uneven performance across populations, the paper introduces the **Value Inequity Index**, abbreviated **Inequity**:

$$
\text{Inequity}_M = \frac{1}{|\mathbb{D}|} \sum_{\mathcal{D}^k \in \mathbb{D}} \sigma\left(\{Acc_M^{g_{k_t}} \mid \forall g_{k_t}\in \mathcal{D}^k\}\right)
$$

Here, $\mathbb{D}$ is the set of **13 demographic dimensions**, $\mathcal{D}^k$ is one demographic dimension, $g_{k_t}$ are the groups within that dimension, $Acc_M^{g_{k_t}}$ is model accuracy on a given group, and $\sigma$ is the standard deviation. Lower values indicate more even performance across groups; higher values indicate greater partiality.

Reported Inequity values include **1.94** for Llama-3.1-70B, **2.55** for GPT-4o-mini, **2.83** for GPT-4-turbo (0409), **2.87** for GPT-4o (0513), **2.97** for Llama-3.1-8B, **3.03** for GPT-4o (0806), **3.06** for Mixtral-8x22B, **3.14** for Claude-3.5 Sonnet, **3.19** for Mixtral-8x7B, and **3.24** for Qwen2-72B. The paper highlights that GPT-4o (0513) and Llama-3.1-70B both have accuracy **63.7**, yet GPT-4o (0513) has higher Inequity (**2.87**) than Llama-3.1-70B (**1.94**). Average accuracy therefore does not determine parity across demographic groups.

The paper also reports concrete partiality patterns. For example, Llama-3.1-8B is most accurate for people from **Oceania**, **high income**, and **upper-middle class** groups. This is presented as evidence that LM reasoning about global individualistic values is demographically uneven.

## 6. IndieValueReasoner and broader significance

The paper introduces **IndieValueReasoner**, a family of models obtained by fine-tuning **meta-llama/Meta-Llama-3.1-8B-Instruct** on IndieValueCatalog. The stated goal is not merely to improve the benchmark score, but also to analyze what patterns of human values can be learned from the data and to estimate a data-driven upper limit for the task [2410.03868].

Training uses **Open-Instruct** on a single node with **8 NVIDIA H100 80GB GPUs**, for **2 epochs**, with learning rate **5e-6**, maximum sequence length **4096**, and batch size **8**. Each training instance contains demonstration statements from one person, a probing question with candidate statements, and the correct choice. Variants differ in demonstration count, statement granularity, presence of demographics, and whether demonstration counts are fixed or mixed.

The best reported model is **[probe=p+r, demo=mixed+200, N=1600]**, with **67.67** accuracy and **2.22** Inequity. This outperforms zero-shot Llama-3.1-8B and also surpasses the **Resemble (top cluster)** baseline, reported at **66.60**. The paper identifies several training regularities: mixed demonstration counts from **50–200** improve generalization more than a fixed number; adding both mixed and full **200-demo** cases improves further; and exposure to both **polar** and **refined** formats yields the best overall robustness.

The regional experiments provide additional structure. Continent-specific models often perform best on their own continent, and some cross-continent transfer is strong, often aligning with geographical or cultural similarity. **Oceania** is identified as an exception because its data comes entirely from **New Zealand**, and the authors hypothesize that the resulting homogeneity causes the continent-specific model to behave anomalously. At the same time, the globally trained model performs comparably or better than regional models on many test sets, suggesting that diverse worldwide training data is valuable for robust individualistic value reasoning.

In aggregate, IndieValueCatalog establishes that person-specific value prediction is feasible but far from solved; that demographics do not substitute for direct evidence of values; and that average benchmark accuracy can conceal substantial demographic partiality. Its significance lies in reframing pluralistic alignment away from bucketed proxies and toward inference over individual value profiles, while providing a concrete evaluation framework for measuring both predictive performance and inequity [2410.03868].

Source: https://www.emergentmind.com/topics/indievaluecatalog