IndieValueCatalog: Individualistic Value Dataset

• IndieValueCatalog is a dataset derived from the World Values Survey that studies individualistic value reasoning by inferring a person’s value system from their statements.
• It standardizes diverse survey responses into natural-language statements, enabling large-scale structured predictions over nuanced value judgments.
• Empirical findings reveal that frontier language models achieve 55%-65% accuracy, highlighting the challenge of person-specific value inference and demographic inequity.

IndieValueCatalog is a dataset derived from the World Values Survey (WVS) for studying individualistic value reasoning: the task of inferring a specific person’s value system from that person’s prior value-expressing statements and then predicting how the same person will answer novel value questions. Its central motivation is a critique of pluralistic alignment methods that rely on predefined groups such as demographics, cultures, or personas. Instead of treating diversity through fixed buckets, IndieValueCatalog frames values as a person-specific latent structure and evaluates whether LLMs can reconstruct and use that structure at the individual level (Jiang et al., 2024).

1. Conceptual basis: individualistic alignment

IndieValueCatalog is designed around the distinction between group-based and individualistic alignment. In the group-based setting, systems commonly sort people into categories such as country, gender, age, class, or personality type, and then model value distributions at that level. The paper argues that this approach is useful but coarse, because people within the same group can differ substantially, group labels can induce stereotyping, and group membership can obscure within-group variation (Jiang et al., 2024).

The alternative proposed is individualistic alignment. In this formulation, the objective is not to predict what a demographic group tends to believe, but to infer what a particular person believes from that person’s own statements. IndieValueCatalog operationalizes this idea by asking whether a model can condition on known statements from one individual and correctly choose that individual’s response in unseen cases. This shifts the unit of analysis from population aggregates to person-specific value profiles.

A common misconception is that individualistic alignment is simply demographic personalization with finer granularity. The reported results argue against that interpretation. The paper finds that a precise description of individualistic values cannot be approximated only with demographic information, and that demographic-only inference risks stereotypical group-based interpretation rather than genuine person-level reasoning (Jiang et al., 2024).

2. Construction from the World Values Survey

The dataset is transformed from World Values Survey Round Seven. The source survey contains heterogeneous response formats, including multiple-choice items, Likert scales, coded responses, and demographic questions. IndieValueCatalog converts these into standardized natural-language statements, so that survey responses become textual propositions suitable for LM prompting and supervised training. One example given is the WVS question “Could you tell me how secure you feel these days?”, which is converted into statements such as “I feel very secure these days.” and “I feel not at all secure these days.” (Jiang et al., 2024).

The paper defines two statement granularities. The polar version is coarser and often collapses response options by polarity. The refined version is more fine-grained and preserves more response distinctions.

Quantity	Value
Original questions	253
Polar statements	567
Refined statements	929
People	93,279
Valid question-answer instances	22.6M
Average valid questions per person	242.03
Standard deviation	17.31
People with full question set	15,819

The dataset also includes 31 demographic questions converted into identity-declaring natural-language statements, such as “I am currently in Andorra.” and “I am a citizen of this country.” These demographic statements can be provided optionally as model input (Jiang et al., 2024).

The scale of the answer space is emphasized as a substantive property of the dataset rather than a decorative statistic. The paper reports an approximate refined answer space of $1.65 \times 10^{139}$ and a polar answer space of $3.94 \times 10^{86}$. This suggests that the benchmark should not be interpreted as a small-label classification problem; it is instead a large structured prediction problem over many interdependent value judgments.

3. Task formulation and benchmark protocol

The benchmark asks: given a subset of a person’s known value statements, can a model infer the person’s underlying values well enough to predict responses to unseen questions? Operationally, each person’s responses are partitioned into demonstration statements and probing statements. The number of demonstration statements ranges from 50 to 200, while the probing set contains 39 held-out statements drawn across 13 WVS categories (Jiang et al., 2024).

At inference time, the model receives three types of information: a list of known value statements from the same person, optionally that person’s demographic statements, and for each unseen question a set of candidate statements corresponding to possible answers. The model must select the candidate statement most likely to match the person’s values. The appendix prompt format includes a rationale field and a choice field, but the underlying decision problem is multiple-choice selection conditioned on person-specific prior evidence.

The benchmark uses three probing splits. Each split has 200 demonstration questions and 39 probing questions, and results are averaged across the three splits to avoid overfitting to one probing partition. The evaluation subset contains 800 held-out individuals, selected to preserve balanced demographic representation (Jiang et al., 2024).

The probe questions span 13 categories: Social Values / Attitudes / Stereotypes; Happiness and Well-Being; Social Capital / Trust / Organizational Membership; Economic Values; Corruption; Migration; Security; Postmaterialist Index; Science / Technology; Religious Values; Ethical Values and Norms; Political Interest / Participation; and Political Culture / Regimes. Consequently, the benchmark covers personal beliefs, moral judgments, trust attitudes, religious practice, political and economic orientations, and subjective well-being in a unified format.

4. Empirical performance of frontier LLMs

The main empirical result is that frontier LMs achieve only about 55% to 65% accuracy on this task under zero-shot or demonstration-based evaluation, indicating that individualistic value reasoning remains difficult (Jiang et al., 2024).

The paper reports the following overall polar-setting accuracies:

Model	Accuracy
Claude-3.5 Sonnet	64.66
GPT-4-turbo (0409)	63.74
GPT-4o (0513)	63.71
Llama-3.1-70B	63.71
Qwen2-72B	63.61
GPT-4o (0806)	63.46
Mixtral-8x22B	61.16
GPT-4o-mini (0718)	60.80
Llama-3.1-8B	58.20
Mixtral-8x7B	56.26

Performance also varies across the three probing splits. For example, GPT-4o (0806) obtains 65.21 / 64.77 / 60.39, GPT-4-turbo (0409) obtains 65.08 / 65.73 / 60.41, Claude-3.5 obtains 65.74 / 66.48 / 61.76, and Llama-3.1-8B obtains 58.72 / 62.09 / 53.80. Probe 3 is generally harder than probes 1 and 2.

The benchmark includes Random and GPT-4o Rand baselines. All models substantially outperform Random, and GPT-4o without demonstrations performs better than Random, suggesting some default preference structure. However, demonstrations materially matter: the text reports 63.5 vs. 54.8 for GPT-4o with and without demonstrations. This indicates that person-specific evidence improves performance, but does not resolve the task.

The paper also compares polar and refined statement formulations. One table reports 63.61 average accuracy for refined and 63.46 for polar. The authors further note that refined expressions are more informative and nuanced for reconstructing values. This suggests that representation granularity affects recoverability of individual value structure.

5. Demographics, inequity, and partiality

A major finding is that demographics alone are only marginally informative for predicting individual value judgments. For GPT-4o, the paper reports 60.31 when using only demographics and 60.59 when using 50 value demonstrations. Adding demographics to value statements yields only marginal improvement and is not statistically significant for GPT-4o (Jiang et al., 2024).

This result is conceptually important because it separates identity descriptors from value evidence. With sufficiently many demonstrations, value statements are more informative than demographics, while heavy reliance on demographic attributes risks reinforcing stereotypes. The paper notes that weaker models such as GPT-4o-mini benefit more from demographics because they struggle more with nuanced value statements. A plausible implication is that demographic cues function partly as a shortcut when value inference capacity is limited.

To quantify uneven performance across populations, the paper introduces the Value Inequity Index, abbreviated Inequity:

$$\text{Inequity}_M = \frac{1}{|\mathbb{D}|} \sum_{\mathcal{D}^k \in \mathbb{D}} \sigma\left(\{Acc_M^{g_{k_t}} \mid \forall g_{k_t}\in \mathcal{D}^k\}\right)$$

Here, $\mathbb{D}$ is the set of 13 demographic dimensions, $\mathcal{D}^k$ is one demographic dimension, $g_{k_t}$ are the groups within that dimension, $Acc_M^{g_{k_t}}$ is model accuracy on a given group, and $\sigma$ is the standard deviation. Lower values indicate more even performance across groups; higher values indicate greater partiality.

Reported Inequity values include 1.94 for Llama-3.1-70B, 2.55 for GPT-4o-mini, 2.83 for GPT-4-turbo (0409), 2.87 for GPT-4o (0513), 2.97 for Llama-3.1-8B, 3.03 for GPT-4o (0806), 3.06 for Mixtral-8x22B, 3.14 for Claude-3.5 Sonnet, 3.19 for Mixtral-8x7B, and 3.24 for Qwen2-72B. The paper highlights that GPT-4o (0513) and Llama-3.1-70B both have accuracy 63.7, yet GPT-4o (0513) has higher Inequity (2.87) than Llama-3.1-70B (1.94). Average accuracy therefore does not determine parity across demographic groups.

The paper also reports concrete partiality patterns. For example, Llama-3.1-8B is most accurate for people from Oceania, high income, and upper-middle class groups. This is presented as evidence that LM reasoning about global individualistic values is demographically uneven.

6. IndieValueReasoner and broader significance

The paper introduces IndieValueReasoner, a family of models obtained by fine-tuning meta-llama/Meta-Llama-3.1-8B-Instruct on IndieValueCatalog. The stated goal is not merely to improve the benchmark score, but also to analyze what patterns of human values can be learned from the data and to estimate a data-driven upper limit for the task (Jiang et al., 2024).

Training uses Open-Instruct on a single node with 8 NVIDIA H100 80GB GPUs, for 2 epochs, with learning rate 5e-6, maximum sequence length 4096, and batch size 8. Each training instance contains demonstration statements from one person, a probing question with candidate statements, and the correct choice. Variants differ in demonstration count, statement granularity, presence of demographics, and whether demonstration counts are fixed or mixed.

The best reported model is [probe=p+r, demo=mixed+200, N=1600], with 67.67 accuracy and 2.22 Inequity. This outperforms zero-shot Llama-3.1-8B and also surpasses the Resemble (top cluster) baseline, reported at 66.60. The paper identifies several training regularities: mixed demonstration counts from 50–200 improve generalization more than a fixed number; adding both mixed and full 200-demo cases improves further; and exposure to both polar and refined formats yields the best overall robustness.

The regional experiments provide additional structure. Continent-specific models often perform best on their own continent, and some cross-continent transfer is strong, often aligning with geographical or cultural similarity. Oceania is identified as an exception because its data comes entirely from New Zealand, and the authors hypothesize that the resulting homogeneity causes the continent-specific model to behave anomalously. At the same time, the globally trained model performs comparably or better than regional models on many test sets, suggesting that diverse worldwide training data is valuable for robust individualistic value reasoning.

In aggregate, IndieValueCatalog establishes that person-specific value prediction is feasible but far from solved; that demographics do not substitute for direct evidence of values; and that average benchmark accuracy can conceal substantial demographic partiality. Its significance lies in reframing pluralistic alignment away from bucketed proxies and toward inference over individual value profiles, while providing a concrete evaluation framework for measuring both predictive performance and inequity (Jiang et al., 2024).