TRAIT Dataset: LLM Personality Assessment

• TRAIT is a domain-specific, psychometrically validated benchmark that employs an 8,000-item multiple-choice framework to evaluate LLM personality traits.
• The methodology integrates classical psychometrics with realistic scenarios from the ATOMIC-10X knowledge graph, ensuring domain relevance and behavioral grounding.
• Experimental insights reveal that LLMs exhibit distinct and stable personality profiles, with alignment tuning significantly influencing traits like Agreeableness and Conscientiousness.

The TRAIT dataset, as introduced in distinct recent research lines, constitutes a set of domain-specific, psychometrically validated benchmarks designed to evaluate and analyze personality traits—particularly in LLMs—under controlled, yet behaviorally grounded, conditions. In its most prominent incarnation, the TRAIT dataset (Lee et al., 20 Jun 2024) is an 8,000-item, multi-choice test constructed for automated, high-fidelity assessment of LLM personality, extending classical psychometrics into automated agent evaluation. This resource builds upon and systematically enhances human personality assessment instruments, offering a robust experimental framework for probing personality expression, stability, and alignment mechanics in LLMs.

1. Construction and Structure of the TRAIT Dataset

TRAIT is constructed as an 8,000-item benchmark, primarily composed of multi-choice questions crafted for the systematic evaluation of LLMs’ “personality” through scenario-based behavioral analogues. Its foundation rests on two established psychological questionnaires: the 44-item Big Five Inventory (BFI) and the 27-item Short Dark Triad (SD-3). These seed items are algorithmically and semi-manually expanded 112-fold to comprehensively span eight personality traits: the canonical five BIG-5 facets (Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism) alongside Machiavellianism, Narcissism, and Psychopathy.

Each TRAIT test item incorporates:

• A real-world scenario generated by sampling and filtering from the ATOMIC-10X @@@@1@@@@, designed to contextualize trait expression in realistic, commonsense environments.
• A four-way multiple-choice question, with two options mapped to “high” and two to “low” trait presence, facilitating discrete signal extraction.

This approach enables not only traditional score assignment but also behavioral and situational generalization, essential for evaluating non-human conversational agents.

2. Methodological Foundations and Content Enrichment

To construct situational realism and ensure coverage, item content for each trait is enriched with domain knowledge from the ATOMIC-10X knowledge graph. For every initial trait item, 20 varied situational prompts are sampled; five are then selected with model (e.g., GPT-4)-in-the-loop ranking to maximize domain relevance. The result is an unprecedented variety of contextualizations that dramatically expand the effective test space compared to rigid self-report questions, allowing scenarios to probe deeper dimensions of trait expression in LLMs.

Answer key assignment is carefully balanced: every scenario is explicitly grounded in behavioral tendencies implicating high or low trait levels, and the mapping of answer options is controlled to reduce response pattern artifacts.

3. Psychometric Validation and Robustness Metrics

TRAIT is subjected to rigorous validation on key axes:

• Content/Construct Validity: Ensured through item generation protocols and expert review that preserve trait specificity and language-model-applicability.
• Internal Consistency and Reliability: Quantified via three primary sensitivity metrics across repeated testings:
  • Prompt Sensitivity: Stability of responses across three distinct prompt templates; low variance indicates robustness.
  • Option-Order Sensitivity: Agreement between standard choice ordering and random or reversed choice orders.
  • Paraphrase Sensitivity: Consistency when query wordings are systematically rephrased.

Mathematically, e.g., prompt sensitivity is computed as

$1 - \frac{1}{n}\sum_i s_i$

where $s_i = 1$ if all responses to an item across prompt templates agree, 0 otherwise, aggregated over all $n$ items.

A critical statistical quality benchmark is the “refusal rate,” the fraction of items for which the LLM does not render a meaningful personality-based decision. For TRAIT, this rate is approximately 0.2%, a substantial improvement over earlier benchmarks.

4. Experimental Insights into LLM Personality

Application of TRAIT to major LLMs (e.g., GPT-4, GPT-3.5, Llama variants, Mistral) reveals two high-level findings:

• Distinct and Consistent Personality Profiles: Contemporary LLMs, when benchmarked on TRAIT, exhibit statistically significant, stable trait score patterns, sensitive to alignment phases and underlying data. For example, alignment-tuned models (such as GPT-4) consistently yield higher Agreeableness and Conscientiousness, and lower Dark Triad metrics, compared to their pre-alignment counterparts.
• Exposure of Data Leverage and Prompting Limits: Model trait scores shift predictably under experimental manipulation of training set trait distributions. However, some traits (notably high psychopathy, low conscientiousness) remain relatively intransigent to both direct prompting and known alignment techniques, reflecting strong inductive biases instilled via training and reinforcement learning from human feedback (RLHF)/direct preference optimization (DPO).

A “Trait Balance Score” is further introduced to quantify the alignment correlation: $$\text{Trait Balance Score} = \frac{\#\text{High Trait Examples}}{\#\text{Low Trait Examples}}$$ in training, correlating with model persona shifts.

5. Coverage, Limitations, and Directions for Further Research

TRAIT’s breadth positions it as the de facto benchmark for LLM personality measurement. Nonetheless, documented limitations include:

• Prompting Saturation and Control: Only a subset of target traits can be shifted via simple prompt engineering; more recalcitrant traits resist control without altering model weights or training corpus composition.
• Cultural and Model Bias: Given ATOMIC-10X and LLM contributors draw heavily from “Global North” samples and English-language contexts, the dataset’s cultural neutrality is partial.
• Assessment Scope: TRAIT focuses on single-turn, scenario-based judgments. More subtle aspects, such as multi-turn behavior or group trait expression, remain largely untested within this framework.

Identified trajectories for extension include multi-step (conversational or behavioral chain) assessments, integration of simulated social interaction tests, and deeper analysis of non-Western/cross-cultural scenario generalization.

6. Practical Utility and Implications for LLM Development

TRAIT enables the quantifiable, comparative, and reproducible paper of “personality” as a behavioral signature in large, parameterized LLMs. Implications for model development and alignment include:

• Safety and Alignment Tuning: By characterizing which traits are modifiable and which are resistant, model designers can focus alignment efforts on trait axes most relevant to risk or harm.
• Benchmarking and Transparency: TRAIT’s sensitivity and reliability metrics set a new standard for “psychometric” evaluation of non-human agents, aligning model evaluation with established behavioral sciences.
• Transfer to Multi-Domain Measurement: The scenario expansion process supports adaptation to other latent trait assessments (e.g., moral reasoning, emotional intelligence), especially where scenario-based proxies are required for non-human agents.

7. Dataset Availability and Research Adoption

TRAIT and associated materials are openly released for academic and commercial research, with code and full item content available. Adoption is encouraged across:

• LLM evaluation and alignment communities;
• Computational social scientists studying emergent behavior or LLM anthropomorphism;
• Psychometric modeling researchers interested in automated assessment beyond human subjects.

By integrating and extending established psychometric theory, commonsense knowledge grounding, and empirical evaluation, TRAIT represents the current state-of-the-art in automatic agent personality assessment, systematically revealing both the expressivity and the inherent limits of contemporary LLM “personas” (Lee et al., 20 Jun 2024).