Do GPT Language Models Suffer From Split Personality Disorder? The Advent Of Substrate-Free Psychometrics (2408.07377v2)

Abstract: Previous research on emergence in LLMs shows these display apparent human-like abilities and psychological latent traits. However, results are partly contradicting in expression and magnitude of these latent traits, yet agree on the worrisome tendencies to score high on the Dark Triad of narcissism, psychopathy, and Machiavellianism, which, together with a track record of derailments, demands more rigorous research on safety of these models. We provided a state of the art LLM with the same personality questionnaire in nine languages, and performed Bayesian analysis of Gaussian Mixture Model, finding evidence for a deeper-rooted issue. Our results suggest both interlingual and intralingual instabilities, which indicate that current LLMs do not develop a consistent core personality. This can lead to unsafe behaviour of artificial intelligence systems that are based on these foundation models, and are increasingly integrated in human life. We subsequently discuss the shortcomings of modern psychometrics, abstract it, and provide a framework for its species-neutral, substrate-free formulation.

• The paper uses the TIPI across nine languages to reveal that GPT models display non-uniform, multi-modal personality traits rather than a coherent personality.
• It employs Bayesian Gaussian Mixture Models and kernel density estimation on 695 data points to capture variations in personality trait distributions.
• The findings imply that human-centric psychometric tools may be inadequate for AI, underscoring the need for new substrate-free evaluation frameworks for safe deployment.

Assessing GPT LLMs Through Psychometric Evaluation

The paper "Do GPT LLMs Suffer From Split Personality Disorder? The Advent Of Substrate-Free Psychometrics" (2408.07377) examines the expression of personality traits in GPT LLMs using a psychometric approach. By deploying personality questionnaires across multiple languages, the paper investigates whether GPT models exhibit stable personality traits, which has implications for the safety and reliability of these AI systems in real-world applications.

Language-Spanning Personality Assessment

The paper implements the Ten Item Personality Inventory (TIPI) across nine languages to rate the GPT model's expression of the Big Five personality traits: Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism. Statistical analyses, including Bayesian Gaussian Mixture Models, are employed to explore the consistency of these traits across languages. Results indicate significant interlingual and intralingual variations, with distribution patterns suggesting potential multi-modal configurations.

Data and Analysis Techniques

• Sample Collection: The paper collected 695 data points from prompts given in Bulgarian, Catalan, Chinese, German, English, French, Japanese, Korean, and Spanish.
• Kernel Density Estimation: Examined the distribution of personality traits, noting variations in skewness, kurtosis, and modality, which implies non-normality.
• Bayesian Gaussian Mixture Models: Detected that only about 20% of latent trait models were best fit by a singular Gaussian distribution, suggesting the presence of multiple underlying personality components within the model.

Interpretation of Results

The findings reveal that GPT models do not develop a coherent and consistent personality akin to human norms but instead exhibit significant variability across different linguistic inputs. The suggestion of multi-modal distributions indicates that elicited traits may reflect multiple latent sub-personalities rather than a singular cohesive identity schema. The instability of these traits across languages could potentially lead to unpredictable and unsafe AI behavior, particularly as GPT systems become more integrated into decision-making contexts that affect human life.

Implications for AI Psychometrics

• Species-Neutral Psychometrics: The paper proposes a framework for substrate-free, species-neutral psychometric evaluation, extending traditional approaches to better accommodate AI applications.
• Training Data Considerations: The variability in personality traits is potentially influenced by biases in training data distributions and the predominance of certain cultures, notably US-centric values.

Methodological Approaches and Limitations

The method depends heavily on pre-existing psychometric instruments adapted for AI, raising questions about construct validity and the possible emergence of non-human trait dimensions. This work challenges the applicability of human-centric testing tools such as the TIPI when applied to non-human agents and highlights the need for the development of AI-specific psychometric instruments.

Conclusion

The paper presents strong evidence suggesting GPT LLMs do not possess a consistent personality across languages, but rather, present multiple personality-like expressions. This work outlines important considerations for the deployment of AI models in sensitive applications, advocating for the evolution of new psychometric frameworks and methods suited to AI's unique characteristics. The integration of AI systems into decision-making domains underscores the urgency of these developments to ensure safe and reliable AI-human interaction.