Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
60 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
8 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Identifying Multiple Personalities in Large Language Models with External Evaluation (2402.14805v1)

Published 22 Feb 2024 in cs.CL and cs.AI
Identifying Multiple Personalities in Large Language Models with External Evaluation

Abstract: As LLMs are integrated with human daily applications rapidly, many societal and ethical concerns are raised regarding the behavior of LLMs. One of the ways to comprehend LLMs' behavior is to analyze their personalities. Many recent studies quantify LLMs' personalities using self-assessment tests that are created for humans. Yet many critiques question the applicability and reliability of these self-assessment tests when applied to LLMs. In this paper, we investigate LLM personalities using an alternate personality measurement method, which we refer to as the external evaluation method, where instead of prompting LLMs with multiple-choice questions in the Likert scale, we evaluate LLMs' personalities by analyzing their responses toward open-ended situational questions using an external machine learning model. We first fine-tuned a Llama2-7B model as the MBTI personality predictor that outperforms the state-of-the-art models as the tool to analyze LLMs' responses. Then, we prompt the LLMs with situational questions and ask them to generate Twitter posts and comments, respectively, in order to assess their personalities when playing two different roles. Using the external personality evaluation method, we identify that the obtained personality types for LLMs are significantly different when generating posts versus comments, whereas humans show a consistent personality profile in these two different situations. This shows that LLMs can exhibit different personalities based on different scenarios, thus highlighting a fundamental difference between personality in LLMs and humans. With our work, we call for a re-evaluation of personality definition and measurement in LLMs.

PDF HTML Abstract

External Evaluation of Personality in LLMs Reveals Role-Dependent Variability

Introduction

The increasing ubiquity of LLMs in societal applications has opened up discussions around their reliability, safety, and ethical use. A crucial aspect of understanding LLM behavior lies in the analysis of their "personalities," a concept traditionally reserved for human psychology. Traditionally, the measurement of personality in LLMs has leaned heavily on self-assessment tests, a method critiqued for its reliability and applicability to non-human entities. This paper presents an alternative approach to personality measurement in LLMs through external evaluation, leveraging a fine-tuned Llama2-7B model. The significant finding of this paper is the variance in personalities displayed by LLMs in different roles, challenging the enduring consistency observed in human personality profiles.

Personality Measurement in LLMs

The cornerstone of this research is the development of a state-of-the-art personality prediction model based on the Myers-Briggs Type Indicator (MBTI) personality framework. Utilizing the Llama2-7B model, the researchers achieved significantly superior predictive performance compared to existing models. Therein, two specific configurations were explored: a binary model assessing each MBTI dimension as a separate binary classification problem, and a holistic 16-class model capable of identifying one of the 16 MBTI personality types directly.

The external evaluation process for measuring LLM personality was meticulously designed. LLMs were prompted to generate content akin to Twitter posts and comments based on real-world events and existing tweets, with careful consideration to avoid any pre-training data overlap. The ensuing responses were evaluated using the fine-tuned personality prediction model to discern the LLM's personality.

Findings and Implications

A pivotal revelation of this paper is the distinct personality profiles manifested by LLMs in their disparate roles as post generators versus commenters. This contrasts sharply with the human psychological paradigm where personality is considered a stable characteristic across different contexts. The LLMs, depending on the scenario, diverged significantly in the type of personality presented, challenging the notion of a singular, consistent personality within these models.

Moreover, the paper elegantly validates the external evaluation method's efficacy by applying the same approach to human-written posts and comments, showcasing consistency in personality profiles among humans, thereby affirming the reliability of their model.

The Path Forward

This research opens several avenues for future exploration in the field of AI and psychology. It underscores the necessity for a nuanced understanding and definition of "personality" within LLMs that is distinct from human-centric interpretations. There is a clear call to action for more foundational work on evaluating LLM behavior and personality, taking into account their unique operational and functional paradigms.

Conclusion

In conclusion, the paper "Identifying Multiple Personalities in LLMs with External Evaluation" makes a compelling case for re-evaluating how we understand and measure personality in LLMs. By demonstrating that LLM personalities vary significantly based on the roles they are assigned, it challenges the direct applicability of human personality assessments to AI models and beckons a tailored approach. This work not only enriches the ongoing discourse on AI ethics and safety but also sets a new direction for future research in AI personality assessment methodologies.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. Musiclm: Generating music from text. arXiv preprint arXiv:2301.11325.
  2. American Psychological Association. 1983. Publications Manual. American Psychological Association, Washington, DC.
  3. American Psychological Association. 2023. Definition of Personality - https://www.apa.org/topics/personality.
  4. Personality testing of gpt-3: Limited temporal reliability, but highlighted social desirability of gpt-3’s personality instruments results. arXiv preprint arXiv:2306.04308.
  5. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  6. Graham Caron and Shashank Srivastava. 2022. Identifying and manipulating the personality traits of language models. arXiv preprint arXiv:2212.10276.
  7. Raymond B Cattell. 1943a. The description of personality: Basic traits resolved into clusters. The journal of abnormal and social psychology, 38(4):476.
  8. Raymond B Cattell. 1943b. The description of personality. i. foundations of trait measurement. Psychological review, 50(6):559.
  9. Unleashing the potential of prompt engineering in large language models: a comprehensive review. arXiv preprint arXiv:2310.14735.
  10. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  11. John M Digman. 1990. Personality structure: Emergence of the five-factor model. Annual review of psychology, 41(1):417–440.
  12. Katrina Faust. 2019. Myers-briggs type indicator (mbti) overview - https://leadx.org/articles/mbti-myers-briggs-type-indicator-overview/.
  13. Lewis R Goldberg. 1990. An alternative" description of personality": the big-five factor structure. Journal of personality and social psychology, 59(6):1216.
  14. Lewis R Goldberg. 1993. The structure of phenotypic personality traits. American psychologist, 48(1):26.
  15. Personality consistency and situational influences on behavior. Journal of Management, 45(8):3204–3234.
  16. Investigating the applicability of self-assessment tests for personality measurement of large language models. arXiv preprint arXiv:2309.08163.
  17. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751.
  18. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.
  19. Chatgpt an enfj, bard an istj: Empirical study on personalities of large language models. arXiv preprint arXiv:2305.19926.
  20. Using a language model to generate music in its symbolic domain while controlling its perceived emotion. IEEE Access.
  21. Jaeho Jeon and Seongyong Lee. 2023. Large language models in education: A focus on the complementary relationship between human teachers and chatgpt. Education and Information Technologies, pages 1–20.
  22. Mpi: Evaluating and inducing personality in pre-trained language models. arXiv preprint arXiv:2206.07550.
  23. Evaluating and inducing personality in pre-trained language models.
  24. John A Johnson. 2014. Measuring thirty facets of the five factor model with a 120-item public domain inventory: Development of the ipip-neo-120. Journal of research in personality, 51:78–89.
  25. Estimating the personality of white-box language models. arXiv e-prints, pages arXiv–2204.
  26. Psy-llm: Scaling up global mental health psychological services with ai-based large language models. arXiv preprint arXiv:2307.11991.
  27. Yash Mehta. 2023. Personality prediction. https://github.com/yashsmehta/personality-prediction.git. GitHub repository.
  28. Bottom-up and top-down: Predicting personality with psycholinguistic and language model features. In 2020 IEEE International Conference on Data Mining (ICDM), pages 1184–1189. IEEE.
  29. Who is gpt-3? an exploration of personality, values and demographics. arXiv preprint arXiv:2209.14338.
  30. David Noever and Sam Hyams. 2023. Ai text-to-behavior: A study in steerability. arXiv preprint arXiv:2308.07326.
  31. OpenAI. 2022. Chatgpt - https://openai.com/blog/chatgpt#OpenAI.
  32. OpenAI. 2023. Gpt-4 technical report - https://cdn.openai.com/papers/gpt-4.pdf.
  33. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.
  34. Keyu Pan and Yawen Zeng. 2023. Do llms possess a personality? making the mbti test an amazing evaluation for large language models. arXiv preprint arXiv:2307.16180.
  35. Improving language understanding by generative pre-training.
  36. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
  37. Personality traits in large language models. arXiv preprint arXiv:2307.00184.
  38. Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. arXiv preprint arXiv:2310.11324.
  39. Have large language models developed a personality?: Applicability of self-assessment tests in measuring personality in llms. arXiv preprint arXiv:2305.14693.
  40. An attention-based denoising framework for personality detection in social media texts. arXiv preprint arXiv:2311.09945.
  41. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  42. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837.
  43. Orders are unwanted: Dynamic deep graph convolutional network for personality detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 13896–13904.
  44. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (9)
  1. Xiaoyang Song (5 papers)
  2. Yuta Adachi (1 paper)
  3. Jessie Feng (1 paper)
  4. Mouwei Lin (1 paper)
  5. Linhao Yu (10 papers)
  6. Frank Li (18 papers)
  7. Akshat Gupta (41 papers)
  8. Gopala Anumanchipalli (30 papers)
  9. Simerjot Kaur (14 papers)
Citations (6)
View on Semantic Scholar