Human Simulacra: Benchmarking the Personification of Large Language Models (2402.18180v5)
Abstract: LLMs are recognized as systems that closely mimic aspects of human intelligence. This capability has attracted attention from the social science community, who see the potential in leveraging LLMs to replace human participants in experiments, thereby reducing research costs and complexity. In this paper, we introduce a framework for LLMs personification, including a strategy for constructing virtual characters' life stories from the ground up, a Multi-Agent Cognitive Mechanism capable of simulating human cognitive processes, and a psychology-guided evaluation method to assess human simulations from both self and observational perspectives. Experimental results demonstrate that our constructed simulacra can produce personified responses that align with their target characters. Our work is a preliminary exploration which offers great potential in practical applications. All the code and datasets will be released, with the hope of inspiring further investigations.
- Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
- R.C. Atkinson and R.M. Shiffrin. 1968. Human memory: A proposed system and its control processes. volume 2 of Psychology of Learning and Motivation, pages 89–195. Academic Press.
- Alan D. Baddeley and Graham Hitch. 1974. Working memory. volume 8 of Psychology of Learning and Motivation, pages 47–89. Academic Press.
- Evaluating the replicability of social science experiments in nature and science between 2010 and 2015. Nature human behaviour, 2(9):637–644.
- Can ai language models replace human participants? Trends in Cognitive Sciences.
- Explicit representation of confidence informs future value-based decisions. Nature Human Behaviour, 1(1):0002.
- Adrian Furnham. 1996. The big five versus the big four: the relationship between the myers-briggs type indicator (mbti) and neo-pi five factor model of personality. Personality and individual differences, 21(2):303–307.
- Ai and the transformation of social science research. Science, 380(6650):1108–1109.
- Handbook of personality psychology. Elsevier.
- Matthew Hutson and Ashley Mastin. 2023. Guinea pigbots. Science, 381(6654):121–123.
- Mistral 7b.
- Carl Jung. 1923. Psychological types. Harcourt, Brace.
- Beyond rating scales: With targeted evaluation, language models are poised for psychological assessment. Psychiatry Research, page 115667.
- Are you in a masquerade? exploring the behavior and impact of large language model driven social bots in online social networks. arXiv preprint arXiv:2307.10337.
- Jailbreaking chatgpt via prompt engineering: An empirical study. ArXiv, abs/2305.13860.
- Robert R McCrae and Oliver P John. 1992. An introduction to the five-factor model and its applications. Journal of personality, 60(2):175–215.
- Situational judgment tests as an alternative measure for personality assessment. European Journal of Psychological Assessment.
- Isabel Briggs Myers. 1962. The myers-briggs type indicator: Manual (1962).
- Ulric Neisser. 1976. Cognition and Reality: Principles and Implications of Cognitive Psychology. W H Freeman/Times Books/ Henry Holt & Co.
- Human problem solving, volume 104. Prentice-hall Englewood Cliffs, NJ.
- Limited individual attention and online virality of low-quality information. Nature Human Behaviour, 1(7):1–7.
- In-context impersonation reveals large language models’ strengths and biases. arXiv preprint arXiv:2305.14930.
- Character-llm: A trainable agent for role-playing. In Proceedings of the 2023 Conference on EMNLP, pages 13153–13187.
- Robin James Stuart Sloan. 2015. Virtual character design for games and interactive media. CRC Press.
- Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
- Characterchat: Learning towards conversational ai with personalized social support. arXiv preprint arXiv:2308.10278.
- Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926.
- Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models. arXiv preprint arXiv:2310.00746.
- How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms. arXiv preprint arXiv:2401.06373.
- Judging llm-as-a-judge with mt-bench and chatbot arena.
- Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593.
- Qiuejie Xie (1 paper)
- Qiming Feng (1 paper)
- Tianqi Zhang (17 papers)
- Qingqiu Li (11 papers)
- Yuejie Zhang (31 papers)
- Rui Feng (67 papers)
- Shang Gao (74 papers)
- Linyi Yang (52 papers)
- Liang He (202 papers)
- Yue Zhang (618 papers)