Using LLMs to Model the Beliefs and Preferences of Targeted Populations (2403.20252v1)
Abstract: We consider the problem of aligning a LLM to model the preferences of a human population. Modeling the beliefs, preferences, and behaviors of a specific population can be useful for a variety of different applications, such as conducting simulated focus groups for new products, conducting virtual surveys, and testing behavioral interventions, especially for interventions that are expensive, impractical, or unethical. Existing work has had mixed success using LLMs to accurately model human behavior in different contexts. We benchmark and evaluate two well-known fine-tuning approaches and evaluate the resulting populations on their ability to match the preferences of real human respondents on a survey of preferences for battery electric vehicles (BEVs). We evaluate our models against their ability to match population-wide statistics as well as their ability to match individual responses, and we investigate the role of temperature in controlling the trade-offs between these two. Additionally, we propose and evaluate a novel loss term to improve model performance on responses that require a numeric response.
- Using large language models to simulate multiple humans and replicate human subject studies. In International Conference on Machine Learning, pp. 337–371. PMLR, 2023.
- Understanding and shifting preferences for battery electric vehicles, 2022. URL https://arxiv.org/abs/2202.08963.
- Together Computer. Redpajama: an open dataset for training large language models, 2023. URL https://github.com/togethercomputer/RedPajama-Data.
- Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 2023.
- BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018. URL http://arxiv.org/abs/1810.04805.
- Can ai language models replace human participants? Trends in Cognitive Sciences, 2023.
- Alpacafarm: A simulation framework for methods that learn from human feedback. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=4hturzLcKX.
- The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.
- The challenge of using llms to simulate human behavior: A causal inference perspective. SSRN Electronic Journal, 2023. ISSN 1556-5068. doi: 10.2139/ssrn.4650172. URL http://dx.doi.org/10.2139/ssrn.4650172.
- Evaluating large language models in generating synthetic hci research data: a case study. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1–19, 2023.
- Lora: Low-rank adaptation of large language models. CoRR, abs/2106.09685, 2021. URL https://arxiv.org/abs/2106.09685.
- Challenges and applications of large language models. arXiv preprint arXiv:2307.10169, 2023.
- Generative agents: Interactive simulacra of human behavior, 2023.
- Catboost: unbiased boosting with categorical features. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper_files/paper/2018/file/14491b756b3a51daac41c24863285549-Paper.pdf.
- Improving language understanding by generative pre-training, 2018.
- Personality traits in large language models, 2023.
- Role-play with large language models, 2023.
- Llama 2: Open foundation and fine-tuned chat models, 2023.
- Attention is all you need. Advances in neural information processing systems, 30, 2017.
- On decoding strategies for neural text generators. Transactions of the Association for Computational Linguistics, 10:997–1012, 2022.
- Large language models are diverse role-players for summarization evaluation, 2023.
- A survey of large language models, 2023.