2000 character limit reached
Beyond prompt brittleness: Evaluating the reliability and consistency of political worldviews in LLMs (2402.17649v3)
Published 27 Feb 2024 in cs.CL and cs.CY
Abstract: Due to the widespread use of LLMs, we need to understand whether they embed a specific "worldview" and what these views reflect. Recent studies report that, prompted with political questionnaires, LLMs show left-liberal leanings (Feng et al., 2023; Motoki et al., 2024). However, it is as yet unclear whether these leanings are reliable (robust to prompt variations) and whether the leaning is consistent across policies and political leaning. We propose a series of tests which assess the reliability and consistency of LLMs' stances on political statements based on a dataset of voting-advice questionnaires collected from seven EU countries and annotated for policy issues. We study LLMs ranging in size from 7B to 70B parameters and find that their reliability increases with parameter count. Larger models show overall stronger alignment with left-leaning parties but differ among policy programs: They show a (left-wing) positive stance towards environment protection, social welfare state and liberal society but also (right-wing) law and order, with no consistent preferences in the areas of foreign policy and migration.
- Standards for educational and psychological testing. American Educational Research Association.
- Probing pre-trained language models for cross-cultural differences in values. In Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), pages 114–130, Dubrovnik, Croatia. Association for Computational Linguistics.
- On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623.
- Kenneth Benoit and Michael Laver. 2006. Party policy in modern democracies. Routledge.
- Manifesto corpus. version: 2021.1. Berlin: WZB Berlin Social Science Center.
- Scaling instruction-finetuned language models.
- Bias identification and attribution in NLP models with regression and effect sizes. Northern European Journal of Language Technology, 8(1).
- Questioning the survey responses of large language models. ArXiv, abs/2306.07951.
- Spurious correlations in reference-free evaluation of text generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1443–1454, Dublin, Ireland. Association for Computational Linguistics.
- Towards measuring the representation of subjective global opinions in language models. ArXiv, abs/2306.16388.
- Robbie: Robust bias evaluation of large generative language models.
- From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair NLP models. pages 11737–11762.
- “fifty shades of bias”: Normative ratings of gender bias in GPT generated English text. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1862–1876, Singapore. Association for Computational Linguistics.
- The political ideology of conversational ai: Converging evidence on chatgpt’s pro-environmental, left-libertarian orientation. SSRN Electronic Journal.
- The curious case of neural text degeneration.
- Dirk Hovy and Shrimai Prabhumoye. 2021. Five sources of bias in natural language processing. Language and Linguistics Compass, 15(8):e12432.
- Mistral 7b.
- Chapel hill expert survey trend file, 1999–2019. Electoral studies, 75:102420.
- Prompt waywardness: The curious case of discretized interpretation of continuous prompts. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3631–3643, Seattle, United States. Association for Computational Linguistics.
- Gender bias and stereotypes in large language models. In Proceedings of The ACM Collective Intelligence Conference, CI ’23. ACM.
- The unified framework of media diversity: A systematic literature review. Digital Journalism, 8(5):605–642.
- Denis McQuail. 1993. Media performance: Mass communication and the public interest.
- Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048–11064, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- More human than human: Measuring chatgpt political bias. Public Choice.
- Large language models propagate race-based medicine. npj Digital Medicine, 6(1).
- Thomas R. Palfrey and Keith T. Poole. 1987. The relationship between information, ideology, and voting behavior. American Journal of Political Science, 31(3):511–530.
- The self-perception and political biases of chatgpt. arXiv preprint arXiv:2304.07333.
- Whose opinions do language models reflect? In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org.
- You don’t need a personality test to know these models are unreliable: Assessing the reliability of large language models on psychometric instruments.
- Margit Tavits. 2007. Principle vs. pragmatism: Policy shifts and political competition. American Journal of Political Science, 51(1):151–165.
- Do llms design.
- Llama 2: Open foundation and fine-tuned chat models.
- Evaluating open-qa evaluation.
- Not all countries celebrate thanksgiving: On the cultural dominance in large language models. CoRR, abs/2310.12481.
- Albert Webson and Ellie Pavlick. 2022. Do prompt-based models really understand the meaning of their prompts? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2300–2344, Seattle, United States. Association for Computational Linguistics.
- Calibrate before use: Improving few-shot performance of language models.
- Tanise Ceron (6 papers)
- Neele Falk (4 papers)
- Ana Barić (3 papers)
- Dmitry Nikolaev (33 papers)
- Sebastian Padó (39 papers)