Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Llama meets EU: Investigating the European Political Spectrum through the Lens of LLMs (2403.13592v2)

Published 20 Mar 2024 in cs.CL

Abstract: Instruction-finetuned LLMs inherit clear political leanings that have been shown to influence downstream task performance. We expand this line of research beyond the two-party system in the US and audit Llama Chat in the context of EU politics in various settings to analyze the model's political knowledge and its ability to reason in context. We adapt, i.e., further fine-tune, Llama Chat on speeches of individual euro-parties from debates in the European Parliament to reevaluate its political leaning based on the EUandI questionnaire. Llama Chat shows considerable knowledge of national parties' positions and is capable of reasoning in context. The adapted, party-specific, models are substantially re-aligned towards respective positions which we see as a starting point for using chat-based LLMs as data-driven conversational engines to assist research in political science.

View on arXiv
Definition Search Book Streamline Icon: https://streamlinehq.com
References (18)
  1. Palm 2 technical report.
  2. Adapting large language models via reading comprehension.
  3. Deep reinforcement learning from human preferences.
  4. Scaling instruction-finetuned language models.
  5. Bold: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 862–872, New York, NY, USA. Association for Computing Machinery.
  6. From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair NLP models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11737–11762, Toronto, Canada. Association for Computational Linguistics.
  7. Opiniongpt: Modelling explicit biases in instruction-tuned llms.
  8. The political ideology of conversational ai: Converging evidence on chatgpt’s pro-environmental, left-libertarian orientation. ArXiv, abs/2301.01768.
  9. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  10. Mixtral of experts.
  11. Scalable agent alignment via reward modeling: a research direction. CoRR, abs/1811.07871.
  12. euandi2019: Project description and datasets documentation. SSRN Electronic Journal.
  13. OpenAI. 2023. Gpt-4 technical report.
  14. Nils Reimers. 2021. Easy NMT - Easy to use, state-of-the-art Neural Machine Translation.
  15. Whose opinions do language models reflect? In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org.
  16. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems, volume 33, pages 3008–3021. Curran Associates, Inc.
  17. Llama 2: Open foundation and fine-tuned chat models.
  18. Chain of thought prompting elicits reasoning in large language models. ArXiv, abs/2201.11903.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (2)
  1. Ilias Chalkidis (40 papers)
  2. Stephanie Brandl (14 papers)
Citations (4)
View on Semantic Scholar

Summary

Investigating the European Political Spectrum through the Lens of LLMs

Contextualizing the Study

In the ever-evolving landscape of NLP, the scrutiny of LLMs extends beyond their linguistic capabilities to encompass their alignment with social and political dynamics. The paper in focus extends this examination into the complex multi-party system of the European Union (EU). It leverages LLMs, with a spotlight on Llama Chat, to dissect the model's inherent political knowledge, its reasoning capabilities within the EU political framework, and the feasibility of model adaptation to reflect specific political stances.

Dataset and Methodological Framework

The paper employs two foundational datasets. First, it introduces the EU Debates Corpus, a comprehensive collection of plenary speeches from the European Parliament, spanning from 2009 to 2023. This corpus serves as the bedrock for model adaptation and auditing. Secondly, it utilizes the EUandI questionnaire, designed to match EU citizens with political parties based on issue stances before the 2019 EU elections. This dataset facilitates the evaluation of the model's political knowledge and reasoning.

The analytical journey bifurcates into two pathways:

  • Contextualized Auditing: This evaluates the model’s inherent political biases and reasoning abilities using the EUandI questionnaire.
  • Political Adaptation / Alignment: Here, the model undergoes fine-tuning with speeches from specific political groups within the European Parliament to examine the shift in its political alignment.

Core Findings

Political Knowledge and Reasoning

Through contextualized auditing, the model displayed a varied understanding of political parties’ stances, showing a stronger affiliation towards certain ideologies. Models aligned closely with left-wing and green parties, evidencing higher predictive accuracies for these groups. This differential understanding underscores the varying complexity of political ideologies and the model's intrinsic alignment with them.

Adapting Models to Political Ideologies

The model adaptation process revealed that fine-tuning on political speeches realigns the model's stance towards the respective political ideologies, albeit with nuances. While the adaptation effectively mirrored more homogeneously ideologized parties (like Greens/EFA), it encountered challenges with parties having a broader ideological spectrum (like EPP and S&D), hinting at the complexity of capturing nuanced political alignments within LLMs.

Implications and Future Pathways

The paper navigates through uncharted territories of employing LLMs to understand and adapt to the multifaceted political landscapes of the EU. It sets a precedent for utilizing LLMs in political science research, offering a new lens to discern political biases and adapt models to serve as aides in political analysis. However, the journey does not end here. The pathway carved by this research beckons further exploration into multilingual models to encompass the EU's linguistic diversity and a deeper dive into chronological analyses that capture the evolution of political stances over time.

In the grand scheme, this research underscores the potent utility of LLMs in transcending mere linguistic understanding to grasp the intricate weave of political ideologies. It stands as a testament to the potential of LLMs in enriching our comprehension of the digital reflection of societal structures and propelling the dialogue on the ethical implications of AI in mirroring and adapting to our political selves.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets