ParlAI Vote: EU Debate & Voting Analysis
- ParlAI Vote is an interactive system that unifies European Parliament debates, votes, speeches, and demographics in one explorable interface.
- It enables detailed auditing of large language models through vote prediction, counterfactual analysis, and subgroup error inspection.
- The platform builds on the EuroParlVote benchmark, merging legislative outcomes with political metadata to assess bias and fairness in LLM performance.
ParlAI Vote is an interactive system for exploring European Parliament debates and votes, and for testing LLMs on vote prediction and bias analysis. It connects debate topics, speeches, roll-call outcomes, and demographic metadata in a single interface, with the stated aim of making parliamentary behavior easier to explore, evaluate, and audit. Built on top of the EuroParlVote benchmark, it is presented as a research-oriented, AI-powered demonstration platform that supports reproducibility, subgroup error analysis, counterfactual inspection, and comparative model evaluation in a politically sensitive domain (Lin et al., 18 Sep 2025).
1. Definition and intended scope
ParlAI Vote is presented as a research-oriented, AI-powered demonstration platform built on top of the EuroParlVote benchmark to make European Parliament behavior easier to explore, evaluate, and audit. Its motivation is twofold. First, LLMs are increasingly being used to analyze political text, but existing evaluations often treat political prediction as a black box and rarely expose whether performance is equitable across demographic groups. Second, the European Parliament is an unusually transparent setting: debates, roll-call votes, and member attributes are public, detailed, and richly linked, yet scattered across sources and formats. ParlAI Vote is designed to unify these signals in one interactive system so that debate language, legislative outcomes, and model behavior can be inspected jointly rather than in isolation (Lin et al., 18 Sep 2025).
The platform is not just a voting archive. Its central premise is that issue-level debate content, speech-level evidence, parliamentary outcomes, and demographic context should remain visible at the same time. The authors introduce it as the first AI-powered system, according to the paper, that unifies European Parliament debates, roll-call votes, and demographic data into a live explorable interface. In that formulation, the system serves not only as a visualization front end, but also as an instrument for auditing the strengths and limitations of current LLMs in political analysis (Lin et al., 18 Sep 2025).
2. Data substrate and benchmark foundation
ParlAI Vote is built on the EuroParlVote benchmark, which links European Parliament debate speeches to roll-call vote outcomes and enriches them with metadata for each Member of the European Parliament (MEP). The benchmark is constructed from roll-call voting records spanning 7 years from HowTheyVote.eu, the corresponding debate speeches, and demographic and political metadata. The alignment step uses the document references in the voting metadata to match each vote to the debate that preceded it, retaining only cases where the MEP was present and actually cast a vote. Abstentions, examples with missing debate topics, and examples with missing speeches are removed. This yields about 22K debate speeches linked to 969 corresponding roll-call votes and 956 unique topics (Yang et al., 7 Sep 2025).
The benchmark includes gender, political group, age, country, and additional MEP attributes. Gender is represented as male or female; political groups are organized across 8 groups, including non-attached members; age ranges from 25 to 83; and country coverage spans 27 EU member states plus the UK. The data are split roughly 8:1:1 into train, development, and test, with the training split preserving original class imbalance and the development and test splits balanced across vote labels (Yang et al., 7 Sep 2025).
| Layer | Contents | Function |
|---|---|---|
| Debate layer | debate topics, titles, dates, report IDs | issue-level context |
| Speech layer | individual speeches | speaker-level evidence |
| Vote layer | roll-call outcomes, participants, final outcome | legislative supervision |
| Metadata layer | gender, age, country, political group | subgroup auditing |
Within the ParlAI Vote interface itself, the system organizes 969 roll-call votes. For each vote, it exposes the debate title, vote date, report ID, number of participants, and the final outcome. This tight coupling between benchmark structure and interface design is what makes the platform simultaneously a visualization environment and a model-auditing surface (Lin et al., 18 Sep 2025).
3. Interface architecture and analytic workflow
The main page supports search, filtering, and sorting over the full vote collection, allowing legislative events to be isolated and trends over time to be inspected. Its principal visualization is a “Vote Breakdown” display that shows how MEPs voted. By default, votes are grouped by political affiliation and ordered along the left–right spectrum, so that ideological structure is immediately visible. The same breakdown can then be pivoted by country, gender, or age, making it possible to inspect how demographic structure intersects with political behavior (Lin et al., 18 Sep 2025).
For each vote, the interface surfaces both metadata and linked speech evidence. Users can browse debates, inspect linked speeches, compare real voting outcomes with predictions from frontier LLMs, and view error breakdowns by demographic group. In the prediction view, a user can select a debate speaker, inspect the ground truth vote and or gender, run predictions using only the speech, and then add demographic or political-group attributes to observe how outputs change (Lin et al., 18 Sep 2025).
A central design feature is the counterfactual module called the “Demographic Impact Explorer.” It allows users to toggle attributes such as political group or demographic markers and to observe how sensitive a model is to those additions. The paper describes four interactive functions that structure this workflow. Ground truth anchoring makes errors interpretable by juxtaposing predictions with actual outcomes. The demographic impact explorer compares speech-only predictions with predictions augmented by attributes. Multiple models can be compared side-by-side on exactly the same input. The interface also exposes model rationales so that users can inspect whether a prediction appears grounded in substantive argumentation or in superficial cues. The figure example in the paper uses the “Ecodesign Regulation” vote by Marc Angel, with topic plus speech as context and Llama3.2 as the predictor (Lin et al., 18 Sep 2025).
4. Predictive tasks and evaluation logic
ParlAI Vote is centered on two predictive tasks: vote prediction and gender prediction. These are the core tasks used in the underlying EuroParlVote evaluation. In the vote task, the model is prompted with the debate topic and the debate speech, and in some conditions additional metadata are appended, such as a gender hint (“You are a male/female MEP”) or a group hint (“You are an MEP from XX political group”). The output space is binary—For or Against—because Abstain is excluded from the benchmark. The prompt asks the model to state its voting position, give a confidence score on a 1–5 scale, and provide reasoning (Yang et al., 7 Sep 2025).
The benchmark reports Accuracy, F1-For, F1-Against, AUC-ROC, and average confidence in some tables. It also defines three baselines: Random, Group-majority, and Intra-group agreement. The last is treated as a soft upper bound because it assumes knowledge of group behavior for that specific vote in the test set; the paper notes 95.29% average within-group agreement. This benchmark structure reflects a central property of European Parliament voting: behavior is highly group-structured, but the task still requires modeling speech semantics beyond simple priors (Yang et al., 7 Sep 2025).
ParlAI Vote turns this benchmark logic into an interactive workflow. Rather than exposing only aggregate summary metrics, it makes instance-level prediction, subgroup disparity, and rationale inspection first-class objects of analysis. A plausible implication is that the platform is designed less as a leaderboard surface than as an audit environment. That interpretation is reinforced by one benchmark result emphasized in the underlying paper: when speech is masked out and only topic plus gender remain, accuracy falls to roughly 50%, near random, and predictions collapse toward the dominant For class. The task therefore depends materially on speech content, not merely on metadata (Yang et al., 7 Sep 2025).
5. Revealed bias patterns
The demo paper reports several substantive findings from the underlying EuroParlVote-based evaluation. On gender, female MEPs are disproportionately misclassified as male in gender prediction, and vote simulation performance drops when speakers are labeled as female. Proprietary frontier models such as GPT-4o and Gemini-2.5 show lower misclassification rates and more balanced treatment of female speakers, whereas open-weight models such as LLaMA-3.2 and Mistral exhibit stronger male bias and weaker calibration. LoRA fine-tuning did not resolve this pattern and often worsened performance for female MEPs (Lin et al., 18 Sep 2025).
The benchmark paper makes these disparities more explicit. In vote prediction without gender, reported accuracies are about 67.10% for LLaMA-3.2-3B, about 75.42% for Mistral-large, about 80.61% for Claude-3.5, about 83.10% for Gemini-2.5, and about 84.20% for GPT-4o. Under gender manipulations, the All Female condition yields the worst performance and the All Male condition often the best or near-best performance. For example, LLaMA-3.2 drops to 63.33% in the All Female setting, while GPT-4o remains comparatively stable at around 83.66%. In gender classification, LLaMA-3.2 has F1-Female = 37.16 and misclassifies 71.13% of female speakers as male; after LoRA fine-tuning on 5,000 training examples, female F1 falls further to 19.94 (Yang et al., 7 Sep 2025).
Politically, the models exhibit a centrist bias. The demo paper states that they predict centrist and liberal groups most accurately, while struggling more with ideologically extreme groups. It also notes that far-right groups are predicted more reliably than far-left groups, which the authors attribute to the possibility that right-aligned rhetoric may be more uniform and therefore easier for models to capture. Adding explicit political-group identifiers improves fairness by increasing accuracy on underrepresented and extreme groups (Lin et al., 18 Sep 2025).
The benchmark paper sharpens that picture by reporting that centrist groups such as RENEW and SD, and often EPP, are easiest for the models, whereas far-left groups such as GUE/NGL and GREEN_EFA are more difficult. Open-weight models show stronger political bias, while proprietary models are more robust and better calibrated. ParlAI Vote makes these asymmetries visible through subgroup views, counterfactual toggles, and side-by-side model comparison rather than leaving them buried in aggregate accuracy tables (Yang et al., 7 Sep 2025).
6. Relation to adjacent research and broader significance
ParlAI Vote sits within a broader line of work on LLMs in electoral and legislative analysis, but its emphasis is distinctive. Related research has examined LLMs as Voting Advice Applications for the 2024 European Parliament elections, finding that MIXTRAL is highly accurate with an 82% accuracy on average but also exhibits a significant performance disparity across political groups, with accuracy spanning roughly 50–95%; expert-curated augmentation improves performance by approximately 9% (Chalkidis, 2024). Other work has studied persona-driven simulation of MEP voting, reporting a weighted F1 score of approximately 0.793 and showing that performance is stronger for center-left and centrist-progressive groups than for ideological edge groups such as GUE/NGL, ECR, and ID (Kreutner et al., 13 Jun 2025).
Against that backdrop, ParlAI Vote’s main contribution is not a new predictive architecture but a unified interface for reproducibility, auditing, and counterfactual analysis. The platform turns a scattered political benchmark into a live, explorable environment where the same inputs and comparisons can be revisited consistently. For auditing, it allows model errors to be inspected by subgroup, by political ideology, and by attribute manipulation rather than only through aggregate accuracy. For education and public engagement, it allows users to trace the chain from debate topic to speech to vote and to see where models succeed or fail (Lin et al., 18 Sep 2025).
The system is therefore best understood as accountability infrastructure for politically sensitive NLP. The demo paper explicitly argues that in such domains, performance alone is not enough; a transparent interface that reveals demographic and ideological disparities is necessary to understand the limits and risks of LLM-based political analysis. This suggests that ParlAI Vote is significant not merely because it visualizes parliamentary data, but because it operationalizes a particular methodological stance: political AI should be inspectable at the level of data linkage, prediction context, subgroup disparity, and rationale exposure, rather than treated as a black-box classifier (Lin et al., 18 Sep 2025).
