EuroParlVote Benchmark for Political NLP

• EuroParlVote benchmark links parliamentary debates to roll-call votes, enabling gender classification and vote prediction analysis.
• It integrates multilingual debates with detailed demographic metadata to assess fairness, bias, and robustness in political NLP models.
• The platform ParlAI Vote offers interactive auditing for subgroup disparities and counterfactual analysis in legislative decision-making.

EuroParlVote is a benchmark for evaluating LLMs in politically sensitive settings by linking European Parliament debate speeches to roll-call vote outcomes and attaching speaker-level demographic and political metadata such as gender, age, country, and political group. It was introduced as a multilingual parliamentary benchmark covering the 24 official EU languages and designed around two tasks—gender classification from speech and vote prediction from debate context—in order to study fairness, bias, and robustness under real legislative conditions rather than hypothetical ideological questionnaires (Yang et al., 7 Sep 2025). A companion demonstration paper presents ParlAI Vote, an interactive platform built on top of the benchmark that exposes its linked debates, speeches, votes, demographic slices, and model predictions for auditing and counterfactual analysis (Lin et al., 18 Sep 2025).

1. Motivation and conceptual scope

EuroParlVote was proposed as a response to two limitations in prior political-bias evaluation. First, earlier work was described as largely U.S.-centric. Second, much of that work relied on survey-style prompts such as the Political Compass Test rather than institutionally grounded political behavior. EuroParlVote instead evaluates whether models can infer demographic information from parliamentary speech and simulate actual legislative behavior from debate content in a real-world parliamentary environment (Yang et al., 7 Sep 2025).

The European Parliament is treated as a particularly suitable domain because it provides public and structured legislative traces: debates, speeches, roll-call votes, and member-level attributes. The benchmark is therefore oriented toward two linked goals emphasized again in the demonstration paper: equitable performance analysis across demographic groups, and transparent connections between language and consequential decisions rather than black-box prediction (Lin et al., 18 Sep 2025).

This design positions EuroParlVote as infrastructure for fairness and accountability research in political NLP. Its stated use cases include political analysis, decision support, representation studies, and public-interest NLP, but its central emphasis is diagnostic: revealing whether LLMs reproduce or amplify disparities already present in parliamentary discourse and representation (Yang et al., 7 Sep 2025).

2. Data composition and linkage structure

The benchmark is constructed by combining roll-call voting records from HowTheyVote.eu, European Parliament debate data aligned to votes using document references in the voting metadata, and Wikipedia pages for MEP demographic metadata. Publicly listed social-media accounts are also collected, although they are not central to the benchmark tasks (Yang et al., 7 Sep 2025). The roll-call source spans seven years and covers more than 1,200 MEPs.

Its defining unit is a linked speech-vote example. The construction pipeline begins from roll-call vote records, uses document references to align each vote to the corresponding debate or debates, and retains only debate speeches by MEPs who were present and actually cast a vote on the associated motion. Each retained example therefore connects a debate topic, a particular speech by an MEP, and that same MEP’s recorded vote on the relevant motion (Yang et al., 7 Sep 2025).

Several filters are applied. The benchmark excludes Abstain votes, excludes examples with missing debate topic, excludes examples with missing speech, and retains only MEPs who both spoke in the debate and cast a vote on the motion (Yang et al., 7 Sep 2025). After filtering, the benchmark paper reports approximately 22K debate speeches linked to 956 unique topics, while also noting that the abstract and introduction mention 969 corresponding roll-call votes; the system paper gives 969 roll-call votes as the operative collection size exposed in the interface (Yang et al., 7 Sep 2025, Lin et al., 18 Sep 2025). This suggests that the benchmark is centered on a speech-to-vote linkage at speaker level, with a small inconsistency in the reported count of vote-linked topics.

The metadata attached to each example includes gender, age, country, political group, date of birth, and public social-media accounts. The benchmark analysis concentrates on gender, political group, country, and age. Country coverage spans 27 EU member states plus the United Kingdom, which remains in the data because it was part of the Parliament during the historical period covered (Yang et al., 7 Sep 2025).

The benchmark includes eight political groups, including non-attached members: GUE/NGL, GREEN_EFA, SD, RENEW, EPP, ECR, ID, and NI. Their ideological labels are taken from ParlGov, CHES, and prior analyses of European Parliament politics. The demo paper additionally stresses that the platform can group, sort, and pivot examples by political affiliation, country, gender, and age, which makes subgroup disparities inspectable at vote level (Lin et al., 18 Sep 2025).

3. Task design, splits, and evaluation protocol

EuroParlVote defines two benchmark tasks. The first is gender classification from speech: given only an MEP’s debate speech, the model predicts whether the speaker is Male or Female and returns a confidence score on a scale of 1–5 together with a rationale. The second is vote prediction or voting simulation: given the debate topic and the speech, the model predicts whether the MEP would vote For or Against, again with a 1–5 confidence score and a rationale (Yang et al., 7 Sep 2025). Abstain is excluded from the benchmarked vote task entirely, making vote prediction a binary classification problem.

The benchmark paper partitions the data into train, development, and test using an approximately 8:1:1 ratio. A notable design choice is that the training set preserves the original class imbalance, whereas the development and test sets are balanced across vote labels (Yang et al., 7 Sep 2025).

Split	Vote counts	Gender share
Train	16,713 FOR, 2,293 AGAINST	55.1% male, 44.9% female
Dev	774 FOR, 774 AGAINST	59.0% male, 41.0% female
Test	774 FOR, 774 AGAINST	59.0% male, 41.0% female

For gender classification, the reported metrics are Accuracy, F1-Female, F1-Male, and AUC-ROC. For vote prediction, the reported metrics are Accuracy, F1-For, F1-Against, AUC-ROC, and Average Confidence (Yang et al., 7 Sep 2025). The benchmark does not provide formal equations for these metrics, nor formal fairness objectives, calibration definitions, subgroup-gap formulas, or loss functions. Bias is operationalized empirically through subgroup-specific F1 scores, confusion matrices, performance changes under prompt perturbations, and group-wise vote-prediction accuracy (Yang et al., 7 Sep 2025).

The vote task includes several comparison points. A uniform random baseline yields Accuracy 50.19, F1-For 50.35, F1-Against 50.03, AUC 50.19, and Average Confidence 2.50. A group-majority baseline, which predicts each MEP’s vote using the most common vote of that MEP’s political group in the training set, yields Accuracy 65.28, F1-For 73.65, F1-Against 48.96, AUC 65.25, and Average Confidence 4.03. A soft upper bound based on the majority decision of the MEP’s political group for that same vote in the test set yields Accuracy 88.28, F1-For 89.44, F1-Against 86.84, AUC 87.98, and Average Confidence 4.76; the paper attributes this to average within-group agreement of 95.29% (Yang et al., 7 Sep 2025).

4. Gender bias findings

The benchmark’s gender-classification results establish that the task is difficult in parliamentary language. All evaluated models are only around 60–65% accuracy, substantially below the 80%+ often reported for gender classification in less formal domains. The benchmark paper attributes this to formal political language, sparse explicit gender markers, and rhetorical devices such as irony or indirect criticism (Yang et al., 7 Sep 2025).

The central empirical result is asymmetric error against female MEPs. Models frequently default to male, and female MEPs are disproportionately misclassified as male. The strongest example is LLaMA-3.2-3B-Instruct, which records Accuracy 60.01, F1-Female 37.16, F1-Male 70.68, and AUC 55.26; 71.13% of female speakers are incorrectly labeled as male, whereas only 18.38% of male speakers are misclassified (Yang et al., 7 Sep 2025). Claude-3.5 Haiku, Gemini-2.5-Flash, and GPT-4o have better female performance and better balance between female and male F1, although none eliminates the disparity.

Gender perturbations in the vote-prediction task reinforce the same pattern. The benchmark introduces five settings: Without Gender, With Gender, All Male, All Female, and Swapped Gender, where the appended hint is “You are a (male|female) MEP.” Across models, All Female produces the worst performance for nearly every model, while All Male is often best or near-best (Yang et al., 7 Sep 2025). The degradation is especially visible for open-weight models. LLaMA-3.2 falls from Accuracy 67.64 in All Male to 63.33 in All Female, and Mistral-large falls from 77.83 to 70.64. GPT-4o is much more stable, with 83.72 in All Male and 83.66 in All Female (Yang et al., 7 Sep 2025).

The benchmark therefore distinguishes two related effects: representational bias in gender inference and performance disparity in vote simulation conditioned on, or associated with, female identity. The paper’s broader comparison is that proprietary models outperform open-weight alternatives not only in raw vote prediction but also in robustness and fairness under gender perturbation (Yang et al., 7 Sep 2025).

5. Political bias, ablations, and failure modes

EuroParlVote reuses the vote-prediction task to study political bias by comparing a baseline prompt without political-group information against a setting that appends “You are a MEP from XX political group.” Across models, centrist groups are easiest to predict. Performance tends to peak for RENEW and often also for SD and EPP. For GPT-4o, RENEW reaches 88.49 without group metadata and 89.93 with group metadata (Yang et al., 7 Sep 2025).

Extreme groups are harder overall, but the pattern is asymmetric rather than uniformly unfavorable to the right or the left. The benchmark reports that far-right groups are modeled better than far-left groups. For GPT-4o without group metadata, ID scores 86.07 and ECR 83.78, whereas GUE/NGL scores 75.19 and GREEN_EFA 66.21 (Yang et al., 7 Sep 2025). The same broad pattern appears across other models. The resulting interpretation is not simply that models are “left-leaning”; rather, center-left and centrist groups are easiest, far-right is predicted more reliably than far-left, and far-left and green-left groups are especially difficult.

Adding explicit political-group metadata generally improves performance and reduces disparities, especially on underrepresented or extreme groups. Concrete examples include GPT-4o on ID improving from 86.07 to 88.57, GPT-4o on GUE/NGL from 75.19 to 77.44, Gemini on ECR from 76.13 to 86.94, and Claude on ECR from 75.68 to 81.17 (Yang et al., 7 Sep 2025). The demo paper frames the same intervention as counterfactual or demographic-impact analysis: users can hold speech constant, add political-group information, and observe how predictions shift (Lin et al., 18 Sep 2025).

Several ablations clarify what the benchmark is measuring. A speech-masking ablation that removes speeches and keeps only topic plus gender drives performance to near random and produces extreme overprediction of For. LLaMA-3.2-3B without speech records Accuracy 50.12, F1-For 66.61, and F1-Against 0.87; GPT-4o without speech records Accuracy 50.39, F1-For 68.67, and F1-Against 3.03 (Yang et al., 7 Sep 2025). This indicates that speech content is essential and that metadata priors alone do not solve the task.

A translation ablation shows that preserving native-language original speeches matters. Using GPT-4o for downstream vote prediction, No translation reaches Accuracy 84.20, compared with 78.10 for GPT-4o-translated, 75.84 for T5-translated, and 76.35 for Google API-translated (Yang et al., 7 Sep 2025). This supports the benchmark’s multilingual design choice.

Attempts at mitigation through LoRA fine-tuning do not succeed. Fine-tuning LLaMA-3.2-3B and Mistral-large on 5,000 training examples with lora_dropout = 0.05, lora_alpha = 16, learning rate 1e-4, and 2 epochs slightly improves male performance while worsening subgroup disparity. LLaMA’s female F1 collapses from 37.16 to 19.94, and Mistral’s female F1 drops from 44.85 to 32.14 (Yang et al., 7 Sep 2025).

Error analysis further ties bias to discourse-modeling weaknesses. In gender classification, high-confidence mistakes are associated with stereotypical language cues: assertive, formal, technical, and analytical language is often interpreted as male, whereas social and environmental concerns are often interpreted as female. Older MEPs, especially older women with age > 70, are disproportionately misclassified (Yang et al., 7 Sep 2025). In vote prediction, recurrent failure modes include over-reliance on issue keywords, confusion of negative tone with an Against vote even when the speech concludes in support, and literal misreading of sarcasm or irony (Yang et al., 7 Sep 2025).

6. ParlAI Vote and the benchmark’s position in adjacent research

ParlAI Vote operationalizes EuroParlVote as an interactive auditing environment. The main page manages a dataset of 969 roll-call votes and provides search, filtering, and sorting tools. Each vote page displays the debate title, vote date, report ID, number of participants, and overall outcome, and centers on a Vote Breakdown visualization. By default, votes are grouped by political affiliation and ordered along the left–right spectrum; the same visualization can be pivoted by country, gender, and age (Lin et al., 18 Sep 2025).

The platform exposes the benchmark’s two core AI modules—Vote Prediction and Gender Prediction—at the level of individual debate speakers. Users can access ground truth, test demographic impact, compare multiple LLMs side by side, inspect reasoning, and use a counterfactual mode to modify variables such as political group or demographic information and observe sensitivity to identity attributes (Lin et al., 18 Sep 2025). Named models in the demo include GPT-4o, Gemini-2.5, LLaMA-3.2, and Mistral. The system paper describes this as a way to trace “the full chain of evidence: starting from a debate topic, navigating to linked speeches, and examining the corresponding roll-call votes” (Lin et al., 18 Sep 2025).

Within the broader landscape of European political NLP, EuroParlVote occupies a distinctive position. The party-level VAA study “Investigating LLMs as Voting Assistants via Contextual Augmentation” evaluates binary stance prediction over the EUandI-2024 questionnaire and reports substantial group disparities, but its target is party responses to 30 political statements rather than MEP-level roll-call voting (Chalkidis, 2024). “EuroCon” uses European Parliament deliberation records to benchmark consensus finding and resolution generation under simulated parliamentary constraints; voting there is part of the evaluation simulator rather than the primary prediction target (Zhang et al., 26 May 2025). “Assessing the Political Fairness of Multilingual LLMs” introduces 21-EuroParl, a 21-way multiparallel parliamentary speech dataset with political metadata, but it studies translation fairness and contains speeches rather than votes (Lerner et al., 23 Oct 2025). “Persona-driven Simulation of Voting Behavior in the European Parliament with LLMs” is closer in subject matter, but it is a zero-shot, persona-conditioned setup over 47 filtered 2024 votes rather than a benchmark with EuroParlVote’s speech-linked corpus design and split protocol (Kreutner et al., 13 Jun 2025).

A plausible implication is that EuroParlVote functions as a bridge resource between speech-centric EuroParl derivatives and vote-centric parliamentary prediction research. Its distinctive contribution is the systematic linkage of debate topics, speeches, roll-call outcomes, and rich demographics in a multilingual European parliamentary setting, together with an evaluation regime centered on subgroup disparities rather than aggregate performance alone (Yang et al., 7 Sep 2025).