NeutQA-440: Political Bias Benchmark

• NeutQA-440 is a benchmark dataset of 440 non-adversarial prompts designed to evaluate fine-grained partisan associations in LLM responses.
• It employs a pairwise logical plausibility protocol with triplicate iterations and refusal tracking to measure consistent directional bias.
• Empirical results reveal high bias rates (up to 98%) and stark representational asymmetries, particularly in USA models, underscoring risks in LLM deployments.

Searching arXiv for the benchmark paper and closely related political-bias evaluation work. NeutQA-440 is a benchmark dataset for evaluating partisan associations in LLMs under non-adversarial prompting conditions. It was introduced together with AdverQA-440 to move evaluation beyond broad left–right orientation and toward fine-grained auditing of concrete associations attached to political leaders and parties across two national contexts, the United States and India. Its defining purpose is to test whether models systematically judge one of two nearly identical politically grounded statements as more logically plausible when the only substantive difference is the referenced political entity. The benchmark is positioned as cross-cultural, standardized, and explicitly concerned with harmful representational asymmetries rather than only aggregate ideological leaning (Kumar et al., 24 Sep 2025).

1. Definition and motivation

NeutQA-440 consists of non-adversarial prompts designed to probe partisan associations in LLMs through balanced, naturalistic descriptors for both positive and negative traits. The benchmark was created in response to two limitations identified in prior political-bias evaluation practice: a largely Western-centric focus, often centered on American or European questionnaires and statement sets such as the Political Compass Test, and a concentration on overall “left” versus “right” leanings rather than harmful or adversarial representational associations (Kumar et al., 24 Sep 2025).

The benchmark’s motivation is therefore not merely to identify generic political tendencies, but to examine whether models attach asymmetric plausibility judgments to concrete political actors in everyday or plausible settings. The inclusion of India alongside the USA is central to that aim. This cross-cultural design addresses underrepresentation of non-Western political contexts and enables comparison between a major Western setting and a major non-Western one. The accompanying empirical framing emphasizes robustness and safety under both neutral and adversarial comparisons, suggesting that NeutQA-440 is intended as an auditing instrument for real-world deployment settings rather than as a narrow ideological questionnaire (Kumar et al., 24 Sep 2025).

A plausible implication is that the benchmark reframes partisan bias evaluation as a problem of representational harm and comparative plausibility assignment, rather than solely one of scalar ideological placement.

2. Taxonomic structure and prompt design

The dataset is organized through a three-level partisan bias taxonomy. At the highest level are themes, defined as broad evaluative dimensions relevant to political actors. There are 11 themes, and they are the same for both countries. At the second level are topics, which operationalize fine-grained manifestations or actions or attributes under each theme. These topics are divided into two main types: non-adversarial descriptors and adversarial actions. At the third level are identity attributes, namely specific political leaders and parties that are country-specific (Kumar et al., 24 Sep 2025).

For NeutQA-440 specifically, the relevant topic type is the non-adversarial descriptor. The dataset covers both leaders and parties. For both non-adversarial and adversarial prompts, there are 44 unique topics for leaders and 44 unique topics for parties, represented in both countries. Prompt templates use placeholders for <identity_1>, <identity_2>, and <topic>, and are organized into four template families: positive leader, negative leader, positive party, and negative party (Kumar et al., 24 Sep 2025).

Each of the paired datasets contains 440 prompts, which gives NeutQA-440 its name. The benchmark covers the USA and India at the level of major political parties and prominent modern and recent leaders. The USA portion includes 6 leaders, split as 3 Democratic and 3 Republican, together with 2 major parties. The India portion includes 5 leaders with mixed affiliation and 3 parties: Indian National Congress, BJP, and Communist Party of India - Marxist (Kumar et al., 24 Sep 2025).

This design makes the dataset modular and combinatorial. The paper states that both NeutQA-440 and AdverQA-440 cover the same combinatorial space, differing only in prompt style. That pairing is methodologically important because it allows comparison between ordinary-seeming prompts and overtly provocative ones without changing the underlying topical coverage.

3. Evaluation protocol and formalization

NeutQA-440 uses a pairwise logical plausibility judgment protocol. For each prompt, a model receives two nearly identical sentences differing only in the named political entity and is asked: “Which sentence is more logically plausible?” The protocol includes three response iterations per prompt, counterbalanced entity order to remove positional bias, and explicit recording of refusals. Only consistent non-refusal choices are treated as evidence of bias direction (Kumar et al., 24 Sep 2025).

The benchmark’s formal definitions are given as follows. Bias detection for model $i$ and prompt $j$ is

$$B_{i,j} = \begin{cases} 1 & \text{if } R_{i,j}^{(k)} \in \{e_1, e_2\} \text{ for all } k \ 0 & \text{if any } R_{i,j}^{(k)} = \text{``refuse''} \end{cases}$$

Directional consistency is defined by the mode over the three iterations:

$$C_{i,j} = \text{mode}(\{R_{i,j}^{(1)}, R_{i,j}^{(2)}, R_{i,j}^{(3)}\})$$

Aggregate asymmetry, or skew, for an entity $e$ is defined as

$$\text{Skew}(e) = \frac{\sum_{i \in P^+} \mathbb{1}[C_{i,*} = e]}{\sum_{i \in P^-} \mathbb{1}[C_{i,*} = e]}$$

where $P^+$ and $P^-$ are positive and negative prompt sets (Kumar et al., 24 Sep 2025).

The labeling scheme does not rely on human annotation. Instead, “ground truth” is operationally defined by the model’s own response consistency across three iterations. Responses are categorized as biased when the model consistently favors one entity over the other, as refusals when it declines to choose, and by sentiment direction according to whether the topic is positive or negative (Kumar et al., 24 Sep 2025).

This protocol is notable because it treats refusal as analytically meaningful rather than simply missing data. The paper interprets refusal as a possible indicator of safety awareness, while consistent choice indicates a directional association.

4. Composition and analytical utility

NeutQA-440 is designed to facilitate precise measurement of whether LLMs make more positive or more negative associations with specific political actors under realistic prompt framing. Because the prompts are non-adversarial, the benchmark is intended to test whether bias appears in naturalistic rather than extreme contexts. The paper states that this allows measurement of the risk present in realistic, everyday conditions, complementing the more overtly polarized scenarios in AdverQA-440 (Kumar et al., 24 Sep 2025).

The benchmark also supports analysis at multiple granularities. Its taxonomy and template structure make it suitable for model comparison, cross-cultural trend detection, evaluation of refusal and safety mechanisms, and quantitative estimation of asymmetry at the levels of themes, topics, entities, and country. The paper further states that the modular template-based design makes it easy to expand, replicate, or adapt to additional contexts (Kumar et al., 24 Sep 2025).

The following table summarizes the benchmark’s core structural elements as explicitly described.

Component	Specification
Countries	USA and India
Prompt type	Non-adversarial prompts
Dataset size	440 prompts
Themes	11
Leader topics	44 unique topics
Party topics	44 unique topics
USA coverage	6 leaders, 2 major parties
India coverage	5 leaders, 3 parties

Because the paper emphasizes major political parties and prominent modern and recent leaders, the dataset is aimed at high-salience public entities rather than diffuse ideological abstractions. This suggests that NeutQA-440 is particularly suited to studying representational concentration around named actors, where downstream harms may be sharper and more socially legible.

5. Empirical findings from NeutQA-440 and comparative patterns

The reported results indicate high susceptibility to biased partisan associations across all tested models. Across adversarial and neutral settings, bias rates range from 91.6% to 100%. For the neutral prompts of NeutQA-440 specifically, the benchmark elicited higher and more consistent bias rates, reported as 98.2–95.6%, than the overtly adversarial prompts in AdverQA-440, reported as 95.0–92.6% (Kumar et al., 24 Sep 2025).

The USA results show pronounced asymmetry. The Democratic party received 14 times more positive associations than Republicans, with counts reported as 600 versus 48, while Republicans reversely received 13x more negative associations. At the leader level, Democratic leaders received approximately 93% of positive associations, whereas Republican leaders received 6.2%. The paper further states that models readily linked Republican figures to extreme harms, including “systemic embezzlement” and “protecting sexual violence offenders” (Kumar et al., 24 Sep 2025).

The India results are described as more mixed in polarity. BJP received both the most positive and the most negative associations, which the paper interprets as indicating salience and controversy rather than a purely one-sided skew. CPIM and INC were also associated with extreme acts, including “silencing whistleblowers through torture” and “rigging elections.” At the leader level, Modi showed balanced sentiment, Vajpayee mostly positive, and both Gandhis mostly negative (Kumar et al., 24 Sep 2025).

Across models, agreement on bias direction is reported to be high despite differences in provider, safety stack, and underlying architecture. The paper reports agreement rates of 90–98% on bias direction. It also states that bias is more pronounced in the USA context than in India, with the Democrat–Republican split substantially starker than the BJP–INC–CPIM pattern (Kumar et al., 24 Sep 2025).

These findings lead to a central interpretive claim: bias appears embedded rather than strongly prompt-sensitive. The fact that neutral prompts elicited higher and more consistent bias rates than adversarial prompts is used to argue that the issue is not confined to obviously inflammatory inputs.

6. Interpretation, limitations, and broader significance

The benchmark is presented as evidence that LLMs can reproduce consistent and sometimes extreme partisan associations even when prompted with balanced and plausible comparisons. The paper characterizes this as a danger to democratic discourse, noting risks such as amplifying information asymmetries, echo chambers, and potentially influencing voter behavior. It also argues that mitigation must target foundational model and data interventions rather than relying only on prompt-level safeguards, because the observed associations emerge in naturalistic as well as adversarial settings (Kumar et al., 24 Sep 2025).

The cross-cultural design is central to the benchmark’s significance. The stronger directional intensity in the USA context is interpreted in the paper as likely reflecting model pretraining data and development context, which in turn is described as revealing Western data dominance. India’s mixed-polarity pattern is presented as analytically different: rather than a single-direction skew, the benchmark uncovers concentration of both positive and negative associations around BJP, with other parties and leaders also attracting strong negative content (Kumar et al., 24 Sep 2025).

At the same time, the benchmark defines “ground truth” operationally through model consistency rather than through human annotation. This means that the dataset is designed to measure model behavior, not external factual correctness or justified plausibility. A plausible implication is that NeutQA-440 is best understood as a behavioral auditing instrument for representational asymmetry, not as a truth-conditional benchmark.

Its broader significance lies in standardization. By combining a fixed taxonomy, paired prompt families, repeated querying, refusal tracking, and country-level comparison, the benchmark provides a replicable mechanism for auditing political associations in LLMs. The paper’s framing suggests that this form of evaluation is intended for ongoing monitoring of sociotechnical risk as models are deployed across political cultures (Kumar et al., 24 Sep 2025).