PoliBiasES: Political Bias in LLMs

Updated 2 July 2026

PoliBiasES is a suite of theory-grounded methodologies and benchmarks for measuring political bias in large language models, integrating political science and validated survey methods.
It employs rigorously pretested survey instruments like EVS/WVS and systematic prompt variations to reveal nuanced economic and cultural bias dimensions.
Through robust statistical analysis and fine-tuned stance classifiers, it establishes reproducible and empirical standards for auditing instruction-tuned and base models.

PoliBiasES

PoliBiasES refers to a suite of theory-grounded methodologies, tools, and benchmarks for measuring, interpreting, and auditing political bias in LLMs using principles from political science and survey methodology. Developed in response to the limitations of ad hoc or non-validated bias audits such as the popular Political Compass Test (PCT), PoliBiasES implements robust, multidimensional, and empirically validated instruments for analyzing political leaning and prompt sensitivity in LLM outputs, with particular attention to both economic and cultural axes of ideology (Faulborn et al., 20 Mar 2025).

1. Theoretical and Empirical Foundations

PoliBiasES is grounded in established political science theories concerning ideological consistency and multidimensionality. It operationalizes bias not as isolated responses to cherry-picked statements but as statistically significant, coherent patterns aligned along at least two orthogonal dimensions: the economic axis (state intervention vs. laissez-faire) and the cultural axis (libertarian vs. authoritarian) (Faulborn et al., 20 Mar 2025).

To ensure construct validity, PoliBiasES employs items from the European Values Study / World Values Survey (EVS/WVS): these surveys are rigorously pretested, balance sampling strategies, and employ psychometrically documented instruments. This approach avoids the deficiencies of popular-quiz-based audits, such as inclusion of loaded or irrelevant items found in PCT (e.g., astrology beliefs), and ensures that every evaluated survey item is mapped to explicit ideological axes by trained annotators (Cohen’s κ ≈ 0.77 for left/right, 0.76 for economic/cultural mapping).

2. Questionnaire Design and Prompt Robustness

The PoliBiasES framework incorporates 89 distinct policy statements, comprising 33 directly from EVS/WVS and a filtered, reworded subset of PCT items. Each statement appears in three versions: original, paraphrase, and a semantic “flip” (where agreeing signals the opposite ideological direction), with further robustness achieved via ten prompt prefixes—for instance, "please respond," "decide on truthfulness," and "emotion_happy."

Each statement × variant × prefix combination results in 30 unique prompt settings per item. Free-text and Likert (1–5) responses are obtained, with stance labels (agree/disagree/neutral) assigned via a BART-Large classifier fine-tuned on in-domain data (macro-F1 ≈ 0.93 at p ≥ 0.9). Only high-confidence responses (about 67% of all completions) contribute to final bias calculations (Faulborn et al., 20 Mar 2025).

3. Mathematical Bias Metrics and Aggregation

Directional bias for a model $m$ on the ideological direction $d$ (left/right) is formulated as:

$P_{agree,m,d} = \frac{A}{A+D+N}$

$Bias_{m,d} = P_{agree,m,d} - P_{disagree,m,d}$

$Bias_{m} = \frac{Bias_{left,m} - Bias_{right,m}}{2}$

where $A$ , $D$ , and $N$ denote the model’s agree/disagree/neutral counts on items labeled for direction $d$ . The final score $Bias_m \in [-1, +1]$ provides a continuous estimation of model leaning (negative = left, positive = right). Robust uncertainty estimates are produced via 10,000-sample bootstrap confidence intervals.

4. Empirical Results: Model Bias and Prompt Sensitivity

Empirical evaluation over 88,110 LLM responses (11 models, including both open-source and commercial) reveals:

Instruction-tuned models, such as GPT-3.5-turbo and Llama-instruct variants, consistently exhibit significant left-leaning bias across both economic and cultural axes.
No evaluated model showed overall right-leaning bias; GPT-4 models tend toward ideological centrism.
The PCT exaggerates left-leaning bias for GPT-3.5 and cultural bias for several other instruct models (max $d$ 0bias ~0.09), with PCT and WVS rankings showing only moderate correlation (Kendall $d$ 1 ≈ 0.6–0.7).
Prompt format exerts substantial influence: especially when using constrained Likert formats, swings up to $d$ 20.15 in bias scores compared to open-text variants are observed. Some "emotion" prefixes can steer GPT-3.5 further left, but other instruction-tuned models are more robust to prompt manipulation.
For base (non-instruct-tuned) models, classification between left/right can flip under different prompts, underlying the necessity of wide prompt sampling for bias audits.

5. Comparison with Previous Political Bias Audits

PoliBiasES advances over previous bias evaluation frameworks, notably those using the PCT. Unlike prior methods, it combines:

Aspect	PoliBiasES	PCT-only Approach
Theoretical basis	Political science, multidim.	None (popular quiz)
Survey instrument	Validated (EVS/WVS)	Ad hoc, non-documented
Content validity	Pretested, documented	Contains loaded/irrelevant
Response format	Open, multi-prompt, Likert	Mostly Likert, no variation
Sensitivity analysis	30 prompt variants	Single/few prompt forms
Stance detection	Fine-tuned classifier (F1 0.93)	Single-token/forced answer
Computation	88,000+ responses, bootstrapped	Small samples, no CI
Results stability	Uncertainty and prompt-robust	No robustness analysis
Validity, reliability	High, theory-anchored	Uncalibrated, less reliable

Key innovations include the use of scientifically validated statements, systematic sampling of prompt wordings, open-ended and Likert response capture, and classifier-based stance detection, all yielding a bias metric with documented construct validity, reliability, and statistical robustness (Faulborn et al., 20 Mar 2025).

6. Implications, Open Questions, and Extensions

Findings underscore the instability and prompt sensitivity of measured political bias, with instruction-tuning producing a reliably leftward shift. While open-ended prompting generally lessens both bias magnitude and prompt sensitivity, constrained-choice formats can amplify or reverse measured stances, particularly for base models. This has direct implications for the design and deployment of LLM-based tools where neutrality or multi-perspective synthesis is desired.

PoliBiasES’s empirical protocol and reporting templates set a new standard for reproducible, ecologically valid, and theory-anchored benchmarks of political bias in LLMs. Its modular structure enables extensions to other national contexts, languages, and ideological spaces, although all such deployments require rigorous adaptation of survey statements and annotation guidelines for local validity.

Future work should address: (i) the development of bias-mitigating training schemes to bring model distributions toward the empirical center under diverse prompt sets, (ii) the integration of multi-axial and non-Western ideological frameworks, and (iii) comparative evaluation alongside domain-specific downstream tasks such as news summarization, legislative analysis, and social media moderation.

7. Bibliographical Note and Selected References

The foundational development and evaluation of PoliBiasES are documented in "Only a Little to the Left: A Theory-grounded Measure of Political Bias in LLMs" (Faulborn et al., 20 Mar 2025). For further context on prompt sensitivity, ideological tuning, and cross-linguistic audits, see also related benchmarks and audits in (Kim et al., 11 Jun 2026, Elbouanani et al., 26 May 2025, Jakob et al., 18 Sep 2025), and (Löhr et al., 9 Oct 2025).