Populism Meets AI: Advancing Populism Research with LLMs (2510.07458v2)

Abstract: Measuring the ideational content of populism remains a challenge. Traditional strategies based on textual analysis have been critical for building the field's foundations and providing a valid, objective indicator of populist framing. Yet these approaches are costly, time consuming, and difficult to scale across languages, contexts, and large corpora. Here we present the results from a rubric and anchor guided chain of thought (CoT) prompting approach that mirrors human coder training. By leveraging the Global Populism Database (GPD), a comprehensive dataset of global leaders' speeches annotated for degrees of populism, we replicate the process used to train human coders by prompting the LLM with an adapted version of the same documentation to guide the model's reasoning. We then test multiple proprietary and open weight models by replicating scores in the GPD. Our findings reveal that this domain specific prompting strategy enables the LLM to achieve classification accuracy on par with expert human coders, demonstrating its ability to navigate the nuanced, context sensitive aspects of populism.

• The paper introduces a chain-of-thought prompting strategy that replicates expert holistic grading of populism using advanced LLMs.
• It rigorously evaluates multiple LLM architectures against human coders using metrics like Pearson’s r, ICC, and CCC, achieving near-human reliability.
• The study shows that reasoning-tuned models such as GPT-5 and Qwen3 offer scalable, transparent insights into political discourse, despite conservative scoring tendencies.

Populism Measurement via LLMs: Holistic Grading Replication and Model Evaluation

Introduction

This paper addresses the persistent challenge of measuring the ideational content of populism, a latent and context-dependent construct central to comparative political science. Traditional approaches, notably Holistic Grading (HG), require expert coders to read entire political speeches and assign scores based on nuanced rhetorical features. While HG is the benchmark for validity and reliability, its scalability is limited by resource constraints, especially in multilingual and cross-national contexts. The authors propose a rubric- and anchor-guided chain-of-thought (CoT) prompting strategy for LLMs, aiming to replicate the human coder training process and evaluate whether LLMs can achieve human-level accuracy in populism classification.

Methodological Framework

The paper adapts the HG methodology, leveraging the Global Populism Database (GPD) and its anchor speeches to train LLMs. The training set is expanded to address scale coverage gaps, ensuring representation across the full 0–2 populism intensity continuum. The CoT prompting architecture is designed to mirror human coder reasoning, comprising five sequential stages: theoretical foundation, methodological instruction, rubric training, anchor speech training, and implementation instructions. This explicit modeling of the reasoning process is critical for domain-specific, latent construct evaluation, as it compels the LLM to articulate step-by-step justifications for each score.

The evaluation pipeline includes multiple proprietary and open-weight LLMs, spanning diverse architectures (dense, MoE-standard, MoE-reasoning) and governance models (closed vs. open-source). Models are tested on 12 speeches from the UK, Turkey, and Montenegro, with five independent runs per model to assess test-retest reliability. Performance is quantified using linear and rank associations (Pearson’s $r$, Spearman’s $\rho$), intraclass correlation (ICC), concordance correlation (CCC), and Krippendorff’s $\alpha$ for pooled reliability.

Empirical Results

Human–LLM Agreement

The top reasoning models—GPT-5 (reasoning high) and Qwen3 235B (reasoning)—demonstrate near-human agreement across all metrics. GPT-5 achieves CCC = 0.947, ICC = 0.951, and Pearson $r$ = 0.974, while Qwen3 attains CCC = 0.954, ICC = 0.958, and Pearson $r$ = 0.961. Krippendorff’s $\alpha$ for pooled reliability exceeds 0.95 for both, well above the threshold for reliable data. Lower-capacity models (GPT-oss 20B, Llama 4 Scout) exhibit substantially weaker agreement and calibration, with errors large enough to undermine their utility for content analysis.

Figure 1: LLM and Human Holistic Grading in the United Kingdom.

Figure 2: LLM and Human Holistic Grading in Türkiye.

Figure 3: LLM and Human Holistic Grading in Montenegro.

Error and Calibration Diagnostics

Error analysis reveals that Qwen3 (reasoning) and GPT-5 (reasoning) have low mean absolute errors (MAE ≈ 0.13–0.15) and root mean square errors (RMSE ≈ 0.22–0.23), with modest negative bias indicating conservative scoring. Calibration regressions show slopes close to 1 and intercepts near zero, confirming minimal scale distortion and level bias. Bland–Altman plots for these models indicate stable error variance across the populism scale, with most disagreements within half a point.

Figure 4: Bland–Altman plots (best runs): GPT-5 Reasoning (run 3), Qwen3 235B Reasoning (run 2), and Llama 4 Scout Standard (run 3).

In contrast, weaker models display larger errors (MAE > 0.3, RMSE > 0.5), severe negative bias, and poor calibration, with wide limits of agreement and inconsistent error patterns. These findings underscore the necessity of high-capacity, reasoning-tuned architectures for reliable automated holistic grading.

Theoretical and Practical Implications

The results substantiate that LLMs, when prompted with structured, domain-specific reasoning frameworks, can replicate the integrated judgments of expert human coders in measuring populism. The chain-of-thought design not only enhances transparency and auditability but also provides a scalable solution for cross-linguistic and cross-national research. The demonstrated interchangeability of top open-weight and proprietary models (Qwen3 235B vs. GPT-5) has significant implications for reproducibility, cost, and accessibility in computational social science.

However, systematic scale compression and conservative scoring tendencies are observed, with LLMs reluctant to assign extreme values. This pattern is consistent across top models and may reflect an inherent cautiousness in automated grading. The paper also highlights the limitations of prompt-heavy setups, including potential context-rot in long prompts and the variable efficacy of CoT prompting across model classes and subjective tasks.

Future Directions

The authors propose expanding the framework to additional countries, languages, and speech genres, releasing prompts, code, and benchmark sets for community evaluation and refinement. The methodology is adaptable to adjacent constructs such as pluralist framing, nationalist boundary work, and crisis rhetoric, facilitating cumulative research on political discourse. Integration with retrieval-augmented generation (RAG) is suggested to further ground model reasoning and mitigate prior-driven drift.

Conclusion

This paper demonstrates that advanced LLMs, when guided by explicit holistic grading protocols and chain-of-thought reasoning, can achieve human-level reliability in the measurement of populist discourse. The approach bridges computational efficiency with conceptual rigor, enabling scalable, transparent, and reproducible content analysis in political science. Model selection and prompt engineering are critical, with only the most advanced reasoning systems meeting the precision requirements for valid measurement. The findings lay the groundwork for broader adoption of AI-based methodologies in the paper of latent, context-dependent political constructs.