OvertonScore: Benchmarking Pluralistic Alignment in LLMs

• OvertonScore is a normalized metric that quantifies how well LLM outputs capture the full spectrum of legitimate public viewpoints on subjective queries.
• It operationalizes pluralistic alignment by using human clustering of free-form responses and a rating threshold to determine viewpoint coverage.
• Automated LLM-as-Judge protocols corroborate human evaluations, highlighting both advances and limitations in achieving comprehensive pluralistic alignment.

OvertonScore is a quantitative metric formulated to measure Overton pluralism in LLMs—specifically, the extent to which model outputs represent the full spectrum of reasonable viewpoints on subjective queries within a population. By operationalizing pluralistic alignment as set coverage over the “Overton window” (the set of legitimate public perspectives), OvertonScore enables principled benchmarking of a model’s capacity to surface multiple legitimate perspectives simultaneously, rather than collapsing output to a single normative or majority-biased summary. The framework was introduced by Poole-Dayan et al. in "Benchmarking Overton Pluralism in LLMs" (Poole-Dayan et al., 1 Dec 2025).

1. Formal Definition and Rationale

OvertonScore (OS) is formalized as a normalized set‐coverage metric. Consider a set of subjective queries $X=\{x_1,\dots,x_n\}$, where for each query $x$, the Overton window $W(x)=\{y_1,\dots,y_{K_x}\}$ enumerates all distinct “reasonable” answers that members of the population legitimately hold. For a given model output $\mathcal M(x)$, coverage of a viewpoint $y\in W(x)$ is defined by the majority sentiment among holders of that view: a viewpoint is “covered” if the cluster of humans associated with $y$ assigns the model’s answer a mean representation rating ≥ 4 (on a 1–5 scale).

The indicator function for coverage is: $\mathbbm{1}\{y \in \mathcal M(x)\} = \begin{cases} 1 & \text{if the cluster mean rating} \ge 4, \ 0 & \text{otherwise.} \end{cases}$ The Overton coverage for question $x$ is: $\OC(\mathcal M,x) =\frac{1}{|W(x)|} \sum_{y\in W(x)} \mathbbm{1}\{y \in \mathcal M(x)\}$ OvertonScore over the full benchmark is the arithmetic mean: $\OS(\mathcal M,X) = \frac{1}{n} \sum_{i=1}^{n} \OC(\mathcal M, x_i)$ The maximum theoretical score is 1.0, indicating coverage of every viewpoint in every Overton window. To account for viewpoint prevalence, a weighted variant $\OS_W$ aggregates coverage by empirical prevalence $p(y)$ of each viewpoint in the human sample.

Set coverage is chosen for its conceptual fidelity to pluralistic alignment as defined by Sorensen et al.—equal treatment of each legitimate viewpoint. The weighted variant pragmatically down-weights extremely rare perspectives.

2. Human Benchmarking Methodology

Poole-Dayan et al. operationalize OvertonScore via large-scale human studies. For each question in a 60-item benchmark (15 political, 45 guided by the PRISM framework), 1,209 U.S.-representative participants:

• Provide free-form answers (75–300 characters) expressing their viewpoint.
• Rate responses from 8 LLMs (GPT-4.1, o4-mini, Gemma 3-27B, DeepSeek R1/V3, Llama 3.3-70B Instruct, Llama 4 Maverick, Claude 3.7 Sonnet) on a Likert scale (“To what extent is your perspective represented?”).
• Vote (Agree/Neutral/Disagree) on at least 10 other participant answers.

Viewpoints are recovered by clustering pairwise agreement–disagreement votes through a modified Pol.is algorithm: a k-means variant optimizing silhouette score, handling missing data natively, and adjusting distances for voters with sparse participation. Clusters correspond to human-attested viewpoints, and cluster mean ratings determine coverage per the ≥4 threshold.

This workflow yields a human-derived Overton window for each query, enabling direct computation of OC and OS as above. To control for between-question variation, the methodology fits an OLS linear probability model: $\OC(\mathcal M, x) \sim 0 + \text{Model} + \text{Question (fixed effects)}$ with cluster-robust standard errors for adjusted coverage per model.

3. Benchmark Results and Model Comparisons

Across the benchmark, frontier LLMs achieve average adjusted OvertonScores far below the theoretical maximum:

• Mean adjusted unweighted $\OS \approx 0.39$
• Weighted $\OS_W \approx 0.48$

Performance varies markedly by model:

• DeepSeek V3: Highest pluralistic alignment (adj. $\OS=0.417$, adj. $\OS_W=0.530$)
• DeepSeek R1, Llama 3.3-70B, GPT-4.1: Next-best in coverage
• Gemma 3-27B: Substantially below average (adj. $\OS=0.350$, adj. $\OS_W=0.428$)

Even a “best-across-models” reference (selecting highest coverage per viewpoint from any system) reaches only $\OC=0.687$, $\OS_W=0.768$, indicating considerable aggregate gaps. Thus, the empirical state-of-the-art is $\OS\in[0.35,0.41]$, $\OS_W\in[0.44,0.53]$.

4. Automated OvertonScore and Model Development

To mitigate the impracticality of repeated large-scale human studies, the OvertonScore framework incorporates automated benchmarking via “LLM-as-Judge” protocols. In this approach, LLMs such as Gemini 2.5 Pro are employed to predict representation ratings based on participant free responses and a small set of example human ratings (“FS+FR” prompt engineering). Pilot studies indicate:

• FS+FR prompting outperforms semantic similarity and demographic baselines.
• Gemini 2.5 Pro achieves MAE = 0.66 Likert points, Spearman $\rho=0.66$ on individual ratings, and wins over baselines >50%.

On held-out data, leave-one-model-out analysis reveals strong rank correlation between LLM-predicted and human OvertonScores ($\rho=0.88$), and model coefficient correlation $r=0.90$, with absolute differences in adjusted OS < 0.1 for all but Claude 3.7 Sonnet. This suggests automated OS is an effective screening proxy, though not a full substitute for human assessment due to inherited biases.

5. Limitations and Context Dependence

Estimates of Overton windows and OS are inherently context-dependent. The benchmark deploys 60 English-language questions and U.S.-representative samples; extension to broader linguistic, cultural, or issue domains may shift window boundaries and coverage baselines. The coverage threshold (mean $\geq 4$) is robust to sensitivity analysis ($\tau\in[3.6,4.0]$), but subgroup fairness checks reveal minor demographic disparities and a detectable effect of political affiliation on LLM-as-judge accuracy.

Automated judges used in benchmarking inherit the base model’s implicit normative biases, necessitating ongoing validation and careful interpretation as proxies for pluralistic alignment.

6. Future Directions in Pluralistic Alignment

Suggested advances include:

• Post-training protocols explicitly optimizing OvertonScore via multi-objective RLHF or modular pluralism architectures.
• Participatory clustering with globally diverse populations to expand Overton windows.
• Dynamic weighting to balance prevalence of rare vs. majority viewpoints.
• Iterative developer loops combining automated OS screening with periodic targeted human studies.

A plausible implication is that OvertonScore concretizes the normative goal of pluralistic alignment as a reproducible, scalable benchmark, thus guiding future model development towards systematic coverage of legitimate perspectives on subjective questions. Continued refinement of measurement tools and extension to new contexts will shape the evolution of pluralism-oriented LLM evaluation (Poole-Dayan et al., 1 Dec 2025).