POPE: Evaluating Pluralistic LLM Policies

• POPE is an offline framework that evaluates LLM policies by combining human preference signals with entropy-inspired diversity to capture pluralism.
• It decomposes rewards into collaborative utility and a diversity term, ensuring balanced responses that avoid convergence on a single mode.
• The framework utilizes decomposed inverse propensity scoring for robust off-policy evaluation and direct policy optimization.

Searching arXiv for the POPE paper and closely related off-policy evaluation work. Pluralistic Off-Policy Evaluation (POPE) is an offline framework for evaluating and optimizing LLM policies against pluralistic human preferences using logged interaction data generated by other policies. It extends off-policy evaluation from utility-only objectives to a pluralistic value function that combines collaborative utility, derived from human preference signals such as upvotes or relevance scores, with a diversity component inspired by entropy-based coverage measures. The framework couples this reward design with decomposed inverse propensity scoring (IPS) estimators and a maximum-entropy lower bound, then uses the resulting off-policy value as a direct optimization objective for pluralistic alignment (Huang et al., 15 Sep 2025).

1. Definition and motivation

POPE is introduced to address a gap between conventional preference alignment methods and the empirical reality of heterogeneous human judgments. The motivating claim is that personalized preference alignment for LLMs requires methods that capture preferences that are diverse and multi-modal, whereas most existing preference alignment datasets are logged under policies that differ substantially from the evaluated models and most existing off-policy estimators focus solely on overall utility while ignoring preference pluralism. In this setting, pluralistic alignment denotes joint optimization for collaborative utility and pluralism or diversity, rather than approximation of a single dominant response mode (Huang et al., 15 Sep 2025).

The framework is positioned against standard RLHF-style alignment and reward-model pipelines that maximize expected utility across users. In the detailed exposition, this focus is said to induce concentration on a narrow set of “majority” or high-reward responses, to underserve minority opinions, niche styles, or long-tail but valid answers, and to reduce diversity in open-ended tasks. POPE instead targets broad coverage of the “Overton window of plausible, acceptable responses,” while preserving high average quality.

The off-policy aspect is central. Logged data typically come from older models, external vendors, or different sampling strategies, and human feedback is attached to those logged responses rather than to the current policy. The framework is therefore designed for offline use, where the target policy $\pi_\theta$ is evaluated and optimized from data collected under a logging policy $\pi_0$, without live deployment or fresh human annotation.

2. Formal problem setting

The formulation is an offline bandit-style setting with slate actions. Queries or contexts are $x_t \in \mathcal{X}$ for $t=1,\dots,n$. The response space is a finite set $\mathcal{S}$ of size $L = |\mathcal{S}|$, and for each query a policy produces a slate $S_t = \{a_{t,1}, \dots, a_{t,K}\}$ of $K \le L$ responses. Human feedback for each response in the slate is represented by a preference signal $\eta_{t,i}(x_t,a_{t,i})$, which can encode upvotes, helpfulness scores, relevance, or a ranking mapped into such signals (Huang et al., 15 Sep 2025).

The logging policy $\pi_0$ generates the observed data through

$\pi_0$0

while the target policy $\pi_0$1 is the LLM policy to be evaluated and optimized. The paper defines two levels of policy probability. At the response level,

$\pi_0$2

where $\pi_0$3 is the token distribution of the LLM and $\pi_0$4 is the response length. At the response-set level,

$\pi_0$5

These quantities provide the propensity terms used in the IPS estimators.

The target of evaluation is a pluralistic policy value over the logged queries. In the paper’s unified formulation,

$\pi_0$6

and

$\pi_0$7

This definition makes pluralistic alignment a value-estimation problem rather than solely a reward-modeling or preference-ranking problem.

3. Unified pluralistic reward

The collaborative utility component aggregates human preference signals across the slate: $\pi_0$8 This term is intended to represent the total helpfulness or quality of the slate as judged by the crowd. Because it sums over all responses in the slate, it does not force the policy toward a single answer instance but toward a set of responses that collectively reflect empirical human preferences (Huang et al., 15 Sep 2025).

The diversity component is defined through normalized response weights

$\pi_0$9

followed by the entropy-like term

$x_t \in \mathcal{X}$0

The paper notes that Shannon entropy would use $x_t \in \mathcal{X}$1, whereas this formulation uses $x_t \in \mathcal{X}$2; the detailed exposition states that, when embedded in the overall objective and lower bound, it effectively plays the role of an entropy regularizer. In the IPS diversity estimator, the per-response diversity reward is simplified to $x_t \in \mathcal{X}$3, directly mirroring entropy or energy terms in soft reinforcement learning.

A per-response reward is introduced in the lower-bound derivation: $x_t \in \mathcal{X}$4 This compact form makes explicit that pluralistic alignment is modeled as the sum of a human-signal utility term and an entropy-like diversity term.

The stated motivation for this design is that utility-only alignment maximizes $x_t \in \mathcal{X}$5, which oversamples the highest-rated responses and collapses diversity, whereas the diversity term penalizes low-entropy response distributions and encourages broader support over plausible candidates. This suggests that POPE formalizes pluralism as a balance between collaborative quality and coverage, not as a post hoc diversity heuristic.

4. Decomposable IPS estimators and theoretical guarantees

The observed dataset is

$x_t \in \mathcal{X}$6

with $x_t \in \mathcal{X}$7. POPE constructs separate IPS estimators for collaborative utility and diversity, rather than applying a single monolithic estimator to the entire reward (Huang et al., 15 Sep 2025).

For collaborative utility, the slate-level estimator is

$x_t \in \mathcal{X}$8

For diversity, the estimator is defined at the response level: $x_t \in \mathcal{X}$9

The combined estimator is

$t=1,\dots,n$0

The IPS properties rely on standard conditions: support, accurate logging of propensities, and independence of logged samples. The support condition is stated as

$t=1,\dots,n$1

with an analogous condition for slates. The exposition also requires consistent computation or approximation of $t=1,\dots,n$2 and $t=1,\dots,n$3, together with i.i.d. or conditionally independent logging assumptions.

The core theoretical statement is a maximum-entropy lower bound. For $t=1,\dots,n$4 responses sampled independently from $t=1,\dots,n$5, the paper proves

$t=1,\dots,n$6

The abstract characterizes the theoretical result as a proof that the decomposed IPS estimators establish a lower bound on their variance, while the detailed derivation presents a lower bound on a unified IPS objective via a maximum-entropy argument. The proof sketch uses an approximation of the slate ratio by a product of per-action ratios and repeated applications of Jensen’s inequality to move from product-and-sum expressions to sums of expectations.

The stated intuition is that direct slate-level IPS with a unified reward can have high variance because the weight $t=1,\dots,n$7 can be large in multi-response settings, whereas decomposing the estimation into a collaborative utility term and a per-response diversity term yields a simpler and more stable objective. A plausible implication is that POPE’s theoretical contribution is not merely an estimator decomposition but a variance-management strategy tailored to pluralistic response generation.

5. Off-policy optimization and alignment procedure

POPE does not stop at evaluation. The off-policy value estimate is used directly as an optimization target: $t=1,\dots,n$8 The framework therefore links offline evaluation and offline alignment within the same objective (Huang et al., 15 Sep 2025).

Using the lower-bound form, the paper derives the policy gradient

$t=1,\dots,n$9

The update rule is gradient ascent,

$\mathcal{S}$0

with learning rate $\mathcal{S}$1; the experiments use AdamW.

The paper contrasts this objective with RLHF and DPO. RLHF typically trains a reward model from human comparisons and then performs on-policy or near-on-policy policy optimization, such as PPO, to maximize expected scalar reward. DPO optimizes a contrastive objective based on pairwise preferences and raises the likelihood of preferred responses relative to dispreferred ones. Both are described as primarily utility-focused. POPE differs by working purely off-policy, directly using logged human signals without training a separate reward model, and explicitly inserting a diversity or entropy term into both the value estimator and the gradient.

This suggests that POPE belongs simultaneously to the OPE literature and to alignment-by-optimization, but its defining feature is the use of pluralistic reward design as the common interface between the two.

6. Empirical evaluation

The experimental study spans a controlled toy setting, in-domain review generation, and cross-domain transfer. The datasets are the Alpaca-GPT4 instruction-response dataset with 500 prompts and 3–5 synthesized pluralistic responses per prompt, Reddit Movie Reviews, Amazon Movies Reviews, Amazon Music Reviews, and Amazon Video Games Reviews. The evaluated models are instruction-finetuned variants of Llama 3 (1B, 3B), Phi-3.5-mini (3.8B), and Qwen 3 (4B). Baselines are Base, SFT, DPO, and POPE. The reported metrics include PL-Score, Pluralistic Coverage, Distributional Alignment, Helpfulness, Relevance, Distinct-1/2, off-policy value $\mathcal{S}$2, Self-BLEU, and AlignScore (Huang et al., 15 Sep 2025).

On the controlled Alpaca-GPT4 task, POPE is reported to improve PL-Score to $\mathcal{S}$3 versus a best baseline of approximately $\mathcal{S}$4, Pluralistic Coverage to $\mathcal{S}$5 versus approximately $\mathcal{S}$6, and Distributional Alignment to $\mathcal{S}$7 versus approximately $\mathcal{S}$8. The paper further states that SFT and DPO exhibit a trade-off in which higher relevance reduces diversity, whereas POPE’s Pareto envelope dominates and attains high PL-Score and high diversity simultaneously.

On Amazon Movies and Reddit, POPE is described as consistently yielding competitive or higher Helpfulness and Relevance than Base and DPO, higher Distinct-1/2, and higher off-policy value $\mathcal{S}$9. One concrete example given is Phi-3.5 on Amazon Movies, where $L = |\mathcal{S}|$0 is $L = |\mathcal{S}|$1 versus a baseline near $L = |\mathcal{S}|$2. On Amazon Music and Video Games, POPE is reported to achieve the highest $L = |\mathcal{S}|$3 across domains and model families, while improving Helpfulness, Relevance, and Distinct-1/2 in most cases. For Phi-3.5 on Amazon Music, the paper reports Helpfulness $L = |\mathcal{S}|$4 versus approximately $L = |\mathcal{S}|$5 for Base, Relevance $L = |\mathcal{S}|$6 versus approximately $L = |\mathcal{S}|$7, and Distinct-2 $L = |\mathcal{S}|$8 versus $L = |\mathcal{S}|$9.

Pluralistic Coverage gains are reported across Amazon Music, Video Games, Movies, and Reddit for both Llama3 and Phi-3.5, with improvements of $S_t = \{a_{t,1}, \dots, a_{t,K}\}$0–$S_t = \{a_{t,1}, \dots, a_{t,K}\}$1 over SFT and DPO. The auxiliary evaluation further states that POPE maintains or improves AlignScore while lowering Self-BLEU, indicating greater diversity without loss of factual alignment. A qualitative case study on a SpongeBob box set review is used to illustrate that a POPE-tuned Phi-3.5 model produces a review with episode counts, notable episodes, special features, and a fan-centered recommendation, whereas baseline models emphasize sparse factual lists or narrow packaging details.

The paper summarizes these results as evidence that POPE efficiently enhances pluralistic response generation and maintains the models’ general capabilities on downstream tasks.

7. Relation to adjacent OPE research, limitations, and open questions

POPE sits within the broader off-policy evaluation literature, but its object of evaluation is a pluralistic preference-alignment value rather than a scalar utility return. General OPE surveys organize the field around importance sampling, direct methods, doubly robust estimators, efficiency bounds, and horizon-dependent variance behavior in bandit, NMDP, TMDP, and MDP settings (Uehara et al., 2022). Against that background, POPE can be read as an extension of IPS-based reasoning to a reward that is explicitly multi-component and alignment-specific.

Two neighboring strands are especially relevant. Representation Balancing MDPs study off-policy policy evaluation by learning a model with a balanced representation, with emphasis on estimating both the individual policy value and average policy value accurately under distribution shift between behavior and evaluation policies (Liu et al., 2018). In ranking, Adaptive IPS addresses diverse user behavior through context-dependent interaction structures and proves unbiasedness and minimum variance within a class of unbiased IPS estimators based on the relevant subset of actions (Kiyohara et al., 2023). These works do not define POPE, but they establish nearby ideas: multi-context robustness in model-based OPPE and heterogeneity-aware IPS under diverse behavior. A plausible implication is that POPE inherits part of its intellectual setting from OPE methods that already distinguish aggregate performance from heterogeneous or structured forms of evaluation.

Empirical benchmarking work also provides context. The COBS study concludes that no single OPE estimator dominates across all environments and that performance depends strongly on horizon, mismatch, stochasticity, and representation; it highlights FQE, IH, and MAGIC-based hybrids as especially strong in different regimes (Voloshin et al., 2019). This does not directly evaluate POPE, but it underscores that estimator design in off-policy settings is highly regime-dependent, which is pertinent to any pluralistic evaluation framework.

The limitations stated for POPE are specific. It assumes support and access to or approximation of logging propensities; real systems may involve unknown or mixed logging policies. Its diversity reward uses $S_t = \{a_{t,1}, \dots, a_{t,K}\}$2, which the paper describes as a relatively simple entropy proxy, leaving semantic diversity and fairness-aware diversity outside the present formulation. The maximum-entropy lower-bound derivation uses a slate-independence approximation that may not hold exactly for complex LLM sampling strategies. Logged preferences may reflect biases in the populations that produced them. The experiments are limited to 1–4B parameter models and a small set of domains.

The open questions listed in the paper follow from these constraints: richer pluralistic rewards that incorporate group-specific preferences, fairness, and persona information; stronger diversity measures; combination with online bandit or reinforcement-learning schemes; extension to chain-of-thought, multi-turn dialogue, and agentic recommender systems; and theoretical analysis of bias-variance trade-offs for more complex decomposed estimators such as doubly robust pluralistic OPE. These directions suggest that POPE is best understood as a first explicit framework for offline pluralistic preference evaluation and alignment in LLMs, rather than as a complete account of pluralistic alignment under all offline logging conditions (Huang et al., 15 Sep 2025).