PERGData: Robust Personalization Benchmark

• PERGData is a benchmark dataset for evaluating robust personalization by testing whether models can produce factually correct answers that appropriately reflect relevant user preferences.
• It is constructed by combining factual questions from established datasets with both relevant and irrelevant user preferences, resulting in a curated set of approximately 7.2K examples.
• The benchmark uses specific metrics such as breakage rate, alignment failure, and robustness error to analyze how well models balance factual accuracy and preference alignment.

PERGData is a benchmark dataset introduced as the evaluation corpus underlying PERG, a framework for evaluating robustness in personalized LLM generation. Its central purpose is to test whether a model can remain factually correct while adapting its response to a user preference. In the formulation used by the benchmark, robustness is defined over a factual query $x$, a preference set $P$, and a model response $y=M(x,P)$, with correctness required in all settings and preference following required only when the preference is relevant to the query (Okite et al., 18 Sep 2025). PERGData therefore operationalizes a specific tradeoff—preference alignment versus factual accuracy—rather than evaluating personalization in isolation.

1. Definition and conceptual role

PERGData was created to address a gap in personalization evaluation: existing personalization benchmarks mostly ask whether a model follows a user’s preferences, but do not check whether the answer remains factually correct. The dataset pairs objective factual questions with user preferences so that a system can be tested for simultaneous factual accuracy and user adaptation (Okite et al., 18 Sep 2025).

Within PERG, a model is defined as robust if, given a query $x$ and a preference set $P$, it produces a response $y=M(x,P)$ that is factually correct and follows relevant preferences, while ignoring irrelevant ones. The benchmark encodes this as

$$\text{Robust}(x, P, y) = \begin{cases} \text{Acc}(y) \land \text{Followed}(y, P), & \text{if } \text{PrefRel}(x, P)=1 \ \text{Acc}(y), & \text{if } \text{PrefRel}(x, P)=0 \text{ or } P=\emptyset \end{cases}$$

This makes PERGData more than a prompt collection. It is the testbed used to determine whether preference conditioning preserves correctness when it should, and whether irrelevant preferences are safely ignored (Okite et al., 18 Sep 2025).

A plausible implication is that PERGData occupies a distinct position among personalization benchmarks: it evaluates response adaptation under an explicit factuality constraint, rather than treating preference adherence as the sole criterion.

2. Source benchmarks and construction pipeline

PERGData is built from three established factual QA benchmarks: TruthfulQA, MMLU, and CommonsenseQA. These source datasets were selected because they provide objective ground-truth answers independent of user preferences (Okite et al., 18 Sep 2025).

The construction procedure has five steps. First, factual questions with gold answers are sampled from the source datasets. Second, relevant user preferences are generated for each question category. The authors manually curated initial triples of the form $$(\text{question}, \text{preference}, \text{explanation})$$, then used these as in-context examples to prompt GPT-4o mini to generate more preferences and rationales. An author manually reviewed the generated preferences and kept the 35 most coherent and justifiable samples (Okite et al., 18 Sep 2025).

Third, irrelevant preferences are drawn from PrefEval, spanning five domains: entertainment, shopping, travel, lifestyle, and education. These are treated as irrelevant because they are semantically unrelated to the factual QA tasks used in PERGData (Okite et al., 18 Sep 2025). Fourth, both relevant and irrelevant preferences are manually verified. Fifth, each final instance is organized to include a question, a gold answer, a relevant preference, and a justification or explanation for that preference (Okite et al., 18 Sep 2025).

An important design choice is that, for each dataset category, preferences are evenly distributed across questions, simulating a between-subjects design. The paper reports that comparison with a within-subjects variant yielded similar accuracy distributions, suggesting that the lighter sampling design does not materially distort the evaluation (Okite et al., 18 Sep 2025).

3. Dataset composition and instance structure

The effective final size of PERGData is around 7.2K examples. The main text reports 7,200 examples, while the appendix table reports 7,208 total. The appendix breakdown is 817 TruthfulQA examples out of 817, 1,221 CommonsenseQA examples out of 1,221, and 5,170 MMLU examples sampled from 14,042 MMLU test examples, corresponding to 37% of MMLU (Okite et al., 18 Sep 2025).

Preference counts are reported per dataset category: 11 preferences for MMLU, 14 preferences for TruthfulQA, and 12 preferences for CommonsenseQA (Okite et al., 18 Sep 2025). The benchmark does not define a standard train/dev/test split, because it is intended primarily for evaluation rather than training. Some model evaluations use subsets for cost reasons; for example, GPT-4.1 evaluations were run on a random subset of 3,000 datapoints (Okite et al., 18 Sep 2025).

A PERGData item contains the factual question, the answer options, the gold answer, the source dataset, a user preference, and a short explanation of why the preference is relevant (Okite et al., 18 Sep 2025). The examples reported in the paper illustrate the intended structure.

Source dataset	Example preference	Gold-answer type
TruthfulQA	“I prefer responses that include a summary of key points before providing the answer.”	Short factual statement
MMLU	“I prefer answers that provide context and background information.”	Multiple-choice factual answer
CommonsenseQA	“I prefer straightforward and concise responses/solutions.”	Multiple-choice commonsense answer

These examples show that the benchmark is designed so that the preference affects presentation or explanatory style, while the gold answer remains objectively determined (Okite et al., 18 Sep 2025).

4. Preference typology and evaluation settings

The benchmark categorizes preferences into five broad types: Thinking, Context, Clarity, Examples, and Creative. These types are used in the paper’s error analysis to study how distinct personalization styles affect robustness (Okite et al., 18 Sep 2025).

PERGData is evaluated under three preference settings. The first is relevant preference only. The second is irrelevant preference only. The third is mixed preference, in which one relevant preference is inserted among several irrelevant ones. In the reported setup, the irrelevant setting uses five irrelevant preferences, and the mixed setting places one relevant preference at position 3 (Okite et al., 18 Sep 2025).

This organization allows the evaluation to separate at least three phenomena. First, it measures whether models can follow a relevant preference without losing correctness. Second, it tests whether irrelevant personal information degrades factual performance. Third, it probes whether models can distinguish relevant from irrelevant personalization signals when both are present (Okite et al., 18 Sep 2025).

A plausible implication is that PERGData treats relevance detection as part of robust personalization rather than as a separate pre-processing stage. That is, a response can fail even if it is stylistically aligned, when that alignment is triggered by a preference that should have been ignored.

5. Metrics and robustness formalization

The benchmark introduces four metrics computed on PERGData: Breakage Rate, Alignment Failure, Robustness Error, and Performance Variation. It also defines PDDF, the Percentage Difference of Delivery Failure, to measure whether preference conditioning harms the model’s ability to produce parseable multiple-choice answers (Okite et al., 18 Sep 2025).

Breakage Rate measures how often conditioning on a relevant preference causes a model to become incorrect on examples it originally answered correctly without personalization. Alignment Failure measures, among examples originally answered correctly without personalization, how often the model fails to follow the preference. Robustness Error is defined as the union of these two failure modes (Okite et al., 18 Sep 2025).

The formal expression for robustness error is

$$\text{Robustness Error} = 1 - \mathbb{E}_{x \in Q^*}\left[\text{Robust}(x, P, y)\right]$$

where $$Q^* = \{x \in Q \mid \text{Acc}_{\text{no-pref}}(y)=1\}$$ (Okite et al., 18 Sep 2025).

Performance Variation is defined as a Jaccard-like distance between the set of questions answered correctly with and without preference conditioning:

$P$0

These metrics collectively shift evaluation away from raw preference following toward failure analysis under perturbation by personalization. This suggests that PERGData functions as a stress benchmark for preference-conditioned inference rather than a benchmark of stylistic adequacy alone.

6. Empirical findings and use in evaluating Pref-Aligner

PERGData reveals that many models are not robust under personalization. The paper reports that even the strongest models, GPT-4.1 and LLaMA3-70B, still show about 5% breakage, while smaller models can fail more often. Mistral-7B is reported to reach breakage rates as high as 26%, and Janus is described as having particularly poor alignment behavior (Okite et al., 18 Sep 2025).

The benchmark also exposes preference-specific effects. Conciseness and clarity preferences are usually safer on TruthfulQA, whereas preferences requesting context, examples, or extended reasoning often induce breakage. The paper suggests that some preferences trigger overthinking or disrupt reasoning chains (Okite et al., 18 Sep 2025).

PERGData is also the main benchmark used to evaluate Pref-Aligner, a two-stage method that first generates a factual answer without conditioning on preferences and then applies a second aligner step only if the preference is relevant. On PERGData, Pref-Aligner improves robustness across models (Okite et al., 18 Sep 2025). Reported robustness-error changes include 20.9 to 18.1 for Llama3-8B, 9.0 to 6.5 for Llama3-70B, 26.1 to 18.9 for Mixtral-8x7B, and 12.6 to 6.8 for Gemma-2-9B (Okite et al., 18 Sep 2025). For Llama-70B, reported breakage-rate reductions are 5.6 to 1.1 in the relevant setting, 6.9 to 1.2 in the mixed setting, and 5.5 to 1.2 in the irrelevant setting (Okite et al., 18 Sep 2025).

The paper states that these improvements are consistent across models and that Pref-Aligner reduces robustness error by about 25% on average (Okite et al., 18 Sep 2025). This indicates that PERGData is not merely diagnostic; it also supports comparative evaluation of mitigation strategies aimed at separating factual generation from personalization.

7. Limitations and terminological scope

The authors explicitly note several limitations of PERGData. It covers only three factual QA datasets, focuses on multiple-choice questions, centers on explicit user preferences rather than implicit profile inference, and is designed for evaluation rather than training or fine-tuning (Okite et al., 18 Sep 2025). The framework is described as scalable and extendable to datasets such as GPQA, ARC, and MATH, and the paper demonstrates extension to MATH in a free-form setting (Okite et al., 18 Sep 2025).

A further point of scope is terminological. Among the materials considered here, PERGData is defined as a named dataset only in the PERG paper (Okite et al., 18 Sep 2025). Other papers in the surrounding literature use superficially similar acronyms for unrelated objects, including PGR for protein graph repositories (Dhifli et al., 2016), PMDG for privacy-preserving process-mining generalization (Hildebrant et al., 2023), GPCP for generalized parallel coordinate plots (Ge et al., 2020), PR-GAN for privacy-preserving publication with correlated attributes (Rezaei et al., 2018), and DP-PQD for differentially private per-query deciders over synthetic data (Patwa et al., 2023). In additional cases, the supplied summaries explicitly state that “PERGData” is not a named dataset in the underlying paper but only an interpretive label attached to another method or representation, such as graph-based permutation patterns for graph signals (Fabila-Carrasco et al., 2023) or synthetic data for privacy-enforcement guardrails in RAG (Zafar et al., 16 May 2026).

This suggests that, in strict bibliographic usage, PERGData should denote the 7.2K-example factual-question-plus-preference benchmark introduced for robust personalization evaluation, rather than a generic label for preference-conditioned, privacy-preserving, or graph-based data representations (Okite et al., 18 Sep 2025).