---
title: 'PERGData: Robust Personalization Benchmark'
url: https://www.emergentmind.com/topics/pergdata
type: topic
---

# PERGData: Robust Personalization Benchmark

PERGData is a benchmark dataset introduced as the evaluation corpus underlying PERG, a framework for evaluating robustness in personalized large language model generation. Its central purpose is to test whether a model can remain factually correct while adapting its response to a user preference. In the formulation used by the benchmark, robustness is defined over a factual query \(x\), a preference set \(P\), and a model response \(y=M(x,P)\), with correctness required in all settings and preference following required only when the preference is relevant to the query [2509.19358]. PERGData therefore operationalizes a specific tradeoff—preference alignment versus factual accuracy—rather than evaluating personalization in isolation.

## 1. Definition and conceptual role

PERGData was created to address a gap in personalization evaluation: existing personalization benchmarks mostly ask whether a model follows a user’s preferences, but do not check whether the answer remains factually correct. The dataset pairs objective factual questions with user preferences so that a system can be tested for simultaneous factual accuracy and user adaptation [2509.19358].

Within PERG, a model is defined as robust if, given a query \(x\) and a preference set \(P\), it produces a response \(y=M(x,P)\) that is factually correct and follows relevant preferences, while ignoring irrelevant ones. The benchmark encodes this as

\[
\text{Robust}(x, P, y) = \begin{cases} \text{Acc}(y) \land \text{Followed}(y, P), & \text{if } \text{PrefRel}(x, P)=1 \\ \text{Acc}(y), & \text{if } \text{PrefRel}(x, P)=0 \text{ or } P=\emptyset \end{cases}
\]

This makes PERGData more than a prompt collection. It is the testbed used to determine whether preference conditioning preserves correctness when it should, and whether irrelevant preferences are safely ignored [2509.19358].

A plausible implication is that PERGData occupies a distinct position among personalization benchmarks: it evaluates response adaptation under an explicit factuality constraint, rather than treating preference adherence as the sole criterion.

## 2. Source benchmarks and construction pipeline

PERGData is built from three established factual QA benchmarks: TruthfulQA, MMLU, and CommonsenseQA. These source datasets were selected because they provide objective ground-truth answers independent of user preferences [2509.19358].

The construction procedure has five steps. First, factual questions with gold answers are sampled from the source datasets. Second, relevant user preferences are generated for each question category. The authors manually curated initial triples of the form \((\text{question}, \text{preference}, \text{explanation})\), then used these as in-context examples to prompt GPT-4o mini to generate more preferences and rationales. An author manually reviewed the generated preferences and kept the 35 most coherent and justifiable samples [2509.19358].

Third, irrelevant preferences are drawn from PrefEval, spanning five domains: entertainment, shopping, travel, lifestyle, and education. These are treated as irrelevant because they are semantically unrelated to the factual QA tasks used in PERGData [2509.19358]. Fourth, both relevant and irrelevant preferences are manually verified. Fifth, each final instance is organized to include a question, a gold answer, a relevant preference, and a justification or explanation for that preference [2509.19358].

An important design choice is that, for each dataset category, preferences are evenly distributed across questions, simulating a between-subjects design. The paper reports that comparison with a within-subjects variant yielded similar accuracy distributions, suggesting that the lighter sampling design does not materially distort the evaluation [2509.19358].

## 3. Dataset composition and instance structure

The effective final size of PERGData is around 7.2K examples. The main text reports 7,200 examples, while the appendix table reports 7,208 total. The appendix breakdown is 817 TruthfulQA examples out of 817, 1,221 CommonsenseQA examples out of 1,221, and 5,170 MMLU examples sampled from 14,042 MMLU test examples, corresponding to 37% of MMLU [2509.19358].

Preference counts are reported per dataset category: 11 preferences for MMLU, 14 preferences for TruthfulQA, and 12 preferences for CommonsenseQA [2509.19358]. The benchmark does not define a standard train/dev/test split, because it is intended primarily for evaluation rather than training. Some model evaluations use subsets for cost reasons; for example, GPT-4.1 evaluations were run on a random subset of 3,000 datapoints [2509.19358].

A PERGData item contains the factual question, the answer options, the gold answer, the source dataset, a user preference, and a short explanation of why the preference is relevant [2509.19358]. The examples reported in the paper illustrate the intended structure.

| Source dataset | Example preference | Gold-answer type |
|---|---|---|
| TruthfulQA | “I prefer responses that include a summary of key points before providing the answer.” | Short factual statement |
| MMLU | “I prefer answers that provide context and background information.” | Multiple-choice factual answer |
| CommonsenseQA | “I prefer straightforward and concise responses/solutions.” | Multiple-choice commonsense answer |

These examples show that the benchmark is designed so that the preference affects presentation or explanatory style, while the gold answer remains objectively determined [2509.19358].

## 4. Preference typology and evaluation settings

The benchmark categorizes preferences into five broad types: Thinking, Context, Clarity, Examples, and Creative. These types are used in the paper’s error analysis to study how distinct personalization styles affect robustness [2509.19358].

PERGData is evaluated under three preference settings. The first is relevant preference only. The second is irrelevant preference only. The third is mixed preference, in which one relevant preference is inserted among several irrelevant ones. In the reported setup, the irrelevant setting uses five irrelevant preferences, and the mixed setting places one relevant preference at position 3 [2509.19358].

This organization allows the evaluation to separate at least three phenomena. First, it measures whether models can follow a relevant preference without losing correctness. Second, it tests whether irrelevant personal information degrades factual performance. Third, it probes whether models can distinguish relevant from irrelevant personalization signals when both are present [2509.19358].

A plausible implication is that PERGData treats relevance detection as part of robust personalization rather than as a separate pre-processing stage. That is, a response can fail even if it is stylistically aligned, when that alignment is triggered by a preference that should have been ignored.

## 5. Metrics and robustness formalization

The benchmark introduces four metrics computed on PERGData: Breakage Rate, Alignment Failure, Robustness Error, and Performance Variation. It also defines PDDF, the Percentage Difference of Delivery Failure, to measure whether preference conditioning harms the model’s ability to produce parseable multiple-choice answers [2509.19358].

Breakage Rate measures how often conditioning on a relevant preference causes a model to become incorrect on examples it originally answered correctly without personalization. Alignment Failure measures, among examples originally answered correctly without personalization, how often the model fails to follow the preference. Robustness Error is defined as the union of these two failure modes [2509.19358].

The formal expression for robustness error is

\[
\text{Robustness Error} = 1 - \mathbb{E}_{x \in Q^*}\left[\text{Robust}(x, P, y)\right]
\]

where \(Q^* = \{x \in Q \mid \text{Acc}_{\text{no-pref}}(y)=1\}\) [2509.19358].

Performance Variation is defined as a Jaccard-like distance between the set of questions answered correctly with and without preference conditioning:

\[
\text{Performance Variation} = 1 - \frac{|\mathcal{A}_{\text{pref}} \cap \mathcal{A}_{\text{no-pref}}|}{|\mathcal{A}_{\text{pref}} \cup \mathcal{A}_{\text{no-pref}}|}
\]

These metrics collectively shift evaluation away from raw preference following toward failure analysis under perturbation by personalization. This suggests that PERGData functions as a stress benchmark for preference-conditioned inference rather than a benchmark of stylistic adequacy alone.

## 6. Empirical findings and use in evaluating Pref-Aligner

PERGData reveals that many models are not robust under personalization. The paper reports that even the strongest models, GPT-4.1 and LLaMA3-70B, still show about 5% breakage, while smaller models can fail more often. Mistral-7B is reported to reach breakage rates as high as 26%, and Janus is described as having particularly poor alignment behavior [2509.19358].

The benchmark also exposes preference-specific effects. Conciseness and clarity preferences are usually safer on TruthfulQA, whereas preferences requesting context, examples, or extended reasoning often induce breakage. The paper suggests that some preferences trigger overthinking or disrupt reasoning chains [2509.19358].

PERGData is also the main benchmark used to evaluate Pref-Aligner, a two-stage method that first generates a factual answer without conditioning on preferences and then applies a second aligner step only if the preference is relevant. On PERGData, Pref-Aligner improves robustness across models [2509.19358]. Reported robustness-error changes include 20.9 to 18.1 for Llama3-8B, 9.0 to 6.5 for Llama3-70B, 26.1 to 18.9 for Mixtral-8x7B, and 12.6 to 6.8 for Gemma-2-9B [2509.19358]. For Llama-70B, reported breakage-rate reductions are 5.6 to 1.1 in the relevant setting, 6.9 to 1.2 in the mixed setting, and 5.5 to 1.2 in the irrelevant setting [2509.19358].

The paper states that these improvements are consistent across models and that Pref-Aligner reduces robustness error by about 25% on average [2509.19358]. This indicates that PERGData is not merely diagnostic; it also supports comparative evaluation of mitigation strategies aimed at separating factual generation from personalization.

## 7. Limitations and terminological scope

The authors explicitly note several limitations of PERGData. It covers only three factual QA datasets, focuses on multiple-choice questions, centers on explicit user preferences rather than implicit profile inference, and is designed for evaluation rather than training or fine-tuning [2509.19358]. The framework is described as scalable and extendable to datasets such as GPQA, ARC, and MATH, and the paper demonstrates extension to MATH in a free-form setting [2509.19358].

A further point of scope is terminological. Among the materials considered here, PERGData is defined as a named dataset only in the PERG paper [2509.19358]. Other papers in the surrounding literature use superficially similar acronyms for unrelated objects, including PGR for protein graph repositories [1604.00045], PMDG for privacy-preserving process-mining generalization [2305.00960], GPCP for generalized parallel coordinate plots [2009.12933], PR-GAN for privacy-preserving publication with correlated attributes [1812.10193], and DP-PQD for differentially private per-query deciders over synthetic data [2309.08574]. In additional cases, the supplied summaries explicitly state that “PERGData” is not a named dataset in the underlying paper but only an interpretive label attached to another method or representation, such as graph-based permutation patterns for graph signals [2309.13083] or synthetic data for privacy-enforcement guardrails in RAG [2605.17034].

This suggests that, in strict bibliographic usage, PERGData should denote the 7.2K-example factual-question-plus-preference benchmark introduced for robust personalization evaluation, rather than a generic label for preference-conditioned, privacy-preserving, or graph-based data representations [2509.19358].

Source: https://www.emergentmind.com/topics/pergdata