---
title: Preference Learning Using Summarization (PLUS)
url: https://www.emergentmind.com/topics/preference-learning-using-summarization-plus
type: topic
---

# Preference Learning Using Summarization (PLUS)

Preference Learning Using Summarization (PLUS) denotes a family of methods in which summarization is used as the substrate for learning preferences, usually through comparisons, feedback, or preference-conditioned generation rather than through reference matching alone. In one explicit formulation, PLUS is a framework for pluralistic, personalized preference learning in which a model learns text-based summaries of each user’s preferences, characteristics, and past conversations and conditions a reward model on those summaries [2507.13579]. Taken together, the surrounding literature suggests a broader research program: summaries, summary pairs, concept selections, refinement trajectories, and user summaries can all serve as carriers of preference information, and those signals can be converted into ranking functions, reward models, or policy updates for downstream summarization [2311.04919].

## 1. Historical emergence and research scope

Early work in this area replaced reference summaries with interactive preference signals. APRIL, instantiated for extractive multi-document summarization, learned from user preferences between candidate summaries and decomposed the problem into **Active Preference Learning (APL)** and **Reinforcement Learning (RL)**, explicitly targeting lower sample complexity than earlier preference-based interactive methods [1808.09658]. “Adaptive Summaries” moved the interaction from whole-summary comparisons to concept-level feedback, allowing a user to **accept** or **reject** a concept, assign a **weight**, and provide a **confidence** value, with the learned preference represented by updated concept weights inside an iterative ILP-based loop [2012.13387]. Subsequent personalized systems such as **Summation** and **SumRecom** framed summarization as user-adaptive concept ranking and policy optimization over summaries or hierarchical concept maps [2307.05696], [2408.07294].

A second line of work brought preference learning into neural summarization and RLHF. “Make The Most of Prior Data: A Solution for Interactive Text Summarization with Preference Feedback” proposed an interactive framework with a pretrained extractive summarizer, a reward model called **ROMSR**, **PPO-based reinforcement learning**, and selective reuse of offline data in active, online, and few-shot settings [2204.05512]. “The Impact of Preference Agreement in Reinforcement Learning from Human Feedback: A Case Study in Summarization” studied the standard RLHF summarization setup and asked which kinds of human comparison data are most useful for learning reward models and improving downstream summarization [2311.04919].

The explicit framework named **Preference Learning Using Summarization (PLUS)** appears in “Learning Pluralistic User Preferences through Reinforcement Learning Fine-tuned Summaries,” where the central claim is that standard RLHF is too coarse for users with conflicting preferences and that user-specific textual summaries can act as compact, human-readable latent representations for personalized reward modeling [2507.13579]. More recent work extends the same logic to automated factuality supervision, clinical hallucination reduction, and structured personalization over review corpora [2605.26840], [2605.28910], [2605.05911]. This scope makes PLUS both a specific architecture and a broader methodological pattern in which summarization is used to elicit, compress, or operationalize preferences.

## 2. Formal problem formulations

In RLHF summarization, PLUS often begins as a **pairwise preference-learning problem**. Given a context \(x\) and two candidate summaries \(y_w\) and \(y_l\), a reward model outputs a scalar score \(r_\theta(x,y)\) and is trained so that the preferred summary receives the higher score. A standard objective is
\[
\mathcal{L}(\theta) = - \mathbb{E}_{(x,y_w,y_l)\sim D} \left[\log \sigma\left(r_\theta(x,y_w)-r_\theta(x,y_l)\right)\right],
\]
where the score difference is interpreted as the log-odds that humans will prefer one summary over the other [2311.04919]. In that setting, the reward model is then used inside an RLHF loop to fine-tune a summarization policy with constrained policy updates.

The personalized PLUS framework generalizes the classical Bradley–Terry–Luce model by conditioning on a user representation. Standard RLHF models a single population-level preference function, whereas PLUS learns a summarizer \(\pi_\theta\) that maps user context \(c\) to a short natural-language summary \(z=\pi_\theta(c)\), and a summary-conditioned reward model \(r_\phi(s\mid z)\) that predicts preferences given that summary [2507.13579]. The comparison probability becomes
\[
p(\mathbbm{1}\{s_A > s_B\}=1 \mid s_A,s_B,z) =
\frac{\exp r_\phi(s_A \mid z)}{\exp r_\phi(s_A \mid z)+\exp r_\phi(s_B \mid z)}
=
\sigma\!\big(r_\phi(s_A\mid z)-r_\phi(s_B\mid z)\big).
\]
Here the summary is not trained for reconstruction; it is trained to extract exactly those aspects of the user history that are useful for predicting future preferences.

Other PLUS variants replace human labels with automatically derived pairwise supervision. In factual consistency optimization, the method generates two summaries from the same source document, scores both with multiple factuality metrics, converts score differences into binary preferences, filters out pairs with metric disagreement, and trains the summarizer with **Direct Preference Optimization (DPO)**:
\[
L(\theta)= \mathbb{E}_{(\mathbf{x},\mathbf{y}_{\{w,l\})} \left[ \log \sigma\left(\beta\left(f_{\theta}(\mathbf{x},\mathbf{y}_w)-f_{\theta}(\mathbf{x},\mathbf{y}_l)\right)\right) \right].
\]
This preserves the pairwise structure while avoiding a separate reward model [2605.26840].

Concept-based personalized summarization adopts related, but not identical, formalisms. Adaptive Summaries maximizes user-preferred concept content under a length budget with an ILP objective, while Summation and SumRecom formulate summary construction as an episodic MDP in which actions add sentences or concepts and terminal rewards reflect learned user importance or reward estimates [2012.13387], [2307.05696], [2408.07294]. The common structural feature is that preferences are first transformed into a reusable scoring object—concept weights, a ranking function, or a reward model—and only then used for summary generation.

## 3. Sources of preference information

Human pairwise summary comparisons remain the canonical supervision source. In the RLHF case study on summarization, the preference dataset from Stiennon et al. (2020) contains **64,832 human judgments** over TL;DR Reddit summaries, with repeated annotations for some pairs so that annotator agreement can be measured [2311.04919]. The paper’s key insight is that **agreement among annotators is not just noise; it reflects the underlying quality differential between the two summaries being compared**. High-agreement examples teach obvious, high-separation quality cues; medium-agreement examples force the model to learn subtler distinctions; low-agreement examples can be ambiguous or noisy. This makes agreement itself a preference feature, not merely a data-cleaning signal.

Personalized PLUS replaces opaque user embeddings with textual summaries. The user context \(c\) can be a set of past preference pairs, but the framework is explicitly flexible enough to learn from past preference pairs, self-stated preferences, survey answers, or attribute lists [2507.13579]. The intended representation is short, editable, and human-readable. The qualitative distinction emphasized in the paper is between untrained summaries that mention superficial style or generic traits and RL-tuned summaries that encode the true preference dimension, such as whether a user wants information about cats rather than dogs. In this setting, the summary is itself a learned representation of preference.

Interactive concept-based methods expose preference structure more directly. Adaptive Summaries asks users to **accept** or **reject** concepts and to specify both importance and confidence; a rejected sentence sets the sentence weight to zero but does **not necessarily update all concepts inside that sentence**, because sentence rejection may reflect redundancy rather than concept irrelevance [2012.13387]. SumRecom similarly learns concept rankings from lightweight pairwise preferences over concepts, then uses those rankings to generate and optimize personalized summaries [2408.07294]. These approaches treat preference learning as concept selection before sentence or summary selection.

Automated supervision extends PLUS beyond explicit human judgments. In factual consistency optimization, **SBERTScore** and **SummaC-Conv** act as multiple weak annotators; a pair is retained only if all selected metrics agree on the preference direction [2605.26840]. In clinical summarization, detector-guided refinement produces trajectories \(y^{(0)}, y^{(1)}, \dots, y^{(T)}\), and later, detector-corrected summaries are treated as preferred over earlier, less faithful versions, thereby forming preference pairs without human annotation [2605.28910]. In personalized review summarization, feedback is scalar rather than pairwise, but it is still mapped into an online preference estimate over latent aspects and updated through entropic online mirror descent [2605.05911]. Across these variants, PLUS broadens the meaning of “preference data” to include disagreement structure, user-authored text, concept choices, metric consensus, and refinement-based factual corrections.

## 4. Learning architectures and optimization strategies

Reward modeling remains central in neural PLUS systems. In the RLHF agreement study, the reward model is a **T5-XXL** model with about **13B parameters**, **24 transformer blocks**, and **64 attention heads**, trained as a binary preference classifier over pairs of summaries; downstream RLHF fine-tunes a **T5-small** summarization policy and constrains policy updates to avoid overly large steps [2311.04919]. In personalized PLUS, the main models use **Qwen2.5-0.5B** as the reward model and **Qwen2.5-3B-Instruct** as the summarizer, both fine-tuned with **LoRA** [2507.13579]. The summarizer receives RL reward from the negative reward-model loss and is updated with **PPO**, while the reward model is updated by gradient descent on preference loss.

This online co-adaptation loop is a defining architectural choice. For a batch \((s'_A, s'_B, c)\), personalized PLUS generates a summary \(z=\pi_\theta(c)\), computes the reward-model loss
\[
\mathcal L(s'_A,s'_B,z) = -\log \sigma\!\big(r_\phi(s'_A\mid z)-r_\phi(s'_B\mid z)\big),
\]
uses the negative of that loss as the RL reward for the summarizer, updates \(\phi\) by gradient descent, and updates \(\theta\) using **PPO** [2507.13579]. The appendix reports a summarizer learning rate of \(5\times10^{-7}\), a critic learning rate of \(9\times10^{-6}\), and a KL penalty of 0.01 generally, or 0.001 for Pets. The paper explicitly motivates PPO as a stable choice because the summarizer’s reward is non-stationary.

Several alternatives bypass reward-model training. Automated factuality PLUS uses lexically similar summary pairs and DPO directly, arguing that careful pair construction and metric-agreement filtering create a high-quality preference dataset from source documents alone [2605.26840]. Clinical summarization follows the same preference-optimization pattern after **Hallucination Detection guided Self-Refinement (HDSR)** produces ordered refinement trajectories; **HDSR-PL** then fine-tunes the model with DPO so that factual corrections are amortized into the model parameters [2605.28910].

Other strands organize learning in stages rather than via online co-adaptation. **AlignSum** builds a **Data Pyramid** of extractive, abstractive, and human-annotated summary data, applies **Gaussian Resampling** to align pseudo-summary lengths with human-annotated summary lengths, and performs two-stage **Hierarchical Fine-Tuning (HFT)**: generic fine-tuning on extractive plus abstractive data, followed by personalized fine-tuning on human-annotated data [2410.00409]. The paper explicitly contrasts this with naive hybrid fine-tuning, arguing that scarce human-preference data can otherwise be overwhelmed by larger generic corpora.

Interactive extractive methods use older, but still structurally relevant, optimization stacks. APRIL learns a linear Bradley–Terry ranker from queried preferences and then uses that learned ranking as a reward signal for RL, with **TD(\(\lambda\))** and **LSTD(\(\lambda\))** evaluated in the policy phase [1808.09658]. “Make The Most of Prior Data” uses an extractive **BERTSUM** backbone, the **ROMSR** reward model, **PPO** fine-tuning, and selective offline replay through **Low-Reward Sampling (LRS)** and **Document-Similarity Sampling (DSS)** [2204.05512]. These systems differ in architecture but share the same decomposition: learn a reward-like object from preference information, then optimize summary generation against it.

## 5. Empirical patterns and benchmark results

The empirical literature repeatedly shows that the composition of preference data changes both predictive accuracy and the notion of quality that a model learns. In the RLHF agreement study, the authors construct four training sets of **2,000 comparisons each**—**MAX**, **MIN**, **DIST**, and **RAND**—from instances with repeated annotations and evaluate on a held-out test set of **1,267 instances** [2311.04919]. The main result is that **DIST** improves faster during training, reaches the **highest test accuracy**, and outperforms **MAX**, **MIN**, and **RAND**. **MIN** is consistently worst and learns the slowest. On **SummEval**, reward models trained on more diverse agreement distributions capture multiple dimensions of quality better, with **DIST** performing especially well on **coherence** and **relevance**. When used inside RLHF, **DIST** yields the best **ROUGE-1, ROUGE-2, and ROUGE-L before overfitting**, whereas **MIN** produces the worst downstream results and **MAX** does not consistently improve all ROUGE metrics despite strong reward-model accuracy.

The personalized PLUS framework reports gains across synthetic and real pluralistic datasets. On **Pets**, **Pets (OOD)**, **UltraFeedback-Personalized**, and **PRISM**, PLUS is evaluated against **BTL**, **DPL**, **VPL**, **ICL**, **PLUS-untrained**, and **Oracle**, with performance measured by preference prediction accuracy on held-out pairs [2507.13579]. The paper reports improvements as large as **126% on Pets**, **17% on UltraFeedback**, and **5% on PRISM**. Under the **Pets (OOD)** shift, where training uses cats and dogs but evaluation uses rabbits and birds, PLUS still achieves very high accuracy, **above 90%**, while **ICL** drops sharply and **VPL** performs poorly. The same paper also evaluates transfer to **GPT-4.1**, **GPT-4o**, and **GPT-4-turbo**: for **GPT-4o** on controversy-guided conversations, judge accuracy improves from **52.2 to 73.0**, and personalized generation win rate improves from **34% to 66%**.

Alignment-oriented preference fitting in conventional summarization also reports strong gains. AlignSum evaluates on the Element-Aware versions of **CNN/DailyMail** and **BBC XSum**, each with a human-annotated set of **200 samples**, split into **100 train and 100 test** [2410.00409]. In automatic evaluation, **BART-Large with full DP** achieves **ROUGE-1 48.83, ROUGE-2 24.11, ROUGE-L 34.16, BERTScore 0.9058** on CNN/DailyMail and **ROUGE-1 42.38, ROUGE-2 17.75, ROUGE-L 31.64, BERTScore 0.8962** on BBC XSum. In human evaluation against **GPT-3 with CoT prompting**, it achieves up to **65% Win** and **72% Win+Equal**. The ablation study identifies **HFT** as the most important component.

Automated factuality-oriented PLUS produces substantial improvements without human ranking. On **XSUM**, the factuality paper reports **AlignScore** gains from **61.9** to **86.6** for **BART**, from **59.7** to **75.8** for **GPT-J**, from **86.1** to **88.7** for **LLaMA**, and from **82.5** to **83.2** for **DeepSeek** [2605.26840]. On **TL;DR**, the corresponding gains are **84.9** to **94.2** for **BART**, **89.6** to **93.8** for **GPT-J**, **91.4** to **93.5** for **LLaMA**, and **89.1** to **90.9** for **DeepSeek**. The paper further reports that **at least 60% of generated pairs were retained** after metric-disagreement filtering and that using **SBERTScore + SummaC together** improves results over either metric alone.

Clinical summarization shows the same pattern under safety constraints. On **Hallucination-Generated-DI** derived from **MIMIC-IV-Note v2.2**, the main **LLaMA-3.1-8B-Instruct** comparison reports **29 hallucinations** for prompting, **57 hallucinations** for **SFT**, **22 hallucinations** for **HDSR (best; MedAlign)**, and **15 hallucinations** for **HDSR-PL (best; MedCat)** [2605.28910]. The paper highlights that **HDSR** reduces hallucinations by about **24%** relative to prompting, while **HDSR-PL** reduces hallucinations by about **48%** relative to prompting and about **74%** relative to **SFT**. Clinician ratings also improve: **HDSR (MedAlign)** reaches **Consistency 4.13, Coherence 4.48, Fluency 4.53, Relevance 3.95, Avg 4.27**, and **HDSR-PL (MedCat)** reaches **Consistency 4.40, Coherence 4.28, Fluency 4.05, Relevance 3.90, Avg 4.16**.

Interactive personalized summarization reports improvements of a different kind: user alignment and interaction efficiency. SumRecom reports **ROUGE-1/2/L** of **0.341/0.078/0.28** on **DUC1**, **0.372/0.083/0.333** on **DUC2**, and **0.382/0.094/0.301** on **DUC4**, outperforming APRIL and SPPI in those tables; its human study reports that **83%** of participants preferred the produced summary, average satisfaction was **8.2 / 10**, and response time was about **13 seconds** [2408.07294]. Summation reports **ROUGE-1 0.731, ROUGE-2 0.651, ROUGE-L 0.681** in its summary evaluation table and states that **86.67%** of workers were completely confident in answering topic questions after using the concept maps [2307.05696]. These results are not directly comparable to RLHF summarization benchmarks, but they reinforce the broader claim that preference-aware summarization can outperform static, one-size-fits-all systems within its own evaluation regime.

## 6. Interpretability, limitations, and research tensions

A recurring misconception in this area is that better preference learning always comes from cleaner or more unanimous labels. The agreement study directly disputes that view: maximizing agreement alone is **not always optimal**, because highly agreed-upon comparisons may be easy and reliable yet comparatively superficial for learning nuanced quality distinctions, while low-agreement comparisons can be too ambiguous if overused [2311.04919]. A second misconception is that one global reward model is sufficient for all users. The personalized PLUS framework argues that a single population-level preference function is systematically wrong when user preferences are multi-modal and conflicting [2507.13579].

Interpretability is one of the field’s most distinctive claims. Textual user summaries in personalized PLUS are described as **interpretable**, **portable**, and **editable**; the paper argues that they can be surfaced to users as a transparency mechanism and even transferred for zero-shot personalization of stronger proprietary models [2507.13579]. Concept-based methods make the same point in a different form: users can inspect the concepts that drive selection and can influence summary construction by comparing concepts rather than drafting summaries from scratch [2012.13387], [2408.07294]. This stands in explicit contrast to vector embeddings, which the personalized PLUS paper characterizes as opaque and difficult to inspect.

The limitations are substantial and differ by regime. Real heterogeneous human data remain difficult to model: **PRISM** is relatively small compared to the diversity it tries to capture, and even non-personalized baselines can sometimes do well by predicting majority preference [2507.13579]. AlignSum depends on access to human-preference data and is limited experimentally to **CNN/DailyMail** and **BBC XSum** because those were the datasets with rewritten versions reflecting the target preferences [2410.00409]. Automated factuality methods report a trade-off in which PLUS summaries are often more conservative and shorter, improving factuality while sacrificing some informativeness or stylistic richness [2605.26840]. Clinical systems operate in a safety-relevant domain where unsupported diagnoses, procedures, medications, times, or test results are not merely stylistic errors, and their evaluation remains partly resource-constrained [2605.28910]. Earlier interactive frameworks are often limited to extractive or multi-document settings and may require careful query selection to remain practical [1808.09658], [2204.05512].

Taken together, these results suggest that PLUS is best understood not as a single algorithm but as a design principle for alignment through summarization. Preferences can be elicited from humans, induced from disagreement patterns, inferred from detector-guided revisions, summarized into human-readable user models, or updated online from scalar feedback. The strongest common conclusion is not that summarization should replace other preference-learning modalities, but that summarization provides a particularly flexible interface for representing what matters—quality differentials, user-specific values, or factual faithfulness—and for turning those representations into trainable objectives.

Source: https://www.emergentmind.com/topics/preference-learning-using-summarization-plus