Preference Learning Using Summarization (PLUS)

• The paper demonstrates that PLUS leverages textual summaries to capture nuanced user preferences and improve reward model training.
• PLUS is a framework where interactive pairwise comparisons and concept-level feedback are used to enhance personalization in summarization systems.
• Empirical evaluations show that PLUS significantly boosts alignment, factual accuracy, and downstream performance across diverse benchmarks.

Preference Learning Using Summarization (PLUS) denotes a family of methods in which summarization is used as the substrate for learning preferences, usually through comparisons, feedback, or preference-conditioned generation rather than through reference matching alone. In one explicit formulation, PLUS is a framework for pluralistic, personalized preference learning in which a model learns text-based summaries of each user’s preferences, characteristics, and past conversations and conditions a reward model on those summaries (Nam et al., 17 Jul 2025). Taken together, the surrounding literature suggests a broader research program: summaries, summary pairs, concept selections, refinement trajectories, and user summaries can all serve as carriers of preference information, and those signals can be converted into ranking functions, reward models, or policy updates for downstream summarization (Gooding et al., 2023).

1. Historical emergence and research scope

Early work in this area replaced reference summaries with interactive preference signals. APRIL, instantiated for extractive multi-document summarization, learned from user preferences between candidate summaries and decomposed the problem into Active Preference Learning (APL) and Reinforcement Learning (RL), explicitly targeting lower sample complexity than earlier preference-based interactive methods (Gao et al., 2018). “Adaptive Summaries” moved the interaction from whole-summary comparisons to concept-level feedback, allowing a user to accept or reject a concept, assign a weight, and provide a confidence value, with the learned preference represented by updated concept weights inside an iterative ILP-based loop (Ghodratnama et al., 2020). Subsequent personalized systems such as Summation and SumRecom framed summarization as user-adaptive concept ranking and policy optimization over summaries or hierarchical concept maps (Ghodratnama et al., 2023, Ghodratnama et al., 2024).

A second line of work brought preference learning into neural summarization and RLHF. “Make The Most of Prior Data: A Solution for Interactive Text Summarization with Preference Feedback” proposed an interactive framework with a pretrained extractive summarizer, a reward model called ROMSR, PPO-based reinforcement learning, and selective reuse of offline data in active, online, and few-shot settings (Nguyen et al., 2022). “The Impact of Preference Agreement in Reinforcement Learning from Human Feedback: A Case Study in Summarization” studied the standard RLHF summarization setup and asked which kinds of human comparison data are most useful for learning reward models and improving downstream summarization (Gooding et al., 2023).

The explicit framework named Preference Learning Using Summarization (PLUS) appears in “Learning Pluralistic User Preferences through Reinforcement Learning Fine-tuned Summaries,” where the central claim is that standard RLHF is too coarse for users with conflicting preferences and that user-specific textual summaries can act as compact, human-readable latent representations for personalized reward modeling (Nam et al., 17 Jul 2025). More recent work extends the same logic to automated factuality supervision, clinical hallucination reduction, and structured personalization over review corpora (Ye et al., 26 May 2026, Seethakantha et al., 27 May 2026, Roy et al., 7 May 2026). This scope makes PLUS both a specific architecture and a broader methodological pattern in which summarization is used to elicit, compress, or operationalize preferences.

2. Formal problem formulations

In RLHF summarization, PLUS often begins as a pairwise preference-learning problem. Given a context $x$ and two candidate summaries $y_w$ and $y_l$, a reward model outputs a scalar score $r_\theta(x,y)$ and is trained so that the preferred summary receives the higher score. A standard objective is

$$\mathcal{L}(\theta) = - \mathbb{E}_{(x,y_w,y_l)\sim D} \left[\log \sigma\left(r_\theta(x,y_w)-r_\theta(x,y_l)\right)\right],$$

where the score difference is interpreted as the log-odds that humans will prefer one summary over the other (Gooding et al., 2023). In that setting, the reward model is then used inside an RLHF loop to fine-tune a summarization policy with constrained policy updates.

The personalized PLUS framework generalizes the classical Bradley–Terry–Luce model by conditioning on a user representation. Standard RLHF models a single population-level preference function, whereas PLUS learns a summarizer $\pi_\theta$ that maps user context $c$ to a short natural-language summary $z=\pi_\theta(c)$, and a summary-conditioned reward model $r_\phi(s\mid z)$ that predicts preferences given that summary (Nam et al., 17 Jul 2025). The comparison probability becomes

$p(\mathbbm{1}\{s_A > s_B\}=1 \mid s_A,s_B,z) = \frac{\exp r_\phi(s_A \mid z)}{\exp r_\phi(s_A \mid z)+\exp r_\phi(s_B \mid z)} = \sigma\!\big(r_\phi(s_A\mid z)-r_\phi(s_B\mid z)\big).$

Here the summary is not trained for reconstruction; it is trained to extract exactly those aspects of the user history that are useful for predicting future preferences.

Other PLUS variants replace human labels with automatically derived pairwise supervision. In factual consistency optimization, the method generates two summaries from the same source document, scores both with multiple factuality metrics, converts score differences into binary preferences, filters out pairs with metric disagreement, and trains the summarizer with Direct Preference Optimization (DPO): $y_w$0 This preserves the pairwise structure while avoiding a separate reward model (Ye et al., 26 May 2026).

Concept-based personalized summarization adopts related, but not identical, formalisms. Adaptive Summaries maximizes user-preferred concept content under a length budget with an ILP objective, while Summation and SumRecom formulate summary construction as an episodic MDP in which actions add sentences or concepts and terminal rewards reflect learned user importance or reward estimates (Ghodratnama et al., 2020, Ghodratnama et al., 2023, Ghodratnama et al., 2024). The common structural feature is that preferences are first transformed into a reusable scoring object—concept weights, a ranking function, or a reward model—and only then used for summary generation.

3. Sources of preference information

Human pairwise summary comparisons remain the canonical supervision source. In the RLHF case study on summarization, the preference dataset from Stiennon et al. (2020) contains 64,832 human judgments over TL;DR Reddit summaries, with repeated annotations for some pairs so that annotator agreement can be measured (Gooding et al., 2023). The paper’s key insight is that agreement among annotators is not just noise; it reflects the underlying quality differential between the two summaries being compared. High-agreement examples teach obvious, high-separation quality cues; medium-agreement examples force the model to learn subtler distinctions; low-agreement examples can be ambiguous or noisy. This makes agreement itself a preference feature, not merely a data-cleaning signal.

Personalized PLUS replaces opaque user embeddings with textual summaries. The user context $y_w$1 can be a set of past preference pairs, but the framework is explicitly flexible enough to learn from past preference pairs, self-stated preferences, survey answers, or attribute lists (Nam et al., 17 Jul 2025). The intended representation is short, editable, and human-readable. The qualitative distinction emphasized in the paper is between untrained summaries that mention superficial style or generic traits and RL-tuned summaries that encode the true preference dimension, such as whether a user wants information about cats rather than dogs. In this setting, the summary is itself a learned representation of preference.

Interactive concept-based methods expose preference structure more directly. Adaptive Summaries asks users to accept or reject concepts and to specify both importance and confidence; a rejected sentence sets the sentence weight to zero but does not necessarily update all concepts inside that sentence, because sentence rejection may reflect redundancy rather than concept irrelevance (Ghodratnama et al., 2020). SumRecom similarly learns concept rankings from lightweight pairwise preferences over concepts, then uses those rankings to generate and optimize personalized summaries (Ghodratnama et al., 2024). These approaches treat preference learning as concept selection before sentence or summary selection.

Automated supervision extends PLUS beyond explicit human judgments. In factual consistency optimization, SBERTScore and SummaC-Conv act as multiple weak annotators; a pair is retained only if all selected metrics agree on the preference direction (Ye et al., 26 May 2026). In clinical summarization, detector-guided refinement produces trajectories $y_w$2, and later, detector-corrected summaries are treated as preferred over earlier, less faithful versions, thereby forming preference pairs without human annotation (Seethakantha et al., 27 May 2026). In personalized review summarization, feedback is scalar rather than pairwise, but it is still mapped into an online preference estimate over latent aspects and updated through entropic online mirror descent (Roy et al., 7 May 2026). Across these variants, PLUS broadens the meaning of “preference data” to include disagreement structure, user-authored text, concept choices, metric consensus, and refinement-based factual corrections.

4. Learning architectures and optimization strategies

Reward modeling remains central in neural PLUS systems. In the RLHF agreement study, the reward model is a T5-XXL model with about 13B parameters, 24 transformer blocks, and 64 attention heads, trained as a binary preference classifier over pairs of summaries; downstream RLHF fine-tunes a T5-small summarization policy and constrains policy updates to avoid overly large steps (Gooding et al., 2023). In personalized PLUS, the main models use Qwen2.5-0.5B as the reward model and Qwen2.5-3B-Instruct as the summarizer, both fine-tuned with LoRA (Nam et al., 17 Jul 2025). The summarizer receives RL reward from the negative reward-model loss and is updated with PPO, while the reward model is updated by gradient descent on preference loss.

This online co-adaptation loop is a defining architectural choice. For a batch $y_w$3, personalized PLUS generates a summary $y_w$4, computes the reward-model loss

$y_w$5

uses the negative of that loss as the RL reward for the summarizer, updates $y_w$6 by gradient descent, and updates $y_w$7 using PPO (Nam et al., 17 Jul 2025). The appendix reports a summarizer learning rate of $y_w$8, a critic learning rate of $y_w$9, and a KL penalty of 0.01 generally, or 0.001 for Pets. The paper explicitly motivates PPO as a stable choice because the summarizer’s reward is non-stationary.

Several alternatives bypass reward-model training. Automated factuality PLUS uses lexically similar summary pairs and DPO directly, arguing that careful pair construction and metric-agreement filtering create a high-quality preference dataset from source documents alone (Ye et al., 26 May 2026). Clinical summarization follows the same preference-optimization pattern after Hallucination Detection guided Self-Refinement (HDSR) produces ordered refinement trajectories; HDSR-PL then fine-tunes the model with DPO so that factual corrections are amortized into the model parameters (Seethakantha et al., 27 May 2026).

Other strands organize learning in stages rather than via online co-adaptation. AlignSum builds a Data Pyramid of extractive, abstractive, and human-annotated summary data, applies Gaussian Resampling to align pseudo-summary lengths with human-annotated summary lengths, and performs two-stage Hierarchical Fine-Tuning (HFT): generic fine-tuning on extractive plus abstractive data, followed by personalized fine-tuning on human-annotated data (Han et al., 2024). The paper explicitly contrasts this with naive hybrid fine-tuning, arguing that scarce human-preference data can otherwise be overwhelmed by larger generic corpora.

Interactive extractive methods use older, but still structurally relevant, optimization stacks. APRIL learns a linear Bradley–Terry ranker from queried preferences and then uses that learned ranking as a reward signal for RL, with TD($y_l$0) and LSTD($y_l$1) evaluated in the policy phase (Gao et al., 2018). “Make The Most of Prior Data” uses an extractive BERTSUM backbone, the ROMSR reward model, PPO fine-tuning, and selective offline replay through Low-Reward Sampling (LRS) and Document-Similarity Sampling (DSS) (Nguyen et al., 2022). These systems differ in architecture but share the same decomposition: learn a reward-like object from preference information, then optimize summary generation against it.

5. Empirical patterns and benchmark results

The empirical literature repeatedly shows that the composition of preference data changes both predictive accuracy and the notion of quality that a model learns. In the RLHF agreement study, the authors construct four training sets of 2,000 comparisons each—MAX, MIN, DIST, and RAND—from instances with repeated annotations and evaluate on a held-out test set of 1,267 instances (Gooding et al., 2023). The main result is that DIST improves faster during training, reaches the highest test accuracy, and outperforms MAX, MIN, and RAND. MIN is consistently worst and learns the slowest. On SummEval, reward models trained on more diverse agreement distributions capture multiple dimensions of quality better, with DIST performing especially well on coherence and relevance. When used inside RLHF, DIST yields the best ROUGE-1, ROUGE-2, and ROUGE-L before overfitting, whereas MIN produces the worst downstream results and MAX does not consistently improve all ROUGE metrics despite strong reward-model accuracy.

The personalized PLUS framework reports gains across synthetic and real pluralistic datasets. On Pets, Pets (OOD), UltraFeedback-Personalized, and PRISM, PLUS is evaluated against BTL, DPL, VPL, ICL, PLUS-untrained, and Oracle, with performance measured by preference prediction accuracy on held-out pairs (Nam et al., 17 Jul 2025). The paper reports improvements as large as 126% on Pets, 17% on UltraFeedback, and 5% on PRISM. Under the Pets (OOD) shift, where training uses cats and dogs but evaluation uses rabbits and birds, PLUS still achieves very high accuracy, above 90%, while ICL drops sharply and VPL performs poorly. The same paper also evaluates transfer to GPT-4.1, GPT-4o, and GPT-4-turbo: for GPT-4o on controversy-guided conversations, judge accuracy improves from 52.2 to 73.0, and personalized generation win rate improves from 34% to 66%.

Alignment-oriented preference fitting in conventional summarization also reports strong gains. AlignSum evaluates on the Element-Aware versions of CNN/DailyMail and BBC XSum, each with a human-annotated set of 200 samples, split into 100 train and 100 test (Han et al., 2024). In automatic evaluation, BART-Large with full DP achieves ROUGE-1 48.83, ROUGE-2 24.11, ROUGE-L 34.16, BERTScore 0.9058 on CNN/DailyMail and ROUGE-1 42.38, ROUGE-2 17.75, ROUGE-L 31.64, BERTScore 0.8962 on BBC XSum. In human evaluation against GPT-3 with CoT prompting, it achieves up to 65% Win and 72% Win+Equal. The ablation study identifies HFT as the most important component.

Automated factuality-oriented PLUS produces substantial improvements without human ranking. On XSUM, the factuality paper reports AlignScore gains from 61.9 to 86.6 for BART, from 59.7 to 75.8 for GPT-J, from 86.1 to 88.7 for LLaMA, and from 82.5 to 83.2 for DeepSeek (Ye et al., 26 May 2026). On TL;DR, the corresponding gains are 84.9 to 94.2 for BART, 89.6 to 93.8 for GPT-J, 91.4 to 93.5 for LLaMA, and 89.1 to 90.9 for DeepSeek. The paper further reports that at least 60% of generated pairs were retained after metric-disagreement filtering and that using SBERTScore + SummaC together improves results over either metric alone.

Clinical summarization shows the same pattern under safety constraints. On Hallucination-Generated-DI derived from MIMIC-IV-Note v2.2, the main LLaMA-3.1-8B-Instruct comparison reports 29 hallucinations for prompting, 57 hallucinations for SFT, 22 hallucinations for HDSR (best; MedAlign), and 15 hallucinations for HDSR-PL (best; MedCat) (Seethakantha et al., 27 May 2026). The paper highlights that HDSR reduces hallucinations by about 24% relative to prompting, while HDSR-PL reduces hallucinations by about 48% relative to prompting and about 74% relative to SFT. Clinician ratings also improve: HDSR (MedAlign) reaches Consistency 4.13, Coherence 4.48, Fluency 4.53, Relevance 3.95, Avg 4.27, and HDSR-PL (MedCat) reaches Consistency 4.40, Coherence 4.28, Fluency 4.05, Relevance 3.90, Avg 4.16.

Interactive personalized summarization reports improvements of a different kind: user alignment and interaction efficiency. SumRecom reports ROUGE-1/2/L of 0.341/0.078/0.28 on DUC1, 0.372/0.083/0.333 on DUC2, and 0.382/0.094/0.301 on DUC4, outperforming APRIL and SPPI in those tables; its human study reports that 83% of participants preferred the produced summary, average satisfaction was 8.2 / 10, and response time was about 13 seconds (Ghodratnama et al., 2024). Summation reports ROUGE-1 0.731, ROUGE-2 0.651, ROUGE-L 0.681 in its summary evaluation table and states that 86.67% of workers were completely confident in answering topic questions after using the concept maps (Ghodratnama et al., 2023). These results are not directly comparable to RLHF summarization benchmarks, but they reinforce the broader claim that preference-aware summarization can outperform static, one-size-fits-all systems within its own evaluation regime.

6. Interpretability, limitations, and research tensions

A recurring misconception in this area is that better preference learning always comes from cleaner or more unanimous labels. The agreement study directly disputes that view: maximizing agreement alone is not always optimal, because highly agreed-upon comparisons may be easy and reliable yet comparatively superficial for learning nuanced quality distinctions, while low-agreement comparisons can be too ambiguous if overused (Gooding et al., 2023). A second misconception is that one global reward model is sufficient for all users. The personalized PLUS framework argues that a single population-level preference function is systematically wrong when user preferences are multi-modal and conflicting (Nam et al., 17 Jul 2025).

Interpretability is one of the field’s most distinctive claims. Textual user summaries in personalized PLUS are described as interpretable, portable, and editable; the paper argues that they can be surfaced to users as a transparency mechanism and even transferred for zero-shot personalization of stronger proprietary models (Nam et al., 17 Jul 2025). Concept-based methods make the same point in a different form: users can inspect the concepts that drive selection and can influence summary construction by comparing concepts rather than drafting summaries from scratch (Ghodratnama et al., 2020, Ghodratnama et al., 2024). This stands in explicit contrast to vector embeddings, which the personalized PLUS paper characterizes as opaque and difficult to inspect.

The limitations are substantial and differ by regime. Real heterogeneous human data remain difficult to model: PRISM is relatively small compared to the diversity it tries to capture, and even non-personalized baselines can sometimes do well by predicting majority preference (Nam et al., 17 Jul 2025). AlignSum depends on access to human-preference data and is limited experimentally to CNN/DailyMail and BBC XSum because those were the datasets with rewritten versions reflecting the target preferences (Han et al., 2024). Automated factuality methods report a trade-off in which PLUS summaries are often more conservative and shorter, improving factuality while sacrificing some informativeness or stylistic richness (Ye et al., 26 May 2026). Clinical systems operate in a safety-relevant domain where unsupported diagnoses, procedures, medications, times, or test results are not merely stylistic errors, and their evaluation remains partly resource-constrained (Seethakantha et al., 27 May 2026). Earlier interactive frameworks are often limited to extractive or multi-document settings and may require careful query selection to remain practical (Gao et al., 2018, Nguyen et al., 2022).

Taken together, these results suggest that PLUS is best understood not as a single algorithm but as a design principle for alignment through summarization. Preferences can be elicited from humans, induced from disagreement patterns, inferred from detector-guided revisions, summarized into human-readable user models, or updated online from scalar feedback. The strongest common conclusion is not that summarization should replace other preference-learning modalities, but that summarization provides a particularly flexible interface for representing what matters—quality differentials, user-specific values, or factual faithfulness—and for turning those representations into trainable objectives.