LongLaMP: Benchmark for Personalized Text Generation

• LongLaMP is a unified benchmark designed to evaluate personalized long-form text generation, encompassing diverse tasks, user profiles, and robust evaluation metrics.
• It employs both zero-shot retrieval-augmented generation and fine-tuning strategies, using metrics like ROUGE, METEOR, and LLM-based quality scores to assess performance.
• Innovative methods such as reasoning-augmented self-training (REST-PG) enhance model alignment with user intent, achieving significant personalization gains.

LongLaMP is a unified benchmark designed to systematically evaluate and advance personalized long-form text generation. It emphasizes the generation of extended, user-tailored text in settings where existing personalization research has focused predominantly on short passages. The benchmark enables robust comparisons of both zero-shot and fine-tuned models, includes a diverse range of real-world tasks, and provides a rigorous suite of evaluation metrics and best-practice recommendations. LongLaMP has also served as the focal point for methodological innovations such as reasoning-augmented self-training frameworks and reference-free LLM-based evaluation, catalyzing progress in personalized text generation research (Kumar et al., 2024, Salemi et al., 7 Jan 2025).

1. Formalization and Framework

Personalized long-form text generation in LongLaMP is formulated as a mapping:

$$(x_k, y_k, P_{u_k}) \longmapsto \, \hat{y} = f(x; P_u)$$

where $x_k$ is the user prompt (such as a partial email or summary), $y_k$ is the user-authored reference response, and $P_{u_k}$ is the user profile comprising a collection of that user's prior documents. The generation model $f$ is evaluated for its ability to produce $\hat{y}$ maximizing similarity to $y$, operationalized through token-overlap-based metrics (ROUGE, METEOR) and learned LLM-based quality rewards. In retrieval-augmented architectures, $f$ is decomposed into query construction, retrieval from $P_u$, personalized prompt construction, and LLM-based generation. Both zero-shot and gradient-based fine-tuned instantiations are supported within the evaluation protocol (Kumar et al., 2024).

2. Benchmark Tasks and Data Composition

LongLaMP comprises four distinct personalized long-text generation tasks spanning diverse domains and annotation regimes, summarized as follows:

Task	Input Format	Output Format	Profile Size (mean)
Email Completion	Title + partial body	Remaining email body	60–85 documents
Abstract Generation	Title + LLM-extracted keywords	Full abstract	118–120 documents
Review Writing	Product description, rating, summary	Full review	34–50 documents
Topic Writing	Reddit TL;DR summary	Full post	43–50 documents

Each dataset includes multiple data split regimes:

• "User" (cold-start): Disjoint sets of users between train/val/test.
• "Temporal": Chronologically sorted, with most recent examples for test/validation (except Topic Writing’s Random split).

Input and output lengths are task-dependent, e.g., review outputs average 304 ± 229 tokens; emails 92.6 ± 60.7 tokens per response (Kumar et al., 2024, Salemi et al., 7 Jan 2025).

3. Evaluation Metrics

LongLaMP employs three main evaluation metrics based on span and overlap:

• ROUGE-1: Unigram recall, precision, F1.
• ROUGE-L: Longest common subsequence, computed as

$$R_{LCS} = \frac{LCS(\hat{y}, y)}{|y|}, \quad P_{LCS} = \frac{LCS(\hat{y}, y)}{|\hat{y}|}$$

and the F1-like score applies:

$x_k$0

• METEOR: Hybrid alignment, unigram F1, length-penalized.

Additionally, a reference-free LLM-based reward model is used as an evaluator. It provides an automatic “quality” score from 0 to 1, based on ten discrete criteria, with scores demonstrating significant agreement (73%) with crowd judgment and a moderate Spearman correlation (∼0.46). LLM-based scores degrade under profile randomization, demonstrating sensitivity to personalization (Salemi et al., 7 Jan 2025).

4. Baseline Architectures and Methodologies

LongLaMP supports two primary personalization paradigms:

• Zero-Shot Retrieval-Augmented Generation (RAG): Off-the-shelf LLMs (e.g., GPT-3.5-turbo, LLaMA 2–7B-chat), combined with either BM25 or dense retrievers (e.g., ReContriever). Retrieval selects top-$x_k$1 documents from $x_k$2, which are concatenated to the prompt ($x_k$3 is task-specific).
• Fine-Tuning: Models such as FLAN-T5-base are fine-tuned on combined “personalized prompts,” optimizing cross-entropy against $x_k$4 using standard hyperparameters (AdamW, learning rate $x_k$5, batch size $x_k$6 with gradient accumulation, max length $x_k$7) (Kumar et al., 2024).

5. Key Experimental Findings

Personalization Gains

Both zero-shot and fine-tuning regimes exhibit substantial personalization improvements, especially for tasks where prior style, intent, or knowledge are crucial. Representative zero-shot ROUGE-1 gains (User split):

• Abstract Generation: +7.4% (GPT-3.5); +11.7% (LLaMA 2)
• Topic Writing: +4.7% (GPT-3.5); +16.3% (LLaMA 2)
• Review Writing: +4.1% (GPT-3.5); +3.6% (LLaMA 2)
• Email Completion: +96.8% (LLaMA 2 only)

Fine-tuning continues to yield significant improvements (e.g., +78.9% ROUGE-1 for email completion, +8.0% for topic writing; all for best $x_k$8) (Kumar et al., 2024).

Reasoning-Augmented Self-Training

REST-PG (Reasoning-Enhanced Self-Training for Personalized Text Generation) extends the baseline with explicit reasoning-chain supervision followed by EM-style reinforced self-training. The procedure generates intermediate “reasoning” targets via LLM, then uses a reward-weighted SFT objective:

$x_k$9

REST-PG achieves a macro-average LLM-reward of 0.4554 (vs. 0.3976 for SFT, a 14.6% relative improvement). REST-PG outperforms all baselines on every LongLaMP task (significant at $y_k$0 on three out of four tasks) (Salemi et al., 7 Jan 2025).

6. Limitations, Best Practices, and Extensions

Limitations:

• Token-overlap metrics (ROUGE, METEOR) may fail to capture deeper elements of personalization or factuality.
• The private nature of certain source data (e.g., Avocado email collection) may constrain reproducibility.
• Privacy risks (e.g., memorization, membership inference in fine-tuning) are noted but not directly addressed.
• Large $y_k$1 for retrieval may degrade performance due to prompt length and noise accumulation.

Best Practices and Extensions:

• Per-task tuning of $y_k$2 is essential; increasing $y_k$3 past a threshold results in no or negative gains.
• Summarization of retrieved profile documents or explicit style-extraction modules is recommended for long user histories.
• Integrating privacy-preserving approaches (on-device retrieval, federated fine-tuning) is encouraged for real-world deployment.
• Research extension to new domains (e.g., legal, educational) and exploration of narrative, discourse, and human-centric evaluation criteria is advised for future iterations (Kumar et al., 2024).

7. Significance and Research Impact

LongLaMP establishes the first standardized, open, and extensible platform for rigorously evaluating personalized long-form generation. It demonstrates substantial and consistent personalization gains utilizing retrieval-augmented LLMs, highlights the need for privacy safeguards and improved evaluation, and supports reference-free LLM-based metrics as viable proxies for human judgment. Subsequent work, such as REST-PG, builds on these foundations, showing that explicit reasoning over user context yields substantial improvements in alignment with user intent across diverse, real-world generation tasks (Kumar et al., 2024, Salemi et al., 7 Jan 2025).