---
title: 'LongLaMP: Benchmark for Personalized Text Generation'
url: https://www.emergentmind.com/topics/longlamp
type: topic
---

# LongLaMP: Benchmark for Personalized Text Generation

LongLaMP is a unified benchmark designed to systematically evaluate and advance personalized long-form text generation. It emphasizes the generation of extended, user-tailored text in settings where existing personalization research has focused predominantly on short passages. The benchmark enables robust comparisons of both zero-shot and fine-tuned models, includes a diverse range of real-world tasks, and provides a rigorous suite of evaluation metrics and best-practice recommendations. LongLaMP has also served as the focal point for methodological innovations such as reasoning-augmented self-training frameworks and reference-free LLM-based evaluation, catalyzing progress in personalized text generation research [2407.11016][2501.04167].

## 1. Formalization and Framework

Personalized long-form text generation in LongLaMP is formulated as a mapping:
$$(x_k, y_k, P_{u_k}) \longmapsto \, \hat{y} = f(x; P_u)$$
where $x_k$ is the user prompt (such as a partial email or summary), $y_k$ is the user-authored reference response, and $P_{u_k}$ is the user profile comprising a collection of that user's prior documents. The generation model $f$ is evaluated for its ability to produce $\hat{y}$ maximizing similarity to $y$, operationalized through token-overlap-based metrics (ROUGE, METEOR) and learned LLM-based quality rewards. In retrieval-augmented architectures, $f$ is decomposed into query construction, retrieval from $P_u$, personalized prompt construction, and LLM-based generation. Both zero-shot and gradient-based fine-tuned instantiations are supported within the evaluation protocol [2407.11016].

## 2. Benchmark Tasks and Data Composition

LongLaMP comprises four distinct personalized long-text generation tasks spanning diverse domains and annotation regimes, summarized as follows:

| Task                     | Input Format                           | Output Format                | Profile Size (mean)  |
|--------------------------|----------------------------------------|------------------------------|----------------------|
| Email Completion         | Title + partial body                   | Remaining email body         | 60–85 documents      |
| Abstract Generation      | Title + LLM-extracted keywords         | Full abstract                | 118–120 documents    |
| Review Writing           | Product description, rating, summary   | Full review                  | 34–50 documents      |
| Topic Writing            | Reddit TL;DR summary                   | Full post                    | 43–50 documents      |

Each dataset includes multiple data split regimes:
- "User" (cold-start): Disjoint sets of users between train/val/test.
- "Temporal": Chronologically sorted, with most recent examples for test/validation (except Topic Writing’s Random split).

Input and output lengths are task-dependent, e.g., review outputs average 304 ± 229 tokens; emails 92.6 ± 60.7 tokens per response [2407.11016][2501.04167].

## 3. Evaluation Metrics

LongLaMP employs three main evaluation metrics based on span and overlap:
- **ROUGE-1**: Unigram recall, precision, F1.
- **ROUGE-L**: Longest common subsequence, computed as
  $$
  R_{LCS} = \frac{LCS(\hat{y}, y)}{|y|}, \quad
  P_{LCS} = \frac{LCS(\hat{y}, y)}{|\hat{y}|}
  $$
  and the F1-like score applies:
  $$
  F_{LCS} = \frac{(1+\beta^2)R_{LCS}P_{LCS}}{R_{LCS}+\beta^2P_{LCS}}
  $$
- **METEOR**: Hybrid alignment, unigram F1, length-penalized.

Additionally, a reference-free LLM-based reward model is used as an evaluator. It provides an automatic “quality” score from 0 to 1, based on ten discrete criteria, with scores demonstrating significant agreement (73%) with crowd judgment and a moderate Spearman correlation (∼0.46). LLM-based scores degrade under profile randomization, demonstrating sensitivity to personalization [2501.04167].

## 4. Baseline Architectures and Methodologies

LongLaMP supports two primary personalization paradigms:
- **Zero-Shot Retrieval-Augmented Generation (RAG)**: Off-the-shelf LLMs (e.g., GPT-3.5-turbo, LLaMA 2–7B-chat), combined with either BM25 or dense retrievers (e.g., ReContriever). Retrieval selects top-$k$ documents from $P_u$, which are concatenated to the prompt ($k$ is task-specific).
- **Fine-Tuning**: Models such as FLAN-T5-base are fine-tuned on combined “personalized prompts,” optimizing cross-entropy against $y$ using standard hyperparameters (AdamW, learning rate $5\times10^{-5}$, batch size $1$ with gradient accumulation, max length $1024$) [2407.11016].

## 5. Key Experimental Findings

### Personalization Gains
Both zero-shot and fine-tuning regimes exhibit substantial personalization improvements, especially for tasks where prior style, intent, or knowledge are crucial. Representative zero-shot ROUGE-1 gains (User split):
- **Abstract Generation**: +7.4% (GPT-3.5); +11.7% (LLaMA 2)
- **Topic Writing**: +4.7% (GPT-3.5); +16.3% (LLaMA 2)
- **Review Writing**: +4.1% (GPT-3.5); +3.6% (LLaMA 2)
- **Email Completion**: +96.8% (LLaMA 2 only)

Fine-tuning continues to yield significant improvements (e.g., +78.9% ROUGE-1 for email completion, +8.0% for topic writing; all for best $k$) [2407.11016].

### Reasoning-Augmented Self-Training
REST-PG (Reasoning-Enhanced Self-Training for Personalized Text Generation) extends the baseline with explicit reasoning-chain supervision followed by EM-style reinforced self-training. The procedure generates intermediate “reasoning” targets via LLM, then uses a reward-weighted SFT objective:
$$
\theta^{t+1} = \arg\max_\theta \sum_{(x, y, P, \hat{y})\in D_t} R(x, y, \hat{y}) \log p_\theta(\hat{y} \mid x, P)
$$
REST-PG achieves a macro-average LLM-reward of 0.4554 (vs. 0.3976 for SFT, a 14.6% relative improvement). REST-PG outperforms all baselines on every LongLaMP task (significant at $p<0.05$ on three out of four tasks) [2501.04167].

## 6. Limitations, Best Practices, and Extensions

**Limitations**:
- Token-overlap metrics (ROUGE, METEOR) may fail to capture deeper elements of personalization or factuality.
- The private nature of certain source data (e.g., Avocado email collection) may constrain reproducibility.
- Privacy risks (e.g., memorization, membership inference in fine-tuning) are noted but not directly addressed.
- Large $k$ for retrieval may degrade performance due to prompt length and noise accumulation.

**Best Practices and Extensions**:
- Per-task tuning of $k$ is essential; increasing $k$ past a threshold results in no or negative gains.
- Summarization of retrieved profile documents or explicit style-extraction modules is recommended for long user histories.
- Integrating privacy-preserving approaches (on-device retrieval, federated fine-tuning) is encouraged for real-world deployment.
- Research extension to new domains (e.g., legal, educational) and exploration of narrative, discourse, and human-centric evaluation criteria is advised for future iterations [2407.11016].

## 7. Significance and Research Impact

LongLaMP establishes the first standardized, open, and extensible platform for rigorously evaluating personalized long-form generation. It demonstrates substantial and consistent personalization gains utilizing retrieval-augmented LLMs, highlights the need for privacy safeguards and improved evaluation, and supports reference-free LLM-based metrics as viable proxies for human judgment. Subsequent work, such as REST-PG, builds on these foundations, showing that explicit reasoning over user context yields substantial improvements in alignment with user intent across diverse, real-world generation tasks [2407.11016][2501.04167].

Source: https://www.emergentmind.com/topics/longlamp