PSE-SU4: Personalized Summarization Metric

• The paper introduces PSE-SU4 as a metric that quantifies personalized summarization by assessing divergence using ROUGE-SU4 based skip-bigram comparisons.
• It integrates a normalized ratio framework with a data augmentation pipeline (PerAugy) to enhance evaluation by directly measuring user-specific summary alignment.
• The metric offers a robust benchmark for personalization, demonstrating strong correlation with human judgments and significant improvements over traditional evaluation methods.

The PSE-SU4 metric is a personalized evaluation measure introduced to quantify the responsiveness of text summarization systems to individual user preferences. It is a variant of the PerSEval metric, specifically tailored to the personalized summarization task where the subjective alignment of generated summaries with user-specific gold references is paramount. The metric leverages ROUGE-SU4, a skip-bigram–based content similarity measure, within a normalized, ratio-based divergence framework that directly assesses how well a summarizer output matches the expectations of a particular user among a pool of users. PSE-SU4 has been shown to exhibit strong correlation with human judgments of personalization quality and is sensitive to improvements from dataset diversity augmentation, notably through the PerAugy pipeline.

1. Formal Definition and Measurement Framework

PSE-SU4 quantifies personalization by evaluating the divergence between a generated summary $G$ and a user-specific gold reference summary $R$ using the ROUGE-SU4 F1 score, which measures the overlap of skip-bigrams (bigrams where up to four words can be skipped) between the summaries. The key elements of the metric are:

• Skip-bigram Recall:

$$R_{SU4} = \frac{\text{Number of matching skip-bigrams between } G \text{ and } R}{\text{Total skip-bigrams in } R}$$

• Skip-bigram Precision:

$$P_{SU4} = \frac{\text{Number of matching skip-bigrams between } G \text{ and } R}{\text{Total skip-bigrams in } G}$$

• F1 score:

$$F1_{SU4} = \frac{2 \cdot P_{SU4} \cdot R_{SU4}}{P_{SU4} + R_{SU4}}$$

PSE-SU4 is defined as the normalized ratio of divergences between the generated summary for a specific user and their gold reference, relative to all other user reference summaries associated with the same document. This ratio-based formulation ensures scale-invariance and focuses specifically on the subjective component of personalization. The metric thereby captures the degree to which a system’s output is more relevant for the intended user than for others, aligning with human expectations of personalized relevance.

2. Relationship to Data Augmentation via PerAugy

The PerAugy technique is a data augmentation pipeline designed to enrich User Interaction Graphs (UIGs) with enhanced diversity in both temporal and thematic dimensions. PerAugy consists of two primary components:

• Double Shuffling (DS): Segments from different user trajectories are interchangeably swapped, with controlled gap parameters to ensure natural interest diffusion across temporal axes.
• Stochastic Markovian Perturbation (SMP): Introduced segment mismatches are smoothed by substituting summary nodes with contextually coherent alternatives, guided by an influence-weighted RMSD computation.

By leveraging this augmentation, models are trained on more varied user preference data, which leads to more nuanced and responsive user representations. The impact of PerAugy augmentation is evaluated intrinsically through PSE-SU4, with experimental findings demonstrating a marked improvement in PSE-SU4 scores for personalized summarizers utilizing PerAugy-augmented training data compared to baselines. Gains as large as 0.012 (e.g., from 0.006 to 0.017) have been reported for models using the DS+SMP variant in the GTP framework with TrRMIo embeddings.

3. Comparison with Conventional and Diversity-Oriented Metrics

Traditional metrics for personalized recommendation and summarization include:

Metric	Measurement Focus	Captures Personalization
AUC	Binary ranking accuracy	No
MRR	Reciprocal rank of correct result	No
nDCG@k	Position-weighted ranking relevance	No
TP, RTC	Trajectory diversity (topics, rate)	Partial
DegreeD	Document/summary shift alignment	Partial
PSE-SU4	Summary–user alignment (SU4 ratio)	Yes

While AUC, MRR, and nDCG@k measure global accuracy or ranking quality, and TP/RTC/DegreeD characterize dataset trajectory diversity, they do not directly assess the subjective alignment of generated summaries to individual users. PSE-SU4 uniquely quantifies this personalization axis, correlating closely with human evaluations (Pearson $r = 0.6$, Spearman $\rho = 0.6$, Kendall $\tau = 0.51$ as reported) and offering robust validity for benchmarking personalized summarization systems.

4. Evaluation Methodology and Empirical Findings

The evaluation methodology for PSE-SU4 in the context of PerAugy involves several steps:

1. UIG Construction: Original user trajectories are sampled and augmented via DS and DS+SMP within PerAugy, introducing both controlled and stochastic diversity.
2. Model Training: Four state-of-the-art user-encoder models (NAML, NRMS, EBNR, TrRMIo) are trained on the augmented UIGs.
3. Integration and Summarization: User embeddings from these encoders are injected into two summarization frameworks (PENS and GTP).
4. Personalization Evaluation: Model outputs are compared against user-specific gold summaries using PSE-SU4 as the principal metric.

Results consistently indicate that models trained on PerAugy-augmented data achieve higher PSE-SU4 scores than those trained on data augmented with baseline methods (PENS-SH, S3, SDAInter). The reported improvements validate the hypothesis that increasing dataset diversity through structured augmentation strategies leads to more personalized and effective summarizer outputs, as measured by PSE-SU4.

5. Implications and Future Research Directions

PSE-SU4's demonstrated sensitivity to dataset diversity and robust correlation with human perceptions of personalization have several notable implications:

• Dataset Engineering: The relationship between UIG diversity and PSE-SU4 suggests that dataset augmentation is an important lever for improving personalized summarization performance, particularly in domains where user–gold reference pairs are scarce.
• Evaluation Standardization: PSE-SU4 provides a benchmark for evaluating the personalization quality of summarization systems, in contrast to standard ranking metrics that cannot detect subjective responsiveness.
• Modular Augmentation Pipelines: The modularity of PerAugy (DS, SMP) allows the technique to be adapted for various domains requiring personalized data synthesis, including recommender systems and dialogue personalization tasks.
• Resource-Efficient Training: The generation of synthetic, diverse training data is particularly advantageous in low-resource scenarios or for novel domains lacking annotated datasets.
• Metric Development: Future work may explore refinements to the ratio and divergence formulations underlying PSE-SU4 or the integration of alternative distance metrics and embedding spaces.

A plausible implication is that metrics analogous to PSE-SU4 could be adapted for other domains where the subjective alignment between system outputs and user preferences is essential, thereby broadening the applicability of personalized evaluation frameworks.

6. Significance in the Landscape of Personalized Summarization Evaluation

PSE-SU4 establishes a domain- and task-relevant grounds for measuring the effectiveness of personalization in automatic summarization. By emphasizing the subjective match between user expectations and summarizer output, PSE-SU4 addresses evaluation gaps unfilled by prior metrics, providing a rigorous tool for both benchmarking and driving methodological improvements in personalized NLP systems.

In summary, PSE-SU4 represents a substantive advancement in the quantification of personalization for text summarization, offering a ratio-based, skip-bigram–sensitive metric that directly evaluates the subjective suitability of generated summaries for individual users. Its adoption, particularly in conjunction with data augmentation methods such as PerAugy, enables the principled evaluation and development of more responsive, user-tailored NLP systems.