Optimizing Abstractive Summarization With Fine-Tuned PEGASUS

Abstract: Abstractive text summarization is the technique of generating a short and concise summary comprising the salient ideas of a source text without making a subset of the salient sentences from the source text. The introduction of transformer models such as BART, T5, and PEGASUS has made this sort of summarization process more efficient and accurate. The objective of this paper is to fine-tune PEGASUS on the XL-Sum English corpus to achieve a better performance compared to the baseline mT5 model. The performance of the generated summaries from the fine-tuned model is evaluated using the ROUGE metric, which basically compares the auto-generated summaries with human-created summaries. To the best of our knowledge, the results from our fine-tuned PEGASUS model give a state-of-the-art performance on the XL-Sum English Corpus. To quantify the improvement, there is a 4.04% improvement in the ROUGE-1 score, a 15.25% increase in the ROUGE-2 score, and a 3.39% improvement in the ROUGE-L score from the baseline model.

• The paper fine-tunes PEGASUS on 20% of XL-Sum’s English training data for five epochs and achieves ROUGE-1/2/L scores of 39.121, 17.467, and 30.894, respectively.
• PEGASUS outperforms the mT5 baseline by 4.04% on ROUGE-1, 15.25% on ROUGE-2, and 3.39% on ROUGE-L, suggesting stronger phrase-level content alignment in news summaries.
• The results are promising but require cautious interpretation because the study uses limited training data, a small evaluation split, and no human, factuality, or statistical significance testing.

Overview

This paper reports a fine-tuning study of PEGASUS for abstractive summarization on the English portion of the XL-Sum corpus (2606.25462). The authors, from BRAC University, fine-tune the pre-trained PEGASUS checkpoint on XL-Sum English data and compare against the mT5-based baseline released by Hasan et al. with the original XL-Sum dataset (Nan et al., 2021). The central claim is that the resulting model, named pegasus_xlsum, achieves state-of-the-art ROUGE performance on the XL-Sum English corpus, exceeding the mT5 baseline on all three ROUGE variants.

Background and motivation

The paper situates itself within the transition from attention-augmented sequence-to-sequence summarizers to transformer-based pre-trained models such as BART, T5, and PEGASUS. The authors note two recurring weaknesses of earlier seq2seq systems: redundant terminology and inconsistent handling of rare terms, as well as slow sequential processing that the transformer's self-attention mitigates through parallel computation over the input sequence.

PEGASUS is described in some detail: it is a sequence-to-sequence transformer pre-trained with gap-sentence generation (GSG), in which salient sentences are masked from the input document and generated jointly from the remaining text — an objective deliberately aligned with downstream summarization. The original work by Zhang et al. introduced PEGASUS_BASE (223M parameters) and PEGASUS_LARGE (568M parameters), with the latter surpassing prior state of the art across twelve downstream summarization datasets (Feng et al., 2020). The literature review also covers extractive-then-abstractive hybrid approaches for long documents (Simoneau et al., 2020) and entity-level factual consistency, noting that models trained on existing datasets exhibit entity hallucination and that data filtering plus supplementary factuality metrics are candidate remedies (Nan et al., 2021). Notably, the paper does not evaluate factual consistency of its own outputs; this remains an unaddressed concern given the acknowledged hallucination problem.

Dataset

XL-Sum is a large-scale multilingual abstractive summarization corpus of over one million BBC article–summary pairs spanning 44 languages, many of which lack other publicly available summarization resources. The newer version adds Traditional Chinese, making it the largest publicly available summarization dataset at the time. The English split is unusual relative to most languages: while most languages use an 80/10/10 train/dev/test split, English uses 93% training, 3.5% development, and 3.5% test. This matters for interpreting the results, since the evaluation set is comparatively small.

Fine-tuning setup

Fine-tuning used 20% of the English training corpus rather than the full split, with the following configuration:

Hyperparameter	Value
Learning rate	2e-05
Training / eval batch size	8 / 8
Optimizer	Adam ($\beta_1=0.9$, $\beta_2=0.999$, $\epsilon=1\text{e-}08$)
LR scheduler	Linear
Epochs	5
Seed	42

The authors state these values were selected via parameter sweeps, though no sweep results or ablations are reported. Early stopping was implemented but, due to resource constraints, training was capped at five epochs regardless.

Results

Evaluation uses ROUGE against human-written reference summaries. The comparison is against the mT5 model fine-tuned on XL-Sum by Hasan et al., which reported ROUGE-1/ROUGE-2/ROUGE-L of 37.60/15.15/29.88.

Model	ROUGE-1	ROUGE-2	ROUGE-L
Baseline mT5	37.601	15.153	29.880
Fine-tuned PEGASUS	39.121	17.467	30.894

Relative to the baseline, this corresponds to a 4.04% improvement in ROUGE-1, a 15.25% improvement in ROUGE-2, and a 3.39% improvement in ROUGE-L. The disproportionately large ROUGE-2 gain suggests improved capture of bigram-level content overlap, which the authors interpret as better handling of complex sentence structure and content relationships. On the strength of these numbers, the authors claim state-of-the-art performance on the XL-Sum English corpus; this claim rests solely on comparison with the single mT5 baseline, and no statistical significance testing or human evaluation is provided to support it.

Limitations

The paper is explicit about several constraints. First, computational resources forced the use of only 20% of the English training corpus and termination after five epochs, despite the convention of longer fine-tuning schedules; the reported scores therefore likely understate what full-corpus, longer-schedule training would achieve, but also mean the "state-of-the-art" claim is established under a reduced training regime. Second, the model degrades on short inputs: coherent summaries require at least several lines of source text, limiting applicability to brief documents. Third, as noted above, the evaluation is confined to ROUGE; no human assessment, factuality check, or significance test accompanies the headline numbers, so hallucination behavior — a problem the paper itself highlights in its literature review — is unmeasured.

Conclusion

This paper demonstrates that PEGASUS, fine-tuned with a modest compute budget on a fraction of the XL-Sum English training data, outperforms the mT5 baseline across ROUGE-1, ROUGE-2, and ROUGE-L, with the largest relative gain in ROUGE-2 (+15.25%). The result supports gap-sentence-generation pre-training as a strong initialization for news-style abstractive summarization. Open questions left by the work include whether the gains hold when trained on the full English corpus with extended schedules, how the model behaves on short inputs, and whether its summaries satisfy factual-consistency criteria beyond n-gram overlap.