BertSumExt: Extractive Summarization Model

• The paper demonstrates how BertSumExt leverages a document-level BERT encoder and inter-sentence Transformer layers to create robust sentence representations for extractive summarization.
• BertSumExt is an extractive summarization model that preprocesses texts with [CLS] tokens and interval embeddings, effectively modeling sentence interactions at the document level.
• Empirical results on datasets like CNN/DailyMail and NYT50 verify that BertSumExt’s architecture and fine-tuning regime significantly improve ROUGE scores, with ablation studies underscoring the value of interval embeddings.

BertSumExt is an extractive text summarization model based on a document-level BERT encoder with additional inter-sentence Transformer layers. Developed to address the need for effective integration of pretrained LLMs in summarization, BertSumExt constructs sentence-level representations and models sentence interaction at the document level, serving as a key module in both extractive and hybrid extractive-abstractive summarization frameworks (Liu et al., 2019).

1. Architectural Overview

The model operates on documents composed of sentences $sent_1, ..., sent_m$, preprocessing them by inserting a special $\mathtt{[CLS]}$ token before each sentence and a $\mathtt{[SEP]}$ token after each. The resulting sequence is

$$X = [\,\mathtt{[CLS]},\,w_1^{(1)},\ldots,w_{n_1}^{(1)},\mathtt{[SEP]},\;\mathtt{[CLS]},\,w_1^{(2)},\ldots,w_{n_2}^{(2)},\mathtt{[SEP]},\;\dots].$$

Each token position $i$ is embedded using a sum of token, segment (interval), and position embeddings: $$x_i = E_{\text{token}(w_i)} + E_{\text{segment}(s_i)} + E_{\text{position}(i)}.$$ "Interval embeddings" alternate between two learned vectors $E_A$ and $E_B$ for odd/even sentences to support document-level modeling.

After token and embedding processing, the input passes through a multi-layer (typically 12-layer) pretrained BERT encoder. To represent sentences, the output from the position of each inserted $\mathtt{[CLS]}$ token is extracted as

$$t_j = h_{\text{pos}(\mathtt{[CLS]}_j)}^{(L_B)}, \quad j=1\ldots m.$$

On top of these sentence vectors, $\mathtt{[CLS]}$0 Transformer encoder layers are stacked with sinusoidal positional encodings $\mathtt{[CLS]}$1 following the formulation of Vaswani et al. The input for the inter-sentence Transformer is

$\mathtt{[CLS]}$2

2. Mathematical Formulation and Sentence Selection

The inter-sentence layers process the stacked sentence representations as follows. For each layer $\mathtt{[CLS]}$3: $\mathtt{[CLS]}$4

$\mathtt{[CLS]}$5

where $\mathtt{[CLS]}$6 is a two-layer MLP with inner size 2048, and multi-head attention uses $\mathtt{[CLS]}$7 heads with a hidden size $\mathtt{[CLS]}$8.

After $\mathtt{[CLS]}$9 Transformer layers, sentence-level extractive classification is performed by computing for each sentence $\mathtt{[SEP]}$0: $\mathtt{[SEP]}$1 with $\mathtt{[SEP]}$2 as the logistic sigmoid. At inference, sentences are ranked by $\mathtt{[SEP]}$3, and the top 3 are selected with trigram-blocking to reduce redundancy.

The extractive objective function is binary cross-entropy over sentences: $\mathtt{[SEP]}$4

3. Fine-Tuning Regime and Optimization

BertSumExt is trained in a two-stage regime when incorporated in the BertSumExtAbs pipeline: first, extractive fine-tuning of both the BERT encoder and inter-sentence Transformer on labeled extractive summarization data. Subsequently, this extractively fine-tuned encoder initializes an encoder-decoder model for abstractive summarization.

Distinct optimizers are used in stage two to reconcile the pretrained (encoder) and randomly initialized (decoder) model components. Both use Adam ($\mathtt{[SEP]}$5), but with different learning rates and schedules. The learning schedule is the “warmup & inverse square-root” schedule: $\mathtt{[SEP]}$6 For the extractive stage, $\mathtt{[SEP]}$7, $\mathtt{[SEP]}$8; for the abstractive stage, encoder and decoder have separate $\mathtt{[SEP]}$9 ($$X = [\,\mathtt{[CLS]},\,w_1^{(1)},\ldots,w_{n_1}^{(1)},\mathtt{[SEP]},\;\mathtt{[CLS]},\,w_1^{(2)},\ldots,w_{n_2}^{(2)},\mathtt{[SEP]},\;\dots].$$0) and $$X = [\,\mathtt{[CLS]},\,w_1^{(1)},\ldots,w_{n_1}^{(1)},\mathtt{[SEP]},\;\mathtt{[CLS]},\,w_1^{(2)},\ldots,w_{n_2}^{(2)},\mathtt{[SEP]},\;\dots].$$1 ($$X = [\,\mathtt{[CLS]},\,w_1^{(1)},\ldots,w_{n_1}^{(1)},\mathtt{[SEP]},\;\mathtt{[CLS]},\,w_1^{(2)},\ldots,w_{n_2}^{(2)},\mathtt{[SEP]},\;\dots].$$2).

4. Datasets, Preprocessing, and Training Protocol

BertSumExt and its variants are evaluated on multiple large-scale summarization datasets:

• CNN/DailyMail: 90,266/1,220/1,093 (CNN) and 196,961/12,148/10,397 (DM) train/validation/test splits; processed with CoreNLP sentence splitting; documents truncated to 512 tokens.
• NYT50: 96,834/4,000/3,452, articles ≥50 words, documents truncated to 800 tokens.
• XSum: 204,045/11,332/11,334 split, documents truncated to 512 tokens.

All datasets use BERT’s WordPiece tokenization and are implemented in PyTorch/OpenNMT using bert-base-uncased.

Training uses 50K steps with gradient accumulation (steps: 2 for extractive, 5 for abstractive), dropout 0.1, and label-smoothing 0.1 in the decoder.

5. Empirical Results

BertSumExt achieves superior ROUGE F1 performance compared to prior baselines on CNN/DailyMail and NYT50. The BertSumExtAbs two-stage model, which leverages BertSumExt for extractive pretraining, further improves the results.

ROUGE F1 on CNN/DailyMail

Model	R-1	R-2	R-L
Oracle	52.59	31.24	48.87
Lead-3	40.42	17.62	36.67
TransformerExt	40.90	18.02	37.17
BertSumExt	43.25	20.24	39.63
BertSumExt (large)	43.85	20.34	39.90
BertSumAbs	41.72	19.39	38.76
BertSumExtAbs	42.13	19.60	39.18

On NYT50, BertSumExt achieves ROUGE-1 = 46.66, ROUGE-2 = 26.35, ROUGE-L = 42.62; on XSum, BertSumExtAbs achieves ROUGE-1 = 38.81, ROUGE-2 = 16.50, ROUGE-L = 31.27. Other published extractive and abstractive baselines are surpassed except the oracle.

6. Ablation and Analysis

Ablation studies indicate that interval (segment) embeddings provide a measurable benefit (removing them drops ROUGE-1 by approximately 0.05 on CNN/DM). Varying the number of inter-sentence Transformer layers reveals that $$X = [\,\mathtt{[CLS]},\,w_1^{(1)},\ldots,w_{n_1}^{(1)},\mathtt{[SEP]},\;\mathtt{[CLS]},\,w_1^{(2)},\ldots,w_{n_2}^{(2)},\mathtt{[SEP]},\;\dots].$$3 achieves optimal results relative to 1 or 3 layers. The two-stage fine-tuning regime (BertSumExtAbs) outperforms single-stage training (BertSumAbs) by approximately 0.4 ROUGE-1 points on NYT. Separate optimizer schedules with learning rates $$X = [\,\mathtt{[CLS]},\,w_1^{(1)},\ldots,w_{n_1}^{(1)},\mathtt{[SEP]},\;\mathtt{[CLS]},\,w_1^{(2)},\ldots,w_{n_2}^{(2)},\mathtt{[SEP]},\;\dots].$$4, $$X = [\,\mathtt{[CLS]},\,w_1^{(1)},\ldots,w_{n_1}^{(1)},\mathtt{[SEP]},\;\mathtt{[CLS]},\,w_1^{(2)},\ldots,w_{n_2}^{(2)},\mathtt{[SEP]},\;\dots].$$5 produce best perplexity on validation.

7. Human Evaluation and Significance

Human evaluations using QA-based informativeness (following Narayan et al., 2018) and Best–Worst scaling for Informativeness/Fluency/Succinctness show that BertSumExtAbs and BertSumAbs are preferred to all prior extractive and abstractive baselines with statistical significance ($$X = [\,\mathtt{[CLS]},\,w_1^{(1)},\ldots,w_{n_1}^{(1)},\mathtt{[SEP]},\;\mathtt{[CLS]},\,w_1^{(2)},\ldots,w_{n_2}^{(2)},\mathtt{[SEP]},\;\dots].$$6). These findings corroborate the automatic ROUGE-based results, establishing the model’s effectiveness on multi-document and single-document summarization tasks across diverse domains (Liu et al., 2019).