---
title: 'BertSumExt: Extractive Summarization Model'
url: https://www.emergentmind.com/topics/bertsumext
type: topic
---

# BertSumExt: Extractive Summarization Model

BertSumExt is an extractive text summarization model based on a document-level BERT encoder with additional inter-sentence Transformer layers. Developed to address the need for effective integration of pretrained language models in summarization, BertSumExt constructs sentence-level representations and models sentence interaction at the document level, serving as a key module in both extractive and hybrid extractive-abstractive summarization frameworks [1908.08345].

## 1. Architectural Overview

The model operates on documents composed of sentences \( sent_1, ..., sent_m \), preprocessing them by inserting a special \(\mathtt{[CLS]}\) token before each sentence and a \(\mathtt{[SEP]}\) token after each. The resulting sequence is 
\[
X = [\,\mathtt{[CLS]},\,w_1^{(1)},\ldots,w_{n_1}^{(1)},\mathtt{[SEP]},\;\mathtt{[CLS]},\,w_1^{(2)},\ldots,w_{n_2}^{(2)},\mathtt{[SEP]},\;\dots].
\]
Each token position \(i\) is embedded using a sum of token, segment (interval), and position embeddings:
\[
x_i = E_{\text{token}(w_i)} + E_{\text{segment}(s_i)} + E_{\text{position}(i)}.
\]
"Interval embeddings" alternate between two learned vectors \(E_A\) and \(E_B\) for odd/even sentences to support document-level modeling.

After token and embedding processing, the input passes through a multi-layer (typically 12-layer) pretrained BERT encoder. To represent sentences, the output from the position of each inserted \(\mathtt{[CLS]}\) token is extracted as
\[
t_j = h_{\text{pos}(\mathtt{[CLS]}_j)}^{(L_B)}, \quad j=1\ldots m.
\]
On top of these sentence vectors, \(L_E=2\) Transformer encoder layers are stacked with sinusoidal positional encodings \(\mathrm{PE}(j)\) following the formulation of Vaswani et al. The input for the inter-sentence Transformer is
\[
h_j^{(0)} = t_j + \mathrm{PE}(j).
\]

## 2. Mathematical Formulation and Sentence Selection

The inter-sentence layers process the stacked sentence representations as follows. For each layer \(l\):
\[
\tilde{h}^{(l)} = \mathrm{LayerNorm}\left(h^{(l-1)} + \mathrm{MultiHeadAttn}(h^{(l-1)}, h^{(l-1)}, h^{(l-1)})\right),
\]
\[
h^{(l)} = \mathrm{LayerNorm}\left(\tilde{h}^{(l)} + \mathrm{FFN}(\tilde{h}^{(l)})\right),
\]
where \(\mathrm{FFN}\) is a two-layer MLP with inner size 2048, and multi-head attention uses \(H\) heads with a hidden size \(d=768\).

After \(L_E\) Transformer layers, sentence-level extractive classification is performed by computing for each sentence \(j\):
\[
\hat{y}_j = \sigma(W_o h_j^{(L_E)} + b_o),
\]
with \(\sigma\) as the logistic sigmoid. At inference, sentences are ranked by \(\hat{y}_j\), and the top 3 are selected with trigram-blocking to reduce redundancy.

The extractive objective function is binary cross-entropy over sentences:
\[
\mathcal{L}_{\text{ext}} = -\sum_{j=1}^m [y_j \log \hat{y}_j + (1-y_j)\log(1-\hat{y}_j)].
\]

## 3. Fine-Tuning Regime and Optimization

BertSumExt is trained in a two-stage regime when incorporated in the BertSumExtAbs pipeline: first, extractive fine-tuning of both the BERT encoder and inter-sentence Transformer on labeled extractive summarization data. Subsequently, this extractively fine-tuned encoder initializes an encoder-decoder model for abstractive summarization.

Distinct optimizers are used in stage two to reconcile the pretrained (encoder) and randomly initialized (decoder) model components. Both use Adam (\(\beta_1=0.9, \beta_2=0.999\)), but with different learning rates and schedules. The learning schedule is the “warmup & inverse square-root” schedule:
\[
\mathrm{lr}(\text{step}) = \tilde{\mathrm{lr}} \cdot \min(\text{step}^{-0.5},\, \text{step} \cdot \mathrm{warmup}^{-1.5}).
\]
For the extractive stage, \(\tilde{\mathrm{lr}}=2\times 10^{-3}\), \(\mathrm{warmup}=10{,}000\); for the abstractive stage, encoder and decoder have separate \(\tilde{\mathrm{lr}}_{\mathcal E}=2\times10^{-3}\) (\(\mathrm{warmup}_{\mathcal E}=20,000\)) and \(\tilde{\mathrm{lr}}_{\mathcal D}=0.1\) (\(\mathrm{warmup}_{\mathcal D}=10,000\)).

## 4. Datasets, Preprocessing, and Training Protocol

BertSumExt and its variants are evaluated on multiple large-scale summarization datasets:
- **CNN/DailyMail**: 90,266/1,220/1,093 (CNN) and 196,961/12,148/10,397 (DM) train/validation/test splits; processed with CoreNLP sentence splitting; documents truncated to 512 tokens.
- **NYT50**: 96,834/4,000/3,452, articles ≥50 words, documents truncated to 800 tokens.
- **XSum**: 204,045/11,332/11,334 split, documents truncated to 512 tokens.

All datasets use BERT’s WordPiece tokenization and are implemented in PyTorch/OpenNMT using bert-base-uncased.

Training uses 50K steps with gradient accumulation (steps: 2 for extractive, 5 for abstractive), dropout 0.1, and label-smoothing 0.1 in the decoder.

## 5. Empirical Results

BertSumExt achieves superior ROUGE F1 performance compared to prior baselines on CNN/DailyMail and NYT50. The BertSumExtAbs two-stage model, which leverages BertSumExt for extractive pretraining, further improves the results.

### ROUGE F1 on CNN/DailyMail

| Model             | R-1   | R-2   | R-L   |
|-------------------|-------|-------|-------|
| Oracle            | 52.59 | 31.24 | 48.87 |
| Lead-3            | 40.42 | 17.62 | 36.67 |
| TransformerExt    | 40.90 | 18.02 | 37.17 |
| BertSumExt        | 43.25 | 20.24 | 39.63 |
| BertSumExt (large)| 43.85 | 20.34 | 39.90 |
| BertSumAbs        | 41.72 | 19.39 | 38.76 |
| BertSumExtAbs     | 42.13 | 19.60 | 39.18 |

On NYT50, BertSumExt achieves ROUGE-1 = 46.66, ROUGE-2 = 26.35, ROUGE-L = 42.62; on XSum, BertSumExtAbs achieves ROUGE-1 = 38.81, ROUGE-2 = 16.50, ROUGE-L = 31.27. Other published extractive and abstractive baselines are surpassed except the oracle.

## 6. Ablation and Analysis

Ablation studies indicate that interval (segment) embeddings provide a measurable benefit (removing them drops ROUGE-1 by approximately 0.05 on CNN/DM). Varying the number of inter-sentence Transformer layers reveals that \(L_E=2\) achieves optimal results relative to 1 or 3 layers. The two-stage fine-tuning regime (BertSumExtAbs) outperforms single-stage training (BertSumAbs) by approximately 0.4 ROUGE-1 points on NYT. Separate optimizer schedules with learning rates \(\tilde{lr}_{\mathcal E}=2\times10^{-3}\), \(\tilde{lr}_{\mathcal D}=0.1\) produce best perplexity on validation.

## 7. Human Evaluation and Significance

Human evaluations using QA-based informativeness (following Narayan et al., 2018) and Best–Worst scaling for Informativeness/Fluency/Succinctness show that BertSumExtAbs and BertSumAbs are preferred to all prior extractive and abstractive baselines with statistical significance (\(p<0.05\)). These findings corroborate the automatic ROUGE-based results, establishing the model’s effectiveness on multi-document and single-document summarization tasks across diverse domains [1908.08345].

Source: https://www.emergentmind.com/topics/bertsumext