---
title: 'BertSumAbs: Abstractive Summarization Model'
url: https://www.emergentmind.com/topics/bertsumabs
type: topic
---

# BertSumAbs: Abstractive Summarization Model

BertSumAbs is an abstractive text summarization model that integrates Bidirectional Encoder Representations from Transformers (BERT) as a document-level encoder and stacks a Transformer-based decoder for summary generation. Developed by Liu and Lapata, BertSumAbs also underpins state-of-the-art systems for news headline generation and general document summarization, utilizing a two-stage fine-tuning schedule with independent optimizers for encoder and decoder components. The architecture generalizes across languages and domains, as evidenced by adaptations to both English and Russian corpora [1908.08345], [2007.05044].

## 1. Model Architecture

### 1.1 Document-Level Encoder

BertSumAbs employs the bert-base-uncased model or a language-specific variant (e.g., RuBERT), typically retaining 12 Transformer layers (or only the first 6 for certain downstream languages such as Russian [2007.05044]), hidden size 768, and 12 heads. For documents with $m$ sentences $[sent_1, \ldots, sent_m]$:

- Each sentence is prepended with a special [CLS] token and terminated with [SEP], yielding the input format:
  ```
  [CLS] sent₁ [SEP] [CLS] sent₂ [SEP] ... [CLS] sentₘ [SEP]
  ```
- Embeddings per token are computed as: $e_i = E_{tok}(x_i) + E_{pos}(i) + E_{seg}(s_i)$, with $E_{tok}$ the WordPiece embedding, $E_{pos}$ positional embedding (extended or sinusoidal if required), and $E_{seg}(s_i)$ an interval segment embedding alternating per sentence (E_A/E_B or 0/1) [1908.08345], [2007.05044].
- Through 12 Transformer layers, each using multi-head self-attention and feed-forward sublayers as in Vaswani et al. (2017), contextual encoder states are produced:
  $$
  \tilde{h}^\ell = LN(h^{\ell-1} + MHAtt(h^{\ell-1}))
  $$
  $$
  h^\ell = LN(\tilde{h}^\ell + FFN(\tilde{h}^\ell))
  $$
- Sentence representations $T = [t_1, ..., t_m]$ are extracted as the final-layer [CLS] positions, $t_i \in \mathbb{R}^{768}$.

### 1.2 Transformer Decoder

BertSumAbs utilizes a standard 6-layer Transformer decoder, composed of the following in each layer:

- Masked self-attention (causal for summary generation): $Z'^\ell = LN(Z^{\ell-1} + MHAtt(Z^{\ell-1}, Z^{\ell-1}, Z^{\ell-1}))$.
- Cross-attention to all encoder output states (not just [CLS]):
  $$
  Z''^\ell = LN(Z'^\ell + MHAtt(Z'^\ell, H^L, H^L))
  $$
- Feed-forward sublayer:
  $$
  Z^\ell = LN(Z''^\ell + FFN(Z''^\ell))
  $$
- At each decoding step $j$, the output $o_j \in \mathbb{R}^{768}$ is projected to the vocabulary via:
  $$
  P(y_j|y_{<j}, X) = softmax(W_o o_j + b_o)
  $$
  where $W_o \in \mathbb{R}^{|V| \times 768}$ [1908.08345], [2007.05044].

### 1.3 Encoder–Decoder Interface

The encoder and decoder are jointly fine-tuned, with the decoder attending over token-level encoder outputs. This enables the model to generate fluent abstractive sequences not constrained to extraction or copying [1908.08345].

## 2. Training Regime and Fine-Tuning Procedures

### 2.1 Two-Stage Fine-Tuning

BertSumAbs adopts a two-stage schedule designed to stabilize training and exploit available supervision:

- **Stage 1: Extractive fine-tuning**. Two inter-sentence Transformer layers are appended atop the sentence embeddings $T$, with additional positional embeddings for sentence order. Each sentence receives a binary label predicted by a classifier $\hat{y}_i = \sigma(W_o h_i^L + b_o)$ and trained via binary cross-entropy:
  $$
  Loss_1 = -\sum_{i=1}^m \left[ y_i \log \hat{y}_i + (1-y_i) \log (1-\hat{y}_i) \right]
  $$
  Encoder parameters are updated accordingly to yield $\theta_E^{(1)}$.

- **Stage 2: Abstractive fine-tuning**. The decoder parameters $\theta_D$ are randomly initialized, while the encoder is loaded from $\theta_E^{(1)}$. The objective becomes cross-entropy over the summary/target tokens:
  $$
  Loss_2 = -\sum_{j=1}^{|y|} \log P(y_j | y_{<j}, X; \theta_E, \theta_D)
  $$
  [1908.08345]

### 2.2 Optimizer and Hyperparameter Scheduling

Distinct Adam optimizers are used for encoder and decoder, with different learning rates and warm-up periods:

- **Encoder optimizer**: 
  $$
  lr_E(step) = 2 \times 10^{-3} \cdot \min(step^{-0.5}, step \cdot warmup_E^{-1.5}),\quad warmup_E=20{,}000
  $$
- **Decoder optimizer**:
  $$
  lr_D(step) = 1 \times 10^{-1} \cdot \min(step^{-0.5}, step \cdot warmup_D^{-1.5}),\quad warmup_D=10{,}000
  $$

This separation addresses the mismatch between the pretrained (encoder) and randomly initialized (decoder) components, facilitating more stable convergence [1908.08345]. In Russian headline generation, learning rates are tuned to $2 \times 10^{-5}$ (encoder) and $2 \times 10^{-3}$ (decoder), with Adam ($\beta_1=0.9$, $\beta_2=0.999$, $\epsilon=1e$-6) and weight decay 0.01 [2007.05044].

### 2.3 Key Hyperparameters

- **Extractive stage**: 50,000 steps on 3 GPUs, gradient accumulation every 2 steps, batch size $\sim$2 documents/GPU, document truncation to 512 tokens (800 for NYT), checkpoint averaging over top-3 by validation loss. 
- **Abstractive stage**: 200,000 steps on 4 GPUs, gradient accumulation every 5 steps, beam search (size 5, length penalty $\alpha$ tuned in [0.6, 1.0]), dropout 0.1, label smoothing 0.1 [1908.08345], [2007.05044].

## 3. Mathematical Formulation

### 3.1 Self-Attentive Encoder/Decoder

Both encoder and decoder use multi-head attention and feed-forward blocks with residual and layer normalization, as defined:

- **Scaled dot-product attention**:
  $$
  Attention(Q,K,V) = softmax \left( \frac{QK^\top}{\sqrt{d_k}} \right) V
  $$
- **Multi-head attention**:
  $$
  head_i = Attention(Q W_i^Q, K W_i^K, V W_i^V),\quad MultiHead(Q,K,V) = Concat(\text{head}_1, ..., \text{head}_h) W^O
  $$
- **Token and sentence encoding**:
  $$
  h^0 = input~embeddings; \quad h^\ell = TransformerLayer(h^{\ell-1})
  $$
- **Decoder prediction probability**:
  $$
  p(y_t | y_{<t}, X) = softmax(W_o s_t + b_o)
  $$
- **Training losses**: binary cross-entropy for extraction, standard cross-entropy for headline/summarization generation [1908.08345], [2007.05044].

## 4. Empirical Results

### 4.1 Main Results on Benchmark Summarization

#### CNN/DailyMail, NYT50, XSum (Test F$_1$)

| Model              | R-1   | R-2   | R-L   |
|--------------------|-------|-------|-------|
| BertSumAbs         | 41.72 | 19.39 | 38.76 |
| BertSumExtAbs      | 42.13 | 19.60 | 39.18 |
| BertSumExt         | 43.25 | 20.24 | 39.63 |

- On CNN/DailyMail, BertSumAbs outperforms BottomUp (41.22/18.68/38.34) and previous extractive models [1908.08345].
- On NYT50 (recall-limited), BertSumAbs achieves 48.92/30.84/45.41, improving over PTGen+Cov and CopyNet.
- On XSum, BertSumAbs delivers 38.76/16.33/31.15.

### 4.2 News Headline Generation (Russian)

| Model       | R-mean (RIA) | BLEU (RIA) | R-mean (Lenta) | BLEU (Lenta) |
|-------------|--------------|------------|----------------|--------------|
| CopyNet     | 35.0         | 53.8       | 22.7           | 40.4         |
| mBART       | 36.1         | 55.1       | 24.0           | 43.2         |
| BertSumAbs  | 39.0         | 57.6       | 24.7           | 45.1         |

- BertSumAbs increases ROUGE-mean by 2.9 (RIA) and 2.0 (Lenta) over prior SOTA models [2007.05044].

### 4.3 Ablation Studies

- **Interval embeddings**: Removing interval segment embeddings has negligible effect on performance.
- **Encoder size**: Scaling to Bert-large increases ROUGE-1 by ~0.6.
- **Inter-sentence layers**: Best performance with $L=2$ layers; deviation from 2 reduces effectiveness.
- **Learning rates**: Decoder learning rate must be substantially larger than encoder for stability; improper tuning leads to high perplexity [1908.08345], [2007.05044].
- **Novel n-gram rate**: BertSumAbs produces more novel n-grams in headlines versus mBART, confirming greater abstractive capacity but also increased risk of hallucination [2007.05044].

## 5. Evaluation Protocols and Metrics

- **ROUGE-N (F$_1$)**: $R_N = \text{Overlap}_N / \text{Total}_N^\text{ref}$; ROUGE-N F$_1 = 2R_N P_N / (R_N+P_N)$.
- **ROUGE-L (F$_1$)**: Based on LCS between system and reference.
- **R-mean**: Mean of F$_1$ for ROUGE-1,2,L.
- **BLEU**: n-gram precision with brevity penalty.
- **Novel n-gram rate**: Fraction of novel n-grams in output relative to source.
- **Evaluation splits**: Macro-averaged scores over withhold (test) splits; for Lenta dataset, zero-shot transfer evaluated after training on RIA [2007.05044].

## 6. Applications, Limitations, and Best Practices

- **Applications**: BertSumAbs applies to general abstractive summarization and domain-specific tasks such as news headline generation, benefiting from large-scale pretrained encoders and adaptable fine-tuning [1908.08345], [2007.05044].
- **Language/Domain Adaptation**: Using language-specific pretrained encoders (e.g., RuBERT for Russian) significantly benefits in-domain performance versus multilingual encoders [2007.05044].
- **Optimizer Strategy**: Independent optimizer and learning rate schedules for encoder and decoder are crucial when the former is pretrained and latter randomly initialized.
- **Abstractive–Extractive Tradeoff**: High novel n-gram rate correlates with more abstractive, but sometimes less faithful, outputs; human evaluation on Russian data yielded 49% preference for BertSumAbs machine headlines and 8% “draw” [2007.05044].
- **Temporal and Domain Bias**: BertSumAbs is sensitive to domain drift and temporal changes (e.g., previously unseen named entities), leading to performance drops on out-of-domain news [2007.05044].
- **Limitations and Research Directions**: Outstanding issues include factual consistency (hallucination of unsupported facts), mitigation of domain drift, and the potential for integrating document-level verification or entity-awareness. Two-stage fine-tuning using headline generation as the first phase is also suggested as best practice before downstream task transfer [2007.05044].

Source: https://www.emergentmind.com/topics/bertsumabs