BertSumAbs: Abstractive Summarization Model

• BertSumAbs is an abstractive summarization model that integrates a BERT-based document encoder with a Transformer decoder for fluent summary generation.
• It employs a two-stage fine-tuning process with independent optimizers to effectively balance the pretrained encoder and the randomly initialized decoder.
• Empirical results demonstrate state-of-the-art ROUGE and BLEU scores on benchmarks, confirming its adaptability across multiple languages and domains.

BertSumAbs is an abstractive text summarization model that integrates Bidirectional Encoder Representations from Transformers (BERT) as a document-level encoder and stacks a Transformer-based decoder for summary generation. Developed by Liu and Lapata, BertSumAbs also underpins state-of-the-art systems for news headline generation and general document summarization, utilizing a two-stage fine-tuning schedule with independent optimizers for encoder and decoder components. The architecture generalizes across languages and domains, as evidenced by adaptations to both English and Russian corpora (Liu et al., 2019, Bukhtiyarov et al., 2020).

1. Model Architecture

1.1 Document-Level Encoder

BertSumAbs employs the bert-base-uncased model or a language-specific variant (e.g., RuBERT), typically retaining 12 Transformer layers (or only the first 6 for certain downstream languages such as Russian (Bukhtiyarov et al., 2020)), hidden size 768, and 12 heads. For documents with $m$ sentences $[sent_1, \ldots, sent_m]$:

• Each sentence is prepended with a special [CLS] token and terminated with [SEP], yielding the input format: $E_{pos}$4
• Embeddings per token are computed as: $e_i = E_{tok}(x_i) + E_{pos}(i) + E_{seg}(s_i)$, with $E_{tok}$ the WordPiece embedding, $E_{pos}$ positional embedding (extended or sinusoidal if required), and $E_{seg}(s_i)$ an interval segment embedding alternating per sentence (E_A/E_B or 0/1) (Liu et al., 2019, Bukhtiyarov et al., 2020).
• Through 12 Transformer layers, each using multi-head self-attention and feed-forward sublayers as in Vaswani et al. (2017), contextual encoder states are produced:

$$\tilde{h}^\ell = LN(h^{\ell-1} + MHAtt(h^{\ell-1}))$$

$h^\ell = LN(\tilde{h}^\ell + FFN(\tilde{h}^\ell))$

• Sentence representations $T = [t_1, ..., t_m]$ are extracted as the final-layer [CLS] positions, $t_i \in \mathbb{R}^{768}$.

1.2 Transformer Decoder

BertSumAbs utilizes a standard 6-layer Transformer decoder, composed of the following in each layer:

• Masked self-attention (causal for summary generation): $[sent_1, \ldots, sent_m]$0.
• Cross-attention to all encoder output states (not just [CLS]):

$[sent_1, \ldots, sent_m]$1

• Feed-forward sublayer:

$[sent_1, \ldots, sent_m]$2

• At each decoding step $[sent_1, \ldots, sent_m]$3, the output $[sent_1, \ldots, sent_m]$4 is projected to the vocabulary via:

$[sent_1, \ldots, sent_m]$5

where $[sent_1, \ldots, sent_m]$6 (Liu et al., 2019, Bukhtiyarov et al., 2020).

1.3 Encoder–Decoder Interface

The encoder and decoder are jointly fine-tuned, with the decoder attending over token-level encoder outputs. This enables the model to generate fluent abstractive sequences not constrained to extraction or copying (Liu et al., 2019).

2. Training Regime and Fine-Tuning Procedures

2.1 Two-Stage Fine-Tuning

BertSumAbs adopts a two-stage schedule designed to stabilize training and exploit available supervision:

• Stage 1: Extractive fine-tuning. Two inter-sentence Transformer layers are appended atop the sentence embeddings $[sent_1, \ldots, sent_m]$7, with additional positional embeddings for sentence order. Each sentence receives a binary label predicted by a classifier $[sent_1, \ldots, sent_m]$8 and trained via binary cross-entropy:

$[sent_1, \ldots, sent_m]$9

Encoder parameters are updated accordingly to yield $e_i = E_{tok}(x_i) + E_{pos}(i) + E_{seg}(s_i)$0.

• Stage 2: Abstractive fine-tuning. The decoder parameters $e_i = E_{tok}(x_i) + E_{pos}(i) + E_{seg}(s_i)$1 are randomly initialized, while the encoder is loaded from $e_i = E_{tok}(x_i) + E_{pos}(i) + E_{seg}(s_i)$2. The objective becomes cross-entropy over the summary/target tokens:

$e_i = E_{tok}(x_i) + E_{pos}(i) + E_{seg}(s_i)$3

(Liu et al., 2019)

2.2 Optimizer and Hyperparameter Scheduling

Distinct Adam optimizers are used for encoder and decoder, with different learning rates and warm-up periods:

• Encoder optimizer:

$e_i = E_{tok}(x_i) + E_{pos}(i) + E_{seg}(s_i)$4

• Decoder optimizer:

$e_i = E_{tok}(x_i) + E_{pos}(i) + E_{seg}(s_i)$5

This separation addresses the mismatch between the pretrained (encoder) and randomly initialized (decoder) components, facilitating more stable convergence (Liu et al., 2019). In Russian headline generation, learning rates are tuned to $e_i = E_{tok}(x_i) + E_{pos}(i) + E_{seg}(s_i)$6 (encoder) and $e_i = E_{tok}(x_i) + E_{pos}(i) + E_{seg}(s_i)$7 (decoder), with Adam ($e_i = E_{tok}(x_i) + E_{pos}(i) + E_{seg}(s_i)$8, $e_i = E_{tok}(x_i) + E_{pos}(i) + E_{seg}(s_i)$9, $E_{tok}$0-6) and weight decay 0.01 (Bukhtiyarov et al., 2020).

2.3 Key Hyperparameters

• Extractive stage: 50,000 steps on 3 GPUs, gradient accumulation every 2 steps, batch size $E_{tok}$12 documents/GPU, document truncation to 512 tokens (800 for NYT), checkpoint averaging over top-3 by validation loss.
• Abstractive stage: 200,000 steps on 4 GPUs, gradient accumulation every 5 steps, beam search (size 5, length penalty $E_{tok}$2 tuned in [0.6, 1.0]), dropout 0.1, label smoothing 0.1 (Liu et al., 2019, Bukhtiyarov et al., 2020).

3. Mathematical Formulation

3.1 Self-Attentive Encoder/Decoder

Both encoder and decoder use multi-head attention and feed-forward blocks with residual and layer normalization, as defined:

• Scaled dot-product attention:

$E_{tok}$3

• Multi-head attention:

$E_{tok}$4

• Token and sentence encoding:

$E_{tok}$5

• Decoder prediction probability:

$E_{tok}$6

• Training losses: binary cross-entropy for extraction, standard cross-entropy for headline/summarization generation (Liu et al., 2019, Bukhtiyarov et al., 2020).

4. Empirical Results

4.1 Main Results on Benchmark Summarization

CNN/DailyMail, NYT50, XSum (Test F$E_{tok}$7)

Model	R-1	R-2	R-L
BertSumAbs	41.72	19.39	38.76
BertSumExtAbs	42.13	19.60	39.18
BertSumExt	43.25	20.24	39.63

• On CNN/DailyMail, BertSumAbs outperforms BottomUp (41.22/18.68/38.34) and previous extractive models (Liu et al., 2019).
• On NYT50 (recall-limited), BertSumAbs achieves 48.92/30.84/45.41, improving over PTGen+Cov and CopyNet.
• On XSum, BertSumAbs delivers 38.76/16.33/31.15.

4.2 News Headline Generation (Russian)

Model	R-mean (RIA)	BLEU (RIA)	R-mean (Lenta)	BLEU (Lenta)
CopyNet	35.0	53.8	22.7	40.4
mBART	36.1	55.1	24.0	43.2
BertSumAbs	39.0	57.6	24.7	45.1

• BertSumAbs increases ROUGE-mean by 2.9 (RIA) and 2.0 (Lenta) over prior SOTA models (Bukhtiyarov et al., 2020).

4.3 Ablation Studies

• Interval embeddings: Removing interval segment embeddings has negligible effect on performance.
• Encoder size: Scaling to Bert-large increases ROUGE-1 by ~0.6.
• Inter-sentence layers: Best performance with $E_{tok}$8 layers; deviation from 2 reduces effectiveness.
• Learning rates: Decoder learning rate must be substantially larger than encoder for stability; improper tuning leads to high perplexity (Liu et al., 2019, Bukhtiyarov et al., 2020).
• Novel n-gram rate: BertSumAbs produces more novel n-grams in headlines versus mBART, confirming greater abstractive capacity but also increased risk of hallucination (Bukhtiyarov et al., 2020).

5. Evaluation Protocols and Metrics

• ROUGE-N (F$E_{tok}$9): $E_{pos}$0; ROUGE-N F$E_{pos}$1.
• ROUGE-L (F$E_{pos}$2): Based on LCS between system and reference.
• R-mean: Mean of F$E_{pos}$3 for ROUGE-1,2,L.
• BLEU: n-gram precision with brevity penalty.
• Novel n-gram rate: Fraction of novel n-grams in output relative to source.
• Evaluation splits: Macro-averaged scores over withhold (test) splits; for Lenta dataset, zero-shot transfer evaluated after training on RIA (Bukhtiyarov et al., 2020).

6. Applications, Limitations, and Best Practices

• Applications: BertSumAbs applies to general abstractive summarization and domain-specific tasks such as news headline generation, benefiting from large-scale pretrained encoders and adaptable fine-tuning (Liu et al., 2019, Bukhtiyarov et al., 2020).
• Language/Domain Adaptation: Using language-specific pretrained encoders (e.g., RuBERT for Russian) significantly benefits in-domain performance versus multilingual encoders (Bukhtiyarov et al., 2020).
• Optimizer Strategy: Independent optimizer and learning rate schedules for encoder and decoder are crucial when the former is pretrained and latter randomly initialized.
• Abstractive–Extractive Tradeoff: High novel n-gram rate correlates with more abstractive, but sometimes less faithful, outputs; human evaluation on Russian data yielded 49% preference for BertSumAbs machine headlines and 8% “draw” (Bukhtiyarov et al., 2020).
• Temporal and Domain Bias: BertSumAbs is sensitive to domain drift and temporal changes (e.g., previously unseen named entities), leading to performance drops on out-of-domain news (Bukhtiyarov et al., 2020).
• Limitations and Research Directions: Outstanding issues include factual consistency (hallucination of unsupported facts), mitigation of domain drift, and the potential for integrating document-level verification or entity-awareness. Two-stage fine-tuning using headline generation as the first phase is also suggested as best practice before downstream task transfer (Bukhtiyarov et al., 2020).