BertSumExtAbs: Unified Summarization Model

• The paper demonstrates a unified approach by combining a pretrained BERT encoder for extractive sentence scoring with an abstractive decoder to enhance summarization performance.
• It employs document-level representations using alternating segment embeddings and shallow Transformer layers to effectively capture inter-sentence relationships.
• A two-stage fine-tuning strategy with distinct optimizers and trigram-blocking is used to mitigate redundancy and refine summary generation.

BertSumExtAbs is an encoder–decoder neural text summarization framework that incorporates a pretrained BERT encoder for document representation and leverages both extractive and abstractive objectives for summary generation. It was proposed by Yang and Zhang (2019) to address limitations in prior approaches by unifying document-level semantic representation with flexible fine-tuning schemes, enabling state-of-the-art performance in both extractive and abstractive summarization tasks (Liu et al., 2019).

1. Document-Level BERT Encoder

BertSumExtAbs processes input documents using a document-level BERT encoder, referred to as “BertSum.” The input consists of individual sentences $$\text{sent}_1, \text{sent}_2, \ldots, \text{sent}_m$$ concatenated into a single token sequence. Each sentence is preceded by a special [CLS] token and followed by a [SEP] token. “Interval” segment embeddings alternate between $E_A$ and $E_B$ across sentences—odd sentences are assigned $E_A$, even sentences $E_B$—enabling BERT’s two-segment embedding scheme to distinguish sentences in a single pass.

For sequences that exceed BERT's default 512-token positional embedding capacity, extra positions are randomly initialized and learned during fine-tuning. After passing through $L_B$ layers of BERT, a token-level representation $t_i$ is obtained for every position. Each sentence is represented by the output at its corresponding [CLS] position, $s_i = t_{[CLS]_i} \in \mathbb{R}^d$ with $d=768$ for BERT-base.

2. Extractive Summarization Module (BertSumExt)

Extractive summarization is realized by stacking shallow inter-sentence Transformer layers (depth $L_T$, best results at $E_A$0) atop the set of sentence embeddings $E_A$1. Learned sinusoidal positional embeddings are added to each $E_A$2, producing $E_A$3 for $E_A$4. For each layer $E_A$5, a standard Transformer block is applied: $E_A$6

$E_A$7

On the final layer, each $E_A$8 is passed through a sigmoid classifier: $E_A$9 where $E_B$0. The extractive loss is the binary cross-entropy over all sentences: $E_B$1 with $E_B$2 indicating gold inclusion in the summary.

During inference, all sentences are scored and the top–$E_B$3 (e.g., $E_B$4 for CNN/DM) are selected after trigram-blocking for redundancy removal—a sentence is skipped if it shares any trigram with those already selected. The optimizer is Adam with $E_B$5, $E_B$6 and learning rate schedule

$E_B$7

with $E_B$8 steps.

3. Abstractive Summarization and Two-Stage Fine-Tuning

The abstractive summarization setup extends BertSumExt by adding a standard 6-layer Transformer decoder ($E_B$9, $E_A$0) with masked multi-head self-attention, multi-head encoder–decoder attention over all token-level BERT outputs $E_A$1, and positionwise feed-forward sublayers. Dropout of 0.1 is applied before all linear transforms and label smoothing $E_A$2 is used.

The training objective maximizes the log-likelihood of the reference summary tokens $E_A$3 given the source: $E_A$4 No explicit pointer-generator or coverage loss is used. The model instead mitigates repetition via subword vocabulary and trigram-blocking during beam search inference.

BertSumExtAbs introduces a two-stage fine-tuning schedule:

• Stage 1 (optional): Fine-tune the shared BERT encoder and inter-sentence layers solely on $E_A$5 (extractive objective). The resulting encoder parameters prime the initialization for the abstractive phase.
• Stage 2: Jointly fine-tune the encoder and decoder on $E_A$6 using separate Adam optimizers: encoder ($E_A$7, warmup$E_A$8 steps) and decoder ($E_A$9, warmup$E_B$0 steps), both with the "noam" learning rate schedule. The rationale is to provide a smaller, slower-decaying learning rate to the pretrained encoder while enabling rapid adaptation of the randomly initialized decoder.

Training details: 4$E_B$11080Ti GPUs, gradient accumulation every 5 steps, 200K total training steps, checkpointing every 2.5K steps. Inference uses beam search size 5, length-penalty $E_B$2 in $E_B$3, and trigram-blocking.

4. Hyper-Parameter Configuration

The table below highlights the primary hyper-parameter decisions for both extractive (“Ext”) and abstractive (“Abs” or “ExtAbs”) modules:

Parameter	Extractive (BertSumExt)	Abstractive (BertSumAbs/ExtAbs)
Optimizer	Adam ($E_B$4=0.9,0.999)	Encoder: Adam ($E_B$5=0.9,0.999); Decoder: Adam ($E_B$6=0.9,0.999)
LR, Warmup	$E_B$7=2e-3, warmup=10K	Encoder: $E_B$8=2e-3, warmup=20K; Decoder: $E_B$9=0.1, warmup=10K
Inter-sentence layers $L_B$0	2	2
Decoder layers	–	6
$L_B$1	768	768
$L_B$2	3072	2048
Attention heads	12	12
Dropout, label smoothing	–	0.1 (drop), 0.1 ($L_B$3)
Train steps, GPUs	50K, 3 (accumulate=2)	200K, 4 (accumulate=5)
Inference	Trigram-blocking, top–K	Beam=5, $L_B$4, trigram-blocking

5. Evaluation and Empirical Results

BertSumExtAbs was systematically evaluated on three benchmarks: CNN/DailyMail (F$L_B$5 ROUGE), New York Times (limited-length recall), and XSum. The models were compared to both extractive and abstractive baselines, and demonstrated consistent improvements.

ROUGE F$L_B$6 scores on test sets:

Dataset	BertSumExt R1/R2/RL	BertSumAbs R1/R2/RL	BertSumExtAbs R1/R2/RL
CNN/DailyMail	43.25/20.24/39.63	41.72/19.39/38.76	42.13/19.60/39.18
NYT	46.66/26.35/42.62	48.92/30.84/45.41	49.02/31.02/45.55
XSum	–	38.76/16.33/31.15	38.81/16.50/31.27

All BertSum variants substantially outperform prior extractive or abstractive baselines on these news-summarization tasks (Liu et al., 2019).

6. Architectural Features, Rationale, and Redundancy Control

BertSumExtAbs provides an end-to-end framework that exploits BERT’s document-level semantic encoding for both sentence scoring (extractive) and token-level sequence generation (abstractive). The alternating segment embedding scheme ensures BERT can process multi-sentence inputs natively. The use of a shallow Transformer stack above BERT’s sentence embeddings introduces inter-sentence interaction explicitly.

The two-stage fine-tuning strategy addresses the pretrained–randomly-initialized parameter mismatch between encoder and decoder, using separate learning rates and warmup schedules. The application of trigram-blocking in both extractive and abstractive inference stages serves as an efficient control against redundancy and repetition.

A plausible implication is that the architectural separation of extractive pretraining and abstractive fine-tuning enables improved knowledge transfer from large pretrained LLMs to generation modules in summarization.

7. Broader Context and Significance

BertSumExtAbs established a general methodology for integrating pretrained encoders like BERT into summarization architectures, with direct mechanisms for sentence selection and sequence generation. The framework’s empirical success on news-oriented benchmarks drove subsequent research on pretrained models for both extractive and abstractive summarization, particularly in leveraging segment-level embeddings and fine-tuning strategies tailored for document processing. The released codebase (https://github.com/nlpyang/PreSumm) further facilitated reproducibility and extension within the summarization community (Liu et al., 2019).