---
title: 'SummaRuNNer: Hierarchical RNN for Summarization'
url: https://www.emergentmind.com/topics/summarunner
type: topic
---

# SummaRuNNer: Hierarchical RNN for Summarization

SummaRuNNer is a recurrent neural network-based sequence model for extractive summarization of documents. It re-casts the extractive summarization problem as sequential binary classification over sentences, using a hierarchical architecture of bidirectional Gated Recurrent Units (GRUs) at the word and sentence levels, and integrates interpretable feature-based scoring for content, salience, novelty, and positional biases. The architecture enables both standard extractive supervision with oracle-generated labels and a novel abstractive training paradigm utilizing only human-written abstracts. SummaRuNNer achieves performance that is comparable to, or better than, state-of-the-art neural extractive models on multiple summarization benchmarks, and offers unique interpretability through per-feature contribution visualizations [1611.04230].

## 1. Hierarchical Model Architecture

SummaRuNNer's architecture consists of two stacked bi-directional GRU layers: a word-level bi-GRU operating over the tokens within each sentence, and a sentence-level bi-GRU that processes the sequence of sentence embeddings representing the whole document. The model sequentially classifies each sentence for inclusion in the summary.

For document $d$ of $N_d$ sentences:

1. **Word-level bi-GRU:** For sentence $i$, each token $t$ is passed through a forward and backward GRU. The hidden states $\{h^w_{i, t}\}$ are average-pooled to obtain a fixed-length sentence embedding $x_i$.
2. **Sentence-level bi-GRU:** The sentence embeddings $\{x_i\}$ are processed bidirectionally to yield sentence representations $h_i = [h_i^f, h_i^b]$.
3. **Document vector:** The document context vector is
   $$
   d = \tanh ( W_d \,\frac{1}{N_d} \sum_{j=1}^{N_d} [h_j^f, h_j^b] + b_d )
   $$
   where $W_d$ and $b_d$ are learnable parameters.

The GRU recurrence is as follows (for generic input $x_j$ and hidden state $h_{j-1}$):
$$
\begin{align*}
u_j & = \sigma( W_{ux} x_j + W_{uh} h_{j-1} + b_u ) \\
r_j & = \sigma( W_{rx} x_j + W_{rh} h_{j-1} + b_r ) \\
h'_j & = \tanh( W_{hx} x_j + W_{hh} ( r_j \odot h_{j-1} ) + b_h ) \\
h_j & = (1-u_j) \odot h'_j + u_j \odot h_{j-1}
\end{align*}
$$
where $\sigma$ denotes the sigmoid function and $\odot$ denotes element-wise multiplication [1611.04230].

## 2. Sentence Scoring and Selection Mechanism

Each sentence is assigned a probability of inclusion in the summary using a logistic classifier with a composite scoring function that incorporates multiple human-interpretable features:
$$
P(y_j=1\mid h_j, s_j, d) = \sigma(
    W_c h_j
    + h_j^T W_s d
    - h_j^T W_r \tanh(s_j)
    + W_{ap} p^a_j
    + W_{rp} p^r_j
    + b )
$$
where:

- $W_c h_j$ scores intrinsic sentence content.
- $h_j^T W_s d$ measures sentence-to-document salience.
- $- h_j^T W_r \tanh(s_j)$ penalizes redundancy via novelty with respect to the accumulated summary representation $s_j$.
- $p^a_j$ and $p^r_j$ are learned embeddings for absolute and relative sentence position; $W_{ap}$ and $W_{rp}$ are their associated weights.
- $b$ is a scalar bias.

The summary vector $s_j$ is a running, probability-weighted sum of all prior sentence vectors:
$$
s_j = \sum_{i<j} h_i \cdot P(y_i=1\mid h_i, s_i, d)
$$
The absence of an explicit summary-length regularizer means sentence extraction at inference is typically governed by selecting top-scoring sentences up to a desired length [1611.04230].

## 3. Training Paradigms

SummaRuNNer supports both extractive and end-to-end abstractive training regimes:

- **Extractive training:** As most summarization corpora only provide abstractive (not extractive) references, labels $y_j$ are generated by a greedy oracle that selects document sentences maximizing ROUGE F1 with respect to the reference summary, adding one at a time until no further improvement is possible. The training objective is the binary cross-entropy loss over sentences:
  $$
  L_{\text{extractive}} = - \sum_{d=1}^{N} \sum_{j=1}^{N_d} \left[ y_j^d \log P(y_j^d=1) + (1-y_j^d) \log (1-P(y_j^d=1)) \right]
  $$
- **Abstractive (end-to-end) training:** An RNN decoder (GRU) is placed on $s_{-1}$, the summary representation after processing the last sentence, and trained to maximize the likelihood of reference summary words (negative log-likelihood loss).
- **Optimization:** Word embeddings are 100d word2vec vectors; each GRU direction uses 200 hidden units. Vocabulary is capped at 150K; max 100 sentences per document, max 50 tokens per sentence. Training uses Adadelta optimization, batch size 64, gradient clipping, and early stopping on validation loss [1611.04230].

## 4. Interpretability and Feature Visualization

A distinctive attribute of SummaRuNNer is the direct interpretability of its sentence selection decisions. Each scoring term in the inclusion probability (content, salience, novelty, absolute/relative position) is independently human-readable, enabling per-sentence visualizations via normalized feature scores. This supports fine-grained analysis and model auditing; for example, the feature "heat-map" (cf. Figure 2 of the original work) shows for each sentence the breakdown of contributions from each abstract feature. Consequently, each selection can be directly correlated with its underlying rationale, addressing a key challenge in neural summarization [1611.04230].

## 5. Experimental Evaluation and Comparative Analysis

SummaRuNNer was evaluated on three primary datasets: DailyMail, CNN+DailyMail, and DUC 2002 (out-of-domain).

The following tables summarize reported ROUGE results under various evaluation settings:

**DailyMail, 75-byte recall:**
| Model             | ROUGE-1 | ROUGE-2 | ROUGE-L |
|-------------------|---------|---------|---------|
| Lead-3            | 21.9    | 7.2     | 11.6    |
| LReg(500)         | 18.5    | 6.9     | 10.2    |
| Cheng et al '16   | 22.7    | 8.5     | 12.5    |
| SummaRuNNer-abs   | 23.8    | 9.6     | 13.3    |
| SummaRuNNer       | 26.2±0.4|10.8±0.3 |14.4±0.3 |

**CNN/DailyMail, full-length F1:**
| Model             | ROUGE-1 | ROUGE-2 | ROUGE-L |
|-------------------|---------|---------|---------|
| Lead-3            | 39.2    | 15.7    | **35.5**|
| Nallapati et al    | 35.4    | 13.3    | 32.6    |
| SummaRuNNer-abs   | 37.5    | 14.5    | 33.4    |
| SummaRuNNer       | **39.6±0.2** | **16.2±0.2** | 35.3±0.2 |

On DUC 2002, SummaRuNNer is competitive but not state-of-the-art, which the authors attribute to domain shift and the use of a generic model [1611.04230].

## 6. Key Contributions and Limitations

Major contributions of SummaRuNNer include:

- Demonstrating that a simple, hierarchical RNN-based sequence model matches or surpasses the previous state-of-the-art in neural extractive summarization on large-scale English news datasets.
- Introducing end-to-end abstractive training for an extractive architecture, removing dependency on noisy oracle labels; however, extractive training achieves higher ROUGE in practice.
- Providing interpretable, decompositional sentence selection criteria, enabling user-facing auditability uncommon among neural models of its time.

Limitations include possible noise in the greedy oracle’s extractive label generation, underperformance of the end-to-end abstractive variant compared to extractive training, a drop in performance when transferring across domains (notably to DUC 2002), and the lack of an explicit summary-length selective regularizer. The authors suggest that improved joint training or pre-training strategies and enhanced domain adaptation remain open problems [1611.04230].

## 7. Relation to Later Work and Comparative Landscape

SummaRuNNer was a precursor to subsequent extractive summarization models integrating scoring and selection in a joint neural framework. Later systems, such as NeuSum, explicitly learn to score and select sentences in a single neural loop and outperform SummaRuNNer by directly modeling the incremental ROUGE gain of candidate sentences given the current partial summary [1807.02305]. SummaRuNNer's structured feature-based decomposition and sequence-labeling perspective influenced the design of interpretable and hierarchical architectures adopted in later summarization research.

Source: https://www.emergentmind.com/topics/summarunner