Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization (1808.08745v1)

Abstract: We introduce extreme summarization, a new single-document summarization task which does not favor extractive strategies and calls for an abstractive modeling approach. The idea is to create a short, one-sentence news summary answering the question "What is the article about?". We collect a real-world, large-scale dataset for this task by harvesting online articles from the British Broadcasting Corporation (BBC). We propose a novel abstractive model which is conditioned on the article's topics and based entirely on convolutional neural networks. We demonstrate experimentally that this architecture captures long-range dependencies in a document and recognizes pertinent content, outperforming an oracle extractive system and state-of-the-art abstractive approaches when evaluated automatically and by humans.

Overview of "Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization"

The paper introduces "extreme summarization," a novel single-document summarization task which is inherently abstractive and demands creating a brief, one-sentence summary that answers the question, "What is the article about?". This form of summarization differs from traditional methods that typically favor extractive strategies. The authors address this task by introducing a large-scale dataset derived from British Broadcasting Corporation (BBC) articles and propose an innovative deep learning model that leverages convolutional neural networks (CNNs) conditioned on the topics of articles.

Methodology and Dataset

The primary contributions of the paper include the creation of the XSum dataset and the development of a topic-aware convolutional sequence-to-sequence (T-ConvS2S) model. The XSum dataset comprises BBC articles and curated summaries, thus providing a valuable resource tailored for abstractive summarization tasks. This dataset exhibits significantly less bias towards extractive methods compared to existing datasets like CNN/DailyMail, evidenced by a higher degree of novel n-grams in the reference summaries.

The authors emphasize the necessity of an abstractive approach by presenting statistical analyses and empirical results highlighting that extractive methods underperform for this task. Specifically, the task's extreme compression ratio (i.e., distilling an article into a single sentence) necessitates the use of sophisticated abstraction techniques involving sentence paraphrasing, fusion, and inference.

Topic-Aware Convolutional Model

To tackle this task, the authors introduced a novel T-ConvS2S model encompassing several key innovations:

1. Convolutional Networks: The T-ConvS2S model employs convolutional neural networks instead of the more commonly used recurrent neural networks (RNNs) for summarization tasks. This choice is motivated by CNNs' superior capability to model long-range dependencies and facilitate the parallel encoding of the input sequence, leading to computational efficiency and better learning through hierarchical feature extraction.
2. Topic Conditioning: The encoder of the T-ConvS2S model is enhanced by associating each word with a topic vector derived from Latent Dirichlet Allocation (LDA). This vector captures the topical salience of individual words within a document and the overall topic distribution of the document, thus improving the sensitivity of the model to contextually relevant content during encoding.
3. Attention Mechanism: The multi-hop attention mechanism used in the T-ConvS2S model ensures the decoder attends to the most pertinent parts of the input document, conditioned on both word-level and document-level topics. This sophisticated attention mechanism surpasses conventional single-step attentions in effectiveness.

Experimental Evaluation

The T-ConvS2S model was evaluated against several benchmarks, including extractive baselines (lead, ext-oracle) and RNN-based abstractive models (Seq2Seq, PtGen, PtGen-Covg). The results, measured using ROUGE scores, demonstrate the superior performance of the T-ConvS2S model. It consistently outperformed existing models both in terms of n-gram novelty and overall ROUGE scores.

Key numerical results include:

• The T-ConvS2S model achieved ROUGE-1, ROUGE-2, and ROUGE-L scores of 31.89, 11.54, and 25.75, respectively.
• The model demonstrated a higher proportion of novel n-grams in its generated summaries compared to other systems, with 94.10% novel trigrams and 98.03% novel 4-grams.

Human Evaluation

In addition to automatic evaluations, the authors conducted human evaluations using Best-Worst Scaling (BWS) and a question-answering (QA) paradigm to assess informativeness and fluency. The T-ConvS2S summaries were preferred over others, and participants could answer more questions correctly based on these summaries, highlighting their higher informativeness and contextual accuracy.

Implications and Future Directions

The implications of this research are twofold:

1. Practical Implications: As an effective solution for extreme summarization tasks, the T-ConvS2S model can be utilized in applications requiring concise content delivery, such as news aggregation, content curation, and summarization for mobile devices.
2. Theoretical Implications: The research underscores the importance of balancing local and global contextual information in summarization models. It pushes the boundaries of convolutional network applications in NLP, particularly for tasks demanding long-range dependency modeling and abstraction.

Future research directions may include further refining topic-aware mechanisms, integrating other linguistic features like co-reference resolution, exploring novel architectures that blend the strengths of CNNs and transformers, and extending the dataset to include a broader variety of document types to test the model's generalizability.