Answers Unite! Unsupervised Metrics for Reinforced Summarization Models (1909.01610v1)

Abstract: Abstractive summarization approaches based on Reinforcement Learning (RL) have recently been proposed to overcome classical likelihood maximization. RL enables to consider complex, possibly non-differentiable, metrics that globally assess the quality and relevance of the generated outputs. ROUGE, the most used summarization metric, is known to suffer from bias towards lexical similarity as well as from suboptimal accounting for fluency and readability of the generated abstracts. We thus explore and propose alternative evaluation measures: the reported human-evaluation analysis shows that the proposed metrics, based on Question Answering, favorably compares to ROUGE -- with the additional property of not requiring reference summaries. Training a RL-based model on these metrics leads to improvements (both in terms of human or automated metrics) over current approaches that use ROUGE as a reward.

An Expert Analysis of "Answers Unite! Unsupervised Metrics for Reinforced Summarization Models"

The paper "Answers Unite! Unsupervised Metrics for Reinforced Summarization Models," authored by Thomas Scialom et al., addresses the limitations of current abstractive summarization evaluation metrics, particularly focusing on the widely used ROUGE metric. This research proposes alternative metrics based on Question Answering (QA) that do not require reference summaries for evaluating abstractive summarization models, and explores their effectiveness within a Reinforcement Learning (RL) framework.

Core Contributions

The paper's main contributions can be outlined as follows:

1. Introduction of QA-based Metrics: The authors extend recent works by introducing new QA-based metrics to evaluate summarization systems. These metrics diverge from traditional n-gram matching techniques utilized by ROUGE. They aim to provide a more holistic evaluation by assessing the ability of a summary to answer questions generated from the source document, thereby aligning evaluation with human-like summarization better.
2. Comparison with Existing Metrics: Through a human-evaluation analysis, the paper quantitatively compares the proposed metrics against existing methodologies, showing that QA-based metrics better align with human judgments on readability and relevance.
3. Reinforcement Learning with New Metrics: The research incorporates QA-based metrics into RL frameworks to drive the learning process of summarization models. It showcases improvements in generating summaries over current approaches reliant on ROUGE optimizations.
4. Unsupervised and Cross-Domain Evaluation: The paper further investigates the applicability of these metrics in unsupervised settings, both within domain (in-domain) using CNN/Daily Mail dataset and out-of-domain using the TL;DR dataset, demonstrating their capability to leverage unannotated corpora for model improvement.

Methodological Insights

Evaluation Metrics Analysis

The paper argues that while ROUGE is the standard for summarization evaluation, it presents significant shortcomings, particularly bias towards lexical similarity. The proposed QA-based metrics (QA_fscore and QA_conf), computed in both supervised and unsupervised settings, are shown to correlate more closely with human assessments of summary quality, as indicated by a quantitative analysis employing Spearman's rank correlation against human evaluations.

Reinforced Training Objectives

By integrating these metrics into RL training, the authors adopt a mixed objective function that combines traditional maximum likelihood estimation with reinforcement signals derived from their QA-based approach. This contrasts with prior works that heavily favored ROUGE, often compromising on summary fluency and readability.

Empirical Findings and Implications

The empirical results highlight that:

• Training models using the proposed metrics yield summaries that rank better in terms of human-evaluated readability and relevance, compared to ROUGE-optimized models.
• Inclusion of in-domain and out-of-domain unsupervised data using QA metrics significantly boosts model performance, suggesting potential pathways for utilizing vast, unannotated text corpora in summarization tasks.

Future Directions

The implications of this paper extend to several promising research avenues:

• Refinement and adoption of QA-based metrics could reframe summarization evaluation, moving towards more content-aware and contextually relevant criteria.
• The paper paves the way for creating more sophisticated RL paradigms that utilize diverse, non-traditional evaluation metrics to adequately capture nuanced aspects of language generation.
• Further exploration of unsupervised training methods, using innovative metrics, might minimize reliance on manually annotated corpora, enhancing scalability and domain adaptability of summarization systems.

This research advances our understanding of abstractive summarization evaluation, challenging existing paradigms and offering practical solutions to enhance model assessment and training, contributing valuable insights into the broader narrative of Natural Language Processing advancements.