FABLES: Evaluating faithfulness and content selection in book-length summarization (2404.01261v2)

Abstract: While long-context LLMs can technically summarize book-length documents (>100K tokens), the length and complexity of the documents have so far prohibited evaluations of input-dependent aspects like faithfulness. In this paper, we conduct the first large-scale human evaluation of faithfulness and content selection on LLM-generated summaries of fictional books. Our study mitigates the issue of data contamination by focusing on summaries of books published in 2023 or 2024, and we hire annotators who have fully read each book prior to the annotation task to minimize cost and cognitive burden. We collect FABLES, a dataset of annotations on 3,158 claims made in LLM-generated summaries of 26 books, at a cost of $5.2K USD, which allows us to rank LLM summarizers based on faithfulness: Claude-3-Opus significantly outperforms all closed-source LLMs, while the open-source Mixtral is on par with GPT-3.5-Turbo. An analysis of the annotations reveals that most unfaithful claims relate to events and character states, and they generally require indirect reasoning over the narrative to invalidate. While LLM-based auto-raters have proven reliable for factuality and coherence in other settings, we implement several LLM raters of faithfulness and find that none correlates strongly with human annotations, especially with regard to detecting unfaithful claims. Our experiments suggest that detecting unfaithful claims is an important future direction not only for summarization evaluation but also as a testbed for long-context understanding. Finally, we move beyond faithfulness by exploring content selection errors in book-length summarization: we develop a typology of omission errors related to crucial narrative elements and also identify a systematic over-emphasis on events occurring towards the end of the book.

Evaluating Faithfulness and Content Selection in Book-Length Summarization

Introduction to Assessment of LLM-Generated Summaries

The evolution of LLMs has drawn significant attention towards summarizing lengthy documents such as books. Deciphering the quality of these summaries, especially in terms of faithfulness and content selection, has been a pivotal step in ensuring the reliability and utility of LLM-generated summaries. This investigation pivots around a comprehensive human evaluation focusing on these critical aspects.

Methodology for Collecting Faithfulness Annotations (FABLES)

Our methodological framework concentrated on the summary of books published in 2023 or 2024 to avoid data contamination. By enlisting annotators who were already familiar with the books, we endeavored to mitigate the challenges linked with the complexity and length of such narratives. Employing a hierarchical summarization strategy, we generated summaries through five different LLM configurations and subsequently decomposed them into discrete claims for detailed annotation. The dataset, named FABLES, encompasses 3,158 claim-level annotations stretched across 26 books, further providing a robust basis for evaluating the book-length summarizers.

Results and Discoveries from the Annotations

Analyzing FABLES unveiled several insightful patterns:

• The predominance of unfaithfulness was primarily associated with misrepresented events and character states, demanding complex, indirect reasoning for validation.
• Interestingly, discrepancies emerged in the reliability of LLMs for generating faithful summaries; with one LLM demonstrating a notable edge in producing the most faithful summaries.
• The inconsistencies in faithfully representing the narrative underscore a substantial area for improvement in book-length summarization techniques.

Delving into Content Selection Errors

Beyond faithfulness, the exploration delved into content selection errors, highlighting omission as a recurrent issue. A developed taxonomy categorizes these omissions with a significant focus on missing critical narrative elements. The paper accentuates the tendency of certain models to disproportionately emphasize events occurring towards narratives' conclusion.

The Challenge of Automatic Faithfulness Evaluation

A concerted attempt was made to automate the evaluation process through various LLM-based raters of faithfulness. Despite these endeavors, the correlation between automated metrics and human judgment remained weak. This particular finding not only echoes the inherent challenges in automating the evaluation of faithfulness but also underscores the complexity of summarizing book-length documents.

Implications and Future Directions

The insights garnered from this paper lay the groundwork for enhancing the strategies employed in the summarization of long documents. The revelations regarding the inconsistencies in content selection and faithfulness offer valuable feedback for refining LLM-based summarization approaches. The quest for devising robust automated evaluation metrics continues, indicating a pivotal area for future research.

Conclusion

By harnessing human judgments to unravel the intricacies of faithfulness and content selection errors in LLM-generated summaries, this paper paves the way for nuanced understanding and methodological enhancements in long-context summarization. The findings spotlight the paramountcy of tackling unfaithfulness and mining deeper into the content selection processes to bolster the reliability and efficacy of summarization technologies.