Cited Text Spans for Citation Text Generation (2309.06365v2)

Abstract: An automatic citation generation system aims to concisely and accurately describe the relationship between two scientific articles. To do so, such a system must ground its outputs to the content of the cited paper to avoid non-factual hallucinations. Due to the length of scientific documents, existing abstractive approaches have conditioned only on cited paper abstracts. We demonstrate empirically that the abstract is not always the most appropriate input for citation generation and that models trained in this way learn to hallucinate. We propose to condition instead on the cited text span (CTS) as an alternative to the abstract. Because manual CTS annotation is extremely time- and labor-intensive, we experiment with distant labeling of candidate CTS sentences, achieving sufficiently strong performance to substitute for expensive human annotations in model training, and we propose a human-in-the-loop, keyword-based CTS retrieval approach that makes generating citation texts grounded in the full text of cited papers both promising and practical.

• The paper introduced the use of cited text spans (CTS) to mitigate hallucinations in citation generation systems that rely on abstracts.
• It employs distant labeling with ROUGE-based retrieval and human-in-the-loop keyword strategies integrated with DPR, RAG, and a Longformer model.
• Evaluation on the CORWA dataset shows that CTS-based models significantly outperform abstract-only approaches in BLEU, METEOR, ROUGE-F1, and human quality assessments.

Evaluation of Cited Text Spans for Scientific Citation Text Generation

The paper "Cited Text Spans for Scientific Citation Text Generation" presents an investigation into improving the grounding of automatic citation generation systems by proposing the use of cited text spans (CTS) as the primary input, as opposed to the conventional approach of utilizing abstracts. This paradigm shift stems from the observation that abstracts often provide an inadequate basis for citation generation due to their summarized content, which may not encompass details crucial for a faithful and accurate portrayal of the relationship between scientific works.

Problem Statement and Hypothesis

The authors identify a critical issue in current citation generation systems — the tendency of models to hallucinate when only provided with abstracts. This problem arises because abstracts too frequently omit detailed information found in the body sections of a paper. Furthermore, human annotations that denote the precise CTS in cited documents are laborious to produce and often suffer from low inter-annotator agreement. Consequently, the work explores scalable alternatives to manual CTS annotation that could provide a more stable foundation for generating factually accurate citations.

Methodology

The paper investigates distant labeling as a surrogate for manual CTS annotations, employing a ROUGE-based retrieval approach that prioritizes lexical overlap with the gold citation text. Additionally, a human-in-the-loop, keyword-based CTS retrieval strategy is proposed. This method integrates user guidance to enhance the retrieval of relevant CTS sentences comprehensively.

In a practical move, the authors employ Dense Passage Retrieval (DPR) and Retrieval-Augmented Generation (RAG) frameworks in combination with a Longformer-Encoder-Decoder model to evaluate the efficacy of these CTS retrieval strategies in improving citation text generation. The experimental setup employs the CORWA dataset, ensuring a rigorous examination of their theoretical contributions.

Results

The findings articulate a clear performance improvement in CTS-based generation over abstract-only generation strategies. Significant metrics such as BLEU, METEOR, and ROUGE-F1 indicate better performance for CTS-conditioned models. The human evaluation corroborates these findings, showing higher Relevance, Coherence, and Overall Quality scores for CTS inputs compared to abstracts. Moreover, distinct improvements in faithfulness, measured by QuestEval and ANLI scores, affirm the potential of CTS in mitigating hallucinations.

Implications and Future Work

The implications of this paper underscore the practicality of grounding citation text generation in CTS. This approach enriches the factual basis of generated citations, potentially elevating the quality of automated scientific literature analysis. By leveraging scalable methods like distant labeling, the research bridges the gap between computational accuracy and the labor-intensive nature of manual annotation.

Future explorations could delve further into enhancing automatic retrieval of CTS, potentially integrating recent advances in LLMs for topic analysis or more sophisticated keyword extraction techniques. Moreover, while addressing hallucinations is paramount, developing systems that also incorporate paraphrasing modules could minimize risks of plagiarism further—ensuring ethical and innovative scientific communication.

Overall, this work marks a notable advancement in citation text generation, presenting a pragmatic framework that other researchers could build upon to enhance the integrity and reliability of scientific knowledge dissemination.