Can Large Language Model Summarizers Adapt to Diverse Scientific Communication Goals? (2401.10415v2)

Abstract: In this work, we investigate the controllability of LLMs on scientific summarization tasks. We identify key stylistic and content coverage factors that characterize different types of summaries such as paper reviews, abstracts, and lay summaries. By controlling stylistic features, we find that non-fine-tuned LLMs outperform humans in the MuP review generation task, both in terms of similarity to reference summaries and human preferences. Also, we show that we can improve the controllability of LLMs with keyword-based classifier-free guidance (CFG) while achieving lexical overlap comparable to strong fine-tuned baselines on arXiv and PubMed. However, our results also indicate that LLMs cannot consistently generate long summaries with more than 8 sentences. Furthermore, these models exhibit limited capacity to produce highly abstractive lay summaries. Although LLMs demonstrate strong generic summarization competency, sophisticated content control without costly fine-tuning remains an open problem for domain-specific applications.

Introduction

Recent exploration in the domain of LLM research has begun scrutinizing the adaptability of LLMs to specific tasks beyond simple text generation, with particular attention on the domain of scientific communication. Contemporary works have put forward the hypothesis that the controllability of LLMs may pave the way for generating different styles of summaries—ranging from paper reviews to comprehensive abstracts—without the need for extensive fine-tuning.

Investigating Controllability

At the heart of this investigation is the determination of whether non-fine-tuned LLMs can be manipulated to generate summaries that adhere to intentional prompts reflective of different scientific communication objectives. This includes managing stylistic features and ensuring coverage of key content. One pivotal paper found that LLMs could outshine humans in generating multi-perspective scientific review summaries, as evidenced through higher lexical overlap with reference summaries. Crucially, this was achieved without fine-tuning, representing notable progress in the field.

Controllable Summarization

When it comes to controlling LLMs, findings suggest that precision can be influenced by presenting models with strategic prompts. This influences factors such as the length of summaries, narrative perspectives, and keyword coverage. Models like LLAMA-2 and GPT-3.5 have demonstrated an impressive compliance with such intents, generating summaries with a remarkable alignment to specific standards set by the prompts. Moreover, introducing classifier-free guidance (CFG) during the decoding process has shown to enhance the alignment of generated summaries with the intended prompts.

Limitations and Future Outlook

Despite these advancements, limitations prevail. One notable observation is LLMs' propensity to struggle with generating more extended, highly abstractive summaries, as observed in lay summary tasks, which continue to pose challenges. Additionally, there's a call to assess these findings' broader applicability beyond controlled experimental settings. There is a collective understanding that while the capacity for content control without costly fine-tuning has been demonstrated, applying these insights domain-specifically remains an area ripe for further research and development.