Delving into the Utilisation of ChatGPT in Scientific Publications in Astronomy (2406.17324v2)

Abstract: Rapid progress in the capabilities of machine learning approaches in natural language processing has culminated in the rise of LLMs over the last two years. Recent works have shown unprecedented adoption of these for academic writing, especially in some fields, but their pervasiveness in astronomy has not been studied sufficiently. To remedy this, we extract words that ChatGPT uses more often than humans when generating academic text and search a total of 1 million articles for them. This way, we assess the frequency of word occurrence in published works in astronomy tracked by the NASA Astrophysics Data System since 2000. We then perform a statistical analysis of the occurrences. We identify a list of words favoured by ChatGPT and find a statistically significant increase for these words against a control group in 2024, which matches the trend in other disciplines. These results suggest a widespread adoption of these models in the writing of astronomy papers. We encourage organisations, publishers, and researchers to work together to identify ethical and pragmatic guidelines to maximise the benefits of these systems while maintaining scientific rigour.

Utilisation of ChatGPT in Scientific Publications in Astronomy

The paper "Delving into the Utilisation of ChatGPT in Scientific Publications in Astronomy" by Simone Astarita et al. investigates the adoption of ChatGPT, a prominent LLM, in the domain of astronomical scientific literature. The paper elucidates the prevalence of ChatGPT-generated text within peer-reviewed and non-peer-reviewed publications by examining shifts in linguistic patterns specific to AI-generated content.

Methodological Approach

The paper employs a methodical approach to identify words that are disproportionately used by ChatGPT compared to human authors. The dataset for this analysis is derived from NASA’s Astrophysics Data System (ADS), encompassing over 1 million articles published between 2000 and 2024. The authors define a list of words favored by ChatGPT by analyzing a corpus created by Liang et al., which consists of AI-generated paraphrases of human-written academic text.

Using the API of NASA’s ADS, the frequency of these words is tracked across publications over the targeted timeframe. Words with significant usage differences between AI and human text are identified, forming the subset $W$. To ensure robustness, a control group $C$, consisting of randomly selected words that appeared in the dataset, is also established. Both sets of words $W$ and $C$ are analyzed to detect temporal variations in their frequency of occurrence.

Results and Statistical Analysis

One of the key findings from the paper is the statistically significant increase in the frequency of AI-favored words in 2024. The variation is particularly pronounced in non-peer-reviewed articles, suggesting a greater penetration of LLM-generated text in this subset. The Kolmogorov–Smirnov test is utilized to confirm the significance of these changes, showing p-values consistently below 0.05 when comparing 2024 against preceding years.

To mitigate the confounding effect of natural language evolution, the paper compares the differences for random control words over the same period. The absence of a significant pattern in these control words further substantiates the claim that the observed increase in AI-favored words is linked to the uptake of LLMs like ChatGPT.

Implications

The results of this paper indicate that there has been a notable increase in the use of ChatGPT in scientific writing within the field of astronomy. This trend raises important questions about the balance between leveraging LLMs for their utility and the potential risks associated with their use. The paper highlights several implications:

• Ethical Considerations: As noted in the literature cited by Astarita et al., there are concerns regarding plagiarism, factual inaccuracies, and the potential erosion of individual authorship credit. These aspects necessitate the development of clear guidelines and disclosure requirements for the use of LLMs in academic writing.
• Quality Control: The paper underscores the need for stringent peer-review processes to ensure that the scientific integrity of published works is maintained. The distinction between refereed and non-refereed publications in the analysis indicates that the peer-review process may currently mitigate some of the risks associated with LLM-generated text.
• Support for Non-Native Speakers: On the positive side, the adoption of LLMs can be seen as a tool to help non-native English speakers produce high-quality scientific text, thereby leveling the playing field in international academic publishing.

Future Directions

Looking ahead, the continuous advancement of LLMs presents both opportunities and challenges:

1. Model Improvement and Specialization: Future developments may include specialized LLMs trained on domain-specific corpora, such as AstroLLaMA for astronomy. This could enhance the relevance and accuracy of AI-generated content in specific scientific fields.
2. Detection and Transparency: The evolving sophistication of LLMs will necessitate ongoing research into detection mechanisms for AI-generated content. Transparent reporting and the development of tools to identify LLM-authored sections can help maintain the credibility of scientific literature.
3. Ethical Frameworks: The establishment of robust ethical frameworks and publishing policies that address the nuances of LLM use in scientific writing will be critical. Such frameworks should balance the benefits of LLMs with the need to preserve the integrity and originality of scientific research.

Conclusion

The paper by Astarita et al. provides a comprehensive analysis of the penetration of ChatGPT in astronomical scientific literature. By highlighting the frequency changes of AI-favored words from 2000 to 2024, the paper presents compelling evidence of the growing influence of LLMs in academic writing. The implications discussed underscore the need for a balanced approach that harnesses the advantages of LLMs while addressing the ethical and quality concerns they introduce. As LLM technology continues to evolve, ongoing dialogue and adaptive policies will be essential to navigate its integration into the scientific publishing landscape.