Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loop (2311.16822v2)

Abstract: LLMs (LLM) are already widely used to generate content for a variety of online platforms. As we are not able to safely distinguish LLM-generated content from human-produced content, LLM-generated content is used to train the next generation of LLMs, giving rise to a self-consuming training loop. From the image generation domain we know that such a self-consuming training loop reduces both quality and diversity of images finally ending in a model collapse. However, it is unclear whether this alarming effect can also be observed for LLMs. Therefore, we present the first study investigating the self-consuming training loop for LLMs. Further, we propose a novel method based on logic expressions that allows us to unambiguously verify the correctness of LLM-generated content, which is difficult for natural language text. We find that the self-consuming training loop produces correct outputs, however, the output declines in its diversity depending on the proportion of the used generated data. Fresh data can slow down this decline, but not stop it. Given these concerning results, we encourage researchers to study methods to negate this process.

Introduction

The environment in which LLMs are trained and subsequently deployed is a dynamic one, closely tied to the vast and ever-evolving content on the internet. A notable phenomenon is how the content generated by LLMs is often recycled back into the data pools from which new LLM generations are trained, leading to what can be described as a "self-consuming training loop". This process raises questions about the long-term effects on the quality and diversity of the output produced by successive LLM generations.

Analyzing the Self-Consuming Training Loop

An investigative paper has been conducted to understand the ramifications of LLMs that are part of this cycle. The researchers adopted an empirical approach, constructing a novel dataset comprised of logical expressions. Unlike natural language, logical expressions allow for straightforward analytical verification of both syntactic and semantic accuracy, providing a clear measure of the correctness of LLM-generated content.

Quality and Diversity Over Generations

The paper revealed that, initially, this self-consuming training loop does appear to enhance the quality and diversity of the outputs. However, it becomes clear that after a few iterations of this cycle, the diversity of the content starts to decline, irrespective of the data cycle (the method by which new data is incorporated for each LLM training generation). The degree of this decline in diversity was also noted to be dependent on the mix of real and synthetic data used in training.

Implications and Future Work

A significant implication drawn from the paper is that while utilizing LLM-generated data can improve correctness in the short term, it may significantly compromise the variety of outputs over time. This points to the necessity for caution and deeper examination of training data by researchers and developers to avoid a potential decrease in the utility and performance of LLMs. The paper suggests further research is needed to explore how the introduction of fresh data in each generation and tactics like fine-tuning might affect such self-consuming training loops.