RecurrentGPT: Interactive Generation of (Arbitrarily) Long Text (2305.13304v1)

Abstract: The fixed-size context of Transformer makes GPT models incapable of generating arbitrarily long text. In this paper, we introduce RecurrentGPT, a language-based simulacrum of the recurrence mechanism in RNNs. RecurrentGPT is built upon a LLM such as ChatGPT and uses natural language to simulate the Long Short-Term Memory mechanism in an LSTM. At each timestep, RecurrentGPT generates a paragraph of text and updates its language-based long-short term memory stored on the hard drive and the prompt, respectively. This recurrence mechanism enables RecurrentGPT to generate texts of arbitrary length without forgetting. Since human users can easily observe and edit the natural language memories, RecurrentGPT is interpretable and enables interactive generation of long text. RecurrentGPT is an initial step towards next-generation computer-assisted writing systems beyond local editing suggestions. In addition to producing AI-generated content (AIGC), we also demonstrate the possibility of using RecurrentGPT as an interactive fiction that directly interacts with consumers. We call this usage of generative models by ``AI As Contents'' (AIAC), which we believe is the next form of conventional AIGC. We further demonstrate the possibility of using RecurrentGPT to create personalized interactive fiction that directly interacts with readers instead of interacting with writers. More broadly, RecurrentGPT demonstrates the utility of borrowing ideas from popular model designs in cognitive science and deep learning for prompting LLMs. Our code is available at https://github.com/aiwaves-cn/RecurrentGPT and an online demo is available at https://www.aiwaves.org/recurrentgpt.

• The paper introduces a simulated recurrence mechanism that overcomes fixed context limitations in Transformers.
• It enables interactive long-text generation using editable, natural language-based memory constructs to maintain narrative coherence.
• Empirical results demonstrate that RecurrentGPT produces thousands of coherent tokens, outperforming existing state-of-the-art models.

RecurrentGPT: Extended Text Generation Through Recurrence Simulation

The paper "RecurrentGPT: Interactive Generation of (Arbitrarily) Long Text" introduces RecurrentGPT, a novel framework designed to extend the capabilities of LLMs, such as ChatGPT, beyond the fixed-size context limitations inherent in the Transformer architecture. This work aims to enable LLMs to generate arbitrarily long texts by mimicking the recurrence mechanism typically associated with Recurrent Neural Networks (RNNs), particularly Long Short-Term Memory (LSTM).

Key Contributions

1. Simulated Recurrence Mechanism: RecurrentGPT is designed to replicate RNN mechanisms using natural language constructs. This approach circumvents the architectural restraints of traditional Transformers, which are limited by fixed context windows. The framework substitutes vector-based operations in LSTMs with natural language segments, allowing it to simulate recurrence without architectural changes.
2. Interactive and Interpretative Long Text Generation: By enabling natural language-based memory constructs, RecurrentGPT affords greater interpretability and user interaction. Users can edit generated memories and plans, allowing for dynamic adaptations during the text generation process. This interactivity also prevents the model from straying from desired narratives, a common issue with autonomous AI agents.
3. Applications Beyond Conventional AI Content Generation: RecurrentGPT is not limited to generating AI-based content but can also function dynamically in applications like interactive fiction, where the model can respond directly to consumer inputs. This paradigm, termed "AI as Contents" (AIAC), highlights the potential for generative models to engage directly with end-users.
4. Empirical Results: The empirical evaluation shows that RecurrentGPT can autonomously produce texts spanning thousands of tokens with maintained coherence and engagement, surpassing the outputs of state-of-the-art baselines like RE$^{3}$ and DOC.

Implications and Future Directions

Practical Implications:

RecurrentGPT holds potential as an advanced computer-assisted writing tool, assisting users with producing extensive content, such as novels, with greater ease. By alleviating the fixed-size context constraint, it offers a robust alternative for scenarios that require sustained narrative continuity.

Theoretical Implications:

The framework demonstrates the feasibility of integrating principles from cognitive science and recurrent models into the domain of LLMs. This integration could lead to hybrid models that leverage the interpretability of RNN mechanisms with the generative prowess of Transformers.

Potential Developments:

Future work can explore leveraging more advanced LLMs as the backbone for RecurrentGPT to further enhance output quality and utility. Moreover, expanding the interactive fiction capabilities may lead to more immersive AI-driven narrative experiences.

In summary, RecurrentGPT presents an innovative approach to tackling the limitations of existing LLMs through simulated recurrence, offering an array of applications in interactive content generation. This work paves the way for further advancements in the intersection of LLMs and interactive AI systems.