Scalability in Memory-Augmented LLMs

Establish scalable memory-bank management strategies for memory-augmented large language models that store learned per-document modulation parameters in an external memory, so that these systems can handle streaming corpora reaching hundreds of thousands or millions of documents while balancing adaptation effectiveness, computational efficiency, and training stability.

Background

The paper discusses memory-augmented frameworks for continual adaptation of LLMs, where learned modulation parameters for each document are stored in an external memory bank and used to condition a frozen base model during inference. This approach avoids gradient-based updates and mitigates catastrophic forgetting by aggregating modulations across the entire memory bank.

However, in real-world streaming scenarios the number of documents can grow to hundreds of thousands or millions, causing the external memory bank to become very large, increasing storage and inference costs. The authors explicitly identify scalability as an open problem in these memory-augmented systems that must be addressed while maintaining adaptation quality, efficiency, and stability.