A Parametric Memory Head for Continual Generative Retrieval

Abstract: Generative information retrieval (GenIR) consolidates retrieval into a single neural model that decodes document identifiers (docids) directly from queries. While this model-as-index paradigm offers architectural simplicity, it is poorly suited to dynamic document collections. Unlike modular systems, where indexes are easily updated, GenIR's knowledge is parametrically encoded in its weights; consequently, standard adaptation methods such as full and parameter-efficient fine-tuning can induce catastrophic forgetting. We show that sequential adaptation improves retrieval on newly added documents but substantially degrades performance on earlier slices, exposing a pronounced stability-plasticity trade-off. To address this, we propose post-adaptation memory tuning (PAMT), a memory-only stabilization stage that augments an adapted model with a modular parametric memory head (PMH). PAMT freezes the backbone and attaches a product-key memory with fixed addressing. During prefix-trie constrained decoding, decoder hidden states sparsely query PMH to produce residual corrections in hidden space; these corrections are mapped to score adjustments via the frozen output embedding matrix, computed only over trie-valid tokens. This guides docid generation while keeping routing and backbone parameters fixed. To limit cross-slice interference, PAMT updates only a fixed budget of memory values selected using decoding-time access statistics, prioritizing entries frequently activated by the current slice and rarely used in prior sessions. Experiments on MS MARCO and Natural Questions under sequential, disjoint corpus increments show that PAMT substantially improves retention on earlier slices with minimal impact on retrieval performance for newly added documents, while modifying only a sparse subset of memory values per session.

• The paper introduces PAMT, a two-stage process that uses a parametric memory head to mitigate catastrophic forgetting in generative retrieval systems.
• It demonstrates that selectively updating sparse memory values retains retrieval performance on both old and new document slices.
• Experiments on MS MARCO and Natural Questions show significant Hit@10 improvements, validating the modular approach for dynamic corpora.

A Parametric Memory Head for Continual Generative Retrieval

Introduction

The research paper "A Parametric Memory Head for Continual Generative Retrieval" (2604.23388) focuses on enhancing generative information retrieval (GenIR) systems for dynamic document collections. GenIR models represent document retrieval as a neural sequence-generation task, translating queries into document identifiers (docids). This paradigm inherently faces challenges related to document collections that evolve over time, as GenIR bases its retrieval indices on model weights rather than modular systems that can dynamically update indices without retraining. The paper introduces post-adaptation memory tuning (PAMT) to address the stability-plasticity trade-off typical in GenIR, particularly catastrophic forgetting when models are incrementally updated to index new documents.

Generative Information Retrieval Challenges

Generative retrieval systems consolidate processes, leveraging neural models to map queries directly to docids. While they achieve architectural simplicity and efficiently utilize semantic priors from pretrained LMs, they are less adaptable to dynamic corpora. When new documents are incorporated, traditional retraining methods can lead to catastrophic forgetting—where retrieval performance on previously indexed data degrades significantly. This exposes a pronounced stability-plasticity trade-off, necessitating a solution that balances improved retrieval on new data with retention on existing content Figure 1.

Figure 1: Post-adaptation memory tuning (PAMT) for continual GenIR. (a) Adapt-then-stabilize pipeline: The parametric memory head (PMH) is attached and co-trained with the backbone on $\mathcal{D}_0$.

Post-Adaptation Memory Tuning (PAMT)

PAMT consists of two stages:

1. Stage 1 adapts the backbone model to newly added slices using conventional methods.
2. Stage 2 introduces the parametric memory head (PMH) which augments the backbone post-adaptation, retaining learned mappings without needing to retrain the backbone (2604.23388).

PMH operates by attaching a product-key memory with fixed addressing to the frozen backbone, querying sparse hidden-space corrections through the memory during constrained decoding. It updates a fixed budget of memory values, selecting which values to update based on decoding-time access frequencies Figure 2.

Figure 2: Stage~1 vs.\ Stage~2 across temporal slices. Hit@10 (\%) over slices D_0--D_5 for NQ and MS~MARCO under Expanded and Fixed protocols.

Methodology and Experimentation

The experiments conducted on MS MARCO and Natural Questions benchmarks showcase PAMT in action over sequential, disjoint document increments. The extensive tests indicate substantial improvements in retention and minimal impact on the retrieval performance of new documents. This is achieved while modifying only a sparse subset of memory values per session.

PAMT's efficacy is demonstrated through clearer retention on earlier slices while maintaining adaptability to newly added documents. Hit@10 scores significantly improved, establishing PAMT's utility in generative retrieval contexts Figure 3.

Figure 3: Stage~1 vs.\ Stage~2 aggregate continual-learning metrics. AP, $\mathrm{FWT}_{\mathrm{diag}$.

Evaluation and Implications

The research conclusively shows PAMT's potential to offset challenges in dynamic generative retrieval systems. Owing to its design, it offers practical implications for real-world applications, notably those with frequently updating corpora. Moreover, PAMT represents a meaningful direction for future explorations into balancing the stability and plasticity of GenIR systems.

While the paper primarily focuses on immediate enhancements regarding catastrophic forgetting, it leaves room for exploring broader implications on memory-augmented models and their adaptation capabilities. The modular nature of the parametric memory head might be adapted further to enhance generalization capabilities across different retrieval contexts.

Conclusion

The paper "A Parametric Memory Head for Continual Generative Retrieval" decisively addresses GenIR systems' adaptability to evolving document collections. PAMT provides a promising avenue for balancing retrieval on new and existing data by utilizing a modular memory head that recalibrates document scoring interfaces without extensive retraining. This approach not only mitigates catastrophic forgetting but also enhances current generative retrieval models' robustness. The adoption of generative retrieval paradigms can benefit significantly from such advancements, making them viable solutions for complex, dynamic information retrieval tasks.