Objective for trading off parametric and non-parametric memory in language models
Develop a principled objective or decision criterion to determine the tradeoff between parametric memory (knowledge encoded in model weights) and non-parametric memory (e.g., long-context conditioning and retrieval augmentation) in language models, including when and how to allocate reliance on each memory type.
References
How to decide the tradeoff between parametric and non-parametric memory (or, how to define the objective for such a tradeoff) is another interesting open problem for future work.
A substituted sentence checked against the entity's attested true value has never populated on this corpus, zero of 295, 54 and 125 built items across three option counts, leaving open whether parametric belief enters here.
Only 2 memory answers were observed, too few to bound how often that case occurs or is caught (a denominator of 2 puts the one-sided 95\% ceiling at 77.6\%); it stays open.
Whether parametric consolidation offers an advantage for such associations remains open, however: internal storage does not guarantee successful recall or composition, and iterative retrieval can also uncover indirect connections.
Under what query frequencies, corpus sizes, and update rates should information be kept as text, materialized as KV states, or compressed into parameters?