Cooperative Memory Paging with Keyword Bookmarks for Long-Horizon LLM Conversations

Abstract: When LLM conversations grow beyond the context window, old content must be evicted -- but how does the model recover it when needed? We propose cooperative paging: evicted segments are replaced with minimal keyword bookmarks ([pN:keywords], ~8-24 tokens each), and the model is given a recall() tool to retrieve full content on demand. On the LoCoMo benchmark (10 real multi-session conversations, 300+ turns), cooperative paging achieves the highest answer quality among six methods -- outperforming truncation, BM25, word-overlap retrieval, a search-tool baseline, and full context -- on four models (GPT-4o-mini, DeepSeek-v3.2, Claude Haiku, GLM-5), confirmed by four independent LLM judges ($p=0.017$, paired bootstrap). We then study the paging design space with a 5x4 ablation over boundary strategies and eviction policies (3,176 synthetic probes, 1,600 LoCoMo probes). Key findings: (1) coarse fixed-size pages (fixed_20) reach 96.7% while content-aware topic_shift collapses to 56.7%; (2) eviction policy choice is data-dependent (FIFO best on synthetic, LFU on LoCoMo); (3) two bookmark generation strategies improve over the heuristic baseline (+4.4 and +8.7 E2E points); (4) the remaining bottleneck is bookmark discrimination -- the model triggers recall() 96% of the time but selects the correct page only 57% when bookmarks are insufficiently distinctive. Keyword specificity alone accounts for a 25 percentage point accuracy difference.

• The paper introduces cooperative memory paging with a novel approach using keyword bookmarks to effectively manage long-horizon LLM conversations.
• It presents a deterministic segmentation, bookmark extraction, and recall API framework achieving over 90% recall accuracy and outperforming baseline methods.
• Empirical evaluations reveal practical design insights, advocating minimal, domain-specific bookmarks and adaptive eviction policies for improved retrieval.

Cooperative Memory Paging with Keyword Bookmarks for Long-Horizon LLM Conversations

Problem Formulation and Limitations of Prior Approaches

The context window limitation in autoregressive LLMs remains a principal challenge for sustaining high-quality responses in long-horizon, multi-turn dialogues. As conversational history grows unbounded, models must evict historical content to stay within the context window, creating a crucial problem: when and how can an LLM reliably recover necessary evicted information? Prior memory management proposals have clear limitations. Approaches such as MemGPT's hierarchical retrieval or Pichay's OS-analogous paging rely on the model to self-diagnose its information gaps or match explicit tool outputs. Compression-based approaches (e.g., AFM, InfiniAttention, and LLMLingua) irreversibly lose detail, limiting recovery fidelity.

The unreliability of purely passive NLL-based fault detection is empirically quantified: output uncertainty, measured by token-level negative log-likelihood (NLL), is not a robust signal for missing context, as LLMs tend to over-confidently generate plausible but incorrect completions when information is absent. In test cases, the mean $\Delta$NLL between full and gist-compressed contexts is statistically indistinguishable from control, with frequent cases of the model being more confident under compression, indicating failure of passive Bayesian surprise as a retrieval trigger.

Figure 1: NLL-based fault detection fails to reliably differentiate between full and compressed contexts; difference in mean NLL ($\Delta$NLL) is not discriminative.

This motivates an explicit, system-informed approach to signaling missing content, rather than implicit detection via model behavior.

Cooperative Paging: Mechanism and Implementation

The paper introduces cooperative memory paging, wherein each evicted conversation segment is replaced with an extremely compact, domain-relevant keyword bookmark ($\sim$8–24 tokens). The core mechanism encompasses three pillars:

1. Segmentation: Conversation turns are grouped into fixed-size pages (default: 5 turns; optimal configurations are determined empirically).
2. Bookmark Generation: Bookmarks are constructed via a deterministic extraction pipeline targeting capitalized tokens, numbers, and dates, filtered via a stopword list to yield domain-discriminative keywords.
3. Recall Tool: The LLM is augmented with a recall() API, enabling explicit page retrieval based on the indexed bookmarks. The LLM is prompted to invoke recall when specific details are required, never to hallucinate.

This design operationalizes a cooperative OS-LLM interface, shifting from the unreliable "detect missing context" paradigm to a "model checks the supplied index" protocol.

Figure 2: Cooperative memory paging pipeline visualized: (left) conversation grows, (center) turns grouped, evicted pages replaced by bookmarks, (right) recall() retrieves full page on demand.

Empirical Evaluation on LoCoMo and Controlled Benchmarks

Controlled Synthetic Experiments

On synthetic dialogues with planted critical information, the cooperative bookmark mechanism achieves 90.9% recall accuracy and 95.2% page selection accuracy (n=22 probes). Importantly, zero false positives are recorded, attesting to the reliability of recall triggering conditioned on discriminative bookmarks.

LoCoMo Benchmark

The LoCoMo benchmark—a realistic evaluation comprising 10 multi-session dialogues, each with $>$300 turns—anchors the method's external validity. Against five representative baselines (truncated context, BM25 retrieval, word overlap, a search-tool interface, and a standard full-context window), Bookmark+Recall decisively outperforms, yielding the highest average quality scores (e.g., 2.18/5 on GPT-4o-mini vs.\ 1.86–2.02 for baselines; $p=0.017$ vs.\ BM25 via bootstrap analysis).

Crucially, the improvement is robust across four models from three provider families (OpenAI, DeepSeek, Anthropic, Zhipu). The advantage persists across QA types, with pronounced gains in temporal reasoning and open-domain queries that necessitate deep historical access.

Figure 3: Per-category accuracy on LoCoMo; Bookmark+Recall dominates across all QA types, with especially large margins on temporal and open-domain questions.

Multi-judge evaluation with four strong LLMs confirms the stability of the ranking; Bookmark+Recall leads under all assessor models.

Fine-Grained Ablation: Paging and Bookmark Strategies

Paging Boundary and Eviction Policy Analysis

A comprehensive 5$\times$4 grid experiment explores the influence of page boundaries and eviction policies:

• Page Granularity: Fixed-size coarse pages (fixed_20, i.e., 20 turns per page) are optimal (96.7% accuracy on synthetic; 63.9% on LoCoMo), whereas topic-shifted or fine-grained boundaries over-fragment, resulting in degraded recall. The primary determinant is the number of bookmarks: minimizing this inventory reduces model search cost and error rate.

  Figure 4: Ablation shows coarse fixed-size pages (fixed_20) substantially outperform finer or content-aware segmentation strategies.

• Eviction Policy: Optimal choice is data-dependent. FIFO excels on forward-moving synthetic data; LFU/LRU outperform on recurrent topic, real-world LoCoMo data. Belády's oracle sets an upper-bound, with 8–14 percentage point gaps to the best practical policy, highlighting the latent room for smarter online adaptation.

  Figure 5: Eviction policy ablation; Belády's oracle upper-bounds achievable recall, revealing substantial room for smarter policies.

• Error Analysis: Current bottleneck is not recall triggering (solved at 96.3%) but page discrimination given indistinct bookmarks (56.6% correct page selection among triggered recalls).

Conversation Topology Effects

Eviction policy performance inversion is explained by underlying conversation topology: FIFO aligns with forward-only synthetic dialogues, but is inadequate in revisit-heavy, real-world settings, where LFU/LRU capture informational locality better.

Figure 6: Forward-moving (left) vs.\ revisit-heavy (right) conversation topologies elucidate rationale for eviction policy selection.

Bookmark Content and Generation Strategies

Bookmark Format Ablation:

Minimal keyword-only bookmarks ($\sim$24 tokens) sharply outperform more verbose or structured formats. Rich bookmarks paradoxically suppress recall invocation, validating the information-gap principle: bookmarks must inject just enough uncertainty to trigger recall, but not so much that the model hallucinates detail.

Figure 7: Pareto analysis—minimal keywords achieve superior accuracy per token vs.\ richer formats.

Figure 8: Information gap principle—minimal bookmarks maximize recall by creating uncertainty, whereas verbose stubs mislead the LLM into less recall and higher hallucination.

Keyword Specificity: Domain-relevant, concrete keywords (e.g., "dietary preference, vegetarian") yield a 25.7pp gain in recall accuracy over generic labels. The discriminative content, not length or structure, is key.

Automated Bookmark Generation: Among six tested strategies, hybrid (heuristic + LLM augmentation) and llm-batch (cross-page non-overlap via LLM) outperform others, notably improving accuracy by $+4.4$–$+8.7$ points (up to $+50$ on open-domain questions). Naive sophistication (TF$\Delta$0IDF, page-level LLM summaries) fails, often by suppressing recall through overinformative bookmarks. The selection rule is simple: use hybrid when surface heuristics are informative; LLM-batch when they are not.

Stress Testing and Robustness

Under maximal eviction pressure (fine boundaries, many bookmarks), only semantically distinguished facts (e.g., food allergy, unique numbers) maintain high recall accuracy; facts using generic tokens (e.g., "medical" or "schedule") collapse under ambiguity, reinforcing the bottleneck on discriminative keyword extraction.

Figure 9: Paging accuracy heatmap; page granularity dominates eviction policy in determining achievable accuracy.

Implications, Limitations, and Future Directions

Practical and Theoretical Implications

• Design Principle: Memory management for LLMs in long-horizon dialogue should prioritize minimal, highly discriminative keyword bookmarks and limit inventory size.
• Retrieval Triggering: The recall call incentive is structural; improving discrimination only helps if recall is still triggered.
• Policy Adaptivity: Online eviction policies should adapt to actual dialogue topology, which can flip the effectiveness of standard heuristics.
• Efficiency: Bookmark+Recall provides quality gains with minimal latency and token overhead—practical for deployment at scale.

Limitations

• All evaluations use LLM auto-judgment; human validation would further strengthen the external claims.
• Experiments are primarily conducted on two LLMs, limiting cross-model generalizability, though robustness is observed across four.
• Bookmark extraction is heuristically driven; supervised or reinforcement learning-based extraction might close remaining accuracy gaps.

Prospective Research

• Learned Keyword Extraction: Discriminatively train keyword extractors with an explicit focus on anticipated future query distributions.
• Adaptive and Predictive Eviction: Model access patterns to further close the gap to oracle eviction.
• Hierarchical Bookmarking: Explore multi-level or hierarchical index structures for extreme-length dialogues.
• Multi-stage Recall and Embedding Reranking: Decouple recall triggering from fine-grained page selection via reranking pipelines at retrieval time.

Conclusion

Cooperative paging with minimal keyword bookmarks and explicit recall tools is demonstrated to be the most effective approach for LLM conversation memory management over long horizons. The core determinant of system efficacy is not advanced retrieval mechanism complexity, but rather disciplined design of compact, highly specific bookmarks and appropriate page granularity. These findings provide actionable insights for both practitioners seeking improved longitudinal dialogue capabilities and researchers advancing LLM memory architectures.

---

References

See full reference list in the original paper (2604.12376).