PACMS: Submodular Context Selection as a Pluggable Engine for LLM Agents

Abstract: Conversational and tool-using LLM agents operate over a context window that fills from several directions simultaneously. As a session proceeds, the agent accumulates user and assistant turns, entries drawn from a persistent memory store, and often largest of all, the verbatim outputs of tool calls such as file reads, search results, and API responses. Once the cumulative context exceeds the model's token budget, the framework must decide what to keep. The prevailing mechanism is recency truncation, sometimes paired with periodic summarization. This is topic-blind: a fact established early in a session is discarded simply because it is old, even when the current user query is about exactly that fact; conversely, verbose but irrelevant recent material is retained. Agents that must recall information across many turns, the defining case for memory, are precisely where recency truncation fails. Existing alternatives sit outside the agent's assembly step. Retrieval augmented generation fetches external documents into the prompt but does not arbitrate the agent's \emph{already-present} pooled context. Context-compression methods reduce token count by rewriting or pruning text, but operate query-blind and lossily. Neither treats memory entries, conversation turns, and tool outputs as a single candidate pool to be selected from by relevance at the moment the prompt is assembled.

• The paper introduces a submodular formulation for context selection that optimizes query-relevant token budgets for LLM agents.
• It employs a facility-location objective via CELF lazy-greedy to balance relevance and redundancy while providing formal optimization guarantees.
• Empirical evaluations show that PACMS significantly improves end-to-end QA accuracy over traditional methods in real-world LLM deployments.

PACMS: Submodular Context Selection for Pluggable LLM Agent Memory

Motivation and Context

The increasing contextual demands of multi-turn, tool-using LLM agents have outpaced recency-based truncation schemes: session transcripts, persistent memory, and tool outputs collectively overflow LLM token budgets. Topic-blind recency truncation discards old but query-relevant evidence, while retaining recent but irrelevant context, critically undermining long-term recall and QA performance in agents. Retrieval-augmented generation and prompt compression methods, though widely deployed, do not arbitrate in-session context; they operate query-blind or lossily, and fail to treat all candidate memory, turns, and tool outputs as a unified pool. PACMS proposes a principled selection objective—budget-aware, submodular, coverage-maximizing—that directly selects a query-relevant, token-budgeted context subset at prompt assembly.

Figure 1: PACMS overview—candidates form a query-anchored similarity graph, with PACMS selecting a token-budgeted subset covering all query-relevant regions, actively skipping redundant duplicates.

Submodular Selection Objective and Implementation

PACMS recasts the context assembly step as a submodular maximization problem under a knapsack token constraint. For pooled candidates $C$, query $q$, fixed budget $B$, and mandatory set $M$, relevance is measured as $\mathrm{rel}(i, q) = \max(0, \cos(e_i, e_q))$, and coverage weights $$w_{ij} = \mathrm{rel}(i, q) \cdot \max(0, \cos(e_i, e_j))$$. The facility-location objective is defined as:

$F(S) = \sum_i \max_{j \in S} w_{ij}$

subject to $$\sum_{j \in S}\mathrm{tok}(j) \le B, \; M \subseteq S$$, and optimized via CELF lazy-greedy, yielding constant-factor approximation guarantees.

Key distinctions from canonical MMR:

• PACMS coverage sums over the entire pool, not just the selected set, rewarding unaccounted regions.
• Coverage is query-relevance weighted: only relevant regions are considered.
• Formal submodular guarantees, not mere pairwise-penalty heuristics.

Runtime integration leverages OpenClaw's pluggable engine architecture, with PACMS implementing assemble(), delegating summarization to the host, and uniformly selecting across heterogeneous context units, including both terse user turns and large tool outputs.

Figure 2: PACMS runtime architecture—OpenClaw plugin invokes local service, running CELF greedy selection with cached embeddings, and exposes debug endpoints for inspection.

Interactive Demonstration and Real-World Use

The demonstration provides real-time keep/drop visualization, showing explicit token allocation and user-adjustable budgets. Attendees can observe facts previously lost to recency truncation returning to the selected set under query-driven selection. PACMS's selection is strictly relevance-driven: tool outputs are token-measured units, the assembly step is directly manipulative and transparent, and redundancy is actively shed.

A real-world OpenClaw workspace scenario illustrates PACMS's operational effectiveness: after several days and multiple sessions, only query-relevant candidates (such as tool outputs indicating Rust installation failures and preceding relevant memory entries) are retained, while unrelated earlier content is dropped, demonstrating robust query-aware recall.

Figure 3: PACMS deployed on an OpenClaw session—17 of 168 candidates retained for prompt assembly, with keep/drop visualizations and token budget explicitly shown.

Empirical Evaluation and Numerical Results

Evaluation on the LongMemEval benchmark (100-question, multi-session) measures evidence-round recall and QA accuracy under injected redundancy at varying budgets. PACMS matches LangChain's production MMR baseline on recall but substantially outperforms it in end-to-end QA accuracy: at $45\%$ token budget and redundancy $R=2$, PACMS yields $q$0 (GPT-5-mini) and $q$1 (GPT-5.4-mini) percentage point gains over MMR, despite similar recall rates. Notably:

• Pure top-$q$2 relevance ranking leads evidence recall at tight budgets, but PACMS overtakes as budget widens and redundancy increases.
• Recency truncation collapses under redundancy, both in recall and QA; PACMS's superiority persists even under stronger readers.
• The divergence between recall and QA metrics is highlighted: facility-location coverage maximizes downstream extractability for LLM readers, outperforming both pairwise diversification and pure relevance approaches.

Practical and Theoretical Implications

PACMS establishes a principled, pluggable layer for LLM agent context selection, uniformly arbitrating across multi-source candidate pools. The facility-location coverage objective produces prompts optimized for reader exploitation, not just evidence retention. Integration as an OpenClaw context engine demonstrates direct applicability in real-world agent deployment. Exposing context selection decisions, token budgets, and policy swappability enhances transparency and manipulability for practitioners.

Theoretically, separation of recall from extractability points to necessary evaluation advances: QA benchmarks should measure downstream reader utility, not just evidence retention. Submodular selection with formal guarantees outperforms pairwise heuristics, and query-relevance weighting is critical for high-fidelity agent memory.

Future Directions

Anticipated research avenues include:

• Adaptive/learned relevance scoring supplanting fixed cosine embeddings
• Memory curation strategies leveraging assembly statistics (persist, compress, evict)
• Multi-agent contexts wherein selection is performed across shared pools
• On-policy evaluation in interactive deployments, surpassing static benchmarks.

Conclusion

PACMS advances LLM agent context management by formulating prompt assembly as submodular, query-driven, token-budgeted selection, delivered as a plugin for OpenClaw. Matching MMR on evidence recall, PACMS achieves superior end-to-end QA accuracy, confirming that coverage-based selection yields more extractable prompts. The facility-location objective, cache-efficient runtime, interactive visualization, and open-source release position PACMS as a foundational component for scalable, transparent agent memory systems. Future work will extend adaptive relevance, session-level memory management, and multi-agent context selection, addressing both practical deployment and evaluation paradigms for LLM agents.