MemRouter: Memory-as-Embedding Routing for Long-Term Conversational Agents

Abstract: Long-term conversational agents must decide which turns to store in external memory, yet recent systems rely on autoregressive LLM generation at every turn to make that decision. We present MemRouter, a write-side memory router that decouples memory admission from the downstream answer backbone and replaces per-turn memory-management decoding with an embedding-based routing policy. MemRouter encodes each turn together with recent context, projects the resulting embeddings through a frozen LLM backbone, and predicts whether the turn should be stored using lightweight classification heads while training only 12M parameters. Under a controlled matched-harness comparison on LoCoMo, where the retrieval pipeline, answer prompts, and QA backbone (Qwen2.5-7B) are held identical, MemRouter outperforms an LLM-based memory manager on every question category (overall F1 52.0 vs 45.6, non-overlapping 95% CIs) while reducing memory-management p50 latency from 970ms to 58ms. Descriptive factorial averaging further shows that learned admission improves mean F1 by +10.3 over random storage, category-specific prompting adds +5.2 over a generic prompt, and retrieval contributes +0.7. These results suggest that write-side memory admission can be learned by a small supervised router, while answer generation remains a separate downstream component in long-horizon conversational QA.

• The paper presents an embedding-based memory routing approach that decouples high-frequency write operations from autoregressive LLM inference.
• It demonstrates significant efficiency gains with a 17× reduction in latency and improved QA F1 scores in single-hop (+14%) and multi-hop (+19%) settings.
• Experimental results confirm scalable, backbone-agnostic performance with benefits in resource-constrained environments and modular upgrade paths.

MemRouter: An Embedding-Based Memory Routing Framework for Conversational Agents

Introduction

Scaling memory for long-term conversational agents is an unresolved challenge as LLMs’ context windows remain insufficient for real-world multi-session tasks. Most prior solutions rely on per-turn autoregressive LLM calls for deciding which dialogue turns to admit to persistent memory, which introduces unacceptable compute overhead and tightly couples memory management to specific LLM answer agents. "MemRouter: Memory-as-Embedding Routing for Long-Term Conversational Agents" (2605.00356) proposes a novel architectural solution: decoupling the write-side admission mechanism via a lightweight, embedding-based supervised router operating entirely in frozen LLM embedding space. This essay discusses the MemRouter methodology, its empirical results, and its implications for memory in agentic LLM systems.

Motivation and Limitations of Prior Work

Prior approaches such as Memory-R1 [yan2025memoryr1], AgeMem [yu2026agemem], MemGPT [packer2023memgpt], and A-MAC [zhang2026amac] couple long-term memory selection with autoregressive LLM inference and/or RL-driven policies. This results in:

• Prohibitive per-turn overhead: Each conversational turn results in one or more full LLM generations—approximately 1,200 generation calls for a 600-turn conversation.
• Backbone coupling: Memory management policies are co-trained with answer agents and must be retrained for each new QA backbone.

MemRouter addresses these points by replacing text-based admission with embedding-level routing, entirely side-stepping token-level autoregression during the write operation.

Architectural Design

MemRouter comprises three main modules (see (Figure 1)):

Figure 1: MemRouter system overview—write path, persistent memory, and read/QA path conceptualized as separate modular components.

1. Memory Router: Processes each turn $x_t$ + recent context, chunked and projected into embedding space, and applies a frozen large transformer (e.g., Qwen2.5-7B) as a feature extractor. Only $\sim$12M lightweight classifier parameters are trained, predicting ADD/NOOP and content-type labels.
2. Memory Store: Admitted turns are stored verbatim with timestamp and speaker metadata. Retrieval is a weighted hybrid of dense BGE embeddings (semantic) and BM25 (keyword), supporting coverage for both factual and entity-specific queries.
3. Answer Agent: Only invoked at question time, with standard text generation (no fine-tuning, prompting only), ensuring that backbone upgrades do not necessitate retraining the router.

This decouples high-frequency write operations from downstream QA, enabling one memory router to serve as the write policy across answer agents and model upgrades.

Training and Admission Policy

Training uses turn-level labels (ADD/NOOP and content types) produced by supervision from a stronger teacher LLM (Qwen3.5-35B-A3B), covering diverse conversational datasets (LoCoMo, LongMemEval, MSC). The router’s trainable parameters consist solely of the (i) chunk embedding projection layers and (ii) classification heads. The backbone and encoder remain strictly frozen—no RL, distilled, or autoregressive fine-tuning is performed.

The key technical distinction is processing the write decision only with a single forward pass of a frozen transformer and trainable head, instead of a full sequence generation, with chunked context embeddings ensuring sufficient semantic and pragmatic awareness.

Retrieval and Answering Pipeline

Upon ADD selection, the verbatim turn is stored (not structured facts)—empirically optimal for token-level F1 recall on LoCoMo [maharana2024locomo]. Query-time retrieval boosts matching using both dense (BGE) and sparse (BM25) signals and query-aware weighting (e.g., temporal, speaker cues).

The downstream answer agent (Qwen2.5-7B or Qwen3.5-35B) receives grouped retrieved memories and executes the only generation pass in the pipeline. Prompts are category-specific: fixed word limits for factual/temporal, unbounded enumerative for multi-hop—a critical decision boosting multi-hop F1 from 26.1 to 52.4.

Experimental Results

Controlled Matched-Harness Comparison: With the same retrieval, prompts, and backbone, MemRouter achieves a total F1 of 52.0 vs. 45.6 for the LLM-based manager, with non-overlapping 95% confidence intervals (Figure 2). The largest gains arise in single-hop (+14%) and multi-hop (+19%) QA. MemRouter also yields a 17× reduction in median per-turn memory management latency (58ms vs. 970ms).

Figure 2: Mean F1 (\%) for each admission policy, prompt style, and retrieval setting after averaging over the other two factors.

Backbone Transfer: The router is backbone-agnostic: swapping only the answer agent to Qwen3.5-35B improves F1 further (to 55.5) without retraining the router. Adapting the router head to LLaMA-3.1-8B yields 49.1 F1, validating cross-family portability.

Ablations (Budget-Matching): With a 62% storage budget (not store-all), MemRouter outperforms MLP-only (50.8 vs. 50.0), keyword heuristics (47.2), random selection (42.8), and recency selection (38.6). The gap with store-all (upper bound) is just ~3 F1, confirming effective prioritization under real constraints.

Efficiency: End-to-end throughput increases 11×, with only a minor peak GPU memory rise due to the small router and additional embedding encoder. Supervised training is more stable and effective than RL-based alternatives, with RL cold-starting and mode collapse reliably observed in embedding-level policies.

Implications and Theoretical/Practical Significance

MemRouter empirically validates that the LLM “cognition” required for memory selection can, in fact, be efficiently approximated through frozen backbone embedding and a lightweight classifier. This contradicts prior assumptions that per-turn text generation is necessary and demonstrates that decoupling write-path policies enables scalable, backbone-independent memory architectures for LLM agents.

Key implications include:

• Scalability: Write-side costs no longer scale with dialogue length; agents can efficiently maintain long-term persistence with minimal per-turn cost.
• Upgrade Path: Memory policies become modular—trained once, reused across LLM backbone upgrades. This facilitates rapid evolution of answer quality without retraining full memory managers.
• Resource-Constrained Deployment: Embedding-level routing enables long-term agents in compute- or latency-sensitive environments.

From a theoretical perspective, this advocates embedding-level policies as primitives for agentic memory operations, orthogonal to downstream QA, and elevates the design of routing/gating mechanisms (as in Mixture-of-Experts, latent routing) to first-class citizens in agent memory stacks. The results motivate further exploration of embedding-based UPDATE/DELETE policies, unsupervised or in-the-wild supervision, and integration with non-LLM memory compression and consolidation methods.

Future Directions

• Extension to UPDATE/DELETE: Extending beyond ADD/NOOP by addressing the challenges of mode collapse and ambiguous update boundaries in dialogue remains necessary for full-fledged CRUD memory agents.
• Self-Supervised Routing: Reducing reliance on teacher LLM labeling, possibly via intrinsic reward or self-distillation from downstream QA improvement signals.
• Memory Compression and Hierarchical Control: Combining router-based admission with hierarchical/recursive summarization [wang2023recursive], reversible compression [wang2025r3mem], or multi-granular memory banks [zhong2024memorybank].

Conclusion

MemRouter offers a modular, efficient, and empirically validated embedding-based architecture for write-side memory admission in long-term LLM agents. Decoupling admission from text-level LLM inference substantially improves efficiency and portability, while maintaining—or exceeding—the answer quality of fully autoregressive baselines. This work shifts the field towards scalable, backbone-agnostic memory stack design, a core enabler for the next generation of agentic LLMs.