Memory Retrieval and Routing in AI

• Memory retrieval and routing are techniques for efficiently organizing and accessing knowledge in multi-tiered memory systems with adaptive eviction and selective admission policies.
• They use methods such as dynamic query-conditioned pipelines, importance scoring, and cost-sensitive store routing to balance latency, cost, and answer quality.
• Applications span telecom, video processing, and multi-agent systems, yielding significant improvements in efficiency and accuracy over uniform retrieval approaches.

Memory retrieval and routing are central mechanisms for scalable, high-fidelity knowledge access in agentic and autonomous AI systems under persistent, high-volume workloads. These systems must not only retrieve relevant facts from vast or growing memory, but also decide—efficiently and adaptively—where (and when) to retrieve from, what to store, and how to route queries and updates under severe latency, cost, and context constraints. The design space is shaped by architectural trade-offs between memory tiering, selective retrieval, dynamic eviction, application-specific routing strategies, and control policies that directly optimize answer quality and compute efficiency.

1. Tiered Memory Architectures and Importance-Aware Retrieval

Memory retrieval under bounded storage and context constraints demands tiering: organizing memory into levels with differing cost, capacity, and access semantics. A typical instantiation, as in HTM-EAR, utilizes a two-tier design: an L1 working set (≤500 facts, HNSW-based approximated nearest-neighbor index) for fast, frequent access, and an L2 archival tier (≤5,000 facts) for less common, but still accessible, information (Singh, 27 Feb 2026). Upon insertion, facts are embedded (bi-encoder, E5-large), scored for importance, and admitted to L1. Overflow triggers importance- and usage-aware demotion, with permanent pruning from L2 when its capacity is reached. Essential facts (importance ≥ 0.85) are explicitly tracked to monitor critical information loss.

Retrieval begins with vector search over L1. If similarity (inner product) or entity coverage is insufficient, a hybrid gating function triggers an L2 fall-back. Both sources' candidates are re-ranked using a composite score that blends similarity ($\mathrm{sim}^3$), entity-overlap, and fact-importance (parameters: $\lambda=0.8$, $\gamma=0.1$) before a final, lightweight cross-encoder rerank. This design achieves MRR = 1.000 for active facts at load, 0.215 for deep history (15,000 facts; L1=500, L2=5000), and 0 essential facts lost, outperforming both LRU and uniformly greedy routing under saturation.

The adaptive eviction score is

$$S_{\mathrm{evict}} = \alpha\,\mathrm{importance} + \beta\,\min\left(\frac{\mathrm{usage}}{10},\,1\right)$$

with $\alpha=0.75$, $\beta=0.25$, prioritizing retention of high-value, recently used facts.

2. Routing Policies: Selective Admission, Store Selection, and Shard Routing

Routing in memory-augmented systems spans multiple decision points: admission (what to store), read-side (where and how to retrieve), and architectural scope (within-tier, across stores, or across shards). These policies optimize cost–accuracy trade-offs (Gaikwad, 8 Mar 2026, Zhao et al., 29 Jan 2026). Key approaches include:

• Cost-Sensitive Store Routing: Viewing store selection as a subset-selection problem under cost constraints, agents select stores $\hat G \subseteq \mathcal{S}$ to maximize expected answer quality minus retrieval cost ($\lambda \sum_{s\in G} c_s$). Oracle routing, which selects only contextually necessary stores, saves 62% of tokens and improves QA accuracy by 5.4 pp over uniform retrieval (Gaikwad, 8 Mar 2026).
• Masking and Shard Routing: ShardMemo partitions evidence memory into sharded indexes (Tier B), routed via a masked mixture-of-experts (MoE) gate. Scope predicates define eligibility (structured masking), after which eligible shards are prioritized by a learned router (cost-aware gating, Top-$B_{\mathrm{probe}}$ or adaptive Top-$P$ selection) (Zhao et al., 29 Jan 2026). This reduces retrieval work, improves F1 by up to +6.87 over cosine-prototype routing under constrained probe budgets, and enforces strict latency/efficiency guarantees.
• Write-Side Router Admission: MemRouter decouples write-side memory admission decisions from autoregressive LLM inference, using a lightweight (12M param) embedding-based classifier to admit new memory items. Only relevant turns are stored, greatly reducing write-path latency (58 ms vs. 970 ms) while improving F1 by +10.3 over random admission (Hu et al., 1 May 2026).

Routing mechanism	Core principle	Measured impact
Importance + usage gate (HTM-EAR)	Evict least valuable/retrieved	0 essential facts lost; MRR=1.000
Store subset selection	Maximize $\lambda=0.8$0	-62% tokens, +5.4pp QA (oracle)
Masked MoE over shards	Eligibility masking, learned priority	+6.87 F1, -20.5% VecScan
Learned admission (MemRouter)	Lightweight embedding classification	+10.3 F1 over random, -94% latency

3. Dynamic Query- and Type-Conditioned Routing

Effective memory retrieval increasingly requires routing pipelines conditional on query properties or types. SelRoute formalizes type-aware routing as a two-stage process (McKee, 2 Apr 2026):

• Type classification: A regex-based classifier $\lambda=0.8$1 predicts which of up to six types (e.g., knowledge-update, multi-session) a query belongs to.
• Pipeline assignment: A routing function $\lambda=0.8$2 assigns the query to lexical, semantic, hybrid, or vocabulary-enriched pipelines. Retrieval within each pipeline uses the appropriate scoring (BM25, cosine, or their weighted combination).

Even with imperfect type prediction (71.9% raw classifier accuracy), effective routing accuracy reaches 83%, and end-to-end retrieval (Recall@5 = 0.689–0.800) outperforms all uniform single-pipeline baselines, confirming that identifying and matching query types to storage/retrieval strategies is essential for high-recall, efficient access.

Dynamic, query-conditioned routing underpins several other frameworks, including

• ICICLE’s [COPY] router: Decides, for each query, whether to copy a docid from in-context memory or to fall back to parametric retrieval, using a calibrated routing signal. This resolves corpus expansion bottlenecks in generative retrieval (Den et al., 26 May 2026).
• PRISM’s intent cascade: Adaptive heuristic-to-LLM gating routes temporal/causal or multi-hop queries through specialized retrieval-cost functions without uniform LLM overhead, saving 42.3% of LLM classifier calls (Peng et al., 12 May 2026).

4. Closed-loop Retrieval and Reflective Control

Robust memory routing in long-horizon or uncertain environments increasingly leverages closed-loop, reflective control rather than open-loop pipelines. MemR³ exemplifies this with a controller that, at each iteration, selects among three actions: retrieve, reflect (reason over current gaps and evidence), or answer, according to the evidence–gap state and an LLM-generated decision constrained by fixed safety policies (Du et al., 23 Dec 2025). This controller dynamically budgets retrieval steps to optimize answer quality, reduces unnecessary queries, and delivers unconstrained improvements (e.g., RAG backbone: 79.46%→86.75% LLM-judge accuracy).

Key design:

• Evidence–gap tracking: The agent maintains sets $\lambda=0.8$3 (gathered evidence) and $\lambda=0.8$4 (unsatisfied requirements), transitioning between retrieval and reflection until all gaps close or a maximum iteration is reached.
• Policy constraints: Hardcoded stopping, reflect-streak, and retrieval opportunity checks guarantee termination and avoid infinite loops.

5. Multi-Agent and Multi-Role Routing Paradigms

In collaborative or multi-LLM agent systems, memory routing becomes interleaved with role assignment, workflow generation, and distributed memory management.

• Role- and stage-aware routing: RCR-Router assigns memory items scores based on their relevance to the agent’s role, task stage, and recency, and routes only the highest-scoring items within per-agent token budgets (knapsack-optimal if item size is uniform) (Liu et al., 6 Aug 2025).
• Heterogeneous graph-based routing: GraphPlanner formulates routing as an MDP over agent roles and LLM backbones, leveraging a dynamic heterogeneous memory graph (GARNet) to aggregate historical and workflow memory into state representations (Feng et al., 26 Apr 2026). At each step, routing decisions are made via attention-read over role-hub embeddings, optimizing for both effectiveness and efficiency under reinforcement learning. Ablations confirm the necessity of dynamic, memory-conditioned routing for attaining up to +9.3% accuracy gain and robust zero-shot generalization.

6. Applications and Domain-Specific Routing

Domain-specialized applications motivate architectures that combine memory retrieval with domain-aware routing:

• Telecom RAG with neural routing: Telco-oRAG routes queries to only the most relevant 3GPP standard series, selected by a compact neural router based on learned series-alignment and semantic features (Bornea et al., 17 May 2025). This reduces memory footprint by 45% and achieves +17.6% accuracy improvement over vanilla RAG.
• Spatio-temporal segment routing in video: SALOVA segments long videos, encodes segments, then routes user queries by cross-attention between segment and query embeddings (SR-Router), enabling scalable retrieval over multi-million frame corpora while maintaining <3× the memory cost and high recall (Kim et al., 2024).
• Agent routing for hybrid LLM/agent architectures: BoundaryRouter maintains an early-experience memory of solved queries, retrieves similar prior cases, and uses a rubric-guided LLM prompt to decide at inference whether to use direct LLM inference or escalate to a full agent, halving average inference time and improving accuracy by 28.6% over naïve LLM-only baselines (Wang et al., 8 May 2026).

7. Architectural Trends and Quantitative Outcomes

Memory retrieval and routing are increasingly recognized as primary drivers of efficiency and answer quality in persistent agent systems—often more so than raw model scale (Liu et al., 24 Mar 2026). Systems with small models but high-quality retrieval/memory architectures recover most of the gains of large models at a fraction of the cost. Hybrid routing (combining multiple retrieval pipelines or memory tiers) consistently achieves optimal or near-oracle accuracy with substantial reductions in token, RAM, and latency budgets. Tables 1 and 2 below summarize quantitative impacts:

Table 1: Performance and Efficiency in Recent Routing Systems

System	Routing Strategy	Accuracy gain / metric	Resource savings
HTM-EAR	Tiered, importance-aware	Active MRR=1.000	0 essential lost
MemRouter	Embedding-based admission	F1=52.0 vs 45.6 LLM manager	17× write-phase speed
SelRoute	Query-type routing	Recall@5=0.800 (bge-base)	No LLM/GPU required
ShardMemo	Masked MoE, scope-mask	+6.87 F1, 20% fewer VecScan	-19% p95 latency
Telco-oRAG	Neural series router	+17.6% (3GPP MCQ)	-45% RAM
PRISM	Intent-based, graph path	0.831 judge, 0.411 eff	13× fewer context tokens

Table 2: Failure Modes and Future Research Directions

Limitation	Observed in	Impact / Quantitative Finding
Routing calibration bottleneck (ICICLE)	(Den et al., 26 May 2026)	Hits@1 on new docs drops as candidates grow (1.0→0.607)
Reasoning-agnostic pipelines (SelRoute)	(McKee, 2 Apr 2026)	Recall@5 = 0.149 (RECOR, multi-hop)
Open-loop retrieval, no reflective control	(Du et al., 23 Dec 2025)	–13% judge accuracy if masked/reflect removed
Over-retrieval (uniform/pattern-based)	(Gaikwad, 8 Mar 2026)	Increased context, lower QA accuracy (coverage 100% but low EM)

Extending current routing paradigms to support more expressive, learned policies (reinforcement learning, differentiable gates, hybrid symbolic-neural routes), multi-hop or compositional retrieval, and domain-specific architectural integration remains an active area with direct application to LLM- and agent-driven reasoning, persistent monitoring, and high-frequency interaction environments.