SmartSearch: How Ranking Beats Structure for Conversational Memory Retrieval

Abstract: Recent conversational memory systems invest heavily in LLM-based structuring at ingestion time and learned retrieval policies at query time. We show that neither is necessary. SmartSearch retrieves from raw, unstructured conversation history using a fully deterministic pipeline: NER-weighted substring matching for recall, rule-based entity discovery for multi-hop expansion, and a CrossEncoder+ColBERT rank fusion stage -- the only learned component -- running on CPU in ~650ms. Oracle analysis on two benchmarks identifies a compilation bottleneck: retrieval recall reaches 98.6%, but without intelligent ranking only 22.5% of gold evidence survives truncation to the token budget. With score-adaptive truncation and no per-dataset tuning, SmartSearch achieves 93.5% on LoCoMo and 88.4% on LongMemEval-S, exceeding all known memory systems under the same evaluation protocol on both benchmarks while using 8.5x fewer tokens than full-context baselines.

• The paper introduces a deterministic retrieval pipeline using rule-based expansion and ranking to achieve superior token efficiency.
• It employs a CrossEncoder and ColBERT reranker to optimize token allocation, achieving 93.5% and 88.4% accuracy on benchmark tests.
• The work challenges reliance on LLMs for memory structuring by revealing that effective reranking is more critical than raw retrieval recall.

SmartSearch: Deterministic Memory Retrieval and Compilation Bottlenecks in Long-Conversation LLM Agents

Overview and Motivation

This paper introduces SmartSearch, a retrieval system for conversational LLM memory whose core claim is that sophisticated LLM-based ingestion and memory structuring are superfluous: nearly all effective retrieval in long-horizon conversational agents can be achieved through deterministic, LLM-free pipelines, provided ranking and truncation are handled with care. SmartSearch queries raw, unstructured conversation logs using a deterministic set of retrieval and rule-based expansion operations, with a CrossEncoder and ColBERT-based reranking stage as the sole learned component, all operating efficiently on CPU.

The empirical results demonstrate that SmartSearch surpasses the state of the art on both the LoCoMo and LongMemEval-S benchmarks, achieving 93.5% and 88.4% accuracy respectively, exceeding baselines while using up to 8.5$\times$ fewer context tokens than full-context approaches. The architecture foregrounds the insight that retrieval recall is not the bottleneck in long-term conversational memory—rather, effective ranking and token-budget allocation are.

Methodology

Pipeline Architecture

SmartSearch eliminates LLM involvement from both the ingestion and query phases, deploying a purely deterministic retrieval process:

1. Query Parsing: Queries are analyzed using SpaCy-based NER and POS tagging. Extracted tokens are linguistically weighted (proper nouns > nouns > verbs; NER entities receive a bonus), shifting importance estimates from corpus-statistical (e.g., BM25/IDF) to linguistically-driven signals.
2. Substring-Based Document Retrieval: Candidates are identified using NER-weighted substring (exact match) search. Rule-based entity extraction on top retrieved passages supplies new expansion terms for multi-hop queries, enabling coverage of multi-step reasoning tasks without learned policies or LLM-generated queries.
3. Ranking: The only learned components are a CrossEncoder (DeBERTaV3-based, 435M parameters, MS MARCO fine-tune) and a ColBERT reranker, fused via Reciprocal Rank Fusion. Both models run efficiently on CPU ($\sim$650ms per query in parallel). Reranker quality is shown to be the dominant factor in pushing gold-standard evidence to high rank for inclusion.
4. Truncation: Instead of a fixed token budget, the ranking stage enables score-adaptive truncation based on relative score thresholds, allowing more dynamic allocation of context for complex/difficult queries.

Query Expansion

Expansion mechanisms compensate for the intrinsic brittleness of exact-match retrieval. Named entity discovery and pseudo-relevance feedback (PRF) are used to surface supplementary candidate passages, especially beneficial in multi-session or temporally-dispersed information queries.

Index-Free Variants

The pipeline admits a fully index-free variant, removing all learned dense retrieval models and semantic indices. On small-to-medium corpora, this variant loses negligible recall (<1pp); on larger corpora, query expansion mechanisms largely recoup any shortfall.

Experimental Results

LoCoMo and LongMemEval-S Benchmarks

Comprehensive evaluation on two prominent long-context benchmarks demonstrates consistent gains:

• LoCoMo (${~}$9k tokens/conv, 1,540 questions): SmartSearch achieves 93.5% accuracy with only 3,100 tokens presented to the LLM answerer, exceeding EverMemOS (92.3%, 2,300 tokens) and Memora (86.3%, $\sim$8,500 tokens). The system's open-ended question accuracy exceeds structured-memory baselines by over 20pp—attributed to preservation of conversational “texture” lost in abstraction-based systems.
• LongMemEval-S (${~}$115k tokens/conv, 500 questions): Both index-free (88.4%) and indexed (87.6%) SmartSearch variants surpass recent SOTA (Memora 87.4%, EverMemOS 83.0%). Notably, the index-free pipeline leverages expansion to nearly close the gap with learned dense retrieval. Large improvements are seen in multi-session and user-centric categories.
• Reranker Ablation: Upgrading reranker quality provides up to +7pp, with CrossEncoder- and ColBERT-based fusion outperforming BM25 and single-model approaches.
• Compilation vs. Retrieval Bottleneck: Oracle trace analysis reveals retrieval recall up to 98.6%, but only 22.5% of gold evidence is retained after token-budget truncation without an intelligent ranking/truncation strategy—solidifying that ranking, not retrieval, governs performance.

Robustness and Failure Analysis

• On both datasets, performance is robust across a range of retrieval/ranking configurations, with maximal sensitivity to the quality of the reranker and token allocation method.
• Index-free performance scales well with expansion, especially on long-dialog tasks, where exact match alone would otherwise fail due to vocabulary/inflection mismatch.
• The dominant error mode in the final system is not retrieval failure but LLM answer synthesis: in over 59% of errors, gold passages are present but the answer LLM fails to derive the correct response.

Comparative Analysis

The findings directly contradict several pervasive assumptions:

• LLMs are not required in the retrieval loop for strong conversational memory: deterministic query analysis and rule-based multi-hop expansion suffice.
• Precomputed indices and dense embeddings are not necessary at conversational scale: for most practical corpora, exact match plus expansion, ranked by a sufficiently strong model, suffice.
• Multi-hop reasoning does not require complex policies: 97% of LoCoMo queries resolve in a single hop; the few true multi-hop cases are handled rule-based entity discovery.
• Memory abstraction/consolidation can degrade open-ended and user-centric recall: structured-memory systems compress away conversational details that are critical for certain query types.

Implications and Future Directions

By shifting focus from maximizing retrieval recall to minimizing compilation losses (i.e., improving reranker precision and optimizing token usage), this work sets a new direction for long-context retrieval: ML models should be concentrated at the ranking and selection stages, with deterministic, high-recall retrieval systems as the backbone. The method’s efficiency (no GPU, sub-second latency) and lack of dependency on expensive, opaque memory structuring make it pragmatically attractive for deployment scenarios.

Future research directions include:

• Passage Deduplication and Compression: Leveraging computed representations to reduce redundancy and further increase information density in the LLM context window.
• Generalization to Real-World, Unstructured Conversational Data: Benchmarks tested exhibit regular structure and high entity density—examining more diverse corpora will clarify limits.
• Domain and Language Transfer: Extending deterministic, expansion-driven retrieval to non-conversational and multilingual corpora remains unexplored.
• Optimized Answer Models: As answer LLM synthesis is the dominant failure point, developing memory-aware answer LLM prompting or custom models could further raise accuracy.

Conclusion

SmartSearch advances the state of conversational LLM memory by demonstrating a regime in which deterministic, LLM-free retrieval—augmented by targeted reranking and expansion—recovers or exceeds the performance of sophisticated, compute-intensive memory structuring pipelines. This shift challenges accepted design patterns in RAG systems, emphasizing that the true bottleneck lies in post-retrieval evidence selection, not recall. As long-context LLM agents scale, architectural simplicity, token efficiency, and transparent performance tuning—as exemplified by SmartSearch—will likely become central to both theoretical and applied research in long-horizon agent memory.