Search Is Not Retrieval: Decoupling Semantic Matching from Contextual Assembly in RAG (2511.04939v2)

Abstract: Retrieval systems are essential to contemporary AI pipelines, although most confuse two separate processes: finding relevant information and giving enough context for reasoning. We introduce the Search-Is-Not-Retrieve (SINR) framework, a dual-layer architecture that distinguishes between fine-grained search representations and coarse-grained retrieval contexts. SINR enhances the composability, scalability, and context fidelity of retrieval systems by directly connecting small, semantically accurate search chunks to larger, contextually complete retrieve chunks, all without incurring extra processing costs. This design changes retrieval from a passive step to an active one, making the system architecture more like how people process information. We discuss the SINR framework's conceptual foundation, formal structure, implementation issues, and qualitative outcomes. This provides a practical foundation for the next generation of AI systems that use retrieval.

• The paper introduces the SINR framework, which decouples semantic matching and contextual assembly using a dual-layer architecture with small search and larger retrieve chunks.
• It utilizes deterministic parent mapping to consolidate overlapping chunks, thereby improving narrative continuity and reducing redundancy.
• Empirical evaluations show SINR achieves 15–25% higher recall and 40–60% reduced search index size, demonstrating efficiency gains over conventional RAG systems.

Decoupling Semantic Matching and Contextual Assembly in Retrieval-Augmented Generation: The SINR Framework

Introduction

Retrieval-Augmented Generation (RAG) architectures underpin contemporary LLM pipelines, enabling large models to dynamically access external knowledge. However, traditional RAG systems conflate two fundamentally distinct retrieval objectives: (1) precise localization of relevant knowledge and (2) assembly of contextually sufficient evidence for downstream reasoning. This paper formalizes and resolves the resulting structural tension by introducing the Search-Is-Not-Retrieve (SINR) framework—a dual-layer architecture designed to independently and optimally address semantic matching and contextual assembly.

Motivation and Theoretical Foundations

Current RAG pipelines typically chunk text into fixed windows, each embedded into a vector space for nearest neighbor search. Small chunks ensure high search precision but fracture discourse continuity; large chunks preserve context but dilute semantic density. This unitary approach forces a compromise, impacting both the relevance and the utility of retrieved contexts.

The SINR framework posits that semantic matching (search) and contextual assembly (retrieve) have orthogonal requirements. Inspired by cognitive models of information foraging, it operationalizes a dual-layer representation:

• Search chunks ($s_i$): Small (100-200 tokens), semantically dense units optimized for embedding-based similarity search
• Retrieve chunks ($r_j$): Larger (600-1000 tokens), structurally or semantically coherent units that serve as complete reasoning contexts

A deterministic parent mapping $f_\mathrm{parent}: S \rightarrow R$ constrains each search chunk to a unique retrieve chunk, enabling efficient and traceable hierarchical retrieval.

SINR Architecture and Algorithms

Corpus Segmentation and Indexing

Documents are segmented into retrieve chunks via structural heuristics (e.g., section headers, paragraph boundaries) or adaptive methods. Each retrieve chunk is further divided into overlapping search chunks with small window and stride (e.g., 150 tokens, 100 token stride). Embeddings for search chunks are created using established dense encoders (e.g., BERT variants, Sentence-BERT), forming a high-granularity semantic index. Metadata, including parent retrieve chunk IDs, is stored alongside embeddings in a vector database supporting ANN search (e.g., FAISS-HNSW, Milvus, Pinecone).

Parent Mapping

A lightweight mapping (hash table or B-tree) tracks parent-child relationships. Storage cost is minimal—approximately 16 bytes per search chunk—compared to embedding and text storage. This mapping enables $O(1)$ or $O(\log n)$ lookup of retrieve chunks for any subset of search chunk hits.

Retrieval Pipeline

Given a query $q$:

1. Embed: Encode $q$ into an embedding vector $\mathbf{q}$.
2. Semantic Search: Use the vector index to find top-$k$ nearest search chunks $S_\mathrm{top}$ to $\mathbf{q}$.
3. Parent Lookup: Map each $s_i \in S_\mathrm{top}$ to its unique parent retrieve chunk $r_j = f_\mathrm{parent}(s_i)$.
4. Context Assembly: Deduplicate and aggregate parent retrieve chunks $R_\mathrm{top}$; supply them in full for downstream model consumption.

This process separates the fine-grained localization of evidence from the assembly of coherent, contextually rich inputs for LLMs. Overlapping search chunks ensure robustness against semantic boundary splits, while parent aggregation eliminates retrieval redundancy.

Complexity and Scalability

For $n$ search chunks and $m$ retrieve chunks ($m \ll n$), retrieval is dominated by ANN search ($O(\log n)$) and mapping/deduplication ($O(k)$, $k \ll n$). Storage and update costs scale linearly with corpus size, but parent mapping overhead remains $<$1% of the embedding storage.

Empirical deployments reported 40--60% reductions in search index size and 20--30% lower average query latency relative to flat RAG pipelines, with improved recall and context fidelity.

Distinctive Properties and Practical Implications

Interpretability and Traceability

SINR offers inherent interpretability: each output can be traced from model response back through the unique chain $$\mathbf{q} \rightarrow S_\mathrm{top} \rightarrow R_\mathrm{top} \rightarrow \text{Answer}$$. This granular path enables transparent auditing of information provenance—critical for high-stakes applications in law, healthcare, and finance.

Modularity and Independent Tuning

SINR's decoupling of search and retrieve enables independent optimization—search chunk size can be tuned for semantic matching without impacting context granularity, and retrieval context size can be aligned with reasoning requirements or token budgets without affecting search index density. This modularity dramatically simplifies experimentation and reduces infrastructure update costs, particularly significant at enterprise and internet scales.

Coherent Contextual Retrieval

SINR addresses the fragmentation endemic to flat chunking. By aligning retrieval units with natural discourse structure, it improves logical flow and reduces model hallucination. This architecture is especially advantageous with long-context models (e.g., GPT-4, Claude 2+) capable of ingesting extended coherent spans.

Efficient Incremental Updates

Incremental corpus modifications necessitate only localized re-segmentation and re-embedding—no global reindexing is required. Parent mappings and embeddings can be updated for affected documents independently, supporting rapid, minimally disruptive corpus evolution.

Evaluation and Empirical Observations

Evaluation Dimensions

SINR's impact is measured across: semantic precision, contextual completeness, faithfulness (groundedness), efficiency (latency and storage), and traceability.

Qualitative assessments and prototype deployments revealed the following:

• Increased narrative continuity: Contexts retrieved under SINR preserved structural integrity and logical flow.
• Stable search precision: Embedding-based search over small chunks maintained or improved recall@20.
• Reduced redundancy: Parent aggregation consolidated overlapping hits, decreasing repeated context.
• Enhanced auditability: Direct mapping between query, matched evidence, and resulting context improved system transparency and facilitated debugging.

In benchmark scenarios, SINR achieved 15--25% higher recall@20 and 30% higher contextual coherence compared to fixed-chunk RAG. Real-world deployments spanning millions to billions of documents demonstrated the approach's scalability and efficiency.

Deployment Considerations and Limitations

System Integration

SINR can be integrated into existing RAG toolkits (e.g., LangChain, LlamaIndex) by decoupling the chunking and retrieval modules and introducing the dual-layer mapping. Optimal technology stacks differ by deployment scale; in-memory hash tables and FAISS suffice for smaller corpora, while cloud-native vector stores and distributed key-value stores (e.g., Redis, DynamoDB) are recommended for large-scale deployments.

Constraints

• Short documents: For inputs $<$200 tokens, dual-layer chunking is unnecessary.
• Non-structured sources: SINR requires discernible document structure for effective retrieve chunk definition.
• Extreme context limits: SINR is most valuable when the downstream model's context window is sufficiently large to admit multi-chunk passages.
• Initialization: Defining retrieve chunk boundaries is nontrivial; semantic or structural heuristics are required.

Research Directions and Broader Impact

SINR introduces several open research avenues:

• Learned chunking strategies: Optimizing retrieve chunk boundaries via data-driven methods rather than static heuristics.
• Dynamic context sizing policies: Adapting retrieve chunk size based on query intent or downstream usage.
• Multi-modal and hierarchical generalizations: Extending SINR beyond text to include images, tables, and deeper document hierarchies.
• Integration with agentic systems: Leveraging SINR as an interpretable memory architecture for autonomous and interactive agents.

SINR also exemplifies "data-centric AI," emphasizing the impact of information structure on ML system reliability and maintainability.

Conclusion

The Search-Is-Not-Retrieve (SINR) framework formalizes and resolves the inherent tension between semantic matching and contextual assembly in RAG systems. By introducing explicit, deterministic separation between search and retrieve layers, it enables simultaneously high search precision and robust, contextually complete reasoning—a balance that flat chunking cannot achieve. SINR’s traceability, modularity, and scalability provide both immediate practical benefits for real-world RAG deployments and a conceptual advance for the design of interpretable, maintainable AI systems. Future exploration of adaptive and learned strategies for chunking and context assembly, especially in multi-modal and agentic settings, will further extend SINR’s applicability and impact on information retrieval architectures.