SPAR: Adaptive Retrieval for Enterprises

• SPAR is a retrieval framework that separates semantic metadata indexing from session-specific vector database construction to improve search relevance.
• It scales retrieval for heterogeneous, large enterprise datasets by leveraging metadata filters and on-demand embedding for efficient, structured searches.
• Experimental evaluations indicate SPAR achieves a 9.2 pp recall improvement and over 2x faster query times compared to traditional RAG systems.

SPAR (Session-based Pipeline for Adaptive Retrieval) is a conceptual framework for retrieval-augmented generation (RAG) in enterprise settings, designed to address the inefficiencies and limitations of legacy file systems that lack semantic organization and structured metadata. By introducing a lightweight, two-stage pipeline that separates semantic metadata indexing from session-specific vector database construction, SPAR achieves scalable, controllable, and more relevant retrieval for large, heterogeneous data corpora inaccessible to conventional global vector-indexing RAG approaches (Nguyen et al., 15 Dec 2025).

1. Motivation and Challenges in Legacy File Systems

Enterprise environments accumulate data in sprawling, unstructured file hierarchies with minimal or inconsistent metadata. Documents of varied types and domains (financial reports, logs, images, etc.) are typically stored with no meaningful semantic organization, resulting in three fundamental challenges for retrieval and analytics:

• Absence of structured metadata: Retrieval is necessarily “blind,” reducing semantic precision as there are no consistent tags or taxonomies.
• High computational costs of global indexing: Conventional RAG approaches require up-front embedding and indexing of all $N$ files: $O(N T_\mathrm{proc})$ preprocessing and $O(N\log N)$ for approximate nearest neighbor (ANN) index construction.
• Synchronizing with a dynamic file system: As files are created, moved, or updated, a monolithic global index rapidly becomes stale or requires expensive re-ingestion.

Conventional RAG architectures fail to address these issues, as they necessitate brute-force embedding, maintenance of a global vector database, and post-hoc filtering at query time. This results in high computational and memory overheads and fails to exploit user knowledge or query context to scope retrieval (Nguyen et al., 15 Dec 2025).

2. SPAR Architecture: Two-Stage Adaptive Retrieval

SPAR reconstructs the retrieval workflow around two central observations: (1) most queries are naturally scoped to manageable subsets of files (by time, project, department, etc.), and (2) enterprise users possess domain knowledge about relevant groupings even when these are not formally represented.

The pipeline comprises two stages:

a. Semantic Metadata Index Construction

A relational index is built over two tables:

• Files: $$(\mathrm{File\_ID, File\_Path, Metadata\_Fields, Tag\_IDs})$$
• Tags: $$(\mathrm{Tag\_ID, Tag\_Value, Parent\_Tag\_ID, Children\_Tag\_IDs})$$

A controlled vocabulary of $M$ enterprise “leaf” tags (e.g., MeSH terms) is assigned via a one-time manual/LLM-assisted process, then organized into a hierarchical taxonomy, often modeled as a DAG through prefix expansion or LLM clustering. This index enables fast tag-based file filtering ($O(M\log M)$ for build; $O(N_\mathrm{candidates})$ per query).

b. On-Demand, Session-Specific Vector Databases

Upon a query:

• The user specifies workspace scope through metadata and hierarchical tag filters, yielding a candidate subset of size $N_\mathrm{filtered} \ll N$.
• Files in the workspace are embedded on demand: $O(N_\mathrm{filtered} T_\mathrm{proc})$.
• A lightweight approximate nearest neighbor index (e.g., HNSW) is built over the embeddings: $O(N T_\mathrm{proc})$0.
• Queries are processed by vector similarity search (cosine, etc.) at $O(N T_\mathrm{proc})$1 per query.
• All session artifacts (embeddings, workspace index) exist only ephemerally; after the session, embeddings can be discarded while retaining metadata for future archive/sharing.

This design avoids the pitfalls of monolithic indices and allows workspaces to be “just large enough” for the task at hand, maximizing interactive responsiveness and control (Nguyen et al., 15 Dec 2025).

3. Complexity Analysis and Comparative Advantages

Theoretical comparison with conventional RAG pipelines reveals substantial computational and memory advantages:

	Traditional RAG	SPAR
Construction time	$O(N T_\mathrm{proc})$2	$O(N T_\mathrm{proc})$3
Query search	$O(N T_\mathrm{proc})$4 (global index)	$O(N T_\mathrm{proc})$5
Memory usage	$O(N T_\mathrm{proc})$6	$O(N T_\mathrm{proc})$7

By ensuring $O(N T_\mathrm{proc})$8 and moderating the number of concurrent workspaces $O(N T_\mathrm{proc})$9, SPAR dramatically reduces both up-front and per-query computational costs, achieving amortization benefits with minimal memory duplication (Nguyen et al., 15 Dec 2025).

4. Experimental Evaluation and Empirical Results

In a synthetic deployment on a 1,000-article biomedical file system (PMC Open Access), SPAR was compared with a canonical RAG system using Pinecone for global vector indexing:

• Retrieval Accuracy (Top-5): 89.5% (SPAR) vs. 80.3% (RAG)
• Average Retrieval Time: 0.015 s (SPAR) vs. 0.039 s (RAG)
• Downstream Answer Accuracy: 68.1% (SPAR) vs. 65.1% (RAG)

Despite utilizing the same LLM agent for downstream generation, SPAR achieves a 9.2 pp improvement in recall, over 2x faster retrieval, and a 3 pp improvement in answer accuracy. These gains derive directly from hierarchical tag filtering and scoping, which concentrate attention on semantically relevant subsets (Nguyen et al., 15 Dec 2025).

5. Design Trade-Offs, Limitations, and Open Research Challenges

SPAR’s efficiency and focus derive from session-scoping, but this approach introduces several trade-offs:

• Startup Overhead: Each new workspace incurs a one-time embedding and ANN index build for its candidate set, mitigated but not eliminated by local caching.
• Transient Storage Redundancy: Embedding the same file in concurrent sessions could consume excess memory; shared caches or deduplication remain an open area.
• Metadata Dependency: The benefits of SPAR are contingent on reasonably complete and accurate tagging. Incomplete or noisy metadata increases false negatives; while LLM-assisted tagging offers partial mitigation, human-in-the-loop or constraint-enforced pipelines may be necessary.

Active research directions foreground:

• Shared or incremental workspace indexing,
• Adaptive pre-warming caches based on query statistics,
• Robust pipelines for automatic and audited metadata tagging,
• Cross-workspace collaboration mechanisms that preserve privacy and provenance (Nguyen et al., 15 Dec 2025).

6. Broader Implications and Significance

SPAR constitutes a pragmatic rethinking of LLM-based RAG for legacy enterprise environments where global vector databases are infeasible. By exploiting latent domain logic already available to users, SPAR achieves superior interactivity, transparency, and efficiency. Rather than seeking completeness through brute-force global indexing, SPAR dynamically constructs sufficient indices, converting intractable file system search into manageable, session-scoped interactive tasks.

The architectural decoupling from large vector databases, combined with robust metadata hierarchies and ephemeral workspaces, renders SPAR particularly well-suited for enterprises with vast, dynamic, and heterogeneously organized corpora, where the costs and risks of building and maintaining monolithic retrieval infrastructure are prohibitive (Nguyen et al., 15 Dec 2025).