SPEAR: Automated Evaluation for RAG

• SPEAR is a framework that automates pseudo ground truth generation with subset sampling and LLM-based minimal fact extraction to evaluate retrievers in RAG systems.
• It significantly reduces annotation costs by aggregating candidate samples while preserving performance ordering and ensuring high precision and comparable recall.
• SPEAR computes robust metrics like PR-AUC and precision across diverse configurations, enabling scalable, domain-specific optimization of retrieval pipelines.

SPEAR (Subset-sampled Performance Evaluation via Automated Ground Truth Generation for RAG) is a framework for evaluating retrievers in retrieval-augmented generation (RAG) systems via automated pseudo ground truth generation and comprehensive retrieval metrics. SPEAR addresses the critical bottleneck of expensive or impractical evaluation in complex or domain-specific RAG applications by leveraging subset sampling and LLM-based minimal fact extraction to construct a scalable, accurate, and low-cost evaluation protocol (Yuheng et al., 9 Jul 2025).

1. Subset Sampling and Automated Pseudo Ground Truth

SPEAR builds its evaluation framework around subset sampling, designed to circumvent the prohibitive annotation costs and time consumption of full ground truth (GT) construction for massive or rapidly-changing knowledge bases. The framework aggregates all knowledge chunks retrieved by any configuration (“candidate retrievers”) into a subset. This sampled subset contains all positive samples (relevant chunks retrieved by any method), with a substantially reduced number of negative samples. The key property of the method is that it maintains the relative performance ordering between retrievers, even as the annotation burden is drastically reduced. Mathematically, if the knowledge base size is $N$ and is reduced to $N’ \ll N$ through sampling, computational cost transitions from $M \times N$ to $M \times N’$. Proofs in the appendix establish that the precision calculated from the pseudo GT is identical to that from the true GT; recall is shown to be closely matched.

Automation is further enabled by using LLMs to extract “minimal retrieval facts” from retrieved chunks—removing overlapping, redundant, or superfluous content and isolating atomic semantic units that correspond to the answer-relevant parts of the knowledge base.

2. Evaluating Retrievers in RAG Systems

SPEAR enables robust, automated evaluation of retrievers under realistic workload scenarios. Candidate retrievers may differ along segmentation strategy (e.g., chunking granularity), embedding model (e.g., retriever-specific representation), or retrieval method (e.g., vector search, hybrid). For each, SPEAR executes batch evaluations over authentic user queries, collects the retrieved chunks, and processes them through LLM-based semantic segmentation to distill minimal ground truth facts.

A comprehensive suite of retrieval metrics is then automatically computed across the sampled pseudo GT:

• Precision (the ratio of relevant retrieved chunks to total retrieved).
• Recall (the ratio of relevant retrieved to total relevant).
• PR-AUC (precision–recall area under the curve), which is preferred over F-score for domain-adaptivity and better performance under imbalanced data.
• Fβ-score is also defined:

$$F_\beta = (1 + \beta^2) \cdot \frac{\text{precision} \times \text{recall}}{(\beta^2 \cdot \text{precision}) + \text{recall}}$$

Although Fβ can trade off precision and recall via the β parameter, PR-AUC is chosen by the authors due to the difficulty of selecting a universal β across distinct RAG domains.

The core process reduces manual annotation overhead and provides a fair and directly comparable performance evaluation among retrievers, even when segmentation granularity or query rewriting are involved.

3. Application Scenarios and Results

SPEAR’s methodology is validated in multiple real-world RAG applications:

• Knowledge-based Q&A: “Linghang Shu,” an internal enterprise consultation assistant, was used to paper 24 different retriever configurations, varying chunk sizes, retrieval, and embedding methods. SPEAR identified optimal configurations—e.g., nms-style segmentation with a conan embedding model and hybrid search—which empirically improved both recall and precision.
• Retrieval-based Travel Assistant: The pipeline included query rewriters, POI/length/engagement filters, and rerankers. SPEAR’s evaluation protocol isolated configuration effects, allowing for selection of optimal pipelines in terms of user-perceived relevance and factuality.

This approach allows for scenario-specific retriever optimization, increasing QA response quality and user satisfaction for practical deployments.

4. Technical Specifics and LLM-based Fact Extraction

The framework’s minimal fact extraction step addresses a key challenge of chunk-based retrieval evaluation: different segmentation policies often produce overlapping content, e.g., with redundant prefix/suffix text. SPEAR employs LLMs for semantic segmentation of each retrieved chunk, extracting only the minimal and relevant text block(s) that match the query context. This normalization is required to ensure that recall and precision are not artificially inflated or deflated by irrelevant segment boundaries.

False positive and recall metrics are thus normalized across retrievers that operate with non-identical chunking strategies. The process is entirely automated—after candidate retrieval, LLMs segment and distill the GT, then metrics are computed without manual intervention.

Mathematical derivations in Appendix A verify that, under the subset sampling and LLM extraction protocol, pseudo GT precision matches true GT, while recall retains a close correspondence.

5. Comparison with Existing Evaluation Paradigms

SPEAR contrasts sharply with traditional RAG evaluation paradigms:

• Cost Efficiency: By labeling and evaluating only a small fraction $K \ll N$ of the knowledge base (the aggregate of all chunks retrieved by any configuration), SPEAR achieves significant reduction in computational and annotation costs.
• Evaluation Fidelity: Unlike synthetic-data-based toolkits (e.g., Ragas), which generate queries and answers in distributional isolation, SPEAR evaluates with real user queries and pseudo GT, maintaining scenario relevance and domain authenticity. Its automation allows ongoing, iterative retriever evaluation as queries and the corpus evolve.
• Adaptivity: By computing PR-AUC instead of fixed-threshold F-scores, SPEAR outcomes remain interpretable even under highly imbalanced or sparsely-answered retrieval scenarios, a known challenge in enterprise and consumer RAG applications.

A summary comparison:

Method	Annotation overhead	Domain-specific evaluation	Handles segmentation issues?
Human GT	High	Yes	No
Ragas	Low	No (synthetic)	No
SPEAR	Low	Yes (true user queries)	Yes (LLM fact extraction)

6. Limitations and Future Directions

The methodology acknowledges trade-offs between computational efficiency (via subset sampling and automation) and evaluation accuracy. Precision is strictly preserved; recall is only “relatively comparable”—in some cases, minor discrepancies can emerge if positive samples are omitted from the aggregate subset. Unanswerable or ambiguous queries are not yet explicitly handled; this could lead to overestimation of failure rates for queries not covered by the corpus. A plausible enhancement is the introduction of query difficulty metrics or retrieval confidence estimation.

Advances in LLM-based factuality detection and minimal fact segmentation are expected to further improve robustness, enabling more sophisticated evaluation in cases of semantic drift or heterogeneous answer formulations. This suggests SPEAR could in future dynamically adapt its evaluation sample size or filtering protocols to match the available resources or accuracy targets.

7. Broader Significance and Impact

By making large-scale, low-cost retriever evaluation accessible for RAG systems across diverse and domain-specific business scenarios, SPEAR directly accelerates the optimization and deployment of high-performing LLM-based assistants, Q&A engines, and retrieval-based recommender systems. The framework’s applicability has been empirically demonstrated in enterprise knowledge management and travel-assistant pipelines, and its design is readily extensible to other real-world retrieval applications where precise, workload-specific retriever tuning is necessary.

In summary, SPEAR provides an automated, scientifically rigorous, and resource-efficient protocol for retriever comparison and selection in RAG systems—addressing practical evaluation bottlenecks and supporting ongoing model improvement in rapidly evolving real-world settings (Yuheng et al., 9 Jul 2025).