Candidate-Constrained Retrieval-Augmented Generation for LongEval-RAG: System Design and Empirical Analysis

Abstract: We present a candidate-constrained retrieval-augmented generation system for LongEval-RAG, where each query is associated with an organizer-provided candidate set and all retrieved evidence and final citations must remain within that set. The system combines deterministic provenance tracking with passage-based retrieval, deterministic query expansion, pseudo-relevance feedback (PRF), reciprocal rank fusion (RRF), lightweight evidence reranking, citation-aware evidence aggregation, and optional MiniLM sentence reranking. We evaluate ten pipeline variants using a primary organizer evaluation and a supplementary self-generated diagnostic protocol. The primary evaluation shows that the strongest balanced variant is rule-minilm: a rule-based chunking pipeline with query expansion, PRF, RRF, reranking, citation prior, and late MiniLM sentence selection. This variant obtains the highest BERTScore, retrieval precision, nugget coverage, and average grade among our submissions. The result suggests that the main gain does not come from more complex semantic or topic-shift chunking, but from pairing stable rule-based evidence units with sentence-level neural selection before generation. The supplementary LLM-judge evaluation remains useful for early diagnosis and additional analysis, but it emphasizes different systems than the primary gold-answer and nugget-based evaluation, highlighting the need for multi-metric RAG evaluation.

• The paper presents a candidate-constrained retrieval-augmented generation approach that limits evidence selection to a fixed document pool.
• It evaluates deterministic provenance tracking, varied passage segmentation techniques, and late-stage MiniLM reranking to enhance semantic alignment and evidence precision.
• Experimental analysis shows that rule-based chunking with multi-query expansion and neural reranking consistently outperforms other configurations.

Candidate-Constrained Retrieval-Augmented Generation for LongEval-RAG: System Design and Empirical Analysis

Problem Setting and Motivation

LongEval-RAG transitions from open-domain RAG by framing answer generation over an organizer-supplied candidate set, rather than unrestricted corpus search. All retrieved evidence and generated citations must be drawn from a pre-specified set of document IDs per query, bounding the retrieval space and focusing methodological rigor on how passages are constructed, ranked, and traced back to their provenance. This candidate-constrained setting removes corpus recall as a variable, thus exposing strengths and weaknesses in evidence selection, passage segmentation, and sentence allocation strategies.

System Architecture

The system pipeline comprises deterministic provenance tracking, passage-based retrieval, multi-faceted query expansion, pseudo-relevance feedback (PRF), reciprocal rank fusion (RRF), lightweight reranking incorporating document citation priors and optional temporal signals, and late-stage cross-encoder MiniLM reranking at the sentence level. This architecture enforces candidate constraints throughout, supported by a data loading/join protocol that loads only the candidate documents per query and segments them into retrieval units according to different chunking strategies.

Figure 1: Overall pipeline used in the supplementary LongEval Task 4 experiments, covering candidate-constrained retrieval, answer generation, and cross-provider evaluation.

Passage construction is parameterized across three main variants: rule-based chunking (fixed word threshold with overlap), semantic chunking (embedding similarity-based merging via all-MiniLM-L6-v2), and topic-shift chunking (local cosine drift detection between sentences). For retrieval, baseline BM25 is supplemented by deterministic query expansion, PRF-derived query variants, and all lists are fused via RRF with a fixed hyperparameter ($k=60$).

Lightweight reranking incorporates citation prior (proportional to document citation graph edges) and an optional temporal encouragement for covering cited documents from diverse publication years, configured for temporally-phrased queries.

Final answer generation operates on a narrowed set of up to 10 candidate sentences, potentially reranked by a MiniLM cross-encoder, and submitted to an LLM generator (OpenAI gpt-5.4-mini) with the original provenance and citation mapping preserved.

Experimental Variants

Ten distinct pipeline configurations are reported, encompassing ablations for baseline retrieval (concat, BM25), multi-query fusion, chunking variant (rule-based, semantic, topic-shift), and with/without late-stage MiniLM sentence reranking. The "rule-minilm" variant—rule-based chunking with full retrieval enrichment and MiniLM evidence reranking—emerged as the most effective balanced strategy under primary evaluation.

Evaluation Protocols

Two contrasting evaluation protocols are used:

• Primary (Organizer) Evaluation: Multi-metric performance across ROUGE-L, BERTScore, retrieval precision (fraction of cited documents judged relevant), nugget coverage/recall (minimal information unit coverage), average rubric grade, and TFC1 (query-term-focused content measure).
• Supplementary Diagnostic Evaluation: LLM-as-judge (Claude Sonnet 4.5) scoring of correctness, completeness, document ID overlap (Recall, Precision, Jaccard) relative to a reference-style answer, and detailed error analysis metrics.

Each protocol exposes different aspects of RAG performance—surface lexical similarity and factual coverage, evidence recall/precision, and provenance faithfulness.

Results and Analysis

Primary Metric Findings

• Rule-minilm is the strongest balanced configuration, achieving the highest BERTScore (0.168), retrieval precision (0.975), nugget coverage (0.367), and average nugget grade (2.929). The configuration pairs stable, deterministic rule-based passage boundaries with late MiniLM cross-encoder reranking at the sentence level to allocate the evidence budget.
• Richer chunking strategies (semantic, topic-shift) modify failure profiles: Topic-shift chunking yields the highest ROUGE-L (0.167) and TFC1 (0.373), while semantic chunking slightly boosts BERTScore and evidence measures, at a minor cost in precision.
• MiniLM sentence reranking is most effective when applied late, after passage selection, rather than substituting semantic chunking for heuristic passage boundaries, particularly for maximizing semantic answer similarity and evidence precision.
• Complex chunking does not outperform strong rule-based baselines under the candidate-constrained regime with tight document pools. Embedding-based segmentation is not harmful but offers limited gains given the experimental constraints.

Supplementary Diagnostic Findings

• Supplementary LLM-judge evaluation prefers the rrf-no-rerank variant (highest mean correctness/completeness score: 2.734) over rule-minilm, suggesting that local fluency and broad coverage are rewarded by the LLM-as-judge protocol, but not always aligned with evidence precision or nugget coverage.
• All retrieval-augmented systems outperform the concat baseline by large margins across both judge and overlap metrics.
• Primary and supplementary metrics disagree informatively: The supplementary protocol can overvalue plausible synthesis, underweighting evidence-precision and provenance alignment, supporting the case for a multi-metric evaluation paradigm.

Methodological and Practical Implications

The primary evaluation results establish that, for candidate-constrained RAG, upstream passage boundary design (rule-based vs. embedding-driven) is less important than late-stage, metric-aware allocation of answer sentence candidates. Maintaining stable, overlapping rule-based chunks preserves local context and minimizes fragmentation, while late neural reranking allows sentence selection to focus on semantic alignment with the query and gold-standard nuggets.

Strong retrieval enrichment (multi-query expansion, PRF, and RRF) remains critical even within fixed candidate sets, echoing findings from Query2Doc [wang2023query2doc], relevance-model [lavrenko2001relevance], and RRF [cormack2009rrf] studies. The architecture allows for modular ablation across retrieval, chunking, and neural reranking for diagnosis and optimization.

Evaluation protocol design is pivotal. The divergence between LLM-as-judge and gold-answer/nugget-based protocols demonstrates that aggregate metrics should be interpreted in the context of their metric biases, and further motivates the use of provenance-aware, multi-factor RAG assessment strategies.

Limitations and Future Directions

Quantitative differences between closely matched variants should be considered with caution given the 47-query slice and fixed ten-document pool, and broader generalization may require extension to larger and variable candidate sets.

Future research should prioritize:

• Enhanced late-stage evidence reranking tuned to specific evaluation targets.
• More granular diagnostics of where neural reranking improves (or harms) nugget coverage and citation precision.
• Metric-aware generator conditioning and more robust candidate chunking hyperparameter tuning.
• Expanding evaluation with human-authored reference answers and further provenance-tracing analysis.

Conclusion

This study demonstrates that, within a controlled candidate-constrained RAG setting, maintaining deterministic, overlap-preserving passage segmentation combined with strong retrieval enrichment and late-stage MiniLM reranking yields superior semantic and provenance-aligned answer generation, as assessed by diverse organizer metrics. Late neural sentence allocation, rather than complex chunking, is the most effective intervention. The strong evidence for multi-metric evaluation nuances received wisdom regarding measurement protocols for RAG system development.

Figure 1: Overall pipeline used in the supplementary LongEval Task 4 experiments, covering candidate-constrained retrieval, answer generation, and cross-provider evaluation.