SciFullBench: Full-Text Scientific Retrieval

• SciFullBench is a full-text, citation-based benchmark designed for realistic paper-to-paper retrieval using full-context queries and segmented document representations over 40K papers.
• Its corpus construction employs rigorous leakage control and citation-graph-derived labels to prevent metadata cues and ensure authentic evaluation.
• The benchmark’s design, including PRISM’s multi-aspect query optimization and rank fusion, demonstrates the critical role of document segmentation and multi-vector aggregation in retrieval performance.

Searching arXiv for the SciFullBench benchmark and closely related scientific retrieval benchmarks. SciFullBench is a full-text, citation-based benchmark for scientific paper-to-paper retrieval in realistic large-corpus settings, introduced alongside "PRISM: Fine-Grained Paper-to-Paper Retrieval with Multi-Aspect-Aware Query Optimization" (Park et al., 14 Jul 2025). It is designed around full-paper queries, full-paper candidates, segmented full-text representations, and citation-graph-derived relevance labels, with evaluation performed at the document level over a corpus of 40,782 papers. The benchmark targets ML/NLP scientific search workflows such as retrieving related work for a given paper, and it should be distinguished from the quantum-computing full-stack benchmarking framework "Scalable Full-Stack Benchmarks for Quantum Computers" (Hines et al., 2023) and the multilingual code-generation benchmark "FullStack Bench: Evaluating LLMs as Full Stack Coders" (Bytedance-Seed-Foundation-Code-Team et al., 2024).

1. Motivation and benchmark scope

SciFullBench was created to address several limitations of existing scientific retrieval benchmarks. Prior work such as SPECTER, SciNCL, SciDocs, SciRepEval, MDCR, CSFCube, and RELISH largely evaluates models using abstracts only or very short contexts, even though real scientific search often depends on methodological details, experimental setups, or problem framing that appear only in the full text (Park et al., 14 Jul 2025). Other datasets with full texts, such as FUTEX, use very small candidate sets, for example 5 candidates per query, which is unsuitable for large-scale information retrieval evaluation. Still others, including SciMMIR and CSFCube, target different granularities or multimodal settings rather than full-paper document retrieval.

The benchmark is explicitly positioned as a full-context, large-corpus benchmark for scientific retrieval in ML/NLP domains, with full-paper queries and full-paper candidates. Its central task is document-to-document retrieval: given a query paper $P$, retrieve a ranked list of candidate papers $D \in C$ from a large corpus $C$ such that the returned papers are relevant to $P$. In contrast to existing studies that use $P_{\text{abstract}}$, SciFullBench uses the complete paper $P$ for query formulation and corpus construction.

A common misconception is that adding full text alone is sufficient. The benchmark’s own analyses indicate that full-context retrieval is not simply a longer-input version of abstract retrieval: representation granularity, segmentation, and ranking fusion materially affect performance. This suggests that SciFullBench is designed not merely as a larger corpus, but as a testbed for multi-aspect retrieval over long scientific documents.

2. Corpus construction and leakage control

SciFullBench is organized around a query set and a retrieval corpus. The query papers come from ICLR 2024 and 2025, NeurIPS 2023 and 2024, ACL 2023 and 2024, and EMNLP 2023 and 2024. ACL papers are sourced from the ACL Anthology website, while ICLR and NeurIPS papers are sourced from the OpenReview API; for some ICLR 2024 and NeurIPS 2023 submissions, full PDFs are taken from SEA to augment OpenReview (Park et al., 14 Jul 2025).

The retrieval corpus is constructed from the arXiv metadata dump up to January 2025, filtered to identifiers corresponding effectively to 2020–2025 and to the subject classes

$$\text{cs.AI, cs.LG, cs.CL, cs.CV, cs.NE, cs.IR, cs.DS, cs.CC, cs.DL, cs.HC, cs.RO, cs.MM, cs.CG, cs.SY}.$$

Because the arXiv API restricts large-scale PDF downloads, Google Cloud APIs are used to fetch the full arXiv PDF dump and then match PDFs to arXiv IDs from metadata. The final corpus contains all gold relevant documents for all queries, plus additional surrounding arXiv papers, for a total of 40,782 papers.

Both query and candidate PDFs are parsed with AllenAI Science Parse. The parsed JSON includes title, abstract, main content as a list of sections with header and contents, and references with both reference entries and structured citation mentions. Since relevance is defined through citation links, leakage control is a central design constraint. The entire reference section is removed where possible; an auxiliary filtering algorithm uses the parsed reference titles to eliminate any section or header whose text contains at least one reference title, using case-insensitive matching. Documents with fewer than 4 parsed references are excluded because the filtering algorithm cannot operate reliably, and documents are also excluded if the removed “problematic passages” would amount to more than half of the total main-content list.

A second pass removes inline citation markers such as numeric citations and author-year citations. This pass uses citation mention spans from Science Parse together with approximately 10 regex patterns. Remaining occurrences of reference titles in the main content are also removed. The resulting design aims to prevent trivial recovery of the citation graph from explicit citation text, so that retrieval performance reflects document semantics rather than metadata leakage.

3. Full-text representations and segmentation

Each document in SciFullBench contains four primary textual representations: title, abstract, full_paper_text, and segments (Park et al., 14 Jul 2025). The full_paper_text is reconstructed by concatenating the cleaned section contents after reference and citation removal. Segmentation is performed with NLTK word tokenization, producing contiguous fixed-length chunks of the cleaned full text.

For a document $D$ with token sequence $(w_1,\ldots,w_{|D|})$ and chunk size $L$, segmentation is described as

$D \in C$0

where

$D \in C$1

The primary experimental configuration uses segments of up to 3,000 tokens, although 6,000-token segments are also examined in ablations. The chunked candidate corpus is written as

$D \in C$2

with each segment retaining a pointer to its source document.

This segmentation strategy is motivated by the length and heterogeneity of scientific papers. The benchmark description notes that papers can exceed 10K–100K tokens, so a single-vector full-document representation can both exceed model context limits and conflate distinct aspects such as motivation, methodology, and experiments. Segmenting into 3K-token chunks therefore enables multi-vector candidate representations and supports late-interaction or multi-vector retrieval strategies.

PRISM introduces aspect-aware query optimization on top of this benchmark structure. Its query optimization functions are

$D \in C$3

where each $D \in C$4 maps the full paper $D \in C$5 to an aspect-specific query $D \in C$6. The final query set is

$D \in C$7

SciFullBench itself does not require this formalism, but its provision of segmented full-text candidates is what enables this type of multi-aspect retrieval.

4. Relevance labels, benchmark splits, and statistics

SciFullBench defines relevance through two citation-derived relations. For each query paper $D \in C$8, the reference-based relevant set is

$D \in C$9

and the citation-based relevant set is

$C$0

Citation-based labels are obtained from the Semantic Scholar API by gathering titles of papers that cite the potential query paper and matching those titles case-insensitively against arXiv metadata. Reference-based labels are obtained from Science Parse reference extraction and the same title-matching procedure. Relevance is binary rather than graded.

Query papers are retained only if at least 10 gold candidates remain in the filtered corpus for each relation type. For each venue-relation split, 400 query papers are randomly sampled, yielding $C$1 query papers in total (Park et al., 14 Jul 2025). No explicit train/validation/test partition is defined at the query level; the benchmark is used as an evaluation suite.

Split	Avg. tokens per paper	Avg. candidates per sample
ICLR-References	9402.35	21.74
ICLR-Citations	8183.37	42.25
NeurIPS-References	11184.39	24.58
NeurIPS-Citations	9035.43	40.37
EMNLP-References	5908.36	19.64
EMNLP-Citations	6226.45	39.24
ACL-References	6190.06	18.05
ACL-Citations	6614.6	33.89

At the corpus level, SciFullBench contains 40,782 papers, an average of 200.39 tokens per abstract, an average of 6987.67 tokens per paper, 115,004 total segments, and an average of 2479.90 tokens per segment. The enforced minimum number of gold candidates per query is 10 for both reference and citation splits. The resulting relevance sets are therefore substantially richer than the small candidate pools typical of earlier benchmarks.

5. Evaluation protocol and retrieval settings

The evaluation setting always uses the full SciFullBench corpus as the candidate pool for the relevant split. The benchmark supports four common retrieval configurations: abstract-to-abstract (A2A), full-to-full (F2F), abstract-to-chunk (A2C), and full-to-chunk (F2C) (Park et al., 14 Jul 2025). Here, A denotes abstract, F full paper, and C chunked full text. When full papers are encoded as single vectors, they are truncated to the model’s context limit.

In the reported experiments, most models, including SciNCL, SPECTER2, Jina-Embeddings, and text-embedding-3-small, use Euclidean distance between embeddings, with retrieval implemented as minimum $C$2 search. SciMult uses maximum inner product. Approximate nearest-neighbor retrieval is implemented with FAISS, specifically FlatL2 or FlatIP indexes.

The benchmark focuses on Recall@K and MRR@K. For a query $C$3 with relevant set $C$4,

$C$5

If $C$6 is the rank of the first relevant document within the top $C$7, then

$C$8

MRR@K is the mean of RR@K over queries. The paper reports metrics such as Recall@50, Recall@100, Recall@200, Recall@300, and MRR@50.

At rank-fusion level, PRISM combines ranked lists from aspect-specific queries using Reciprocal Rank Fusion: $C$9 with $P$0 in the experiments. Although RRF is part of the PRISM method rather than the benchmark definition, its use is tightly coupled to SciFullBench’s segmented full-text and multi-aspect design.

6. Empirical behavior, benchmark difficulty, and limitations

SciFullBench is used to evaluate both domain-specific abstract retrievers and domain-agnostic long-context embedding models. The benchmark’s main findings indicate that abstract-only state-of-the-art scientific encoders remain competitive in A2A settings, but aspect-aware retrieval over segmented full text can outperform them. The paper reports that PRISM with Jina-Embeddings or text-embedding-3-small outperforms all domain-specific baselines on SciFullBench, with an average improvement of 4.3% over existing retrieval baselines and detailed gains reported as 4.3–7% depending on configuration (Park et al., 14 Jul 2025).

Several ablations clarify what the benchmark measures. First, simply switching from abstracts to full text does not guarantee better retrieval: F2F, A2C, or F2C without aspect-aware query optimization yields limited improvements and can underperform A2A. Second, candidate granularity matters. The corpus granularity ablation shows that 3K-token chunking yields better Recall and MRR than abstract-only representations, single full-document embeddings, and coarser 6K-token chunking. Third, the benchmark appears sensitive to multi-aspect query decomposition: increasing the number of specialized query agents improves retrieval, and specialized multi-agent setups outperform a single agent generating all aspect queries.

Rank aggregation is another important design point. On ACL-Citations, naive multi-vector aggregation is reported as extremely poor, MaxSim is better but still much worse than RRF, and RRF achieves Recall@200 of approximately 41.54% compared with approximately 28.83% for MaxSim and approximately 4.55% for naive aggregation. This suggests that the benchmark’s multi-aspect, multi-segment structure is not well captured by simple embedding-level pooling.

SciFullBench also supports diversity analyses of retrieved sets. The reported metrics include cosine distance, centroid distance, and convex hull volume after 3D projection. PRISM retrieval produces higher cosine distance, larger centroid distance, and larger convex hull volume than A2A baselines, while t-SNE visualizations show broader coverage of embedding space and better overlap with ground-truth relevant clusters. A plausible implication is that, under citation-grounded relevance, broader semantic coverage can improve retrieval of heterogeneous but relevant related work.

The benchmark has several explicit limitations. Its domain coverage is restricted to ML, NLP, and closely related computer-science arXiv categories; other scientific fields are not represented. Segmentation is rule-based and fixed-length, which can split discourse at arbitrary boundaries and risk losing contextual coherence. Label construction depends on Semantic Scholar coverage, Science Parse quality, and title matching against arXiv metadata, so missing labels and parsing noise are possible. Relevance is binary rather than graded, and citation links do not encode the strength or type of topical relevance. The corpus size of approximately 40K candidates is large relative to prior scientific retrieval benchmarks but still smaller than real-world scholarly indexes. The released dataset is available on Hugging Face under the name SciFullBench: $P$1

Within the benchmark landscape, SciFullBench occupies a specific position: it is a full-text, citation-grounded evaluation suite for realistic paper-to-paper retrieval, not a quantum full-stack hardware benchmark and not a multilingual code-generation benchmark. Its distinctive combination of full-paper queries, leakage-controlled full-paper candidates, segmented representations, and large-corpus document-level evaluation makes it a specialized benchmark for studying retrieval methods that exploit the internal structure and multi-faceted content of scientific articles.