ChemRxiv Retrieval Benchmark

Updated 7 July 2026

The paper introduces two key retrieval formulations—one for ad-hoc paragraph ranking and another for QA-grounded retrieval using curated ChemRxiv preprints.
It employs advanced corpus construction with quality filtering, LLM-generated synthetic queries, and domain-specific adaptations like ChemVocab token augmentation.
The benchmark utilizes metrics such as nDCG, MRR, and MAP to demonstrate significant improvements over baseline models for chemical literature search.

Searching arXiv for the cited benchmark and closely related chemistry retrieval papers. {"query":"(Kasmaee et al., 3 Aug 2025) ChEmbed Enhancing Chemical Literature Search Through Domain-Specific Text Embeddings", "max_results": 5} {"query":"ChemRxivQuest curated chemistry question-answer database extracted from ChemRxiv preprints (Amiri et al., 8 May 2025)", "max_results": 5} {"query":"Benchmarking Retrieval-Augmented Generation for Chemistry (Zhong et al., 12 May 2025) ChemRAG-Bench", "max_results": 5} The ChemRxiv Retrieval Benchmark denotes a chemistry-specific evaluation setting built from ChemRxiv preprints for testing retrieval over current chemical literature. In the most explicit usage, introduced in "ChEmbed: Enhancing Chemical Literature Search Through Domain-Specific Text Embeddings," it is a standard ad-hoc retrieval task over held-out ChemRxiv paragraphs; related work in "ChemRxivQuest: A Curated Chemistry Question-Answer Database Extracted from ChemRxiv Preprints" describes a retrieval benchmark as the retrieval-oriented framing of validated question-answer pairs linked to source passages (Kasmaee et al., 3 Aug 2025, Amiri et al., 8 May 2025). Both formulations emphasize source-grounded retrieval from preprint literature rather than retrieval from general encyclopedic or textbook collections.

1. Scope and benchmark variants

The literature describes two closely related ChemRxiv-centered retrieval formulations. One is the paragraph retrieval benchmark introduced with ChEmbed, where the task is to rank held-out ChemRxiv paragraphs for synthetic natural-language queries. The other is the ChemRxivQuest retrieval benchmark, where validated chemistry questions are used to retrieve source passages extracted from ChemRxiv preprints.

Benchmark formulation	Query set	Retrieval collection
ChemRxiv Retrieval	5,000 held-out queries	69,457 held-out paragraphs
ChemRxivQuest retrieval benchmark	970 questions $\{q_i\}$	All 400-char passages with 100-char overlap extracted from 155 preprints

In the ChEmbed benchmark, each query has exactly one relevant passage and all other paragraphs are negatives. In ChemRxivQuest, each question $q_i$ has a set of one or more positive passage IDs $P_i$ , and often $|P_i|=1$ . This suggests two complementary evaluation regimes: one optimized for paragraph-level embedding retrieval, and one aligned with retrieval-augmented question answering over source-grounded passages (Kasmaee et al., 3 Aug 2025, Amiri et al., 8 May 2025).

2. Corpus construction for paragraph retrieval

The ChEmbed benchmark is built from ChemRxiv preprints under CC-BY and CC-BY-NC licenses, totaling around 30 k manuscripts. PDFs are processed with GROBID and segmented into paragraphs. Quality filtering follows S2ORC methodology: table and figure captions, conclusions, and funding notes are removed, and paragraphs with fewer than 50 words or unigram log-probability below $-20$ are discarded. The resulting corpus contains 139,057 paragraphs in the training split, drawn from CC-BY material, and 69,457 paragraphs in the held-out evaluation split, drawn from CC-BY-NC material (Kasmaee et al., 3 Aug 2025).

Synthetic query-passage pairs are then generated to create realistic supervision for contrastive retrieval. The prompt design instructs LLMs to generate exactly one clear, chemistry-focused question answerable by the paragraph, while forbidding yes/no questions and questions phrased as “according to this paragraph.” The LLMs used for training data are o3-mini, gpt-4.1-nano, and gpt-4o-mini; the LLM used for evaluation data is Claude Sonnet 3.7, explicitly chosen to minimize generation bias. Paragraphs for which the LLM refused to generate a query, approximately 29 k, are removed. The final dataset comprises approximately 1.7 million synthetic query-passage pairs for training and 5,000 held-out queries on 69,457 passages for test. No separate dev set is explicitly reported; models are tuned on the synthetic data and directly evaluated on the held-out test set (Kasmaee et al., 3 Aug 2025).

3. QA-grounded passage retrieval in ChemRxivQuest

ChemRxivQuest starts from 155 ChemRxiv preprints evenly sampled, approximately 9 per category, across 17 chemistry subfields. OCR is performed with OLMocr at 300 DPI. The extracted text is cleaned by removing headers, footers, page numbers, references, and figures via regular-expression heuristics; 10% of documents are manually inspected to refine the regexes. The cleaned text is then split into overlapping passages of 400 characters with 100-character strides, and these passages later serve as the retrieval unit (Amiri et al., 8 May 2025).

Question-answer generation uses GPT-4o on each preprint’s cleaned text with temperature set to 0.2. The model is instructed to produce exactly three kinds of questions—conceptual, mechanistic, and applied—and each output must include a single well-formed question $Q$ , a concise answer $A$ , and a verbatim or nearly verbatim snippet $S$ from the source document supporting $A$ . This process generates 1,550 candidate QA triples, 10 per preprint. Answer verification is then performed against the 400-character passages using exact substring matching and approximate matching based on a partial Levenshtein ratio,

$\mathrm{similarity}(A,S)=\frac{|A|+|S|-d_{\mathrm{lev}}(A,S)}{|A|+|S|},$

with acceptance if

$q_i$ 0

A 400-character window is slid in 100-character steps over each preprint, the highest-scoring passage is recorded, and the passage is buffered by $q_i$ 1 characters to avoid truncating mid-sentence. Any QA pair failing both checks is discarded. A random 10% of retained pairs, approximately 97, is manually validated by a chemist; post-filtering accuracy is estimated at greater than 95%, and 970 QA pairs survive from the 1,550 candidates, a 62.6% retention rate (Amiri et al., 8 May 2025).

The validated dataset contains 970 QA pairs from 155 preprints. Its question-type distribution is Conceptual 231 (23.7%), Mechanistic 242 (24.9%), Applied 248 (25.5%), and Synthetic & Experimental 249 (25.7%). The 17 subfields are Materials Chemistry, Theoretical & Computational Chemistry, Inorganic Chemistry, Physical Chemistry, Organic Chemistry, Nanoscience, Biological & Medicinal Chemistry, Analytical Chemistry, Chemical Engineering & Industrial Chemistry, Catalysis, Energy Chemistry, Earth, Space & Environmental Chemistry, Polymer Science, Materials Science, Agriculture & Food Chemistry, Organometallic Chemistry, and Chemical Education. The retrieval task is then framed with query set $q_i$ 2 equal to the 970 questions, document collection $q_i$ 3 equal to all 400-character passages extracted from the 155 preprints, and gold labels given by the positive passage IDs that passed fuzzy-match verification. Suggested retrieval back ends are sparse retrieval via BM25, for example Lucene, and dense vector retrieval via FAISS using off-the-shelf embeddings. After top- $q_i$ 4 retrieval, a reader model may be applied to the top passages to extract or generate the final answer $q_i$ 5 (Amiri et al., 8 May 2025).

4. Retrieval task definitions and metrics

In the ChEmbed formulation, the task is standard ad-hoc retrieval: given a natural-language query $q_i$ 6, rank a fixed corpus $q_i$ 7 of paragraphs so that the gold passage $q_i$ 8 appears as high as possible. Because each query has exactly one relevant passage, the principal metrics can be written compactly. Precision at cutoff $q_i$ 9 is

$P_i$ 0

Recall at cutoff $P_i$ 1 is

$P_i$ 2

with $P_i$ 3 in this setup, and mean reciprocal rank is

$P_i$ 4

Mean Average Precision at cutoff $P_i$ 5 and normalized discounted cumulative gain at cutoff $P_i$ 6 are also reported, with nDCG@10 given by the average of per-query $P_i$ 7 values (Kasmaee et al., 3 Aug 2025).

ChemRxivQuest adopts the standard metrics typically used for retrieval-augmented QA. Recall@ $P_i$ 8 measures how often at least one correct passage appears in the top- $P_i$ 9; MRR averages the reciprocal of the best relevant rank; MAP averages average precision over all questions with respect to the positive passage set $|P_i|=1$ 0. Because the retrieval benchmark is passage-grounded and permits one or more positive passages per question, these metrics evaluate both early precision and robustness to multiple valid supporting contexts. The original ChemRxivQuest paper does not report empirical retrieval scores; instead, it suggests that future work should compare BM25 against dense FAISS retrieval, and no subfield-specific or question-type-specific retrieval performance is yet available (Amiri et al., 8 May 2025).

5. Baselines, ChEmbed, and measured performance

The ChEmbed paper compares a range of open-source and proprietary embedding baselines on ChemRxiv Retrieval, ranking them by nDCG@10. Selected reported results are as follows.

Model	Parameters	nDCG@10
all-MiniLM-L6-v2	22.7 M	0.674
e5-base	109 M	0.770
e5-large-v2	335 M	0.821
nomic-embed-text-v1	136.7 M	0.821
Qwen3-Embedding-4B	4 020 M	0.861
Qwen3-Embedding-8B	7 570 M	0.865
text-embedding-ada-002	—	0.770
cohere-embed-english-v3	—	0.781

ChEmbed itself starts from nomic-embed-text-v1, a 136.7 M-parameter encoder-only model with rotary positional embeddings, SwiGLU activations, FlashAttention, and support for up to 8192 tokens. Its tokenizer augmentation, termed “ChemVocab” in the paper, is derived from 2,083,502 unique IUPAC names from PubChem. A WordPiece model is trained, tokens already present in bert-base-uncased are removed, and the top 900 domain tokens are injected into bert-base-uncased’s 994 unused slots; new token embeddings are initialized from $|P_i|=1$ 1. Fine-tuning uses approximately 1.7 M query-passage pairs with an InfoNCE loss, cosine similarity, pure in-batch negatives, batch size 16,384 on 4 $|P_i|=1$ 2A100 40 GB GPUs via GradCache and mixed precision, maximum learning rate $|P_i|=1$ 3, and linear warm-up over 5% of steps. Training uses a 2048-token context length, dynamically scalable to 8192 at inference (Kasmaee et al., 3 Aug 2025).

On ChemRxiv Retrieval, ChEmbed_vanilla reports MAP@10 = 0.878, MRR@10 = 0.878, and nDCG@10 = 0.902, corresponding to a gain of 8.1 percentage points over nomic-v1. ChEmbed_progressive reports MAP@10 = 0.889, MRR@10 = 0.889, and nDCG@10 = 0.911, a gain of 9.0 percentage points over nomic-v1. The tokenizer-adaptation ablation further reports nDCG@10 values of 0.902 for vanilla, 0.895 for full, 0.903 for plug, and 0.911 for progressive, indicating that progressive unfreezing best leverages domain tokens while preserving general-domain knowledge. Checkpoint-level tracking shows steady gain on ChemRxiv Retrieval as fine-tuning proceeds but continuous decline on ChemTEB Retrieval, which the authors interpret as evidence of distributional mismatch. The paper also reports a speed of 189 samples/sec on an NVIDIA A10 24 GB and characterizes ChEmbed as an order of magnitude smaller, at 136.7 M parameters, and 10–30× faster than the best large models in the 4 B–8 B range, while achieving state-of-the-art chemical literature retrieval accuracy with nDCG@10 = 0.911 (Kasmaee et al., 3 Aug 2025).

The ChemRxivQuest retrieval benchmark explicitly identifies several limitations. Automatic QA generation can introduce plausible but unsupported claims; fuzzy matching mitigates but does not eliminate semantic errors. Subfield imbalance remains present, with Organic and Materials Chemistry overrepresented and areas such as Chemical Education and Agriculture & Food Chemistry comparatively small. The source material consists of preprints, so some content may later be revised or retracted. Verification granularity is also limited, since the current fuzzy-token matching does not capture nuanced semantic drift. Future directions stated in the paper include human-expert validation, stratified sampling to rebalance under-represented subfields, expansion to peer-reviewed journals including Europe PMC, PubChem, and ACS open access, semantic verification using transformer-based similarity such as Sentence-BERT, and community-driven versioning through GitHub and open annotation (Amiri et al., 8 May 2025).

Within the broader chemistry-RAG landscape, ChemRAG-Bench provides a complementary benchmark rather than a ChemRxiv-only one. Its ChemRAG Corpus combines PubChem, PubMed Abstracts, USPTO Patents, Semantic Scholar Full-Text, OpenStax Textbooks, and Wikipedia into approximately 101 M snippets. ChemRAG-Toolkit supports BM25, Contriever, SPECTER, E5, and Reciprocal Rank Fusion, and the benchmark unifies four datasets comprising 1,932 QA pairs across six task categories. Reported experiments show an average relative improvement of 17.4% for retrieval-augmented generation over direct inference, with the combined ChemRAG Corpus yielding more robust retrieval across diverse tasks, Contriever identified as the most stable single retriever, RRF recommended for robustness, and top- $|P_i|=1$ 4 reported as a reliable default passage depth (Zhong et al., 12 May 2025).

ChemRxiv-centered benchmarks are explicitly envisioned for several applications. ChemRxivQuest is proposed as a testbed for chemistry-specific retrieval systems, including BM25, dense retrieval, and hybrid RAG; as a fine-tuning corpus for domain-adapted LLMs such as “ChemGPT”; as a benchmark for scientific QA with slice analyses by subfield or question type; as an embedding evaluation suite comparing systems such as SciBERT, Specter, E $|P_i|=1$ 5, or OpenAI embeddings on Recall@ $|P_i|=1$ 6 or nDCG; and as a basis for educational tools and longitudinal studies of knowledge drift across successive LLM versions (Amiri et al., 8 May 2025). A plausible implication is that the ChemRxiv Retrieval Benchmark is valuable not only as a standalone retrieval test, but also as a tightly domain-matched substrate for chemistry-aware search engines, retrieval-augmented generation systems, and domain-adapted representation learning.