R2MED Benchmark for Clinical Reasoning

• R2MED benchmark is a medical IR testbed that evaluates systems’ ability to retrieve evidence through implicit reasoning, moving beyond surface-level text matching.
• It covers 876 queries across diverse clinical scenarios, including Q&A, clinical evidence, and case retrieval, emphasizing diagnostic inference and latent connections.
• The benchmark guides future research by highlighting the gap in current models, with advanced methods like GAR and LRM achieving up to 41.7 nDCG@10 yet leaving room for improvements.

R2MED is a reasoning-driven medical retrieval benchmark designed to evaluate and advance information retrieval (IR) systems capable of supporting complex clinical decision-making. Unlike prior medical IR benchmarks that focus primarily on lexical or shallow semantic similarity, R2MED targets the ability to retrieve documents based on implicit reasoning links, as commonly required in clinical workflows when physicians seek evidence and authoritative references aligned not with the surface form but with the inferred diagnosis or reasoning chain emerging from patient presentation.

1. Purpose and Distinctive Aims

R2MED addresses a key limitation of existing benchmarks: their reliance on direct query-document overlap. In practical clinical scenarios, relevant information for a query may only be connected to the query text via intermediate, often latent, reasoning steps. For example, the supporting literature for a diagnosis may not directly mention symptoms but instead substantiate a diagnostic or therapeutic conclusion drawn from those symptoms.

The primary objective of R2MED is to introduce a testbed that:

• Probes retrieval models for reasoning-centric tasks where relevance assumes implicit or latent connections.
• Fosters the development of retrieval systems that move beyond surface matching to support clinical evidence retrieval and case-based reasoning.
• Facilitates robust comparison of retrieval approaches under reasoning-heavy conditions, intentionally constructed to be challenging for current systems.

Formally, for query $q$ and document collection $\mathcal{D}$, R2MED’s annotation protocol defines the set of relevant documents as

$$\mathcal{D}_q^+ = \{D_{q,1}^+, D_{q,2}^+, ..., D_{q,m}^+\}$$

where relevance is mediated not merely by text overlap but by a latent answer or reasoning step $\mathcal{A}$ linking $q$ and $D_q^+$.

2. Benchmark Construction: Tasks and Scenarios

R2MED consists of 876 queries spanning three reasoning-intensive retrieval tasks, stratified across five clinical scenarios and twelve body systems. This design yields broad topical coverage and ensures diversity of reasoning demands, as confirmed by low inter-dataset Jaccard similarity in the diversity analysis.

a. Q&A Reference Retrieval

• Source: StackExchange (Biology, Bioinformatics, Medical Sciences).
• Challenge: Retrieve external references that substantiate the logic behind accepted answers. Relevant documents are those cited in the answer, with relevance relying on the answer's (possibly hidden) clinical or scientific logic rather than surface match to the original query.

b. Clinical Evidence Retrieval

• Source: MedXpertQA, MedQA, MedRBench.
• Scope: Spanning examination, diagnosis, and treatment questions; queries are often vignettes or clinical scenarios.
• Challenge: Documents are relevant if they support the decision or clinical reasoning indicated by the answer, requiring inference from query to the type of required evidence.

c. Clinical Case Retrieval

• Source: PubMed, IIYi clinical records.
• Purpose: Retrieve cases with the same diagnosis and supporting diagnostic logic, though the query may lack explicit diagnostic labels.
• Challenge: Demands diagnostic inference from the described case, then matching other cases with congruent reasoning paths.

3. Evaluation Protocol and Empirical Findings

R2MED employs nDCG@10 as its principal evaluation metric, measuring the ranking quality of the top ten returned documents with respect to expert-defined relevance.

A wide array of 15 retrieval systems have been benchmarked on R2MED, including:

• Sparse methods: BM25.
• Dense retrievers: Contriever, BGE, NV-Embed-v2, GritLM.
• Domain-specific models: BMRetriever, MedCPT.
• Proprietary systems: OpenAI, Voyage.
• Rerankers: MonoBERT, BGE-Reranker, RankLLaMA.
• Generation-augmented methods (GAR): HyDE, Query2Doc, LameR.
• Large reasoning models (LRMs) with agentic, search-enhanced frameworks: Search-o1, Search-R1.

Main Results

• The best vanilla dense retriever (NV-Embed-v2) achieves 31.4 nDCG@10.
• Sparse BM25 baseline achieves 15.1 nDCG@10.
• Advanced generation-augmented retrieval (GAR) methods, which use large LLMs to generate inferences or queries, substantially close the gap: Query2Doc (GPT-4o, NV-Embed) achieves 41.7 nDCG@10.
• Large Reasoning Models, such as o3-mini paired with NV-Embed-v2, set the upper bound at 41.4 nDCG@10.
• Classical reranking offers only marginal improvement for strong dense retrievers and sometimes negative gain; it provides more benefit when the base retriever is weak (BM25).

These results demonstrate that reasoning-augmented models and GAR pipelines provide significant advancement, yet the best performing models still retrieve less than half of the relevant reasoning-linked documents—exposing a large headroom for future research.

A strong correlation exists between intermediate answer accuracy and retrieval nDCG, underscoring the importance of integrating reasoning modules or stages within retrieval architectures.

4. Architectural and Methodological Considerations

R2MED’s challenge derives from explicit disentanglement of lexical matching from reasoning linkage. The construction protocol ensures that:

• Positive labels require reasoning chains, frequently invisible in the query surface form.
• Datasets minimize overlap and foster strong generalization pressure for retrieval systems.
• Tasks are structured to prevent shortcut learning via spurious correlations or superficial matching.

Retrieval systems evaluated on R2MED exhibit the following trends:

• Standard dense or sparse retrievers, even if medical-domain pretrained, struggle with reasoning-intensive queries.
• Reranking and generation-augmented retrieval are beneficial, particularly when using larger LLMs as generators or reasoners.
• Integration of intermediate inference (explicit answer generation) with retrieval forms the primary path for further improvements—as highlighted by GAR, LRM, and agentic-LLM frameworks.
• Direct fine-tuning on R2MED or similar data is not reported in the source, suggesting the benchmark is intended for out-of-distribution, zero-shot, or transfer evaluations.

5. Implications and Future Directions

R2MED exposes a substantial research gap between existing retrieval techniques and the needs of reasoning-driven clinical tasks. Its design encourages:

• Further integration of LLM-based or agentic reasoning modules into IR workflows.
• Exploration of joint modeling or co-training of reasoning and retrieval components.
• Development of novel rerankers or augmentation strategies explicitly tailored for medical reasoning, including architecture-level support for multi-step inference guidance.

The benchmark’s roadmap includes plans to extend to multimodal retrieval, integrating imaging data, which is essential for comprehensive medical reasoning. Other possible future directions involve optimizing for efficiency, understanding the interaction between computational cost and retrieval quality, and enhancing dataset diversity.

6. Data Availability and Usage

R2MED is publicly released for academic use and reproducibility:

• Data and code: https://github.com/R2MED/R2MED
• Supplementary hosting: https://huggingface.co/R2MED
• Format: JSONL files for queries (including text, intermediate answer, relevant documents), corpus, and binary relevance associations.
• Licensing: All source datasets used are open source or permissively licensed (CC-BY, MIT, etc.), and detailed licensing information is tabulated in the paper appendix.

Prerequisites include Python and standard NLP/IR tooling, with GPU recommended for running large-scale model inference. The benchmark is strictly intended for research and not for deployment in clinical practice.

7. Summary Table: Core Characteristics

Dimension	Description
Query count	876
Tasks	Q&A reference retrieval, clinical evidence retrieval, clinical case retrieval
Scenarios covered	5 (Q&A, exam, diagnosis, treatment, case similarity)
Organ systems	12
Evaluation metric	nDCG@10
Best result (as of 2025)	41.7 (GAR, Query2Doc/GPT-4o/NV-Embed-v2)
Public access	GitHub, HuggingFace

In summary, R2MED establishes a new standard for evaluating medical IR systems that must retrieve clinically relevant evidence via implicit reasoning, rather than explicit query-document similarity. Its design, results, and public availability provide a rigorous foundation for research and system development bridging information retrieval and clinical reasoning.