---
title: 'QuarkMedSearch: Medical Deep-Search Benchmark'
url: https://www.emergentmind.com/topics/quarkmedsearch-benchmark
type: topic
---

# QuarkMedSearch: Medical Deep-Search Benchmark

Searching arXiv for the specified QuarkMedSearch and QuarkMedBench papers to ground the encyclopedia entry.
QuarkMedSearch Benchmark is a human-verified evaluation set introduced as part of QuarkMedSearch, a medical deep-search system built on Tongyi DeepResearch for the Chinese medical domain. It is designed to measure long-horizon, tool-using, retrieval-dependent medical reasoning in an online environment rather than short-horizon reasoning over a fixed corpus. The benchmark contains \(N=140\) held-out questions, is stratified across six medical categories, and is paired with an evaluation protocol that measures answer correctness as well as search efficiency and completion behavior. Its design directly targets settings in which an agent must plan over multiple steps, invoke external tools, synthesize evidence across documents, and produce a single unambiguous answer supported by retrieval [2604.12867].

## 1. Origins and problem setting

QuarkMedSearch Benchmark was proposed in response to a domain-specific gap in medical deep search. The motivating claim is that medical deep search requires multi-step planning, precise cross-document synthesis, authoritative sources, and interpretable reasoning chains, while existing training data are general-domain, short-horizon, and English-only. In that setting, models either answer from parametric memory or fail at long-horizon retrieval, and existing evaluation benchmarks are described as focusing on short-horizon reasoning or fixed corpora, thereby insufficiently measuring agentic abilities in open medical environments [2604.12867].

The benchmark therefore occupies a distinct niche within medical LLM evaluation. It is not centered on standardized exam items, nor on fixed-document question answering, nor on open-ended conversational grading. Instead, it is constructed for deep search: a mode of evaluation in which the agent is expected to operate in an online Web setting, traverse multiple retrieval steps, and resolve questions that are intentionally difficult to answer from memory alone. This suggests that the benchmark is intended to probe the operational boundary between retrieval-augmented QA and fully agentic medical information seeking.

A useful contrast is provided by QuarkMedBench, which evaluates real-world medical LLM responses on 20,821 single-turn queries and 3,853 multi-turn sessions using automated fine-grained rubrics grounded in expert-audited scoring. QuarkMedBench emphasizes ecologically valid patient and professional queries, safety, coverage, and response quality, whereas QuarkMedSearch Benchmark emphasizes long-horizon deep search, explicit tool use, and retrieval-supported uniqueness of the final answer [2603.13691].

## 2. Construction principles and benchmark composition

The evaluation set consists of 140 human-verified questions. These questions are stratified across six categories to avoid imbalance, and they are held out at the entity level from all SFT and RL training data. The benchmark has no public dev split; evaluation is performed end-to-end on all 140 samples [2604.12867].

The benchmark’s category taxonomy is as follows:

| Category | Scope |
|---|---|
| Biomedical Fundamentals | Molecular mechanisms |
| Drugs & Medical Products | Chemicals, biologics, devices |
| Medical Research & Knowledge | Clinical trials, EBM standards |
| Diseases & Clinical Manifestations | Diagnosis, differential points |
| Clinical Procedures | Labs, surgeries, protocols |
| Medical Institutions | Hospitals, associations |

The verification criteria are explicitly manual and expert-driven. Each question must satisfy linguistic clarity, meaning no ambiguity or grammatical errors; a difficulty threshold, meaning it cannot be answered from parametric memory alone by experts or frontier LLMs; and ground-truth uniqueness, meaning it has a single unambiguous answer supported by retrieval [2604.12867].

These criteria are central to the benchmark’s identity. Linguistic clarity reduces ambiguity in scoring. The difficulty threshold targets retrieval dependence rather than latent memorization. Ground-truth uniqueness constrains the task to questions that admit a definite answer, making token-overlap and exact-match metrics meaningful. A plausible implication is that the benchmark is engineered to minimize the confounds that often arise when open-ended medical questions have multiple partially correct answers.

## 3. Agentic task model and execution environment

QuarkMedSearch Benchmark is embedded in a ReAct-style agent loop with alternating Thought \(\rightarrow\) Action \(\rightarrow\) Observation. In the system description, Thought \((\tau_t)\) denotes internal planning and sub-goal decomposition, Action \((a_t)\) is one of \(\{\text{Search}, \text{Visit}, \text{MedicalProfessionalSearch}, \text{LLMCheck}\}\), and Observation \((o_t)\) is a retrieved summary, webpage content, or an LLM-only answer judgment [2604.12867].

The tool ecosystem is correspondingly specialized. Search is a general Web search over top-\(K\) results; Visit performs full-text fetch and summarization; Medical Professional Search is a vertical medical engine emphasizing authority and timeliness; and LLM Check supplies frontier parametric-model answers without tools, serving as an in-loop verifier and reward signal. The benchmark therefore does not merely test final-answer generation; it tests whether an agent can orchestrate a heterogeneous tool stack in a medically constrained retrieval setting [2604.12867].

Queries are presented as textual prompts containing complex medical questions with constraints and multiple entities. The agent must emit a succession of Thought\(\rightarrow\)Action\(\rightarrow\)Observation steps until the final Answer action. Task types explicitly include multi-hop reasoning across at least 10 retrieval hops, tool invocation, reflection and verification, and context management through “Discard-all” when the input window exceeds 128 K tokens [2604.12867].

The evaluation environment is online rather than static. It uses Google Search API, in-house medical search, and live webpage fetching. A unified prompt template is used across baselines for fair comparison. The runtime limits are a maximum context length of 128 K tokens, a maximum of 128 tool calls, and temperature \(0.6\) [2604.12867].

This execution model is important because it defines what the benchmark regards as competence. Correctness is inseparable from planning, search depth, dynamic verification, and the ability to recover when context length becomes a constraint.

## 4. Metrics and scoring formalism

The benchmark evaluates both answer quality and search behavior. Let \(N\) be the number of test questions, \(\hat a_i\) the model’s answer for question \(i\), and \(a_i\) the ground truth. Exact Match is defined as

\[
EM = \frac{1}{N}\sum_{i=1}^N \mathbb{1}(\hat a_i = a_i).
\]

F1-Score is defined through token-level precision and recall:

\[
\text{precision} = \frac{|\text{Tokens}(\hat a_i)\cap \text{Tokens}(a_i)|}{|\text{Tokens}(\hat a_i)|}, \qquad
\text{recall} = \frac{|\text{Tokens}(\hat a_i)\cap \text{Tokens}(a_i)|}{|\text{Tokens}(a_i)|},
\]

\[
F1 = \frac{2 \cdot \text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}}.
\]

Termination Rate measures the proportion of queries completed within tool-call budget \(C_{\max}\):

\[
TR = \frac{1}{N}\sum_{i=1}^N \mathbb{1}(\text{calls}_i \le C_{\max}).
\]

Average Tool Calls (Correct) is defined as

\[
\overline{T}_{\text{correct}} =
\frac{1}{|\{i:\hat a_i = a_i\}|}\sum_{i:\hat a_i = a_i} \text{calls}_i.
\]

The protocol also reports Avg@3, where each question is sampled three times and the average score is used to smooth randomness [2604.12867].

The metric set reflects the benchmark’s dual objective. Exact Match and F1 evaluate answer correctness against a unique retrieval-supported ground truth, while Termination Rate and average correct tool calls quantify search efficiency and budget compliance. This suggests that the benchmark is not only measuring whether an agent can eventually solve a question, but whether it can do so within practical constraints of iterative online search.

## 5. Relationship to the data synthesis pipeline

Although the benchmark itself contains 140 questions, it is paired with a larger data-construction pipeline used to create long-horizon medical deep-search training data. QuarkMedSearch builds, in parallel, a large-scale synthetic training corpus consisting of tens of thousands of multi-hop tasks and the human-verified evaluation benchmark [2604.12867].

The four-phase pipeline begins with a large in-house Chinese medical knowledge graph spanning diseases, drugs, genes, anatomy, and related entities. Seed questions are formed by stratifying entities by graph frequency, sampling from the long tail, extracting 4–6-hop subgraphs, verbalizing them into QA pairs, and retaining only samples for which multiple closed-source LLMs agree on the answer. In the second phase, agentic multi-tool exploration expands each hop with real Web evidence, while LLMCheck is used after each generated question to test whether the answer is still recoverable from memory; only samples with at least 10 verified retrieval hops are retained. The third phase performs entity obfuscation, such as rewriting dates to “early 2020s” or replacing drug names with symptom descriptions, under multi-round self-checklists for Replacement Naturalness, Coverage Completeness, Difficulty Effectiveness, and Answer Uniqueness, with at least five iterations per sample. The fourth phase applies single-model multi-rollout consistency checks, cross-model validation, and data recovery for API failures [2604.12867].

These pipeline details matter for the benchmark because they explain its operational assumptions. The benchmark is not an isolated test set created independently of the training regime; it is the evaluation component of a broader framework whose explicit aim is to produce truly retrieval-dependent, multi-hop medical tasks. The paper reports that this process yields tens of thousands of such tasks with average tool calls of approximately 27, and states that entity obfuscation and multi-model verification ensure answer uniqueness and difficulty comparable to BrowseComp [2604.12867]. A plausible implication is that the benchmark inherits a difficulty profile calibrated to stress agentic retrieval rather than memorized factual recall.

## 6. Empirical behavior, comparisons, and scope

On the QuarkMedSearch Benchmark, the reported medical-domain results place QuarkMedSearch \((30\ \text{B-A3B})\) at \(55.71\%\), equal to Seed1.8 at \(55.71\%\), above Qwen3.5-Plus at \(51.42\%\), above Tongyi DeepResearch \((30\ \text{B-A3B})\) at \(40.71\%\), and below GLM-5 at \(58.57\%\). Medium-scale open-source baselines score \(43.57\%\) for Qwen3.5-35B-A3B and \(44.28\%\) for Qwen3.5-27B [2604.12867].

The accompanying interpretation in the source is that QuarkMedSearch achieves state-of-the-art performance among open-source models of comparable scale on the QuarkMedSearch Benchmark, and that on the dedicated medical benchmark it rivals closed-source giants while outperforming large-model Qwen3.5-Plus. The same work also reports strong competitiveness on general deep-search benchmarks, with QuarkMedSearch scoring \(47.0\) on BrowseComp-EN, \(57.6\) on BrowseComp-ZH, and \(81.0\) on Xbench-DeepSearch among 30 B-parameter open-source models [2604.12867].

Several ablation-style insights in the paper further clarify what the benchmark rewards. Phase I SFT at up to 32 K tokens is reported to improve QuarkMedSearch Benchmark performance from \(40.7\) to \(50.0\), while Phase II SFT at 32 K–128 K tokens yields a further \(+2.2\) points. RL from Verifiable Rewards is reported to reduce average tool calls and increase termination rate by \(+1.6\) to \(+2.2\) points. Context management through “Discard-all” is reported to yield \(+10.6\) points on BrowseComp-EN and \(+9.6\) points on BrowseComp-ZH [2604.12867]. These findings indicate that the benchmark is sensitive to long-horizon planning, search efficiency, and context-control mechanisms rather than only to answer extraction.

A common misconception would be to treat QuarkMedSearch Benchmark as interchangeable with broader medical QA benchmarks. The comparison with QuarkMedBench makes the distinction clearer. QuarkMedBench evaluates open-ended medical generation over genuine online healthcare logs, including colloquial and fragmented queries, wellness health, professional inquiry, and multi-turn sessions, with automated rubric-based scoring and expert-audited concordance of \(91.8\%\). QuarkMedSearch Benchmark instead uses a smaller, manually verified set of 140 retrieval-dependent questions with unique answers, judged by EM, F1, termination behavior, and tool-use efficiency [2603.13691]. The two benchmarks therefore target different layers of medical intelligence: one centered on real-world response quality in open-ended clinical and health communication, the other on long-horizon search agency in open Web medical environments.

The limitations stated for QuarkMedSearch include expansion to multilingual and multicenter medical Web environments, incorporation of structured domain ontologies for finer-grained reasoning, and broader benchmark coverage for long-answer synthesis, patient-simulated dialogue, and multimodal medical data [2604.12867]. This suggests that the current benchmark should be understood as a focused instrument for retrieval-intensive, short-answer deep search, not as an exhaustive evaluation of all medically relevant agent behaviors.

Source: https://www.emergentmind.com/topics/quarkmedsearch-benchmark