- The paper introduces DBench-Bio, a monthly updated benchmark that automatically converts post-release biology abstracts into filtered causal and mechanistic QA tasks across 12 biomedical domains.
- The paper finds that all evaluated models perform well below the maximum score, while agentic ReAct and workflow systems improve results more consistently than naive retrieval and reasoning-mode gains vary by model.
- The paper shows that high static-benchmark accuracy does not predict discovery performance, identifies mechanism errors, generic answers, refusals, and overconfidence as common failures, and highlights the need for process-level evaluation.
Motivation and problem statement
The paper addresses a specific evaluation gap: existing benchmarks for scientific knowledge discovery (Auto-Bench, ResearchBench, LLM-SRBench, HiSciBench) are built on static datasets, so a model's score may reflect memorization of training data rather than genuine discovery. The authors argue that a valid knowledge-discovery benchmark must satisfy temporal separation — the evaluated knowledge must post-date the model's release date — and that because LLMs are retrained and re-released on short cycles (e.g., GPT-5.2 one month after GPT-5.1), static benchmarks become obsolete almost immediately. Sustaining temporal separation therefore requires a benchmark that is both dynamic and automatically constructed, since manual curation cannot keep pace with model release cycles.
The DBench-Bio pipeline
DBench-Bio is a three-stage, fully automated pipeline:
- Data acquisition: abstracts are crawled from journals in the JCR "Biology & Biochemistry" category, restricted to Q1 quartile by Journal Impact Factor, and published strictly after the target model's release date. Abstracts are used rather than full texts because they densely summarize core findings with less experimental noise.
- QA extraction: an LLM (DeepSeek-V3.2-thinking) converts each abstract into one QA pair, where the question is a causal-inference or mechanistic-explanation hypothesis (e.g., "Does protein X regulate phenotype Y?") and the answer captures the paper's core conclusion as concise bullet points, deliberately avoiding fine-grained numerical details.
- QA filter: an LLM judge scores each pair on three 1–5 dimensions — relevance (alignment with the sub-domain), clarity (self-containedness; explicitly screening out context-dependent phrases like "based on the text" that would reduce the task to reading comprehension), and centrality (whether the pair targets the primary finding). Retention thresholds are strict: Relevance ≥ 4, Clarity = 5, Centrality = 5.
The benchmark covers 12 biomedical sub-domains (from biochemistry/molecular biology to mathematical/computational biology) and is updated monthly; the initial instantiation uses publications from December 2025 and January 2026.
A notable methodological claim is the validation of LLM annotation against human experts using the Alt-test [(Lefgoum et al., 5 Mar 2025) refers to this work via Calderon et al., ACL 2025]. Three experts scored 100 QA pairs; winning rates exceeded 0.5 for all metrics (statistically significant), and advantage probabilities reached 0.99 (relevance), 0.89 (clarity), 0.99 (centrality), and 0.86 for overall evaluation. This supports replacing human annotators with LLM judges throughout the pipeline.
Evaluation setup
The authors evaluate base LLMs (GPT-5.2, GPT-5, GPT-5-Mini, Gemini-3-Pro/Flash, DeepSeek-V3.2, Kimi-K2, GLM-4.6/4.7, Qwen3-Max, Baichuan-M2) in four configurations: base models with and without thinking modes, RAG-style tool use, ReAct agents, and a multi-agent workflow (Planner, Tool Caller, Reasoner, Reporter, Critic). Tool-using variants use GPT-5-Mini as backbone with PubMed search restricted to pre-cutoff literature to prevent leakage through retrieval. Scoring is LLM-as-a-judge on a 1–5 Likert scale, again validated via the Alt-test.
Main findings
Low absolute performance across all models. No model approaches the maximum score of 5 on either monthly snapshot, indicating that current frontier LLMs do not reliably derive biological findings that post-date their training cutoffs. This is the paper's central empirical result.
Divergent benefit of thinking modes. Explicit reasoning yields clear gains for some models (Kimi-K2, GPT-5 family) but negligible gains for others (GLM-4.6/4.7, DeepSeek-V3.2), suggesting the utility of structured reasoning is model-dependent rather than universal.
Limited value of naive retrieval augmentation. Single-tool RAG produced no significant improvement, which the authors attribute to retrieved information largely overlapping the model's internal knowledge. In contrast, agentic architectures (ReAct and Workflow) both improve performance, with a marginal gap between them — implying that integrating reasoning with tools matters more than the specific orchestration paradigm. On a 500-instance subset, agent methods built on GPT-5.2 outperform their backbone, and agent performance scales positively with backbone capability, positioning agent frameworks as amplifiers of intrinsic model capacity.
Domain difficulty is uneven. Mathematical & computational biology is consistently the hardest sub-domain for all models, pointing to persistent weaknesses in quantitative reasoning within discovery tasks.
Memorization does not transfer to discovery. Comparing MMLU-Pro (biology) with DBench-Bio reveals a sharp dissociation: thinking models exceed 90% accuracy on MMLU-Pro yet perform poorly on DBench-Bio. Model rankings also misalign between the two benchmarks — Gemini-3-Flash leads on MMLU-Pro but underperforms on DBench-Bio. The authors draw two implications: high static-benchmark scores may be inflated by contamination, and knowledge discovery is not acquired by scaling pre-training data alone, requiring specialized mechanisms instead. This positions DBench-Bio as more discriminative than static alternatives.
Failure taxonomy
Case analysis identifies four recurring failure modes:
- Mechanism error: the model proposes plausible mechanisms, none matching the actual finding (e.g., attributing brusatol's anti-gastric-cancer effect to NRF2/PI3K-AKT pathways rather than the P4HA2–glycolysis–histone lactylation–TTK cascade).
- Generic mechanism substitution: the model recites textbook-level associations (e.g., melatonin's antioxidant receptor signaling) instead of the experiment-specific results.
- Refusal to answer: the model declines outright, indicating retrieval failure or safety-triggered disengagement.
- Overconfident reasoning: notably observed in agentic settings, where the model bypasses available tool calls and hallucinates a confident, internally coherent but incorrect mechanism from prior knowledge.
The overconfident-reasoning case is particularly relevant for agent design: providing tools does not guarantee their use when the model's priors produce fluent, plausible-sounding answers.
Limitations and open questions
The paper concedes several constraints. Evaluation is outcome-based only: a model reaching the correct conclusion through flawed reasoning would still score highly, motivating process-level evaluation as future work. Only a representative subset of SOTA models was tested due to cost. The agent experiments use GPT-5-Mini as backbone for cost reasons, and the GPT-5.2-based comparison relies on a 500-instance subsample, leaving open whether full-scale results would hold. Quality filtering depends entirely on LLM judgment; while Alt-test validation is strong, it is based on only 100 QA pairs and three experts. Finally, the benchmark evaluates answering questions about discoveries already made by humans — it does not assess generating genuinely novel hypotheses beyond published literature, which remains an unaddressed evaluation target.
Conclusion
DBench-Bio contributes the first dynamic, fully automated framework for evaluating new-knowledge discovery, instantiated as a monthly-updated biology benchmark spanning 12 sub-domains. Its pipeline is domain-agnostic by construction — swapping the JCR category extends it to other sciences. Empirically, the benchmark exposes a substantial gap between models' mastery of established knowledge and their ability to derive findings outside their training distribution, along with a concrete taxonomy of failure modes. The central open question the paper leaves is how to build specialized training or architectural mechanisms that close this memorization-to-discovery gap, and how to evaluate the reasoning process rather than only its outcome.
