DSBio: Bioinformatics Benchmark in DSGym
- DSBio is a bioinformatics benchmark featuring 90 expert-derived tasks designed to assess data science agents’ scientific grounding in biomedical analyses.
- It integrates specialized data modalities—such as single-cell biology, genetics, and spatial transcriptomics—to challenge realistic and precise data interpretation.
- Rigorous quality control through expert review ensures tasks require bottom-up reasoning and exact-match answers beyond superficial retrieval.
Searching arXiv for the primary DSBio benchmark paper and closely related benchmark/ontology work to ground the article in current literature. DSBio is the bioinformatics component of DSGym-Tasks: a curated benchmark of 90 scientific data-science questions grounded in real biomedical research datasets and top-tier literature. It is described as “an expert-derived scientific analysis suite of 90 bioinformatics tasks grounded in academic literature,” and its purpose is to test whether data science agents can reason correctly in realistic scientific settings rather than only on generic tabular analysis. In the DSGym formulation, DSBio targets scientific-domain grounding by requiring agents to interpret unfamiliar biomedical data modalities, understand domain-specific terminology and conventions, use specialized bioinformatics libraries and workflows, and perform bottom-up reasoning from raw data rather than superficial retrieval (Nie et al., 22 Jan 2026).
1. Conceptual scope and role within DSGym
DSBio is not a standalone execution framework. Its role is defined within a larger layered system: DSGym provides the infrastructure and environment, DSGym-Tasks is the benchmark suite, and DSBio is the domain-specific bioinformatics task collection inside that suite. The benchmark therefore inherits DSGym’s standardized task abstraction, isolated execution environments, default agent interface, and reproducible evaluation, while specializing the task content to bioinformatics problems (Nie et al., 22 Jan 2026).
The benchmark was created in response to a specific deficiency in prior data-science evaluation. The DSGym paper argues that existing benchmarks overrepresent generic statistics and tabular tasks and do not sufficiently test domain-specific scientific workflows. DSBio was introduced to fill that gap with tasks that require biological context, specialized data modalities such as .h5ad, careful preprocessing, and correct scientific interpretation. A central claim is that frontier models perform comparatively well on generic analysis yet remain weak on scientific grounding, and DSBio is designed to expose that weakness (Nie et al., 22 Jan 2026).
A common misconception is to treat DSBio as a pure reproduction benchmark. The benchmark does include reproduction of reported findings, but it also contains expert-derived follow-up analyses requiring bottom-up reasoning from raw data, statistical modeling, multi-dataset integration, and minimal reliance on pre-wrapped domain-specific workflows. In that sense, DSBio is intended to evaluate scientific analysis behavior rather than mere extraction of answers already present in publications (Nie et al., 22 Jan 2026).
2. Corpus composition and benchmark construction
The released benchmark contains 90 tasks total. Its domain distribution is explicitly reported as follows (Nie et al., 22 Jan 2026):
| Domain | Tasks |
|---|---|
| Single-cell biology | 56/90 |
| Genetics | 21/90 |
| Spatial transcriptomics | 13/90 |
The construction process began with eight papers spanning single-cell omics, spatial omics, multi-omics integration, and human genetics. Papers were selected only if they had publicly available datasets and the data size was manageable in the sandbox environment. This selection criterion tied the benchmark to executable analysis rather than to literature reading alone (Nie et al., 22 Jan 2026).
Task generation followed two complementary modes. The first mode reproduced reported findings: the authors identified conclusive claims or quantitative results in the original paper and transformed them into executable questions. A task qualified only if it could be answered purely from the dataset, did not require visual inspection of figures, and had a deterministic numerical or factual output. The second mode consisted of expert-derived follow-up analyses, written after domain experts conducted deep exploratory analyses in Jupyter notebooks. These tasks were designed to require bottom-up reasoning from raw data, statistical modeling, and multi-dataset integration, thereby extending the benchmark beyond strict replication (Nie et al., 22 Jan 2026).
Quality control was deliberately iterative. A primary expert drafted the task and produced a Gold Notebook solution; an independent expert then reviewed task quality and difficulty, attempted to solve the task from the prompt and data alone, and compared results. Tasks were accepted only when the solutions matched and the task was judged sufficiently deep; otherwise they were discarded or refined and re-reviewed. This review protocol was intended to eliminate tasks that were too simple, ambiguous, or non-deterministic (Nie et al., 22 Jan 2026).
3. Task formalization, execution model, and evaluation protocol
Within DSGym, every task is represented as a tuple
where denotes the data files required for execution, the prompt or query, the evaluation metric, and metadata such as task category, domain label, and tags. This representation allows DSBio tasks to be evaluated in the same standardized environment as general analysis and prediction tasks, while preserving their domain-specific content (Nie et al., 22 Jan 2026).
The evaluation setup for DSBio uses the default CodeAct-like agent, fixes temperature at , and disables all tools. For analysis tasks such as DSBio, the reported metric is exact-match accuracy with slight numerical tolerance. This protocol emphasizes whether an agent can derive a correct, data-grounded answer under a controlled interface, rather than whether it can invoke an external toolchain or recover from a highly interactive environment (Nie et al., 22 Jan 2026).
The appendix examples illustrate the benchmark’s analytic style. Reported tasks include identifying the largest co-expression module in endothelial cells, estimating the fraction of response eQTL-caQTL pairs showing chromatin QTL activity, and finding the cell type with highest correlation with a spatial cluster. These examples show that DSBio is oriented toward exact, data-grounded answers coupled to biological interpretation and structured analytical reasoning, not toward free-form essay generation (Nie et al., 22 Jan 2026).
4. Empirical performance and characteristic failure modes
The benchmark reports the following overall accuracies for frontier and smaller models under the standardized DSBio evaluation protocol (Nie et al., 22 Jan 2026):
| Model | Accuracy |
|---|---|
| Kimi K2 Instruct | 43.33% |
| Claude Sonnet 4.5 | 42.22% |
| DeepSeek-v3.1 | 40.00% |
| Qwen3 235B Instruct | 38.89% |
| GPT-5.1 (high) | 38.89% |
| Claude Sonnet 4 | 36.67% |
| Qwen3-Coder 480B | 34.44% |
| GPT-4o | 33.33% |
| GPT-OSS-120B | 25.56% |
| Qwen3-4B-Instruct | 6.67% |
| Qwen2.5-7B-Instruct | 4.44% |
Subdomain results indicate heterogeneous difficulty. In single-cell biology, the best reported score is 44.83% by Kimi K2 Instruct. In genetics, the best score is 42.86%, achieved by Kimi K2 Instruct and Qwen3 235B Instruct. In spatial transcriptomics, the appendix reports a best value of 45.45% in some cases, with Claude Sonnet 4.5 and GPT-4o tied there, while GPT-5.1 is reported as “none” for the best slot in the cited breakdown. The paper’s main conclusion is that DSBio performance is substantially lower than performance on general data analysis benchmarks (Nie et al., 22 Jan 2026).
The dominant error category is not generic coding failure but domain-grounding error. Across sampled failed trajectories, 85–96% of failures are attributed to this class. Reported examples include misreading biological metadata, using the wrong cell-type labels, mishandling sparse or high-dimensional biological data, and choosing inappropriate analysis procedures and libraries. This result is central to the benchmark’s interpretation: the principal obstacle is scientific grounding in the target domain, not merely code synthesis or surface-level reasoning (Nie et al., 22 Jan 2026).
DSBio also functions as a downstream test for training effects within DSGym. The paper reports that Qwen3-4B-DSGym-SFT-2k reaches 21.11% on DSBio, compared with 6.67% for the base Qwen3-4B-Instruct. The demonstrated training corpus was built from general analysis tasks, yet some behavior transferred to DSBio. This suggests that execution-grounded analysis training can improve scientific reasoning, although the resulting absolute performance remains limited (Nie et al., 22 Jan 2026).
5. Relation to adjacent biomedical benchmarks and semantic infrastructures
A useful comparison is BioDSA-1K, a benchmark for biomedical hypothesis validation that is positioned directly in the DSBio / data-science-for-biomedicine line of work. BioDSA-1K contains 1,029 scientific hypotheses and 1,177 analysis tasks / analysis plans, derived from 329 publications. Its structure is explicitly hypothesis-centric and evaluates agents along four axes: hypothesis decision accuracy, evidence alignment, reasoning correctness or analysis quality, and code executability. It also includes non-verifiable hypotheses, including 100 strictly non-verifiable hypotheses in a dedicated analysis, thereby testing whether an agent can recognize when available data are insufficient to support or refute a claim (2505.16100).
The contrast with DSBio is methodological rather than adversarial. DSBio evaluates exact, data-grounded scientific analysis tasks in bioinformatics settings, whereas BioDSA-1K organizes evaluation around structured scientific claims, supporting evidence, and analysis plans extracted from published studies. BioDSA-1K reports modest evidence alignment scores, around 0.20–0.25, and emphasizes that correct final decisions can coexist with incorrect or incomplete analytical pathways. This suggests a complementary perspective on trustworthy biomedical agents: DSBio stresses realistic bioinformatics workflows and domain-grounding, while BioDSA-1K expands evaluation toward explicit claim-evidence reasoning and the handling of non-verifiability (2505.16100).
A different but related infrastructure appears in the Anthology of Biosurveillance Diseases (ABD), which was built as a practical, globally applicable disease ontology for biosurveillance. ABD includes infectious diseases of biosurveillance relevance and syndromic categories, and it is implemented as a SQL database with a Django application overlay, a REST API built using Django REST Framework, and exports in JSON, XML, and RDF/XML compatible with OWL. Its design addresses synonym resolution, multi-resolution organism associations, syndromic categories, and multilingual extensibility, thereby providing a machine-readable disease vocabulary intended to connect surveillance resources, disease models, and analytic systems under a shared semantic framework (Daughton et al., 2016).
Although ABD is not a benchmark, it clarifies an adjacent requirement for DSBio-style work. Real scientific and biosurveillance workflows often need standardized descriptions of diseases, organisms, transmission, and related metadata across heterogeneous resources. A plausible implication is that benchmarked bioinformatics agents and curated biomedical task suites may increasingly rely on such semantic normalization layers when tasks involve cross-system integration rather than isolated dataset analysis (Daughton et al., 2016).
6. Limitations, interpretation, and future directions
The current DSBio release is explicitly limited. The paper notes that the scientific-domain component presently focuses on bioinformatics only. It excludes visualization-centric tasks and more open-ended exploratory tasks, and the authors propose extending the framework to additional scientific domains such as chemistry, materials science, geoscience, astronomy, and economics (Nie et al., 22 Jan 2026).
These limitations matter for interpretation. DSBio is designed to avoid shortcut-solvable tasks and to favor determinism, which makes it suitable for controlled benchmarking but also narrows the range of scientific activity represented. It does not aim to cover the full space of scientific inquiry, full protocol semantics, or unconstrained exploratory analysis. This suggests that DSBio should be read as a rigorous slice of scientific data-science evaluation rather than as an exhaustive model of bioinformatics practice (Nie et al., 22 Jan 2026).
The future directions proposed in the paper are correspondingly technical. The authors argue that improving performance on DSBio will likely require better domain-adaptive learning, stronger scientific tool abstractions, and possibly continued pretraining or finetuning on scientific corpora and verified traces. Given the observed dominance of domain-grounding errors, the benchmark’s significance lies less in ranking models by a few percentage points than in operationalizing a failure mode that generic data-analysis benchmarks often miss. In that sense, DSBio serves both as an evaluation instrument and as a specification of what remains difficult for data science agents in realistic bioinformatics settings (Nie et al., 22 Jan 2026).